TEC programme data review and evaluation: summary report

Recommendations and evaluation options

In this section, we provide practical recommendations of steps that could be taken to work towards the achievement of the nine principles set out above. Some are high level recommendations on the approach to evaluation – the ‘how’ to measure, and others are recommendations on evaluation content – the ‘what’ to measure. These are cross-cutting recommendations drawn from all the research activities carried out as part of this project. There is no hierarchy as such, but some will require a larger scale/resource level (e.g. implementation) than others (e.g. quality of life impacts).

1. Adopt the RRRP approach to evaluation

We recommend that the Scottish Government formally adopts this approach and uses the nine principles set out above to guide evaluators. In many instances, this is what the workstreams are already doing. Formalising the approach provides academic and methodological credibility for work that is already being undertaken. As part of this we would recommend building economic analysis more routinely into evaluations. Finally, it would be useful to document the process of adopting and developing the RRRP as this will increase its legitimacy as well as contribute to a process of ongoing learning and development.

2. Prioritise implementation and the impact of the TEC programme

Much of the evaluation to date has focused on the technologies at the workstream level, rather than the programme as a whole. In future evaluations, and as the programme matures, it will be helpful to shift the emphasis on to the impact of the programme, despite the continued existence of data gaps at the workstream level.

Given that the programme focuses largely on implementation, this will be facilitated by a greater focus on evaluating this area. From the exploratory research in this study, we recommended exploring the following evaluation options.

A. Scoping research on implementation needs and the effectiveness of existing strategies.

Implementation has already been addressed as part of evaluations to date. Nonetheless, this has been identified as an area that requires ongoing investigation with a view to identifying the most significant barriers (and new barriers as they emerge) and crucially the most useful implementation strategies to address them as the programme shifts to national scale up.

B. Identification of an IS framework for future TEC rollouts.

It is recommended that the programme use, or adapt, an existing IS framework in future project planning and evaluation that can support both national and local implementation and contribute to the implementation of the new Digital Health and Care Strategy.

C. Evaluating the success of the implementation framework.

Should an implementation framework be selected, it will be important to evaluate its effectiveness. This would be highly beneficial for the programme and the future development of TEC and wider digital health and care priorities in Scotland. It would also be of wider academic, policy and practice interest, and contribute further to innovation in technology-enabled care.

3. Conduct a review of monitoring data capture and use evaluation to address gaps in the logic models

The way in which monitoring data is captured should be reviewed. Alongside this, a long-term plan for automating data collection should be developed as well as a short/medium-term plan for the interim. This would seek to address the inconsistent and time-consuming nature of existing data collection and explore how to better support this. We recommend increasing the use of technological solutions where possible. Assuming a better system can be developed, this should seek to extend what is being captured to include outcomes, economic, and implementation data.

4. Adopt a more consistent approach to evaluation, especially economic evaluation

In general, the programme could benefit from more consistency in how evaluations are approached. This would enable comparison but also improve the quality of evaluations by ensuring that good indicators and approaches to data collection are shared across projects. However, consistency is especially important for economic evaluations. This could be improved by adopting a standardised approach to the measurement of outcomes, wherever possible ^[3] . This could include the development of a bank of indicators and values that enable measurement and valuation to be undertaken more easily for multiple stakeholders and in the same way across different studies. This will make it easier to aggregate and compare results, or at a minimum read across different studies.

5. Develop guidance, in-house skills and a microsite

To improve consistency and ensure high quality approaches are used, the programme would benefit from developing bespoke guidance for the TEC programme. This would include templates for evaluation, surveys, interviews and guidance on economic analysis and other ‘off the shelf approaches’ that are consistent with the RRRP. This also applies to economic analysis.

To host these materials, we recommend the development of a microsite specific to the TEC programme. This would act as a ‘one stop’ resource hub for TEC evaluations. To manage this site and the guidance development, we recommend fostering some in-house evaluation skills within the TEC programme (e.g. a part-time staff role). This person could also support evaluations of delivery staff. Finally, they could be responsible for keeping abreast of the latest international research and disseminating this internally.

6. Conduct more multi-stakeholder research

Evaluations to date have tended to focus on a single stakeholder, even in contexts where there are multi-stakeholder impacts. We recommend taking a wider approach, which is again consistent with the RRRP. This is especially important for economic evaluations to ensure that benefits are being measured holistically. In our review, only one in ten studies adhered to this even though it is considered a best practice approach (Treasury, 2003). This is a further area where bespoke guidance may be beneficial.

7. Further research on future benefits, sustainability and mainstreaming

There are two types of sustainability relevant to the TEC programme: sustainability of the use of technology and sustainability of the outcomes (i.e. the benefit period for outcomes). The first is concerned with whether people cease to use a technology over the time and the second is concerned with whether the technology use continues to provide them with the initial benefit. Measuring this over time with users will be important especially for economic studies that want to project benefits into the future or understand the rate at which benefit declines over time.

An additional future impact is to understand the point at which a technology becomes mainstream. This would include some exploration of what constitutes ‘critical mass’ for each technology. This links to questions of when a technology to have been fully implemented and there is no further need for a TEC programme. This may also help identify the most promising set of activities to achieve mainstreaming.

8. More research on some secondary outcomes

Unsurprisingly, the evaluations have tended to focus on the primary outcome of a technology, often a clinical or care outcome. However, there are some gaps relating to secondary outcomes, which could benefit from some further research. The two we highlight here are quality of life/well-being impacts and health inequalities.