Cluster working: Monitoring and evaluation

Monitoring refers to setting targets and milestones to measure progress and achievement, and check whether the inputs are producing the planned outputs. Evaluation is not just about demonstrating eventual success; it also provides insights into why things don’t work (as learning from mistakes has equal value). Monitoring and evaluation are not about finding out about everything (which is intimidating) but are focused on the things that matter.

Monitoring and evaluation resources

There are a wealth of resources to guide monitoring and evaluation activities, including:

• Introductory guide to evaluation (Data Cymru): This guide will support your understanding of what evaluation is and why it is important for your projects, programmes and policies; give you the ‘basics’, so you understand why and when you might undertake evaluation; provide you with direction as to the approaches and processes you might use to undertake effective evaluation; and provide pointers to further guidance and support.
• The Magenta Book: Guidance for evaluation (HM Treasury): The Magenta Book is the recommended central government guidance on evaluation that sets out best practice for departments to follow; recommended by Data Cymru as the “go to” evaluation resource.
• Public Health Intervention Responsive Studies Teams (PHIRST), funded by the National Institute for Health and Care Research (NIHR), provide timely and accessible evaluations of public health interventions. The PHIRST scheme links up academic teams with local government and public health organisations to evaluate public health interventions that are already happening across the UK.

The PHW Evaluation community of practice, can provide access to training and resources to support monitoring and evaluation activities. Sign up by emailing [email protected]

System measures include:

• Healthy Days at Home Wales
• Primary Care Model for Wales key indicators – Under development

Logic models

Logic models can help sense-check the elements that must come together to successfully plan, deliver and evaluate a project. They can be integrated into project plans from the outset, or inform a bespoke monitoring and evaluation plan by teasing out the following:

• Inputs: The key things we need to invest/ have in place to support the activity.
• Activities: What we do with the inputs.
• Outputs: What we produce as a result of the activities.
• Outcomes: What our products will achieve for people or services (aims).
• Impacts: High-level, ultimate ambitions e.g., the quadruple aims of A healthier Wales.
• Barriers: What we may find difficult to influence or overcome (e.g., external factors).
• Assumptions: What we hope is already in place (supportive conditions, etc.).

A logic model tries to establish sequential links between the above elements, in multi-row table or diagram form. Sometimes they are easier to populate right-to-left, instead of left-to-right (starting with inputs). Advice on developing logic models include:

• Logic models (Data Cymru): Using logic models to plan, map and identify the activities and inputs that lead to results, and to understand desired changes and who would be accountable for them.
• Using logic models in evaluation (The Strategy Unit): This briefing has been prepared for NHS England, by the Strategy Unit, as part of a programme of training to support national and locally based evaluation of the Vanguard programme and sites.

Upscaling projects

Pilot projects serve to differentiate that which works from that which does not. Taking things that do work on a small scale to a larger scale (e.g., health board or all-Wales footprint) can be challenging; advice is contained with the following resources:

• Bevan Exemplars Programme (Bevan Commission): Bevan Commission innovation programmes support and enable change makers to transform health and care services for all citizens in Wales and beyond.
• The Dragons Heart Institute train healthcare’s future leaders and prepare teams to scale their solutions across their system and the wider world.
• Against the odds: Successfully scaling innovation in the NHS (Innovation Unit, undated): This report from the Innovation Unit and the Health Foundation calls for new approaches to scaling tried and tested health care innovations; it highlights the need to create the right conditions to spread these successfully across the NHS.
• Spreading and scaling up innovation and improvement (BMJ 2019;365:l2068): Disseminating innovation across the healthcare system is challenging but potentially achievable through different logics: mechanistic, ecological, and social.