This section will help you to:
- Consider how progress and performance might be best measured and evaluated
Measuring progress and performance
Assessing progress and performance is fundamental to most evaluations, and, where possible, there is an understandable keenness to quantify and measure.
Before considering how to measure adaptation progress and performance for an intervention, consider what you are measuring against. Three useful lenses through which to view progress and performance are:
- Against the objectives of the intervention
- Against emerging understanding of good adaptation
- Against a baseline.
Questions to consider
- What were the objectives of the adaptation intervention and have these been achieved?
- Did these objectives remain relevant and appropriate?
- How can the principles of good adaptation be reflected in your evaluation criteria?
Use the following questions to help you think about a baseline and the selection of indicators. As mentioned before, ensure your investment in baseline data is proportionate.
- Will your baseline provide a clear picture of the type and nature of both climate and non-climate vulnerabilities and impacts? As climate change is unlikely to be the only issue under consideration, it is important to understand non-climate issues too. For example, understanding local economic conditions may help you to understand how community members’ contributions change as the project develops.
- For medium and long term interventions, are you able to distinguish the differences between your actions and changes in baseline conditions? For example, an agricultural adaptation project may enable crops to be harvested and achieve good market prices, yet due to changes in the baseline climate net yields may actually reduce.
- How often should you check your baseline to assess how conditions have changed? This will be influenced by the length of the proposed activity (both in terms of delivery and expected impacts), the timing of key decision points and the likely rate of change from the baseline.
- How will data availability change during the course of the project? Can new data be incorporated into your baseline?
- Critically, do you think your baseline will help you make better decisions during and after the intervention?
Measuring performance against objectives
Comparing outputs and outcomes of your project programme against the original purposes and objectives is one of the simplest ways of evaluating performance. This might include evaluating changes in behaviour and practice which support your adaptation objectives (adaptive capacity). This approach does not consider whether these objectives were right in the first place – an important point, as we are still learning how best to adapt to a changing climate. By developing an Adaptation Logic Model, you can examine the underlying assumptions and test the logic of the objectives as well as evaluating whether the objectives have been met.
Measuring performance against good adaptation
The characteristics of good adaptation can also be a useful way to measure performance. These can form the basis of evaluation criteria alongside the assessment of project-specific objectives. The 6 ‘guiding principles’ of good adaptation developed by Defra (2010) provide a useful starting point and emphasise that adaptation interventions should be:
Sustainable: Sustainable development will ensure that we are best placed both to minimise the threats posed by the impacts of climate change and to capitalise on potential opportunities presented by it.
Proportionate and integrated: Assessing climate risks should become business as usual and part of normal risk management. Action must relate to the level of risks and the desired outcomes, and will need to be taken at the most appropriate level and timescale.
Collaborative and open: Adapting to climate change is a challenge for the whole of our economy and society, and will require action from a range of individuals and organisations, within and across sectors working together.
Effective: Actions should be context specific, implementable, and enforceable. They should incorporate flexibility to adjust to a range of future climate scenarios, as well as socio-economic, technical and other changes.
Efficient: Actions should weigh costs, benefits and risks involved. Measures should be timed appropriately.
Equitable: The distributional consequences of different options should be considered to inform decision makers of the effects of the activity on the natural environment and different social groups, especially vulnerable ones, to ensure that individuals or groups do not bear a disproportionate share of those costs or residual risks.
You should also consider the degree of flexibility preserved or promoted through adaptive actions taken (Defra, 2010) as we are not yet able to evaluate whether or not our decisions are optimal or appropriate. Given this uncertainty, it is important to evaluate whether we have retained flexibility to ‘change direction’ at a later date.
Measuring performance against a baseline
A baseline is commonly used to assess performance against a snapshot of conditions before the start of an intervention, and uses a set of indicators relevant to your objectives. Progress is determined by comparing the baseline to the indicators at a set point during the implementation.
In the context of adaptation, establishing a baseline is difficult as it may move over time, especially for projects with a long lifetime. The underlying conditions of an intervention may change dramatically, irrespective of that intervention. For example, if a project is aiming to conserve a natural wetland environment, then what is natural if baseline conditions change?
It is important to challenge assumptions you may have about prevailing conditions and what these mean for your objectives (are we doing the right thing?). It also means a broader range of indicators may be required which capture possible future changes in climatic and socio-economic conditions. This can be potentially time consuming and expensive, hence it may be appropriate to use secondary data wherever possible.
