You cannot measure impact if you do not know what impact you are trying to achieve. Good planning is therefore a prerequisite of good monitoring. Most planning systems encourage an inspirational vision with a measurable goal (defining the state of biodiversity) and objectives (often defining threats to be reduced), based on a conceptual model of the main issues and how they relate to each other. These elements of a strategy should include a theory of change, preferably elucidated through a set of results chains (or pathways to impact) to demonstrate the logic of how you expect a given action to lead to a specified output onto a specified outcome and then a specified impact (CMP, 2013). Key issues in setting goals and objectives include the identification and engagement of stakeholders in the planning process and the consideration of root causes in addition to proximate threats. But the key element for ensuring the roll up and aggregation of results and the eventual measurement of impact is that project goals and objectives must link to higher-level programme goals.

Therefore, the formulation of the goal of the project should reflect and share language with that of the programme. An example of such a goal hierarchy is demonstrated in Fig. 1 and Table 1.

Fig. 1.

Diagram showing (A) generic goal and objective hierarchy, (B) an example from the hypothetical elephant programme. If common indicators are used to monitor the same types of goal and objective at project level then data can be aggregated to demonstrate programme impact and organisational impact.

(0.5MB).

Project indicators will need to be chosen to measure the goals and objectives identified in step 1 following best practices for indicator development (e.g. CMP, 2013). This includes the need to ensure the indicators will provide measures of change in the timeframe relevant to the project. The conservation community has widely adopted the pressure-state-response-benefit (PSRB) model of interlinked indicators for biodiversity monitoring (Sparks et al., 2011). The relationship between the indicators (Fig. 2) ensures they create a more complete picture of what is happening in the project and why. The types of indicators in this model are:

• •

  Pressure indicators: monitoring the extent and intensity of the causes of biodiversity loss that responses aim to address (e.g. habitat loss, levels of exploitation or offtake, nitrogen deposition rate, invasive species, climate change impacts).

• •

  State indicators: analysing the condition and status of aspects of biodiversity (e.g. species abundance and diversity, habitat extent and quality).

• •

  Response indicators: measuring the implementation of policies or actions to prevent or reduce biodiversity loss (e.g. PA coverage, PA management effectiveness, area under sustainable management, funding).

• •

  Benefit indicators: quantifying the benefits that humans derive from biodiversity (e.g. livelihoods, fuelwood availability, populations of utilised species). Benefit indicators may also represent an impact if the project has a benefit-related goal (Brichieri-Colombi et al., 2018).

Fig. 2.

Relationship between PSRB indicators. Adapted from Sparks et al. (2011).

(0.13MB).

The key to ensuring the eventual measurement of aggregated impact is that at least a small sub-set of the indicators are chosen from a suite of common indicators used by all of the projects contributing to the same programme goals and objectives. For example, if the state indicator for the elephant programme is the number of elephants in Tanzania, the state indicator for the project in Ruaha National Park needs to be the number of elephants in the park. If a programme-level pressure indicator is the number of incidents of human-elephant conflict (HEC), the project indicator is the number of HEC incidents in Ruaha. Note that most of a project’s indicators will likely be unique to the project to monitor project-specific objectives that have no direct link to programme objectives (in this instance, for example, the Ruaha project has an indicator concerning boundary markers in place, which is not relevant to the programme). If the programme was developed after some of its constituent projects were established (a regular occurrence in NGOs, especially if new funding is secured), programme-level indicators may need to be identified retroactively, with relevant common indicators identified from existing project indicators.

In turn, the common indicators used to monitor a programme’s progress against its goals can be harmonised where appropriate with higher-level global indicators. Many government agencies and NGOs use common measures that are aggregated to show contributions towards the Aichi Biodiversity Targets of the Strategic Plan for Biodiversity 2011–2020 and the UN Sustainable Development Goals (SDGs) (Bubb, 2013; Tittensor et al., 2014). In our fictional example, the NGO goal of ‘no threatened mammal species go extinct in Africa’ could be measured over the long-term with the IUCN Red List Index (which is also an indicator of Aichi Target and SDG delivery). Recent studies comparing species’ status with predicted or calculated counterfactuals (Hoffmann et al., 2015; Akçakaya et al., 2018) demonstrate how conservation impact action can be estimated with such indicators (assuming monitoring is continued long enough to measure change). In addition, common indicators can monitor scaling up “at the project as well as portfolio level” (GEF IEO, 2019).

Sometimes it is useful to convert indicators into relative scores, such as 0–4 for no progress to high progress, to allow comparisons across sites (e.g., UNEP-WCMC, 2018). However, this can lose detail and the chance to aggregate raw data. The approach can tell you that two out of three projects have increasing elephants, but not that elephants have increased across the programme by x%.

Since some pressure and state indicators will show results only after several years, projects need to have key performance indicators (KPIs) against annual goals to provide some intermediary measures of progress. The KPIs monitoring the performance of projects and programmes will answer the question: how well did organisations deliver on what they set out to do? Some agencies use a performance KPI scoring system, such as WWF’s Conservation Achievement KPI (Stephenson and Reidhead, 2018), where the degree of delivery of an intermediary result for the year is scored. For example, if the Ruaha project staff trained one community in HEC mitigation last year instead of the three they had planned to train, they would score 33% against the intermediary result for HEC.

Project indicators need to be measured regularly. A suite of methods and protocols are available, especially for state and pressure monitoring (e.g. Wilson et al., 1996; Birdlife International, 2006; Sutherland, 2006). There are increasing numbers of technological solutions to enhance data collection, from camera traps to environmental DNA sampling, but tools need to be chosen that are appropriate to local capacity, budgets and needs (Stephenson, 2019). Data collection should follow best practices for ensuring robust sampling design, statistical power and consistent replication of methods (da Silva Dias et al., 2017; Stephenson, 2019).

National governments generally collect their own biodiversity data (at least the key elements of relevance to them) but, in some cases, also use global data sets managed by NGOs or international organisations for certain indicators (e.g. the IUCN Red List of Threatened Species managed by IUCN; the Living Planet Index managed by WWF and ZSL; Protected Planet managed by UNEP-WCMC). These data are then used to report against indicators at the national level in National Reports to CBD, the Convention on Biological Diversity (https://www.cbd.int/reports/), and at the global level through synthesis analyses (Secretariat of the Convention on Biological Diversity, 2014; Tittensor et al., 2014). Only by collecting data on common indicators that are relevant at the site, national and global levels are we able to scale up monitoring, aggregate and disaggregate data and demonstrate progress against global goals such as the Aichi Targets.

If governments, NGOs and the private sector use similar indicators, and in many cases the same ones as for CBD and SDG reporting, they will also be able to use common monitoring protocols and collect data in a systematic, standardised way. Standardised monitoring protocols used widely include the Spatial Monitoring and Reporting Tool (SMART; http://smartconservationsoftware.org) to monitor pressures, the IUCN Red List categories and criteria (IUCN, 2012) to assess species status, and Important Bird and Biodiversity Area monitoring (BirdLife International, 2006). More protocols are also being introduced to monitor often-neglected species such as invertebrates and plants (e.g. Van Swaay et al., 2015; Borges et al., 2018).

Data collected in the same way are easier to store in the same (or interlinked) national, regional and global databases and, in turn, are easier to share among partners. For example, if all the elephant PAs in the fictional programme used SMART, poaching levels could be compared across sites, and compared with other agencies also using the tool, nationally, regionally and globally. Similarly, if population data were collected systematically, they could be shared through the African Elephant Database (sensu Thouless et al., 2016) and the Living Planet Index.

Partnerships, such as those between governments, NGOs and academic institutions, are widely encouraged to support data collection (Lindenmayer et al., 2012; Stephenson et al., 2015b; Vugteveen et al., 2015). Local communities can also play a role and citizen science projects are expanding in biodiversity-rich countries.

Monitoring should be implemented at least as long as the project. However, as noted by Lindenmayer and Likens (2010), many projects are funded for only one to three years, a timeframe too short to monitor ecological change. An internal review of WWF programmes found that, due to the time taken to implement work and for species and habitats to recover, it took an average of 10 years for interventions to result in an impact (Sheila O’Connor and PJS, unpublished data). Therefore, wherever possible, monitoring should continue for a relevant period after a project ends. If feasible, data should also be collected at similar sites not subject to conservation interventions to establish counterfactuals (see below). This will need to be factored into project budgets.

Programme managers and other decision makers and policymakers need data in formats that are easy to interpret and act on. Each project or programme should identify which questions it wants the data to answer and how it would like the data to be presented. Trends over time are the key to any meaningful analysis, presented in ways that demonstrate outliers (good or bad) in terms of scale or rate of change. The spatial presentation of data in map form is a powerful way to understand what is happening where. Dashboards that present data in various graphic forms are widely used in the corporate sector (e.g. Eckerson, 2010; Kerzner, 2013) and have been proposed as tools to visualise biodiversity data that monitor NGO goals and Aichi Targets (e.g. Han et al., 2014; Stephenson et al., 2015a). A system developed by WWF (Stephenson and Reidhead, 2018; Fig. 3) collected data through a standardised report format and summarised it in rows: a performance KPI score (showing delivery of annual goals), a summary of achievements and challenges (putting data into context) and then a suite of common PSRB indicators. By aligning common indicators across programmes, reading down columns allows managers to identify outliers (high or low performing programmes) for each measure, facilitating the identification of priorities and decision-making that leads to action. For example, data presented in dashboards that demonstrated the species declining fastest influenced WWF decisions on resource allocation (Stephenson et al., 2015a). In some cases, the development of decision triggers (Foster et al., 2019), defining the level of a measure at which action needs to be taken, can enable data use in adaptive management.

Fig. 3.

An example of a dashboard to present indicator data. From Stephenson et al. (2015a).

(0.65MB).

Every project team should use monitoring data on a regular basis, formally and informally, to assess and evaluate progress and make decisions for adaptive management (e.g. redefine objectives, implement new actions and approaches, adapt planning assumptions). While widely advocated in conservation, however, adaptive management is rarely applied in practice (Gillson et al., 2019). There is a tendency for project staff to save data for use in written reports, yet project teams should also discuss the information as it comes in and schedule regular meetings for such analyses.

Formal evaluations are a key element of project cycle management. Evaluations use appropriate methods (e.g. UNEP, 2008; Mascia et al., 2014; Moore et al., 2018) to assess efficiency of delivery of outputs, effectiveness of delivery of intermediate results and outcomes, and impact on conservation targets. Potential adaptive management responses to evaluations are numerous, but include altering strategies, improving plans and systems, and changing human resources. Learning from evaluations should also help shape the future project portfolio. For example, if elephants continued to decline in the Ruaha project site but increased in the Serengeti site, the management team would need to compare strategies and external factors to see what lessons from the successful Serengeti project could be applied to the less successful Ruaha project, or what additional external factors might need addressing (perhaps through a change of objectives). Similarly, if HEC mitigation in the hypothetical elephant programme was far more successful in one project than another, managers could exchange lessons and identify what needs changing.

A project evaluation will go a long way in assessing delivery. However, ultimately, at least for large-scale projects or programmes, conservation policy and practice will be much better informed through impact evaluations (the systematic process of measuring the intended and unintended causal effects of conservation interventions, with emphasis on long-term impacts on ecological and social conditions) and systematic reviews (the review of existing research findings to assess evidence regarding the impacts of conservation interventions) (Mascia et al., 2014). Currently, impact evaluations are far less common in environmental projects than in sectors such as health, education, agriculture and development (McKinnon et al., 2015; Cameron et al., 2016). By using counterfactuals, impact evaluations provide the ultimate method of establishing whether a project had a significant influence on the state of biodiversity or a reduction in threats. It can also facilitate learning and promote accountability (McKinnon et al., 2015; GEF IEO, 2016).

Learning can be facilitated in various ways. Peer review of project reports (having project managers or monitoring staff review each other’s reports) can be productive and facilitate lesson sharing. No-one should ever write a report that just gets filed. Every report writer should receive feedback from, at least, their line manager and, at best, a cohort of their peers, and have opportunities to discuss lessons and decide on responses. The tendency is for these responses to await the final project evaluation, but that may be too late. Feedback should be constant and iterative.

For adaptive management to work, it is vital that managers support organisational monitoring standards and reporting systems (Stephenson et al., 2015b). However, there is often a paucity of institutional support for monitoring (Lindenmayer et al., 2012). Not everyone appreciates data since the trends may not reflect the messages they want to send. For example, many people perceive the need to report positive results to their line managers or donors. Wahlén (2014) found that “organisational and managerial pressure to report success is greater than donor pressure”, a reality I have also witnessed. Organisations need to address this by developing a results-based management culture that creates the space to fail, encouraging people to learn and share lessons (Redford and Tabler, 2000). Identification of problems (such as project objectives and results not being met), and the adaptation of strategies and policies to try to rectify them, should be seen as strengths, not weaknesses.

Creating a results-based management culture also means circumventing common excuses that prevent data use. Some people continue to perceive monitoring as “blind data gathering” (Noss, 1990), diverting resources from conservation action. But all action and no monitoring means we have no idea if we attained what we wanted to. Conservation activities that are poorly planned and monitored will deliver fewer successful conservation outcomes and impacts than those that are planned well, monitor progress and adapt in response to lessons learned. A return on investment in planning and monitoring will be a higher proportion of successfully implemented conservation actions (Fig. 4).

Fig. 4.

Graphic to explain the theory of change behind results-based management (RBM). Investment in RBM, especially planning and monitoring, will lead to a higher proportion of successful conservation actions and an improved return on investment. Note that the proportions assigned to different categories of effort are only indicative and are not meant to propose a specific level of RBM as that will vary between projects. (Adapted from Foundations of Success and WWF, Conservation Measures Partnership).

(0.14MB).

Some managers do not see a value in collecting data if their staff cannot prove change was a result of the organisation’s work. There are three main ways to provide an appropriate assessment of attribution depending on the level of resolution available, through what I term the “Three Cs of Attribution” (Table 3): counterfactuals — you can provide evidence of the impact of your actions by demonstrating what happened where conservation was not applied; context — by relating your results to your theory of change you can show that what was expected to happen is happening, suggesting the impact of your actions; and comparison — if you look at similar projects applying similar strategies and demonstrating similar results, you can assume the impact of your actions.

Impact evaluation methods use counterfactuals to compare project sites with sites where no interventions were made (Ferraro and Hanauer, 2014; Gertler et al., 2016). This approach should be the norm, at least for large-scale projects and programmes, as “the differences between the observed conditions and the counterfactual conditions are, by definition, the impacts that we seek to understand” (Ferraro and Pressey, 2015). Sometimes the counterfactual can be a simple measure, especially if collected remotely (like habitat cover or wetland extent), and can provide the necessary extra data without incurring huge costs. By using the context approach from the Three Cs of Attribution, and looking at a project’s theory of change, in some cases lower-level results can be used to make assumptions and lay the basis for some degree of attribution. Additionally, if more than one step in the results chain is measured, it could show interlinkages between PSRB indicators. Comparison can also extend to neighbouring project sites. If a project nearby experiences a dip in habitat cover or water levels due to climatic factors at the same time as your project, it may be easier to demonstrate change due to external rather than project factors.

Adoption of project management tools and building capacity for their use contributes to project success (Golini et al., 2015; Brooks, 2017), especially if adapted to the local context (Thapa et al., 2014). However, capacity for planning and monitoring in the conservation sector remains weak (Barlow et al., 2016; Stephenson et al., 2017a), especially in high biodiversity countries in Africa and Asia (Stephenson et al., 2017b).

Steps to Conservation Impact 1 and 2 need to follow a structured approach to project design. The Conservation Measures Partnership’s Open Standards for the Practice of Conservation (CMP, 2013) offer a series of guidelines and tools to take users through the project cycle. Other models (e.g. Andrews et al., 2013; Kapos et al., 2008; Cook et al., 2014; Dickson et al., 2017), or combinations of models (Battisti, 2018), exist. Every organisation will need to decide which system to follow and then build internal capacity for its use to ensure projects are planned and monitored properly, and the data used in reporting, learning, and decision-making for adaptive management. Whichever system is followed, however, the Five Steps to Conservation Impact need to be incorporated and emphasised, otherwise impact may remain elusive. Assigning an appropriate portion of project budgets for planning and data collection is also obviously key. Organisations might consider building national or regional hubs of monitoring expertise comprised of staff who can help projects in-country.