ReviewA global comparative analysis of impact evaluation methods in estimating the effectiveness of protected areas
Introduction
Impact evaluation methods are pivotal for measuring the effectiveness of conservation interventions (Ferraro and Pattanayak, 2006). All these methods aim to estimate the impact of interventions by comparing conditions in the presence of interventions with those in the absence of interventions. These diverse methods have been sparsely applied in conservation science and vary in rigor and reliability. According to the Millennium Ecosystem Assessment (2005) “few well-designed empirical analyses assess even the most common biodiversity conservation measures.” The need for studies evaluating the effectiveness of conservation actions is even stronger considering limited budgets for conservation and ever-increasing rates of environmental change (Pullin and Knight, 2001; Sutherland et al., 2004; Ferraro and Pattanayak, 2006).
Protected areas (PAs) are common and extensive conservation interventions, considered crucial to limit the decline of biodiversity and ecosystem services (Ferraro et al., 2013). However, the effectiveness of PAs in reducing biodiversity loss is usually implicitly or explicitly assumed in many practical situations (Pfaff et al., 2009; Vieira et al., 2019). Therefore, accurately measuring the effectiveness of PAs has been a recurring challenge (Chape et al., 2005). A common way to measure the effectiveness of PAs involves comparisons of areas inside and outside PAs, aiming to estimate how protection prevented deforestation (Millennium Ecosystem Assessment, 2005; Blackman, 2015; Burivalova et al., 2019). However, such an approach can disregard the fact that PAs are usually established in places with low opportunity costs (e.g. on steep terrains with low suitability for agriculture; Hoekstra et al., 2005; Brooks et al., 2006; Vieira et al., 2019), whereas the control areas might not be. Comparison of areas with different opportunity costs for extractive uses can therefore result in biased estimates of PA effectiveness, leading to overestimation (Oldfield et al., 2004; Andam et al., 2008). In theory, to overcome this bias, protected and unprotected areas should be randomly distributed across the landscape. However, it is easy to realize that the use of experimental randomization is often not feasible or even desirable in conservation biology (Khandker et al., 2009).
In general, we classify as “naïve” (Pfaff et al., 2009) methods that, in attempting to estimate the impacts of PAs in avoiding deforestation, disregard the differences in opportunity costs and deforestation probabilities of protected and control areas (Joppa and Pfaff, 2009). In view of this problem, other methods derived from the naïve ones have been proposed to control for the non-random distribution of PAs. Such methods limit the control areas to being close to PAs (Mas, 2005). This approach intends to make control areas and PAs as similar as possible. However, such similarity is rarely validated (Vanclay et al., 2001) because the proximity of control areas does not guarantee similarity in terms of likelihood of protection or deforestation. We call these methods “naïve within buffer zones,” because they generally disregard differences across PAs boundaries in the likelihood of deforestation. In addition, the widths of buffer zones are usually not theoretically justified. Estimates of impact from methods we term naïve and naïve within buffer zones can also be affected by spillover (Ament and Cumming, 2016) and leakage effects (Renwick et al., 2015).
In view of the problems with naïve methods, linear models have been used to improve impact estimates (White, 2006; Imbens and Wooldridge, 2009). These models include as explanatory variables a dummy variable indicating the presence or absence of protection and different covariates, such as distance to urban centers, distance to highways, distance to navigable rivers and streams, altitude, and slope (Newman et al., 2014). These covariates are selected to allow for factors that theoretically affect the likelihood of deforestation or protection. The measured outcome, such as avoided deforestation due to protection, is the response variable. The basic idea underlying these models is to evaluate the impact of protection, after considering the covariates as confounding factors. Different types of linear models can be used for impact evaluation and, due to the wide variation of approaches, we grouped them as “methods based on linear models”. Such methods have problems due to high parameterization, which in turn limit their applications to a small number of covariates (Pfaff et al., 2009). In addition, other problems suggest that linear models cannot represent intervention-outcome responses if those responses are not linear, and also, cannot represent different interactions between covariates and their effects on other covariates or outcomes (Pearl and Mackenzie, 2018). Thus, for example, linear models can provide biased estimates of the impact of PAs due to non-linear relationships between the explanatory and response variables (Rubin, 2001). We consider as “traditional methods” of impact evaluation all those cited so far (i.e. naïve, naïve within buffer zones, and based on linear models).
Only recently, other methods seeking to control for the non-random spatial distribution of conservation interventions have been proposed in the specialized literature. These methods, based on counterfactual thinking, are common in the medical, political, psychological, social, and econometric sciences (Imbens, 2004; Höfler, 2005; Brookhart et al., 2006; Morgan and Harding, 2006; Banerjee and Duflo, 2009; Ferraro, 2009; Stuart et al., 2011). Counterfactual methods aim to estimate the impact of interventions by comparing factual states (in the presence of the intervention) and counterfactual states (in the absence of intervention). Thus, the question would be through a counterfactual state, for example: how much deforestation would there be, in a given area if there were no intervention such as the establishment of a PA? However, as described by McConnachie et al. (2015): “because the counterfactual is unobservable, researchers must use the outcome from an area that was not treated or before the intervention happened as a surrogate for the counterfactual outcome.” To estimate the counterfactual state and the impact of PAs, counterfactual methods are distinguished from naive methods by selecting control areas as similar as possible to PAs, considering characteristics that influence the likelihood of these areas being protected and deforested (Gertler et al., 2016). Counterfactual methods should therefore provide more accurate impact estimates than traditional methods (Ferraro, 2009; Imbens and Wooldridge, 2009). Among counterfactual methods, we highlight covariate matching and propensity score matching (Khandker et al., 2009; Schleicher et al., 2019). Furthermore, there are methods commonly used in environmental sciences that are similar to counterfactual methods, and that have also been applied to impact evaluation purposes, such as multivariate ordination (Eklund et al., 2016).
Using a systematic review, our main objective was to quantify the differences between estimates of PA effectiveness in avoiding deforestation, considering the results obtained by traditional and counterfactual methods (similar to the categories described by Burivalova et al., 2019). PAs are key management tools in conservation (Geldmann et al., 2013) and were therefore selected as the intervention of interest. In addition, we used avoided deforestation per year as the response variable. Our hypothesis is that traditional methods overestimate PA effectiveness compared to counterfactual methods. We expect that the largest differences in effectiveness would occur between naïve methods (including naïve within buffer zones) and counterfactual methods. Naïve methods, in principle, will overestimate the effect of PAs in avoiding deforestation by comparing control areas, no matter how close to PAs, that are different from PAs in terms of deforestation probabilities (Andam et al., 2008; Hanauer and Canavire-Bacarreza, 2015). For example, consider two scenarios. In the first, PAs are located on steep terrains, whereas control areas are on adjacent flat terrains. In the second, both PAs and control areas are located on steep terrains. We expect that the estimate of avoided deforestation would be much higher in the first scenario than in the second because other factors – terrain in this example – explain, at least partially, deforestation rates. On the other hand, we expect that the estimates obtained by methods based on linear models would be similar to the estimates provided by counterfactual methods. This expectation can be justified given linear models, even considering their limitations (see Imbens and Wooldridge, 2009; for discussion on this topic), account for confounding factors by adjusting the outcome for different covariates, We also assessed methods that did not fall into the previous impact evaluation classes, such as those based on multivariate ordinations.
Section snippets
Overview of systematic review
The systematic review was based on a review protocol available on the Open Science Framework platform (Ribas and Bini, 2017). The systematic review protocol followed Pullin and Stewart (2006) and Moher et al. (2009). We divided the review into two phases. The first phase focused on articles that used counterfactual impact evaluation methods. The second, complementary, phase focused on articles that used either or both counterfactual and traditional impact evaluation methods. Both phases
Distribution of estimates
We found 391 effect sizes: 145 in samples classified as belonging to the paired group and 246 to the unpaired group. About one quarter (27%) evaluated the impact of only one PA, 17.6% evaluated the impacts of between two and 20 PAs, and 25.4% evaluated the impact of more than 20 PAs. We were unable to find the number of PAs for 30% of the case studies because of missing information. Some 42% of the estimates evaluated PAs with restricted use (IUCN categories Ia, Ib, II, III and IV).
General trends of impact evaluations of protected areas
Studies evaluating whether PAs are effective in avoiding deforestation when compared to non-protected areas are accumulating in the specialized literature (e.g. Nepstad et al., 2006; Andam et al., 2008; Chape et al., 2008; Joppa and Pfaff, 2010a). Also, there is a consensus that the establishment of PAs is one of the main interventions to reduce biodiversity loss (Chape et al., 2008). However, we are far from fully understanding the impacts of PAs in biodiversity conservation (Chape et al., 2005
Conclusions
- (i)
Counterfactual methods are preferable to most traditional ones because they control for different confounding factors and, therefore, it is reasonable to assume that they provide more accurate estimates. However, counterfactual methods should not be considered viable for all impact evaluation studies, especially because good counterfactual models depend on a thorough knowledge of the systems studied.
- (ii)
At least indirectly, our results, showing lower effectiveness of PAs when the studies use
CRediT authorship contribution statement
Luiz Guilherme dos Santos Ribas: Conceptualization, Methodology, Formal analysis, Investigation, Data curation, Writing - original draft. Robert L. Pressey: Conceptualization, Writing - review & editing. Rafael Loyola: Writing - review & editing. Luis Mauricio Bini: Conceptualization, Methodology, Validation, Writing - original draft.
Declaration of competing interest
The authors are not aware of any actual or potential conflicts of interest.
Acknowledgments
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001 (scholarships to LGSR), LGSR acknowledges the importance of the CNPq Sandwich program for the collaboration involved in the present paper. RL and LMB research activities are funded by CNPq (grants #306694/2018-2 and #304314/2014-5). RLP acknowledges the support of the Australian Research Council. This paper is a contribution of the INCT in Ecology, Evolution and
References (100)
Strict versus mixed-use protected areas: Guatemala’s Maya biosphere reserve
Ecol. Econ.
(2015)- et al.
Paper park performance: Mexico’s natural protected areas in the 1990s
Glob. Environ. Chang.
(2015) - et al.
Evidence types and trends in tropical forest conservation literature
Trends Ecol. Evol.
(2019) - et al.
Contrasting spatial and temporal trends of protected area effectiveness in mitigating deforestation in Madagascar
Biol. Conserv.
(2016) - et al.
Estimates of reserve effectiveness are confounded by leakage
Trends Ecol. Evol.
(2008) On regression adjustments to experimental data
Adv. Appl. Math.
(2008)- et al.
First, do no harm: a systematic review of deforestation spillovers from protected areas
Glob. Ecol. Conserv.
(2019) - et al.
Effectiveness of terrestrial protected areas in reducing habitat loss and population declines
Biol. Conserv.
(2013) - et al.
Temporal instability of evidence base: a threat to policy making?
Trends Ecol. Evol.
(2019) - et al.
Long-distance benefits of marine reserves: myth or reality?
Trends Ecol. Evol.
(2019)
Assessing deforestation and fragmentation in a tropical moist forest over 68 years; the impact of roads and legal protection in the Cockpit Country, Jamaica
For. Ecol. Manag.
A gap analysis of terrestrial protected areas in England and its implications for conservation policy
Biol. Conserv.
Governance, location and avoided deforestation from protected areas: greater restrictions can have lower impact, due to differences in location
World Dev.
Changing agents of deforestation: from state-initiated to enterprise driven processes, 1970–2000
Land Use Policy
The need for evidence-based conservation
Trends Ecol. Evol.
Forest protection and tenure status: the key role of indigenous peoples and protected areas in Panama
Glob. Environ. Chang.
The residual nature of protected areas in Brazil
Biol. Conserv.
Large sample properties of matching estimators for average treatment effects
Econometrica
Density-dependent spillover from a marine reserve: long-term evidence
Ecol. Appl.
Successful conservation of global waterbird populations depends on effective governance
Nature
Scale dependency in effectiveness, isolation, and social-ecological spillover of protected areas
Conserv. Biol.
Measuring the effectiveness of protected area networks in reducing deforestation
Proc. Natl. Acad. Sci.
The experimental approach to development economics
Ann. Rev. Econ.
Wildlife population trends in protected areas predicted by national socio-economic metrics and body size
Nat. Commun.
Fitting Linear Mixed-Effects Models Using lme4
A conservation planning approach to mitigate the impacts of leakage from protected area networks
Conserv. Biol.
Introduction to Meta-Analysis
Dependence of sustainability on the configuration of marine reserves and larval dispersal distance
Ecol. Lett.
Principles for the design of marine reserves
Ecol. Appl.
Inferring causal impact using Bayesian structural time-series models
Ann. Appl. Stat.
Variable selection for propensity score models
Am. J. Epidemiol.
Global biodiversity conservation priorities
Science
Forecasting conservation impact to pinpoint spatial priorities in the Brazilian Cerrado
Biol. Conserv.
Measuring the extent and effectiveness of protected areas as an indicator for meeting global biodiversity targets
Phil. Trans. R. Soc. B
The World’s Protected Areas: Status, Values, and Prospects in the Twenty-First Century
Marine reserves: size and age do matter
Ecol. Lett.
The Costs and Benefits of Protected Areas for Local Livelihoods: A Review of the Current Literature
Controlling bias in observational studies: a review
Sankhyã
Deforestation driven by urban population growth and agricultural trade in the twenty-first century
Nat. Geosci.
Counterfactual thinking and impact evaluation in environmental policy
N. Dir. Eval.
Money for nothing? A call for empirical evaluation of biodiversity conservation investments
PLoS Biol.
Conditions associated with protected area success in conservation and poverty reduction
Proc. Natl. Acad. Sci. U. S. A.
More strictly protected areas are not necessarily more protective: evidence from Bolivia, Costa Rica, Indonesia, and Thailand
Environ. Res. Lett.
Examining protected area effectiveness in Sumatra: importance of regulations governing unprotected lands
Conserv. Lett.
A global-level assessment of the effectiveness of protected areas at resisting anthropogenic pressures
Proc. Natl. Acad. Sci.
Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting
Methods Ecol. Evol.
Impact Evaluation in Practice
The impact of marine reserves: do reserves work and does reserve size matter?
Ecol. Appl.
Marine reserves have rapid and lasting effects
Ecol. Lett.
Implications of heterogeneous impacts of protected areas on deforestation and poverty
Phil. Trans. R. Soc. B
Cited by (50)
Assessment and management for future habitat risks under the implementation of China's territorial spatial planning: A case study from Hainan Island
2024, Environmental Impact Assessment ReviewAssessment of the effectiveness of China's protected areas in enhancing ecosystem services
2024, Ecosystem ServicesAssessing and optimizing the effectiveness of protected areas along China's coastal region: A social-ecological protected area network study
2024, Journal of Environmental ManagementAssessing the conservation effectiveness of theWorld's protected areas: A habitat quality and human activities perspective
2023, Journal of Cleaner ProductionWildlife response to management regime and habitat loss in the Terai Arc Landscape of Nepal
2023, Biological Conservation