Review
A global comparative analysis of impact evaluation methods in estimating the effectiveness of protected areas

https://doi.org/10.1016/j.biocon.2020.108595Get rights and content

Highlights

  • Traditional methods estimated higher effectiveness of protected areas.

  • Counterfactual methods estimated lower effectiveness of protected areas.

  • Counterfactual methods should be considered more accurate than others.

  • Some traditional and counterfactual methods estimated similar effectiveness.

  • Evidence of the effectiveness of protection is unevenly distributed across the world.

Abstract

Impact evaluation aims to estimate the effect of an intervention on intended, and perhaps unintended, outcomes compared to the outcomes of no intervention or different intervention. Traditional impact evaluation methods used in environmental sciences tend to compare protected and control areas that differ in several characteristics, thereby hampering the attribution of causality such as lower rates of deforestation occurring as consequence of protection. To overcome this problem, counterfactual methods have been developed to improve impact evaluation in environmental sciences, including studies that aim to measure the effects of protected areas in avoiding deforestation. The goal of counterfactual methods is achieved by identification of carefully selected and comparable control areas. Here, we report on a systematic review to evaluate whether estimates about the effectiveness of protected area differ between traditional and counterfactual impact evaluation methods. We found that estimates from traditional methods of avoided deforestation due to the establishment of protected areas were generally higher than those from counterfactual methods. However, estimates based on traditional linear models and multivariate ordinations were similar to those obtained by counterfactual methods. Although rarely used, linear methods and ordinations appear promising as parts of the impact evaluation toolbox, although their limitations need to be better understood.

Introduction

Impact evaluation methods are pivotal for measuring the effectiveness of conservation interventions (Ferraro and Pattanayak, 2006). All these methods aim to estimate the impact of interventions by comparing conditions in the presence of interventions with those in the absence of interventions. These diverse methods have been sparsely applied in conservation science and vary in rigor and reliability. According to the Millennium Ecosystem Assessment (2005) “few well-designed empirical analyses assess even the most common biodiversity conservation measures.” The need for studies evaluating the effectiveness of conservation actions is even stronger considering limited budgets for conservation and ever-increasing rates of environmental change (Pullin and Knight, 2001; Sutherland et al., 2004; Ferraro and Pattanayak, 2006).

Protected areas (PAs) are common and extensive conservation interventions, considered crucial to limit the decline of biodiversity and ecosystem services (Ferraro et al., 2013). However, the effectiveness of PAs in reducing biodiversity loss is usually implicitly or explicitly assumed in many practical situations (Pfaff et al., 2009; Vieira et al., 2019). Therefore, accurately measuring the effectiveness of PAs has been a recurring challenge (Chape et al., 2005). A common way to measure the effectiveness of PAs involves comparisons of areas inside and outside PAs, aiming to estimate how protection prevented deforestation (Millennium Ecosystem Assessment, 2005; Blackman, 2015; Burivalova et al., 2019). However, such an approach can disregard the fact that PAs are usually established in places with low opportunity costs (e.g. on steep terrains with low suitability for agriculture; Hoekstra et al., 2005; Brooks et al., 2006; Vieira et al., 2019), whereas the control areas might not be. Comparison of areas with different opportunity costs for extractive uses can therefore result in biased estimates of PA effectiveness, leading to overestimation (Oldfield et al., 2004; Andam et al., 2008). In theory, to overcome this bias, protected and unprotected areas should be randomly distributed across the landscape. However, it is easy to realize that the use of experimental randomization is often not feasible or even desirable in conservation biology (Khandker et al., 2009).

In general, we classify as “naïve” (Pfaff et al., 2009) methods that, in attempting to estimate the impacts of PAs in avoiding deforestation, disregard the differences in opportunity costs and deforestation probabilities of protected and control areas (Joppa and Pfaff, 2009). In view of this problem, other methods derived from the naïve ones have been proposed to control for the non-random distribution of PAs. Such methods limit the control areas to being close to PAs (Mas, 2005). This approach intends to make control areas and PAs as similar as possible. However, such similarity is rarely validated (Vanclay et al., 2001) because the proximity of control areas does not guarantee similarity in terms of likelihood of protection or deforestation. We call these methods “naïve within buffer zones,” because they generally disregard differences across PAs boundaries in the likelihood of deforestation. In addition, the widths of buffer zones are usually not theoretically justified. Estimates of impact from methods we term naïve and naïve within buffer zones can also be affected by spillover (Ament and Cumming, 2016) and leakage effects (Renwick et al., 2015).

In view of the problems with naïve methods, linear models have been used to improve impact estimates (White, 2006; Imbens and Wooldridge, 2009). These models include as explanatory variables a dummy variable indicating the presence or absence of protection and different covariates, such as distance to urban centers, distance to highways, distance to navigable rivers and streams, altitude, and slope (Newman et al., 2014). These covariates are selected to allow for factors that theoretically affect the likelihood of deforestation or protection. The measured outcome, such as avoided deforestation due to protection, is the response variable. The basic idea underlying these models is to evaluate the impact of protection, after considering the covariates as confounding factors. Different types of linear models can be used for impact evaluation and, due to the wide variation of approaches, we grouped them as “methods based on linear models”. Such methods have problems due to high parameterization, which in turn limit their applications to a small number of covariates (Pfaff et al., 2009). In addition, other problems suggest that linear models cannot represent intervention-outcome responses if those responses are not linear, and also, cannot represent different interactions between covariates and their effects on other covariates or outcomes (Pearl and Mackenzie, 2018). Thus, for example, linear models can provide biased estimates of the impact of PAs due to non-linear relationships between the explanatory and response variables (Rubin, 2001). We consider as “traditional methods” of impact evaluation all those cited so far (i.e. naïve, naïve within buffer zones, and based on linear models).

Only recently, other methods seeking to control for the non-random spatial distribution of conservation interventions have been proposed in the specialized literature. These methods, based on counterfactual thinking, are common in the medical, political, psychological, social, and econometric sciences (Imbens, 2004; Höfler, 2005; Brookhart et al., 2006; Morgan and Harding, 2006; Banerjee and Duflo, 2009; Ferraro, 2009; Stuart et al., 2011). Counterfactual methods aim to estimate the impact of interventions by comparing factual states (in the presence of the intervention) and counterfactual states (in the absence of intervention). Thus, the question would be through a counterfactual state, for example: how much deforestation would there be, in a given area if there were no intervention such as the establishment of a PA? However, as described by McConnachie et al. (2015): “because the counterfactual is unobservable, researchers must use the outcome from an area that was not treated or before the intervention happened as a surrogate for the counterfactual outcome.” To estimate the counterfactual state and the impact of PAs, counterfactual methods are distinguished from naive methods by selecting control areas as similar as possible to PAs, considering characteristics that influence the likelihood of these areas being protected and deforested (Gertler et al., 2016). Counterfactual methods should therefore provide more accurate impact estimates than traditional methods (Ferraro, 2009; Imbens and Wooldridge, 2009). Among counterfactual methods, we highlight covariate matching and propensity score matching (Khandker et al., 2009; Schleicher et al., 2019). Furthermore, there are methods commonly used in environmental sciences that are similar to counterfactual methods, and that have also been applied to impact evaluation purposes, such as multivariate ordination (Eklund et al., 2016).

Using a systematic review, our main objective was to quantify the differences between estimates of PA effectiveness in avoiding deforestation, considering the results obtained by traditional and counterfactual methods (similar to the categories described by Burivalova et al., 2019). PAs are key management tools in conservation (Geldmann et al., 2013) and were therefore selected as the intervention of interest. In addition, we used avoided deforestation per year as the response variable. Our hypothesis is that traditional methods overestimate PA effectiveness compared to counterfactual methods. We expect that the largest differences in effectiveness would occur between naïve methods (including naïve within buffer zones) and counterfactual methods. Naïve methods, in principle, will overestimate the effect of PAs in avoiding deforestation by comparing control areas, no matter how close to PAs, that are different from PAs in terms of deforestation probabilities (Andam et al., 2008; Hanauer and Canavire-Bacarreza, 2015). For example, consider two scenarios. In the first, PAs are located on steep terrains, whereas control areas are on adjacent flat terrains. In the second, both PAs and control areas are located on steep terrains. We expect that the estimate of avoided deforestation would be much higher in the first scenario than in the second because other factors – terrain in this example – explain, at least partially, deforestation rates. On the other hand, we expect that the estimates obtained by methods based on linear models would be similar to the estimates provided by counterfactual methods. This expectation can be justified given linear models, even considering their limitations (see Imbens and Wooldridge, 2009; for discussion on this topic), account for confounding factors by adjusting the outcome for different covariates, We also assessed methods that did not fall into the previous impact evaluation classes, such as those based on multivariate ordinations.

Section snippets

Overview of systematic review

The systematic review was based on a review protocol available on the Open Science Framework platform (Ribas and Bini, 2017). The systematic review protocol followed Pullin and Stewart (2006) and Moher et al. (2009). We divided the review into two phases. The first phase focused on articles that used counterfactual impact evaluation methods. The second, complementary, phase focused on articles that used either or both counterfactual and traditional impact evaluation methods. Both phases

Distribution of estimates

We found 391 effect sizes: 145 in samples classified as belonging to the paired group and 246 to the unpaired group. About one quarter (27%) evaluated the impact of only one PA, 17.6% evaluated the impacts of between two and 20 PAs, and 25.4% evaluated the impact of more than 20 PAs. We were unable to find the number of PAs for 30% of the case studies because of missing information. Some 42% of the estimates evaluated PAs with restricted use (IUCN categories Ia, Ib, II, III and IV).

General trends of impact evaluations of protected areas

Studies evaluating whether PAs are effective in avoiding deforestation when compared to non-protected areas are accumulating in the specialized literature (e.g. Nepstad et al., 2006; Andam et al., 2008; Chape et al., 2008; Joppa and Pfaff, 2010a). Also, there is a consensus that the establishment of PAs is one of the main interventions to reduce biodiversity loss (Chape et al., 2008). However, we are far from fully understanding the impacts of PAs in biodiversity conservation (Chape et al., 2005

Conclusions

  • (i)

    Counterfactual methods are preferable to most traditional ones because they control for different confounding factors and, therefore, it is reasonable to assume that they provide more accurate estimates. However, counterfactual methods should not be considered viable for all impact evaluation studies, especially because good counterfactual models depend on a thorough knowledge of the systems studied.

  • (ii)

    At least indirectly, our results, showing lower effectiveness of PAs when the studies use

CRediT authorship contribution statement

Luiz Guilherme dos Santos Ribas: Conceptualization, Methodology, Formal analysis, Investigation, Data curation, Writing - original draft. Robert L. Pressey: Conceptualization, Writing - review & editing. Rafael Loyola: Writing - review & editing. Luis Mauricio Bini: Conceptualization, Methodology, Validation, Writing - original draft.

Declaration of competing interest

The authors are not aware of any actual or potential conflicts of interest.

Acknowledgments

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001 (scholarships to LGSR), LGSR acknowledges the importance of the CNPq Sandwich program for the collaboration involved in the present paper. RL and LMB research activities are funded by CNPq (grants #306694/2018-2 and #304314/2014-5). RLP acknowledges the support of the Australian Research Council. This paper is a contribution of the INCT in Ecology, Evolution and

References (100)

  • M.E. Newman et al.

    Assessing deforestation and fragmentation in a tropical moist forest over 68 years; the impact of roads and legal protection in the Cockpit Country, Jamaica

    For. Ecol. Manag.

    (2014)
  • T.E.E. Oldfield et al.

    A gap analysis of terrestrial protected areas in England and its implications for conservation policy

    Biol. Conserv.

    (2004)
  • A. Pfaff et al.

    Governance, location and avoided deforestation from protected areas: greater restrictions can have lower impact, due to differences in location

    World Dev.

    (2014)
  • T.K. Rudel

    Changing agents of deforestation: from state-initiated to enterprise driven processes, 1970–2000

    Land Use Policy

    (2007)
  • W.J. Sutherland et al.

    The need for evidence-based conservation

    Trends Ecol. Evol.

    (2004)
  • G. Vergara-Asenjo et al.

    Forest protection and tenure status: the key role of indigenous peoples and protected areas in Panama

    Glob. Environ. Chang.

    (2014)
  • R.R.S. Vieira et al.

    The residual nature of protected areas in Brazil

    Biol. Conserv.

    (2019)
  • A. Abadie et al.

    Large sample properties of matching estimators for average treatment effects

    Econometrica

    (2006)
  • R.A. Abesamis et al.

    Density-dependent spillover from a marine reserve: long-term evidence

    Ecol. Appl.

    (2005)
  • T. Amano et al.

    Successful conservation of global waterbird populations depends on effective governance

    Nature

    (2018)
  • J.M. Ament et al.

    Scale dependency in effectiveness, isolation, and social-ecological spillover of protected areas

    Conserv. Biol.

    (2016)
  • K.S. Andam et al.

    Measuring the effectiveness of protected area networks in reducing deforestation

    Proc. Natl. Acad. Sci.

    (2008)
  • A.V. Banerjee et al.

    The experimental approach to development economics

    Ann. Rev. Econ.

    (2009)
  • M. Barnes et al.

    Wildlife population trends in protected areas predicted by national socio-economic metrics and body size

    Nat. Commun.

    (2016)
  • D. Bates et al.

    Fitting Linear Mixed-Effects Models Using lme4

    (2014)
  • M. Bode et al.

    A conservation planning approach to mitigate the impacts of leakage from protected area networks

    Conserv. Biol.

    (2015)
  • M. Borenstein et al.

    Introduction to Meta-Analysis

    (2009)
  • L.W. Botsford et al.

    Dependence of sustainability on the configuration of marine reserves and larval dispersal distance

    Ecol. Lett.

    (2001)
  • L.W. Botsford et al.

    Principles for the design of marine reserves

    Ecol. Appl.

    (2003)
  • K.H. Brodersen et al.

    Inferring causal impact using Bayesian structural time-series models

    Ann. Appl. Stat.

    (2015)
  • M.A. Brookhart et al.

    Variable selection for propensity score models

    Am. J. Epidemiol.

    (2006)
  • T.M. Brooks et al.

    Global biodiversity conservation priorities

    Science

    (2006)
  • F.T. Brum et al.

    Forecasting conservation impact to pinpoint spatial priorities in the Brazilian Cerrado

    Biol. Conserv.

    (2019)
  • S. Chape et al.

    Measuring the extent and effectiveness of protected areas as an indicator for meeting global biodiversity targets

    Phil. Trans. R. Soc. B

    (2005)
  • S. Chape et al.

    The World’s Protected Areas: Status, Values, and Prospects in the Twenty-First Century

    (2008)
  • J. Claudet et al.

    Marine reserves: size and age do matter

    Ecol. Lett.

    (2008)
  • L. Coad et al.

    The Costs and Benefits of Protected Areas for Local Livelihoods: A Review of the Current Literature

    (2008)
  • W.G. Cochran et al.

    Controlling bias in observational studies: a review

    Sankhyã

    (1973)
  • R.S. DeFries et al.

    Deforestation driven by urban population growth and agricultural trade in the twenty-first century

    Nat. Geosci.

    (2010)
  • P.J. Ferraro

    Counterfactual thinking and impact evaluation in environmental policy

    N. Dir. Eval.

    (2009)
  • P.J. Ferraro et al.

    Money for nothing? A call for empirical evaluation of biodiversity conservation investments

    PLoS Biol.

    (2006)
  • P.J. Ferraro et al.

    Conditions associated with protected area success in conservation and poverty reduction

    Proc. Natl. Acad. Sci. U. S. A.

    (2011)
  • P.J. Ferraro et al.

    More strictly protected areas are not necessarily more protective: evidence from Bolivia, Costa Rica, Indonesia, and Thailand

    Environ. Res. Lett.

    (2013)
  • D.L.A. Gaveau et al.

    Examining protected area effectiveness in Sumatra: importance of regulations governing unprotected lands

    Conserv. Lett.

    (2012)
  • J. Geldmann et al.

    A global-level assessment of the effectiveness of protected areas at resisting anthropogenic pressures

    Proc. Natl. Acad. Sci.

    (2019)
  • K. Gerstner et al.

    Will your paper be used in a meta-analysis? Make the reach of your research broader and longer lasting

    Methods Ecol. Evol.

    (2017)
  • P.J. Gertler et al.

    Impact Evaluation in Practice

    (2016)
  • B.S. Halpern

    The impact of marine reserves: do reserves work and does reserve size matter?

    Ecol. Appl.

    (2003)
  • B.S. Halpern et al.

    Marine reserves have rapid and lasting effects

    Ecol. Lett.

    (2002)
  • M.M. Hanauer et al.

    Implications of heterogeneous impacts of protected areas on deforestation and poverty

    Phil. Trans. R. Soc. B

    (2015)
  • Cited by (50)

    View all citing articles on Scopus
    View full text