The science behind the Guardian piece is fatally flawed

On January 18, The Guardian published an article based on three scientific papers (West et al. 2020, West et al. 2023 and Guizar Coutiño et al. 2022). These papers use a synthetic control approach in an attempt to create “proxy” areas for individual REDD+ projects as an alternative to the reference areas (VCS, 2020) required for developing a baseline scenario under the VCS standard.

The authors used their selected control sites in an attempt to assess the accuracy of the baseline deforestation used by the projects for evaluating performance in reducing deforestation and carbon emissions. Based on this approach, the authors concluded that the majority of REDD+ project baselines were overstated, suggesting that the actual carbon emissions reductions generated by the projects were far less than claimed.

However, the underlying analysis on which the article is based is fundamentally flawed, resulting in inaccurate and misleading conclusions. The methodological weaknesses, significant uncertainties, lack of peer-reviewed science for one critical reference, and misrepresentation of another fundamentally undercut the validity of the conclusions drawn by the Guardian authors. The reporting by the Guardian reflects no acknowledgement of these issues, and creates a strong impression, if not the clear reality, of bias and low journalistic integrity. As the world reckons with this unprecedented challenge of ending deforestation by 2030 and safeguarding life on Earth, we must hold ourselves to a higher standard of scientific and journalistic integrity.

Here is what they got wrong:

1. The studies do not compare “like” with “like.”  In principle, variables that influence local deforestation risk should be matched between a “control area” and a corresponding project area – for example, the presence of a road or of recent deforestation in adjacent areas. However, the authors selected highly unrealistic matches based on a simplistic, generic set of variables that were applied across all projects – many of which were not relevant to the local deforestation drivers associated with each specific project area. Many of the control sites were located very far from the project in question, and feature very different deforestation, socioeconomic and land tenure circumstances. For example, in the case of one large REDD+ project, the authors selected protected areas as the control sites for a project that would have been a commercial logging concession in the baseline (“without project”) scenario. As a result, these “control areas” naturally show far lower deforestation than sites with similar baseline land tenure conditions, i.e., logging concessions, leading to an incorrect and misleading conclusion that this project has overstated its baseline.   

2. Results vary wildly. Slight differences in the chosen matching criteria can produce substantial differences in the results of synthetic control analyses, and it is easy for researchers to subtly alter matching characteristics to produce desired results (Desbureaux, 2021; Schleicher et al. 2019). We find that the control site results differ significantly between West (2020, 2023) and Guizar Coutiño (2022) –  when comparing the deforestation rates in control areas attributed to the same project, rates differ from between +18% to -1883%. If similar methods produce significant result divergences and the results cannot be replicated, no valid substantive conclusions can be drawn.

3. The underlying data is highly uncertain and simplistic. All data used in estimating deforestation has some degree of error. The level of error impacts how much we can rely on the results. The selected forest change datasets used in the abovementioned studies are known to be highly uncertain and oversimplified for the analytic purposes of the cited studies. For example, the West et al. (2020, 2022) datasets are limited to a simple forest/non-forest classification; much of which is known to have errors of up to 65% in the studied regions (Tyukavina et al. 2015). Based on this potential for error, it is unsurprising that the rates of project area deforestation vary significantly between the three studies: West et al. results are on average 256% higher than Guizar Coutiño. Yet, neither West et al. nor The Guardian authors communicated about the uncertainty of their findings, nor did they assess or account for these uncertainties, nor the significant discrepancies between the studies. 

4. No peer review. The article relies heavily on the work of West et al. (2023) in particular, yet this paper was not accepted by a scientific journal and has not been peer-reviewed. The Guardian’s conclusions are based principally on work that the authors have been unable to validate by the scientific community.

5. Misrepresentation of key reference. In addition, the Guardian refers to the work of Guizar et al. (2022) as corroborating its conclusions. However, this paper used control areas to compare deforestation in project areas to the control areas, not to assess REDD+ project baselines. In fact, the study “found consistent reductions in deforestation and forest degradation when comparing the effectiveness of REDD+ relative to matched pixels outside protected areas,” and “provides promising evidence that site-based REDD+ projects have helped reduce deforestation, particularly in areas of high deforestation threat.” The Guardian journalists took these published results out of context and misrepresented them inappropriately to corroborate their conclusions.

A growing body of peer-reviewed research – including Guizar et al. (2022) as well as our own (Pauly, Crosse, and Tosteson (2022)) – confirm the substantial effectiveness of REDD+ in reducing deforestation and emissions. Forthcoming research by a large international group of collaborators will provide a more comprehensive, methodologically rigorous evaluation of REDD+ baselines; this research will be published later in 2023. 

REDD+ is one of the most effective proven mechanisms, developed over decades by pioneers in the sector to provide economic value for conservation. It works, and it must be scaled up urgently if we are to have any legitimate hope of ending deforestation by 2030, as 140 countries have committed to doing.

Signed,

Joshua Tosteson, MSc.
President, Everland

Maren Pauly, PhD.
Director – Evaluation & Research, Everland

Works cited

Desbureaux, Sébastien. 2021. Subjective modeling choices and the robustness of impact evaluations in conservation science. Conservation Biology 35(5). 

Guizar‐Coutiño, A., Jones, J.P., Balmford, A., Carmenta, R. and Coomes, D.A., 2022. A global evaluation of the effectiveness of voluntary REDD+ projects at reducing deforestation and degradation in the moist tropics. Conservation Biology, p.e13970.

Schleicher, J., Eklund, J., D. Barnes, M., Geldmann, J., Oldekop, J.A. and Jones, J.P., 2019. Statistical matching for conservation science. Conservation Biology, 34(3), pp.538-549.

Tyukavina, A., Baccini, A., Hansen, M.C., Potapov, P.V., Stehman, S.V., Houghton, R.A., Krylov, A.M., Turubanova, S. and Goetz, S.J., 2015. Aboveground carbon loss in natural and managed tropical forests from 2000 to 2012. Environmental Research Letters, 10(7), p.074002.

West, T.A., Wunder, S., Sills, E.O., Börner, J., Rifai, S.W., Neidermeier, A.N. and Kontoleon, A., 2023. Action needed to make carbon offsets from tropical forest conservation work for climate change mitigation. arXiv preprint arXiv:2301.03354.

West, T.A., Börner, J., Sills, E.O. and Kontoleon, A., 2020. Overstated carbon emission reductions from voluntary REDD+ projects in the Brazilian Amazon. Proceedings of the National Academy of Sciences, 117(39), pp.24188-24194.

Pauly, M., Crosse, W. and Tosteson, J., 2022. High deforestation trajectories in Cambodia slowly transformed through economic land concession restrictions and strategic execution of REDD+ protected areas. Scientific Reports, 12(1), pp.1-9.

Verified Carbon Standard, 2020. VM0007 REDD+ Methodology Framework (REDD+ MF). Version 1.6. Sectoral Scope 15.