Morning mist in Gunung Halimun-Salak National Park, Java, Indonesia. Mokhamad Edliadi/CIFOR photo

Forest conservation could benefit from more, and better, impact evaluation to understand what works and what not, but evaluation efforts must account for the complexities inherent in the sector.

This was the conclusion of experts in impact evaluation and conservation following a three-day international workshop on Evaluating Forest Conservation Initiatives, held in Barcelona in December. The workshop brought together about 40 researchers, practitioners and policymakers.

Presentation:

Sven Wunder from Barcelona

“It’s a healthy process for us to think about the impacts of these programs — what works, what doesn’t,” said Jan Börner,  Associate Scientist at the Center for International Forestry Research (CIFOR) and  affiliated with the Center for Development Research (ZEF) in Bonn (Germany), and a member of the workshop organizing committee.

“That has been happening much more in other sectors, although impact evaluation is less difficult there than for the kind of socioeconomic–biophysical interaction that we’re dealing with in forest conservation,” he added.

This complexity has left conservation “in the dark ages compared to most other sectors,” said Paul Ferraro, professor of economics at the Andrew Young School of Policy Studies at Georgia State University and one of the workshop’s keynote speakers, on the sidelines of the event.

“This is not only because we don’t have the evidence base but because conservation is one of the most difficult evaluation contexts,” Ferraro said.

Multiple examples of the complexity inherent in evaluating conservation initiatives emerged during presentations delivered at the workshop, with researchers and practitioners describing programs on payments for environmental services (PES) in Colombia and Costa Rica, protected areas in Brazil and Indonesia, biodiversity conservation in Mexico, marine parks in Indonesia and reciprocal watershed agreements in Bolivia.

QUESTION AND ANSWER

The difficulty of evaluating impacts in forest conservation was similarly noted by Alexander Pfaff, associate professor at the Sanford Institute of Public Policy at Duke University, also a keynote speaker.

“What’s easy to see is if the tree is standing,” Pfaff said on the sidelines of the workshop. “What’s harder to know is if the tree would be standing if you hadn’t done anything — and that means that we’re in the business of guessing.”

Pfaff offered the example of Costa Rica, renowned for its success in forest conservation and use of PES policies. Costa Rica, he pointed out, had tried several other policies before PES, so the data had to be scrutinized to determine whether the success was in fact attributable to PES or to one or more of the other policies.

In this case, as in any other, he said, the first step in increasing or improving impact evaluation is to get researchers, practitioners and policymakers alike asking about impact in the first place.

“Just asking the question, ‘Can we check whether this was the thing that changed the forest?’ — if we can ask it in a neutral and supportive way, and use the answers to guide more policy, I think that’s very important,” he said.

Yet given the complexities associated with forest conservation, subsequent questions must be asked carefully and simplistic answers avoided, argued event co-organizer Sven Wunder, Principal Scientist at CIFOR.

“The tendency for many impact studies has been to concentrate on a single bottom line or number, and that’s something that we have to get away from,” Wunder said.

Often single numbers are not telling the whole story. This became clear in a presentation by Jonah Busch of the Center for Global Development, simulating the effect of Indonesia’s moratorium on forest concessions. Busch showed that the proportion of forest the policy may have “saved” could vary from 3.6 percent to 13 percent depending on the analytical assumptions.

“We also need to see the interplay with causal factors — why it works or not,”  Wunder added.

“The single  bottom line can be more or less accurate, but it won’t tell you enough about what has happened, and why. You need a combined story about causality backed up with the evidence.”

Further complicating the answers is that conservation policies affect different groups — socioeconomic groups, ethnic groups, gender groups — in different ways, and no single number can capture the heterogeneity of effects across social groups.

As Pfaff noted, “Numbers are great, but you can make up a lot of stuff with numbers.”

BODY OF EVIDENCE

The advice from Philip Davies, Deputy Director for Systematic Reviews at International Initiative for Impact Evaluation, for the forest conservation sector to catch up is to take a strategic approach to building evidence.

“We need to distinguish between building strategic evidence and operational evidence. You need to think about three, five, 10 years — that’s thinking strategically,” Davies said during his presentation at the workshop.

Davies gave the example of the public health sector, which over the past two decades has compiled a huge evidence base, including impact evaluation studies and systematic reviews — the same kind of thinking that led CIFOR and its partners to launch the Evidence-Based Forestry initiative in 2013.

By starting now, the forest conservation sector will have a solid evidence base by 2020, Davies said, and that would help inform and guide policy.

“It’s very hard to influence policy in real time unless you’ve got that evidence base that you can pull out and put into practice,” he said.

Putting that evidence into practice is, of course, the ultimate aim — and it must happen as early in the process as possible, Ferraro advised.

Arild Angelsen:

Using experiments for ex-ante evaluation of forest conservation initiatives

“We talk about monitoring and evaluation, but what we’re really doing is just monitoring,” he said. “We’re talking about trends and status of the forest, but that’s not the same as evaluation. It’s hard to do that without designing the program at the beginning to be evaluated.”

So with all those at the workshop in agreement that the sector needs more impact evaluation, the next question was: How much does it need? Despite an entertaining debate on the question between Wunder and Ferraro, no single number or definite answer appears to apply here either.

“All I know is that zero is not the right number, and that’s the situation we have now,” Ferraro said. “And I would say it’s not 100 percent [either].”

Because, he argued, the solution is not to try and evaluate the impact of every program — “That isn’t possible” — but to focus resources on larger  programs, such as national-level PES and protected area networks, so as to test their societal benefits.

Or, as Pfaff put it: “We need as much impact evaluation as would actually improve policy.”

For further information on the topics discussed in this article, please contact Sven Wunder at s.wunder@cgiar.org.

The international workshop “Evaluating Forest Conservation Initiatives: New Tools and Policy Needs” was organized by ZEF (Center for Development Research, University of Bonn) and Institute for Environmental Science and Technology, with support from the Robert Bosch Foundation through ZEF’s project “Shaping environmental policies for sustainable forest bioeconomies”, the European Association of Environmental and Resource Economists (EAERE), and IDDRI.

Recording of presentations made at the workshop are available here.