Biodiversity of forest plant and animal species (such as the clear-Winged Forest glory pictured) is inextricably linked to carbon and climate change. Kadavoor.

In retrospect I went in blind. Of course I had familiarized myself with systematic reviews and read the Collaboration for Environmental Evidence guidelines, but in reality I was unprepared for what waited for me when I promised to conduct my first systematic review.

In case you are new to the world of evidence-based policymaking, systematic reviews aim to provide an exhaustive summary of literature relevant to a research question. They are different from normal (“narrative”) literature reviews in that they are more rigorous and try to incorporate all the possible evidence presented: not only scientific journals, but also from other sources of information, such as reports, working papers, and doctoral theses that are less often accessed.

Figure 1. The main elements of a systematic review, adapted from Petrokofsky et al. 2011 and reproduced by kind permission of the Commonwealth Forestry Association.

And that is the beauty of them: it is real evidence not just a collection of random papers – however good these may be. Now that I have experienced the review process from the beginning to submission of the manuscript, I am in love with them. So in case you are about to board the train called evidence based forestry (or conservation/development), I would like to share some lessons from my first experience.

Lesson 1: Choose a topic and team mates that you love

If you have this luxury, take it, because you will be working hard! At the beginning I was not overly enthusiastic about the topic of our review on the impacts of biofuel crop cultivation on biodiversity, but it grew on me. Mainly because I thought there is little added value on our review. How wrong I was: though opinions are abound, very little empirical data about the impacts of biofuel crops exists.

In many ways a systematic review is like a marathon — you may reach the point where you feel that you have done quite enough but you are still not at the end. But unlike a marathon, you may be dragging your teammates with you and sometimes they may be the ones to push instead of yourself. I was lucky enough to work with people who care. Although I did the bulk of the work alone, they were supportive throughout the process, and offered invaluable scientific and practical comments.

Lesson 2: Devote time to preparing the protocol and testing the search terms

Of course there are things that can make your life easier during the systematic review process. The importance of your systematic review protocol cannot be emphasized enough. The protocol charts the course for your review: In it you write where you will look for the information (the sources), how you will find it (the search terms), what are you going to include in the final review (the inclusion and quality criteria), how you will analyze the data and synthesize your findings.

Testing the search terms is especially important. When I tested our search terms, I was more concerned about accidentally leaving things out so I ended up including a lot of unrelated material – we started with around 10,000 articles (even just reading each title takes a long time!) and then whittled it down to 25 final studies that fulfilled all the inclusion and quality criteria. Here is a link to the protocol we developed for our systematic review.

Lesson 3: Stakeholder workshops are a great way forward at various stages of the review 

At the beginning of the review process, we elicited comments on the protocol from various stakeholders and colleagues by email communication. However because the protocol is so important, in retrospect, we should have held a stakeholder workshop to better garner input face to face. For the final push we organized a writing workshop and that was hugely beneficial, not only for the review but also for me personally in terms of meeting new people with the same interests.

Lesson 4: Allow time to contact authors to get all the data you need

Once you are actually writing the review you may notice how poor we researchers sometimes are at reporting not only the data we collect, but also our experimental design and our methods of proceeding with the research. So be prepared to contact authors to get clarity on these and budget time for that. Although some may not answer, the ones that replied to me were so supportive they made me smile.

Lesson 5: Be prepared to confront strong opinions and conflicting evidence in your review.

One of the best things about systematic reviews is that it makes a calm, scientific assessment of the evidence. This is especially useful when views are strongly contested and where there can be a tendency for the subject to raise strong feelings– such as in our review. Almost like two camps in a war against each other recognizing only evidence that supports their view and ignoring the evidence produced by ‘the other side’. In systematic review such bias cannot exist as the review takes into account all available evidence.

Lesson 6: Systematic reviews really are ideal for scientists who cannot be at the bench or in the field either permanently or temporarily, but who want to be part of mainstream high-quality science. 

In my case, the additional demands (and pleasures) of motherhood meant it was an ideal time for me to finish my first systematic review. Competing demands often make it difficult to spend extended time in the field, but writing the review allowed me to do science at the time when I needed to be in office.

Finally, remember a systematic review is just a starting point. As important is what you do with the information and how you disseminate it. For our team that has meant writing a concept note to get funding to fill some of the knowledge gaps we identified in the review. So the work continues.

Many thanks to Gill Petrokofsky for her valued comments on this blog.

Dr Sini Savilaakso is an associate professional officer at CIFOR. Contact her at s.savilaakso@cgiar.org

CIFOR’s Evidence Based Forestry Initiative is part of the CGIAR Research Program on Forests, Trees and Agroforestry and is supported by the UK Department of International Development. For more information, please visit //www.cifor.org/ebf or contact CIFOR-EBFInitiative@cgiar.org