Assessing quality in impact-oriented research

Four main attributes of transdisciplinary research quality have been identified: relevance, credibility, legitimacy, and effectiveness. Phil Roeder

Contemporary research in the social and environmental realms places strong emphasis on achieving ‘impact’. Research programs and projects aim to generate new knowledge with a focus on promoting and facilitating the use of that knowledge to enable change, solve problems and support innovation.

Much of the research conducted by CIFOR under the Forests, Trees and Agroforestry program (FTA) takes this approach. The social, environmental and economic problems we face today are complex and multi-dimensional and they operate at multiple scales. They also usually involve diverse stakeholders with different perspectives and interests.

Successfully addressing these problems requires combinations of new knowledge and innovation, alongside action and engagement. Therefore, the research we do tends to cross disciplinary lines (referred to as “interdisciplinary research”) and often involves stakeholders and other actors in the research process (referred to as “transdisciplinary research”).

As the boundaries between disciplines are crossed, traditional academic definitions, criteria and measures of research quality no longer suffice. They may even constrain or hinder effective work. For example, reviews of grant applications or journal article submissions that rely primarily on traditional academic research quality criteria may overlook or under-estimate the value of projects that cross disciplines or that include important engagement, capacity-building or knowledge translation components.

Likewise, researchers and research managers need appropriate criteria to guide the design and implementation of inter- and transdisciplinary projects and programs. As research approaches evolve, we need a parallel evolution of the principles and criteria that we use to define and evaluate research quality.

To help meet this need, a team of researchers from CIFOR and Royal Roads University recently conducted a systematic review of the literature as part of CIFOR’s Evidence-Based Forestry Initiative. We tackled this project while asking the question: “What are appropriate principles and criteria for defining and assessing transdisciplinary research quality?”

The study provides an overview of the relevant literature and summarizes the main aspects of quality identified. Four main principles emerged: relevance (including social significance and applicability); credibility (including criteria of integration and reflexivity added to traditional criteria of scientific rigor); legitimacy (including criteria of inclusion and fair representation of stakeholder interests); and effectiveness (with criteria that assess actual or potential contributions to problem solving and social change).

We organized the main criteria within these four principles and then developed a research quality assessment framework that can be used by researchers, supervisors, research managers and research evaluators to guide and assess research design and implementation. All in the hope of achieving greater ‘impact.’