Terms of Reference
Evidence gap map and systematic review consultancy
Issue date: 30 June 2016
Deadline: 23:59 GMT, 30 July 2016
1. Introduction
The International Initiative for Impact Evaluation (3ie) seeks to improve the lives of people in
low- and middle-income countries (L&MICs) by generating and synthesising high-quality
evidence on development interventions. Impact evaluations examine not just what development
interventions work, when, why, but also at what cost. One of the ways it does this is by
providing both technical and financial support for impact evaluations of development
programmes. As of 2016, 3ie has funded over 200 studies in over 50 countries. 3ie grants are
provided through three main grant windows, including the Development Priorities Window,
thematic windows which support studies in a specific sector and policy window grants that
match research teams with implementing agencies to do impact evaluations of programmes
identified by the country. 3ie is registered as a non-governmental organisation in the United
States of America. It has offices in New Delhi, London, and Washington, DC.
The World Agroforestry Centre (ICRAF) is one of 15 CGIAR Research Centers. ICRAF’s
headquarters are in Nairobi, Kenya with six regional offices located in Cameroon, China, India,
Indonesia, Kenya and Peru. ICRAF's mission is to generate science-based knowledge about
the diverse benefits — both direct and indirect — of agroforestry, or trees in farming systems
and landscapes, and to disseminate this knowledge to develop and promote policies and
practices that improve livelihoods and benefit the environment.
2. Background
There is increasing interest in promoting agroforestry as part of sustainable intensification
initiatives that reconcile agricultural production with the provision of other important ecosystem
services, such as the provision of clean water, flood and erosion control, enhancement of soil
health, and biodiversity conservation. A recent policy-oriented working paper of the Food and
Agricultural Organization of the United Nations, for example, states the following:
When designed and implemented correctly, agroforestry combines the best practices of tree
growing and agricultural systems resulting in more sustainable use of land. Agroforestry takes
place in both tropical and temperate regions, producing food and fibre for better food and
nutritional security. It also sustains livelihoods, alleviates poverty and promotes productive,
resilient agricultural environments. In addition, when practiced at scale, it can enhance
ecosystems through carbon storage, prevention of deforestation, biodiversity conservation,
cleaner water and erosion control, while enabling agricultural lands to withstand events such as
floods, drought and climate change. 1
The integration of trees into farming systems is promoted in many high-level policy documents
in many countries. 2 While several systematic reviews (SRs) have focused on trees and the
effects of agroforestry practices on specific outcomes, such as productivity and ecosystem
service provision 3, a focused and holistic SR of high-quality empirical studies on the causal
effects of agroforestry interventions and practices (particularly outside the context of tightly
controlled, research station-based experimental trials) is yet to be undertaken. Consequently, it
will be particularly useful for policymakers, researchers and other stakeholders to understand
the nature of this evidence to inform both policy decision-making and prioritising research.
Given 3ie’s focus on generating high-quality evidence on what works and why, particularly on
policy-relevant topics in L&MICs, and ICRAF’s interest in understanding the nature of evidence
on agroforestry and gaps that remain to be addressed, the two organisations have joined forces
in calling for and supporting this programme.
3. Scope of work
3.1 Nature of consultancy
3ie and ICRAF are seeking applications from qualified research teams with significant expertise
and experience in developing an EGM and conducting a high-quality SR of agroforestry
interventions in L&MIC as per 3ie’s approaches (Appendix A).
3ie seeks to recruit a team for these specific objectives:
I.
II.
To systematically collect, appraise, map and synthesise all available high-quality
evidence on the effectiveness (and where possible, cost-effectiveness and cost-benefit)
of tree-based interventions to inform policy and practice.
To identify key gaps in the evidence base on the effectiveness of such tree-based
interventions to guide future impact evaluations on this topic.
The team will work under the direction of the 3ie Evaluation Office and seek quality assurance
advice from 3ie’s Synthesis and Reviews Office as well as ICRAF’s Monitoring, Evaluation, and
Impact Assessment (MEIA) unit.
1 FAO. (2013) Advancing the Agroforestry on the Policy Agenda: A guide for decision-makers. Agroforestry Working
Paper, No. 1. Rome, Italy. http://www.fao.org/3/a-i3182e.pdf
2 See, for example: Republic of Kenya. (2014) Forest Policy, 2014. Ministry of Environment, Water and Natural
Resources; Government of Malawi. (2011) The National Agricultural Policy. Ministry of Agriculture and Food Security.
3 See, for example: Foli, S. et al. (2014) To what extent does the presence of forests and trees contribute to food
production in humid and dry forest landscapes?: a systematic review protocol. Environmental Evidence, 3:15;
Rosenstock, T. et al (2016) The scientific basis of climate-smart agriculture: A systematic review protocol. CCAFS
Working Paper no.138. Copenhagen, Denmark: CGIAR Research Program on Climate Change, Agriculture and Food
Security (CCAFS).
Page 2 of 11
The team will be responsible overall to the Head of Evaluation at 3ie and will report on day-today activities to the Evaluation Specialist in New Delhi. Coordination and technical support will
also be provided by ICRAF’s MEIA unit who will coordinate scientific inputs from other units
within the organisation as necessary. 3ie and ICRAF will have full ownership of the data,
protocols, templates and reports that will be produced. The researchers on the team will be
acknowledged as co-authors of the final product of the study upon discussion with 3ie and
ICRAF.
3.2 Key responsibilities
The scope of work and responsibilities are as following:
A. EGM of agroforestry interventions and practices in L&MICs
1. Develop a theory of change (ToC): Review all relevant literature, including academic and
policy journals and grey literature, to inform the development of a ToC and specific
impact pathways pertaining to agroforestry interventions and practices in L&MICs.
2. Validate the ToC: Engage with key stakeholders, including relevant ICRAF scientists
and other agroforestry experts and practitioners to inform this preliminary ToC and its
specific impact pathways, as well as to understand issues in the design and delivery of
agroforestry promotion programmes, including evidence needed to strengthen them. In
this role, the team may be required to travel to ICRAF headquarters in Nairobi;
3. Develop the study design: This includes a population, intervention, comparator, outcome
and study design (PICOS) framework to structure both the EGM and SR, informed by
the resulting ToC and the preliminary framework described in Appendix B;
4. Develop a robust search strategy: This will include identifying databases as well as
defining both inclusion and exclusion criteria;
5. Select studies and extract relevant data: Screen, review and code studies, including any
available grey literature;
6. Populate the EGM: Present the gap map in the 3ie-prescribed format;
7. Report and disseminate findings: The team will submit a report on the EGM and is also
expected to participate in and help organise a stakeholder workshop in Nairobi, Kenya.
B. SR of the existing evidence base on the effectiveness (and, where possible, the
cost effectiveness) of agroforestry interventions and practices in L&MICs
The literature review for the agroforestry EGM will inform this SR, which should be
structured around the ToC and impact pathways developed (as explained in Section 4.1).
The team will carry out the following tasks:
1. Draft a review protocol, as per 3ie’s systematic review guidelines, to be registered with
one of the collaborating groups.
2. Finalise the PICOS framework, including the development of search terms and search
strings to identify relevant impact studies for potential inclusion (if required, 3ie will
provide access to relevant databases).
3. Review and quality assess the included studies in accordance with 3ie’s checklist.
4. Undertake a statistical meta-analysis on an appropriate sample of the included studies,
where possible.
5. Analyse data, including a full causal chain analysis (based on the ToC and impact
Page 3 of 11
pathways) of agroforestry interventions and practices in L&MICs and include a limited
number of qualitative studies.
6. Maintain detailed documentation of the review process.
7. Write the review report, summary and policy brief in close consultation with 3ie and
ICRAF.
Refer to Appendix A for a better understanding of 3ie’s approach to EGM and SRs.
4. Required skills, experience and qualifications
4.1 Eligibility
The consultancy is open to any team with the requisite expertise, skills, time and general
capacity to successfully deliver on the scope of work, as well as eligibility to receive grants from
3ie and ICRAF. 3ie especially encourages women, ethnic minorities and qualified differentlyabled candidates to apply.
4.2 Experience and qualifications
The team leader must have a minimum of three to four years of experience in a role that
includes conducting registered SRs, EGMs and statistical meta-analysis. All members of teams
applying for this consultancy must possess these minimum requirements:






Experience of working successfully in a research team;
A post-graduate degree preferably in social sciences, statistics or other fields that
involved rigorous training in statistics;
Proven knowledge of impact evaluation design and methods, sampling and analysis;
Demonstrated knowledge and experience in forestry and agroforestry interventions;
Proficiency in MS Office suite and statistical software, such as Stata or SPSS for
quantitative and Atlas.ti or Nvivo for qualitative data collection and analysis; and
Familiarity with reference management softwares, such as Endnote or EPPI Reviewer.
Following are the required key personal competencies in team members:







Excellent project management skills;
Relationship management skills, including being cognizant of interpersonal and/or
multicultural sensitivities;
Demonstrated ability to initiate and take work forward with a team;
Ability to deliver under pressure with strict deadlines;
High ethical and work standards;
Good oral and written communication skills in English; and
Willingness and ability to travel to Nairobi, Kenya.
5. Contract terms and remuneration
The selected team should be available to start as soon as possible. The contract will be for a
period of one year. 3ie and ICRAF will contribute up to US$100,000 to the development of the
EGM and SR.
Page 4 of 11
6. Selection criteria
The proposals will be reviewed by a selection committee on the basis of following criteria:
 Overall approach to the tasks described;
 Relevant experience, especially working with non-profit organisations, international
organisations and multilateral or bilateral organisations; and
 Cost proposal without travel costs, which will be borne by 3ie if agreed as part of the
proposal.
7. Submission
Completed applications must include the following documents and information:
1. A two-page cover letter clearly mentioning the names of the team leader and the other
team members. The letter should highlight the team’s experience with conducting
EGMs, SRs, statistical meta-analysis, risk of bias assessment and knowledge of impact
evaluations;
2. A three to four page proposal (using information in Appendix B as reference) organised
along the following topics:
• Understanding of the scope and work required;
• A brief description of why an SR is required and why it is important to focus on
tree-based interventions;
• A collated list of search databases that will be used;
• A timeline for the proposed tasks (as listed in Section 4);
• Brief descriptions of the proposed team members, their experience and
qualifications in conducting research with respect to agroforestry or forestry,
EGM and SR;
• A table showing the division of tasks and time allocation of team members; and
• The proposed budget for undertaking the study. The budget must follow 3ie’s
direct cost and indirect cost policies.
3. Curriculum vitae (CV, not to exceed three pages). Applicants are required to provide
only information relevant to this call in their CV including their educational background,
professional experience and publications on agroforestry or forestry.
4. Contact details of three references.
5. One working paper or a published paper of an SR or an EGM.
Please submit the completed applications (not to exceed 10MB) via email to
[email protected] with a copy to [email protected].
Please include ‘Agroforestry EGM and SR consultancy’ in the subject line and mail it no later
than 23:59 GMT, 30 July 2016.
Incomplete applications, including applications without a cover letter, will not be considered.
Only shortlisted candidates will be contacted.
3ie is an equal-opportunity employer fully committed to diversity and equality in all of its
employment practices.
Page 5 of 11
Appendix A: 3ie’s approach to evidence gap maps and systematic reviews
3ie’s Evidence gap maps 4 (EGMs) are thematic collections of evidence on effects covering a
range of international development thematic areas, such as education for peace, HIV and AIDS,
education, and agriculture. They take stock and present a visual overview of existing and
ongoing SRs, and impact evaluations in a sector or sub-sector; schematically representing the
types of interventions evaluated and outcomes reported. A 3ie EGM identifies key gaps, where
little or no evidence from impact evaluations or SRs is available and where 3ie recommends
future research be focused. In general, EGMs have two main objectives:
•
•
Facilitate evidence-informed decision-making in international development policy and
programming by providing user-friendly tools for accessing evidence on a topic, enabling
decision-makers to explore the findings and quality of the existing evidence on a topic
quickly and efficiently.
Facilitate the strategic use of scarce development research funding and enhance the
potential for future evidence synthesis by identifying gaps in the available evidence that
can inform prioritisation of future research.
3ie-funded systematic reviews have these requirements:
•
•
•
A clear theory or theories of change that specifies how, why and under what conditions a
policy or intervention is supposed to work in order to achieved the desired outcomes.
This requires analysis of the evidence from included studies along the causal chain;
A stakeholder engagement and communication plan (SECP), which is a detailed plan for
engaging in ongoing two-way communication and, where appropriate, collaboration with
key users when conducting the SR in order to ensure maximum uptake into policy and
programming; and
The presentation of the findings of a review in a user-friendly style, using language that
does not require a technical background in research or evaluation.
3ie SRs are generally restricted to interventions in L&MICs, though they may sometimes draw
on evidence from high-income countries. A list of 3ie SRs that have been completed, and others
that are currently in progress and is available in 3ie’s SRs database. Guidelines to write a
strong SR application is also available for a better understanding.
3ie partners with the Campbell Collaboration in the production and quality assurance of SRs.
Similar collaborations exist between 3ie and the Cochrane Collaboration, the Evidence for
Policy and Practice Information and Co-ordinating Centre (EPPI Centre), and the Collaboration
for Environmental Evidence. All 3ie-funded review teams should aim to register their reviews
with one of these collaborating groups. 3ie’s staff can advise successful teams on the most
appropriate group with which any particular review should be registered. However, 3ie staff
cannot give advice on individual proposals during the application stage or be named as part of
teams submitting proposals.
4
Snilstveit, Birte; Vojtkova, Martina; Bhavsar, Ami; Gaarder, Marie. 2013. Evidence gap maps - a tool for promoting
evidence-informed policy and prioritizing future research. Policy Research working paper ; no. WPS 6725.
Washington DC ; World Bank Group.
Page 6 of 11
Appendix B: Policy relevance, motivation, and preliminary framework
for agroforestry evidence gap map and systematic review
The role of trees in enhancing and restoring productivity and wellbeing in
smallholder farming systems: A systematic review of the evidence
Objectives • To systematically collect, appraise, and synthesise all available highquality evidence on the effectiveness (and, where possible, costeffectiveness and costs versus benefits) of tree-based interventions for
enhancing and restoring the productivity of and wellbeing in smallholder
farming systems to inform policy and practice.
• To identify key gaps in the evidence-base on the effectiveness of treebased interventions to guide future impact evaluations on this topic.
Policy Relevance Globally, the number of small farms (< 2 hectares) is estimated at 475
million1. They represent only above 12 percent of all farmland, but they
produce the majority of the food consumed in the developing world2. Many of
those that farm these small parcels of land have benefited from the research,
technologies, and extension services associated with the peek of ‘green
revolution’ in the 1960s and 1970s, albeit to varying degrees3. However, the
benefits have come with a price—adverse environmental externalities,
including soil degradation, chemical pollution, biodiversity loss, pest
upsurges, and aquifer depletion4. Indeed, the expansion and intensification
of ‘modern’ agriculture is recognized as a key driver of global environmental
degradation5. This, coupled with decades of underinvestment in the
agricultural sector, have left many smallholders vulnerable and
While the ‘green
impoverished6. Moreover, in contrast with other continents, agricultural
revolution’ brought
productivity in sub-Saharan Africa has actually declined7, which has been
benefits for many, it
locked others out and exacerbated by poor and deteriorating soil fertility8.
generated adverse
Unsurprisingly, a number of alternative concepts and approaches have
environmental
surfaced in a bid to steer global agriculture to a more positive trajectory. In
externalities,
resulting in a plethora general, they share the threefold objective of (1) meeting current and
projected global biomass demand (both for food and bioenergy); (2)
of alternative
sustaining/enhancing land and ecosystem health; and (3) sustainably
concepts and
improving the livelihood prospects and general wellbeing of farmers,
approaches.
particularly smallholders. Popular examples include: sustainable agriculture9;
climate smart agriculture10; sustainable intensification11; agroecology2;
organic agriculture12; permaculture13; conservation agriculture14; integrated
soil fertility management15; and agroforestry (the integration of woody
perennials into farming systems)16.
At the same time, there is considerable interest in and momentum for
restoring degraded landscapes17. This ambition is clearly encapsulated in the
Bonn Challenge—an international platform set up in 2011 to facilitate the
Page 7 of 11
implementation of several international commitments associated with land
restoration—with its ambitious call for the restoration of 150 million hectares
of deforested and degraded lands by 202018. While the global assessment
that informed the setting of this target has been criticized19, 75% of the two
billion hectares of the prospective land it identified for restoration is
earmarked for ‘mosaic restoration.’ Here, forest- and tree-focused restoration
is co-located with agroforestry and smallholder agriculture systems, smallscale settlements, and other appropriate land uses20.
Evidenced-informed
decision-making is
challenged by the
myriad of alternative
options, coupled with
the absence of
synthesized evidence
on their relative
effectiveness and
cost-effectiveness.
A useful way of focusing and structuring systematic reviews is the Population,
Intervention, Comparator and Outcome (PICO) framework.21 The following is
a preliminary one, which the contracted consultancy team can build from:
Population: Farmer field, land unit, livestock population, or smallholder
farming household 5 in low and middle income countries where practices
falling under the definition of agroforestry below are applied, promoted, or
taken up.
Intervention: The application, uptake, adoption or promotion of any land use
system or practice that involves (or has the potential to involve) the inclusion
of woody perennials in smallholder farming systems and their management
in rural landscapes to enhance productivity, profitability, diversity and
ecosystem services and sustainability 28.
Keywords:
Agroforestry,
agro-forestry,
perennial
intercropping,
silvopastoral,
silvo-pastoral,
agro-silvo-pastoral,
silvopasture,
agrisilviculture, improved fallow, tree fallows, fertilizer trees, multipurpose
trees, legume trees, leguminous trees, living fence, contour hedgerow,
alley cropping, alley farming, hedgerow intercropping, biomass transfer,
Farmer Managed Natural Regeneration, perennial garden, shade-grown,
multistory agriculture, taungya, farm forestry
Comparator: Smallholder field/plot, livestock or land unit or smallholder
farming household where any of the practices that fall under the definition of
Preliminary agroforestry above have not been substantively treated.
PICOS Framework
Keywords: impact, evaluation, assessment, effect, effectiveness, costeffectiveness, cost-benefit, efficacy, response
Outcomes: The review will include all relevant, high quality studies
irrespective of the outcomes examined. However, key outcomes of interest
at varies levels include:
5
Applying a universal definition of a smallholder household based on land holding size may prove problematic. For
example, cultivating four hectares of land in an agriculturally productive area cannot be compared with cultivating the
same size of land in a semi-arid, marginal area. There is furthermore significant regional variation is landholding size
among what are locally regarded as smallholder farmers.
Page 8 of 11
Farm: Crop yields; yields of fruit, fuelwood, or other products from perennial
species; total agricultural yield; Total Factor Productivity; yield variability;
farm profitability; farm income
The outcomes
associated with
agroforestry are
broad, so the review
should cast a broad
net to ensure that
these are captured.
Environment: Erosion; soil fertility (organic matter, nitrogen, potassium,
and phosphorous); soil health; carbon exchange capacity; micronutrients;
nutrient cycling; biodiversity; habitat; carbon sequestration; water;
infiltration; run-off; energy; environmental services; ecosystem services;
landscape health; deforestation; re-afforestation
Household: Wealth; consumption/expenditure; income; food security;
dietary diversity; nutrition; climate change adaptation; resilience; risk
mitigation
Study Design: Experimental (RCT), Quasi-Experimental (DID, RDD, PSM,
IV), SRs, SR Protocols, Meta-analysis.
Any empirical study conducted in a low to middle income country that
compares smallholder fields/farms/farming systems/households where
agroforestry has and has not been significantly promoted and/or practiced
based on a credible identification strategy, i.e. random assignment,
instrumental variable, regression discontinuity, difference-in-differences,
propensity score matching, selection and regression models, particularly
where the assumptions upon which they depend have been both clearly
Eligibility Criteria
acknowledged and defended. Rigorous qualitative studies will be considered,
provided that explicit efforts have been made to (a) evidence causal
pathways/mechanisms; and (b) both identify and rule out possible alternative
explanations for any observed changes that have taken place.
Description studies of particular agroforestry systems or practices without
empirical assessments of impact will be excluded, as will ex-ante studies,
those emphasizing the likelihood or potential of impact, or those based on
weak identification strategies. Studies testing the technical efficacy of one or
more woody perennial-based interventions in tightly controlled settings (e.g.
agricultural research stations) will also be excluded. However, the review will
reference and document other systematic reviews and literature that have
analysed, synthesised and/or examined such studies.
Page 9 of 11
References:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
Lowder, S. K., Skoet, J. & Singh, S. What do we really know about the number and
distribution of farms and family farms in the world ? ESA Working Paper 14-02, (2014).
FAO. Smallholders and Family Farmers. (2012).
Hazell, P., Poulton, C., Wiggins, S. & Dorward, A. The Future of Small Farms:
Trajectories and Policy Priorities. World Dev. 38, 1349–1361 (2010).
Evenson, R. E. & Gollin, D. Assessing the impact of the green revolution, 1960 to 2000.
Science 300, 758–762 (2003).
Wade, M. R., Gurr, G. M. & Wratten, S. D. Ecological restoration of farmland: progress
and prospects. Philos. Trans. R. Soc. B Biol. Sci. 363, 831–847 (2008).
IFAD. Smallholders, food security and the environment. (2013).
van Ittersum, M. K. et al. Yield gap analysis with local to global relevance—A review. F.
Crop. Res. 143, 4–17 (2013).
Sanchez, P. A. et al. in Replenishing Soil Fertility in Africa 1–46 (Soil Science Society of
America and American Society of Agronomy, 1997).
Velten, S., Leventon, J., Jager, N. & Newig, J. What Is Sustainable Agriculture? A
Systematic Review. Sustainability 7, 7833–7865 (2015).
Lipper, L. et al. Climate-smart agriculture for food security. Nat. Clim. Chang. 4, 1068–
1072 (2014).
Garnett, T. et al. Sustainable Intensifi cation in Agriculture: Premises and Policies. Sci.
Mag. 341, 33–34 (2013).
Niggli, U. Sustainability of organic food production: challenges and innovations. Proc.
Nutr. Soc. 74, 83–88 (2014).
Ferguson, R. S. & Lovell, S. T. Permaculture for agroecology: Design, movement,
practice, and worldview. A review. Agron. Sustain. Dev. 34, 251–274 (2014).
Hobbs, P. R., Sayre, K. & Gupta, R. The role of conservation agriculture in sustainable
agriculture. Philos. Trans. R. Soc. B Biol. Sci. 363, 543–555 (2008).
Vanlauwe, B. & Zingore, S. Integrated soil fertility management: an operational definition
and consequences for implementation and dissemination. Better Crop. with Plant Food
95, 4–7 (2011).
Nair, P. R. The coming of age of agroforestry. J. Sci. Food Agric. 87, 1613–1619 (2007).
Rey Benayas, J. M. & Bullock, J. M. Restoration of Biodiversity and Ecosystem Services
on Agricultural Land. Ecosystems 15, 883–899 (2012).
GPFLR, G. P. on F. L. R. The Bonn Challenge on Landscape Restoration. (2015).
Veldman, J. W. et al. Where Tree Planting and Forest Expansion are Bad for Biodiversity
and Ecosystem Services. Bioscience XX, biv118 (2015).
Laestadius, L. et al. Mapping opportunities for forest landscape restoration. Unasylva 62,
47–48 (2011).
Higgins JP, T. & Green, S. Cochrane Handbook for Systematic Reviews of Interventions.
Version 5.1.0. (The Cochrane Collaboration, 2011).
Gibbs, H. K. & Salmon, J. M. Mapping the world’s degraded lands. Appl. Geogr. 57, 12–
21 (2015).
Neely, C. & Fynn, A. Critical choices for crop and livestock production systems that
enhance productivity and build ecosystem resilience. (2011).
White, H. & Waddington, H. Why do we care about evidence synthesis? An introduction
to the special issue on systematic reviews. J. Dev. Eff. 4, 351–358 (2012).
Page 10 of 11
25.
26.
27.
28.
Waddington, H. et al. How to do a good systematic review of effects in international
development: a tool kit. J. Dev. Eff. 4, 359–387 (2012).
Buttoud, G., Place, F. & Gauthier, M. Advancing Agroforestry on the Policy Agenda.
Agroforestry Working Paper No.1 FAO, Rome, (2013).
Government of India. National agroforestry policy. (2014).
ICRAF. Strategy 2013-2022. (2013).
Page 11 of 11