A quantitative framework has been
developed to help direct international
agricultural research. It is based on
methods that measurably outperform
current practice. The quantitative analysis
of decisions and the supporting metrics will
have a profound effect on intervention
policy.
Global
Intervention
Decision
Model
Using Applied Information
Economics for Developing
World Agricultural Research
I
Executive Summary
The Global Intervention Decision Model (GIDM) is a probabilistic decision analysis tool produced
through a cooperative effort between the Consultative Group on International Agricultural Research
(CGIAR) and Hubbard Decision Research (HDR). The work was carried out under the CGIAR program on
Water, Land and Ecosystems1 (WLE). The objective was to provide a modeling framework to support
future decisions and to provide guidance for what should be tracked in a metrics database. The
quantitative methods used are supported by published research showing how these methods provide a
measurable improvement on expert decisions done without the aid of such models.
The decision model framework was developed by identifying and creating “pilot” models representing
different types of intervention. Six groups of CGIAR scientists picked an agricultural research proposal or
set of proposals for which we collaboratively built risk return analyses. Each analysis provides a model to
optimize the investment decision and simultaneously updates critical metrics in sustainable agriculture.
We selected each case study from a different research portfolio to sample the different types of
interventions being considered by the WLE program, to help identify what types of information systems
could have high value for improving intervention decisions, and to also reflect the CGIAR objective of
integrating the work of the centers.
The process we use for improving decision quality is based on a probabilistic risk return analysis called
Applied Information Economics, which uses Monte Carlo simulations to produce a distribution of
potential outcomes. This method allows the potential stakeholder to consider uncertainty explicitly and
to calculate the risk of a negative outcome or loss. Another primary output of an Applied Information
Economics model is the calculation of the economic value of information for each uncertain variable2. By
collecting information values on a variety of interventions, we can identify priorities for a metrics
database as well as provide better informed intervention decisions. These metrics could inform CGIAR
research directed toward achieving their four system level outcomes.
The project took place over the first 9 months of 2013, after development of the Global Intervention
Decision Model concepts and project approach at the end of 2012. Individual modeling efforts for
different interventions were completed on different schedules – the completion of the first modeling
effort was in May 2013 and the last intervention model was finished at the end of September. Each
group met for 6-12 workshops, consisting of a calibration workshop, an intensive series of workshops,
follow-up workshops, and a final meeting for presentation and model “hand off” to the group. The
intensive series of workshops typically started with focusing on what decision was being affected. This
intensive workshop period also included selecting variables and starting construction of the model.
1
2
http://wle.cgiar.org/
Information value in this context is a derivation of the benefit of reducing uncertainty on a variable.
I
Workshops were largely done remotely, although twice there were week-long series of workshops done
onsite (one in Nairobi and one in Kunming).
This report contains a summary of our efforts, reviews some key challenges facing CGIAR, and presents
both the current method and a proposed approach for dealing with these challenges. We then give an
overview of the pilot projects, present information values, and present other modeling results. We
conclude by returning to a discussion of next steps for the proposed Global Intervention Decision Model,
as well as suggesting specific actions based on our findings. Here is a summary of key observations in
this study.
1. The risk return analyses had some significant impacts on recommendations for interventions
and even how interventions would be defined. Some investments turned out to be marginal or
negative where others were overwhelmingly positive; these findings often surprised those
involved in the modeling. There were also cases where modifications to an intervention made
the difference in recommendations – such as establishing a higher level of government subsidy
before committing to a human manure (or “humanure”) production facility in Ghana.
2. High priority measurements as identified by the information value calculations were often
surprising. Among the surprising findings of variables with high information value were the
potential “points of failure” in an intervention related to a distributed seed distribution network
in Ghana, and the estimates of effects of an intervention in the Tana River Basin on urban
migration and the economic value of preventing migration. It was unlikely that these variables
would have been chosen for further detailed measurement without the explicit information
value calculations.
3. The process was a new way of thinking for the researchers involved in the modeling of
decisions. Researchers often struggled to identify a development intervention that they were
actively engaged in supporting through their research and for which there was a clear
opportunity to influence stakeholder decisions. This finding supports the current CGIAR drive for
researchers to better articulate impact pathways.
II
Contents
EXECUTIVE SUMMARY ....................................................................................................................... I
CONTENTS ....................................................................................................................................... III
BACKGROUND AND CONCEPTS ...........................................................................................................4
THE CHALLENGE .......................................................................................................................................... 4
CGIAR BACKGROUND .................................................................................................................................. 4
CGIAR’S CURRENT DECISION METHODS ......................................................................................................... 5
THE PROPOSED DECISION METHOD: APPLIED INFORMATION ECONOMICS ............................................................. 5
SOLVING THE THREE CHALLENGES .................................................................................................................. 7
THE GLOBAL INTERVENTION DECISION MODEL ................................................................................................. 7
DESCRIPTION OF THE PROCESS ...........................................................................................................9
MAJOR PROJECT PHASES .............................................................................................................................. 9
DELIVERABLES............................................................................................................................................. 9
AN INTRODUCTION TO CALIBRATION ............................................................................................................. 10
MODELING WORKSHOPS ............................................................................................................................ 11
PILOT PROJECTS OVERVIEWS............................................................................................................ 14
INTEGRATED RAINFED AGRICULTURE – TANA RIVER BASIN ................................................................................ 14
PAYMENT FOR ENVIRONMENTAL SERVICES – SASUMUA DAM............................................................................ 16
BASINS – WATER MANAGEMENT IN THE LOWER MEKONG BASIN ...................................................................... 17
RESOURCE RECOVERY REUSE – HUMANURE FACTORY IN GHANA ....................................................................... 18
IRRIGATION – TORGORME IRRIGATION SCHEME IN GHANA ............................................................................... 19
BIODIVERSITY – OPTIMIZED SEED DISTRIBUTION NETWORK .............................................................................. 20
THE RESULTING GLOBAL INTERVENTION DECISION MODEL ............................................................... 22
COMMON SECTIONS .................................................................................................................................. 23
OTHER SECTIONS....................................................................................................................................... 24
HOW SYSTEM LEVEL OUTCOMES ARE MODELED ............................................................................................. 24
THE VALUE OF INFORMATION ANALYSIS........................................................................................... 25
OVERVIEW ............................................................................................................................................... 25
DETAILED FINDINGS ................................................................................................................................... 26
RECOMMENDED NEXT STEPS............................................................................................................ 29
APPENDICES .................................................................................................................................... 31
APPENDIX A - PROJECT TASK DETAIL ............................................................................................................. 31
APPENDIX B – REVIEW OF PROJECT PLANS AND BACKGROUND .......................................................................... 32
APPENDIX C – DOCUMENT INVENTORY ......................................................................................................... 34
APPENDIX D – BASS MODEL, UTILITY CURVES AND DISCOUNT RATES ................................................................. 35
APPENDIX E: THE NEED FOR AN INTERVENTION DECISION MODEL...................................................................... 37
III
Background and Concepts
The Challenge
In the last 40 years, The Consultative Group on International Agricultural Research (CGIAR) has had an
immense impact on agricultural outcomes around the world. From improvements in crop genetics, to
integrated pest management and biological control, to improvements in livestock and fish production
systems, CGIAR has contributed billions of dollars of value to agricultural systems of the developing
world. Recently, a growing recognition has emerged that climate change will provide new challenges to
agricultural progress in these regions. In addition to this new challenge, there is a sentiment that some
interventions have not been successful due to the complexities of developing world agricultural systems.
These realities contribute to the perception that what has worked in the past may not work in the
future, and that the decision about what to do has become more uncertain.
CGIAR has therefore identified a need to develop a comprehensive, quantitative model to assess the
value of intervention decisions. The primary outcome of this project is the creation of a Global
Intervention Decision Model (GIDM), which can form the basis for prioritizing interventions, determine
how to measure impact on the health of an agro-ecosystem (i.e. on agricultural productivity, the
environment, and human welfare), and calculate the value of the research itself.
CGIAR Background
CGIAR is a collaboration of 15 research centers and hundreds of partner organizations that investigate
sustainable development. Often, institutions conducting research in sustainable agricultural practices
and use of the environment face the following challenges:
1. Estimate the impact of intervention: Institutions attempting to solve big, complex problems like
sustainable development should always have more ideas than they have resources. How do we
decide which ideas to pursue? Or put more formally: given limited resources, how does CGIAR
determine which interventions should have higher priority? Which interventions will reduce risk,
increase security, and improve lives the most? These decisions are always made with significant
uncertainty such as uncertainty about the adoption of a new intervention, about the costs of the
intervention, about the long-term impacts (positive and negative) of any change to a system, etc.
Despite these uncertainties, CGIAR must still be able to make recommendations about how to use
limited resources. In the face of so many uncertainties, what is the best process for researchers
deciding which interventions to research and recommend?
2. Determine how to measure agro-ecosystem health: There are vast combinations of data that could
be gathered about the health of an agricultural system or ecosystem – but not all metrics will be of
equal value when it comes to supporting important intervention decisions. Of all of the variables
that might be an indication of health, some will have more direct impact on real decisions than
others. And even if a variable may have an impact on intervention decisions, costs of gathering data
will vary greatly. How should researchers determine what data gathering costs are justified?
4
3. Show the value of research: Because donors would often rather directly fund intervention
programs than support research, researchers must be able to show how the expense of research is
justified by better intervention decisions.
CGIAR’s Current Decision Methods
The prevailing method for making decisions on project implementations is to have a table of activities
and outcomes (e.g. a log frame). Recently, researchers and aid organizations have spent time on impact
pathways and outcome mapping, reflecting an increased interest in impact. Currently, the practice of
carrying out a cost/benefit analysis is rare while use of a probabilistic method in constructing
cost/benefit analyses is rarer.
For making decisions on topics of research, the current method of decision analysis usually involves
some combination of researcher logic (and experience) and input obtained by interviewing stakeholders.
There are isolated examples of hiring consulting firms to do a risk assessment on a project related to
research but largely these decisions are made on the basis of internal dialogue among researchers and
external dialogue with stakeholders. While the existing process is important, it seldom involves a
quantitative calculation of information values or a true quantitative risk analysis.
The Proposed Decision Method: Applied Information Economics
The World Agroforestry Centre (ICRAF), one of the research centers of the CGIAR, has identified a
consolidated approach that will address all three of the challenges outlined above. The solution will
involve the use of HDR’s primary method called Applied Information Economics (AIE). Applied
Information Economics (AIE) was developed as a robust method for addressing investment dilemmas
that are large, risky, and full of difficult measurements. It is designed to perform even in the presence of
“intangibles” and significant uncertainty. This approach is well suited to developing world agricultural
research because decisions often involve opaque actors (e.g., stakeholder governments), poor and/or
unreliable data (e.g., pastoralist regions), and effects that are perceived as difficult to measure (e.g., the
effect of climate change on agriculture).
Unlike traditional methods that produce arbitrary "scores" or deterministic returns on investment, AIE
conducts a true "Risk/Return" analysis with the same degree of rigor used by actuaries to estimate loss
rates in insurance pools. The method involves five steps – define the decision(s), model what we know
now, compute the value of information, measure what matters, and make better decisions.
5
Applied Information Economics (AIE) combines several methods from decision theory, economics,
actuarial science, and other mathematical methods. The method has been widely used in business,
governmental, and NGO settings – in decisions as diverse as wildlife preservation, mine flooding, and IT
security. AIE makes use of methods that have been shown to improve on human expert judgments in
multiple independent studies. Here is a brief summary of the method.

Define the Decision(s): As obvious
as this step may first appear, it is the
key to better understanding what to
measure and real decisions are often
different from what they first appear
to be. Is the dilemma whether to
simply approve a project or how to
conduct a project given a vast
combination of alternatives? Or is
the decision a matter of when a
given initiative should be approved?
The costs, benefits, timing, risks and
even external factors are identified
and the real decision is clarified.

Operations
Research
Actuarial
Science
Economics
Applied
Information
Economics
Measurement
Science
Options
Theory
Modern
Portfolio
Theory
Model What We Know Now: Cost estimates, forecasts of benefits, project risks, and other
variables in a typical big investment decision are almost never known exactly. The uncertainty
about some variables, especially long term forecasts, can seem extreme. But the consequences
of even extremely uncertain variables can be assessed using the “Monte Carlo” method and a
special method for training experts to assess probabilities. The Monte Carlo method is a
method for conducting decision analysis by sampling variables that do not have exactly known
values (i.e. most variables in a model). This initial model is effectively a snapshot of the current
state of uncertainty about a problem before additional measurements are made.

Compute the Value of Information: Not all variables in a decision model are worth
measuring and those worth measuring are often a surprise to the decision makers. In fact,
normally a kind of “measurement inversion” exists in most decisions – that is, the most
uncertain variables tend to be ignored while the variables that usually receive a lot of attention
actually have less bearing on the decision. With AIE, every variable in a model will have an
“information value” that allows identification of high value variables in a decision. This
approach targets only the variables in a decision that are the most likely to significantly reduce
overall uncertainty in the decision.

Measure What Matters: Once the high-value measurements are identified, a variety of
empirical methods can be used. Contrary to what is sometimes assumed, relatively little data or
simple observations may be required for extremely uncertain variables. AIE often uses efficient
“Bayesian” methods, which exploit prior knowledge and can be used even when data is messy
6
or sparse. The measured variables will have less uncertainty and then the model of uncertainty
can be updated.

Make Better Decisions: The output of the Monte Carlo model, updated with targeted
measurements, is compared to the risk/return preferences of the organization. Research shows
that the actual risk aversion and other preferences of decision makers changes frequently and
unconsciously. Different preferences are applied to different investments even when
management believes they are being consistent. AIE addresses this major source of decision
error by quantifying and documenting preferences such as risk tolerance and the value of
deferred benefits so that the results of analysis can be assessed in a controlled, uniform
manner. Finally, sometimes decisions have large combinations of outcomes and have to be part
of a portfolio of decisions. When necessary, AIE applies optimization methods to determine the
best decision even from a large set of alternatives. The AIE process can help scientists and
planners to clarify and improve intervention decisions even in complex multi-stakeholder
situations.
The Global Intervention Decision
Model
Solving the Three Challenges
The GIDM is both a template and a methodology
for evaluating interventions and research (See
concept note, Appendix E). The template
consists of a set of sections and variables
common to many of the potential interventions
in the CGIAR universe. The template also
includes specialized utility curves such as income
utility and an environmental discount rate to
properly address common preferences in the aid
community such as targeting vulnerable
segments of the population for benefits and
properly emphasizing sustainable solutions.

An intervention decision model built on the
principles of AIE simultaneously addresses the
three challenges listed on pages 4-5. Even
uncertain and difficult to measure interventions
can be assessed and prioritized. And because of
the information value calculations, the best agroecosystem health metrics can be identified and the
value of research itself can be articulated.
Given the critical nature of the decisions CGIAR
supports, perhaps the single most important task
ahead is to determine the best way to make
decisions. Uncertainties and risks must be
quantified, policy preferences must be clarified,
and the measurements most important to the
decisions must be identified. The GIDM is a major
advance in this direction.
Forecasts of Objective Outcomes: The
costs of the intervention may be
uncertain and the long-term effects of
any intervention will be uncertain. The
GIDM will determine the uncertainty of
onsite and offsite impacts as well as
behavioral factors like the adoption rate
of a new practice or how incentives change behavior. Some of the elements of this model will
be based on known science such as yield improvements from additional irrigation. But some
factors, such as long-term changes in behavior will be much more uncertain. Quantifying the
difference in this uncertainty will be critical in determining what to measure.
7

Quantified Preferences & Policies: Preferences about what risks are acceptable, how to value
long-term effects, or the value of equitable improvements in income, need to be quantified and
documented as a matter of policy. These preferences are captured as a set of “utility curves”
that make policies – such as the relative value of a near-term certain impact versus a long term
and uncertain impact – unambiguous. Such clarity will mean that various interventions can be
evaluated against the same standards of risk aversion and other preferences.

Quantified Values: Ultimately, the effects of an intervention and the quantified preferences are
combined into a single monetized value so that interventions of different types and sizes can be
compared. Each intervention creates a set of estimated impacts over a period of time. The
timing and uncertainty of these impacts are adjusted so that they can be rolled into a single
number. The quantified values can also adjust outcomes for differences in how benefits of a
program are distributed equitably. Separately, the GIDM can assess the likely impacts on
individual System Level Outcomes (SLO) such as health, food security, poverty, and
sustainability.
Figure 1: Initial Outline of the Global Intervention Decision Model
Quantified Values
Certain Present Value
Equivalent
System Level Outcomes
(Poverty, food security,
sustainability, health &
nutrition)
Distribution for
Aggregate Present Value
Forecasts converted to monetized net
impact, by year, with uncertainty
Objective
Forecasts
Intervention
Cost
Onsite &
Offsite
Impacts
Provisioning
Supporting/Regulating Services
Proposed Programmatic Interventions
8
Quantified
Preferences
Certain Value
Curve
Time Value
Curve
Marginal Income
Value Curve
Behavioral Change
(adoption, crop
choices, practices,
etc.)
Description of the Process
Major Project Phases
The Global IDM would be based on a series of pilot projects representing different types of
interventions. Six different intervention types were chosen and subject matter experts were recruited
to participate in each of the projects. The model structures from each of these intervention pilot
projects were then combined to create the global model. For each of the pilots, there were five major
tasks as follows (See Appendix A for detailed task list):
1. Decision Clarification Workshop: The purpose of this half-day workshop was to
remove the ambiguity about what the decision was and the objectives of the
investment. We asked questions like “what is the purpose of the investment, project,
or intervention?” or “What are the choices to be analyzed? What are the decision
criteria?”
2. Calibrated Probability Training Workshop: This half-day workshop trained the
participants to assess uncertainties in a quantitative manner. Each participant
became “calibrated” so that estimates they gave could be expressed probabilistically.
3. Detailed Decision Modeling Workshop(s): Much of this work was done in an
“intensive” period of half-day workshops where the teams built the detailed decision
model. In some cases the intensive workshops started with more decision
clarification, in which case the modeling was moved to follow-up workshops. Every
variable in the model was estimated by the “calibrated estimators.”
4. Risk/Return Analysis and Preparation of Deliverables: This takes the input of all
previous steps to produce a quantitatively-sound and complete analysis of the
proposed investment.
5. Value of Information Analysis (VIA): This step computed the economic value of
measuring each of the uncertain variables. The teams gained insight regarding which
variables to measure in more detail and how much measurement effort is required
and justified.
Deliverables
The deliverables for this project included:
1. A detailed spreadsheet model of the decision which uses probabilistic methods to
assess a decision
2. A “value of information” (VIA) analysis showing the economic benefit of measuring
each uncertain variable in the investment so that effort can be spent measuring the
right things
9
3. A risk/return analysis of the proposed investment including a “probability
distribution” of the possible returns for the project or investment and how the
investment compares to the risk tolerance of the organization
4. Recommendations on what to measure to reduce uncertainty and risk in the
intervention
5. A summary presentation of the findings
An Introduction to Calibration
The order of our workshops is an important aspect of the process. Calibration workshops come before
detailed decision modeling because AIE decision models are built with ranges of uncertainty on many of
the variables. Therefore, before a subject matter expert or participant can contribute ranges on a
variable, they must be able to accurately assess their uncertainty. This skill – the ability to accurately
assess one’s uncertainty – can be taught and we call this process “calibration.”
Following methods designed by various academic researchers 3,4 and Doug Hubbard5, experts can
measure how well they subjectively assess uncertainty with explicit probabilities. The vast majority of
people enter training in a state of overconfidence – they predict they will be correct more often than
they are. In other words, when most people say they are 90% confident in each of some large number of
predictions, the frequency of correct answers will be significantly less than 90%. Once an initial
assessment has been conducted, experts learn several techniques for achieving a measurable
improvement in estimating. By the end of a 3 hour training workshop, 85-90% of participants achieve a
state of calibration – that is they are able to give estimates which are correct as often as they predict
them to be. Even those who don’t achieve calibration in the workshop can still participate once their
overconfidence has been measured.
The experience with the researchers was consistent with observed results for professionals in many
other fields. Virtually all researchers started out in a state of extreme overconfidence about their
estimates. But after training most were performing to almost an ideal level of calibration (i.e., they
could not be statistically differentiated from ideally calibrated persons given the samples size of
estimates they provided).
3
D. Kahneman and A. Tversky, "Subjective Probability: A Judgement of Representativeness". Cognitive Psychology.
4: p. 430-454(1972).
4
D. Kahneman, P. Slovic, and A. Tversky, Judgement under Uncertainty: Heuristics and Biases, Cambridge:
Cambridge University Press. 1982.
5
D. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business: John Wiley & Sons. 2007
10
Modeling Workshops
The modeling workshops begin by defining the specific decision that the group chose to analyze.
Arguably the most important (and often the most difficult step) is specifying what decision is actually
being evaluated. In our work with CGIAR groups this step proved the most challenging; often
participants were not accustomed to thinking in terms of directing uncertainty reduction in the context
of decision analysis. Researchers often presumed that some study or type of intervention is desirable
but did not have specific actions in mind. We thought this experience was noteworthy and participants
commented that gaining a decision analysis perspective was a positive externality of their participation.
Uncertainty in the model is assessed using a Monte Carlo simulation. This is a way of computing the
uncertainty of a system or outcome given the uncertainty of the inputs to the model. A Monte Carlo
simulation method is used for multiple reasons. First, there is evidence that those using Monte Carlo
simulations are better at forecasting than those who do not6,7. Second, these decisions have significant
risks and uncertainties and there is conclusive evidence that left to their own intuition, even
quantitatively sophisticated decision makers will introduce several types of inference errors when it
comes to the use of probabilities to describe uncertainty and risk8. A Monte Carlo simulation will make
the mathematics of these inferences explicit and avoid several types of inference errors. Finally, Monte
Carlo simulations are often the only mathematical solution to assess a large number of uncertain
variables in complex relationships.
Hubbard Decision Research had previously developed an Excel-based Monte Carlo tool for these types
of uncertain decision models. The calibrated researchers defined the decision model in this Monte Carlo
model and populated the model with their initial calibrated estimates. A key part of this process is
balancing analysis effort with uncertainty reduction – the more aggregated a model the higher the level
of uncertainty; the more detailed the model, the more time it takes to complete the process of
composing logic and gathering estimates. The most efficient method is to err on side of less detail –
building a simple model that works – and make decompositions and refinements once there is a working
model.
The HDR Monte Carlo spreadsheet tool consists of an “Inputs” tab which summarizes all of the
important variables that go into the decision. We divide the variables into sections and each of these
sections has both certain (deterministic) and uncertain (probabilistic) elements. We elicit estimations of
each unknown variable from (the now calibrated) participants and the estimations include a best
estimate and a range which represents a 90% confidence interval (See Table 1) or, in some cases, binary
probabilities (such as the probability of a project failure or drought in a given year).
6
D. Hubbard The Failure of Risk Management: Why It’s Broken and How to Fix It, John Wiley & Sons. 2009
Fiona Macmillan. “Risk, Uncertainty and Investment Decision-Making in the Upstream Oil and Gas Industry.” PhD
diss., University of Aberdeen, 2000.
8
D. Kahneman, P. Slovic, and A. Tversky, Judgement under Uncertainty: Heuristics and Biases, Cambridge:
Cambridge University Press. 1982.
7
11
The best estimate values of the variables feed into an analysis of costs and benefits (Table 2) over a
period of time (usually models are between 5 and 20 years). The range estimates flow into an analogous
probabilistic cash flow calculation using the same calculations and logic as the deterministic analysis.
Table 1: An illustration of part of the input worksheet for the Sasumua model
Lower Bound
Formula &
Best Estimate
Increase in use of fertilizer over period
122%
250%
600%
Triangular
Fertilizer threshold for eutrophication
175%
300%
600%
Triangular
Inputs - Variable Name
Upper Bound
Distribution
Type
Water Quality
Cost of Fertilizer Runoff
Removing algae growth $/year
$
5,000
$
9,758
$
100,000
Triangular
Cost of chemically cleaning water
$
50,000
$
200,000
$
500,000
Triangular
<----click here to expand/shrink sections
Goodwill Variables
Reduction of legal liabilities
Frequency per year
-
Cost per incident
$
Reduction, attributable to PES
2
50,000
$
5%
Triangular
5
50,000
$
25%
200,000
50%
Log Normal
Triangular
Blocking of intakes
Frequency per year
…
2
16
…
…
12
32
…
Triangular
…
Table 2: Snapshot of part of the Tana Model showing best estimates for adoption variables and how
they flow into the deterministic version of cash flow calculations.
13
Pilot Projects Overviews
In selecting interventions and groups we largely followed the original proposal. Table 3 summarizes the
pilots chosen including the selected decision, a brief description of the project, and the parties involved.
The primary differences between pilots chosen and the original proposal were the added modeling
efforts for the Sasumua Dam and the changed subject matter for the irrigation intervention. The
remainder of this section describes each pilot project and its initial findings in further detail. Most
showed a potentially very positive, yet highly uncertain impact.
Table 3: Actual Research Groups and Projects for Investigation
Focus/Group
Name of Intervention
Core Team
Description of Actual Project
Rainfed
Agriculture, ICRAF
Tana River Basin
Shepherd, De Leeuw,
Hubbard, Millar
Design of integrated interventions to increase
productivity and resilience of rainfed agriculture in
Africa; Stakeholder: Benevolent Sponsor
Sasumua Dam,
ICRAF
PES/Sasumua
Namirembe, Gathenya,
Nyongesa, Mwangi, Millar
Payment for Environmental Services (terraces and
grass strips) for Sasumua Dam, Kenya; Stakeholder:
Nairobi Water Company
Basins
Laos Water Management
on Mekong River
McCartney, Rebelo,
Luedeling, Hubbard,
DauSchmidt
Comparing two dams and alternative water
management interventions; Stakeholder: Laotian
Government
Resource Recovery
Reuse
Humanure Factory
Gebrezgabher, Nikiema,
Manning, Millar
Business case for manufacturing fertilizer from
human manure. Stakeholder: Entrepreneur
Irrigation, GIDA
Torgorme Irrigation
Scheme
Bell, Tawiah, Gbireh,
DauSchmidt
Evaluating an Irrigation Scheme in Togorme, Ghana
to measure viability over a 10 year period;
Stakeholder: Ghana Irrigation Development
Authority
Biodiversity,
Bioversity
Optimized Seed
Distribution Network
De Clerck, Bruskiewich,
Arnaud, Fadda, Atieno,
Clapp, Millar
Constructing a regional system of seed storage
nodes in the Volta Basin. Stakeholder: Benevolent
Sponsor/Aid Organization
Integrated Rainfed Agriculture – Tana River Basin
The Tana model considers a variety of interventions related to intensification of rainfed agriculture
including proposed changes in soil management, water resource management, intensification of trees,
and improved mixed livestock and grazing systems. We specified the stakeholder selected in this
intervention – an actor from the government or aid community whose preferences on outcomes can be
specified – as a “benevolent stakeholder.” The scale of the model is a large river basin, incorporating
14
interventions in three separate climactic zones across Kenya. Given the complexity and size of the
proposed model, our initial effort was done on an aggregate level, meaning many of the variables were
estimated as averages or estimated for an entire system.
This aggregate approach has both advantages and disadvantages: while it might appear that we are deemphasizing variables that might be the specific focus of agricultural research choices, it allowed the
group to “see the forest for the trees” in the sense that we were targeting the most significant variables
for system wide success. In no way does this imply that there would not be high information values
found in a more granular analysis of each type of intervention and zone. On the contrary, information
values were high enough to justify a model built around specific individual interventions and even
specific variables. However, without first modeling on the aggregate level, we may not have seen the
relative importance of uncertainty reduction on some of the system level variables such as urban
migration and climate change. The Tana model included all but one of the proposed “common sections”
of the Global IDM model which we outline later in this report.
We used a total of 83 uncertain variables in this model. The model indicated a highly uncertain
intervention outcome – there was a high risk of making the wrong choice and the potential size of loss
from making the wrong choice was large. In 10,000 Monte Carlo scenarios, the range of impacts was
negative $5 billion to positive $10.5 billion. Of the 10,000 scenarios, 74% were calculated as a net
benefit over the whole system and 26% were calculated as a net loss. The net benefit averaged over all
scenarios was $779 million.
Figure 2: Histogram generated from 10,000 Monte Carlo simulations of the Tana interventions
The enormity of the impact reflects the scope of the project – affecting millions of Kenyans over a 20
year period. The fact that the range is so wide, and the trials so evenly split between negative and
positive outcomes, means that the value of reducing uncertainty was also large (Table 5, we discuss
value of information in a later section).
15
Payment for Environmental Services – Sasumua Dam
The Sasumua model provided an analysis on the value of an investment in grass strips and terraces in
the upland watershed of the Sasumua catchment in Kenya. The stakeholder in this model is the Nairobi
Water Company (NWC) – although they did not participate in the modeling effort, the research group
had significant contact with the stakeholder and the model is built around the financial impact on NWC.
We estimated the annual cost the NWC incurs from cleaning the water of excess sediment including
chemical costs, direct and indirect costs of cleaning filters, and loss of reservoir capacity over time. We
also included a cost of flooding which included both direct (flood mitigation and damage) and indirect
(public relations) aspects. All of these costs would potentially be altered by an investment in grass strips
and terraces – so we compared the cost/benefit of this Payment for Environmental Services scheme
with the baseline costs they would otherwise incur. Because we were modeling from the perspective of
the NWC, benefits to the upland farmers were considered positive externalities and did not affect cash
flow calculations.
The Sasumua model has 58 uncertain variables and includes a climate section to predict the effects of
variation in precipitation. This model had a relatively large percentage (88%) of positive Net Present
Value (NPV) outcomes in the Monte Carlo simulation. The range of outcomes was from a loss of $1.99
million to a gain of $27.7 million over a twenty year period. The average NPV was a gain of $1.96 million
for the Nairobi Water Company. Given the relatively low percentage of losing scenarios, the value of
information was much more modest than in the Tana model.
Figure 3: Histogram generated from 10,000 Monte Carlo simulations of the Sasumua intervention
16
Basins – Water Management in the Lower Mekong Basin
This model compares several different options for water management in the Lower Mekong Basin in
Laos:
a. Building a hydroelectric dam at Don Sahong
b. Building the Thakho hydroelectric structure combined with some investments in alternative
water management
c. Building alternative water management systems such as small reservoirs and ponds
d. Doing nothing to alter the current Lower Mekong Basin
The stakeholder for this model is the Laotian government. In making the comparison, we assumed that
approximately equivalent monetary investments would be made in each of the first three options. We
consider the costs and benefits from hydroelectricity generation, food production, flood reduction, and
several other factors in determining the optimal investment decision.
This model was one of the larger with 146 uncertain variables and comparing three intervention plans to
a baseline scenario of no intervention. Additionally this model is over a comparatively long time period,
40 years, and looked at the effects on a relatively large geographic region, the Lower Mekong Basin. The
cumulative effects of both “good” and “bad” interventions are quite large in magnitude.
The histogram below (Figure 4) clearly shows that the preferred course of action is the alternative water
management intervention. This approach consists of the construction of small reservoirs, wells, and
water pumps to increase agricultural yields across the region.
Figure 4: Comparison of 10,000 Monte Carlo simulations for each Mekong Basin option
17
Resource Recovery Reuse – Humanure Factory in Ghana
The RRR model was developed to provide the entrepreneurial investor with a means of objectively
examining the potential benefits of a project aimed at processing human waste matter into an organic
fertilizer. In the many areas of sub-Saharan Africa considered within the scope of this model, the
current approach towards treatment of waste matter, including human excreta, involves little or no
processing of the waste matter.
This model will assist the entrepreneur in ascertaining whether investment in a facility capable of
processing raw fecal sludge into an alternative, organic fertilizer product is justified in the sense that the
entrepreneur may reasonably expect to realize a return on their investment.
Since this particular model has been constructed from the point of view of a commercial entrepreneur
rather than from the point of view of a philanthropic benefactor, many of the secondary benefits to the
region (e.g., increased crop yields due to soil building, improvements to public health due to increased
sanitation) do not play into the model with much weight. While these benefits are significant, and may
even be primary considerations for a philanthropist or country stakeholder, they do not directly impact
the entrepreneurial investor’s bottom line and as such must carry little to no weight for this decision
making process.
Given that the estimated costs provided for this model were relatively higher than estimates provided
for benefits, the model output indicates that this would not be an attractive investment for an
entrepreneur. In roughly 80% of Monte Carlo scenarios the direct benefits to a potential entrepreneur
are exceeded by the costs that they are expected to incur, often by a significant margin. The average
NPV for the intervention is negative 1.5 million Ghana Cedis, with roughly 80% of the scenarios resulting
in a negative NPV.
Figure 5: Results from 10,000 Monte Carlo simulations for the RRR model
600
NPV Histogram for the RRR model
500
Frequency
400
300
200
100
0
(₡10)
(₡8)
(₡6)
(₡4)
(₡2)
NPV
18
₡0
₡2
₡4
Millions of Ghanaian Cedi
Our subject matter experts acknowledged some level of conservatism with respect to the cost/benefit
estimates and expressed an intention at some point in the near future to present the model to other
thought leaders in this field for additional input. The result of these subsequent conversations may be
updated variable estimates, or perhaps the identification of previously unidentified benefit streams that
directly impact the potential investor.
Irrigation – Torgorme Irrigation Scheme in Ghana
This model was built to determine the risks and returns from the Torgorme irrigation scheme either with
or without an initial grant from the United States. The stakeholder in this model is the Ghana Irrigation
Development Authority (GIDA), which is the scheme management entity and one of the stakeholders in
the irrigation scheme, which is currently under construction. The Scheme Management Entity (SME),
which includes farmer representatives, public, and private employees, is the umbrella entity for all
stakeholders involved. The scheme is designed to help small landholders in Farmer Based Organizations
receive a transfer of knowledge concerning irrigation methods in addition to increased access to
international markets. Anchor tenants consisting of larger, more advanced farmers serve as the
propagators of this increased knowledge and access to markets. The Ghanaian government intends to
partially support the scheme during the initial years as the farmers increase crop yield towards full
capacity.
The catalyst for this project was a grant from USAID for the vast majority of the construction costs.
Since the initial decision and funding had already been made, the decision we modeled centers on
whether the scheme is still profitable after 10 years. The histogram below shows two outcomes: if the
Scheme Management Entity had to come up with the construction costs equivalent to the size of the
grant themselves (“Scheme NPV without grant”); and if the Scheme Management Entity is only
responsible for the ongoing costs not covered by the initial grant (“Scheme NPV including grant
money”).
19
Figure 6: Results from 10,000 Monte Carlo simulations for the Irrigation Model
The scheme’s success appears to depend on the initial grant money—with an average positive return of
approximately 8.5 million Ghanaian Cedi with the grant but an average negative return of 22.2 million
Cedi without the grant. Since these are preliminary results, there could be significant revisions based
upon further communication with our contacts at GIDA.
Biodiversity – Optimized Seed Distribution Network
The Optimized Seed Distribution Network is an analysis of proposed funding for a seed distribution
network in the Volta region of Burkina Faso and Ghana. Like the Tana model, this would be constructed
from the perspective of a “benevolent stakeholder.” The proposal would aim to increase farm yields and
decrease farm income variability by creating access to a wider diversity of seeds per crop and
encouraging a wider diversity of crops per farm. The benefits of this proposal would be achieved
through several mechanisms:
a. Increasing genetic diversity would reverse increased system vulnerability from genetic
erosion, (potentially) decreasing volatility in farm income
b. Farmers could increase yields by increasing access to seeds specialized to their relevant
micro-climates
c. Seed specialization would potentially cut down on the need for fertilizer, irrigation, and
other input costs
20
d. Genetic diversity would potentially increase resilience to climate change, meaning the
effects of increased average yield and decreased annual farm income volatility could grow in
magnitude over time
The costs involved in this proposal include initial set-up costs for brick and mortar, inventory system,
and IT costs such as climate information, pest and disease information, and market information.
Ongoing costs would include storage, transportation of seeds, and maintenance of the information
systems.
This model considered 74 uncertain variables, ran over a time frame of 50 years, and was the third
model to consider an entire basin (the Volta basin in Ghana and Burkina Faso). As we would expect, the
range of results is commensurately large.
Initial results from the Optimized Seed Distribution Network suggested that a positive NPV for the
business case was essentially a sure thing. 99.7% of scenarios had a positive NPV and the average over
50 years was a $54 billion benefit.
Figure 7: Results from 10,000 Monte Carlo simulations for the Biodiversity Model
While highly uncertain, the initial results showed that a positive return is virtually certain and that
extremely high returns are a good possibility. Is it truly a case of new information coming to light that
would create a breakthrough in productivity? Or are there other possibilities that we aren’t properly
considering that might interfere? In this case, we determined it was the latter – there were at least five
“points of failure” that weren’t originally considered:
1.
2.
3.
4.
Significant infrastructure degradation
Poor quality control of seeds coming into the system and matching seeds to needs
A communication breakdown that meant farmers lost access to the system
Information system error
21
5. Political instability/war/conflict
Each of these points of failure had a significant chance of occurring and a significant potential negative
effect on outcome. With their inclusion, the picture had changed significantly. Now only 82% of
scenarios had a positive NPV, meaning the possibility of a negative NPV had increased by a factor of
fifty. The average NPV was still quite high at $17 billion but was less than 1/3 of the initial result.
The Resulting Global Intervention Decision Model
The cases modeled here are examples - from a variety of potential interventions - of large investment
decisions that have significant decision risk. From these a common framework was identified based on
model components that appeared in multiple pilots and would be likely to appear in future modeling
efforts as well. Figure 8 below illustrates the major components. Further detail on components
common to all models, components unique to some models, and how System Level Outcomes are
modeled is provided in the text below.
Figure 8: Schematic of Model Construction
22
Common Sections
1. Climate – usually a rain/drought model that generated stochastic annual or seasonal
precipitation amounts. Average precipitation and precipitation variation are estimated. Monte
Carlo selects a random scenario for the “true” value of each variable. These then use another
random number to generate each season’s “actual” rainfall. Simple models like this would be
made more robust (relationships between consecutive seasons, better specification of the
distribution of precipitation, etc.) if the variables had high information value. This section is
critical in a wide variety of possible interventions – from the effect it has on resilience outcomes,
to crop yields, to dams flooding.
Table 4: Average precipitation and precipitation variation feed into the random cash flow page. A
Monte Carlo simulation runs thousands of such scenarios.
2. Geography and Demographics – This includes variables such as the number of hectares in
different zones or basins, percentages of land in agriculture versus agroforestry versus livestock,
density of farmers.
3. Adoption – This includes variables such as the maximum projected adoption of a practice or
intervention, the rate of adoption, the likelihoods of project cancellation, the likelihood of
abandonment or individual cancellation, and other variables likely to affect rates of adoption.
Given these parameters we use an S-curve derived from the Bass Model (see Appendix D) to
calculate annual adoption rates.
4. Utility Curves – We use three utility curves (See Appendix D for more details) to show the
preferred tradeoffs for the value of time, income distribution and risk. One utility curve
providing a time discount for future costs and benefits is a standard element of any cash flow
analysis. We settled on use of the “environmental discount rate” as a way to capture both longterm concerns and large short-term discount rates. The second utility curve is an income
distribution utility curve which allows the stakeholder to specify preference for benefits accruing
to recipients at different income levels. The final utility curve called the risk equivalence utility
curve helps decision makers make consistent choices when deciding which projects to fund
based on their risk/return profile.
5. Macro-Financial and Commodity – This includes crop prices and yields, chemical and other
input prices, and interest rates.
6. Costs – The cost section consists of both initial costs (usually the initial donor grant) as well as
ongoing costs which includes both donor and participant costs required for maintenance of the
23
intervention (such as annual labor costs for care of trees) and indirect costs (such as increased
displacement, pollution, or erosion that results from an adopted practice).
a. Initial costs
b. Ongoing or annual costs
i. Direct
ii. Indirect (negative externalities)
7. Benefits – The benefit section is divided in a similar way to costs, and includes both onsite and
offsite benefits, as well as both direct and indirect benefits.
a. Onsite benefits
i. Direct
ii. Indirect (positive externalities)
b. Offsite benefits
i. Direct
ii. Indirect (positive externalities)
Other Sections
In using the Global IDM, an individual project may require a section that is relatively unique. In this case,
it might not make sense to include it in the global framework because it won’t have a common enough
application. These sections could be a detailed or expanded version of one of the existing sections (such
as a complicated demographics model or a detailed section on fish and fishing). Other possibilities
include:
a. Politics and their effect on intervention outcomes
b. The possibility of war and its effect on outcomes; in our models we modeled shocks as
coming from climate, but war is another common source of shock
c. Farm income volatility and its effect on bankruptcy rates and investment levels
d. Food security and its effect on other elements in the model
How System Level Outcomes Are Modeled
The modeling effort showed that the System Level Outcomes (SLOs) - reduced poverty, improved food
security, sustainability, health and nutrition - are not entirely orthogonal measures; the SLOs are highly
interrelated. The effects of one SLO are often implied when other SLOs are modeled.
For example, up to a point food security and health can be “bought” with higher incomes. So increased
incomes do have an effect on food security and health just as they would (by definition) on reduced
poverty. If an intervention program improves health or food security, the decision maker can consider
how much incomes would have to be raised to get an equivalent effect. This would often require direct
increases in income much larger than what could be achieved with direct improvements in food security
and health. Likewise, methods that improve the yields of farms improve food security even though this
24
benefit was often monetized by computing the yield at the market price. This would be both a
reduction in poverty as well as an increase in food security.
Sustainability was another SLO that was indirectly modeled in the pilots. Interventions can reduce the
risk of long term negative impact on land and other resources. The pilots addressed this by modeling
uncertain future outcomes. In effect, sustainability is realized as the reduction of the probability of
some future negative event, the reduction in the magnitude of this event, and even the delay of the
event. While simply “delaying” a negative event may not always be perceived as sustainable, it is
considered an improvement worth including in the model. As long as this delay is not offset by
increased likelihood and magnitude of future negative events, it should be counted as a net gain.
As a result, other than reduced poverty (through increased income), none of the pilots specifically
mentioned the SLOs by name. Yet the SLOs are fully included in the Global IDM just the same.
The Value of Information Analysis
Overview
Not only does this analysis produce a quantitative picture (Figures 4-9) of the uncertainty facing decision
makers for any given intervention, it also delivers another important result: it mathematically derives
the value of reducing uncertainty on uncertain variables in the decision making process. Thus, even as
we focused on evaluating individual investments, the discovery of information values of the variables in
these decisions was another primary outcome from this effort.
Of course, it is not practical to reduce uncertainty on all variables in a model. If there are 5 dozen
uncertain variables, each of which could be decomposed into a further dozen variables, it is obvious that
decisions need to be made to direct research toward some variables and away from others. The best
way to prioritize these measurement efforts is to use a method developed in game theory and decisions
theory for computing the economic value of information. This is based on the idea that uncertainty in
decisions results in the risk of an opportunity loss when the wrong decision is made. Reducing
uncertainty on some variables has a large impact on the reduction in this decision risk. This change in
monetized risk from measuring a particular variable is the value of information for that variable.
Unfortunately, there is evidence that experts often choose to focus on measurements that are less likely
to inform uncertain decisions. It has been shown, in fact, that there is almost an inverse relationship
between the measurement effort focused on a variable and its information value 9. This phenomenon is
known as the “Measurement Inversion” and it pervades many – if not all – professions and industries.
For complex decision models with a large number of variables, it is even very difficult for an expert –
without the aid of information value calculations – to choose the measurements that would have the
9
D. Hubbard, How to Measure Anything: Finding the Value of Intangibles in Business: John Wiley & Sons. 2007
25
greatest benefit for uncertain decisions. Given the vast array of variables to consider in agricultural
interventions (our six models had roughly 500 uncertain variables), the process of calculating
information values will help inform CGIAR on future research efforts.
One particular pilot project provides a useful illustration of the Measurement Inversion even during
initial modeling efforts. For the biodiversity project the team held five modeling workshops, exchanged
many dozen emails related to modeling and estimation, and cumulatively spent hundreds of hours
modeling. The six variables with the highest information value were discussed less than an hour. The
only defense against this misallocation of effort appears to be deliberate calculations of the value of
information.
The value of information can be approximated first by computing the Expected Value of Perfect
Information (EVPI). In its simplest form, the EVPI is the cost of being wrong multiplied by the chance of
being wrong. In the more complex situations we assessed, however, the calculation becomes more
involved since being “wrong” or “correct” is not a simple binary outcome. Instead, when a decision is
made about an intervention, one can be wrong by a little or wrong by a lot. The EVPI calculations used
take this into account using a series of algorithms in Excel macros.
Obviously, all real, empirical measurements will fall short of perfect certainty but the EVPI can be used
as a guideline for measurement efforts. Also, the point of diminishing returns tends to be early in
measurements. In fact, for highly uncertain variables in particular, large uncertainty reductions can be
gained from a few observations. This tends to make small, iterative measurements preferable.
Consequently, the optimal effort tends to be a small fraction of the EVPI.
Detailed Findings
In Table 5, we list the variables with the highest EVPI for each intervention. Note that the differences in
scale produce EVPIs that are orders of magnitude different among the pilot projects. The first 3 models
are a country wide intervention or series of interventions that would potentially affect millions of
people, whereas others are for specific infrastructure investments that would affect either a single
company or entrepreneur. It is also important to note that the EVPI of a variable or group of variables
assumes no other variables are measured. As such, EVPIs are not strictly additive. The EVPI of
measuring two variables together could be much less or much more than the sum of the individual
EVPIs.
For example the sum total of the EVPIs for the Tana River Basin pilot exceeds the cost of the entire
project, even though this intervention could exceed $1 billion. However, the optimal measurement
would be measuring just one or two of the variables with the largest information values and then only a
fraction of the EVPI for those variables. An initial measurement effort for such a large and impactful
project could realistically be a few million dollars and still be considered a prudent level of analysis.
Biodiversity, on the other hand, produced initial EVPIs where no single variable measured by itself would
have an information value greater than zero. This means that perfect information on any one
parameter could not have changed the outcome of the decision but measuring groups of parameters
could. For the Biodiversity pilot therefore, Table 5 shows groups of variables and their EVPIs.
26
Table 5: Variables with Highest Expected Value of Perfect Information for All Pilots
Model
Categories of Variables in Models
EVPI
Cost per ton CO2
Household benefits: for crop, mixed livestock, and for humid and
semi-arid
zones and change
CO2, Methane,
NO2 emissions
$ 382 million
$393 million
Urban migration and economics of migration decision
$471 million
Project cancellation risk (grazing only)
$35 million
Density of cattle, mixed livestock and grazing intervention
$45 million
Tree related variables: coverage, density, time to maturity
$24.3 million
Participation levels in programs and adoption rate of new methods
and technologies
Initial labor and maintenance
labor, tree intensification
$14.6 million
Initial Funding for all interventions
$1.1 million
Loss of lifetime generation (all Tana dams)
$840,000
Variables related to Infrastructure degradation, poor seeds
Average hectare per farm
$104 million
$48 million
Variables related to project failure risks
$39 million
Change in production and costs with intervention
$45 million
Per farmer cost of seed distribution
$21 million
Adoption of program by farmers
$2 million
Information system error
$2 million
Basins, Mekong
Impact on fish migration due to each dam
$90 million
Sasumua
Chemical price per ton (alum, soda ash, chlorine)
$15,000
Tons of sediment runoff per year
$8,000
Labor costs
$53,000
Facility production capacity
$39,000
Vegetable crop yield with irrigation
$5.80 million
Unit price of for box of vegetables
$3.93 million
Inputs cost per hectare
$1.86 million
Increased yield per meter precipitation
$1.28 million
Rainfed, Tana
River Basin
Biodiversity,
Optimized Seed
Distribution
Network
Resource
Recovery Reuse
Irrigation,
Togorme
27
$ 285 million
$1.4 million
Comparing these results with more typical measurement priorities demonstrates another instance of
the Measurement Inversion. Perhaps some variables such as “net gain in rice yield using a water pump
in the Mekong delta” and “crop and mixed livestock benefits in the humid zone of the Tana River basin”
would be somewhat familiar areas or topics of study for CGIAR scientists. However, many other
variables listed in Table 5 would probably not have been prioritized for data collection by CGIAR
scientists.
Even though the information value calculations were done for only a handful of specific pilot decisions,
there are some findings that can be extrapolated into guidance for a metrics database. The information
value calculations have identified some potential gaps in the data collection efforts of CGIAR scientists.
The five areas below do not cover every high EVPI mentioned in Table 5 but indicate potential recurring
themes based on these results and observations in other government and business decisions.
Potential Gaps in Current Measurement Efforts




Market Prices: Anything regarding a market price – such as bulk chemicals, crop market prices,
the price of carbon offsets, and labor costs – has not been a focus of data gathering. Yet, a
market price for some item had one of the highest information values in four of the six pilots.
During the pilot project analysis, scientists felt somewhat uncomfortable even giving broad
estimates for market prices of any kind. Clearly this was outside of their field of expertise and
other sources for this sort of data should be utilized.
Project Failure Risks: The two pilot projects that included some type of probability of project
failure showed a high information value for that risk. This is also consistent with other
observations of information values on projects in many industries and government agencies. It
is likely that had other projects included the risk of failure that this would have been one of the
high information values in those projects as well. Project failure risks include probability of total
cancellation of the project (failure to complete with nothing to show), radical reduction in scope
(cancellation of parts of the original plan after expenditures on those parts had been made) and
massive delays. Data collection about success rates of projects and predictive models for
projects with various characteristics will be key.
Negative Consequences: For some projects there is the possibility that the project actually has
an overall detrimental effect with a loss much greater than merely losing the invested resources.
Projects that intensify farming practices for near-term benefits but with long term costs could
fall into this category. This is a type of project failure risk but in this case the original project was
successfully completed but with unintended results. Variables related to this also tend to have
high EVPIs in many industries and organizations. Again, historical project characteristics and
outcomes could be gathered to reduce this uncertainty.
Adoption Rates: Most interventions require some sort of change in behavior of a population. If
farming techniques, policies and technologies are not adopted by households then the benefits
may never materialize. Understanding and predicting the adoption characteristics of a
population (see Appendix D) will be a recurring uncertainty in many interventions. This is also
28


an observation that is very consistent with EVPI calculations in other fields in business and
government.
Detailed Household Demographics: There were recurring uncertainties about details of
households and individual farms that had significant bearing on intervention decisions. The
decisions of individual households on urban migration, the sizes of their farms, the number and
type of livestock and other types of demographic information were required for the analysis.
This sort of information is already gathered in some programs but this project finds that for
some intervention decisions a higher resolution and broader scope of this data may be required.
There is a lot of data that could be gathered but even small samples of the population would
have been informative. The specific data gathered should be driven by the information values.
Land Properties: The specific size of different lands, density of trees, erosion rates and other
characteristics of the land had high EVPIs in some projects. This is also information that is
gathered to some degree already but the information values indicate that higher resolutions of
this data may be required for some decisions. A Geographic Information System (GIS) type of
data base may be the ideal format but the data gathered should be driven by specific
information values.
Recommended Next Steps
As part of next steps for incorporating AIE in the most useful way for CGIAR, we have several specific
recommendations:
 The Global IDM should be immediately applied to additional major intervention decisions. This
analysis provided significant changes to how interventions should be assessed and it will likely
have some bearing on any potential intervention being analyzed.
 Now that information values have been computed for the initial pilot projects, those
measurements should begin immediately. The best approach will be small, incremental
measurements prioritized by their EVPIs. The practitioners of the AIE method within CGIAR
(primarily Eike Luedeling and Keith Shepherd) will document this process.
 The Global IDM will evolve. As new intervention decisions are analyzed there needs to be a
process for managing the changes to the model. A model manager will be appointed to
oversee how the model is used for individual decisions and how the basic template should
change over time. The manager will implement a version control process to accommodate
these changes. This will be necessary as several decision models may be developed in parallel.
Each new intervention will have three potential consequences for the Global IDM which must
be identified:
1. Identify which parts of the intervention decision can be captured in the existing Global
IDM. As more decisions are analyzed and as the model evolves, more of the new
decisions can be entirely modeled within components that already exist.
29

2. Identify which parts of the intervention decision will require changes to the Global IDM
which should be kept as part of the model for future intervention decisions. This will
happen frequently at first.
3. Identify which parts of the intervention are unique to that decision. They need to be
included in that instance of the model but do not need to be kept for future modeling
efforts. They will not have a lasting effect on the Global IDM. This should be a rarer
occurrence. Any model component that could happen more than once should be part
of the growing Global IDM.
The metrics database should gather data on key variables identified in the initial pilots. As
modeling continues for other interventions, common variables having information value across
multiple models need to be documented and added to the metrics database. Like the Global
IDM itself, the metrics database will evolve as new interventions are assessed and new
information values are computed. CGIAR now has a dynamic template for researching,
reporting, and updating information values on variables in agricultural intervention systems.
1. As indicated by the AIE methods, this process should start with small, iterative
measurements across the highest EVPI variables.
2. Metadata about the metrics should also be gathered in parallel, such as the cost of
gathering the data, its statistical errors, date gathered, source details, and a history of
subsequent updates when further observations reduce uncertainty.
3. CGIAR needs to decide what to do with high information variables outside of their
mandate. If metrics like future carbon dioxide levels and cost of migration are crucial
but outside their mandate is the best path forward cooperation with other
organizations, changing the mandate, or accepting this level of uncertainty? Even
without making this decision CGIAR can benefit from decomposing these variables and
how they interact with the variables that are in their mandate.
4. Variables that have high information values across multiple interventions (such as
project risks, household data, market prices, etc.) should have consolidated
measurement efforts across multiple interventions. The managers of the metrics
database may be in a position to facilitate this.
30
Appendices
Appendix A - Project Task Detail
Approximate Relative
Effort
As % of All
Phases
25%
Phase
As % of
Phase
Task
Preparatory period: Initial Planning and Analysis (4-6 weeks)
Introduction: Project announcement, scheduling initial workshops, and sharing
20%
material.
10% Help in vetting and hiring of applicant for Senior Scientist Decision Analyst
35%
Initial Workshops (3 generic workshops for each research group to determine
participants, potential interventions, and 2 calibration workshops)
10%
Gathering materials and literature
10%
Finalizing onsite workshop tasks
15%
Status updates and calls
30%
Intensive: Onsite or concentrated workshops (3 to 4 days)
40%
Specific onsite modeling work and related remote support
10%
Miscellaneous stakeholder presentations
5%
45%
45%
Team calls and status updates
Additional tasks determined during the
Follow Up: Model Creation and Recommendations (6 to 8 weeks)
50%
Complete construction of model
10%
Run VIA and give measurement recommendations
Measurement consulting and post-measurement model adjustments
Creating a template for future applications of method, including a method for
10% gathering utility curves from stakeholders, generating estimates, measurement
techniques, and model building
5% Status Updates
Synthesize global results – i.e. focus on results that have relevance across
10%
different research portfolios
15%
31
Appendix B – Review of Project Plans and Background
We have designed an overall schematic to help tackle the complexity of the problem CGIAR and its
stakeholders are facing. In 2013, we will focus on 4-5 interventions or projects, each representing a
different research group. For each research group we will use a model similar to the one outlined in the
schematic and one of our outcomes will be to continue to develop and refine a “best practice” for the
application of the Hubbard Method to the types of problems faced by CGIAR. In each case, we will be
looking at a different type of intervention, and we plan to apply our findings both to the specific cases
and to provide a more global framework for research decisions.
For each specific intervention or program, we will map relevant utility curves, objective forecasts,
monetization of impacts, and distill out a unifying certain present value equivalent. Incentives and utility
curves will vary by stakeholder, so part of the global model will include accounting for these differences,
but the primary considerations are:
1.
2.
3.
4.
Rural poverty
Food security
Improvement of nutrition
Ecological considerations (both gain/loss in local fertility and global ecological issues)
The weight of each of these considerations will vary by stakeholder, and part of our work will be to
complete a categorization of stakeholder objectives.
1. Preparatory Tasks for All Projects
a. Project announcement (including dates for calibration training) by early January
b. Schedule 2 more calibration webinars at times (6-9:30 PM, 7-10:30 AM) that make it
possible for attendees from Asia and Africa to attend by the end of January
c. Share workshop preparation materials (HTMA, 1st workshop) by January 24th
d. Finalize travel in January
e. Three generic two-hour workshops in February and March:
i. Workshop “A” is an offsite introduction to AIE process including:
1. Tailored Module I (30-45 minutes)
2. Take questions and concerns relating to specific project
3. Better specification of the candidate problems for case study AIE
analysis
ii. Workshops “B” and “C” vary according to how far along we are; actively
modeling versus scoping
f. Gather materials and literature on problems specific to each case
g. Finalize onsite workshop tasks specific to each case – dates vary with the date of each
projects onsite or intensive dates
2. Onsite Tasks for All Projects
a. Specific onsite modeling work
b. Miscellaneous stakeholder presentations
c. Additional tasks will be determined on a specific basis for each case.
32
3. Follow up Tasks for All Projects
a. Majority of estimation for populating models
b. VIA calculations
c. Generate model management recommendations to each group
d. Streamlined deliverable for each individual case
e. Model management activity - are there global model implications (crossover between
groups)
f. Data tracking/management recommendations (What data bases should be augmented)
4. Identifying typical stakeholders and their objectives
a. Identifying the size, frequency and types of decision problems typically facing CGIAR and
its stakeholders,
b. Assessing the existing historical data about decisions, analysis, forecasts, and outcomes
c. Continuing work on the taxonomy of recurring decision types; these findings will
motivate final recommendations on the application of the AIE framework
5. Develop the framework procedures and templates for identifying and analyzing the important
decisions affecting sustainable intensification of agriculture
a. Roles and responsibilities of individuals will be defined
b. “Risk/Return” boundaries for quantifying the risk-aversion of decision makers will be
determined
c. Methods to define and monetize sustainability with the use of “utility curves” will be
employed to quantify sustainability as a function of factors like biodiversity, top soil
depletion, etc.,
d. Examples of templates will be developed and other potential templates will be
identified
e. Potential methods for tracking and developing templates for agro-ecosystem health
metrics will be proposed; (actual procedures will depend on the outcomes of future
risk/return analysis and the value of information calculations)
33
Appendix C – Document Inventory
1.
2.
3.
4.
5.
6.
7.
8.
9.
Global Integrated Decision Model 2013 (Spreadsheet)
Rainfed Tana 2013 (Spreadsheet)
Sasumua Dam 2013 (Spreadsheet)
Mekong Basins 2013 (Spreadsheet)
RRR 2013 (Spreadsheet)
Irrigation Volta 2013 (Spreadsheet)
Biodiversity ODSN 2013 (Spreadsheet)
CGIAR Intervention Decision Model Concept Paper (Word Doc)
Statement of Work for 2013 (Word Doc)
34
Appendix D – Bass Model, Utility Curves and Discount Rates
Bass Model
The Bass Model is a useful way to model the presence of both innovators and imitators in a population.
Although the model was created as a marketing tool to project sales, the logic of the Bass Model is
attractive for matters relevant to agricultural interventions such as modeling adoption of a farming
technique or practice or the adoption or purchase of a new device or technology (like a cell phone). The
theory is that certain segments of the population called "innovators" (usually a small proportion) like to
try new things based on ideas; they are open to trying a new idea even if they have no personal
anecdotal evidence of its effectiveness. The other segment of the population (called imitators) generally
prefers to see something work anecdotally before they are willing to try it.
The presence and rate of innovation and imitation are represented as coefficients. The values of these
coefficients (where "p" denotes rate of innovation and “q” denotes rate of imitation) in a community
will determine how quickly you get to saturation; saturation is the point where everyone or nearly
everyone who will adopt a new technology or practice has already done so. In our application to the
CGIAR models, we back out the values of coefficients “p” and “q” by taking the subject matter expert’s
estimation for range of years to maximum adoption and calculating what values would give a sufficient
approximation to those estimates.
35
Environmental Discount Rate
Rather than use the traditional static discount rate we used an “environmental” discount rate by having
capital in the distant future valued at some minimum fixed point. The purpose of this is so one can
model both a high short-term discount rate and still asymptotically approach some minimum value for
which you value the distant future. (See Weitzman "On the Environmental" Discount Rate in the Journal
of Environmental Economics and Management). The hypothesis that we have declining discount rates
makes intuitive sense and can be checked anecdotally simply by asking yourself a series of simple
equivalence questions ("Would you rather have x now versus y in the future" & "would you rather have
x 20 years from now or y 21 years from now?") The status quo application of taking a static 3% discount
rate is obviously problematic for even simple business applications and could be improved upon by
either mapping utility curves in more detail or by looking at detailed interest rate expectations for each
year through ten years using the government yield curve or interest rate forwards (like a 5 year/5 year).
Static discount rates 50 years out are not operationally useful and we have replaced them with a better
alternative.
Income Distribution Utility Curve
The income distribution utility curve was created to reflect the reality that different agents (think
“donor” or “stakeholder national government”) may place different values on a dollar going into the
pocket of a poor person than a rich person. In the initial example below, we model the utility of a
marginal dollar as a decreasing function of the income of the recipient. Feel free to switch it. We can
imagine a stakeholder having the opposite view (e.g. working to attain an optimal capital base instead of
supporting the most vulnerable segments of the population). The key points are that:
1. We don’t assume the stakeholder views marginal increases in income to different income
classes as equivalent
2. The stakeholder who will be funding the project specifies the relationship – so it isn’t imposed
by the model builder
36
Appendix E: The Need for an Intervention Decision Model
A Concept Paper by CGIAR
October, 2012
The Consultative Group on International Agricultural Research (CGIAR) needs to develop a
comprehensive, quantitative model to assess the value of intervention decisions. The Intervention
Decision Model (IDM) will be the basis for prioritizing interventions, determining how to measure the
health of an agro-ecosystem, and calculating the value of the research itself.
Background
The Consultative Group on International Agricultural Research (CGIAR) is a collaboration of 15 research
centers and hundreds of partner organizations that investigate sustainable development.
Often, institutions conducting research in sustainable agricultural practices and use of the environment
face the following challenges.
1. Estimate the impact of intervention: Institutions attempting to solve big, complex problems like
sustainable development should always have more ideas than they have resources. Given limited
resources, how does CGIAR determine which interventions should have higher priority? Which
interventions will reduce risk, increase security, and improve lives the most? These decisions are
always made with significant uncertainty about acceptance of a new intervention, the future risks if
the intervention is not funded, the costs of the intervention, the long-term impacts (positive and
negative) of any change to a system, and so on. Yet, still CGIAR must be able to make
recommendations about how to use limited resources. In the face of so many uncertainties, how
should researchers decide which interventions to recommend?
2. Determine how to measure agro-ecosystem health: There are vast combinations of data that could
be gathered about the health of an agricultural system or ecosystem – but not all metrics will be of
equal value when it comes to supporting important intervention decisions. Of all of the variables
that might be an indication of health, some will have more direct impact on real decisions. And
even if a variable may have an impact on intervention decisions, costs of gathering data will vary
greatly. How should researchers determine what data gathering costs are justified?
3. Show the value of research: Donors would often rather directly fund intervention programs than
support research and researchers must be able to show how the expense of research is justified by
better intervention decisions.
37
The Method: Applied Information Economics
ICRAF, one of the research partners in CGIAR, has found a consolidated approach that will address all
three of these issues. The solution will involve the use of a method called Applied Information
Economics (AIE). Applied Information Economics (AIE) was developed as a scientifically proven method
developed for addressing investment dilemmas that are large, risky, and full of difficult measurements.
AIE uses a unique approach to assess major business decisions even when there are “intangibles.” Even
so-called intangibles like information-value have proven economic formulae that are exploited by AIE.
AIE synthesizes several quantitative methods from economics, actuarial science, decision theory and
statistics. AIE has been applied to over 70 real major decisions, of high uncertainty and complexity in
both government and commercial organizations. HDR has previously applied these methods to
problems as diverse as IT investments, logistics, environmental policy, new pharmaceutical products,
projects in the entertainment industry, the value of industry standards, and engineering and mining
risks.
Unlike traditional methods that produce arbitrary "scores" or unrealistic ROI's, AIE conducts a true
"Risk/Return" analysis with the same degree of rigor used by actuaries to estimate loss rates in
insurance pools. The method involves five steps – define the decision(s), model what we know now,
compute the value of information, measure what matters, make better decisions.
Figure 1: The Steps of The Applied Information Economics Process
Define the decision(s) - Identify Relevant Variables. Set
up the “Business Case” for the decision.
Model the current state of uncertainty – Initially
use calibrated estimates and then actual
measurements.
Compute the value of additional Information –
Determine what to measure and how much effort to
spend on measuring it.
No
Is there significant
value to more
information?
Yes
Measure where the information value is high –
Reduce uncertainty using proven empirical
methods.
Optimize the decision – Use the quantified Risk/Return
boundary of the Decision makers to determine which
decision is preferred.
38

Define the Decision(s): As obvious as this step may first appear, it is the key to better
understanding what to measure and real decisions are often different from what they first
appear to be. Is the dilemma whether to simply approve a project or how to conduct a project
given a vast combination of alternatives? Or is the decision a matter of when a given initiative
should be approved? The costs, benefits, timing, risks and even external factors are identified
and the real decision is clarified.

Model What We Know Now: Cost estimates, forecasts of benefits, project risks, and other
variables in a typical big investment decision are almost never known exactly. The uncertainty
about some variables, especially long term forecasts, can seem extreme. But the consequences
of even extremely uncertain variables can be assessed using the “Monte Carlo” method and a
special method for training experts to assess probabilities. The Monte Carlo method is a
method for conducting decision analysis by sampling variables that do not have exactly known
values (i.e. most variables in a model). This initial model is effectively a snapshot of the current
state of uncertainty about a problem before additional measurements are made.

Compute the Value of Information: Not all variables in a decision model are worth measuring
and those worth measuring are often a surprise to the decision makers. In fact, normally a kind
of “measurement inversion” exists in most decisions – that is, the most uncertain variables tend
to be ignored while the variables that usually receive a lot of attention actually have less bearing
on the decision. With AIE, every variable in a model will have an “information value” that allows
identification of high value variables in a decision. This approach targets only the variables in a
decision that are the most likely to significantly reduce overall uncertainty in the decision.

Measure What Matters: Once the high-value measurements are identified, a variety of
empirical methods can be used. Contrary to what is sometimes assumed, relatively little data or
simple observations may be required for extremely uncertain variables. AIE often uses efficient
“Bayesian” methods, which exploit prior knowledge and can be used even when data is messy
or sparse. The measured variables will have less uncertainty and then the model of uncertainty
can be updated.

Make Better Decisions: The output of the Monte Carlo model, updated with targeted
measurements, is compared to the risk/return preferences of the organization. Research shows
that the actual risk aversion and other preferences of decision makers changes frequently and
unconsciously. Different preferences are applied to different investments even when
management or believes they are being consistent. AIE addresses this major source of decision
error by quantifying and documenting preferences such as risk tolerance and the value of
deferred benefits so that the results of analysis can be assessed in a controlled, uniform
manner. Finally, sometimes decisions have large combinations of outcomes and have to be part
of a portfolio of decisions. When necessary, AIE applies optimization methods to determine the
best decision even from a large set of alternatives.
39
The Intervention Decision Model
This process simultaneously addresses all three challenges. Even uncertain and difficult to measure
interventions can be assessed and prioritized. And because of the information value calculations, the
best agro-ecosystem health metrics can be identified and the value of research itself can be articulated.
In order to get this process started, classes of typical intervention decisions will need to be modeled in
this structured way. To this end, CGIAR proposes developing the Intervention Decision Model (IDM).
The IDM will be a guide to all of the major intervention decisions in sustainable agriculture and
ecosystem maintenance. It consists of four major components: a model for all major classes of
interventions, the model for forecasting the outcomes of those decisions, a set of “preference curves”
that represent risk aversion and other policy choices, and the quantified values engine that combines
them all for recommendations.

Proposed Programmatic Interventions: The IDM will cover all major types of intervention
decisions including improvements to irrigation, improvements to rainfed-productivity including
crop and rangeland-productivity, and climate change adaptation or mitigation strategies. These
could be implemented as direct infrastructure investments, policies, training or incentive
programs. Investments in these areas have impacts that improve incomes in both the near and
long term as well as improve access and security for water, energy, and food. The IDM will
include all of these types of decisions.

Objective Forecasts: The costs of the intervention may be uncertain and the long term effects of
any intervention will be uncertain. The IDM will determine the uncertainty of onsite and offsite
impacts as well as behavioral factors like the adoption rate of a new practice or how incentives
change behavior. Some of the elements of this model will be based on known science such as
yield improvements from additional irrigation. But some factors, such as long term changes in
behavior will be much more uncertain. Quantifying the difference in this uncertainty will be
critical in determining what to measure.

Quantified Preferences & Policies: Preferences about what risks are acceptable, how to value
long-term effects, or the value of equitable improvements in income, need to be quantified and
documented as a matter of policy. These preferences are captured as a set of “utility curves”
that make policies – such as the relative value of a near-term certain impact vs. a long term and
uncertain impact – unambiguous. Such clarity will mean that various interventions can be
evaluated against the same standards of risk aversion and other preferences.

Quantified Values: Ultimately, the effects of an intervention and the quantified preferences are
combined into a single monetized value so that interventions of different types and sizes can be
compared. Each intervention creates a set of estimated impacts over a period of time. The
timing and uncertainty of these impacts are adjusted so that they can be rolled into a single
number. The quantified values can also adjust outcomes for differences in how benefits of a
40
program are distributed equitably. Separately, the IDM can assess the likely impacts on
individual System Level Outcomes (SLO) such as health, food security, poverty, and
sustainability.
Quantified Values
System Level Outcomes
(Poverty, Food Security,
Sustainability, Health &
Nutrition)
Certain Present
Value Equivalent
Distribution for
Aggregate Present
Value
Forecasts converted to of
monetized net impact, by year,
with uncertainty
Objective
Forecasts
Intervention
Cost
Onsite &
Offsite
impacts
Provisioning
Supporting/ Regulating
Services
Quantified
Preferences &
Policies
Certain
Value Curve
Time Value
Curve
Marginal
Income Increase
Value
Behavioral
Change
(adoption, crop
choices,
practices, etc.)
Proposed Programmatic Interventions
Figure 2: Outline of the Intervention Decision Model
41