Evaluation of Nutrition-Sensitive
Programs*
Presented by: Deanna Olney, PhD
October 19, 2016
* Olney, Leroy and Ruel. Evaluation of Nutrition Sensitive Interventions. Chapter 44. In: De Pee, Taren and
Bloem (eds.) Nutrition and Health in a Developing World. 3rd Edition (forthcoming).
* Leroy, Olney and Ruel. Evaluation of Nutrition Sensitive Interventions. AOTR
Background
• The Lancet Series on Maternal and Child Nutrition calls
for greater investments in large-scale nutritionsensitive programs.
• Rigorous evidence of their effectiveness, how they
work and cost-effectiveness is scant attributed to:
• Weaknesses in program design and implementation
• Poor evaluation designs.
• There is a need for rigorous, theory based-evaluations
that can provide evidence of:
•
•
•
•
What works to improve nutrition
How programs work to achieve nutrition impacts
What other impacts programs have
Cost and cost-effectiveness of achieving impacts
Key challenges: Program
complexity
• Complex in design and implementation:
• Multiple goals, inputs, pathways of impacts
• Span across different sectors (e.g. health, agriculture,
education)
• Within each program intervention potential variability in:
• Delivery (quantity and quality)
• Utilization
• Adherence to the program protocol
Key challenges:
Long time frames and impact
pathways
• Long time frames for implementation and evaluation:
• Long pathways from program inputs to impact
• Implement program at the level of quality desired
• Achieve meaningful impacts on some biological outcomes
such as child growth
• Design and carry out a rigorous and comprehensive
evaluation
• … vs often short time frames imposed by donors
Key challenges:
Differing priorities, expectations,
incentives, and perceptions
• Priorities of implementers:
• Deliver a high quality program
• Meet targets within the specified budget and time frame
• Priorities of evaluators:
• Rigorously evaluate the program
• Answer key questions related to what impact was achieved
why or why not it was achieved and at what cost
• Misperceptions that can undermine trust:
• Process evaluation viewed as evaluating the performance of
the program implementers vs generating evidence to
improve programming and understand impacts
• Cost studies can be perceived as auditing the program’s
finances
Key challenges:
Implementation constraints vs.
evaluation rigor
Cluster randomized controlled design
T24
Treatment 1
T18
Treatment 2
Treatment 3
TNFP
Control
Control
Regular program
Regular
program
Key challenges:
Assessing benefits beyond
targeted beneficiaries
• Potential “spillover” of benefits beyond the
targeted beneficiaries:
• E.g. siblings, neighbors, etc.
• Adds complexity:
• Capturing all impacts of nutrition-sensitive programs
• Measuring the cost-effectiveness of programs
Two essential ways to
address these challenges
• Build a strong partnership and collaboration
between implementers and evaluators
• Design and use a rigorous and comprehensive
evaluation framework:
In-depth understanding of the program, program theory,
program impact pathways, rigorous comprehensive
evaluation design
Comprehensive evaluation:
Embrace the complexity of
the program
Inputs
Village Model
Farms (VMF)
established and
VMF owners
trained
Cost of
the
program?
HKI partners
with local
NGOS
Work with
local authorities
(e.g. village
chief)
Cost
study
Women’s
groups
established
Village Health
Support Group
identified and
trained
Processes
Outputs
Agriculture
inputs (e.g.
seeds, saplings
and poultry
distributed)
Improved and
developed
gardens
established
Provision of
agriculture
training
Provision of
nutrition, health
and hygiene
education
Outcomes
Increased
production of
micronutrientrich fruits and
vegetables
How and why
does the
program (not)
have an
impact?
Impacts
Increased
Income
Women’s
increased
control over
resources
resource
Small animal
production
established
Increased
poultry
production
Agriculture
training
received and
understood by
beneficiaries
Beneficiaries’
agriculture
knowledge
improved
Increased
household
consumption
Health, hygiene
and nutrition
education
received and
understood by
beneficiaries
Beneficiaries’
health, hygiene
and nutrition
knowledge
improved
Improved child
care and
feeding
practices
Process + impact evaluation
Improved
women’s
empowerment
What is
the
impact of
the
program?
Improved
maternal and
child health and
nutritional
status
Impact
evaluation
Impact evaluation:
Experimental designs
• Designed to assess program impacts at different levels
• Uses quantitative methods
• Ideally conducted before the program starts and at the end of the
program but can include intermediary time points
• Requires a valid counterfactual and appropriate target groups,
duration and indicators (with adequate sample sizes for each)
• Experimental (or randomized) designs considered gold standard for
impact evaluations.
• Randomization: individual or group (cluster) level.
• *If* done well, one can assume:
• that both groups are comparable;
• that the only difference between the groups is the program;
• that the control group provides a valid counterfactual for the intervention group
exposed to the program.
differences found in the outcomes of interest attributable to
the program.
80.0
75.075.0
70.070.0 68.2
68.2
60.060.0
64.164.1
64.364.3
65.0
64.764.7
60.0
2010
2010
Impact (pp)
75.0
68.2
70.0
68.2
65.065.0
74.874.8
75.0
74.8
68.268.2
68.2
64.1
63.063.0
64.3
62.362.3
70.0
68.2
74.8
74.8 74.874.8
68.2 68.268.2
68.2
74.8
65.0
65.765.7
64.664.6
64.164.1
64.7
64.764.1
64.3
64.1
63.0
62.3
60.0
-5.0
-5.5-5.5
-5.5 pp
2010-2014
2010-2014
Control
Control
-5.5
-5.2-5.2
+ 0.2 pp
-7.1-7.1
0.0 0.0
-5.0-5.0
-4.0-4.0
-5.5
-5.2
- 10.5 pp
-7.1
-7.1
-10.0
-10.0
68.2
64.6
63.0
2010
2014
2010-2014
2010-2014
2010-2014
2010-2014
2010-2014
2010-2014 2010-2014
2010-2014
2010-2014
2010-2014
2010-2014
Wave
Wave
Treatment
Treatment
Control
Wave
Impact
(95%
CI) CI)
(95%
Impact
Treatment
Control
74.8
74.8
68.2
2014
2014
2010
2014
2010
2014
2014
2010
2014
2014
2010
2014
2010 2010
2014 2014
2010 2010
2014 2010
2010
2014
2010
0.0
TNFP T18
T24
80.0
74.874.8
Impact on stunting (pp)
Impact on stunting (pp)
74.874.8
With-andT24without
AllTNFP
TNFP T18
Impact on stunting (pp)
80.080.0
Before-andT24
T18
All
T18
T24 after
Impact on stunting (pp)
Randomized
All All
design
Stunting (%)
Unadjusted stunting (%)
Impact evaluation:
Invalid counterfactuals
74.8
68.2
65.7
64.1
62.3
2014
2010
2014
0.0
-5.0
-4.0
-5.2
-10.0
2010-2014
2010-2014
Wave
Impact
(95% CI)
Treatment
How and why does the
program (not) have an impact?
• Two ways to assess how impact is achieved :
• Measurement of intermediary measures (outcomes) in impact
study
• Process evaluation
• Can also be used to improve on-going or future programs
• Solid understanding of program theory and program
impact pathways is key
Process evaluation
• Examines the primary inputs, processes, outputs and
outcomes along the primary program impact pathways
• Conducted about one year after program implementation
begins and can be repeated
• Use a mix of quantitative and qualitative methods to
examine the different steps along the impact pathways
• Include different key stakeholders (e.g. program
implementers and beneficiaries) and different program
delivery points
• Share the results:
• Workshop with implementers
• Identify what is feasible to improve and how improvements can
be made
Cost study
• Designed to estimate the overall cost of the program, the cost
of the main program components and its cost-effectiveness
• Can also be used to estimate savings or costs associated with
adding, changing or dropping program activities or
beneficiaries
• One method that can be used is the Activity Based Costing
Ingredients (ABC-I) approach.
• Using the program impact pathways: provide detailed description of
all program activities.
• Identify the program’s main activities and “ingredients” needed for
each.
• Define the unit cost algorithms, i.e. the different types, quantities
and costs of the “ingredients” necessary for each activity.
• Calculate cost of each program activity and of full program.
In summary
• Evidence of what works in nutrition-sensitive programs,
how and at what cost is extremely limited.
• Evaluation of nutrition-sensitive programs is complex but
necessary.
• Strong partnerships between program evaluators and
implementers are needed to carry out rigorous program
evaluations and to ensure that results are used to improve
programming.
• Comprehensive evaluations should include rigorous impact
and process evaluations and when possible cost studies.
• Guidance for future investments requires strong evidence
from rigorous, theory-based comprehensive evaluations of
different nutrition-sensitive program models that bring
together interventions from a variety of sectors.