Institute for International Programs
An international evaluation consortium
A common evaluation framework
for the scale-up to achieve
the health MDGs
Including the results framework for
maternal, newborn and child health
International Evaluation Consortium
Institute for International Programs,
Johns Hopkins Bloomberg School of Public Health
May 2008
Presentation Outline
1. Background, rationale, methods and
partners
2. The strategic framework
3. The results framework for maternal and
child health and nutrition
4. Evaluation challenges
Section 1:
BACKGROUND, RATIONALE,
METHODS AND PARTNERS
Rationale for a common framework
 To support the comparison of results and
costs within and across countries that may use
different approaches to implementation
 To promote systematic attention to the
methodological challenges of evaluating largescale approaches
 To inform pay-for-performance approaches
Victora CG, Black RE, Bryce J. Learning from new initiatives in maternal and child health. Lancet 2007; 370: 1113-1114.
What is included in an
evaluation framework?
1. A strategic framework including general
principles
2. A results framework that includes:

A conceptual model specifying how activities will
lead to outcomes and impact

A set of compatible designs for evaluation of
country-level initiatives, to allow comparisons
across places and time

A set of common indicators and other measures
Evaluators &
products
Initiatives
Framework development process
Global business
plan
IHP+
Global campaign
Catalytic initiative
BMGF/PMNCH
Rapid Scale Up
Doris Duke
Overall strategic
framework
MCH results
framework
Common evaluation
framework
Section 2:
THE COMMON STRATEGIC
FRAMEWORK
Framework Terminology
Term
Definition
Evaluation
rigorous, science-based collection of information about
program activities, characteristics, outcomes and impact
that determines the merit or worth of a specific program or
intervention
Monitoring
routine tracking and reporting of priority information about a
program and its intended outputs and outcomes…to assess
whether resources are spent according to plan and whether
the program is resulting in the expected outputs
Monitoring
performance
project or programme monitoring which aims to provide
feedback for improved performance/ implementation
Performancebased funding
provision of financial incentives based on measured
progress
Data quality
audits
a method to assess recording and reporting systems in the
context of performance-based funding
Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of
the scale up for better health. Draft 4, February 2008.
Principles
Collective action: Primary focus on the contribution of the collective efforts to
scale-up the health sector response in countries
Alignment with country processes: build upon national processes that countries
have established to M&E progress in the implementation of national plans
Balance between country participation and independence: driven by country
needs but conducted in a manner which maintains independence of evaluation
Harmonised approaches: common protocols and standardized outcome indicators
and measurement tools, with appropriate country adaptations
Capacity building and health information system strengthening: systematic
involvement of country institutions
Adequate funding: between 5% and 10% of the overall scale-up funds set aside for
monitoring performance, evaluation, operational research and strengthening health
information systems
Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of
the scale up for better health. Draft 4, February 2008.
Strategic Framework
Process
Capacity building
Programmes
Institutions
People
Harmonization
Aligned international
efforts with national plan
Well coordinated and
harmonized support
Accountability
Performance monitoring
Results focus and
evaluation
Improved services
Plan
Coherent, prioritised and
funded
Outcomes
Access, safety, quality, efficiency
National plan
implementation
Systems strengthening
Priority interventions
scale-up
Health system strengthened
Funding
Domestic sources
International sources
Outputs
Governance, HR, medical products, information
Inputs
Increased service
utilization and
intervention
coverage
Reduced inequity
(e.g. gender, socioeconomic position)
Responsiveness
No drop-off nonhealth sector
interventions (e.g.
water & sanitation)
Use for better practices
Implementation
Monitoring
Health system monitoring
Improved survival
Child mortality
Maternal mortality
Adult mortality due to
infectious diseases
Improved nutrition
Children
Pregnant women
Reduced morbidity
HIV, TB, malaria, repr.
health
Improved equity
Social and financial risk
protection
Reduced impoverishment
due to health expenditures
Coverage monitoring
Impact monitoring
Strengthen country health information systems
Evaluation: process, health systems strengthening, impact
Adapted from Boerma T, Bos E, Walford V et al. International Health Partnership+. A common framework for monitoring performance and evaluation of
the scale up for better health. Draft 4, February 2008.
M & E action
Aid process
monitoring
Resource tracking
Impact
Section 3:
THE RESULTS FRAMEWORK
FOR MATERNAL, NEWBORN
AND CHILD SURVIVAL
Three components
of a results framework
1.
A conceptual model specifying how
activities will lead to outcomes and impact
2.
A set of compatible designs for evaluation
of country-level initiatives, to allow
comparisons across places and time
3.
A list of common indicators and other
measures
What is a conceptual model
in the context of a results framework?
A description of the activities and pathways
that will lead from program inputs
to impact on health and nutrition
A generic conceptual model based on the strategic framework
Inputs
Process
Outputs
Outcomes
Impact
Funding
Training &
Capacity building
Health services
delivery
Mortality
Morbidity
Nutrition
Procurement and
supply
Quality
Service
utilization and
intervention
coverage
Planning &
policies
Harmonization &
efficiency
Guidelines
IEC
Community
mobilization
Behavioural
Interventions
& knowledge
Behavioural
change
Reduced
inequity
Why is a conceptual model
essential in a results framework?
 To clarify expectations of program planners/ developers




To define the evaluation questions and select indicators
To support the design and estimate sample sizes
To guide analysis and attribution of results
To compare and interpret results across sites
 To track changes in assumptions as they evolve in
response to evaluation findings
 To stay honest about what was expected
The generic conceptual model must be adapted to reflect
the plans and expectations of each country/program.
Common elements of the
conceptual model for the global scale-up
Inputs
Process
Outputs
Country-specific
implementation & health
system strengthening
Outcomes
High and equitable coverage
with proven interventions
Impact
 MORBIDITY AND MORTALITY
Designs linked to national monitoring
National program monitoring
Policies/plans/
resources
Implementation
(provision & quality)
Outcomes
Impact
Impact evaluation
The evaluation framework must be implemented in ways
that build institutional capacity for evaluation in countries.
Evaluation challenges
 Complete tracking of program and
contextual factors
 Full consideration of equity
 Evaluating “scaling-up” and “catalytic
actions”
 Measuring “community engagement”
 Assessing “health system strength”
 Capturing changes in mortality within
short time frames – new approaches
needed
Summary
The common evaluation framework for the
global scale-up to the health MDGs
 There is a common evaluation framework; all countries
and partners should use it as a starting point for planning
their evaluations
 Main purpose of evaluating global scale up is
assessment of program effectiveness
 Requires comprehensive prospective evaluation designs
 Complementary evaluation goals of independence and
capacity building
 Key constructs need further definition (e.g., “scaling-up”;
“catalytic”, “health system strength”)