Impact Evaluation Workshop for Health Sector Reform
Cape Town, South Africa, December 7-11, 2009
Impact Evaluation of Malaria
Prevention and Treatment
Jed Friedman (World Bank)
Evaluating Health
Programs is Different
• Evaluation methods often used in medicine to
determine efficacy of treatment
• What we know less about:
– How to get people to utilize prevention / treatment
services?
– What is the most cost effective mode of
prevention / treatment given behavioral
response?
– What are the socioeconomic effects of health
interventions?
Some Differences
• HIV/AIDS: Largely behaviorally driven
• Malaria: Vector-born disease
…But the behavioral aspects are important
Treat Infected
- Early Diagnosis &
Drug Type (AMT, ACT)
Problems
- Access
- Compliance
- Cost-Effectiveness
- Long-Term Effect
Protect Everyone
- ITN, LLIN
- Mosquito Proofing
Problems
- Valuation/Usage
- Compliance
MAN
“THE HOST”
PLASMODIUM
“THE AGENT”
MOSQUITO
“THE VECTOR”
Vector Control
- Prevent Breeding (DDT)
- Prevent Entry (Proofing)
- Prevent Bite (ITN, Spray)
Problems
- Resistance to Insecticides
- Compliance
Malaria Control
Is Not Principally a Question of
Technical Innovation
“Widespread use of ITNs and state-of-the-art drugs
succeeded in cutting malaria deaths half in 2 countries most
heavily affected by the disease, Rwanda and Kenya.”
Washington Post (01/31/08)
“This is a genuinely historic achievement. This is not
theoretical. We do not have to wait for a vaccine or new
drugs. If we implement today’s technologies aggressively on
a national scale we will have a big impact.”
Richard Feachem, former Director of the Global Fund
“With the resources available, we should be able to eradicate
malaria before I hang up my lab coat.”
Peter Agre, Malaria Research Institute, JHU
Impact of Policy Change on Malaria
Prevalence: South Africa
National Geographic 07/07
Policy Regime Switches
1. Pesticide Resistance
- DDT Stops: 1996
- Spraying Resumes: 2000
Drug Resistance
- Multidrug Therapy: 2000
Aggravating Exogenous Factors
- Refugee Flow
- Heavy Rains
- Worsening Resistance
In Kwazulu-Natal the combined effect of switching
from SP to the fixed combination of AL and IRS
with DDT was associated with a decrease in cases
of 78% and an increase in cure rate of 87%
(Barnes K, unpublished data; in Yeung et al. 2004).
Case 1: DDT - India
Quasi Experiment (Cutler et al. 2007)
• Long-term Effects of Malaria Eradication
• Outcome Measure: Educational Gains
1. Literacy Rate (LR); 2.School Completion Rate (SCR)
• Method: Quasi Experiment using Diff-in-Diff
1947 Pre-Eradication
1953 Intervention
Gains
- Population: 334 million
- Cases: 75 million (annual)
- Prevalence: 22% (annual)
- Mortality: 800,000 (annual)
- Mortality: 10% of total deaths
1st: 1953 NMCP Launched
DDT Spraying
- 2 Rounds per Year
- 125 Malaria Control Units
2nd: 1958 NMEP Launched
12% ↑ in LR, SCR
- Malaria explains
half of these gains
- Income ↓ through
malaria 7-10%
Sample: 300,000 Households, 1 million Individual Observations (NSS)
- Control: Pre-Eradication Cohorts (C0): 1912-1952
- Treatment: Post-Eradication Cohorts (C1): 1962-1972
- Omitted: Eradication Cohorts: 1953-1961
Case 2: ITN – Kenya
Field Experiment (Cohen and Dupas 2008)
• Explore tradeoffs between cost-sharing (CS) & free distribution for ITNs
• Randomize price of ITNs (0 ≤ p <pPrevailingCS) in prenatal clinics in Kenya
• Evaluate impact on pregnant women
1. Demand / Uptake
- Cost-sharing (C/S) does considerably dampen demand.
- Uptake Drops: i) by 75% from 0 price to prevailing CS price; ii) by 20% for
↑10Ksh.
2. Usage
- No evidence that C/S reduces wastage on those who do not use the net.
- Free ITN owner is not less likely to use net than those who paid higher prices.
- Coverage (Uptake + Usage): 63% (Free Net) v. 14% (40Ksh)
3. Need (Health)
- No evidence that C/S induces selection of those who need net more.
- Those paying higher prices appear no sicker (anemia) than control group.
4. Compare Cost Effectiveness (Externality Assumptions)
- Number of child lives saved highest under free distribution.
- Free distribution is more CE when externality threshold is medium level.
Source: Cohen and Dupas 2008
Average Number of ITNs
Sold/Distributed per Month
1) Demand for ITNs: Monthly
Net Sales by ITN Price
200
180
160
140
120
100
80
60
40
20
0
0
10
20
ITN Price (in Ksh)
Source: Cohen and Dupas 2008
40
2) ITN Usage Rates by Price: Share of
“Takers” who Report Using ITN at Home
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
10
20
40
ITN Price (Ksh)
All
First Prenatal Visits Only
Source: Cohen and Dupas 2008
First Pregnancy Only
Case 3: ITN – Uganda
Experimental Evidence on ITNs from
Rubagano and Kimuli Villages (Hoffman1)
Do free goods stick to poor households?
Design
NHH=193 T1: Free Net: 71, T2: Cash Transfer: 72, C: Uncompensated: 50
Findings
- Wealth and endowment effects result in very few HHs selling free net (FN).
- Only 6% of FN would be sold in frictionless market. Accounting for
transaction cost would further reduce this number.
- No significant gender gradient in average compensated valuation of ITNs.
- Man have higher income elasticity of supply for ITNs. Men have higher ATP.
Implications
- Distributing FN to women ►less leakage.
- Marketing among men ► more effective.
Source: Hoffmann, Barrett, and Just (2007)
Case 3: ITN – Uganda
Experimental Evidence on ITNs from
Rubagano and Kimuli Villages (Hoffman2)
Psychology, gender, intra-HH allocation of ITNs
Design:
NHH=143 T1: Free Net: 71, T2: Cash Transfer: 72
Findings
- Free nets lead to greater number of children covered, even for HHs with ATP.
- Net retention is higher for free nets (Endowment Effect): NFN > NCT
- Women tend to cover larger proportion of HH with nets.
- Intra-HH allocation of purchased nets (CT) depends on cost-benefit
calculations, with income-earners, net purchasers, and people often suffering
from malaria receiving it.
- Accompanied with a message, in-kind nets (FN) induce allocation to children.
Implications
Beyond price, mode of allocation and communication are important.
Source: Hoffmann (2007)
Overview of the World Bank’s Malaria
Impact Evaluation Program (MIEP)
• Program launched to conduct malaria policy relevant operational impact
evaluation under the Booster Program for Malaria Control
Goals:
• to generate evidence on effective approaches to increase utilization of
malaria prevention and treatment services
• to increase familiarity with IE approaches and build evaluative capacity in
national malaria control programs
•currently conducting evaluation studies in 6 countries assessing
effectiveness of a variety of control strategies
Access to effective treatment
Nigeria MIEP is assessing the involvement of trained community- and
private sector-based agents (CDDs and PMVs) in malaria prevention, and
case management to increase access to prompt diagnosis with RDTs and
treatment with ACTs.
Zambia estimating the gains in access through the introduction of RDTs
and ACTs through village Community Health Workers (CHW)
Overview of the World Bank’s Malaria
Impact Evaluation Program (MIEP) (cont.)
Access to effective treatment (cont)
Nigeria assessing the introduction of RDTs and ACTs to the workforce of
a large sugar cane plantation in order to estimate productivity costs of
adult malaria infection
India estimating the gains in quality of care through the involvement of
local organizations in supportive supervision and training of CHWs
Integrated vector control
Eritrea determining the cost-effectiveness of continued Indoor Residual
Spraying (IRS) in a low-endemic setting
School based programs
Kenya and Senegal asking whether the introduction of preventive
therapy through schools results in higher attendance and learning as well
as better student health
MIEP Portfolio Summary
Country
Project
PIs
Impact Evaluation
Project Value
Eritrea
IDF Grant –
Capacity building
for evidence-based
policy making in
the health sector
P. Carneiro
Experimental Design (RCT) of
impact of indoor residual spraying,
and incentives for improving larval
habitat management
$485,000
India
Leveraging local
capacity to assist
malaria control
efforts
J. Friedman
Experimental design of CBO
assistance to government malaria
prevention and fever case
management initiatives
$200 million
Kenya
School-based
Malaria Prevention
S. Brooker
Experimental Design (RCT) of
teacher training and school-based
intermittent preventive treatment
$4 million
approved +
$10.4 million
Pipeline
Malaria Control
Booster Program
P. Carneiro
Experimental Design (RCT) of
Community- and Private Sectorbased Malaria Control
$180 million +
$100 million
Additional
Financing
School-based
Malaria Prevention
S. Brooker
Experimental Design (RCT) of
teacher training and school-based
intermittent preventive treatment
$5 million
Zambia Access to
ACTs Initiative
J. Friedman
Quasi Experimental Design
(Matching and RCT) of Public Sector
Supply Chain Management and
Community- and Private Sectorbased Malaria Control
$26.85 million
Nigeria
Senegal
Zambia
J. Keating
M. Jukes
E. Velenyi
M. Jukes
E. Velenyi
Conclusions
• Impact evaluation studies have already contributed much
information for understanding the efficacy and costeffectiveness of various malaria prevention options
• Recent impact evaluation studies will help policy makers
work through certain malaria control decisions such as free
ITN distribution vs. cost-recovery
• However many critical questions remain. For example, how
do we affect people's behavior to ensure adoption and
proper usage of nets? How do we increase the proportion of
fever cases seeking treatment at facilities with adequate
diagnostic and curative care?
• Impact Evaluation of Malaria Programs (See Handout)
– Friedman, Legovini, and Velenyi; Development Dialogue Notes Vol. 1 (2009)
• This week we will discuss methods through which we
can answer these questions.
References 1
1.
2.
3.
4.
Malaria Control http://www.malariasite.com/malaria/ControlOfMalaria.htm
Brown, D. “Malaria Deaths Drop in Rwanda, Kenya.” Washington Post January 31, 2008
Finkel, M. “Raging Malaria – Stopping a Global Killer.” National Geographic July 2007
Yeung et al. 2004. “Antimalarial Drug Resistance, Artemisinin-Based Combination Therapy, and the
Contribution of Modeling to Elucidating Policy Choices.” Am. J. Trop. Med. Hyg., 71(Suppl 2), 2004, pp.
179–186.
5. Cutler et al. 2007. “Mosquitoes: The Long-term Effects of Malaria Eradication in India.” NBER.
6. Cohen and Dupas. 2008. “Free Distribution or Cost-Sharing? Evidence from a Randomized Malaria
Prevention Experiment.” Draft. January. Presented at the Brookings Institution. January 24, 2008.
7. Hoffmann, Barrett, and Just. 2007. “Do free goods stick to poor households? Experimental evidence on
insecticide treated bednets.” Draft. November. Department of Applied Economics, Cornell University.
8. Hoffmann. 2007. “Psychology, gender, and the intra-household allocation of free and purchased
mosquito nets.” Draft. November. Department of Applied Economics, Cornell University.
9. Clarke et al. 2008. “Health and educational impact of intermittent preventive treatment of malaria in
schoolchildren: a cluster-randomized controlled trial.” Draft. LSHTM.
10. Arrow, K.J. et al., Eds. 2004. “Saving lives, buying time: economics of malaria drugs in an age of
resistance.” Board on Global Health. Washington, D.C.: Institute of Medicine.
11. Laxminarayan, Over, and Smith. 2005. “Will a Global Subsidy of Artemisinin-Based Combination
Treatment (ACT) for Malaria Delay the Emergence of Resistance and Save Lives?” World Bank Policy
Research Working Paper 3670, July.
References 2
12. Gollin and Zimmermann. 2007. “Malaria: Disease Impacts and Long-Run Income Differences.”
Discussion Paper No. 2997. IZA.
13. McCarthy, Wolf, and Wu. 1999. “Malaria and Growth.” World Bank Working Paper. WPS 2303.
14. Gallup and Sachs. 2001. “The economic burden of malaria.” Am. J. Trop. Med. Hyg. 64(1,2)S: 1-11.
15. Acemoglu and Johnson. 2006. “Disease and development: The effect of life expectancy on economic
growth.” NBER Working Paper 12269.
16. Weil. 2007. “Accounting for the effects of health on economic growth.” The Quarterly Journal of
Economics. August. Vol. 122, No. 3, Pages 1265-1306.
17. Bleakley. 2007. “Malaria in the Americas: A retrospective analysis of childhood exposure.” Department
of Economics, University of Chicago.
18. Lucas. 2005. Economic Effects of Malaria Eradication: Evidence from the Malarial Periphery.
Manuscript. Department of Economics, Brown University.
19. Hong. 2007. "A Longitudinal Analysis of the Burden of Malaria on Health and Economic Productivity:
The American Case." University of Chicago.
20. Barreca. 2007. "The Long-Term Economic Impact of In Utero and Postnatal Exposure to Malaria." UC
Davis.
21. Coleman et al. 2004. “A Threshold Analysis of the Cost-Effectiveness of Artemisinin-Based
Combination Therapies in Sub-Saharan Africa.” Am. J. Trop. Med. Hyg., 71(Suppl 2), 2004, pp. 196–
204.
22. Breman et al. 2006. “Conquering Malaria.” Eds. Jamison et al. Disease Control Priorities. Chapter 21.
Impact Evaluation Workshop for Health Sector Reform
Cape Town, South Africa, December 7011, 2009