Beyond Beneficiaries: The Use of
Information Systems for Cost-Effective
Evaluation
Suzanne Duryea
Research Department
Inter-American Development Bank
December 10, 2003
Information Systems
• Information systems such as SISBEN, CAS, SIPO
were developed to better target beneficiaries
(predominantly the poor and at-risk populations).
• A quantitative score is used as a proxy for poverty
and/or the lack of income generating resources
within the family. Families scoring below a
threshold are eligible to participate in certain
programs.
Information Systems for Social Programs
• Can facilitate certain types of evaluations
• With some complementary additions: the
databases can form a stronger foundation
for evaluation
What do we mean by
“impact evaluation?”
• “What would have happened to the
beneficiaries in the absence of the
program?”
• “What was the effect of the program?”
Some options
1) Randomized Designs
• Examples:
a) Geographic regions are randomly selected for later
phase-in of a program (Oportunidades, PRAF)
b) Individuals are randomly selected to form a control
group (such as school vouchers Colombia)
• Randomized evaluations are viewed by the experts as
the best option because they solve many of the
problems encountered in evaluation. (Duflo and
Kremer, 2003)
• However, randomization can be politically difficult and
is not always possible or appropriate.
Some options
2) Ex-Post Evaluation
• Aim is to compare outcomes of beneficiaries and nonbeneficiaries who were similar BEFORE program
participation
• Only observe individuals at one point in time -- after
enough time has passed for program to have an effect.
• Critical assumption: After controlling for observed
characteristics, beneficiaries and non-beneficiaries do
not differ in unobserved characteristics
Some options
3) Before and After Comparisons -- “Double Difference”
• Monitor the beneficiaries and non-beneficiaries over the
length of time necessary for the program to have effects
• I.e., Take baselines for both groups then compare the
change in the beneficiaries’ behavior to the change in
the non-beneficiaries’ behavior. All time invariant
unobservable differences are removed.
• Critical assumption: No time varying unobserved
differences between beneficiaries and non-beneficiaries.
Creation of Counterfactual Group
• The most critical part of evaluation design: who
can be used as a natural comparison
• Information Systems Can Plan an Important Role
• There are families who will not quality for
programs because they are slightly above the
eligible score (threshold). These families look
very similar to families with similar scores who
have qualified. Discontinuity Design:
(Campbell, 1969, Buddlemeyer and Skoufias
2003)
• Assign program and then compare outcomes
across the families who are close to the threshold.
Important Caveats
• This is a local average treatment effect: not
necessarily comparing the effect on the poorest
families
• If there are heterogeneous effects at different
levels of scores this is only measuring the effect
at the cutoff score
• Threshold must exist: enforcement of rules
necessary.
Example of an ex-post
evaluation: Superémonos
S. Duryea and A. Morrison (2003)
Relied heavily on information systems from
Instituto Mixto de Ayuda Social (IMAS), Costa Rica:
Sistema de Información sobre la
Población Objetivo (SIPO)
Sistema de Atención a Beneficiarios (SAB)
Superémonos
• Food coupon (U.S. $30 per month during
school)
• Targets poor households with school age
kids (ages 6 to 18) at risk for poor school
attainment using SIPO (proxy means test)
• Conditional transfer: families agree that all
children will regularly attend school
Sistema de Información sobre la
Población Objetivo (SIPO)
• Targeting mechanism
• SIPO score depends on
Occupation of household head
Material used in house construction
Household income
Education of household head
Net household wealth
• Over 250,000 households
SIPO and SAB
• Very efficient
• We provided IMAS with a list of characteristics
(requested 3 regions, ages 10-16, started program
in 2001) and they provided a cross-referenced list
of beneficiaries within minutes
• Were in the field with our survey within two
months
• Very fast in contrast to adding questions to a
national household survey
First however: we were missing a critical component
for evaluation
• No information was available in SAB regarding potential
counterfactual group (those just above the threshhold)
We formed a counterfactual group by conducting surveys
in the same neighborhoods and getting families with
similar probabilities of participating in Superemonos
(propensity matching approach)
Cost of data collection for evaluation of
Superémonos 1788 households under $30,000
Mexico Progresa 24,407 households $450,000
Argentina Trabajar 2,800 households $350,000
Source: Blomquist 2003
Summary Information on Survey Samples
SuperemonosIMAS list
number of total observations
average age of child
percentage female
percentage in San Jose
percentage in Alajuela
percentage in Cartago
percentage of mothers with
incomplete primary education
percentage of households
lacking working electricity
Not from list
746
12.90
50.27
60.19
6.03
33.78
1032
12.95
48.55
56.88
4.94
38.18
36.46
35.76
4.29
3.88
Survey design and implementation
•
•
•
•
Tailor-made survey
Pilot tested
Power tested
Sample size:
746 beneficiaries of Superémonos
1,042 non-beneficiaries
• Data collected in 3 urban centers: San Jose, Alajuela
and Cartago
• Collected information on labor force participation,
school attendance, school performance, and a host of
household characteristics
• Approx. cost: $15 per survey (no addresses)
Duryea Morrison Strategy: Ex-Post Evaluation
Information System Used to Identify Beneficiaries Only
We found that beneficiaries ages 12-15 were 5
percentage points more likely to attend school
than non-beneficiaries.
In this program we have no reason to think there
are serious selection problems. Academic record
not considered for eligibility. Ex-post
methodology may be appropriate.
Alternative Strategy: What we might have done
Double Difference: Information Systems Used to Identify
Treatment and Control Groups
• Create comparison group from those who fall just
above the threshold on SIPO for Superemonos
participation
• How: Match 3 non-beneficiaries to each
beneficiary (based on SIPO score within same
geographic unit)
• Follow beneficiaries and non-beneficiaries over
time (double difference estimation)
Hypothetical Example:
Effect of Conditional Transfer Program on Hours of School Activities
Double Difference Measures Positive Effect of Program
Ex-Post Measure Misses Positive Effect of Program
Hours Spent on School Activities
40
35
B
30
25
A
Double Difference = C - B = 27 - 18 = 9
amount of change by beneficiaries is larger
A = ExPost
Difference
NonBeneficiaries
C
20
15
10
Beneficiaries
5
Before
After
0
0
1
2
3
4
5
6
7
Months of Execution of Project
8
9
Policy Conclusions and Recommendations
1. Information systems are critical, both for
targeting as well as for evaluation.
2. Information systems can facilitate cost-effective
ex-post evaluations.
3. Possible to modify information systems to
provide more rigorous evaluations at reasonably
low cost. Monitoring non-beneficiaries in
addition to beneficiaries is the most important
modification. Part of evaluation strategy for
Chile Solidario.
4. Encourage stricter enforcement of the
implementation of scoring thresholds.