Validating Ex Ante Impact
Evaluation Models: An Example
from Mexico
Francisco H.G. Ferreira
Phillippe G. Leite
Emmanuel Skoufias
The World Bank
PREM Learning Forum-April 22, 2008
1
Introduction
 Conditional Cash Transfer (CCT)
programs are becoming an important
element of social policy in LAC
 Distinguishing characteristic of CCTs:
 social accountability supported by
rigorous impact evaluation (IE)
Introduction
 Alternative IE designs:
 Experimental design: hh randomly
assigned to T and C groups, prior to
implementation of program. Typically
hh surveyed in baseline and for 1 or
more rounds after the start of the
program.
 +: provide most reliable estimates (gold
standard) of program impacts
 -: costly, likely to be of small scale
 -: large time lags involved
Introduction
 Quasi-experimental designs: typically
comparison/control hh are obtained expost (after the start of the program),
attempting to equalize selection bias
between treatment and control groups
 +: less costly
 -: lack of baseline data (and/or pre-program
differences)
 Overall, ex-post methods do not provide
ANY information about the possible effects
of the program prior to its
implementation.
Introduction
Ex ante methods: simulate the effects of
the program on the basis of a structural
(or reduced form) model of household
behavior
Easily implemented using a representative hh
data set (e.g. BFL, 2003)
Expand the set of policy-relevant questions
that can be addressed, e.g. useful in designing
the program, size of transfer, etc.
Based on the concept of treatment and
comparison/counterfactual group
However, require some strong assumptions
about:
Functional form
Perfect implementation of the program
Introduction
This paper is one of the first to provide a
validation test of the ex-ante evaluation
methodology
Approach: Use household survey data
from two CCT programs (PROGRESA in
Mexico and BDH-Bono de Desarollo
Humano in Ecuador) where experimental
designs were employed to (ex post)
evaluate program impact
Use the baseline data from each survey to
apply ex-ante evaluation methods to
predict program impact.
Introduction
Compare the impact predictions obtained
with the ex-ante method to the impact
estimates obtained using the experimental
(ex-post) methods.
Some Background on
PROGRESA
 What is PROGRESA?
Targeted cash transfer program conditioned on
families visiting health centers regularly and
on children attending school regularly.
Cash transfer-alleviates short-term poverty
Human capital investment-alleviates poverty
in the long-term
Started in 1998. By the end of 2004:
program (renamed Oportunidades) covered
nearly 5 million families, in 72,000 localities in
all 31 states (budget of about US$2.5 billion).
Transfers given to mothers: 20% of hh
consumption expenditure
Some Background on
PROGRESA
 Two-stage Selection process:
Geographic targeting (used census data to
identify poor localities)
Within Village household-level targeting
(village household census)
Used hh income, assets, and demographic
composition to estimate the probability of being poor
(Inc per cap<Standard Food basket).
Discriminant analysis applied separately by region
Discriminant score of each household compared to a
threshold value (high DS=Noneligible, low
DS=Eligible)
Initially 52% eligible, then revised selection
process so that 78% eligible. But many of the
“new poor” households did not receive benefits
Ex ante model: BFL
 Why BFL instead of Attanasio, Meghir et Santiago (2005)
ou Todd et Wolpin (2005)?
Simplicity since dynamic Ex ante models as AMS and
TW are data intensive depending on panel data.
Is a behavioral model based on four key
assumptions:
Do not model household behavioral, i.e., do not
debate who makes child’s decision;
Adults are unafected by children’s choice;
Siblings interaction are ignored;
Household composition is exogeneous
10
Ex ante model: BFL
 The model
Child’s occupational choice
(0) Not going to school;
(1) Going to school and paid work;
(2) Going to school and non-paid work
Ui (0)  Z i   0  (Yi  y i0 )   0  v i0
Ui (1)  Z i   1  (Yi  y i1 )   1  v i1
Ui (2)  Z i   2  (Yi  y i2 )   2  v i2
11
Ex ante model: BFL
 The model
Child’s contribution to income in each state 0, 1 and
2
y i 0  w i ; y i 1  M  wi ; y i 2  D  wi
Then
Log wi  X i    m  Ind(Si  1)  ui
where
M  exp(m)
12
Ex ante model: BFL
 The model
Child (household) i chooses the alternative that
yields the highest utility
Ui (0)  Zi   0  Yi   0  w i   0  v i0
Ui (1)  Zi  1  Yi  1  w i  1  v i1
Ui (2)  Zi   2  Yi   2  w i  2  v i2
where
 0   0 ;  1  1 M;  2   2 D
13
Ex ante model: BFL
 The model
Child (household) i chooses the alternative that
yields the highest simulated utility
ˆ v
ˆ i  
ˆ0  w
Ui* (0)  Zi ˆ0  Yi 
0
i0
ˆ  v if potential benef
ˆ 1  wˆ i  
Ui* (1)  Zi ˆ1 (Yi T )
1
i1
ˆ  v otherwise .
ˆ i  
ˆ1  w
Ui* (1)  Zi ˆ1  Yi 
1
i1
ˆ  v if potential benef
ˆ 2  wˆ i  
Ui* (2)  Zi ˆ2 (Yi T )
2
i2
ˆ v
ˆ i  
ˆ2  w
Ui* (2)  Zi ˆ2  Yi 
2
i2
ˆ  
ˆ  
ˆ  
ˆ ; 
ˆ M; 
ˆ D
where 
0
0
1
1
2
2
14
Ex ante estimator
 Average Intent to Treat effect (AIT) which provides an
estimate of the average impact of the availability of the
program to eligible households (in treatment
communities) by simulating impact of the program on
the sample of eligible age group of children;
 Assumes good implementation of program
 Attention: Ex ante model is static, i.e., no time or trend
effects.
 So, it is best to compare AIT (ex ante) with AIT (ex
post) obtained using 2DIF (which removes the trend
effect from the estimated impact) whenever is possible.
15
Results: PROGRESA
Table 1: Children's Occupational Choices in Mexico
Actual and Counterfactual Enrollment Rate for Target Population
Observed Ex Post ITT1 Ex Ante ITT
B
o
y
s
G
i
r
l
s
74.5%
8-11 years-old
93.8%
1.8%
0.7%
**
0.0%
0.4%
12-17 years-old
57.5%
5.8%
2.1%
**
5.9%
0.8%
**
8-17 years-old
69.4%
**
8-11 years-old
93.9%
-0.3%
1.0%
4.3%
0.7%
-0.2%
0.4%
12-17 years-old
47.9%
9.5%
2.2%
6.6%
0.8%
**
Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation
Note:
1: Results from Skoufias and Parker (2001) - tables 6.
**
Significant at 5% level; * Significant at 10% level.
-
4.0%
0.4%
**
8-17 years-old
-
**
16
Results: PROGRESA
Table 2: Children's Occupational Choices in Mexico
Actual and Counterfactual Child Labor Rate for Target Population
8-17 years-old
B
o
y
s
Observed
Ex Post ITT1
23.0%
-
-2.6%
**
0.3%
8-11 years-old
12-17 years-old
6.7%
37.3%
-1.1%
0.3%
0.0%
0.4%
-4.7%
**
0.0%
8-17 years-old
G
i
r
l
s
Ex Ante ITT
9.7%
-3.7%
**
0.5%
-
-0.7%
*
0.3%
8-11 years-old
12-17 years-old
4.2%
14.6%
0.0%
0.4%
0.0%
0.3%
-2.3%
0.0%
Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation.
Note:
1: Results from Skoufias and Parker (2001) - tables 5.
**
Significant at 5% level; * Significant at 10% level.
Standard Deviation computed by bootstrap method.
*
-1.3%
*
0.5%
17
Results: PROGRESA
Table 3: Poverty Index: Observed and Simulated
FGT(0)
%

"Mexico"
Treatment Comunities - Baseline
FGT(1)
%

FGT(2)
%

60.9% 1.1% 30.8% 1.1% 21.5% 1.1%
 Ex Post: Skoufias and Di Maro (2006) -16.5% 1.6% -24.3% 1.5% -29.2% 1.4%
 Ex Ante: Simulated
-13.1%
-26.6%
-33.1%
Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation;Skoufias and Di Maro (2006) table 4.
Note: Prices 11/99; Pesos
Poverty Line = Mean of Nov 98 Consumption per Capita. Obtained from Skoufias and Di Maro (2006)
18
Results: PROGRESA
Table 4: Estimated and Observed Progresa's Cost
BFL Simulation1,2 January 20021,3
Number of Families with Children Receiving
Benefits:
Average Monthly Transfer for Families with
Children
Estimated Total Annual Transfer
Scaling Up Total Transfer: Simulated average
times families in 2002
4,567
1,681,254
$301
$300
$16,517,580
$6,061,221,243
$6,080,632,364
n.a.
Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation;Skoufias and Di Maro (2006) table 4.
Note:
1: 11/1999 prices
2: Estimated based on the Baseline survey of 1997
3: National oficial numbers: http://www.oportunidades.gob.mx/indicadores_gestion/ene_feb_02/indice.htm
19
Conclusion
 Ex Ante model analysis indicates so far that they can be
very useful as well as powerful in predicting program
impacts.
 But work is still in progress.
Useful for simulating the design or re-design of a
transfer program.
Increasing demand from governments as Panama,
Jamaica and Ecuador
20