A stochastic dominance approach
to program evaluation
And an application to child nutritional
status in arid and semi-arid Kenya
Felix Naschold University of Wyoming
Christopher B. Barrett Cornell University
May 2012 seminar presentation
University of Sydney
Motivation
1.
Program Evaluation Methods
 By design they focus on mean
Ex: “average treatment effect” (ATE)
 In practice, often interested in broader distributional impact
 Limited possibility for doing this by splitting sample
2.
Stochastic dominance
 By design, look at entire distribution
 Now commonly used in snapshot welfare comparisons
 But not for program evaluation. Ex: “differences-in-differences”
3.
2
This paper merges the two
 Diff-in-Diff (DD) evaluation using stochastic dominance
(SD) to compare changes in distributions over time
between intervention and control populations
Main Contributions
1. Proposes DD-based SD method for program evaluation
2. First application to evaluating welfare changes over time
3. Specific application to new dataset on changes in child
nutrition in arid and semi-arid lands (ASAL) of Kenya
3

Unique, large dataset of 600,000+ observations collected by the
Arid Lands Resource Management Project (ALRMP II) in Kenya

(One of) first to use Z-scores of Mid-upper arm circumference
(MUAC)
Main Results
1.
Methodology
 (relatively) straight-forward extension of SD to dynamic
context: static SD results carry over
 Interpretation differs (as based on cdfs)
 Only feasible up to second order SD
2.
Empirical results
 Child malnutrition in Kenyan ASALs remains dire
 No average treatment effect of ALRMP expenditures
 Differential impact with fewer negative changes in treatment
sublocations
 ALRMP a nutritional safety net?
4
Program evaluation
(PE) methods
 Fundamental problem of PE: want to but cannot observe a
person’s outcomes in treatment and control state
i  xiT  xiC
 Solution 1: make treatment and control look the same
(randomization)
 Gives average treatment effect as
E    E  xT   E  xC 
 Solution 2: compare changes across treatment and control
(Difference-in-Difference)
 Gives average treatment effect as:
5
E     E  xT ,t  xT ,t 1   E  xC ,t  xC ,t 1 
New PE method
based on SD
 Objective: to look beyond the ‘average treatment effect’
 Approach: SD compares entire distributions not just their
summary statistics
 Two advantages
Circumvents (highly controversial) cut-off point
Examples: poverty line, MUAC Z-score cut-off
2. Unifies analysis for broad classes of welfare indicators
1.
6
Stochastic Dominance
 First order: A FOD B up to z  xmin , xmax  iff
FB  x   FA  x   0  x  xmin , z 
Cumulative %
of population
FB(x)
FA(x)
xmax
 Sth order: A sth order dominates B iff
0
MUAC
score
Z-
FBs  x   FAs  x   0  x  xmin , z 
7
SD and single differences
 These SD dominance criteria
 Apply directly to single difference evaluation (across time OR
across treatment and control groups)
 Do not directly apply to DD
 Literature to date:
 Single paper: Verme (2010) on single differences
 SD entirely absent from the program evaluation literature (e.g.,
Handbook of Development Economics)
8
Expanding SD to DD
estimation - Method
 Practical importance: evaluate beyond-mean effect in non



9
experimental data
Let   xt  xt 1 , and G denote the set of probability
density functions of Δ, with 𝑔𝐴 ∆ , 𝑔𝐵 ∆ ∈ 𝑮
The respective cdfs of changes are GA(Δ) and GB(Δ)
Then A FOD B iff GB     GA     0 min , max 
A Sth order dominates B iff GBs     GAs     0 min , max 
Expanding SD to DD:
interpretation differences
1. Cut-off point in terms of changes not levels.
 Cdf orders change from most negative to most positive  ‘initial
poverty blind’ or ‘initial malnutrition blind’.
 (Partial) remedy: run on subset of ever-poor/always-poor
2. Interpretation of dominance orders
 FOD: differences in distributions of changes between intervention
and control sublocations
 SOD: degree of concentration of these changes at lower end of
distributions
 TOD: additional weight to lower end of distribution. Is there any
value to doing this for welfare changes irrespective of absolute welfare?
Probably not.
10
Setting and data
 Arid and Semi-arid districts in Kenya
 Characterized by pastoralism
 Highest poverty incidences in Kenya, high infant mortality and
malnutrition levels above emergency thresholds
 Data
 From Arid Lands Resource Management Project (ALRMP) Phase II
 28 districts, 128 sublocations, June 05- Aug 09, 602,000 child obs.
 Welfare Indicator: MUAC Z-scores
 Severe malnutrition in 2005/6:
 Median child MUAC z-score -1.22/-1.12 (Intervention/Control)
 10 percent of children had Z-scores below -2.31/-2.14 (I/C)
 25 percent of children had Z-scores below -1.80/-1.67 (I/C)
11
The pseudo panel
 Sublocation-specific pseudo panel 2005/06-2008/09
 Why pseudo-panel?
Inconsistent child identifiers
2. MUAC data not available for all children in all months
3. Graduation out of and birth into the sample
1.
How?





12
14 summary statistics for annual mean monthly sublocation specific stats: mean & percentiles and ‘poverty measures’
Focus on malnourished children
Thus, present analysis median MUAC Z-score of children z ≤ 0
Control and intervention according to project investment
Results: DD Regression
 Pseudo panel regression model
where D is the intervention dummy variable of interest
NDVI is a control for agrometeorological conditions
L are District fixed effects to control for unobservables
within major jurisdictions
 No statistically significant average program impact
13
DD regression panel results
(1)
median of
MUAC Z <0
(2)
10th
percentile
(3)
25th
percentile
(4)
median of
MUAC Z <-1
(5)
median of
MUAC Z <-2
0.0735
0.0832
0.0661
0.0793
0.0531
(0.248)
(0.316)
(0.371)
(0.188)
(0.155)
1.308*
2.611***
2.058***
0.927*
0.768*
(0.0545)
(0.00294)
(0.00754)
(0.0997)
(0.0767)
(change in NDVI)2 2005/06-08/09
-12.91**
(0.0293)
-8.672
(0.136)
-12.70*
(0.0510)
-0.954
(0.802)
1.924
(0.479)
Constant
0.501***
0.892***
0.839***
0.203***
0.120***
(2.99e-07)
(1.40e-08)
(8.70e-09)
(0.000133)
(0.00114)
114
114
114
114
106
0.319
0.299
0.297
0.249
0.280
VARIABLES
intervention dummy
change in NDVI 2005/06-08/09
Observations
R-squared
14
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
SD Results
Three steps:
 Steps 1 & 2: Simple differences
 SD within control and treatment over time:
No difference in trends. Both improved slightly.
 SD control vs. treatment at beginning and at end:
Control sublocations dominate in most cases, intervention
never dominates.
 Step 3: SD on Diff-in-Diff (results focus for today)
15
Expanding SD to DD –
controlling for covariates
 In regression Diff-in-Diff: simply add (linear) controls
 In SD-DD need a two step method
Regress outcome variable on covariates
2. Use residuals (the unexplained variation) in SD-DD
1.

16
In application below, use first stage controls for agrometeorological conditions (as reflected in remotely-sensed
vegetation measure, NDVI).
FOSD Difference Intervention vs. Difference Control
.6
.4
.2
0
-1
-.4
.2
.8
1.4
2
difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-200
Control
intervention
FOSD Difference Intervention vs. Difference Control
0
.1
.2
Median MUAC of obs<0. Categorization by Investment
-.2
% of sublocations
For (drought-adjusted) median
MUAC z-scores:
 Below z=0.2, intervention
sites FOD control sites,
although not at 5% statistical
significance level.
 ALRMP interventions appear
moderately effective in
preventing worsening
nutritional status among
children.
-.1
% of sublocations
.8
1
Median MUAC of obs<0. Categorization by Investment
-1
17
-.4
.2
.8
1.4
2
difference in median MUAC Z-score with MUAC<0. drought adjusted. 2005/06-2008/09
Confidence interval (95 %)
Estimated difference
Similar results at other
quantile breaks
25th percentile MUAC. Categorization by Investment
0
.2
.4
.6
.8
1
FOSD Difference Intervention vs. Difference Control
-1.5
-.8
-.1
.6
1.3
2
difference in 25th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
18
Control
intervention
Similar results at other
quantile breaks
10th percentile MUAC. Categorization by Investment
0
.2
.4
.6
.8
1
FOSD Difference Intervention vs. Difference Control
-1.5
-.8
-.1
.6
1.3
2
difference in 10th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09
Control
19
intervention
Conclusions
 Existing program evaluation approaches focus on estimating
the average treatment effect. In some cases, that is not really
the impact statistic of interest.
 This paper introduces a new SD-based method to evaluate
impact across entire distribution for non-experimental data
 Results show the practical importance of looking beyond
averages
 Standard Diff-in-Diff regressions: no impact at the mean
 SD DD: intervention locations had fewer negative observations
and of smaller magnitude, especially median and below
 ALRMP II may have functioned as nutritional safety net (though
only correlation, there is no way to establish causality)
20
Thank you for your time,
interest and comments
21
SD, poverty & social
welfare orderings (1)
1. SD and Poverty orderings
 Let SDs denote stochastic dominance of order s and Pα stand
for poverty ordering (‘has less poverty’)
 Let α=s-1
 Then A Pα B iff A SDs B
 SD and Poverty orderings are nested
 A SD1 B  A SD2 B  A SD3B
 A P1 B  A P 2 B  A P3 B
22
SD, poverty & social
welfare orderings (2)
2. Poverty and Welfare orderings (Foster and Shorrocks 1988)
 Let U(F) be the class of symmetric utilitarian welfare functions
 Then A Pα B iff A Uα B
 Examples:
 U1 represents the monotonic utilitarian welfare functions such that
u’>0. Less malnutrition is better, regardless for whom.
 U2 represents equality preference welfare functions such that u’’<0. A
mean preserving progressive transfer increases U2.
 U3 represents transfer sensitive social welfare functions such that
u’’’>0. A transfer is valued more lower in the distribution
 Bottom line: For welfare levels tests up to third order make sense
23
The data (2) – extent of
malnutrition
24
Table 3 10th percentile MUAC Z-score – whole sample
Year
Garissa Kajiado Laikipia Mandera Marsabit
Mwingi
Narok
Nyeri
Tharaka
Turkana
2005/06
-2.4
-2.14
-1.75
-2.65
-2.33
-2.36
-2.55
-1.67
-1.87
-2.26
2008/09
-1.88
-2.22
-2.1
-2.13
-2.29
-2.14
-2.35
-1.54
-1.74
-2.25
Table 4 25th percentile MUAC Z-score – whole sample
year
Garissa Kajiado Laikipia Mandera Marsabit
Mwingi
Narok
Nyeri
Tharaka
Turkana
2005/06
-1.97
-1.67
-1.16
-2.06
-1.79
-1.84
-1.96
-1.2
-1.45
-1.85
2008/09
-1.45
-1.76
-1.4
-1.69
-1.69
-1.68
-1.76
-1.15
-1.28
-1.86
DD Regression 2
 Individual MUAC Z-score regression
 To test program impact with much larger data set
 Still no statistically significant average program impact
25
Results – DD
regression indiv data
Dependent variable: Individual MUAC Z-score
VARIABLES
time dummy (=1 for 2008/09)
0.0785
(0.290)
control - intervention by investment
-0.0576
(0.425)
Diff in diff
0.0245
(0.782)
Normalized Difference Vegetation Index
1.029***
(6.25e-07)
Constant
-1.391***
(0)
Observations
R-squared
26
Robust p-values in parentheses
*** p<0.01, ** p<0.05, * p<0.1
District dummy variables included.
271061
0.033
Full SD results
Sublocation panel
Median MUAC of obs < 0
% below -1 SD
Dominance
I.1 Intervention 05/06-08/09
FOSD
SOSD
TOSD
I.2 Control 05/06-08/09
FOSD
SOSD
TOSD
II.1 Intervention vs. Control 05/06
FOSD
SOSD
TOSD
II.2 Intervention vs. Control 08/09
FOSD
SOSD
TOSD
III. Diff Intervention vs Diff.
Control
FOSD
SOSD
Which*
Signif.
Y
Y
Y
08/09
08/09
08/09
Y
Y
Y
Individual data
MUAC Z-Score
Dominance
Which*
*
Signif.
NS
S
S
Almost
Y
Y
08/09
08/09
08/09
NS
NS
NS
08/09
08/09
08/09
NS
NS
NS
Y
Y
Y
08/09
08/09
08/09
Y (almost)
Y
Y
Control
Control
Control
NS
NS
NS
Almost
Y
Y
#
N
Unclear
Unclear
-
NS
NS
NS
N
Y?
-
NS
NS
Dominance
Which*
Signif.
Y
Y
Y
08/09
08/09
08/09
S
S
S
NS
NS
NS
Y
Y
Y
08/09
08/09
08/09
S
S
S
Control
Control
NS
NS
NS
Y
Y
Y
Control
Control
Control
S
S
S
N
Y
Y
Control
Control
Control
NS
NS
NS
Y
Y
Y
Control
Control
Control
S
S
s
N
Y
Interve
ntion
NS
NS
* Lower curves to the right are dominate for these indicators for which a greater number indicates ‘better’.
** For parts I. and II. higher curves to the left dominate for the proportion of observations below -1SD, as lower
proportions are ‘better’. In contrast, for changes from 2005/06-2008/09 in part III. larger positive changes are better,
27 curves to the right dominate.
so lower
# Control sites dominate up to MUAC Z-score of -0.1. Intervention sites dominate for MUAC Z-score > 0.
FOSD Difference Intervention vs. Difference Control
0
.2
.4
.6
.8
1
Median MUAC of obs<0. Categorization by Investment
-1
-.4
.2
.8
1.4
2
difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-200
Control
28
intervention