Practical Issues
in Applying CIE
Daniele Bondonio
University of Piemonte Orientale
1
How to incorporate policy relevant
issues in CIE
•Estimating average treatment effect for all the treated
units is often non-informative….”black box evaluation”
•Relevant policy issues can be incorporated in CIE by
estimating different impacts for different types of treated
units or different types of interventions
•E.g. [enterprise support] different impacts can be
estimated for:
-soft loans, grants, technical assistance
-different economic values of the subsidies
-different types of assisted firms (small, large)
(textile, chemicals, …services,…..)
2
•E.g. job training:
-different ages of participants
-different education levels
-previous work experience
•Urban revitalization:
-different degrees of initial distress
•Tourism promotion programmes:
-cultural events
-infrastructure improvements
-urban renovations
3
When different impacts are estimated
for different groups of treated units
&/or different variations of how the
treatment is provided
CIE can offer insights on
how a programme works
4
Single-programme vs multipleprogramme evaluations
•E.g. for enterprise support, local economic
development ……multiple different programs are
frequently available and affecting a same outcome
variable of interest
(firm sales, productivity, employment, investments….
indicators of quality of life from survey-data……)
•Caution with single-programme evaluations using
outcome variables that can be affected by other
programmes not included in the analysis
5
I)
Solution: try to include all programmes in your
analysis (e.g. enterprise support taking into account
multiple programs)
e.g. enterprise support: estimate the impact of different
economic intensity of support, different types of
incentives
II) Solution: try to focus on intermediate outcomes that
are affected solely by the programme
Y=number of tourism visits instead of more distant
outcomes like local employment (for tourism promotion
programmes)
Y= innovation outcomes instead of sales growth,
productivity…(for R&D support to enterprises)
6
Measuring the timing of the
intervention
•E.g. enterprise support. Data on program incentives
-dates on when the application was approved
-dates on incentive payments (1 installment, 2
installment…..)
Which dates are to be used in measuring the
beginning of the treatment?
•Overlooked issue with very strong consequences on
impact estimates
-when should we locate T?
•
Approval
•
1st payment
•
2nd payment
Time
7
•
If the programme support is wrongly placed in a time
earlier than the time in which the outcome of interest
could be potentially affected, the outcomes of such
time would be erroneously considered as exposed
to the treatment
•
By contrast, if a programme intervention is wrongly
placed in a time later than the time in which the
outcome of interest could be potentially affected, the
outcomes of such period would be erroneously
considered as not-exposed to the treatment
8
How to measure changes in
the outcome variable?
•Percentage change is often used:
D%=
(Ypost- Ypre)
(Ypre)
•D% adequate for many CIE (e.g. unit of observations:
individuals in need of assistance, geographic areas)
•D% to be used with caution for enterprise support:
the outcomes produced by the program (against the
estimated counterfactual) has a social utility which may
be independent from the initial dimension of the assisted
firms
[1_example]
9
How to choose the control
variables
•With pre- post- intervention data, what really matters is
separating changes due to the program from changes due to
other factors
•What characteristics may be different between treated and
non-treated units that put non-treated units at risk of being
exposed to different external factors generating changes in
the outcome at the same time of the intervention
•Even with discontinuity check with control variables that
units around the threshold are indeed similar: this is done
even with pure randomized experiments (with small numbers
sometimes randomization is unlucky….)
10
“Attrition bias” in outcome data
• “Attrition bias”
is generated when units with particularly
bad (or good results) drop out from the data used to
construct the outcome variable
•Overlooked issue with strong consequences on impact
estimates
•E.g.
enterprise support (balance sheet data contains only
corporations: treated firms with bad results may close or
may loose their corporate status and drop out from the data)
Intuitively: “attrition bias” = programme impact is
estimated only for the treated firms with no bad
results)
11
E.g.
•R&D and innovation support: treated firms (can be
start-ups, small firms) doing particularly well may be lost
in the data because of changes in their company names or
corporate status occurring in the growing process
•Enterprise support, Job training and social
programmes: treated & non-treated units with
particularly good (or bad) results may not be anymore
available to answer the questionnaires or interviews used
to build the outcome variable
“attrition” bias” can severely compromise CIE
check for possible reasons why units may
drop out from the data or check if characteristics of
respondents are similar to non-respondents
12
Long-term program effects & CIE
• For many programmes (e.g. local economic
development, enterprise support, R&D-innovation
support) extreme caution is needed in attempting to
estimate long-term effects
• In the medium and long-run it is likely that a positive
programme impulse spreads (with positive or negative
spillovers) also to the non-treated units
• Outcomes from the non-treated units become affected
by the intervention and cannot be anymore used to
estimate the counterfactual
13
Regional macro-effects & CIE
•In principle, every type of policy may produce long-run
impacts on macro-economic outcomes measured at
regional/country level
Estimating such regional impacts “spill-over effects”
is to be avoided when the economic importance of
the activities produced by the policy is negligible
compared to size of the regional economy and to the
importance of the large number of socio-economic
events (unrelated to the program intervention) that do
affect the macro regional/country outcomes
14
•When the economic importance of the activities produced by
the programme is not negligible compared to size of the
regional economy proceed as follows:
I) Use rigorous CIE the proximate impacts of the program
intervention on firm-level (e.g. how much of the subsidized
investment would have been done anyway?)
II) Include the results of CIE in macro-economic regional
simulation models yielding multiplying effect for the
regional economy
Keep in mind: in the absence of CIE the multipliers used
by regional simulations would be applied directly to
measures of programme activity (over-estimating the
macro-regional impact)
15
Continuing programmes
•With similar programmes available at pre-intervention
times, pre-intervention data may have been affected by
early treatments
•Caution in using pre-intervention characteristics as
control variables if you do not include in the analysis the
early rounds of the treatment
•Make efforts in gathering data to detect which units
were treated also in pre-intervention times
16
Practical advantages of PSM
•Post-intervention outcome data are not always
readily available
(delays in releasing recent data by statistical offices
or need of collecting data through questionnaires,
interviews, direct observation….)
•Estimation of PS requires exclusively preintervention data on observable characteristics
•PSM reduces the number of non-treated unit in
which post-intervention outcome data has to be
collected
17
How to choose the outcome
variable of the evaluation
•Designing CIE starts from understanding the rationale
of the programme intervention
(….. what is the market imperfection/negative externality
that the policy wants to correct)
•E.g. Enterprise support policies:
A) Aimed at producing firm-level investment, that
would never take place in the absence of the program
intervention (correction of credit market imperfections)
18
B) Aimed at boosting business activities in distressed
areas
•Rationale: market forces produce non-optimal allocation of
economic development with negative externalities (e.g. urban
sprawl, traffic congestion, pollution, abandoned areas that
may be conducive to crime…..)
•Goal: modifying the geographic allocation of firm-level
investment (e.g. new investments wanted in Region A, not in
Region B)
C) Aimed at boosting business activities in times with
economic crises (Countercyclical policies)
•Goal: modifying the temporal allocation of firm-level
investments (firms activities wanted in time I, not in time II)
19
Example I
Time
Region A
(disadvantaged)
Region B
+1 Million €
investment
FIRM X (treated)
Region A
(disadvantaged)
Region B
+1 Million €
investment
FIRM Y (Non-treated)
20
•For policies A) [aimed at correcting credit market
imperfections] the outcome variable has to keep tracks of all
firm-activities (e.g. investments) recorded anywhere:
Example I)= zero impact
•For policies B) [targeting distressed areas] the outcome
variable has to keep track of firm-activities recorded solely
in region A (disadvantaged)
Example I)= positive impact
•For policies C) [countercyclical interventions] the
outcome variable has to keep track of firm-activities recorded
solely in time I
Example I= uncertain( impact depends on
the timing of the investments, not location)
21
Example II
Time
Period II
Period I
+1 Million €
investment
+1Million €
investment
FIRM X (Assisted)
FIRM Y (Non-Assisted)
22
•For policies A) [aimed at correcting credit market
imperfections] the outcome variable has to keep tracks of all
firm-activities recorded at any time:
Example II= Zero impact
•For policies B) [targeting distressed areas] the outcome
variable has to keep track of firm-activities recorded solely
in region A (disadvantaged)
Example II)= uncertain (impact depends on
location of the investment not time)
•For policies C) [countercyclical interventions] the
outcome variable has to keep track of firm-activities recorded
solely in time I
Example II= positive impact
23
CIE with survey outcome data
• If surveys are planned to cover both treated and non
treated units, all CIE methods can be applied on survey
outcome data
•PSM can be used to generate surveys outcome data:
Surveys are to be run on:
all treated units + the matched non-treated units
•Radius matching procedures are preferable to ensure to
have a representative sample of comparable non-treated
units
24