INCLUSIVE GROWTH AND EMPLOYMENT IN EUROPE
3.11. 2015
Bratislava
What can a CIE tell us about the origins
of negative treatment effects of a
training programme
Miroslav Štefánik
miroslav.stefanik(at)savba.sk
Motivation
• Data availability (Official registers of
unemployed and Social insurance data)
• Critique of the training programme
• Counterfactual impact evaluation studies
come in a wide stream of literature
Description of the training programme
• Training activities are implemented (also subcontracted) by regional
offices of the Centre of Labour, Social Affairs and Family (COLSAF)
• The content of the trainings provided is widely defined (increasing
employability)
• All registered unemployed are eligible to participate in the training,
capacities are very limited
• Evaluation period 2007-2013
• Data allow us to follow the participants 24 months after the training
– From 1/2007-4/2008 trainings were designed and organised (also
subcontracted) by regional PES offices
– From 5/2008 training providers are selected by a public procurement at the
national level. Content of the trainings is decided based on the requests from
regional offices (general skills). Bratislava remains out of this mechanism.
– In 7/2010 new national projects are introduced with a rapid decline in
numbers of participants and the accessibility of the trainings
Periods of implementation
Source: Database on registered unemployed provided by COLSAF
Outcome indicators
• Working income
– constructed from the assessed base of social
insurance payments at the end of each month
• Employment
– constructed using the information about the
registration for social insurance payments (for
each month)
Propensity score matching
Probit model to predict the propensity score variable (PSV)
log Pr(Ii  1 | Xi ) 
0  2X  
• I- Participation in the training(0,1)
• X- vector of observed characteristics (all information available from the
database):
– Individual characteristics (gender, age, region, level and field of education, ...)
– Previous participation in other ALMM
– Pre-treatment unemployment (date of entering, length and no. of previous
unemployments, ...)
– Previous working experiences (days of previous working experience, economic
sector and occupation, ...)
– Family background (kids, marital status, ...)
– Declared skills (PC skills, languages, ...)
PSM model applied:
–1:1 matching of the nearest neighbour
–Replacement was allowed
–Exact matching/Subgrouping based on
regional offices
–Two matching variables
• PSV
• The date of entering unemployment
Sensitivity analysis:
• PSM using caliper radius (0.00075)
– Marginal improvement in balance
– 46,6% of participants were excluded, leaving us
with 21 288
• OLS estimation
Assumptions behind ex-post (control
group selection) selection
• Unconfoundedness assumption
After ensuring the balance on observable
characteristics, non-participants outcomes have the
same distribution that participants would have
experienced if they had not participated.
There are no unobservable characteristics influencing
the outcome.
• Assumption of common support
An area of common support exists=characteristics of
participants and non-participants overlap.
For each analysed participant, there is a non-participant
which is sufficiently similar.
Distribution of the PSV before
matching
0
.2
.4
psvar
.6
.8
1
0
Graphs by p46
1
PSV – Balance achievement
N
Log likelihood
Prob > chi2
Pseudo R2
Sensitivity
Specificity
Positive predictive value
Negative predictive value
Correctly classified
1 758 123
181 862,2
0,0000
0,5574
28,24%
99,75%
68,54%
98,65%
98,42%
1
0
1
Density
2
3
0
0
.5
1
0
.5
Pr(p46)
Graphs by p46
Source: Database on registered unemployed and Social insurance database
1
Proportion in %
Mean
Control group
N
mean (date of entry)
mean(length of previous
u)
mean(age)
mean(psvar)
Male
NP
Single
ISCO 1
ISCO 2
ISCO 3
ISCO 4
ISCO 5
ISCO 6
ISCO 7
ISCO 8
ISCO 9
Foreign language
Graduate
No elementary
Elementary
Lower socondary
Vocational secondary
Upper socondary vocational
Upper secondary general
First stage university
Second stage university
Ph.D.
Field of education 1
Field of education 2
Field of education 3
Field of education 4
Field of education 5
Field of education 6
Field of education 7
Field of education 8
Participants
Database
32.651
32.651
2.354.850
25.12.08
26.12.08
2.9.10
511,36
530,02
38,49674
38,32553
0,4148981
0,4173019
45,22
47,97
9,79
12,48
37,39
37,7
Previous occupation
15,57
17,59
2,9
2,93
4,8
4,73
14,07
13,99
8,04
7,6
13,36
13,7
0,58
0,62
15,42
15,12
15,16
14,37
75,85
76,02
2,76
2,73
Level of highest education achieved
0,09
0,08
18,53
19,15
0,43
0,43
26,11
26,11
39,29
37,72
5,46
5,36
0,44
0,44
21,09
20,3
0,02
0,03
Field of highest education achieved
19,26
19,94
0,34
0,53
24,5
24,17
17,3
17,15
6,23
6,31
0,79
1,04
20,36
20,59
9,14
8,58
312,59
34,95795
0,0622178
54,12
36,42
50,77
Balance
improvement
99,84%
91,42%
94,92%
99,32%
55,28%
88,76%
97,63%
30,70
1,58
3,18
7,62
4,7
11,69
0,93
13,21
17,23
66,19
2,54
84,59%
97,78%
95,48%
98,74%
84,83%
83,08%
87,10%
84,29%
72,38%
98,27%
84,21%
0,51
24,16
1,07
28,21
30,05
4,12
0,99
17,24
0,14
97,67%
87,62%
100,00%
100,00%
79,53%
91,94%
100,00%
74,18%
90,91%
26,26
0,64
21,94
15,68
5,18
1,51
19,8
7,57
89,24%
-72,73%
85,20%
89,80%
92,92%
46,81%
70,89%
44,55%
Date of entering unemployment
Treatment
0
5.0e-04
Density
.001
.0015
Control
01jul2006
Graphs by p46
01jan2009
01jul2011
01jan2014
01jul2006
01jan2009
01jul2011
01jan2014
Imputing the date of end of treatment
for the control group
Participants
Entering unemployment
(Balanced)
Number of days until the
end of training
End of the treatment
Start of the reference
period
Control group
Entering unemployment
(Balanced)
Imputed end of the treatment
Results ATT on earnings:
Comparison of methods
OLS
Month Coef.
S.E.
PSM NN
p.
N
Coef.
S.E.
p
PSM Caliper
N
Coef.
S.E.
p
N
6
-101,5
2,87
0,000 1757898 -20,25
2,36
0,000 60168 -49,87 2,0666 0,000
41380
12
-82,64
3,19
0,000 1757805 -16,44
2,96
0,000 59889 -38,80 2,4935 0,000
41380
18
-29,02
2,66
0,000 1757296 -3,03
3,19 0,3433 58907 -25,71 2,9141 0,000
41380
24
30,77
3,63
0,000 1756729 13,2
3,54 0,0002 57802
-7,07
6,97
Source: Database on registered unemployed and Social insurance database
0,311
41380
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Employment)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
PSM estimations by period of
implementation (Earnings)
Source: Database on registered unemployed and Social insurance database
CBA scenarios
Positive scenario (200701200804)
Additional
employment
Negative scenario (201201201212)
Real scenario (200701201312)
Employment
Employment
Employment
Additional Additional
Additional Additional
Additional
rate of
rate of
rate of
income employment
income employment
income
participants
participants
participants
1. Year
4,92%
45,75%
29,55
-17,85% 35,28%
-130,16
-7,75%
40,23%
-53,12
2. Year
12,00%
61,10%
64,12
-7,63%
40,77%
-97,52
-3,14%
54,60%
-36,67
3+ years 14,00%
65,95%
68,85
0,00%
40,77%
0,00
0,00%
58,38%
-35,00
CBA, 3 scenarios
Positive scenario
Negative scenario
Realistic scenario
60,000,000
50,000,000
40,000,000
30,000,000
20,000,000
10,000,000
0
2007
-10,000,000
-20,000,000
-30,000,000
-40,000,000
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
Findings
• Evaluated training measure seems to have initial negative
impact on participants chances to get employment and on
their income
• The length of this initial (negative) impact varies between
periods of implementation
• Positive impact of the measure is observed after 24 months
(on average). In some periods of implementation positive
impact is observable even earlier, in some periods there is
none positive impact observable.
• Provided trainings seem to be less effective during and
after the crisis.
• The way of implementation also plays a role in shaping the
impact of the measure.
Thank you for your attention
Miroslav Štefánik
miroslav.stefanik(at)savba.sk