Evaluation of an ESF funded
training program to firms:
The Latvian case
L. Elia, A. Morescalchi, G. Santangelo
Centre for Research on Impact Evaluation (CRIE)
Joint Research Centre, European Commission
[email protected]
Andrea Morescalchi
Ministry of Finance, Riga (LV)
10-11 March 2015
1
CRIE: What is it & Who is in it?
What?
Centre for Research on Impact Evaluation
Joint DG EMPL-DG JRC initiative
Established in June 2013
Support to MS and DG EMPL to set up the necessary arrangements for
carrying out Counterfactual Impact Evaluations (CIE) of ESF
funded interventions
Who?
Researchers with a background in Economics and Statistics
2
CRIE: Where?
at DG JRC 
in its Ispra (Italy)
site 
3
How was CRIE born?
Difficult economic times
 Even more pressing need for the EU to demonstrate the achievements of
ESF and other financial instruments
2014-2020 programming period:

shift toward impact evaluation
DG EMPL’s initiative to help MS in this shift toward impact
evaluation
...
CRIE in collaboration with DG JRC
4
1. The intervention
5
The intervention – 1st call
• Type: training to employees of firms. The type, duration and number of
trainings can vary over firms and employees within firm.
• Motivation: improve the performance and competitiveness of firms by
enhancing the skills and the productivity of their employees.
• Eligibility: associations formed by at least 5 firms belonging to the same
industrial sector can apply.
• Duration: March 2009 - April 2010.
• Participants: 14 projects selected for a total of 111 firms
6
The intervention – 2nd call
• Duration: January 2011 – July 2015.
• Very similar to 1st call except:
1.
total turnover of project co-applicants should exceed 142.29 million Euro
in the last year
2.
the applicant association should have been at least 3 years in operation
3.
Larger group of recipients (>1,000 firms)
7
2. Related literature on
training to firms
8
Human capital theory
Returns to on-the-job training estimated by:
• individual productivity measures
• wages: proxy for individual productivity
• employment prospects:
results of enhancing individuals’ skills and competencies
(Latvia: Dmitrijeva and Hazans, 2007)
• overall firm level outcomes (non-CIE methods)
• value added or sales (absolute or per-capita value):
measures of firm performance
Training characteristics to be taken into account:
• Type
• Hours
• Duration
• Costs
9
Explanatory variables
Firm
• Capital
• Labor
• R&D
• Sector of activity
• Size
• Location
Workforce (shares)
• Occupation
• Education
• Experience
• Age
• Gender
• Worker turnover
10
3. The data
11
Data description
• Data sources:
(1) project database from the Investment and Development Agency of
Latvia (IDAL): # of employees receiving training;
(2) business data collected from the State Revenue Service;
• Period: Yearly firm level data available for 2008-2013
• Variables: Firms’ age, # of employees, location, NACE code, (aftertax) turnover, production costs, profit, salaries, assets, capital
investments, fixed assets, long-term intangible investments
12
Data description – 4 main firms groups
1. Treated in 1st call: 111
2. Treated in 2nd call: 1,118
3. Never treated: 30,557 (nearly 40% of the country total),
4. Treated in both calls: 38, not used in the analysis
Sample adjustments:
•
missing data and outliers removed from the sample: group (1) drops to 66
•
firms starting 2nd call before December 2010 are removed: group (2) drops to 391!
13
Data description – sample statistics
Group
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Treated 1st call
Treated 2nd call
Never treated
Variable
profit
profit
profit
profit p.e.
profit p.e.
profit p.e.
turnover
turnover
turnover
turnover p.e.
turnover p.e.
turnover p.e.
nr. employees
nr. employees
nr. employees
% treated
% treated
% treated
age
age
age
2008
124.1
208.7
-3.9
1.8
4
-0.5
2487.8
10135.3
444.9
40.4
65.2
35.5
53.7
250
13.2
0
0
0
7.7
7.6
5.7
2009
-2.5
71.8
-11.2
1
3.8
-2
1762.3
7710.3
295.6
39.2
82.6
27.6
43.5
199.9
11
40
0
0
8.9
8.2
6.4
Year
2010
37.8
-38.8
-0.7
0.5
3.7
-1.1
1300.4
7665.8
323.1
37.4
75.2
30.6
30.1
180.2
11
40
0
0
9.6
8.9
6.9
2011
-16.9
92.5
9.6
-0.5
4.3
0.2
1417.8
8227.4
429.1
41.7
83.5
41.1
27.9
187.5
12.3
40
0
0
10.8
9.3
7.2
2012
26.5
309
10.3
0
3.8
-0.7
1573.6
8763.6
454.7
41.2
86
41.6
30.2
181.4
12.7
40
0
0
11.7
10
7.5
2013
62.1
345.4
15
-0.1
4
1.6
1284.6
9479.1
465.9
39.7
78.4
43.4
29.2
196.9
13.3
40
0
0
12.8
11.1
7.7
14
Data description – profit per employee
15
Data description – turnover per employee
16
4. Methodology
17
DiD combined with PSM
• Technique: combination of Difference-in-differences (DiD)
with Propensity Score Matching (PSM).
• PSM: used to select a control group of firms similar to treated
firms in terms of observed characteristics.
• DiD: applied on the sample of treated and untreated peers.
18
Propensity score matching
• Propensity score matching (PSM) is used for selecting control units
from two comparison groups in 2008
• Variables used for matching: capital investment, long-term intangible
investments, production costs, pre-program profit and turnover (all
normalized by the number of employees), age, number of employees,
NACE sector of activity and geographical location
• Four algorithms are employed: one-nearest neighbor, with and without
replacement, three-nearest neighbors and kernel matching
19
Difference-in-differences (DiD)
• DiD regression model estimated after matching untreated to treated:
𝑇
𝑦𝑒𝑎𝑟𝑡 + 𝝆 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 ∙ 𝑑𝑡 + 𝛽′ 𝑥𝑖𝑡 + 𝑢𝑖𝑡
𝑦𝑖𝑡 = 𝛼𝑖 + 𝜆𝑡
𝑡=1
• 𝑡𝑟𝑒𝑎𝑡𝑚𝑒𝑛𝑡𝑖 is the share of employees in training
• 𝑑𝑡 = 1 if 𝑡 ≥ 2008
•
𝑇
𝑡=1 𝑦𝑒𝑎𝑟𝑡
are time dummies
• 𝑥𝑖𝑡 are observed firms’ characteristics
• 𝛼𝑖 captures time-invariant unobserved firms’ attributes
• One model is estimated for each of the two control groups
20
5. Results
21
Results for profit per employee
Treatment
invest.p.e.
int.inv p.e.
age
employees
Obs.
Nr. of
firms
Control gr.
Years
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
1nn
1nn_nr
3nn
kern
1nn
1nn_nr
3nn
kern
1nn
1nn_nr
3nn
kern
-2.343
-2.343
-1.571
1.566
-2.574
-2.574
-1.254
2.180
1.295
1.428
1.412
-0.149
(3.003)
(3.002)
(1.930)
(1.080)
(3.269)
(3.269)
(2.071)
(1.386)
(0.995)
(1.130)
(1.098)
(1.101)
1.249***
1.256***
1.161***
0.363
1.691
1.691
1.107
0.002
1.654**
2.153**
2.103**
0.411
(0.450)
(0.446)
(0.412)
(1.258)
(1.539)
(1.539)
(1.443)
(3.757)
(0.660)
(1.013)
(0.945)
(0.578)
0.863
0.861
1.048
1.434
2.311
2.311
2.038
5.784
0.324
0.127
0.122
0.628
(0.996)
(0.995)
(0.868)
(1.613)
(2.557)
(2.557)
(1.951)
(4.767)
(0.921)
(0.722)
(0.600)
(0.932)
0.448
0.454
0.353
0.077
3.524
3.524
2.099
0.744
-0.439
-1.154
-1.096
0.093
(0.658)
(0.657)
(0.402)
(0.176)
(3.818)
(3.818)
(2.521)
(0.658)
(0.517)
(0.851)
(0.741)
(0.401)
3.929**
3.925**
3.403**
7.915***
6.889
6.889
5.217
17.490**
3.976**
0.502
0.625
2.006*
(1.596)
(1.592)
(1.388)
(3.011)
(4.787)
(4.787)
(3.668)
(7.719)
(1.713)
(3.314)
(2.892)
(1.063)
444
449
669
40,476
232
232
342
24,136
278
322
369
753
95
98
156
9,228
84
84
124
9,228
96
111
127
254
no treat
20082013
no treat
20082013
no treat
20082013
no treat
20082013
no treat
20082010
no treat
20082010
no treat
20082010
no treat
20082010
2nd call
20082010
2nd call
20082010
2nd call
20082010
2nd call
20082010
*** p<0.01, ** p<0.05, *p<0.1; clustered standard errors in parenthesis
Short-term impact
22
Results for (log) turnover per employee
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
1nn
1nn_nr
3nn
kern
1nn
1nn_nr
3nn
kern
1nn
1nn_nr
3nn
kern
0.021
0.021
0.097
0.076
0.033
0.033
0.179
0.137
0.116
0.088
0.110
-0.088
(0.087)
(0.087)
(0.098)
(0.079)
(0.093)
(0.093)
(0.111)
(0.087)
(0.095)
(0.093)
(0.094)
(0.098)
0.075
0.076
0.152***
0.139***
0.089
0.089
0.136
0.196***
0.390***
0.393***
0.380***
0.303***
(0.058)
(0.058)
(0.058)
(0.035)
(0.107)
(0.107)
(0.088)
(0.045)
(0.075)
(0.074)
(0.074)
(0.081)
0.139*
0.139*
0.117
0.054
0.249
0.249
0.195
0.035
0.079
0.077
0.068
0.167
(0.078)
(0.078)
(0.078)
(0.046)
(0.235)
(0.235)
(0.210)
(0.113)
(0.154)
(0.122)
(0.097)
(0.213)
-0.014
-0.013
0.007
-0.025
-0.057
-0.057
-0.143***
-0.104***
-0.056
-0.047
-0.062
0.085
(0.029)
(0.029)
(0.025)
(0.019)
(0.060)
(0.060)
(0.050)
(0.030)
(0.047)
(0.042)
(0.038)
(0.062)
0.200
0.200
0.186
0.121
0.044
0.044
0.092
0.084
0.256
0.164
0.146
0.111
(0.171)
(0.171)
(0.143)
(0.084)
(0.231)
(0.231)
(0.219)
(0.130)
(0.214)
(0.197)
(0.174)
(0.264)
Obs.
444
449
669
40,476
232
232
342
24,136
278
322
369
753
Nr. firms
95
98
156
9,228
84
84
124
9,228
96
111
127
254
no treat
no treat
no treat
2008-2013
2008-2013
2008-2013
no treat
20082013
no treat
20082010
no treat
20082010
no treat
20082010
no treat
20082010
2nd call
20082010
2nd call
20082010
2nd call
20082010
2nd call
20082010
Treatment
invest p.e.
int. inv p.e.
age
employees
Control gr.
Years
*** p<0.01, ** p<0.05, * p<0.1; clustered standard errors in parenthesis
Short-term impact
23
6. Limitations
24
Results to be taken with extreme caution…
1. Information on the kind, duration and number of training
courses taken by employees is not available
2. exact start and end date of treatment not available
3. strong selection into treatment
4. association identifier not present in the data
5. information not available at the individual level
6. size of the treatment group is very small
7. only 1 year available before the intervention
25
7. Concluding Remarks
26
Insights and recommendations
• The limitations encountered in the present evaluation make
it difficult to discuss the replicability and the scaling up of
the intervention under scrutiny.
• Future availability of more detailed data and information
will be critical to produce more reliable and robust impact
evaluations
27