+
Measuring Investment in Human Capital Formation:
An Experimental Analysis of Early Life Outcomes
Orla Doyle, Colm Harmon, James Heckman,
Caitriona Logue, Seong Moon
UCL, 25th-26th June 2012
+ Preparing for Life Programme

One of the first experimental early childhood
intervention in Ireland

Funded by Irish Government (DCYA) & The Atlantic Philanthropies

Community-led initiative operated by Northside Partnership in Dublin

~ 6,400 inhabitants

33% dependent on social welfare

60% live in social housing (>3 times national average)

47% lone mothers (29% national average)

16% unemployed (3 times national average)

66% early school leavers (38% national average)

5% third level education (29% national average)
Source: Census (2006)- CSO

Belief within the community that children were lagging behind their peers
+ Design of PFL

Aim: Improve levels of school readiness by assisting parents in
developing skills to prepare their children for school

Bottom-up approach: community initiative involving 28 community
groups, service providers, & local representatives

Theoretical Framework: Grounded in several psychological theories
of development including the theory of human attachment, socioecological theory of development and social-learning theory

Evaluation: Randomised Control Trial design
+
Design of Preparing for Life
+ PFL Evaluation
Impact Evaluation

Data collection: Pre-intervention (baseline), 3 mnts (WASI), 6mths,
12mths, 18mths, 24mths, 3yrs, 3.5yrs, 4yrs

Informant: Mother is the primary informant, but also fathers, child, other
independent data sources (birth records)
Implementation Evaluation

Aim: Delve into the blackbox of programme effectiveness & evaluate
fidelity to the PFL model

Data collection:
1.
Implementation data on the Database Management System
2.
Focus groups with participants
3.
Semi-structured interviews with mentors/IO
+ Recruitment

Eligibility Criteria:



Recruitment:



Cohort of pregnant women residing in PFL catchment area between Jan
2008-August 2010 (32 months)
Includes preparious and non-preparious women
Maternity hospital at first booking visit (b/w 12-26 weeks)
Within the local community
Population-based recruitment rate, based on all live births during
the recruitment phase, was 52%
+ Computerized Randomisation Procedure

Unconditional probability randomisation strategy

After informed consent obtained, mother ask to press the key to allocate
her assignment condition

Evaluation & Implementation team automatically received an email with
the assigned PFL ID number and treatment condition

Preserves the integrity of the procedure as no opportunity for recruiter
to intentionally influence assignment

115 allocated to High treatment group
118 allocated to Low treatment group

+ Baseline Data Collection

Conducted post randomisation, pre intervention

Baseline data wave completed in August, 2010

123 measures:

Interviews conducted
Demographics & SES, health behaviour & pregnancy, parenting,
social support, psychological assessments (maternal well-bring, personality traits, selfesteem, self efficacy, attachment, time preferences)



PFL High treatment group: 104
PFL Low treatment group: 101
Used to determine the effectiveness of the randomisation
procedure
+ Methodology
 Permutation based hypothesis testing (Heckman et al. 2010)



Classical hypothesis tests unreliable when the sample size is small and the
data are not normally distributed
Permutation test are distribution free, thus suitable in small samples
Based on the assumption of exchangeability between treatment
conditions under the null hypothesis
 Stepdown procedure (Romano & Wolf, 2005)



Ignoring the multiplicity of tests may lead to the rejection of “too many”
null hypotheses
Test multiple hypotheses simultaneously by controlling overall error rates
for vectors of hypotheses using the family-wise error rate (FWER) as a
criterion
Less conservative & more powerful than other methods as takes account
of statistical dependencies between tests
+ Summary of Permutation Tests Examining
Differences at Baseline
Proportion of Measures Not Significantly
Different at Baseline
Topic
PFL Low – PFL High
Family Socio-demographics
33/33 (0%)
Maternal Well-being
24/24 (0%)
Maternal Health & Pregnancy
35/35 (0%)
Parenting & Childcare
10/13 (74%)
Social Support
17/18 (94%)
Total NOT Statistically Different

Randomisation worked!
119/123 (97%)
+
Permutation Tests Comparing Baseline Differences
in Selected Family Socio-Demographics
Low Treatment – High Treatment
Instrument
Mother’s Age
First-time Mothers
Mother Married
Mothers with Junior Certificate
Qualification or Lower
Mothers with Primary Degree
Mothers Unemployed
Annual Income of Working
Mothers (in Euros)
Fathers Unemployed
Residing in Social Housing
In Possession of a Medical Card
N
(nLOW/ nHIGH)
205
MLOW
(SD)
25.30
MHIGH
(SD)
25.46
(101/104)
(5.99)
(5.85)
205
0.50
0.54
(101/104)
(0.50)
(0.50)
205
0.18
0.14
(101/104)
(0.38)
(0.35)
205
0.40
0.34
(101/104)
(0.49)
(0.47)
205
0.03
0.03
(101/104)
(0.17)
(0.17)
205
0.41
0.43
(101/104)
(0.49)
(0.50)
75
19,602
19,224
(38/37)
(8,093)
(9,851)
198
0.31
0.43
(97/101)
(0.46)
(0.50)
204
0.55
0.55
(101/103)
(0.50)
(0.50)
205
0.66
0.60
(101/104)
(0.47)
(0.49)
p
Effect Size
(d)
ns
.03
ns
.09
ns
.09
ns
.12
ns
.01
ns
.05
ns
.04
ns
.24
ns
.00
ns
.14
+ Analysis of 6 Month
Outcome Data

Test for treatment effects across 8 main domains including
160 measures








Child development
Child health
Parenting
Home environment
Maternal health
Social support
Childcare & service use
Household factors & SES

25 Step-down categories defined

258 interviews conducted


PFL High treatment group: 84
PFL Low treatment group: 90
12
+ Attrition & Disengagement up to 6
months

Official dropout between
baseline - 6 months:



High treatment – 13%
Low Treatment – 6%
Disengagement between
baseline - 6 months :

High treatment – 9%

Low Treatment – 10%
Baseline Characteristics Associated with Attrition
Variables
Weeks in preg. at programme entry
Mother's age
Partnered
Married
Living with parent(s)
First time mother
Low education
Mother employed
Saves regularly
Social housing
Cognitive Resources (WASI) at 3MO
Physical Health Condition
Mental Health Condition
Smoking during pregnancy
Drinking during pregnancy
Drug ever used
Vulnerable attachment (VASQ)
Positive parenting attitudes (AAPI)
Self efficacy (Pearlin)
High Treatment
Group
p
Low Treatment
Group
p
0.651
0.563
0.505
0.063
0.410
0.503
0.541
0.699
0.477
0.556
0.629
0.073
0.121
0.131
0.010**
0.772
0.337
0.545
0.639
0.759
0.570
0.097
1.000
0.511
0.760
0.197
1.000
0.367
0.439
0.308
0.159
0.675
0.324
0.622
0.565
0.757
0.263
0.389
+ RESULTS FOR HIGH & LOW TREATMENT GROUPS:
CHILD DEVELOPMENT
ASQ Scores & Difficult
Temperament
ASQ Gross Motor Score
ASQ Communication Score
*Difficult Temperament
ASQ Personal Social Score
*ASQ Social-Emotional
ASQ Fine Motor Score
ASQ Problem Solving
N
(nHIGH/ nLOW)
MHIGH
(SD)
MLOW
(SD)
Individual
Test
p1
Step Down
Test
p2
Effect Size
d
ns
ns
0.18
ns
ns
0.16
ns
ns
0.09
ns
ns
0.05
ns
ns
0.03
ns
ns
0.06
ns
ns
0.07
173
40.78
38.50
(83/90)
(11.93)
(12.99)
173
53.07
51.78
(83/90)
(7.84)
(8.49)
173
11.70
12.21
(83/90)
(5.71)
(5.50)
172
46.52
45.94
(82/90)
(12.09)
(13.57)
173
14.76
15.17
(83/90)
(10.68)
(13.75)
173
50.84
51.39
(83/90)
(9.46)
(10.17)
173
51.87
52.56
(83/90)
(9.39)
(9.92)
Notes: 1 one-tailed (right-sided) p value from an individual permutation test with 1000
replications. 2 one-tailed (right-sided) p value from a Step Down permutation test with 1000
replications. * indicates the variable was reverse coded for the testing procedure. ‘ns’ indicates
the variable is not statistically significant.
+ Summary of Six Month Results
Proportion of Measures Significantly Different at
Six Months
Child Development
0% (0/13)
Step Down
categories
0/2
Child Health
10% (3/30)
0/3
Parenting
23% (5/22)
1/5 (PSI)
Home Environment & Safety
36% (8/22)
1/2 (HOME)
Maternal Health
5% (1/20)
1/4 (Health)
Social Support
38% (5/13)
0/2
Childcare & Service Use
7% (1/14)
0/2
Household Factors & SES
0% (0/26)
0/5
14% (23/160)
3/25 categories
Domain
Total Statistically Different
PFL Low – PFL High
+ Summary of 6 month results

Majority of the results for High V’s Low are in hypothesized direction

A few significant differences identified (14%)

In line with other home visiting programmes
+ Additional Analysis


Interaction & Subgroup analysis

Gender, lone parents v partnered parents, first time v non first time
parents, high versus low IQ mothers, high v low family risk

Programme appears to benefit high IQ mothers, families with multiple
children & families with multiple risks
Analysis of engagement (treatment intensity)

On average, high treatment group received 14 home visits of ~1 hr in
duration= total treatment = 14 hours

Higher maternal IQ & vulnerable attachment style & joined programme
earlier associated with higher engagement, while smoking during
pregnancy associated with lower engagement
+ Contamination in RCTs

Contamination occurs when the control groups either actively or
passively receive the intervention intended for the treatment group
(Cook & Campbell, 1979)

Potential for contamination is high in PFL

Members of the treatment and control groups may be friends, neighbours,
colleagues, same family!

Community comparison group used to safeguard against
contamination

Included a range of direct (‘blue-dye’) in each survey
+ Contamination in PFL
 If contamination DID NOT take place:
• High & Low treatment group will differ in their responses
• Low treatment group will not differ from comparison group
• High treatment group will differ from the comparison group
Contamination Questions in 6 month PFL survey
Heard the phrase
Mutual Gaze
Circle of Security
MHIGH
(SD)
MLOW
(SD)
MLFP
(SD)
0.59
0.08
0.04
(0.49)
(0.27)
(0.19)
0.49
0.12
0.05
(0.50)
(0.33)
(0.21)
Low – High
p
High –
Comparison
p
Low –
Comparison
p
<0.01
<0.01
ns
<0.01
<0.01
ns

Little evidence of contamination among treatment groups

Evidence of absorption of PFL knowledge among high treatment group
+ Update on PFL

Oldest PFL child is almost 4 years and old & youngest is 13 months

18, 24, & 36 month surveys are currently in the field

12 month results available during Summer

Over 1,500 interviews conducted as part of impact evaluation

Conducted four school readiness surveys with junior infant cohorts in
2008, 2009, 2010, 2011

Conducted focus groups with low & high treatment groups & semistructured interviews with all PFL mentors

Evaluation will continue until all children start school
Programme website: www.preparingforlife.com
Evaluation website: http://geary.ucd.ie/preparingforlife/