Labour Market Knowledge and
A-level Choices:
A Blinded Cluster Randomized Controlled Trial with Linked
Administrative Data
Neil Davies, Peter Davies, Tian Qiu
Randomised Controlled Trials in the Social Sciences
Eleventh Annual Conference
09/09/16
1
Policy context
• English school students specialise at an early age
• Students chose A-level subjects age 15/16
• These subject affect university subjects
• Degree subject is associated with labour market outcomes,
such as wages
• Research hypothesis
• Does providing labour market information to students affect
their choices and beliefs?
2
3
4
Background
5
Background
6
Methods
• Cluster randomised controlled trial
• Intervention compared to control lesson
• 50 schools (20 private and 30 state)
• 5,593 students
• Two surveys
• Baseline
• Follow-up
• Linkage to National Pupil Database
• Actual A-level choices returned by schools
• Primary outcome: Actual A-level subject choices
7
Timeline of experiment
Week 0
50 eligible schools
randomly sampled
Week 2
Baseline
survey
Schools randomly
allocated to
intervention or
control arm
Intervention or
control lesson
Week 4
Follow-up
survey
Week 6
Actual Alevel choices
reported
Linkage to
National Pupil
Database
8
Baseline survey
• Questions
• Socioeconomic demographics
• Expected GCSE grades
• Prior intentions to study each subject at A-level (Likert scale)
• Measures of cultural capital (number of books)
• Intentions and motivations towards university study (1 to 5)
• Expected effect of attending university on salary (seven point scale)
• Expected effect of each degree on salary (seven point scale)
9
Intervention and control lessons
• Structured lessons
• Intervention lesson
• Information on the graduate premium for 10 subjects:
Business, Education, Engineering, History, Languages, Law,
Mathematics, Politics, Psychology, and Science
• Control lesson
• Based on advice from public available websites
• Advice from Russell group of universities
• Subject difficultly
• No information on relative wages
10
Average Graduate Salaries O’Leary and Sloane (2011)
Science
Psychology
Politics
Maths or Computing
Law
Languages
History
Engineering
Education
Business or Financial
Art
£0
£5,000
£10,000
Male average graduate Salary
£15,000
£20,000
£25,000
£30,000
£35,000
£40,000
Female average graduate salary
11
Statistical methods 1
• Sample size calculation
• Gave 80% power to detect a 0.3 percentage point difference in outcomes
at 𝛼 = 0.05, with an ICC of 0.1 83 A-level students per year and 48
schools.
• Block randomization by external statistician
• Stratified by three variables:
• State or private school
• Single or mixed sex school
• Average pupil achievement above or below the median
• Imputation of missing data
• Permutation calculation to correct p-values for multiple testing
• Protocol registration: AEARCTR-0000468
• Following CONSORT guidelines
12
Statistical Methods 2
• Sampling
• Age range 13-18.
• Have at least 100 students in their ‘sixth forms’ (students in the
academic year between 16 and 18).
• Stratified sample to include 20 private schools and 30 state schools.
• Sampled schools were randomly sampled from list of eligible schools.
• Blinding
• Participating schools blinded to their allocation.
• Deviations from study protocol
• Six schools (571 students) allocated to the treatment arm did not take
part in the intervention.
• Four schools (487 students) allocated to the control arm did not
participate in the second round of questions.
13
Statistical methods 3
All analysis conducted using intention-to-treat.
Primary outcome
• Logistic regression of intervention on subject choice
• Raw and adjusted for baseline variables (including prior intentions)
Secondary outcomes
• Earnings expectation (average and own graduate salaries)
• Multivariate logistic regression of intervention on intention
• Raw and adjusted for baseline variables (including prior intentions)
• All standard errors and statistical tests allow for clustering by
school, permutation p-values also reported in paper.
14
Motivations for attending university
5.0
4.5
Importance ranking 1=high, 6=low
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Higher salary
Job status
Expert knowledge
Females
Enjoyment
Friendship
Cultural awareness
Males
15
Motivation for choosing subject
5.0
4.5
Importance ranking 1=high, 6=low
4.0
3.5
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Higher salary
Job status
Technical knowledge
Females
Creativity
Contribution to
society/environment
Opportunity to care for
others
Males
16
Balance of baseline covariates
Salary very important for choice of subject
Eligible for free school meals
Graduate mother
Graduate father
Student aspires for professional job
Mother professional or managerial
Father professional or managerial
White
English GCSE grade*
Maths GCSE grade*
Male
0
Control
20
40
60
80
100
Intervention
17
Proportion of students taking each
subject in intervention and control arms
60%
50%
40%
30%
20%
10%
0%
Intervention
Control
18
Effects of the intervention on subject
choices: unadjusted
Odds
Subject
ratio (95% CI)
Biology
0.96 (0.62, 1.48)
Business
0.89 (0.54, 1.44)
Chemistry
1.36 (0.89, 2.09)
Computing
0.68 (0.39, 1.19)
Economics
1.06 (0.61, 1.84)
English
1.07 (0.85, 1.34)
Geography
0.89 (0.60, 1.33)
History
1.06 (0.75, 1.50)
Languages
0.97 (0.58, 1.63)
Maths
1.42 (0.94, 2.14)
Physics
1.19 (0.77, 1.84)
Psychology
0.98 (0.63, 1.51)
.5
Less likely to take
1
2
More likely to take
19
Effects of the intervention on subject
choices: adjusted
Odds
Subject
ratio (95% CI)
Biology
0.73 (0.54, 1.00)
Business
1.00 (0.69, 1.45)
Chemistry
1.08 (0.82, 1.43)
Computing
0.61 (0.38, 0.99)
Economics
1.05 (0.73, 1.52)
English
1.18 (0.94, 1.48)
Geography
0.88 (0.66, 1.19)
History
1.09 (0.83, 1.43)
Languages
0.95 (0.68, 1.33)
Maths
1.39 (1.06, 1.82)
Physics
0.98 (0.74, 1.29)
Psychology
0.96 (0.66, 1.39)
.5
Less likely to take
1
2
More likely to take
20
5
Effects of the intervention on income
expectations.
4
Change in expectations (£k)
3
2
1
0
-1
-2
-3
-4
-5
Art
Business or
Financial
Education
Engineering
History
Languages
Average salaries
Law
Maths or
Computing
Medicine
Physics
Politics
Own salary
21
Strengths and limitations
• Strengths
• Cluster randomized
• Administrative data on educational attainment, and data on hard
choices
• Large sample
• Active lesson and engagement rather than passive online website
• Limitations
•
•
•
•
•
Multiple testing
Power
Missing data
Deviations from study protocol
Representativeness
22
Conclusions and future work
• Students’ educational choices were affected by the intervention
• Potentially cheap intervention
• Some evidence that students beliefs are not accurate
• Little evidence of interactions by school type or gender
• Benefit of linkage for randomised trials in social science
• Replication needed - Ioannidis (2014)
• Future – linked NPD and HESA data
• Compare with randomly non-sampled schools “untreated
control”
23
24
References
• Walker, I. & Zhu, Y. (2011). Differences by degree: Evidence of the net financial return to undergraduate study for
England and Wales, Economics of Education Review, 30, pp. 1177-1186.
• O’Leary NC, Sloane PJ. The wage premium for university education in Great Britain during a decade of change: wage premium
for university education. The Manchester School 2011;79:740–64. doi:10.1111/j.1467-9957.2010.02189.x
• Britton J, Dearden L, Shephard N, et al. How English domiciled graduate earnings vary with gender, institution
attended, subject and socio-economic background.
2016.http://www.ifs.org.uk/uploads/publications/wps/wp201606.pdf
• McGuigan, M. McNally, S. & Wyness, G. (2012). Student Awareness of Costs and Benefits of Educational Decisions:
Effects of an Information Campaign. Centre for Economics of Education Working Paper CEE DP 139. (London,
London School of Economics).
• Manski, C. (2004). Measuring expectations, Econometrica, 72, 5, pp. 1329-1376.
• Manski (1993) Adolescent econometricians: How do youth infer the returns to schooling? in Studies of supply and
demand in higher education (University of Chicago Press, 1993; http://www.nber.org/chapters/c6097.pdf), pp. 43–60.
• Jensen, R. (2010). The (Perceived) returns to education and the demand for schooling. Quarterly Journal of
Economics, 125, No.2:515-548.
• Jerrim. J. (2011). Do UK Higher Education Students Overestimate their starting salary? Fiscal Studies, 32, 4, pp.
483-509.
• Wood A.M., White, I. R. & Thompson, S. G. (2004). Are missing outcome data adequately handled? A review of
published randomized controlled trials in major medical journals. Clinical Trials. 1(4):368–76.
• Ioannidis J.P.A. (2014). How to Make More Published Research True. PLoS Medicine. 21;11(10):e1001747.
25