Building Evidence in Education:
Conference for EEF Evaluators
11th July: Theory
12th July: Practice
www.educationendowmentfoundation.org.uk
Panel session 4: Analyse this
Testing younger pupils
Amy Skipp
[email protected]
Pupil testing in the Children and Young
people team
ARK Maths Mastery (40 schools of year 2
pupils)
- IoE evaluating
- ‘Singapore method’ of maths tuition
- Pre and post test of maths, with
waiting
control
Creative Futures (19 schools of 900 year 2
pupils)
- NatCen evaluating
-3 arm RCT within classes of Sing, Play,
Act
Challenges of testing
• Getting good data out of 6 year olds
• Measuring the correct outcomes for the intervention
• Need to minimise burden on schools – time, resource and ‘extras’
• SEN recording
Choosing a test
• Age and ability appropriate – suitable entry level but capturing top end
• Simple and quick to administer (no specialist knowledge)
• Group administration
• Standardised outcomes
• Paper based or PC/online
Creative Futures = PIPS
ARK Maths = Number Knowledge Test
Lessons learnt
Children vs testing
• Difficult to get a quiet private space in primary schools
• Need to factor in toilet breaks
• Children like to copy and see how you’re scoring them
• Little experiencing of ‘being tested’ at this age
• Time taken to get to test
Helping pupils give their best
• Use of appropriate words
• Group by matched ability
• Language support workers
Issues with schools
• Block bookings
• Changes of staff / school location / IT
• Contacting correct staff / getting past the
receptionist
- Web pages
- Over recruitment
- Value of intervention and EEF
Testing children – our
specialism
• Group of experienced ‘testers’
• All full DBS clearance
• Many former teachers
• Familiarity with tests
• Enthusiastic about new interventions
Panel session 4: Analyse this
Trials and tribulations – evaluation of
intervention programmes
Beng Huat See, Stephen Gorard and Nadia Siddiqui
Durham University
Randomising
within
clusters
Implications for
analysis
July 2013
Some definitions
Types of randomisation (see CONSORT):
• Simple
• Restricted
• Stratified
• Blocked
• Paired
Restricted randomisation in EEF
transitions trials
All pupil randomised:
• Chatterbooks (block=school)
• Rhythm for Reading (block=school; not pair!)
• Speaking and Listening (block=timetable
group)
• Vocabulary Enrichment (block=timetable
group or national curriculum level)
Are we justified in restricting?
• To improve balance of important covariates
No – can adjust for
covariates in analysis
• For practical reasons
Yes – teachers need to know
numbers for planning purposes
Fig 1 Correlation in mean survival time between treatment groups under simple and stratified
randomisation (simulated data).
Kahan B C , and Morris T P BMJ 2012;345:bmj.e5840
©2012 by British Medical Journal Publishing Group
What’s the problem?
• Introduces correlation between treatment
groups
• Violates statistical assumption of
independence
• P-values too large and confidence intervals
too wide
• More likely to miss a genuine effect
How big is the problem?
• Only 26% of a recent sample of medical trials
made adequate adjustments (Kahan and
Morris, 2012)
• Of course it is difficult to work out the extent
of the type I error since making and not
making adequate adjustments was NOT
RANDOMISED
How do we solve it?
• We need to include the stratification variable
as a covariate in the analysis
• ANCOVA with dummy variables to identify
school
• Multi-level model
• And this is why pairing before randomisation
is a BAD IDEA
One last thing
Is adjustment necessary for straightforward
blocking during rolling randomisation?
To remember!
If you have restricted your randomisation using
a factor that is associated with the outcome
(e.g. school) THEN
INCLUDE THE FACTOR AS A COVARIATE IN
YOUR ANALYSIS