The effectiveness of volunteer tutoring programmes
Dr Sarah Miller
Centre for Effective Education
Queen’s University Belfast
Campbell Collaboration Colloquium 2012
Copenhagen, Denmark
www.qub.ac.uk/cee
The review team
Dr Sarah Miller
Deputy Director, Centre for Effective Education, QUB
Professor Gary Ritter
College of Education and Health Professions, University of
Arkansas
Pauline Connolly Centre for Effective Education, QUB
Nicole Craig Centre for Effective Education, QUB
Naoimh Fox Centre for Effective Education, QUB
Overview
• Purpose of the current review
• Summary of the methods and findings from the original
Ritter et al 2006 review
• Rationale for updating and extending Ritter et al 2006
review
• Value of systematic reviews
• Potential limitations of reviews
The current review
• The current review is an update and extension of Gary
Ritter’s 2006 review of the effectiveness of volunteer
tutoring programmes
• Context of the Ritter et al 2006 review: limited and
conflicting nature of existing evidence
• Objective of the Ritter et al 2006 review: to summarise
the cumulative empirical evidence on the effectiveness
of volunteer tutoring programmes in improving
academic skills of elementary school children
Summary of methods (Ritter et al 2006)
Types of studies
Randomised field trials, standard care control, published after
1985, English language studies in the United States.
Specialised programmes were excluded.
Types of participants
Tutors: adult, non-professional tutors
Tutees: Kindergarten to Grade 8 (age 5 to 13/14 years old)
Types of interventions
Regular tutoring sessions with academic focus at least 1 month
in duration
Summary of methods (Ritter et al 2006)
Types of outcome measures
Standardised assessment of math and reading skills:
• Reading global
• Reading letters and words
• Reading comprehension
• Reading oral fluency
• Writing
• Mathematics global
Summary of methods (Ritter et al 2006)
Search results
• 969 unique study reports identified
• Application of the inclusion and exclusion criteria resulted in
28 study cohorts to be included in the review
Methodological quality
• Transparency of allocation procedures, integrity of random
assignment design, existence of high level of attrition,
existence of problems re programme fidelity
• No studies eliminated from
the analysis on this basis
Summary of findings (Ritter et al 2006)
Data synthesis
• Total n=1,676 study participants (873=intervention,
803=control)
• Reading: n=1,077 students, 24 studies (excluding 1 outlier
study)
• Maths: n=643, 5 studies
Types of tutors
• Primarily parents: n=338, 5 cohorts
• College age tutors: n=899, 12 cohorts
• Community volunteers: n=439,
11 cohorts
Summary of findings (Ritter et al 2006)
Types of tutees
• Grade 1 (age 6-7 years): n=770, 14 cohorts
• Grade 2 and above (age 7+ years): n=906, 14 cohorts
Programme structure
• Highly structured programmes: n=919, 15 cohorts
• Not highly structured: n=757, 13 cohorts
Source of publication
• Refereed journals: n=772, 15 cohorts
• Other sources: n=904, 13 cohorts
Summary of findings (Ritter et al 2006)
Outcome
N of
studies
N of tutored
students in
analysis
Effect size
95% CI
Reading - overall
24
550
0.30*
(.18, .42)
Reading - global
13
195
0.26*
(.05, .48)
Letters & words
15
403
0.41*
(.27, .56)
Comprehension
8
293
0.18
(-.6, .42)
Oral fluency
12
336
0.30*
(.14, .46)
Writing
6
111
0.45*
(.19, .71)
Mathematics
5
292
0.27
(-.18, .72)
Summary of findings (Ritter et al 2006)
• No significant difference in effect size by tutor group (i.e.
parent, college student, community volunteer)
• No significant difference in effect size by Grade (i.e.
programmes aimed at children in Grade 1 or Grade 2+)
• Highly structured programmes were significantly more
effective (ES=.59) than programmes that were not highly
structured (ES=.14)
• No evidence of publication bias
Rationale for updating and extending the review
• To include studies conducted outside the United States
• To include programmes that have employed both
volunteer and paid tutors to improve literacy and
numeracy outcomes
• To include studies that aim to improve literacy for
children who are learning English as an additional
language
• To include relevant studies that have been conducted
since 2006
Context for updating and extending the review
• PhD students designing evidence based programmes in
the area of literacy, numeracy and language. These are
being evaluated using an RCT design.
• We have recently conducted two large trials of a
volunteer tutoring programme (with children aged 8-9
years) and found small effect sizes related to reading:
Context for updating and extending the review
RCT 1 (n=734)
• Reading comprehension (ES -0.002, p=0.98)
RCT 2 (n=512)
• Decoding (ES +0.15, p=0.01)
• Reading rate (ES +0.22, p=0.01)
• Reading accuracy (ES =.07, p=0.28)
• Reading Fluency (ES +0.14, p=0.05)
• Comprehension (ES +0.05, p=0.55)
Value of systematic reviews
• Systematic reviews produce an unbiased account of the
cumulative state of evidence in relation to a research
question/hypothesis
• They provide information relating to the strengths and
weaknesses of the existing evidence and the rigour of
the methods used
• Determine the magnitude of the relationship under
investigation
• Locate and include all relevant research
Potential limitations
Included studies in reviews often contain small sample
sizes
• Only four trials in four major reviews in the area of tutoring
had a sample size greater than 250
• Trials with small sample sizes and large effect sizes are
potentially over represented in the literature (Slavin & Smith,
2009)
– Use of less robust methodologies
– Super-realisation
– Treatment inherent
measures
Potential limitations
Linking outcomes to theory - theory development should
not be neglected. In the case of volunteer tutoring:
– Chall’s stages of reading development
– Provides a useful framework for understanding children’s
transition through the developmental stages
– Successful transition at age 8-9 (to Stage 3 in Chall’s
theory) requires practice reading familiar materials
– Precisely the aim of many reading focused tutoring
programmes
Challenges associated with conducting a
systematic review
• Narrowing down the research question and defining
variables
• The time required to undertake a review and volume of
data to process and extract
• Studies not reporting sufficient data to use in meta
analysis and non response of authors