Systematic Reviews
and evidence based syntheses of research
Pete Smith
Department of Languages
February 2013
Which of these would you
trust to answer your question?
•
•
•
•
Best practice is …
The latest research shows that …..
A large scale study indicates …
A high quality randomised trial by Prof
Blogs supports …
• Experts think that …
• The overall body of research evidence
indicates that …
Norris & Ortega
Effectiveness of L2 Instruction: A
Research Synthesis and
Quantitative Meta-analysis
John Norris, Lourdes Ortega, Language
Learning, Volume 50, Issue 3, pages 417–528,
September 2000
Norris & Ortega
This study employed (and reports in detail) systematic procedures for research
synthesis and meta-analysis to summarize findings from experimental and
quasi-experimental investigations into the effectiveness of L2 instruction
published between 1980 and 1998. Comparisons of average effect sizes from
49 unique sample studies reporting sufficient data indicated that focused L2
instruction results in large target-oriented gains, that explicit types of instruction
are more effective than implicit types, and that Focus on Form and Focus on
Forms interventions result in equivalent and large effects. Further findings
suggest that the effectiveness of L2 instruction is durable and that the type of
outcome measures used in individual studies likely affects the magnitude of
observed instructional effectiveness. Generalizability of findings is limited
because the L2 type-of-instruction domain has yet to engage in rigorous
empirical operationalization and replication of its central research constructs.
Changes in research practices are recommended to enhance the future
accumulation of knowledge about the effectiveness of L2 instruction.
Doing more harm than good?
•
•
•
•
•
20 years of guess work in ELT
Cochrane Collaboration in 1996
Evidence Based Medicine
5000+ reviews now
Evidence based decision making for
social policy – NICE, EPPI, Campbell,
Cabinet Office Policy Hub
• Evidence based tokenism & objections
The Cochrane Library
• http://www.cochrane.org/cochranereviews/top
The Cochrane Collaboration
The Cochrane Library
• Is aspirin a good treatment for
migraines??
• http://www.dallasnews.com/health/headlin
es/20110620-getting-help-for-migrainesand-why-aspirin-isnt-the-answer.ece
• http://www.thecochranelibrary.com/view/0/i
ndex.html
Does praying for someone
ill make them better ?
• How would you find out?
• Would a good randomised
controlled trial answer the
question?
• Would a peer-refereed article by an
expert in a high-impact, renowned
journal provide a reliable answer?
Why a systematic review
might answer the Question
• Sums up the best available research on a
specific question.
• Synthesizes the results of several studies.
• Transparent procedures to find, evaluate
and synthesize the results of relevant
research.
• Studies included in a review are screened
for quality
Cochrane reviews
The Cochrane Handbook outlines eight general steps for
preparing a systematic review:
• Defining the review question and developing criteria for
including studies
• Searching for studies
• Selecting studies and collecting data
• Assessing risk of bias in included studies
• Analysing data and undertaking meta-analyses
• Addressing reporting biases
• Presenting results and "summary of findings" tables
• Interpreting results and drawing conclusions
Campbell reviews
•
•
•
•
•
•
•
•
•
Clear inclusion & exclusion criteria
An explicit search strategy
Systematic coding and analysis of included studies
Meta-analysis (where possible)
a systematic search for unpublished reports (to avoid
publication bias).
international scope.
A protocol (project plan) for the review is developed in advance
and undergoes peer review.
Study inclusion and coding decisions are accomplished by at
least two reviewers who work independently and compare
results.
peer review and editorial review.
EPPI
• Evidence-informed policy and practice - basing policy and practice
on sound evidence
• Interest in systematic reviews and evidence-informed education and
health promotion is part of a general move in the UK and elsewhere
towards basing policy and professional practice on sound evidence.
Systems, such as the Cochrane Collaboration and the Campbell
Collaboration, have been set up to help professionals, policy makers
and users base their decisions on up-to-date and reliable evidence
by making the results of systematic reviews accessible. The two
collaborations develop systems and methods in health care
interventions (Cochrane Collaboration) and education and social
care, employment and crime and justice (Campbell Collaboration).
• http://eppi.ioe.ac.uk/cms/
Controlling for bias
•
•
•
•
•
•
Transparency & replication (protocol)
Literature searches (publication bias)
Types of studies (selection bias)
Sample selection (robustness)
Meta-analysis where appropriate
Limitations
Strategy training
• In 2004 we had a question
• Does Strategy Training in
language learning work?
• http://eppi.ioe.ac.uk/cms/
Components
• Protocol – prior to looking at
data
• Review – in depth
• Update – evolution of body of
evidence & dialogue
Protocol
•
•
•
•
•
•
•
Title
Question
Background
Objectives
Initial scope of the review
Criteria for considering studies for this review
Types of studies
– Mapping
– In-depth review
•
•
•
•
•
Types of participants
Types of interventions
Types of outcome measures
Search strategy for identification of studies
Methods of the review
Review
BACKGROUND
Aims and rationale for current review
Definitional and conceptual issues
Policy and practice background
Research background
Authors, funders and other users of
the review
Review questions
IDENTIFYING AND DESCRIBING
STUDIES:
Studies included from searching and
screening
Characteristics of the included studies
Identifying and describing studies:
quality-assurance results
IN-DEPTH REVIEW: RESULTS
METHODS USED IN THE REVIEW
User-involvement
Identifying and describing studies
In-depth review
Selecting studies for the in-depth review
Nature of involvement of users in the
review and its impact
(meta analysis or other synthesis)
FINDINGS AND IMPLICATIONS
Strengths and limitations of this
systematic review
Implications
User perspectives
Your review…..
• What’s your question?
• Outline the protocol for a
systematic review