Session 1.
Introduction to Systematic
Reviews and Meta-Analysis
Michael A. Stoto, PhD
ICSA
June 4, 2008
© Michael A. Stoto
Plan for the day
1. Introduction to systematic reviews
and meta-analysis
– Identify/clarify the question
– Identify studies to be included
2. Analyze the data & draw conclusions
– Basic statistical methods
3. Dealing with heterogeneity
– Meta-regression
4. Meta-analysis for drug safety
1
Systematic Reviews
• State-of-the-art literature review
– Logical framework for systematic and objective
review of scientific results
– Basis for evidence-based medicine/practice
– Also known as research synthesis, quantitative
literature review, etc.
• Meta-analysis
– Subset of systematic review: Statistical analysis of a
large collection of analysis results from individual
studies for the purpose of integrating the findings
– Increasingly popular: views of health scientists range
from complete aversion to skepticism to enthusiasm
Meta-analyses recorded in
MEDLINE, 1995-2004
Number of meta-analyses
2000
1500
1000
500
0
1995 1996
1997 1998
1999 2000
2001 2002
2003 2004
Year
2
Proponents of systematic reviews
and meta-analysis
• Archie Cochrane
– Early advocated for RCTs; by 1979 suggested
that they be systematically summarized
– World-wide Cochrane Collaboration
• AHRQ (formerly AHCPR)
– PORTs, Evidence-based practice centers
• Now central to evidence-based practice
– US Preventive Services Task Force
– Community Preventive Services Task Force
Critics of systematic reviews and
meta-analysis
• Shapiro (1994)
– “Meta-analysis, shmeta-analysis”
• Bailar (1995)
– Frequent and serious problems
• including bias by meta-analyst
– “any attempt to reduce the results to a single value
with confidence bounds is likely to lead to
conclusions that are wrong, perhaps seriously so”
– Prefer standard narrative review
3
Proponents of systematic reviews
and meta-analysis
• Oxman and Guyatt (193)
– Typical narrative review relies on expert judgment, and
“experts … write reviews of inferior quality” possibly
“related to the strength of prior opinions”
• Chalmers (2005)
– “unscientific and unethical to embark on new research
without first analysing what can be learned from
existing research”
– Impossible to assess the contribution of a new study
without systematic up-to-date review
• Lancet now requires RCT authors to include a
summary of previous research findings
Purposes of systematic reviews
• Systematic reviews identify, analyze, and
present available data and offer the potential to
– identify areas of agreement
– clarify nature and causes of disagreement
– establish what is known prior to further
research
– identify areas needing more research
– frame results so they can be translated into
practice and policy
4
Purposes of meta-analysis
• Statistical methods combine results from
multiple studies to
–
–
–
–
increase precision of summary estimates
statistical power for average estimates
resolve issues relating to conflicting results
estimate relationships between effects and
study-level variables (meta-regression)
Prophylactic lidocaine post MI
Study
Chopra
Mogensen
Pitt
Darby
Bennett
O'Brian
Lidocaine
N # dead
39
2
44
4
107
6
103
7
110
7
154
11
p
0.051
0.091
0.056
0.068
0.064
0.071
Control
N # dead
43
1
44
4
110
4
100
5
106
3
146
4
p
0.023
0.091
0.036
0.050
0.028
0.027
d
0.028
0.000
0.020
0.018
0.035
0.044
s
0.042
0.061
0.029
0.033
0.028
0.0248
5
Forest plot of lidocaine data
Chopra
Mogensen
Pitt
Darby
Bennett
O'Brian
Combined
-.15
-.1
-.05
0
d
.05
.1
.15
Meta-analysis of lidocaine data
Chopra
Mogensen
Pitt
Darby
Bennett
O'Brian
Combined
-.15
-.1
-.05
0
d
.05
.1
.15
6
CHD and passive smoking
Cumulative meta-analysis:
MI and rofecoxib (Vioxx)
7
Guidelines for systematic reviews (1/3)
• Use a protocol to specify objectives,
hypotheses, scope, and review methods
• Compile a comprehensive set of relevant
primary studies
– Document search methods and sources
– Select based on a priori specifications
• Assess methodological quality of studies
• Identify common definitions for outcome,
explanatory, and confounding variables
Guidelines for systematic reviews (2/3)
• Standardize abstraction of outcome
measures and study and subject
characteristics
• Meta-analyze using appropriate methods
where warranted
– Narrative summary where data are too sparse,
heterogeneous, or of low quality
• Explore robustness to choices and
assumptions
– Impact of study quality/inclusion criteria
– Likelihood and possible impact of publication
bias
8
Guidelines for systematic reviews (3/3)
• Clearly present methods, assumptions, and
results to enable critical appraisal and
replication
• Appraise methodological limitations of
primary studies and systematic review
• Make practical, explicit, and evidencebased clinical or policy recommendations
• Propose future research including clinical
and methodological requirements
Reporting guidelines
• Quality of Reporting of Meta-analyses
(QUORUM) consensus statement
– Moher, et al., 1999, Lancet 354: pp. 1896-1900
• Meta-analysis of Observational Studies in
Epidemiology (MOOSE) consensus
statement
– Stroup, et al, 2000, JAMA 283: pp. 2008-2012
9
Plan for the day
1. Introduction to systematic reviews
and meta-analysis
– Identify/clarify the question
– Identify studies to be included
2. Analyze the data & draw conclusions
– Basic statistical methods
3. Dealing with heterogeneity
– Meta-regression
4. Meta-analysis for drug safety
Guiding principles
• In systematic reviews/meta-analysis,
individual study results are the raw data
• Systematic reviews are retrospective
research, and thus are potentially subject to
the same biases as other retrospective
studies
• Therefore, we need an a priori protocol for
data selection and analyses to ensure
replicability
10
Study purpose
(systematic review)
•
•
•
•
•
•
•
•
•
Summarize large and complex body of data
Resolve conflicting reports in the literature
Clarify strengths/weaknesses of studies
Document need for a clinical trial
Avoid time/expense of a clinical trial
Increase statistical power
Improve precision of treatment effect estimate
Investigate variation in treatment effects
Improve generalizeability of known effects
Clinical/public health question
•
•
•
•
•
Frequency/rate (burden of illness)
Etiology and risk factors
Diagnostic test performance
Effect of an intervention
Prediction and prognosis
11
Public health policies
• Regulation (EPA, OSHA, FDA, etc.)
– Allowable levels of contaminants in air,
ground, water, workplaces, etc.
• Infectious disease control (CDC, state and
local health departments)
– Immunization, contract tracing, etc.
• Education and public campaigns:
– Tobacco, obesity, AIDS prevention, etc.
• Compensation and other legal issues
Clinical policies
(Black N., Evidence based policy, BMJ, 2001)
• Practice policies
– address the use of resources by practitioners
• including the recommendations that practitioners
make about what patients should (should not) do
• Service policies
– address how resources are allocated to deliver
services to a population or a group of people
• Governance policies
– Address organizational structure and finance
12
Topic clarification
• Defining the question
–
–
–
–
target condition
patient population/clinical context
intervention/exposure
outcomes of interest
• Broad/narrow topic (Glasziou et al.)
– mortality reduction in colorectal cancer from
yearly fecal occult blood screening in 40-50year old women
– effect of cancer screening in the general
population
– mortality reduction in colorectal cancer from
fecal occult blood screening in adults
Evidence model/ analytic framework
• Clarifies the questions to be answered
• Avoids inappropriate focus on isolated
linkages in a causal path
• Makes explicit the outcomes for an
intervention to be judged effective
• Systematic review of potential benefits and
harms
13
Evidence model - immunization
Increasing
community
demand
Population
Environment
Enhancing
access to
vaccination
Attendance in
healthcare
systems
Providerbased
interventions
Vaccine
efficacy
Vaccination
coverage
Vaccinepreventable
disease
Treatment
of vaccine
preventable
disease
Morbidity
and
mortality
Exposure to
vaccinepreventable
disease
Reducing
exposure
USPSTF analytic framework
Harris, AJPM, 2001
14
Plan for the day
1. Introduction to systematic reviews
and meta-analysis
– Identify/clarify the question
– Identify studies to be included
2. Analyze the data & draw conclusions
– Basic statistical methods
3. Dealing with heterogeneity
– Meta-regression
4. Meta-analysis for drug safety
Identification of articles to be
included in the review
• Inclusion/exclusion criteria
– population, intervention, principal outcomes,
– study design, publication date, language
• Study selection is major source of disagreement
among meta-analyses
– Choice must be objective
• not based on desired outcome!
– Need an a priori
• search strategy
• set of inclusion and exclusion criteria
15
Admissible evidence
• Sources of published evidence
– experimental studies (randomized or not)
– observational/epidemiological studies
– case studies/series
– abstracts
– editorials/letters to the editor
– non-human studies (animal, in vitro)
– research syntheses (reviews, metaanalyses, decision analyses)
• Include proprietary/unpublished data?
Search methods
• Bibliographic sources
– MEDLINE
• Ovid, PUBMED, Grateful Med
– EMBASE
– Cochrane library
– Web of Science
– Google (Scholar)
• Search methods
–
–
–
–
MESH terms, exploding
keywords vs. full text
methodological filters
inclusion/exclusion criteria
16
Search methods
• Additional sources
–
–
–
–
–
existing reviews
cross-checking citations (“snowballing”)
“hand” search of key journals
surveying investigators in the field
searching registries of pertinent studies
• Search suggestions
– don’t rely exclusively on computerized searches
– avoid multiple publications from single studies
– go beyond abstracts
Searching results: Breast cancer
and alcohol use (Lemeshow, J Clin Epi, 2005)
#
Found
#
Relevant
Found
%
Sensitivity
95%
Confidence
Limits
#
Unique
Found
Biosis
642
63
79
(70, 88)
3
Dissertation
Abstracts Online
44
0
0
-
0
EMBASE
881
65
81
(73, 90)
2
ETOH
373
58
73
(63,82)
0
NIH CRISP
35
0
0
-
0
Database
NTIS
42
0
0
-
0
MEDLINE
537
66
83
(74, 91)
1
Pre-MEDLINE
21
2
3
(-1, 6)
1
1032
65
81
(73, 90)
2
SCI-EXPANDED,
SSCI
Meta-analyses and
reviews,hand search
TOTAL ARTICLES
3607
3
2
80
11
17
Reporting on search methods
in meta-analyses
• Report a priori protocol and changes
• Report inclusion/exclusion criteria
– selection bias is main reason for discrepant
results among meta analyses
– legitimate to include secondary comparisons
• List studies excluded and why
• Describe criteria for extracting data
– operational definitions for explanatory and
response variables
• Include measure of inter-extractor reliability
Study flow diagram
Bolen, S. et. al. Ann Intern Med 2007;0:0000605-200709180-00178-E-70
18
Data extraction forms
• Rationale: Data repository Æ analysis
– Visual link between formulated review question and
planned assessment of included studies
– Historical record of decisions (and changes) made
during the course of the study
• Include
– Verification of study eligibility
– All variables needed for review
• Paper vs. electronic forms
– Paper: simpler, easier for small reviews
– Electronic: complex to program, but
• Helps manage large reviews with many reviewers
• Facilitates double-abstraction process
Data extraction
• Be explicit, unbiased, reproducible
• Use multiple readers
• Include all relevant measures
– Including study quality
• Contact authors for clarification
• Evaluate reproducibility of all measures
• Data extraction errors
– Gøtzsche et al., JAMA 2007
– Serious errors (> 0.1 in standardized mean
differences in >1/2 trials) in 10/27 meta-analyses
19
Publication bias
• Small, negative studies less likely to be published
– “File drawer problem”: investigators don’t submit
– Negative secondary analyses
• Example: survival ratio for multiagent therapy for
ovarian cancer (Simes, 1987)
– Published studies (k=16) 1.16 (1.06-1.27)
– Registered studies (k=13) 1.05 (0.98-1.12)
• Steps to reduce/identify publication bias
– Search registries (if they exist)
– Write to scientists working in the field
– Funnel plots
– Statistical tests and adjustments
Funnel plot
(with “missing” studies)
150
Sample size
100
50
0
.4
1
2
Odds Ratio
4
10
20
Funnel plot
150
Sample size
100
50
0
1
.4
2
Odds Ratio
4
10
ETS and lung cancer
Funnel Plot of Precision by Log odds ratio
10
Precision (1/Std Err)
8
6
4
2
0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Log odds ratio
21
ETS and lung cancer
Funnel Plot of Precision by Log odds ratio
10
Precision (1/Std Err)
8
6
4
2
0
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Log odds ratio
Publication Bias and Language Bias
• RCTs are more likely to be published in an Englishlanguage journal if the results are statistically significant
(Egger 1997)
• Exclusion of non-English literature did not change
estimates of effectiveness (Moher 2000)
• Language restriction depends on topic area
• Exclusion of the “gray” literature can lead to exaggerated
estimates of intervention effectiveness (McAuley 2000)
• Gray literature includes unpublished studies and those
with limited distribution
• Most meta-analyses assess if publication bias is present but
the question remains how to proceed if bias is suspected
(Sutton 2000)
22
Evaluating quality (validity) of
individual studies
• Can be used as
– inclusion criteria
– explanation for heterogeneity
– sensitivity analysis
• Quality scores
– transparent and parsimonious
– don’t confuse with quality of reporting
– reports scoring methods
Quality issues
• Intervention studies
– Study types: RCT, cohort, case-control
– Issues:
• randomization
• completeness of follow-up
• blinding of patients and clinicians
• Etiology/risk factor studies
– Study types: cohort, case-control
– Issues:
• groups differ in exposure, adequate “adjustment”
• outcomes measurement
• reasonable evidence of causation
23
Jadad Quality Score for an RCT
(i) Was the study described as randomized?
(ii) Was the study described as double-blind?
(iii) Was there a description of withdrawals and drop
outs?
• Give a score of 1 for each “yes”
• Give additional point each if randomization and
blinding are appropriate
• Deduct a point each if randomization or blinding are
inappropriate
• Range: 0-5; > 3 = “high quality”
Jadad AR, et al. Assessing the quality of reports of randomized clinical
trials: is blinding necessary? Control Clin Trials 1996; 17:1-12.
Summary: Bias in meta-analysis
• Decisions about studies to include are major
source of disagreement among meta-analyses
– Choice must be objective
• not based on desired outcome!
– Need a priori
• search strategy
• set of inclusion and exclusion criteria
• Bias (poor quality) in included studies
– Allocation of subjects, follow-up, assessment of
results, etc. should not be related to the treatment,
exposure or outcome under study
24
BCG vaccine background
• TB cases increasing since 1986
– HIV associated
– multiple drug resistant TB
• BCG vaccine used widely since 1921
– >3 billion doses administered
– not recommended for general use in U.S.
• efficacy ranged from 2% to 90% in 1987
review
– Madras trial
What is the policy question?
• Is BCG effective against TB?
– Is BCG effective in infants?
• How effective is BCG against TB?
– How effective is BCG in infants?
• In which groups is BCG effective?
• Which strains of BCG are most effective?
• Is BCG harmful?
• Should BCG be recommended?
– For whom?
25
Literature searching
• MEDLINE searches
– BCG vaccine, tuberculosis, human
•
•
•
•
•
Scanning reference lists
Previous reviews
Contacting experts at CDC and WHO
All languages, translated if necessary
Multiple publications reviewed for
most complete and up to date results
Inclusion/exclusion criteria
• Include
– efficacy of BCG in preventing TB cases/death
– random, concurrent comparison groups
– equivalent surveillance, follow-up
• Exclude
– prevalence studies, control programs, tuberculin
reactions, review articles
• Search results
– 1264 titles/abstracts examined
– 70 reviewed in detail
– 26 included in analysis
26
Begg's funnel plot with pseudo 95% confidence limits
13 BCG vaccine trials
1
logrr
0
-1
-2
0
.2
.4
s.e. of: logrr
.6
.8
Data extraction
• Items extracted
–
–
–
–
–
–
–
–
year of publication, program began
years of follow-up
study design (randomization details), n
age range of study population
location (geographical latitude)
strain and dose of BCG used
outcomes measured
vaccine efficacy
• Two readers with adjudication by a third
27
Validity (quality) scoring
• Vaccine trials
–
–
–
–
–
method of vaccine assignment
availability for follow-up
equality of surveillance
TB diagnosis criteria
preparation of BCG vaccine
• Case-control studies
– selection bias
– TB diagnosis criteria
• Assessed prior to analysis of results
Take Away Points
• Systematic Reviews
– Logical framework for systematic and objective
review of scientific results
• Meta-analysis
– Statistical analysis of a collection of results from
individual studies to integrate the findings
• Guiding principles
– Individual study results are the raw data
– Retrospective research, thus potentially subject to
same biases as other retrospective studies
– Need a priori protocol for data selection and analyses
to ensure replicability
28