PCORI Methodology Standards:
Academic Curriculum
© 2016 Patient-Centered Outcomes Research Institute. All Rights Reserved.
Module 5: Step 3—Search the
Literature
Category 11: Systematic
Reviews
Prepared by Eric Bass, MD, MPH
Tianjing Li, MD, PhD
Presented by Eric Bass, MD, MPH
Step 3: Search the Literature
 Select highest-yield electronic databases:
 General (e.g., PUBMED®, EMBASE®, Cochrane database of reviews and trials)
 Subject-specific databases (e.g., CINAHL, PsychINFO, OTSeeker)
 Consider other sources:
 Citation databases, such as Web of Science and Scopus
 Registries, such as ClinicalTrials.gov and WHO portal
 Regulatory agencies, such as Food and Drug Administration (FDA)
 Manufacturers
 Grey literature (dissertations, conferences), such as OpenGrey
 Use other methods:
 Citations in reviews and eligible articles
 Table of contents of relevant journals
 Query experts
3
Is It Sufficient to Use PUBMED and EMBASE?
Topic
Bicycle-related
brain injuries
Database
Articles
available
% of
total
Unique
articles
EMBASE
31
61
8
MEDLINE
33
65
3
PsychINFO
16
34
2
Web of Science
27
53
5
Source: Lawrence, D. W. (2008). What is lost when searching only one literature database for articles relevant to injury prevention and safety promotion? Injury
Prevention: Journal of the International Society for Child and Adolescent Injury Prevention, 14(6), 401–404. http://doi.org/10.1136/ip.2008.019430
4
Build a Boolean PubMed Search Strategy
 Start with:
 (Population OR synonym) AND
(intervention OR synonym) AND
(comparison OR synonym)
 Broaden or limit search with Boolean terms (AND, OR,
NOT)
 Consider limiting by study type
 Cochrane has a highly sensitive search strategy for
randomized controlled trials
5
Building a Search Strategy
 Use controlled vocabulary (medical subject headings) and key words
 If you use only controlled vocabulary, you miss articles because indexing is
imperfect
 If you use only key words, you miss articles that don’t use the same terms
 Use Clinical Queries to search by study category
 Etiology, diagnosis, therapy, prognosis, prediction
 Search for systematic reviews
 Keep a record of search strings and results using Advanced Search Builder and My NCBI
 See online tutorial at http://www.nlm.nih.gov/bsd/disted/pubmed.html
 Adapt search to other databases
 Document when, where, how, and what!
6
Criteria for Assessing the Search Strategy
 Accurate translation of question into search concepts
 Correct choice of Boolean operators and line numbers
 Adequate translation of strategy for each database
 Inclusion of relevant subject headings
 Absence of spelling errors
Source: Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009). An evidence-based practice guideline for the peer review of
electronic search strategies. Journal of Clinical Epidemiology, 62(9), 944–952. http://doi.org/10.1016/j.jclinepi.2008.10.012.
7
How Important Is It to Include Unpublished Results?
 Analysis of 42 systematic reviews of
nine new drugs, comparing results
with and without unpublished FDA
data:
 For one harm outcome, summary
estimate of harm was greater with
unpublished data included
Efficacy estimate
Lower
Higher
Source: Hart, B., Lundh, A., & Bero, L. (2012). Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses.
BMJ (Clinical Research Ed.), 344, d7202.
Identical
8
Review Citations
Titles (n=10,475)
Abstracts (n=3,612)
Full-text articles (n=449)
Eligible articles (n=63)
9
Potential Causes of Bias and Error in Searching the Literature
 Bias
 Publication bias
 Redundant reporting
 Exclusion of important sources
 Incomplete development of search strategy
 Error
 Inadequate monitoring of search process
 Imperfect reliability of reviewers
10
What Can Be Done to Minimize Bias and Error When
Reviewing Citations for Eligible Studies?
 Establish process for reviewing citations with quality control
 Use trained reviewers with documented instructions
 Use independent dual-reviewer process
 Use key articles to validate search
 Assess samples early
 Discuss discrepancies
 Watch for duplication
11
Relevant IOM Standards for Searching the Literature

Standard 2.6: Develop a systematic review protocol
 Describe context and rationale for review from decision-making and research perspective
 Describe study screening and selection criteria (inclusion and exclusion)
 Describe which outcomes, time points, interventions, and comparisons will be addressed
 Describe search strategy for identifying relevant evidence
 Describe procedures for study selection and data extraction, and how to identify and
resolve disagreement between researchers in study selection and data extraction
 Describe approach to critically appraising studies
 Describe method for evaluating body of evidence, including quantitative and qualitative
synthesis
 Describe and justify planned analyses of differential treatment effects according to
subgroups, how intervention is delivered, or how outcome measured
 Describe proposed timetable
12
Relevant IOM Standards for Searching the Literature
 Standard 3.1: Conduct a comprehensive systematic search for evidence
 Work with information specialist trained in systematic reviews to plan search
strategy
• PCORI acknowledges that such expertise is not limited to librarians
 Design the search strategy to address each question
 Use an independent information specialist to peer-review the search strategy
 Search bibliographic databases and citation indexes
 Search literature cited by eligible studies
 Update search at intervals appropriate to pace of generation of new information
 Search subject-specific databases and regional bibliographic databases if other
databases are unlikely to provide all relevant evidence
13
Relevant IOM Standards for Searching the Literature
 Standard 3.2: Take action to address potentially biased reporting of results
 Search grey literature databases, clinical trial registries, and other sources of
unpublished information about studies
 Invite researchers to clarify information about study eligibility, study
characteristics, and risk of bias
 Invite study sponsors and researchers to submit unpublished data, including
unreported outcomes
 Hand-search selected journals and conference abstracts
 Conduct a web search
 Search for studies reported in languages other than English, if appropriate
14
Relevant IOM Standards for Searching the Literature
 Standard 3.3: Screen and select studies
 Include or exclude studies on the basis of protocol’s prespecified criteria
 Use observational studies as well as randomized trials to assess harms of
interventions
 Use two or more members of the team, working independently, to screen and
select studies
 Train screeners using written documentation, and test and retest screeners to
improve accuracy and consistency
 Use one of two strategies to select studies:
• Read all full-text articles identified in the search
• Read full text of articles identified after screening titles and abstracts
 Taking account of risk of bias, consider using observational studies to address gaps
in evidence from randomized trials on the benefits of interventions
15
Relevant IOM Standards for Searching the Literature
 Standard 3.4: Document the search
 Provide a line-by-line description of the search strategy, including date of every
search for each database, web browser, etc.
 Document disposition of each report identified, including reasons for exclusion
16