Issues and Challenges around
Searching the Literature
Dr Andrew Booth
Overview
• What is Already Known on Searching the
Qualitative Research Literature
• Overview of Methodological Issues/Challenges
• Recent Developments (with focus on literature
of last two years)
• Outstanding Issues/Challenges
What is already known
What is Already Known? – 1
•
Finding relevant qualitative studies arduous due to inadequate indexing (Dixon
Woods et al, 2007; Ring et al 2011)
•
Papers often lack abstracts or include non-informative titles, making it difficult to
establish relevance (Dixon Woods et al, 2007).
Several papers outline strategies (filters) for searching for qualitative
studies (Walters et al, 2006; Wilczynski et al, 2007; Wong et al, 2004; McKibbon et al, 2006)
Helpful techniques involve electronic or hand searching (Greenhalgh et al 2005;
Bates, 1989):
•
•
–
–
–
–
–
–
•
•
Reference or footnote tracking (looking backwards at references in articles found).
Citation tracking (tracking forward subsequently citing articles).
Personal knowledge and personal contacts.
Contacting the authors of known papers or experts in the field.
Hand searching relevant journals.
Internet browsing such as berry picking (where one search leads to another and ‘clusters’ of
papers are often found together).
Greenhalgh et al (2005) found only 30% of primary sources from predefined
search strategy. 51% found by other predefined methods (i.e. reference,
footnote and citation tracking).
However this may reflect a priori decisions to concentrate on non-electronic
sources
What is Already Known? - 2
• “Literature searches…open-ended iterative processes
where the topic or research question of interest is
honed over time as the nature of the evidence
becomes more apparent” (Finfgeld-Connett & Johnson
2013)
• Number of articles not only critical factor – reports
“may lack enough thick description to fully develop
concepts and the interrelationships among them”
(Finfgeld-Connett & Johnson 2013)
• Unpublished studies may contain rich, thick description
• Goal may not be aggregative – theoretical saturation
may play a part – selection of sample is crucial
References for What is Known Already - 1
• Bates MJ: The design of browsing and berrypicking techniques for
on-line search interface. Online Rev 1989, 13:407-424.
• Dixon-Woods M, Bonas S, Booth A et al.: How can systematic
reviews incorporate qualitative research? A critical perspective.
Qual Res 2006, 6:27-44.
• Finfgeld‐Connett, D., & Johnson, E. D. (2013). Literature search
strategies for conducting knowledge‐building and theory‐generating
qualitative systematic reviews. Journal of Advanced Nursing, 69(1),
194-204.
• Greenhalgh T, Peacock R: Effectiveness and efficiency of search
methods in systematic reviews of complex evidence: audit of
primary sources. BMJ 2005, 331:1064-1065.
• Ring N, Jepson R, Ritchie K: Methods of synthesising qualitative
research for health technology assessment. Int J Technol Assess
Health Care 2011, 27:384-390
References for What is Known
Already - 2
• Walters LA, Wilczynski NL, Haynes RB: Developing optimal search
strategies for retrieving clinically relevant qualitative studies in
EMBASE. Qual Health Res 2006, 16:162168. PubMed Abstract | Publisher Full Text
• Wilczynski NL, Marks S, Haynes RB: Search strategies for identifying
qualitative studies in CINAHL. Qual Health Res 2007, 17:705710. PubMed Abstract | Publisher Full Text
• Wong SL, Wilczynski NL, Haynes RB: Developing optimal search
strategies for detecting clinically relevant qualitative studies in
MEDLINE. In Medinfo 2004: Proceedings of the 11th World Congress on
Medical Informatics; San Francisco. Edited by Fieschi M, Coiera E, Jack
Li YC. Amsterdam: IOS Press; 2004:311-314.
• McKibbon KA, Wilczynski NL, Haynes RB: Developing optimal search
strategies for retrieving qualitative studies in PsycINFO. Eval Health
Prof 2006, 29:440-454. PubMed Abstract | Publisher Full Text
Overview of Methodological
Issues/Challenges
• Bias towards/Predominance of quantitative
research and publication of resultant reports
• Non-optimal indexing of qualitative studies
(CINAHL more evolved than MEDLINE)
• Qualitative research represents various research
methodologies, including ethnography,
phenomenology, grounded theory and narrative
analysis, which may hinder retrieval
• Lack of informative manuscript titles and
abstracts
Overcoming barriers:
• Searches should be as transparent as possible
without jeopardizing the creativity and
complexity of the process (Finfgeld-Connett &
Johnson 2013)
• Key “test” for knowledge-building/theory
generating review – would small amounts of
conflicting information substantially change the
findings? [Theoretical Saturation/ Qualitative
Sensitivity Analysis]
• Weaknesses in indexing mean that sensitivity of
searches may need to be reduced to allow time
for other search strategies (Pearson et al, 2011)
Simple search strategies vs complex
ones
• Three broad-based terms (i.e. “qualitative”, “findings”
and “interviews”) as effective as more complex search
strategies in identifying relevant qualitative research
reports (Flemming & Briggs 2007).
• Search strategies with broad search terms (“qualitative
research” or “qualitative studies” or “interviews”)
[combined with CAM terms] had highest recall and
precision (Franzel, 2013).
• Within time-limited context, protocol-driven, targeted,
and reference-checking search strategies most effective
(Pearson et al, 2011)
Intervention Searching vs Condition
Searching (Lorenc et al, 2012)
• Tying search terms of SR of qualitative
evidence too closely to interventions may
compromise consistency of the review.
• Dilemma: Performing condition-wide searches
(with no other change to strategies) would
become highly over-inclusive and volumes of
records impracticably large. (Suggests need for
alternative sampling stategies)
Is More Necessarily Better?
• For knowledge-building and theory-generating
systematic reviews, “more is better only when it
helps to fully explicate a concept, substantiate an
interconnection between or among concepts, or
build a line of argument….collecting more of the
same may merely escalate the cost of a study,
clutter the database and obfuscate important
inferences. Concepts and interrelationships
among them can only be more fully explicated
based on data that adds depth, breadth,
meaning and understanding to a phenomenon”
(Finfgeld-Connett & Johnson 2013).
Searching for religion and mental health
studies required health, social science, and grey
literature databases (Wright et al, 2014)
• PsycINFO = best performing database
• ArabPsyNet, CINAHL, Dissertations and Theses, EMBASE,
Global Health, Health Management Information
Consortium, MEDLINE, PsycINFO, and Sociological Abstracts
essential to retrieve included references.
• Citation tracking and personal library of one of the research
teams contributed unique, relevant references.
• Religion studies databases (Am Theo Lib Assoc, FRANCIS)
did not provide unique, relevant references.
• Literature searches for reviews/evidence syntheses of
religion and health studies should include social science,
grey literature, non-Western databases, personal libraries,
and citation tracking activities.
Exhaustive versus Expansive
That is the Question!
Exhaustive versus Expansive (FinfgeldConnett & Johnson 2013)
• Exhaustive searches when conducting
summative and aggregative systematic
reviews
• Expansive searches when conducting
knowledge-building and theory-generating
systematic reviews
Exhaustive versus Expansive
•
•
•
•
Multiple databases
Comprehensive list of terms
Assumption of homogeneity
Other approaches are
“supplementary”
“Combing the Area”
• Databases to reflect
contributing disciplines
• Terminology may
characterise “narrative” of
each discipline
• Likelihood of divergent
cases
• Sibling studies (shared
context)
• Cluster searching
• Citation searching
• “Supplementary”
techniques may be more
important
“Following Up Leads”
Recent Developments
• Importance of Context/Theory
• Database Coverage
• Importance of Supplementary Search
Methods
• Appropriate Selection of Sampling
Methods
• Reporting Standards (e.g. ENTREQ)
Searching for Contextual Richness
Requires identification of related (sibling)
reports i.e. cluster searching (cluster becomes
unit of analysis, not study) (Booth et al, 2013)
Cluster searching for “Siblings” (Booth
et al, 2013)
Element
Procedural Steps
Citations
Backwards reference chaining
Lead Authors
Author searching; Backwards reference chaining
Unpublished materials
Web searches; repositories
Scholar searches
Citations
Theories
Backwards reference chaining
Early Examples
Cited works (Forward reference chaining)
Related Projects
Co-citations
Searching for Theories
• Theory not typically reported in Abstracts
• Theoretical base differs by discipline cp. HSR vs
Public Health vs Nursing vs Psychology vs Sociology
• Reporting of Theory differs by discipline
• Level of Theory may vary e.g. Individual versus
Society (Psychology vs Sociology)
• When is a “theory” a Theory? – labelling (model,
framework, concepts ) and naming (“Health Belief
Model”)
Database Coverage
• CAM search - PubMed yielded 87% of relevant
included qualitative studies (Franzel et al, 2013).
• Five different QES PubMed coverage values 35/44
(79.5%); 9/10 (90%); 10/11 (91%); 9/9 (100%);
7/28 (25% - Grey literature) (Booth 2012
[Unpublished])
• But 5/28 of studies located only through
supplementary searches of three sources
(Stansfield et al, 2012). 21 search sources
required to locate all studies. [Explanation - Role
of Grey Literature and Geographical focus – UK
only]
Where to find qualitative research?
• 61% of [qualitative] research studies published within
journals. 29% were published as research reports and 5%
were published in books. Remainder were theses,
conference papers and raw datasets.
• 66% of studies (66%) located across 19 bibliographic
databases. 15 databases provided studies that were not
identified elsewhere.
• PubMed a good source for all reviews.
• Supplementary information sources important for
identifying studies in all publication formats.
• Undertaking sensitive searches across range of
information sources is essential for locating views studies
in all publication formats. (Stansfield et al, 2014)
Supplementary Strategies
More diffuse topic, move beyond
electronic searching
Supplementary Strategies - 1
• Multiple search strategy more likely to identify
relevant QR than sole reliance on electronic
searching.
• Purpose of synthesis determines appropriate
sampling/search strategy. E.g. mapping out key
conceptual developments – if aim not aggregative,
omission of papers unlikely to have dramatic effect
on results.
• Difficult to maintain sufficient familiarity with > 40
papers (Campbell et al, 2011)
Supplementary Strategies - 2
• Need ‘belt and braces’ (hand-searching; consultation
with experts)
• Searching for books/theses particularly challenging
(not indexed in same way as journal papers) (Campbell
et al, 2011)
• Snowballing and consultation with experts for a realist
review (Pawson et al, 2004)
• Obtaining authors' suggestions - resource-intensive
process with negligible results (Pearson et al, 2011)
• Additional search techniques essential to locate
further high quality references (Papaioannou et al,
2010)
Sampling: Appropriate ≠ Comprehensive (Suri,
2011)
• 16 strategies for sampling in QES
• E.g. Snowball sampling - seeking information from key
informants about other ‘information-rich cases’.
– ‘Chain of recommended informants would typically diverge
initially as many possible sources are recommended, then
converge as a few key names get mentioned over and
over’ (Patton, 2002, p. 237).
• Identify most cited primary research reports by ‘footnote
chasing’ (searching citation indices, browsing through
bibliographies, previous research syntheses, primary research
reports, policy documents, papers written by practitioners
and papers written for practitioners).
ENTREQ & Searching (Tong et al, 2012)
3
4
5
6
Approach to
searching
Indicate whether search was pre-planned (comprehensive
search strategies to seek all available studies) or iterative (to
seek all available concepts until they theoretical saturation is
achieved).
Inclusion criteria
Specify inclusion/exclusion criteria (e.g. in terms of
population, language, year limits, type of publication, study
type).
Data sources
Describe information sources used (e.g.electronic
databases (MEDLINE, EMBASE, CINAHL, psycINFO,
Econlit), grey literature databases (digital thesis, policy
reports), relevant organisational websites, experts,
information specialists, generic web searches (Google
Scholar) hand searching, reference lists) and when searches
conducted; provide rationale for using data sources.
Electronic Search
strategy
Describe literature search (e.g. provide electronic search
strategies with population terms, clinical or health topic
terms, experiential or social phenomena related terms, filters
for qualitative research, and search limits).
Improvement in Search Reporting
1988-2004
2005-2008
(Hannes & Macaitis, 2012)
Outstanding Issues/Challenges
• How to understand publication bias in qualitative
research (not around positive/negative findings)?
• How to systematise (and document) more
intuitive search approaches e.g. cluster searching
and searching for theory?
• How does sampling strategy translate into search
strategy?
• How to construct sampling frames for studies?
• How to sample for diversity?
• How many sources are enough?
• How to retrieve rich data? How to retrieve data
on theory and context?
Publication Bias in Qualitative
Research? – Part One
• “This does not mean that publication biases
do not exist in….qualitative research…
…qualitative studies may be less frequently
conducted, submitted for publication and/or
published in high quality and easily
accessible journals….raw data (i.e. research
findings)….needed to conduct a qualitative
systematic review may not be readily
available” (Finfgeld-Connett & Johnson 2013)
Publication Bias in Qualitative
Research? – Part Two
• “ignoring grey literature, such as
dissertations/theses, government reports,
monographs and books, on the basis that it may
be of lesser quality (as with quantitative
research) is empirically and logically
invalid…..[such] documents may be particularly
rich sources of qualitative data as page limits are
not generally imposed...”. (Finfgeld-Connett &
Johnson 2013)
• Qualitative researchers often choose to publish in
book form (“Truncation bias”)
Conclusions
• Increasing Importance of Explanatory
Sources (e.g. Context and Theory)
• Need for Ongoing Investigation of
Database Coverage and Supplementary
Search Techniques
• Requires Exploration/Selection of
Appropriate Sampling Methods
• Value versus Effort Trade-Off
References - 1
• Booth A, Carroll C. Systematic searching for theory to
inform systematic reviews: is it feasible? Is it desirable?
Health Info Libr J. 2015 Jun 11. doi: 10.1111/hir.12108.
• Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E.
Towards a methodology for cluster searching to provide
conceptual and contextual "richness" for systematic
reviews of complex interventions: case study (CLUSTER).
BMC Med Res Methodol. 2013 Sep 28;13:118.
• Campbell R, Pound P, Morgan M, Daker-White G, Britten N,
Pill R, et al. Evaluating meta-ethnography: systematic
analysis and synthesis of qualitative research. Health
Technol Assess 2011; 15(43).
• Flemming, K., & Briggs, M. (2007). Electronic searching to
locate qualitative research: evaluation of three
strategies. Journal of Advanced Nursing, 57(1), 95-100.
References - 2
• Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B. (2013). How to
locate and appraise qualitative research in complementary and alternative
medicine. BMC complementary and alternative medicine, 13(1), 125.
• Guise, J. M., Chang, C., Viswanathan, M., Glick, S., Treadwell, J., Umscheid,
C. A., ... & Trikalinos, T. (2014). Systematic Reviews of Complex
Multicomponent Health Care Interventions [Internet].
http://www.ncbi.nlm.nih.gov/books/NBK194851/
• Hannes, K., & Macaitis, K. (2012). A move to more systematic and
transparent approaches in qualitative evidence synthesis: update on a
review of published papers. Qualitative Research, 12(4), 402-442.
• McGinn, T., Taylor, B., McColgan, M., & McQuilkan, J. (2014). Social Work
Literature Searching Current Issues With Databases and Online Search
Engines. Research on Social Work Practice, 1049731514549423.
• Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R. (2010).
Literature searching for social science systematic reviews: consideration of
a range of search techniques. Health Information & Libraries
Journal, 27(2), 114-122.
• Pearson, M., Moxham, T., & Ashton, K. (2011). Effectiveness of search
strategies for qualitative research about barriers and facilitators of
program delivery. Evaluation & the Health Professions, 34(3), 297-308.
References - 3
• Stansfield, C., Brunton, G., & Rees, R. (2014). Search wide, dig deep: literature
searching for qualitative research. An analysis of the publication formats and
information sources used for four systematic reviews in public health. Research
Synthesis Methods, 5(2), 142-151.
• Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The
selection of search sources influences the findings of a systematic review of
people’s views: a case study in public health. BMC Medical Research
Methodology, 12(1), 55.
• Suri, H. (2011). Purposeful sampling in qualitative research synthesis. Qualitative
Research Journal, 11(2), 63-75.
• Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing
transparency in reporting the synthesis of qualitative research: ENTREQ. BMC
medical research methodology, 12(1), 181.
• Suri, H. (2013). Towards Methodologically Inclusive Research Syntheses:
Expanding Possibilities. Routledge.
• Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies
required health, social science, and grey literature databases. J Clin Epidemiol.
2014 Jul;67(7):800-10.