So… You Want to Produce a
Systematic Literature Review?
Daniel Amyot
[email protected]
January 2017
Bits of Wisdom
• If I have seen a little farther than others, it is
because I have stood on the shoulders of
giants.
Sir Isaac Newton
• Science is supposed to be cumulative, not
almost endless duplication of the same kind of
things.
Richard Hamming, 1968 Turing Award
So… You Want to Produce a Literature Review
p.2
Why Do a Literature Review?
• Mandatory part of the
program…
• My supervisor wants
me to…
• Part of the thesis
template I got…
• I want to graduate!!!
The literature
So… You Want to Produce a Literature Review
p.3
Are There Better Reasons?
http://www.phdcomics.com/comics/archive.php?comicid=789
So… You Want to Produce a Literature Review
p.4
Why Do This Systematically?
Better Reasons…
• Understand the domain and the state of the art
• Use a recognized methodology (not ad hoc)
• Avoid missing important (or sometimes better)
related work
• Be evidence-based and avoid bias
• Synthesis of work and relationships
• Find gaps/trends/agreements/disagreements in
existing work, or realize there are no gaps!
• Framework to compare/position your own work
• Support repeatability (rigor) and evolution
So… You Want to Produce a Literature Review
p.5
Another (More Selfish) Good Reason
• Get a publication!
– Reviews are usually well cited. Make yours
serious enough to be published.
– There are just too many papers being
published… Help people save time with a
good review, with useful insights.
So… You Want to Produce a Literature Review
p.6
Why Systematic? Evidence-Based!
• Evidence-based medicine has changed
research practices (Cochrane 1972)
• Failure to organise existing medical research cost
lives
• Clinical judgement of experts worse than systematic
reviews
• Evidence-based paradigm adopted by other
disciplines providing service to public
– Social policy, Education, Psychiatry…
• Sloooowwwlllly getting there in Management,
IS, CS, and Software Engineering!
– Work of Kitchenham and others
So… You Want to Produce a Literature Review
p.7
Systematic Reviews 1/2
[Kitchenham]
• A systematic literature review (SLR) is
– An overview of research studies that uses
explicit and reproducible methods
• Systematic reviews aim to synthesise
existing research
– Fairly (without bias)
– Rigorously (according to a defined
procedure or protocol)
– Openly (ensuring that the review procedure
is visible to and auditable by other
researchers)
So… You Want to Produce a Literature Review
p.8
Systematic Reviews 2/2
[Kitchenham]
• Support the evidence-based paradigm
1. Start from a well-defined question
2. Define a repeatable strategy for searching
the literature
3. Critically assess relevant literature
4. Synthesize literature
• With thanks to Cochrane, since 1972
http://en.wikipedia.org/wiki/Archie_Cochrane
So… You Want to Produce a Literature Review
p.9
What is So Systematic about This?
http://dilbert.com/
So… You Want to Produce a Literature Review
p.10
What is So Systematic about This?
•
•
•
•
•
Key search words and queries
Searching method and databases
Criteria for including/excluding references
Clear structure in appraising the evidence
Explicit discussion of limitations (threats to
validity)
• Many guidelines, but there is still room for
adaptation to your context and domain
So… You Want to Produce a Literature Review
p.11
SLR PROCESS AND
PROTOCOL
So… You Want to Produce a Literature Review
p.12
A Systematic Review Process
Develop Review Protocol
Plan Review
Validate Review Protocol
Identify Relevant Research
Select Primary Studies
Conduct Review
Assess Study Quality
Extract Required Data
Synthesise Data
Write Review Report
Document Review
Validate Report
[Kitchenham, 2007]
So… You Want to Produce a Literature Review
p.13
Research Questions…
• Assessing the effect of a technology (causality?)
• Assessing the frequency or rate of a project
development factor
– E.g. Rate of project failures
• Identifying cost and risk factors
• Identifying the impact of technology on
reliability, performance, cost…
• Pragmatic, well-focused questions
• More general questions for other purposes
– Review of research in domain X?
So… You Want to Produce a Literature Review
p.14
SLR Protocol
• Methods to be used for a systematic review
• Predefined protocol
– Avoid bias… Too easy and tempting to change the
research questions to fit search results!
– In practice however, for a first review, chances are
you will be cheating… Iterative approach!
So… You Want to Produce a Literature Review
p.15
SLR Protocol Content
• Content
–
–
–
–
–
–
–
–
–
Background
Research question(s)
Search strategy
Selection (inclusion/exclusion)
Quality assessment criteria
Data extraction
Synthesis
Threats/limitations
Reporting
Schedule/plan
• Sample SLR protocol template
• Sample protocol
So… You Want to Produce a Literature Review
p.16
Executing the Review Protocol…
So… You Want to Produce a Literature Review
p.17
IDENTIFICATION OF
RELEVANT RESEARCH
So… You Want to Produce a Literature Review
p.18
Pearl Growing
• Using 4-5 key paper(s) to find more
– Make use of paper’s subject headings (and other
index details) in key databases
– Also make use of its terminology (for your search
strings), the databases in which it is indexed (for
your databases), the journal it comes from (for
browsing), and so on
• Make sure the initial 4-5 papers are covered!
Some Potential Search Databases
•
•
•
•
•
•
•
•
•
•
•
Scopus (http://www.scopus.com/)
Web of Science (https://webofknowledge.com/)
PubMed (http://www.ncbi.nlm.nih.gov/pubmed/)
Medline (Ovid)
(http://www.ovid.com/site/catalog/databases/901.jsp )
IEEE Xplore (http://ieeexplore.ieee.org/)
ACM Digital Library (http://dl.acm.org/)
SpringerLink (http://www.springerlink.com/)
Google Scholar (http://scholar.google.ca/)
Even plain Google/Bing, sometimes (especially for commercial
products)
Many more on Wikipedia and at the uOttawa library
In your protocol, choose those that make sense in your
context.
So… You Want to Produce a Literature Review
p.20
Non-Academic Databases
• Canadian Periodical Index Quarterly (CPIQ)
• National Bureau of Economic Research (NBER)
• Organisation for Economic Co-operation and
Development (OECD)
• …
• But in practice:
– Make sure selected academic and non-academic are
reliable and have sufficient search options to handle
the complexity of your queries
– Past issues with Factiva, Ebrary, Social Science
Research Network …
So… You Want to Produce a Literature Review
p.21
Some Important Advice
• Be ready to use the Advanced Search
options, always
• If you are an employee, get your VPN access
if you want to work from home
• For technologies, do not underestimate
patents (Google Scholar)
• Your university librarian can help!
– http://biblio.uottawa.ca/en/research-help/researchguides-and-librarians
– See also research guides and relevant databases at
the same URL.
So… You Want to Produce a Literature Review
p.22
Keywords and Queries
• Quickly test your queries, especially for scope
– Not too broad (scope creep, unfeasible)
– Not too narrow (empty net, nothing to learn)
• Think of all possible synonyms for a same concept
– “literature review” vs “literature survey”
• Think of possible spellings
– modelling/modeling, behaviour/behavior
• Check already retrieved papers for possible keywords
and their combinations
• Make sure to follow the search engines rules. All
engines have different limitations…
So… You Want to Produce a Literature Review
p.23
Search Operators
•
•
•
•
•
Boolean operators: AND, OR, NOT…
Grouping: ()
Exact phrase: " "
Truncation: *
Proximity operators: NEAR, FAR, FOLLOWED BY,
WITHIN…
• Search fields: title, subject, abstract, keywords, full-text
• Limiters: date range, publication type, language, peerreviewed, study design…
So… You Want to Produce a Literature Review
p.24
Play a Little, for Scoping and for
Understanding Engine Limitations
•
•
•
•
•
Interested in a review of "Computer Security"?
– Google Scholar returns 360,000 results! (Jan. 2017)
– When do you want to finish your thesis? 
"Computer Security" AND Cloud?
– 25,800 results
"Computer Security" AND Cloud AND Amazon (since 2016)
– 902 results
"Computer Security" AND Cloud AND Amazon AND Prolog (since 2014)
– 12 results… and many irrelevant. Too restrictive?
"Computer Security" AND Cloud AND Amazon AND (Prolog OR ".NET") (since
2016)
– 435 results… but much garbage (net instead of .NET)…
– …
So… You Want to Produce a Literature Review
p.25
Even Advanced Queries Have
Limitations (Google Scholar)
So… You Want to Produce a Literature Review
p.26
Exercise on Google Scholar
• How would you search this abstract query?
(UCM OR "Use Case Map") AND (IP OR "Internet Protocol")
• Need to split it into multiple queries, and merge results!
UCM AND (IP OR "Internet Protocol")
"Use Case Map" AND (IP OR "Internet Protocol")
So… You Want to Produce a Literature Review
p.27
More Specialized, But Still Limited
(SpringerLink)
So… You Want to Produce a Literature Review
p.28
Getting There (PubMed)
So… You Want to Produce a Literature Review
p.29
Just How Complex Can Queries Get?
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
42.
43.
44.
45.
46.
47.
Consent/ or "consent (research)"/
"confidentiality (research)"/ or "Privacy and Confidentiality"/
HIPAA.mp.
"Health Insurance Portability and Accountability Act"/
(Opt-in or opt in or opt-out or opt out or non-consent or no consent or full consent or explicit consent).tw.
(waive$ adj3 consent).tw.
(waiver or consent status).tw.
(consent adj3 model$).tw.
(identifiable adj3 (data or information)).tw.
Data Protection Act.tw.
(Health and Social Care Act).tw.
Human Rights Act.tw.
(Caldicott or PIPEDA or Personal Data Protection Directive).tw.
(privacy adj3 act$).tw.
De-identif$.tw.
(Personal information protection and electronic documents act).tw.
(double-cod$ or double cod$ or single-cod$ or single cod$ or Re-identif$ or reidentif$ or deidentif$ or anonymiz$ or anonymis$ or pseudonymiz$ or pseudonymis$ or reconsent$ or
anonymity or identifiability).tw.
((express$ or knowledgeable) adj3 consent).tw.
(data adj3 unlink$).tw.
((strip$ or remov$ or delet$) adj3 identifier$).tw.
((linked or linkable or coded) adj3 (information or data)).tw.
((unidentif$ or non-identif$ or nonidentif$) adj3 (data or information)).tw.
or/1-22
"bias (research)"/ or nonresponse bias/ or sampling bias/ or selection bias/ or research subject recruitment/
patient selection/
sample size/
"costs and cost analysis"/ or cost-benefit analysis/
Time Factors/
((consent or response or recruitment or participation or refusal$) adj3 rate$).tw.
(bias or biases).mp.
(survey$ adj3 (response$ or participation)).tw.
(Predictor$ adj3 consent).tw.
accrual.tw.
or/24-33
(audit$ or registr$ or observational or epidemiolog$).tw.
((health service$ or medical record$) adj3 (research or study or studies)).tw.
research ethics/ or research methodology/ or data collection/ or data collection methods/ or data collection, computer assisted/ or data mining/ or exp observational methods/ or
"record review"/
"quality of health care"/ or "outcomes (health care)"/ or outcome assessment/ or quality assessment/ or nursing audit/ or "process assessment (health care)"/ or program evaluation/
nutrition assessment/ or exp Medical Records/ or sampling methods/
CROSS SECTIONAL STUDIES/ or PROSPECTIVE STUDIES/
epidemiological research/ or exp health services research/
registries, disease/ or registries, implant/ or registries, organ/ or registries, trauma/ or surveys/ or exp vital statistics/
"QUALITY OF CARE RESEARCH"/ or Quality Assurance/ or Audit/ or databases/
or/35-43
and/23,34,44
limit 45 to english
limit 46 to yr="1990 - 2007"
So… You Want to Produce a Literature Review
p.30
On Getting (Only/All) Relevant Items
• Precision: how many retrieved items are relevant?
• Recall: how many relevant items were retrieved?
• See:
https://en.wikipedia.org/wiki/Precision_and_recall#Defi
nition_.28classification_context.29
So… You Want to Produce a Literature Review
p.31
How to Handle Complex Queries?
• Queries might have to be split into multiple
sub-queries, and results (list of papers)
merged manually
• Lists from multiple engines require merging
too
– This is where a citation manager can help
• Downloading the papers themselves is also
painful…!
So… You Want to Produce a Literature Review
p.32
Document your Queries
• List your exact queries in your thesis!
– Allows repeatability by others
– Allows YOU to repeat the queries before submitting
your thesis…
– As your thesis work spans many years, it is good to
refresh your review before submission, just in case!
• Different engines have different syntaxes and
limitations
– Document your abstract queries in your thesis, as if
you were not limited by the intricacies of the
concrete search engines
– You might want to keep concrete queries for
yourself, to run them again later
So… You Want to Produce a Literature Review
p.33
Beware of Common Mistakes
"Business Process" OR "Business Process Management"
Business Process
Business
Process
Management
So… You Want to Produce a Literature Review
p.34
Is This Query Uselessly Complex?
("Business process compliance" OR "Business Process
Compliance Management" OR "Regulatory Compliance"
OR "Regulatory Compliance Management" OR
"Legal Compliance")
AND
("Systematic review" OR "Systematic Survey" OR
"Literature Review" OR "Literature survey" OR
"State-of-the-art")
AND
("Assess*" OR "Measure*" OR "Monitor*")
So… You Want to Produce a Literature Review
p.35
Is This Query Uselessly Complex?
("Business process compliance" OR "Business Process
Compliance Management" OR "Regulatory Compliance"
OR "Regulatory Compliance Management" OR
"Legal Compliance")
AND
("Systematic review" OR "Systematic Survey" OR
"Literature Review" OR "Literature survey" OR
"State-of-the-art")
AND
("Assess*" OR "Measure*" OR "Monitor*")
So… You Want to Produce a Literature Review
p.36
Simpler and Better Query
("Business process compliance" OR
"Regulatory compliance" OR "Legal compliance")
AND
("Systematic review" OR "Systematic survey" OR
"Literature review" OR "Literature survey" OR
"State-of-the-art")
AND
("Assess*" OR "Measur*" OR "Monitor*")
How many concrete Google Scholar queries
do we need to run here?
So… You Want to Produce a Literature Review
p.37
What Years Should Be Covered?
• This depends on the novelty of the subject
• One option is to start with the year not
included in the last literature review of the field
[Antonova]
– I personally do not like that option…
• If no previous reviews, go by when the first
relevant study was published
• If overlapping, but not identical topic of review,
ok to use the same studies
• Can also be systematic for a recent period, and
ad hoc before (to include seminal papers)
So… You Want to Produce a Literature Review
p.38
Keep a Table of Returned Papers
• This is your raw data!
• You may take note of different characteristics
(columns)
– source engine(s), year published, type
(conference/journal/thesis/patent), country,
query that returned it, number of Google
Scholar references…
• Can be used for statistics and trends
• Can form an (online) Appendix in your thesis
or paper
So… You Want to Produce a Literature Review
p.39
What To Do Once You Have 300
Papers to Read? (1/2)
• Use specific and explicit inclusion/exclusion
criteria
• Might want to focus on the abstract and
perhaps on the introduction/conclusion
sections first (filtering)
• A paper is of low quality or irrelevant or in a
language you don’t understand? Exclude!
• A paper cites another interesting paper not
returned by your initial queries? Include!
So… You Want to Produce a Literature Review
p.40
What To Do Once You Have 300
Papers to Read? (2/2)
• Reduce to a manageable number of papers
– <40 for a systematized review, but can be a
lot more for a systematic map
• Document how many (and which) papers got
filtered out (or added) at each step
• Involve a second person to do the same
exercise in parallel and compare (hard and
costly!), or minimally to review your decisions
(cheaper), in order to avoid bias
So… You Want to Produce a Literature Review
p.41
Citation Chaining
• Backwards chaining: reference list checking
• Forward chaining: citation checking
– Web of Science
– Scopus
– Google Scholar
When to Stop?
• How much time have we got left?
• Saturation!
• Discussion on literature searching (example):
– Conventional subject search (30 of 41)
– Citation searching (3 + 2 serendipitous)
– Reference list checking (4)
– Contact with experts (2)
– Pearl growing (abandoned)
Summary
of Selection
Results
PRISMA 2009,
Flow Diagram
(Word version)
So… You Want to Produce a Literature Review
p.44
ASSESSMENT AND
REPORTING
So… You Want to Produce a Literature Review
p.45
Time to Assess the Papers
• Define criteria for their categorization and
evaluation
• Read and evaluate the papers!
• Do not simply report on Author W did X and author
Y did Z…
• Look for commonalities/differences, trends, and
cross-cutting observations. Critical analysis!
• Include a tabular summary of your evaluation
– This will allow you to add one row at the end of your
thesis, with your own work!
– Do not make your evaluation scale too simplistic
(yes/no); include ranges or partial satisfaction levels
So… You Want to Produce a Literature Review
p.46
Organization (Potential Orders)
• Topical
– by main topics or issues
• Chronological
– by the dates the research was published
• Problem-Cause-Solution
– moves from the problem to the solution
• General-to-Specific
– examine broad-based research first and then focus on
specific studies that relate to the topic
• Specific-to-General
– discuss specific research studies so conclusions can be
drawn
• Others…
So… You Want to Produce a Literature Review
p.47
Write About…
• Significance of what exists
• Gaps or areas of disagreement that beg for
more research
– From you if thesis, for others if papers!
• The 3-4 approaches closest to what you have
in mind for your research (your competition)
• Conclusions
So… You Want to Produce a Literature Review
p.48
Do Not Underestimate…
• Limitations of your own review
– Perry D.E., Porter A.A. and Votta L.G.: Empirical studies of
software engineering: a roadmap. Conf. on the Future of
Software Engineering, ICSE’2000, ACM, 2000, 345–355.
– Brewer, M.: Research Design and Issues of Validity. In
Reis, H. and Judd, C. (eds.) Handbook of Research
Methods in Social and Personality Psychology. Cambridge
University Press, 2002
• In essence, show your maturity by stating
explicitly how to attack the quality of your review!
So… You Want to Produce a Literature Review
p.49
Limitations = Threats to Validity
• Construct validity: Specifies how well the methodology
and protocol helped answered the research questions
• Internal validity: Examines any bias and confounding
factors
• External validity: Specifies how much of the results can
be generalized
• Many other types…
– http://en.wikipedia.org/wiki/Validity_%28statistics%29
– https://en.wikipedia.org/wiki/Internal_validity
• Consider them (with mitigations) in advance
– Could influence your literature review protocol
• Report on threats mitigated and on the remaining ones
So… You Want to Produce a Literature Review
p.50
Typical Threats for Literature
Reviews (1/2)
• Publication bias
– Negative findings may not get published
– Positive findings may get published more than once
• Selection bias
– Inclusion and exclusion criteria can create bias
– Tempting to use only criteria where we know our
own approach will score well and show uniqueness!
• Language bias (i.e. English Language)
– One language only… Enough?
– Authors might be more likely to report positive
findings in an international English language journal
and negative findings in a local journal
So… You Want to Produce a Literature Review
p.51
Typical Threats for Literature
Reviews (2/2)
• Protocol
– Inconsistent use of terms by the community
– Bias introduced by having only 1-2 person(s)
involved in the review for data collection, data
filtering, data analysis and data reporting
• But we can’t expect more in a thesis lit review!
– Time limits! You cannot spend 5 years doing a
review
• Still, you may want to budget several months to
do this well
• … and more!
So… You Want to Produce a Literature Review
p.52
In the Conclusions
• Make sure you state something new that was not
obvious prior to the review!
• Answer your research questions
• Indicate concretely (i.e., operationalize) how your
results can impact and be used by:
– Researchers
– Practitioners
So… You Want to Produce a Literature Review
p.53
REVIEW TYPES
So… You Want to Produce a Literature Review
p.54
Review Types, Systematic or Not
• There are many, many types of reviews.
– Grant, M.J., Booth, A.: A typology of reviews: an
analysis of 14 review types and associated
methodologies. Health Info Libr J. 2009
Jun;26(2):91-108.
– http://guides.mclibrary.duke.edu/c.php?g=158155&
p=1035849
• Typical ones for this course:
– Systematized Review
– Systematic Map
– Meta-analysis
So… You Want to Produce a Literature Review
p.55
Typical Systematized Review
• Almost an SLR! Usually narrative with tables
• Examples from previous years
– Almoaber, B., Amyot, D. (2017) Barriers to Successful
Health Information Exchange Systems in Canada and
the USA - A Systematic Review. Int. J. of Healthcare
Information Systems and Informatics, 12(1), 44-63
– Lessard, L., Okakwu, C.P. (2016) Enablers and
Mechanisms of Value Cocreation in KnowledgeIntensive Business Service Engagements: A
Research Synthesis. 49th Hawaii International
Conference on System Sciences (HICSS), IEEE CS, 16241633
So… You Want to Produce a Literature Review
p.56
Typical Systematic Map
• Goes wide, but with less depth. More about a general
overview and trends
• Example:
– Horkoff, J., et al. (2016) Goal-Oriented Requirements
Engineering: A Systematic Literature Map.
Requirements Engineering Conference (RE), 2016 IEEE
24th International, IEEE CS, 106-115
So… You Want to Produce a Literature Review
p.57
Reviews Can Go Meta!
• Primary study
– Paper on phenomenon of interest, raw material
– May involve interviews, questionnaires, observations…
– Not much interpretation, limited scope
• Secondary study
– Publication with interpretation (journal, thesis, review)
– Meta-analysis use data from primary studies
• Tertiary study?
– Literature reviews as input!
• Meta-analysis and tertiary studies often look for
quantitative results (e.g., aggregation of statistical
results)
So… You Want to Produce a Literature Review
p.58
Good Example of Meta Analysis
• Nery, P.B., Belliveau, D., Nair, G.M., Bernick, J,
Redpath, C.J., Szczotka, A., Sadek, M.M., Green, M.S.,
Wells, G., Birnie, D.H. (2016) Relationship Between
Pulmonary Vein Reconnection and Atrial
Fibrillation Recurrence. JACC: Clinical
Electrophysiology, 2(4), 474-483
• Aggregates data from primary studies and get more
global (statistical) insight
• Very complete, and yet concise and to the point.
So… You Want to Produce a Literature Review
p.59
GETTING HELP!
So… You Want to Produce a Literature Review
p.60
At the uOttawa Library
• http://biblio.uottawa.ca/en/health-scienceslibrary/health-sciences-library-workshops
– BiblioClinic Series
– Planning for your Systematic or Scoping Review
– Citation Manager BiblioClinic
• Next Week!
So… You Want to Produce a Literature Review
p.61
[ http://www.prisma-statement.org/ ]
Beware of Common Pitfalls
• Do not focus on getting rid of the review; target the
Truth!
• Do not be superficial
– Say something about the papers and their
contributions
– Do not include just counts per year
– Show something that was not obvious before
• Do not use quantitative statistics if you do not use a
substantial number of papers
– Consider descriptive statistics instead, if useful
So… You Want to Produce a Literature Review
p.63
Be Careful with References
• In your bibliography or list of references, be
attentive to detail…
• Completeness of information for each item
– Authors, title, journal (with editors, volume and issue
number) or conference (maybe with series and
number), publisher, pages, year, maybe the DOI
• Stick to consistent conventions
– Order, style (e.g., italic), periods/colons/quotes,
author first full name or first letter, “and”, “In:”, …
• Cite properly in the text!
So… You Want to Produce a Literature Review
p.64
Be Careful with Plagiarism/Fraud
• Real problem, more general than for just literature
reviews
• Reference words, data, ideas properly
• Plagiarism and fraud hurt everybody's reputation!
• MUST READ
– “Beware of plagiarism” document
– Academic integrity at uOttawa and at Carleton
– Consequences at uOttawa
– Academic fraud and student guide
So… You Want to Produce a Literature Review
p.65
CONCLUSIONS
So… You Want to Produce a Literature Review
p.66
Is This Systematic Stuff Easy?
• No! Requires more effort than informal
reviews
• Difficult for lone researchers
– Standards require two researchers, to minimize
individual bias
– Involve your supervisor!
• Often incompatible with requirements for short
papers
– But pretty good for journals and theses!
So… You Want to Produce a Literature Review
p.67
Take My Advice Seriously!
So… You Want to Produce a Literature Review
p.68
From My Own Experience
• Not perfect (still learning!)
• All of my Ph.D. students who did a systematic literature
review had it published!
• All of my Ph.D. students who did a more ad hoc or
narrative literature review did not get it published!
• One of my best cited paper is a literature review
So… You Want to Produce a Literature Review
p.69
References and Further Reading
• Guidelines for performing Systematic Literature
Reviews in Software Engineering, Version 2.3
(2007)
• Systematic literature reviews in software
engineering – A systematic literature review
(2009)
• A Guide to Conducting a Systematic Literature
Review of Information Systems Research (2010)
• A typology of reviews: an analysis of 14 review
types and associated methodologies (2009)
So… You Want to Produce a Literature Review
p.70
References (Online Presentations)
• B. Kitchenham: Evidence-Based Software
Engineering and Systematic Reviews (2005)
• E. Antonova: Systematic Literature Review
• L. Ann: Literature Review (2009)
So… You Want to Produce a Literature Review
p.71