Study protocol: descriptive analysis
of controlled before-after (CBA)
and interrupted time series (ITS)
studies in Cochrane reviews
1
2
1
Stephanie Polus , Dawid Pieper , Jacob Burns , Atle
3
4
5
Fretheim , Craig Ramsay , Julian PT Higgins , Lisa M
1
1
Pfadenhauer , Eva Rehfuess
1
Institute for Medical Informatics, Biometry and
Epidemiology, LMU Munich, Germany
2
Institute for Research in Operative Medicine
Chair of Surgical Research, Faculty of Health,
School of Medicine, Witten/Herdecke University
3
Norwegian Institute of Public Health, Oslo,
Norway
4
Health Services Research Unit, University of
Aberdeen, UK
5
School of Social and Community Medicine,
University of Bristol, UK
Background
Randomized controlled trials (RCTs) are considered
to be the gold standard for effectiveness studies in
healthcare (Oxford Centre for Evidence-based
Medicine, Oliver et al., 2008, Shadish W et al.,
2002). They are, however, less available and in
certain circumstances not applicable when
assessing
the
effectiveness
of
complex
interventions (Craig et al., 2008, Deeks et al.,
2003). An increasing recognition of multi-layered
public health interventions that target whole
systems calls for approaches that are capable of
evaluating their effects (Sanson-Fisher et al., 2014),
and complex interventions in public health and
health care services are therefore often assessed
by nonrandomized study designs (Norris and
Atkins, 2005, Higgins et al., 2013).
As NRSs are increasingly used to assess the
effectiveness of complex interventions, systematic
reviewers have been faced with whether and how
to incorporate these studies into systematic
reviews (Norris and Atkins, 2005, Glenton et al.,
2013, Higgins et al., 2013, Peinemann et al., 2013).
Although review authors more and more recognize
the importance of including nonrandomized
studies in systematic reviews (Norris and Atkins,
2005), they are found to be often neglected, and
poor reporting and indexing of their characteristics
complicate the searching and screening for review
authors (Glenton et al., 2013, Ramsay et al., 2003,
Norris and Atkins, 2005). Along with the inclusion
of NRSs comes an increased workload (Glenton et
al., 2013), as well as additional concerns regarding
evidence synthesis (Valentine and Thompson,
2013) and risk of bias (Higgins et al., 2013).
Little is known about the nonrandomized studies
that are most often included in Cochrane
systematic reviews, namely interrupted time series
(ITS) and controlled before-after (CBA) studies (Ijaz
et al., 2014, Norris and Atkins, 2005). Tension
between study names and study features lead to
confusion in the research world regarding what
can be defined an ITS or CBA study (Higgins et al.,
2013). The same plurality of definitions exists
about naming a study experimental. Depending on
the perspective of the researcher, an experimental
study can be an “interventional study” without
necessarily investigator control over the allocation
to the intervention (AFMC, n.d.) or a study where
the investigator has control over the allocation to
the intervention (Shadish W et al., 2002). Different
classifications
into
experimental,
quasiexperimental or non-experimental studies exist for
ITS and CBA studies, depending on their
characterized study features (Hartling et al., 2011,
Shadish W et al., 2002, Deeks et al., 2003,
Norwegian Satellite of the Cochrane Effective
Practice and Organisation of Care Group, 2014,
Cochrane Bias Methods Group, 2013). The
Cochrane Effective Practice and Organisation of
Care (EPOC) Group (Ballini et al., 2011) developed
standards for the inclusion of certain
nonrandomized studies. These include ITSs and
CBAs with clearly defined features, satisfying a
specific level or rigour and thus reducing risk of
bias (Norwegian Satellite of the Cochrane Effective
Practice and Organisation of Care Group, 2014).
The EPOC criteria have been widely used within
the Cochrane Collaboration but also beyond
(Rockers et al., 2012). Yet, taking the recent
literature into account, different definitions, study
labels and other factors related to the specific
study designs reveal that no coherent picture of
these study designs exists in health care research
(Higgins et al., 2013, Reeves BC, 2008, Rockers et
al., 2012).
ITS study
As “stronger quasi-experimental designs”, ITS
studies can provide excellent effect estimates
(Shadish W et al., 2002). ITS in the field of health
systems interventions proved to yield similar
results and particularly controlled ITS revealed
possibly more informative results than the C-RCTs
due to their ability to control for the time trend
(Fretheim et al., 2015). In ITS studies, the
intervention effect is measured against the preintervention trend. According to EPOC criteria, the
study must use at least three data points before
and three after the intervention and must clearly
define the point in time when the intervention
occurred (Norwegian Satellite of the Cochrane
Effective Practice and Organisation of Care Group,
2014). Usually, there is no investigator control over
the allocation of the intervention and the study is
conducted retrospectively. Common data sources
are routine data, although multiple data sources
are often combined. The Cochrane Collaboration
defines an ITS as a study without a control group,
referring to the retrospective, more common ITS
design that uses routine data (Norwegian Satellite
of the Cochrane Effective Practice and
Organisation of Care Group, 2014, Higgins and
Green, 2011). Higgins et al. (2013) and Hartling et
al. (2011) outline the possibility of a control group
that does not receive the intervention, which
mostly occurs when the ITS is designed
prospectively. ITS studies measure the intervention
effect on an aggregate/group level, not on the
individual level (Norwegian Satellite of the
Cochrane Effective Practice and Organisation of
Care Group, 2014).
In the public health field, few papers have
examined ITS studies (Ramsay et al., 2003, SansonFisher et al., 2014, Grimshaw et al., 2000, Rockers
et al., 2012). Ramsay et al. (2003) investigated the
use of nonrandomized studies in systematic
reviews and found inappropriate quality appraisal
and analyses of ITSs in a number of systematic
reviews that included ITS studies, leading to the
frequent judgment of non-significant effects as
significant. Furthermore, there seems to be no
clear distinction between ITS studies and similar
study designs using different or similar analysis
methods, such as the regression discontinuity
design (Reichardt and Henry, 2012, Higgins et al.,
2013).
An increasing number of routine databases, such
as national registers that collect data on a regular
basis over prolonged periods of time, are available
that could be used for evaluation purposes.
Therefore, ITS will be increasingly used for the
evaluation of intervention effects (Fretheim et al.,
2013). Clarity about this study design in terms of its
particular strengths and weaknesses and
challenges in relation to study design, data
collection and data analysis would be helpful for
researchers conducting ITS studies as well as for
review authors including ITS studies in systematic
reviews of complex interventions.
CBA study
In CBA studies “observations are made before and
after the implementation of an intervention, both
in a group that receives the intervention and in a
control group that does not” (Higgins and Green,
2011). The criteria of methodological rigour of the
EPOC group determine that there are at least two
intervention
and
two
control
sites,
contemporaneous data collection, and an
appropriate choice of control site (Norwegian
Satellite of the Cochrane Effective Practice and
Organisation of Care Group, 2014, Cochrane Bias
Methods Group, 2013).
Although CBAs are the most utilized
nonrandomized study designs included in
systematic reviews (Ijaz et al., 2014), no studies
have – to our knowledge – thoroughly examined
how CBA studies are designed, defined and used,
and there are hardly any studies that have defined
and outlined the strengths and weaknesses of
CBAs (Grimshaw et al., 2000, Sanson-Fisher et al.,
2014).
A more in-depth look into the characteristics of
CBAs could help clarify conflicting issues about the
methodology of CBAs, such as a clear description
of study features of CBAs and similar study designs.
Whereas the EPOC group states that in CBAs the
investigator has no control over the allocation to
the study or control groups (Norwegian Satellite of
the Cochrane Effective Practice and Organisation
of Care Group, 2014), Deeks et al. (2003) suggest
that a CBA “can also be considered an
experimental design if the investigator has control
over, or can deliberately manipulate the
introduction of the intervention”, which the
Cochrane Bias Methods Group points out as well
(Cochrane Bias Methods Group, 2013). In a
classification tool developed by Hartling et al.
(2011), a CBA study is not classified as
experimental, which they further define as
investigators having “direct control over study
conditions including: allocation to 2 or more
interventions, timing of interventions, choice of
outcomes and timing of outcome assessments”.
The study authors mention in their discussion that
in one third of the studies testers did not agree
whether a study was experimental or not. They
explicitly call for clear guidelines to interpret the
extent of control that an investigator has (Hartling
et al., 2011), making it clear that confusion in this
area is prevalent.
The incoherent terminology, too, urges for a clear
differentiation between features of CBA studies
and other study designs, such as controlled clinical
trials (Ijaz et al., 2014, Sanson-Fisher et al., 2014,
Hartling et al., 2011). For systematic review
authors but also policy makers, a clear terminology
would help to better find, distinguish and
understand decisions concerning this study design.
Clarity about in which fields and for which types
and levels of interventions CBA studies are most
often used could increase our knowledge for
further decisions.
outcome assessment, statistical
methods)
From this, we will derive implications for
terminology, definitions and strengths and
weaknesses of the two study designs, and explore
consequences for reporting guidelines and quality
appraisal of these studies, including informing
application of the Cochrane Risk Of Bias
Assessment Tool: for Non-Randomized Studies of
Interventions (ACROBAT-NRSI).
Methods
Overview
Ijaz et al (2014) searched the Cochrane Database of
Systematic Reviews until May 2012 for systematic
reviews that included nonrandomized studies. We
will base our methods on those of Ijaz et al. (2014)
and update their searches until March 2015. We
will collect and examine systematic review
characteristics for all systematic reviews identified
through our search as well as identified by Ijaz et
al. (2014). For a subset of all identified reviews
published between June 2012 and March 2015, as
outlined below (figure 1), we will randomly select
and extract CBA and ITS studies included in the
systematic
reviews
and
describe
their
characteristics.
Research objective
Our primary objective is to descriptively analyse a
sample of CBA and ITS studies included in
Cochrane systematic reviews published between
May 2012 and March 2015 in terms of
(i)
Publication characteristics (e.g.
publication year, country, journal)
(ii)
Application characteristics (e.g.
objectives, populations,
interventions, outcomes, type of
intervention, level of intervention)
(iii)
Methodological characteristics in
relation to study design, data
collection and data analysis (e.g.
Inclusion criteria
For systematic reviews
(i)
published between June 2012 and March
2015, as both the use and quality of these
study designs in primary research and
their inclusion in SRs is likely to have
increased in recent years
(ii)
including nonrandomized studies. For
analysis, however, we include only the
reviews where a predefined inclusion
criterion was the use of CBA and ITS
studies (as specified by review authors)
and having at least 2 CBA or 2 ITS studies
(adhering or not adhering to EPOC
standards) included to ensure applicability
to a given type of intervention and a
certain degree of representativeness in
study conduct
After examining the characteristics of all included
systematic reviews (see data collection and
analysis), we will purposively select 2 systematic
reviews from each type of intervention as
categorized beforehand (see Appendix A for
definitions). This is expected to yield 20 systematic
reviews, provided CBAs and ITS studies have been
sufficiently applied across all types of intervention.
In the included systematic reviews we will
randomly select 2 CBA or 2 ITS to conduct data
extraction and analysis. We will undertake the
selection process separately for ITS and CBA
studies.
Primary studies
The following information will be extracted from
selected studies:
a. Publication characteristics
i. Year of publication
ii. Journal
iii. Country of study
iv. Language of study
v. Funding source
vi. Terminology (e.g. study described as
a CBA or cohort)
b.
Application characteristics
i. Study objective
ii. Population, intervention, comparison,
and outcome (PICO)
iii. Type of intervention (behavioural,
clinical, environmental, health policy,
health system, nutrition,
pharmaceutical, occupational,
screening, and vaccination)
iv. Level of intervention (individual,
organisational/large scale, population
level)
c.
Methodological characteristics concerning:
i. Study design (e.g. setting, control,
allocation, timing)
ii. Data collection (e.g. measurements,
outcome assessments, source of data,
timing)
iii. Data analysis (e.g. statistical methods,
unit of analysis, study conduct)
iv. Quality assessment by review authors
(used quality tool, risk of bias
judgement by authors)
Search methods
Our search methods will be based on Ijaz et al
(2014) and include reviews from May 2012 until
March 2015.
(("case-control"[tiab] OR "quasi random"[tiab] OR
"quasi experimental"[tiab] OR ecological[tiab] OR
"non-random"[tiab]
OR
"interrupted-timeseries"[tiab] OR "before-after"[tiab] OR "beforeand-after"[tiab] OR cohort[tiab]) AND "Cochrane
Database
Syst
Rev"[Journal])
AND
("2012/06/01"[PDAT] : "3000"[PDAT])
Data collection and analysis
We will collect data on systematic review and
primary study level. One author will do the data
extraction of the included systematic reviews. Two
assessors will independently extract the data of
the primary studies using the data extraction
tables below for publication, application and
methodological characteristics.
Data extraction
Systematic reviews
For all SRs identified by Ijaz et al (2014) and
identified by our updated searches that meet the
inclusion criteria as specified above, we will
describe:
(i)
The type of studies and number that
are included in each review
(ii)
The respective Cochrane Group
(iii)
The type of intervention
(iv)
The level of intervention
Descriptive analysis
Systematic reviews
In their study, Ijaz et al. (2014) show the number of
nonrandomized studies they found in relation to
the respective Cochrane Group. We will show a
modified table of the data specifically for CBA and
ITS studies extended to the type and level of
intervention of the study, and to the time period of
our search (see Appendix A for the definitions of
the intervention types and levels).
Primary studies
Table 1 to 3 will be used as data extraction
templates for our updated search. This includes an
individual analysis of characteristics for ITS and
CBA study, respectively.
Table 3. Publication and application characteristics
[Study ID]
[Reference]
CBA
ITS
Publication characteristics
Year of publication
Journal
Country of study
Language of study
Funding source
Authors’ conflicts of interests
Terminology
Definition of study design provided
Application characteristics
Study objective
PICO Logic Model components
Who is the target
Population
population/recipient?
Intervention
design
Comparison
Outcomes
Setting
Components
☐ nonprofit
☐ for-profit
☐ no funding
☐ Funding sources not
reported
☐ other
☐yes, explanation:
☐potentially, explanation:
☐no, explanation:
[Key terms]
[Key terms]
☐ no
☐ yes:
☐ EPOC
☐ other:
☐ nonprofit
☐ for-profit
☐ no funding
☐ Funding sources not
reported
☐ other
☐yes, explanation:
☐potentially, explanation:
☐no, explanation:
[Key terms]
[Key terms]
☐ no
☐ yes:
☐ EPOC
☐ other:
☐ population based on
disease (patients)
☐ population based on risk
factors (high-risk groups)
☐ population based on socioeconomic/-cultural/demographic/occupational/ot
her characteristics
☐ general population
☐ technology &
Infrastructure:
☐ education:
☐ regulations and policies:
☐ other:
☐ no
☐ yes:
☐ intermediate outcomes
☐ health outcomes
☐ non-health outcomes
☐ country
☐ region
☐ household
☐ community
☐ school/other educational
institution
☐ company
☐ hospital
☐ population based on
disease (patients)
☐ population based on risk
factors (high-risk groups)
☐ population based on socioeconomic/-cultural/demographic/occupational
/other characteristics
☐ general population
☐ technology &
Infrastructure:
☐ education:
☐ regulations and policies:
☐ other:
☐ no
☐ yes:
☐ intermediate outcomes
☐ health outcomes
☐ non-health outcomes
☐ country
☐ region
☐ household
☐ community
☐ school/other educational
institution
☐ company
☐ hospital
Type of
intervention
Level of
intervention
☐ other, please specify:
☐ health policy
☐ health systems
☐ behavioural
☐ environmental
☐ occupational
☐ clinical
☐ pharmaceutical
☐ nutrition
☐ screening
☐ vaccination
☐other, please specify
☐ individual
☐ organisational
☐ large scale/population
☐ other, please specify:
☐ health policy
☐ health systems
☐ behavioural
☐ environmental
☐ occupational
☐ clinical
☐ pharmaceutical
☐ nutrition
☐ screening
☐ vaccination
☐other, please specify
☐ individual
☐ organisational
☐ large scale/population
Table 4. Methodological characteristics of ITS
[Study ID]
[Reference]
Methodological characteristics of ITS
Study design
Complies with EPOC
criteria
Temporal design
Population
Comparison/Control
group
Allocation directly linked
to study?
Allocation (if applicable*)
☐ Yes
☐ No
☐ Retrospective
☐ Prospective

Eligible population
o Number of target participants, please specify:
 Selected population:
o Number of participants at each measurement, please specify:
o Characteristics of population, please specify (e.g. gynaecologists
working in primary care):
☐ No
☐ Yes (please see below*):
☐ No (natural experiment)
☐ Yes
 Assignment to intervention/control:
☐ Random
☐ Quasi-random, please specify (if reported):
☐ Non-random:
o Allocation by:
☐ Time differences
☐ Location differences
☐ Healthcare decision makers
☐ Participants’ preferences
☐ On the basis of outcome
☐ Other process:
☐ Not reported/not clear

Reported explanation for allocation (i.e. why group/area A, B received the
intervention and group/area C,D did not):
☐ Not reported
☐ Yes, reported, please specify:
Unit of allocation (if
applicable*)
☐ Country
☐ Region
☐ Household
☐ Community
☐ School/other educational institution
☐ Company
☐ Hospital
☐ Other, please specify:
Sites


Number of intervention sites, please specify:
Characteristics of intervention sites, please specify (detailed description if
available):
If applicable:
 Number of control sites, please specify:
 Characteristics of control sites, please specify:
 Likely contamination between sites:
☐ Yes
☐ No
☐ Unclear

Timing of intervention
Time periods of
measurement before and
after intervention
Data collection
Source of data
Outcome assessment
Number of
measurements
Number of observations
and/or events per time
point
Data analysis
Time periods of analysis
before and after
intervention
Statistical analysis:






What type of data(e.g. routine data, please specify all sources):
Are the sources of data before and after intervention the same:
☐ Yes
☐ No, please specify:
☐ Non-validated subjective assessment (e.g. recall of diet)
☐ Validated subjective assessment (e.g. recall of diet through validated
questionnaire)
☐ Non-validated objective assessment (e.g. observed/ weighed food intake)
☐ Validated objective assessment (e.g. observed/weighed food intake with
validated protocol)
 Data points before intervention:
☐ <3 datapoints
☐ 3-5 datapoints
☐ 6-11 datapoints
☐ ≥12 datapoints
 Data points after intervention:
☐ <3 datapoints
☐ 3-5 datapoints
☐ 6-11 datapoints
☐ ≥12 datapoints
☐ <100, specify exact number:
☐ ≥100, specify exact number:


Before intervention, please specify:
After intervention, please specify:
☐ ARIMA
☐ Time series regression models
☐ Adjustment for trends (specify):
☐ Adjustment for auto-correlation
☐ Other (specify):
☐ Other, please specify:

Unit of analysis
Comparability of intervention and control sites according to baseline
characteristics
☐ Yes
☐ No
☐ Unclear (no baseline assessment)
Time point of intervention (when), please specify:
Time period of intervention (how long) , please specify:
Before intervention, please specify:
After intervention, please specify:
Adjustment for confounders
☐ Yes
☐ No
☐ Country
☐ Region
☐ Household
☐ Community
☐ School/other educational institution
☐ Company
☐ Hospital
☐ Other, please specify:
Quality (strengths and
weaknesses) at primary
study level by study
authors
Effect sizes of primary
outcomes as defined by
study authors:


Strengths, please specify:
Weaknesses, please specify:

Direction of effect:
☐ Positive
☐ Negative
Significance of effect:
☐ Yes
☐ No
Used quality appraisal tool:
Risk of bias judgment of study:

Quality assessment at
systematic review level
by review authors


Table 5. Methodological characteristics of CBAs
[Study ID]
[Reference]
Methodological characteristics of CBAs
Study design
Complies with EPOC study design
☐ Yes
☐ No
Temporal design
☐ Retrospective
☐ Prospective
Population

Eligible population:
o Number of target participants, please specify:

Selected population:
o
Number of participants at each measurement, please
specify:
Intervention
Control
Pre intervention
During
intervention
Post intervention
o
Characteristics of population at baseline (e.g.
gynaecologists working in primary care), please specify:
Allocation directly linked to study?
☐ No (natural experiment)
☐ Yes (please see below*):
Selection of control/intervention sites

Assignment to intervention/control:
☐ Random
☐ Quasi-random, please specify (if reported):
☐ Non-random:
o Allocation by:
☐ Time differences
☐ Location differences
☐ Healthcare decision makers
☐ Participants’ preferences
☐ On the basis of outcome
☐ Other process:
☐ Not clear/not reported

Reported explanations for allocation (i.e. why group/area A, B
received the intervention and group/area C,D did not):
☐ Not reported
☐ Yes, reported, please specify:
Unit of allocation
☐ Country
☐ Region
☐ Household
☐ Community
☐ School/other educational institution
☐ Company
☐ Hospital
☐ Other, please specify:
Sites


Number of intervention sites , please specify:
Number of control sites, please specify:

Characteristics of sites, please specify (detailed description if
available):

Likely contamination between sites:
☐ Yes
☐ No
☐ Unclear

Comparability of intervention and control sites according to
baseline characteristics
☐ Yes
☐ No
☐ Unclear (no baseline assessment)
Timing of intervention


Time point of intervention (when), please specify:
Time period of intervention (how long), please specify:
Time periods before and after
intervention
Study group:
 Before intervention, please specify:
 After intervention, please specify (including follow-up
measurements):
Data collection
Outcome assessment
☐ Non-validated subjective assessment (e.g. recall of diet)
☐ Validated subjective assessment (e.g. recall of diet through validated
questionnaire)
☐ Non-validated objective assessment (e.g. observed/weighed food
intake)
☐ Validated objective assessment (e.g. observed/ weighed food intake
with validated protocol)
Data Analysis
Statistical analysis


Unit of analysis
Controls
Please specify:
Adjustment for confounding
☐ Yes  Methods (specify):
☐ No
 Adjustment for time trends
☐ Yes  Methods (specify):
☐ No
 Adjustment for attrition
☐ Yes  Methods (specify):
☐ No
☐ Country
☐ Region
☐ Household
☐ Community
☐ School/other educational institution
☐ Company
☐ Hospital
☐ Other, please specify:
 Methods used:
☐ Matching by design (specify):
☐ Matching by analysis (specify):
☐ Other (specify):
Quality (strengths and weaknesses) at
primary study level by study authors
Effect sizes of primary outcomes as
defined by study authors::

If applicable, method for correcting imbalance, please specify:



Strengths (specify):
Weaknesses (specify):
Direction of effect
☐ Positive
☐ Negative
Significance of effect
☐ Yes
☐ No
Used quality appraisal tool:
Risk of bias judgement:

Quality assessment at systematic
review level by review authors


References
AFMC n.d. Chapter 5 Assessing Evidence and Information. The Association of Faculties of Medicine of
Canada Public Health Educators’ Network,.
BALLINI, L., BERO, L., ECCLES, M., GRIMSHAW, J., GRUEN, R. & LEWIN, S. 2011. About The Cochrane
Collaboration (Cochrane Review Groups (CRGs)). [Online].
COCHRANE BIAS METHODS GROUP. 2013. Research Projects:Cochrane Risk of Bias tool A. Extending
the Cochrane Risk of Bias tool to assess risk of bias in randomised trials with non-parallelgroup designs, and non-randomised studies
[Online]. The Cochrane Collaboration. Available: http://bmg.cochrane.org/researchprojectscochrane-risk-bias-tool [Accessed March 2015].
CRAIG, P., DIEPPE, P., MACINTYRE, S., MICHIE, S., NAZARETH, I., PETTICREW, M. & MEDICAL
RESEARCH COUNCIL, G. 2008. Developing and evaluating complex interventions: the new
Medical Research Council guidance. BMJ, 337, a1655.
DEEKS, J. J., DINNES, J., D'AMICO, R., SOWDEN, A. J., SAKAROVITCH, C., SONG, F., PETTICREW, M.,
ALTMAN, D. G., INTERNATIONAL STROKE TRIAL COLLABORATIVE, G. & EUROPEAN CAROTID
SURGERY TRIAL COLLABORATIVE, G. 2003. Evaluating non-randomised intervention studies.
Health Technol Assess, 7, iii-x, 1-173.
FRETHEIM, A., SOUMERAI, S. B., ZHANG, F., OXMAN, A. D. & ROSS-DEGNAN, D. 2013. Interrupted
time-series analysis yielded an effect estimate concordant with the cluster-randomized
controlled trial result. J Clin Epidemiol, 66, 883-7.
FRETHEIM, A., ZHANG, F., ROSS-DEGNAN, D., OXMAN, A. D., CHEYNE, H., FOY, R., GOODACRE, S.,
HERRIN, J., KERSE, N., MCKINLAY, R. J., WRIGHT, A. & SOUMERAI, S. B. 2015. A reanalysis of
cluster randomized trials showed interrupted time-series studies were valuable in health
system evaluation. J Clin Epidemiol, 68, 324-33.
GLENTON, C., LEWIN, S., MAYHEW, A., SCHEEL, I. & ODGAARD-JENSEN, J. 2013. Nonrandomized
studies are not always found even when selection criteria for health systems intervention
reviews include them: a methodological study. J Clin Epidemiol, 66, 367-70.
GRIMSHAW, J., CAMPBELL, M., ECCLES, M. & STEEN, N. 2000. Experimental and quasi-experimental
designs for evaluating guideline implementation strategies. Fam Pract, 17 Suppl 1, S11-6.
HARTLING, L., BOND, K., SANTAGUIDA, P. L., VISWANATHAN, M. & DRYDEN, D. M. 2011. Testing a
tool for the classification of study designs in systematic reviews of interventions and
exposures showed moderate reliability and low accuracy. J Clin Epidemiol, 64, 861-71.
HIGGINS, J. & GREEN, S. 2011. Cochrane Handbook for systematic reviews of interventions version
5.1.0.: The Cochrane Collaboration.
HIGGINS, J. P. T., RAMSAY, C., REEVES, B. C., DEEKS, J. J., SHEA, B., VALENTINE, J. C., TUGWELL, P. &
WELLS, G. 2013. Issues relating to study design and risk of bias when including nonrandomized studies in systematic reviews on the effects of interventions. Research Synthesis
Methods, 12-25.
IJAZ, S., VERBEEK, J. H., MISCHKE, C. & RUOTSALAINEN, J. 2014. Inclusion of nonrandomized studies
in Cochrane systematic reviews was found to be in need of improvement. J Clin Epidemiol,
67, 645-53.
NORRIS, S. L. & ATKINS, D. 2005. Challenges in using nonrandomized studies in systematic reviews of
treatment interventions. Ann Intern Med, 142, 1112-9.
NORWEGIAN SATELLITE OF THE COCHRANE EFFECTIVE PRACTICE AND ORGANISATION OF CARE
GROUP. 2014. What study designs should be included in an EPOC review and what should
they be called ? [Online]. Available:
http://epocoslo.cochrane.org/sites/epocoslo.cochrane.org/files/uploads/What%20study%20
designs%20should%20be%20included%20in%20an%20EPOC%20reviews%20and%20what%2
0should%20they%20be%20called.pdf.
OLIVER, S. R., BAGNALL, A., THOMAS, J., SHEPHERD J, SOWDEN A, WHITE I, DINNES J, REES R,
COLQUITT J, OLIVER K & Z, G. 2008. RCTs for policy interventions? A review of reviews and
meta-regression [Online]. Birmingham: University of Birmingham.
OXFORD CENTRE FOR EVIDENCE-BASED MEDICINE. Levels of Evidence (March 2009) [Online].
Available: http://www.cebm.net/oxford-centre-evidence-based-medicine-levels-evidencemarch-2009/.
PEINEMANN, F., TUSHABE, D. A. & KLEIJNEN, J. 2013. Using multiple types of studies in systematic
reviews of health care interventions--a systematic review. PLoS One, 8, e85035.
RAMSAY, C. R., MATOWE, L., GRILLI, R., GRIMSHAW, J. M. & THOMAS, R. E. 2003. Interrupted time
series designs in health technology assessment: lessons from two systematic reviews of
behavior change strategies. Int J Technol Assess Health Care, 19, 613-23.
REEVES BC, D. J., HIGGINS JPT, ET AL. [SEPTEMBER ] 2008. Including non-randomized studies. ,
Chichester, UK, John Wiley & Sons, Ltd.
REICHARDT, C. S. & HENRY, G. T. 2012. APA handbook of research methods in psychology,
Washington, DC, US, American Psychological Association.
ROCKERS, P. C., FEIGL, A. B., ROTTINGEN, J. A., FRETHEIM, A., DE FERRANTI, D., LAVIS, J. N.,
MELBERG, H. O. & BARNIGHAUSEN, T. 2012. Study-design selection criteria in systematic
reviews of effectiveness of health systems interventions and reforms: A meta-review. Health
Policy, 104, 206-14.
SANSON-FISHER, R. W., D'ESTE, C. A., CAREY, M. L., NOBLE, N. & PAUL, C. L. 2014. Evaluation of
systems-oriented public health interventions: alternative research designs. Annu Rev Public
Health, 35, 9-27.
SHADISH W, COOK T & D, C. 2002. Experiments and generalized causal inference. Experimental and
quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin
Company.
VALENTINE, J. C. & THOMPSON, S. G. 2013. Issues relating to confounding and meta-analysis when
including non-randomoized studies in systematic reviews on the effects of interventions.
Research Synthesis Methods, 4, 26-35.
Appendix A: Definitions
Types of intervention
Behavioural / educational intervention (behav):
The intervention is targeted at changing behaviour of an individual or population through information or
education, for example, an information campaign aimed to decrease smoking rates. Behavioural interventions
in the workplace, however, would be categorized as occupational interventions.
Clinical intervention (clin):
The intervention is targeted at improving health through treatment, where treatment usually comprises
medical procedures (e.g. diagnosis and any form of treatment exceeding pure drug treatment) and is
administered in a healthcare setting. For example, a new surgery to treat an eye disease.
Pharmaceutical intervention (pharm):
The intervention is targeted at improving health through pharmaceuticals only, for example, ACC-treatment for
the reduction of hypertension in pregnancy.
Environmental intervention (env):
Interventions that are targeted at the environment to improve the health of a population, for example, the
introduction of traffic lights or seatbelt regulation to prevent car accidents.
Occupational intervention (occ):
Interventions in the occupational setting and/or that are targeted at employees, for example, interventions to
reduce air pollution at the workplace to decrease occupational asthma. Interventions in the healthcare setting
are excluded and would be classified under health systems intervention.
Health policy intervention (hp):
Interventions that involve laws, regulations or policies with the primary goal to change health and are decided,
implemented or usually funded by the health sector, for example, the introduction of a policy to control the
costs of pharmaceuticals.
Health systems intervention (hs):
Interventions that are targeted at healthcare institutions or the health system, for example, the introduction of
a new supervisory health staff to improve the treatment and care of community healthcare centres.
Nutrition intervention (nutr):
Interventions that are specifically targeted at modifying the nutrition of people, for example, the vitamin A
supplementation in pregnant women. If the intervention also includes a behavioural element, for example
aimed at changing the diet of a person, this intervention would still be classified as a nutrition intervention.
Vaccination intervention (vacc):
Interventions that are targeted specifically at improving health through vaccinations, for example, the
introduction of HPV vaccination for girls to reduce the incidence of cervical cancer. If the intervention also
includes a behavioural element, for example to increase the motivation of persons to get vaccinated, this
intervention would still be classified as a vaccination intervention.
Screening intervention (screen):
Interventions entailing screening to improve the health of a population, for example, the screening of colon
cancer in men over 50. If the intervention is also another type of intervention, for example a health policy
intervention, changing the policy to make colon cancer screening obligatory to all men over 50, this would be
classified as a screening intervention.
Levels of intervention
Individual (ind):
The intervention is targeted at individuals, such as patients or high-risk groups, and the intervention is
conducted on the individual level.
Organisational/group (org):
An intervention aimed at a specific group or cluster of individuals, and conducted on group/cluster level, such
as schools, the workplace or healthcare institutions.
Large scale / population (pop):
An intervention that is targeted at the general population, such as a law or regulation, or targeted at a largescale population, such as a screening intervention for a broad target group or a nutrition media campaign
targeting adolescents.
Appendix B: Cochrane Group Abbreviations
ARI CRG
Air CRG
Anaesth CRG
Back CRG
BJMT CRG
BC CRG
ChildC CRG
ColoC CRG
C&C CRG
CFGD CRG
DCI CRG
DAN CRG
DPLP CRG
DA CRG
ENT CRG
EPOC CRG
Epil CRG
EV CRG
FR CRG
GC CRG
HM CRG
Heart CRG
HB CRG
HIV CRG
Hypn CRG
Incont CRG
Infec CRG
Inflam CRG
Injuries CRG
LC CRG
MDS CRG
MED CRG
MR CRG
MD CRG
MS&RDCNS CRG
Musc CRG
Neo CRG
ND CRG
OSH CRG
OH CRG
PPSC CRG
PVD CRG
PC CRG
PDUC CRG
PH CRG
Renal CRG
Schiz CRG
STI CRG
Skin CRG
Stroke CRG
TA CRG
UGPD CRG
Wounds CRG
Acute Respiratory Infections Group
Airways Group
Anaesthesia Group
Back Group
Bone, Joint and Muscle Trauma Group
Breast Cancer Group
Childhood Cancer Group
Colorectal Cancer Group
Consumers and Communication Group
Cystic Fibrosis and Genetic Disorders Group
Dementia and Cognitive Improvement Group
Depression, Anxiety and Neurosis Group
Developmental, Psychosocial and Learning Problems Group
Drugs and Alcohol Group
Ear, Nose and Throat Disorders Group
Effective Practice and Organisation of Care Group
Epilepsy Group
Eyes and Vision Group
Fertility Regulation Group
Gynaecological Cancer Group
Haematological Malignancies Group
Heart Group
Hepato-Biliary Group
HIV/AIDS Group
Hypertension Group
Incontinence Group
Infectious Diseases Group
Inflammatory Bowel Disease and Functional Bowel Disorders Group
Injuries Group
Lung Cancer Group
Menstrual Disorders and Subfertility Group
Metabolic and Endocrine Disorders Group
Methodology Review Group
Movement Disorders Group
Multiple Sclerosis and Rare Diseases of the Central Nervous System Group
Musculoskeletal Group
Neonatal Group
Neuromuscular Disease Group
Occupational Safety and Health Group
Oral Health Group
Pain, Palliative and Supportive Care Group
Peripheral Vascular Diseases Group
Pregnancy and Childbirth Group
Prostatic Diseases and Urologic Cancers Group
Public Health Group
Renal Group
Schizophrenia Group
Sexually Transmitted Infections Group
Skin Group
Stroke Group
Tobacco Addiction Group
Upper Gastrointestinal and Pancreatic Diseases Group
Wounds Group