Complexity
Co
ple ty a
and
d evidence
ev de ce
synthesis:
a complex
l methodological
h d l i l
challenge!
Jane Noyes
Bangor University
Lead Convenor Cochrane Qualitative and
Implementation
l
Methods
h d Group
Methodological
g
Investigation
g
of Cochrane
Reviews of Complex Interventions (MICCI)
Acknowledge fellow methodologists who met at Montebello
January 2012
Key
y methodological
g
issues
What do we mean by a systematic review of complex interventions?
•
Simple questions can be asked of complex interventions
•
C
Complex
l questions
ti
can b
be asked
k d off non-complex
l interventions
i t
ti
•
Complex questions can be asked of complex interventions
• Purpose of the ‘complex’ review must be to explore ‘complexity’
rather than just ‘effect’
• Question development: PICO is too restricting when the purpose of
the review is to explore complexity
Slum upgrading strategies involving physical
environment and infrastructure interventions
and their effects on health and sociosocio
economic outcomes.
Turley R, Saith R, Bhan N, Rehfuess E, Carter B.
What do we mean byy complexity?
p
y
• Complex intervention (synthesis of all published definitions)
• Intervention complexity (i.e. situation in which we expect the effect of an
intervention to be modified by variant properties or characteristics of the
intervention itself);
• Implementation complexity (i.e. situation in which we expect the effects
of an intervention to be modified by variant characteristics of the
i l
implementation
t ti process))
• Context complexity (i.e. situation in which we expect the effects of an
intervention to be modified by variant properties or characteristics of the
setting or context in which an intervention is implemented); and
• Complexity in participant responses (i.e.
(i e situation in which we expect the
effects of an intervention to be modified by variant characteristic of
participants receiving the intervention).
Acknowledgement Ian Shemilt, Laurie Anderson, Susan Michie and co-authors
What type of evidence is likely to be useful in
answering complex systematic review questions?
A range of evidence – RCTs, Non randomised studies,
process evaluations, qualitative evidence
Cochrane methodology is adapting to accommodate
different types of evidence –
g development
p
of a new Chapter
p
in
Laurie and Mark leading
Cochrane handbook.
J ki Ch
Jackie
Chandler
dl – One
O off the
h Cochrane
C h
H
Handbook
db k Edi
Editors
What methods are appropriate for answering complex systematic review
questions?
Synthesis Options for Complex
Intervention Reviews
Qualitative
synthesis to
Quantitative
synthesis to
determine effects
and explore context,
heterogeneity etc.
Meta-analysis, meta
regression or
narrative summary
configure/ summarise /
integrate qualitative
data to address specific
questions on
complexity in relation to
a Cochrane
intervention review
Thematic analysis
without theory generation
E.g. Framework
synthesis
Qualitative synthesis
to develop explanatory
theory or models to
explore/explain
complexity
Meta ethnography
Meta-ethnography
Thematic analysis with
theory generation
Grounded theory
Mixed-method synthesis
to integrate and interpret
quantitative, mixed-method
and qualitative evidence within
a single approach or integrated
model to develop explanatory
theory and explore/explain
complexity
Realist review
Qualitative Case Analysis
EPPI approach
Narrative synthesis (Popay et
al)
Bayesian synthesis
Critical interpretive synthesis
Product
Pooled effect size
and/or description of
individual studies
Product
Aggregated/configured
narrative findings from
source papers
Product
Explanatory theory,
analytical or conceptual
framework or interpretative
framework/ mechanism
Product
Integrated synthesis of
quantitative and qualitative
evidence to explore/explain
complexity
Adapted from Noyes and Lewin 2011
Product
Integrated synthesis of
quantitative effects evidence
and qualitative evidence to
explore/explain
l /
l i complexity
l it
Key issue is that
the question
and
d issue
i
off
complexity
should drive the
choice of
design/methods
Reviews teams
stick with
familiar
methods rather
than the most
appropriate
pp p
method
What conceptual frameworks and tools are
available to help make sense of complexity?
MICCI project – developing a toolkit that may help
Cochrane authors. Examples include:
The BeHEMoTh Procedure
Booth et al
Guidance on identifying theory to inform
design, conduct and analysis of systematic
reviews of behaviour change interventions.
Glasziou
Gl
i
et all Taking
T ki h
healthcare
lh
interventions from trial into practice.
Guidance
G
id
on d
detailed
il d d
description
i i off an
intervention and how it is implemented
Michie S et al The behaviour change
wheel.
A framework for describing the components of
behaviour change interventions
Perera et al. Graphical method for
depicting randomised trials of complex
i t
interventions.
ti
A graphical method for depicting randomised
trials of complex interventions
What new tools are in development?
MICCI project: iCAT_SR: Intervention Complexity
Assessment Tool for Systematic Reviews
Aid to disaggregating
intervention ‘components’
and their delivery, and rating
interventions across a set of
10 ‘dimensions’ that have
b
been
identified
id tifi d as pertinent
ti
t iin
assessing their complexity
Acknowledge Simon Lewin, Andy Oxman, Jackie Chandler
and all the people who have supported iCAT_SR
development
Use of logic models
Anderson et al. 2011 Using logic models to capture
complexity in systematic review
Turley et al 2013
Slum upgrading
review:
methodological
challenges that arise
in systematic reviews
of complex
interventions
Future Research and Development agenda
Final paper in the JCE Montebello series outlines a R&D
agenda:
High level overview. Need to:
• Address how complexity is conceptualised, framed, analysed and interpreted
in systematic reviews that ask complex questions of interventions with varying
g
of complexity.
p
y
degrees
• Test and evaluate how the range of existing synthesis methods and tools is
useful for addressing questions about complexity, and to determine where
further methodological development is needed to fill important gaps.
AND
• Dimensions of complexity need to be explicitly described in primary study/
systematic review reports – an intervention complexity extension to the
CONSORT and
d PRISMA statements
t t
t would
ld help
h l with
ith standardised
t d di d reporting.
ti
Jane Noyes
[email protected]
Bangor University
Acknowledgements:
Cochrane MIF Fund.
Fund CIHR
CIHR.
All colleagues who have contributed to
the debate and methodological work
Thank you!