Realist Synthesis 101
Dr Andrew Booth (with
acknowledgements to Dr Janet
Harris)
Illustrating a Realist Review approach
• For Which Schools Do School Performance
League Tables Work?
• How Do School Performance League Tables
Work?
• Do they work in a similar way to Hospital
Performance League Tables?
• What about Individual Surgeon Performance League
Tables?
Developing Programme Theory
• Programme Theory #1: Proponents of performance league tables believe
their publication stimulates competition,
• Programme Theory #2: and that, as each provider adopts “best practice,”
the quality of services will improve.6
• Supporting Data: Some published evidence seems to support this. New
York probably has the best established system for providing the public
with information… Soon after publication of performance league tables
based on providers, the risk adjusted mortality for bypass surgery
declined, leading some to conclude that this was a direct result.7
• Counter Argument: One alternative explanation being that, once
providers know their data will be used for comparative purposes, they
may resort to “creative reporting.” 8 9
• Programme Theory #3: Performance league tables may also improve
patients' choice, and proponents argue that this is necessary for an
efficient market economy by encouraging consumers to seek out high
ranking providers.9
Developing Propositional Statements
(IF-THENs)
• IF Performance Tables are published THEN:
– Staff at Well-performing schools (Context1) will change
their reasoning/actions because (Mechanisms1) (e.g.
Maintain competitive advantage)…..with the result
(Outcome1) that……
– Staff at Average-performing schools (Context2) will change
their reasoning/actions because (Mechanisms2) (e.g.
Identification of good practice)…..with the result
(Outcome1) that…….
– Staff at Poor-performing schools (Context3) will change
their reasoning/actions because (Mechanisms3) (e.g.
Shame)……with the result (Outcome1) that……..
• CMO configurations where Mechanisms = changes that
occur in people's reasoning and actions
Realist synthesis
Context
Altered context
Context triggers
and Interacts
with
Mechanisms
Mechanisms
Outcomes (intended and unintended)
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Formulating the review question
Formulating the review question
• Realist questions aim to identify diverse literature on
the intervention of interest, in order to evaluate what
works for whom in what circumstances
• They are relatively open ended, aiming to capture a
broad spectrum of relevant primary research in the
first instance
• They aim to identify the relationships between
Intervention, context, mechanisms and outcomes
Our realist questions
(1) What approaches to community engagement are most
effective in promoting peer support, to which people
and in what circumstances?
• (2) How does community-based peer support impact on
understanding of existing health information and use of
health information and health services to improve
health and reduce health inequalities?
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Preliminary theory for COPES
• Community engagement aiming to promote
peer support
• increases health literacy
• improves health
• reduces health inequalities
Assumptions
• Communities initiate and foster peer support
• The process of providing community-based peer support influences
the ability to develop health literacy
Community
Peer support
Health literacy
• Health literacy creates a sense of empowerment, which is
associated with a reduction in health inequalities
• Community-based peer support raises awareness of issues with
literacy and empowerment
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Scoping the literature
• For the initial scoping, do a comprehensive search to
‘map the territory’
• Abstract sift: Identify and categorise abstracts by
programme theory, type of intervention
Scoping the literature for key concepts
• Community engagement: Very small number of articles documenting
‘working collaboratively with and through’ community engagement for
peer support
• There were variations in community-based peer support in terms of
• Amount of ‘emotional, appraisal and informational
assistance’
• Degree of ‘created social network member’
• Degree of ‘experiential knowledge of a specific
behaviour or stressor’
• Definitions of ‘Similar characteristics’
• The term health literacy was not used in peer support articles
• so included all articles on health education/health
promotion relating to ‘Understanding and use of
health information and health services’
• Reduction of health inequalities – or any explicit discussion of health
inequalities – was absent
Identifying sources of information in real life settings
Health
librarian
Qualitative
researchers
Systematic
reviewers
CBPR
researchers
Propose a model
Test the emerging theory
Identify what works
Explore gaps
Practitioners
Community
organizations
Peer
supporters
Expert patients
Compare definitions
Define concepts
Network with participants
Research knowledge
Scope the literature
Community knowledge
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
Formulate review question
Scope the primary sources of information
Decide on the scope of the review
Map the programme theory(s)
Identify programme components
Assess effectiveness of components, considering context and
mechanisms
7. Develop propositional statements
8. Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in
what circumstances and at what point in time (mid-range theory)
Defining key concepts:
community engagement and peer support
• Community engagement is ‘the process of working collaboratively with
and through groups of people affiliated by geographic proximity, special
interest, or similar situations to address issues affecting the well-being of
those people…It often involves partnerships and coalitions that help
mobilize resources and influence systems, change relationships among
partners, and serve as catalysts for changing policies, programs, and
practices (CDC, 1997)
• Peer support within a healthcare context is “...the provision of emotional,
appraisal and informational assistance by a created social network member
who possesses experiential knowledge of a specific behaviour or stressor
and similar characteristics as the target population.” (Dennis, 2003).
Emotional support includes expressions of caring, empathy, and
encouragement, and is seen to enhance self-esteem. Appraisal support
involves encouraging persistence and optimism for resolving problems,
affirmation of a peer’s feelings, and reassurance that frustrations can be
handled. Informational support is providing advice, suggestions, alternative
actions, feedback and information relevant to the issue with which the
peer is dealing.
Defining key concepts: health literacy
• Health literacy – ‘the personal, cognitive and social
skills which determine the ability of individuals to gain
access to, understand, and use information to promote
and maintain good health. These include such
outcomes as improved knowledge and understanding of
health determinants, and changed attitudes and
motivations in relation to health behaviour, as well as
improved self-efficacy in relation to defined tasks.
Typically these are outcomes related to health
education activities…’ (Nutbeam, 2001)
• From a number of definitions for health literacy – we
chose one that related to the notion of empowerment
because it was relevant to health inequalities
Improved health outcomes,
healthy choices
and opportunities
Engagement in
social
action/advocacy
for health
Skills in
social organization
and advocacy
Changed health
behaviours and
practices
Improved
Health Literacy
Developed
knowledge
and capability
Participation in
changing social
norms and service
practices
Skills in
negotiation and
self management
Tailored information,
communication, education
Prior understanding of individual
capacity - reading fluency, numeracy,
existing knowledge
Figure 3: Developing interactive and critical health literacy skills
(Nutbeam, 2009)
Health literacy model proposed by Sorensen et al (2012)
• A year into our review, a conceptual analysis of the definitions
was conducted, and a unifying model was produced
(Sorensen et al, 2011). This model focuses on the capabilities
of accessing, understanding, appraising and applying health
information.
Decide on the
scope of the review
Me
Team
World
World
• Things to consider:
– Amount of literature: Comprehensive versus sampling?
– Diversity of literature: Consensus on programme theory?
Variations on theory by what, who, when, setting
– Access to other sources of information
– Interests of those who commissioned the review
•
•
•
•
What was the original problem that needs research evidence?
International, national, regional, local?
What will the findings be used for?
Who is the audience? (this is sometimes wider than just
commissioners)
– Time and money
– Experience
Setting the parameters via sampling
•
Programme theory was not used for sampling because
– Theory was only explicitly cited in 28 instances
– Studies combined elements from the Heath Belief Model,
Theory of Planned Behaviour, social cognitive and social
learning theories, the social support literature, social
comparison theory, social network approaches, and
empowerment models
– Studies used both individual behaviour change and
community-based theories, including community
organization, diffusion of innovation, and social marketing
– Domains for all the individual behaviour change theories
are included in the Capabilities Opportunity Motivation
Behaviour theory (COM-B, Michie et al 2011)
•
Pragmatic sampling on location: the most common United Kingdom health
conditions in the peer support literature were reviewed first
Mapping literature topics to UK settings
TOPIC
Advisory Network
Breastfeeding
Sheffield Well Being Consortium; Sheffield
Breastfeeding study
Darnall Wellbeing
AGE UK; Stroke Association
Irish Gypsy & Travellers Movement in Britain;
Leicester Health Ambassadors
Centre for HIV & Sexual Health; Parent to Parent,
Shout; Sheena Amos Trust; Expert Patients
Programme
Sheffield Community Chronic Pain; Manor & Castle
Development Trust
MECOPP Minority Ethnic Carers of People Project;
Autism Plus
Sharrow ShipShape; Sheffield PCT
MIND; Mencap
Sheffield City Council
SOAR (Southey Owlerton Area Regeneration)
ZEST Community Development Trust
Diabetes
Healthy Living Older People
HIV/AIDS
Health Trainers
Nutrition
Smoking
Summary of searching, sifting and sampling
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Identify the programme components
• Programmes categorised by Stage
– Recruitment of peers
– Training and supporting peers
– Implementing peer support - what was
delivered
– Outcomes
• Within each Stage, components were
identified in the literature and through
workshops with community members
• Components put together in CMO
configurations by the topic leads
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Mapping programme theories
• The theories and assumptions were documented
using Context Mechanism Outcome (CMO)
configurations for each health topic
Type of
finding
authors'
thoughts
Ref ID
2668
What works
well/not well
CONTEXT
Retaining peer supporters
Amongst those consented to be
peer supporters attrition rates
were low 87% of those
nominated agreed to become
peer supporters and 82%
completed the trial
study
protocol
6895 &
LC001
Retaining PS
study
protocol
6895
High ratio trainer :
student
£10 voucher and certificate of
achievement offered to PS who
completed the trial
maximum of 1:15 trainers to
students
MECHANISM
PS felt pride in being nominated by
their peers. It was novel to be taken
out of school for training days and to
miss lessons to attend follow up
sessions and they may have been
committed to reducing smoking
amongst their peers, also incentive
offered to those who completed trial
OUTCOME
The target of 15% of the cohort were
trained and delivered the
intervention to the end of the trial
period
PS responded to this incentive and
remained committed to trial
Low attrition and majority of PS
completed trial
Group work easily facilitated and
trainers developed good
relationships with peer supporters.
Role play activities were successful
Students motivated throughout
training and follow up visits. Role
play helped students anticipate the
types of different reaction they might
receive when attempting to
disseminate information
Dealing with diverse theories
• Some topics used a single theory, for example Diffusion of
Innovation
• Other topics did not explicitly state how theory informed
design, but contained implicit assumptions about what would
work
• Concepts from both stated/implicit theories identified via
CMOs
• Dimensions of concepts for one topic used to question
concepts in another topic area
• For example, ‘recognising someone as a peer’ was an
important mechanism. We asked ‘what does
recognising someone as a peer mean across different
health topics?
• The dimensions we discovered included age, ethnicity,
experience with the problem or condition
• We also used Advisory Network as a source of propositional
statements
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and
mechanisms
Iterative
7. Develop propositional statements
8. Identify patterns in propositional statements across cases
(demi-regularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for
whom, in what circumstances and at what point in time (midrange theory)
Turning CMOs into propositional statements: analysis
of the literature
Overarching
propositional
statement
IF target
community
recognise PS as
their peer
THEN PS is
effective
Supporting data by topic
Smoking cessation
young people
Healthy living in older
people
Smoking in ethnic
minorities
I 5/6:
Target community
must recognise the
peer supporters as
their peers.
(Propositional
statement)
I12:
When age is only
common characteristic,
and differences like
ethnicity exist (meaning
potentially different
needs and value systems),
the peers may have a
negative experience.
(Propositional statement)
D88
Smoking cessation
service is offered by
health professionals,
who created a formal
and impersonal
atmosphere, but
smokers feel a sense of
distrust towards
providers
(Mechanism)
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Searching for related theories
• CMOs and propositional statements (IF-THENs)
considered in terms of higher order concepts, for
example
Overarching propositional statements
Concepts
IF the target community recognise the PS as their peer
THEN peer support is effective
Social learning
Relational learning
IF there is a positive relationship between the PS and the
peers (trust and a sense that the person is believable,
credible) THEN peer support is effective
Participatory parity
Trust
Steps for conducting a realist synthesis
1.
2.
3.
4.
5.
6.
7.
8.
Formulate review question
Scope the primary literature
Decide on the scope of the review
Identify the programme components
Map the programme theory(s)
Assess effectiveness of components, considering context and mechanisms
Develop propositional statements
Identify patterns in propositional statements across cases (demiregularities)
9. Search sources for theories related to the propositions
10. Produce a set of summary statements on what works for whom, in what
circumstances and at what point in time (mid-range theory)
Mid-range theory
• ‘Theories that lie between the minor but necessary working hypotheses
that evolve in abundance during day-to-day research and the all-inclusive
systematic efforts to develop a unified theory that will explain all the
observed uniformities of social behaviour, social organization and social
change.’ (Merton, 1967: 39)
Mid-range theory
• Mid-range theories relate empirical
data to higher order constructs, and
are ‘sufficiently abstract to deal with
different spheres of social behaviour
and social structure, so that they
transcend sheer description or
empirical generalisation’ (Merton,
1967: 68).
• Middle range theory involves
abstraction, of course, but…close
enough to observed data to be
incorporated in propositions that
permit empirical testing.”
References
Dennis, C.-L. (2003). Peer support within a health care context: a concept analysis.
International journal of nursing studies, 40(3), 321–32.
Kincheloe, J. and McLaren. Rethinking theory and research. (2005) In NK Denzin & YS
Lincoln (eds) Sage Handbook of Qualitative Research.
Merton R. On Theoretical Sociology. Five Essays, Old and New. New York: The Free
Press, 1967
Nutbeam, D. (2008). The evolving concept of health literacy. Social science & medicine
(1982), 67(12), 2072–8.
Pawson, R. (2013) the science of evaluation: A realist manifesto. London: Sage.
Sørensen, K., Van den Broucke, S., Fullam, J., Doyle, G., Pelikan, J., Slonska, Z., & Brand,
H. (2012). Health literacy and public health: a systematic review and integration of
definitions and models. BMC public health, 12(1), 80.
Wong, G., Greenhalgh, T., Westhorp, G., Buckingham, J., & Pawson, R. (2013). RAMESES
publication standards: realist syntheses. BMC medicine, 11(1), 21. doi:10.1186/17417015-11-21
Acknowledgements
• Funded by the National Institute of Health Research
Public Health programme PHR - 09/3008/04:
Community-based peer support: A realist synthesis
and model for promoting health literacy (COPES)
• Published as: Harris J, Springett J, Croot L, Booth A,
Campbell F, Thompson J, Goyder E, Van Cleemput P,
Wilkins E, Yang Y. Can community-based peer support
promote health literacy and reduce inequalities? A
realist review. Southampton (UK): NIHR Journals
Library; 2015 Feb.