A Brief Introduction to Realist
Evaluation and Realist Synthesis
(and implications for work protection!)
Ray Pawson
Helsinki 2009
Realism’s place on the evaluation tree
Rossi & Chen
Weiss
Theories
of change
Theorydriven
evaluation
Realist
evaluation
Connell et al
Pawson &
Tilley
Mark, Henry
& Julnes
Theory-driven evaluation
in a nutshell
~ Policies and programmes are ‘theories’
~ Evaluation is the process of testing and
refining those theories
~ The overall purpose is explanatory –
answering the question ‘what works for whom in
what circumstances and in what respects and over
what duration etc etc.
A health education programme from the 90s
• Girls are rather sedentary and spend too much
time mooching around in bedroom culture …
• Here and via the influence of girls magazines
they come under a range of unwholesome,
unhealthy influences …
• Their role models being film stars, soap starts,
rock stars etc. …
• What if we can persuade the editors of these
magazines to pursue a ‘fitness agenda’ and use
fit young blokes (i.e. sports stars!) as the role
models …
… And so ‘Dishy David Beckham
Theory’ was born
Evaluation is theory testing
‘Interviewer: But do you think the fact that
these good-looking blokes are footballers
has any effect on girls' attitude to playing
football?
Girl: No, I think it has more effect on them
watching football, well not the football - the
guys (general laughter and agreement)’.
Mitchell K (1997) ‘Encouraging young women to exercise: can teenage
magazines play a role?’ Health Education Journal 56(2) pp. 264-273
Where to start
1. Start with a careful interrogation of the programme
theory.
2. Theory elicitation may itself be a crucial part of the
method.
3. Search for hypotheses, testable propositions and
formalise them.
4. Programmes are complex adaptive systems. There will
always be more theories available than you can test.
Prioritisation is a must.
5. Anything goes. RE/RS makes use of any form of data
(it favours neither the quantitative or qualitative, primary
or secondary etc.)
Two very broad strategies
1. SUB-GROUP DESIGNS
Hypotheses: the
intervention works
for X, Y, Z
Findings: The
intervention works for
X1, X2, Z1 and Z4
… And the other one
2. PROGRAMME THEORY - ‘flows and blockages’
Hypothesis: Programme theory posits a sequence of
mechanisms
Findings: Empirical research indicates significant unintended
consequences
Example 1 – prisoner education programme
The Simon Fraser Campus-in-a-Prison program
Can a full-time, liberal-arts, university-level education
program in a medium/maximum security prison slow the
revolving door of recidivism and re-incarceration?
In which we discover
the massive, and
entirely typical,
outcome differences
across sub-groups of
recipients.
Theory elicitation again
“ The men who are more likely to be changed are best described as
‘mediocre’. You shouldn’t look for high-flyers. They’re likely to come
from a deprived background with a poor and maybe non-existent
school record. They will be mediocre criminals too. They’ll have gone
on from petty crime, street crime to drugs or armed robbery or
something. Then when they come onto the program, they’re mediocre
or worse. They just survive the first semester but gradually they build
up getting C’s and B’s. So by the end, they’ve actually come a long,
long way. And that’s what changes ‘em. It’s not so much a case of
‘rehabilitation’ as ‘habilitation’.”
Generating the outcomes data
Track prison and
background
records
Reconviction
prediction score
Track prison
education and post
release records
Actual reconviction
record
Explaining the outcome pattern
Programme outcomes are always complex. M & C theories are tested
by examining outcomes for different sub-groups of subjects
Who benefited most? Audience
participation
Improved rehabilitation by age
17-21 22-25 26-30 31-35 36over
1st
4th
5th
2nd
3rd
Improved rehabilitation by prior education
Grade 10 Grade11 Grade 12 Post-Secondary
2nd
1st
3rd
4th
Data in a bit more detail – note the
differences in the differences!
Age at current conviction – post release success
Subgroups
Predicted
Actual
Difference
17-21 y.o.
52
88
36
22-26 y.o.
54
60
6
26-30 y.o.
55
64
9
31-35 y.o.
57
87
30
35 + y.o.
65
84
19
Example 2: Youth Mentoring Programmes
What goes on?
Helping, coaching, tutoring, counselling,
sponsoring, role modelling, befriending, bonding,
trusting, mutual learning, direction setting,
progress chasing, sharing experience, providing
respite, sharing a laugh, widening horizons,
building resilience, showing ropes, informal
apprenticeships, providing openings, kindness of
strangers, sitting by Nellie, treats for bad boys
and girls, the Caligula phenomenon, power play,
tours of middle class life, etc. etc.
A primitive theory for ‘engagement’ mentoring
Mentoring as
‘sponsoring’
Mentoring as ‘coaching’
Mentoring as ‘direction setting’
Mentoring as ‘befriending’
Study 1: De Anda (2001)
Joe had been raised in a very chaotic household with his mother as the
primary parent, his father’s presence erratic… He was clearly
heading towards greater gang involvement… He had, in fact, begun
drinking (with a breakfast consisting of a beer), demonstrated little
interest in school and was often truant… The Mentor Program and
the Captain who became his mentor were ideal for Joe, who had
earlier expressed a desire to become a firefighter. The mentor not
only served as a professional role model, but provided the nurturing
father figure missing from his life. Besides spending time together
socially, his mentor helped him train, prepare and discipline himself
for the Fire Examiners test. Joe was one of the few who passed the
test (which is the same as the physical test given to firefighters). A
change in attitude, perception of his life, and attitudes and life goals
was evident…[further long, long story omitted] He also enrolled at
the local junior college in classes (e.g. for paramedics) to prepare for
the firefighters’ examination and entry into the firefighters academy.
He was subsequently admitted to the fire department as a trainee.
Study 2: Colley (2003)
Adrian spoke about his experience of mentoring with evangelical fervour:
‘To be honest, I think anyone who’s in my position with meeting people,
being around people even, I think a mentor is one of the greatest things
you can have… If I wouldn’t have had Pat, I think I’d still have problems
at home… You know, she’s put my life in a whole different perspective.
Adrian was sacked from the scheme after 13 weeks. He was placed in an
office as filing clerk and dismissed because of lateness and absence.
Colley reports that, despite his profuse excuses, the staff felt he was
‘swinging the lead’. Pat (the mentor) figured otherwise: ‘Pat, a former
personnel manager and now student teacher, was concerned that
Adrian had unidentified learning difficulties that were causing him to
miss work though fear of getting things wrong. She tried to advocate on
his behalf with New Beginnings staff, to no avail.’ At this point Adrian
was removed from the scheme.
Study 3: Rhodes et al (2003)
Path model of reported effects of mentoring
school
achievement
.26
.25
.08
mentoring
.22
quality of
parental
relationship
grades
.25
self
worth
.22
-18
- .09
.26
value of
school
skipping
school
- .11
-.26
Study 4: Grossman (1998)
“Taken together, the results presented here show that having a Big
Brother or a Big Sister offers tangible benefits for youth. We found that
Little Brothers and Little Sisters were less likely to have started using
drugs or taking alcohol, felt more competent about doing school work,
attended school more, got better grades, and have better relationships
with their parents and peers [than non participants] ”
BUT on the BB/BS there are:
Long waiting lists
Screening for entry into the programme
Screening for entry into the research
Self report on outcomes (conducted by telephone survey)
Extensive infrastructure and programme repute
Questions on researcher independence?
Service Delivery: Pathways of youth mentoring
honestbroking
advocacy
coaching
direction
confidence
building
resilience
building
setting
befriending
Work &
Careers
troubleshooting
Education &
Training
fencemending
Youth and
Community
trust
building
Family &
Peers
peacemaking
Example 3:
Banning smoking in
cars carrying
children as
passengers
Can one review a
policy proposal?
PLATO “Good
people do not
need laws to tell
them to act
responsibly,
while bad people
will find a way
around them.”
Public Health
Law in ‘theory’
Behaviours leading
to poor health
1. DRAFTING
Implementation Chain
(Logic Model)
Proscription of unhealthy /
unsafe activities. Prescription
of healthy / safe behaviour
2. ENACTMENT
Penalties / threat of penalties
for failures to comply
3. ENFORCEMENT
Behavioural accommodation
1. Problem Misidentification
DRAFTING
‘Threats’ to
legislation
2. Criminalisation
3. Compensating / Displaced Behaviour
4. Lack of Public Support
ENACTMENT
5. Lobby Group Opposition
6. Obfuscating the New Regulations
ENFORCEMENT
7. Low Perceived Threat of Enforcement
8. Insufficient Enforcement Resources
Smoking in cars carrying children –
three evidence nuggets
Note:
• Original evidence comes from a wide
variety of studies using a wide range of
research methods.
• Quite different bodies of data are used to
test the respective theories
• In some cases evidence is drawn from
‘parallel’ interventions to the one under
review
Theory one: correct problem identification?
• Evidence drawn from toxicity data from air
quality monitors.
• Good evidence on high levels of toxic pollutants
(c.f. smoky pubs)
• More contested evidence on:
i) efficacy of ventilation countermeasures
ii) exposure risks – is there a dose / response
relationships?
iii) comparable risks – home, work, leisure,
transport (and cars)
Theory two – lobby group opposition?
•
Evidence drawn from media reports, tobacco
company outputs, existing reviews.
‘Unlike all other advocacy for smoke free areas,
this debate was not contested by the tobacco
industry or other groups motivated by the
potential to see restrictions reduce sales and
further denormalise use. Indeed, one tobacco
company was supportive of the legislation’.
[Freeman et al, 2007]
Theory three – enforcement difficulties?
• Few applications, fewer studies and no direct
evidence on enforcing such a ban
• Use parallel interventions and studies (on other
hard-to-enforce, in-car bans such as for mobile
phones and seat belts):
i) Evidence of short term compliance and then
fade
ii) Evidence that compliance level may be
boosted by a) publicity, b) crackdowns, c) public
surveillance
And Finally
THANK
YOU
FOR
YOUR
ATTENTION