‘Nothing as Practical
as a Good Theory’
RAY PAWSON
2002 EES Conference
Sevilla
Tasks for the plenary lecture
The basic justification for theorydriven evaluation (a reminder)
New (lesser known) roles for
theory in evidence based policy
• Systematic Review
• Knowledge Transfer
• Complexity
• Enlightenment
Without theory there can be no evaluation
WHY?
• Evaluation tests
programmes
AND
• Programmes are
‘theories’
AND THEREFORE
• Evaluation is
‘theory-testing’
“If these resources are
provided to these people it
may change their behaviour”
Programmes are ‘theories’(I)
‘Dishy-David-Beckham-theory’
‘Interviewer: But do you think the
fact that these good-looking blokes
are footballers has any effect on
girls' attitude to playing football?
Girl: No, I think it has more effect on
them watching football, well not
the football - the guys (general
laughter and agreement)’.
Programmes are ‘theories’(II)
•
•
•
•
Megan’s Law - what is the
optimum ‘radius of notification’?
3 blocks radius (urban)
2 miles (rural)
‘eyeball’ the local map
‘we look at how far the offender
has to travel to buy cigarettes’
Evaluation is theory testing (I)
Theories-of-change strategy
‘surfaced’ theory - short term, intermediate and long-term outputs
process observations - do the anticipated steps come to fruition?
Evaluation is theory testing (II)
Realist strategy
the rare occurrence of the expected …
WILLIAM CARLOS WILLIAMS
From ‘At Kenneth Burke’s Place’
The same programme mechanism will have
different outcomes in different contexts
Let’s review the evidence
Types of Systematic Review
• Meta-analysis
• Narrative review
• Hybrid reviews
• Theory-of-change review
• Realist synthesis
www.evidencenetwork.org
Megan’s Law - Programme Theory
STEP ONE
STEP TWO
STEP THREE
STEP FOUR
Problem
Identification
Public
disclosure
Sanction
Instigation
Offender
response
Identify high-risk
released sex
offenders and
create valid and
reliable registers
Issue bulletins,
press releases,
call meeting to
identify released
offenders to
their community
Community
joins with police
and probation to
increase
surveillance of
suspicious
behaviour
Community
actions shame
offenders and
decrease
opportunity of
further offence
Evidence fragment one: could the law have made
a difference? - a ‘retrospective simulation’
136
serious
sex
offences
12
previous
offence
stranger
predatory
offences
potentially
respond to
community
notification
24
6
100 no
previous
offence
Petrosino & Petrosino
6 could
36
known to
victim
offender
from out
of state
Evidence fragment two: did the law
effect recidivism? A matched trial
Pre-intervention sample
sex recidivism 22%
Post-intervention sample
sex recidivism 19%
Pre-intervention sample
Post- intervention sample
Schram & Milloy
arrest slow
arrest significantly quicker
Evidence fragment three: how did
practitioners respond? Office talk
“The Law is an unfunded mandate”
“Special Bulletin Notification added more work to
already over-worked agents”
“There is more pressure to baby sit with SBN cases
simply because they are SBN cases”
Zevitz and Farkas
Megan’s Law - an emerging theory
• Opportunity for and
effectiveness of
surveillance by the
community is low.
• Increased surveillance and
control by law
enforcement - though this
might lead to detection
rather than deterrence.
On the shoulders of giants?
Are evaluation’s lessons transferable?
Can we recycle evaluation findings?
How should we answer these questions?
Where does the learning lie?
• Replication and the search for
enduring empirical generalisation
OR
• Comparison and the confederation of
of diverse findings under ‘theory’
“THE GENERIC TOOLS OF
GOVERNMENT ACTION”
Rather than focusing on individual programs, as is
now done, or even collections of programs grouped
according to major ‘purpose’ as is frequently
proposed, the suggestion here is that we should
concentrate on the generic tools of government
action that come to be used, in varying
combinations in particular public programs.
Lester M Salamon 1981
Carrot theory - a Cook’s Tour
How Incentives Fare:
• New York Tenements
Early 20th C
• USA - Erewon State
University Campus
Late 20th C
• Breakfast Room
Domesticville, Canada
Late 20th C
Earmarking ‘death money’
A significant proportion of
charitable poor relief,
intended for basic nutrition
and child welfare, was paid
over as ‘death money’ to be
used for the extravagant
funerals of family members.
The neighbours would talk if
their wasn’t a “fine layout”.
V Zelizer
Earmarking ‘blood money’
“I kind of considered it like getting
twenty bucks from grandmother.
It’s 20 free dollars…. Your going to
go blow it on something…. I never
really needed it essentially but it
was always useful.”
L. Anderson et al
Earmarking survey response incentives
• Charities - NO!
• Lotteries - NO!
• Cash - YES!
$2
K. Warriner et al
$5
$10
“The response incentive lies in
the middle ground between
more subtle concepts of
helping behaviour and naked
economic self interest”
Carrot Theory
Refined…….
•
•
•
•
•
•
•
•
Payment is the measure - but incentives are the intended mechanism
But subjects act on the measure rather than the intended mechanism
The targeting of incentives will always be distorted via ‘earmarking’
Earmarking can be benign or a blight in policy terms
Earmarking lies between ‘egoism’ and ‘altruism’
Earmarking practices will vary according to recipient and context
Earmarking practices must be examined prior to any new incentive
Policy-makers who earmark usage (vouchers) will be earmarked back
COMPLEXITY
New Deal for Communities - The Implementation Chain
Policy Architects →
→ Practitioners
→
→ Subjects
Social exclusion →→ Social mobilisation →→ Organisational →→ Programme
Dealing with Complexity
• Don’t try to evaluate everything - assume some
programme theories work
• Use suites of research - both live evaluations and
retrospective reviews
• Remember to cull evidence from other policy domains
• Use horizontal (theories-of-change) cuts and follow
implementation sequences
• Use vertical (realist) cuts and test programme
mechanisms in different contexts
• THE BOTTOM LINE - IT IS IMPOSSIBLE TO EVALUATE
AND CONTROL EVERY CONSTITUENT PROCESS
How does knowledge speak to power?
Political
Arithmetic
Enlightenment
Discursive
Constructions
But isn’t “enlightenment” all rather ephemeral?
Metaphors for utilisation:
• Diffusion of innovation
• Knowledge creep
• Percolation of ideas
• Decentralised learning
• Informed opinion
• The longer view
• Critical outlooks
…because these describe the ‘medium’ rather than the ‘message’.
“Thinking it through”
Be aware ………
• that in getting to Z you need to
plan to get though W, X and Y
• that success at A was followed
by failure at B and the new
location C is different again
• that intended effect M is often
accompanied by unintended
consequence N
EES 2002 : CALL-TO-ARMS
GO
FORTH
AND
THEORISE