Kittens are Evil
Heresies in Public Policy
Dr Toby Lowe
Language

“Outcomes Based Evaluation”

“Outcomes Based Accountability”

“Results Based Management”

“Payment by Results”
= measure performance by the impact a
person/team/organisation/project has in the
world
Research Findings
•
•
•
•
Outcomes don’t measure impact in
people’s lives
Outcomes aren’t delivered by an
organisation
Outcomes distort organisations’ priorities
Outcomes undermine good frontline
practice
Outcomes don’t
measure impact
“One clear and compelling answer to the
question of "Why measure outcomes?" is:
To see if programs really make a
difference in the lives of people.”
(emphasis in original)
Outcome Measurement :What and Why?
An overview, United Way of America,
1996
Outcomes don’t
measure impact
How should we measure impact?
A set of subjects and a control group with similar
characteristics
•
Matching pairs of subjects – one who receives the
intervention, one who does not
•
All other types of study are “weak”, or of limited
“certainty, comparability, and generalizability”
•
Robert Schalock, Outcomes-Based Evaluation, 2001
Outcomes don’t
measure impact
Control for:
“age, gender, functional level, educational
status, intellectual level, risk level,
diagnosis, prognosis or the individual’s
time within the program”
Robert Schalock, Outcomes-Based
Evaluation, 2001
Outcomes don’t
measure impact
What research methodology do you need?
•
•
Qualitative and quantitative methods
Enable participants to design own survey
questions
Robert Schalock, Outcomes-Based
Evaluation, 2001
Minimum recommended follow-up time
to understand impact of an intervention
in an individual’s life?
2 years
What does get measured?
Accountability
Dimension
Quality
Dimension
Key aspects of Accountability
•
•
•
•
Simple, “efficient” data sets
Data collection formats that can lead directly
(and electronically) to data management and
entry and data analysis
Data collection timelines that are consistent
with the organisation’s strategic plan
Standardised report formats that will allow
annual outcome reports (such as report
cards)”
What does get measured?
Accountability
Dimension
Quality
Dimension
Understanding impact
•
Person with mild to medium mental
health problems. Caring responsibilities.
Problems with substance misuse. But in
recovery.
•
They’re on an employability programme
•
They get a job
Impact….?
•
Better or worse mental health?
•
Is the family better off financially?
•
Impact on dependents?
•
On extended family?
•
On the neighbourhood?
•
Impact on use of other state services?
Outcomes aren’t delivered…
•
What is an outcome?
•
How is an outcome made?
Let’s pick one…
•
Preventing an offender from re-offending
Making an outcome
•
Employment and/or availability of legal
income
•
Family circumstance/availability of housing
•
Relationships
•
Individual disposition
•
Criminality in peer network
•
Substance misuse
What is an outcome?
•
A state of affairs in the world
•
Subject to enormous complexity
What is an outcome?
Programme Logic Model
Robert Schalock & Gordon
Bonham “Measuring outcomes
and managing for results”,
Evaluation and Program
Planning, 2003
Attributability
•
Simplify the world
•
Use statistical regression analysis
Programme Logic Model
Robert Schalock & Gordon
Bonham “Measuring outcomes
and managing for results”,
Evaluation and Program
Planning, 2003
What is an outcome?
An employability-support
programme
•
•
•
•
•
It invests time in building meaningful relationships
with participants
offers support and activities which transform
participants sense of what is possible for themselves
helps participants to build key employability skills
related to the local labour market,
helps them with interviews and placements within
local firms
Offers post-employment, in-work support
Outcomes-Based
Accountability
•
•
Payment by Results – no ‘results’
payment
Soft-outcomes: how were these affected
by loss of confidence from labour market
collapse?
What else is missing?
Program participants?
?
?
“Outcomes are by definition results over
which organizations do not have complete
control”
John Mayne, “Challenges and Lessons in
Implementing Results-Based
Management”, Evaluation, 2007
Summary
Theoretical problems:
•
•
Outcomes don’t measure impact in
people’s lives
Outcomes aren’t delivered by an
organisation
The uncertainty principle in action?
Implementing
outcomes approaches
OBA creates “goal displacement …which leads to
emphasis on the wrong activities and encourages
creaming and other means of ‘making the
numbers’ without improving actual outcomes. As a
result, they frequently distort the direction of
programs, diverting attention away from, rather
than towards, what the program should be
doing.”
Burt Perrin “Effective Use and Misuse of Performance
Measurement”, American Journal of Evaluation,
1998
•Implementing
outcomes approaches
“unintended consequences” of using outcomes
information:
•
•
focussing on those who are easiest to help, at
the expense of those most in need
“difficult” clients are skipped in favor of the
“easy” ones
S van Thiel and F. L. Leeuw “The Performance
Paradox in the Public Sector”, Public
Performance and Management Review, 2002
“Target based performance management
always creates ‘gaming’” (my emphasis)
Bevan, G. and Hood, C. “What’s measured
is what matters: targets and gaming in the
English public health care system”, Public
Administration, 2006
“A4e employee forged
signatures to boost job
placement numbers”
The Guardian, 6th March, 2012
What’s the answer?
•
It’s a technical problem!
•
Let’s measure better!
•
Let’s measure
differently!
Outcomes data
Review questions…
•
Q: What is my performance being measured on?
•
A: The production of appropriate outcomes data
•
•
•
•
Q: How do I produce this data? What are the factors which
will create the data I need?
A: Who are the clients which will give me this data?
(creaming/cherry picking)
A: How can I target my resources towards these clients to
produce the relevant data? (teaching to the test)
A: What other ways are there that I can produce this data?
(reclassifying/making things up)
Unintended
consequences?
Commissioning
for outcomes
•
•
Make people promise to deliver outcome
targets (otherwise they won’t win
contracts)
Require people to produce outcome data
(otherwise they won’t get paid)
= entirely knowable, predictable results.
Implementing
outcomes approaches
“Always results in gaming”:
•
•
Creaming (helping the easiest to help)
Targeting resources to produce data
(teaching to the test)
•
Reclassifying results (pretending)
•
Making things up
Impact on
frontline practice
•
Focus on outcomes undermines the
requirements to build relationships with
service users
Frontline practice
OBA makes “it more difficult to engage with and build
relationships with homeless and at risk young
people”
has significant impacts on the daily practice of workers
It reduces the time available to create a sense of
belonging – to build relationships
It reduces the time to “develop young people’s life skills
Lynn Keevers (et al) “Made to Measure: Taming
Practices with Results-based Accountability”,
Organization Studies, 2012
Frontline practice
• 86 per cent of time is system driven - filling in forms for
accountability and discussing them with colleagues.
The 14 per cent of time spent face to face with a family
member is not developmental.
•
“The dialogue between Ryan and Tom is dictated by the forms
and their need for data and information. This squeezes out
any possibility of the sort of conversation that might be
needed to develop a supportive relationship as a first step in
fostering change.”
Hilary Cottom, Relational Welfare, 2011
Frontline practice
= Reversal of relationship between
worker/client
From: how can I help you achieve
your goals?
To: how can you help me achieve
my targets?
Summary
•
•
•
•
Outcomes don’t measure impact in
people’s lives
Outcomes aren’t delivered by an
organisation
Outcomes distort organisations’ priorities
Outcomes undermine good frontline
practice
If not outcomes,
then what?
•
•
•
Bottom up is key – start from
actual people’s needs
Systems thinking is helpful
Ground this theory in social
science
If not outcomes, then what?
•
We know about human
behaviour in society:
–
Social context
–
Human agency
Thanks for listening
Toby Lowe
E: [email protected]
Twitter: @tobyjlowe