Issues in evaluation
Nick Tilley
Why evaluate?
• To learn lessons for other places and times,
– though care needs to taken in replications – they are
never exact
• For accountability,
– though performance indicator driven evaluation can
produce perverse incentives.
• To inform scheme adjustments,
– though it is important to give schemes time to bed in.
The track record of evaluation
•
•
•
•
•
•
Relatively little is evaluated.
There is much lying in evaluation.
There is little competence in evaluation.
Methodology is heavily debated.
Masses of implementation failure is found.
In the most useful evaluations researchers
have been involved in project design.
Problems for evaluation
• Record keeping regarding crime and disorder
• Data provision, data protection and data
security.
• Data quality
• Records tracking interventions
• Political/administrative pressure on evaluators
• Ideology
• Technical skills
What’s worth evaluating?
• Not everything!
– It’s too expensive
– It’s too difficult
– Nothing will be learned
• Prioritise!
–
–
–
–
Where there are significant decisions are at stake
Where there is a chance that evidence will be heard
Where competent implementation is likely
Where project workers and data custodians will play
ball
– Where there is inadequate or insufficient research to
date
Rules for evaluation
• Work out the scheme theory – read and consult
– How is the scheme expected to work and for whom?
– What side effects might be expected, and for whom?
• Work out what to measure to test the theory.
• Measure properly.
• Don’t expect to be able to prove conclusively
what works.
• Tell the truth about unwelcome as well as
welcome findings.
• Don’t make wild generalisations.
• Don ‘t come to premature conclusions.
Example: a scheme for evaluation
• The scheme starts in April 2003.
• The scheme focuses on reducing council
house burglary in a local area.
• An evaluation report is asked for in July
2003.
• The LA wants to decide whether to cancel
it, continue it or roll it out.
• Was it effective?
Before and after 1
700
600
500
400
300
200
100
0
Jan-Mar 2003
April-May 2003
Before and after 2
700
600
500
400
300
200
100
0
Jan-Mar 2003
April-Jun 2003
Time series 1 – longer term trend
1600
1400
1200
1000
800
600
400
200
0
Jan-Mar
2002
Apr-Jun
2002
Jul-Sept
2002
Oct-Dec
2002
Jan-Mar
2003
April-Jun
2003
Time series 2: regression to the
mean
900
800
700
600
500
400
300
200
100
0
Jan-Mar
2002
Apr-Jun
2002
Jul-Sept
2002
Oct-Dec
2002
Jan-Mar
2003
April-Jun
2003
nM
ar
Ap
20
r-J
01
u
Ju n 2
00
l-S
1
ep
t2
O
ct
00
-D
ec 1
Ja
20
n01
M
a
Ap r 2
r-J 002
un
Ju
20
l-S
02
ep
t2
O
ct
00
-D
ec 2
Ja
n- 200
M
2
a
Ap
r2
00
ril
-J
un 3
20
03
Ja
Time series 3: seasonal patterns
900
800
700
600
500
400
300
200
100
0
nM
ar
Ap
20
r-J
01
u
Ju n 2
00
l-S
1
ep
t2
O
ct
00
-D
ec 1
Ja
20
n01
M
ar
Ap
20
r-J
0
un 2
Ju
20
l-S
02
ep
t2
O
ct
00
-D
ec 2
Ja
n- 200
M
2
a
Ap
r2
00
ril
-J
un 3
20
03
Ja
Time series 4
900
800
700
600
500
400
300
200
100
0
Time series 5 (with 4 project data)
3000
2500
2000
1500
1000
500
0
Jan-Mar Apr-Jun Jul-Sept Oct-Dec Jan-Mar Apr-Jun Jul-Sept Oct-Dec Jan-Mar
2001
2001
2001
2001
2002
2002
2002
2002
2003
Project area
Rest of LA
AprilJun
2003
Time series 6 (with data from 5)
Per cent
40
30
20
10
0
JanMar
2001
AprJun
2001
JulSept
2001
OctDec
2001
JanMar
2002
AprJun
2002
JulSept
2002
Project area share of BCU
OctDec
2002
JanMar
2003
AprilJun
2003
Time series 7 (using 6 data)
600
500
400
300
200
100
0
JanMar
2001
Apr-Jun
2001
JulSept
2001
OctDec
2001
JanMar
2002
Private
Apr-Jun
2002
JulSept
2002
Council
OctDec
2002
JanMar
2003
AprilJun
2003
Query
• Is there an anticipatory benefit here?
Per cent
Time series 8 (using 7 data)
30
25
20
15
10
5
0
Jan- Apr- Jul- Oct- Jan- Apr- Jul- Oct- Jan- AprilMar Jun Sept Dec Mar Jun Sept Dec Mar Jun
2001 2001 2001 2001 2002 2002 2002 2002 2003 2003
Council house share
Per cent
Time series 9 (using 8 data)
30
25
20
15
10
5
0
JanMar
2001
AprJun
2001
JulSept
2001
OctDec
2001
JanMar
2002
Private project share of LA
AprJun
2002
JulSept
2002
OctDec
2002
JanMar
2003
LA project share of LA
AprilJun
2003
Conclusions
• It is easy to lie/mislead with data.
• Some technical skills are needed in evaluation –
untutored self-evaluations tend to be very weak
and self-serving.
• Side-effects, notably diffusion of benefit and
displacement, should be explored.
• It is useful to find the active ingredients in
initiatives. They will not always be obvious.
• It is dangerous to draw premature conclusions.