1
CJE3444
Crime Prevention
Chapter 3
Evaluation and Crime Prevention
Dr. Elizabeth C. Buchholz
Types of Evaluation
 Evaluation of crime prevention refers to investigating the impact of a prevention technique or
intervention on the level of subsequent crime, fear, or other intended outcome.
 Investigating the impact
 Impact Evaluation
 Focus on what changes occur after the introduction of the policy, intervention, or
program.
 For instance: Do neighborhood watch problems reduce fear and actual crime rates once
introduced?
 Do police patrols reduce drug sales in an area?
Impact (outcome) Evaluation
cont’d
 In crime prevention, pose interesting problems
 Crime prevention initiatives rarely rely on a single intervention or approach
 The target of the initiatives is a neighborhood or other geographic area
 Neighborhoods cannot be isolated
 Many interventions are not uniformly applied across an area or adopted by all
residents
 Competing issues of crime displacement and diffusion of benefits
Impact (outcome) Evaluation
 Problems:
 The target of the initiatives (unit of analysis for evaluation) is a neighborhood or
other geographic area
 Neighborhoods cannot be isolated
 Many interventions are not uniformly applied across an area or adopted by
all residents
 Issues of crime displacement and diffusion of benefits
Impact Evaluation
 Problems (cont’d):
 Crime prevention initiatives rarely rely on a single intervention approach.
Programs use a “menu” of different activities at the same time
 Watch scheme
 Property identification
 Neighborhood cleanup
 Periodic meeting
 Newsletter
Obstacles with Impact Evaluation
 Which of the program initiatives are most effective?
 What is the unit of analysis?
 Is crime displaced (moved to another area) or is it prevented?
09.03.13
2
Process Evaluation
 Considers the implementation of a program or initiative and involves determining the
procedures used to implement a specific program.
 More of a qualitative approach to evaluation.
 Offer a detailed descriptive account of the program and its implementation
 Information is pivotal in answering questions about the context of an intervention and what
actually took place in the initiative
 Unfortunately, many evaluations look only at the process
Process Evaluation
 Mission/goals of the program
 Level and quality of the program staff
 Funding and other resources of the program
 Obstacles faced in implementing and sustaining the initiative
 Degree to which the project was carried out as planned
 Level of support for the program
 Degree to which the clients complied with the operation
Problems with Process Evaluation
 The impact of the program is often difficult to comprehend
 Success is often determined by “false” results:
 Number of meetings
 Number of members involved
 Length of program
 Financial support
 Are not true measures of success
Cost Benefit Evaluations
 Seeks to assess whether the costs of an intervention are justified by the benefits or outcomes
that accrue from it.
 A cost benefit analysis should involve an impact and process evaluation to determine the
program’s worth.
 You cannot determine if costs are justified if you do not measure whether or not the program
is able to bring about the expected change.
Problems with Cost-Benefit Evaluations
 Largest problem involves setting monetary values on factors that are not easily enumerated
(fear, trauma, emotional loss, etc.)
 Another problem is making certain that all of the costs involved in the program (and related
to the program operations) are counted
Theories and Measurement
 Programs are often created in a theoretical vacuum
 Those implementing and evaluating the intervention do not pay attention to the
theoretical assumptions underlying the prevention program.
 Evaluations undertaken in a theoretical vacuum may still provide answers regarding whether
the program had the intended impact, just not why
 Basic questions on the assumed causal mechanism are often ignored
 Residents and planners want results but often fail to ask the questions of why or how:
09.03.13
3
 Why would an educational program reduce aggressive behavior?
 This results in “Basic” evaluations of yes it was a success or no it was not
 These evaluations fail to tell us “why”
Theories and Measurement
 Programs that are not evaluated:
 Fail to tell us why the program is or is not successful
 Can provide only limited insight to whether the program can be implemented to other
places or at other times
 Many investigations might not be necessary if the underlying theory for the
intervention was examined
• Implementing curfew laws when most crimes involving youths occur during
the afternoon
Why the resistance?
1. “Outcome Myopia”
 Programs and the evaluators are only interested in whether the program works
and not how or why it works.
 Does not take into account the possibility that other factors are at work
 Does not tell why a program does not work
2. Many program administrators assume that a program works
 They believe it’s only “common sense” that it works
 Blind belief in programs
 Have the ear of politicians who can provide legislative and funding
3. Many programs are the results of grassroots efforts
 Not interested in evaluations, as long as they’re happy with it
Theories and Evaluation
 Evaluation of crime prevention strategies is key
 Most programs fail to have adequate evaluation strategies
 Examples
 Juvenile curfew laws
 Weed and Seed programs
Measurement Problems
 Measuring key outcome variables when the intervention is geographically based
• Is area being measured the same as the area the police intervention was
implemented?
• Many crime prevention programs are based on neighborhoods or other small
geographic areas that do not coincide with specific police reporting areas
 Conundrum in crime prevention whereas programs often try to simultaneously reduce the
level of crime while increasing the reporting of crime to the police
 Possible solutions
• Using victim survey data
THEORY AND MEASUREMENT
 Measurement Issues cont’d
Victim data is not always available, and the collection of that data can be both timeconsuming and costly
Operationalizing key variables such as fear is not a straightforward enterprise
09.03.13
4
Finding ways to uncover the competing influences in the project that mask the outcomes is
difficult
THEORY AND MEASUREMENT
 Follow-Up Periods
How long after the implementation of the program or intervention will changes in crime (or
other outcome) appear?
Is there a possibility that over time any initial changes will diminish or disappear?
There is no rule on the appropriate follow-up time
Evaluation should look to the underlying theory for guidance
Ideal would be one where follow-up data is gathered at different intervals
METHOD FOR EVALUATION
Experimental Design
 The preferred approach by many evaluators due to a number of strengths:
 A true experimental design is also known as a randomized control trial
o the gold standard in evaluation
 Relies on the random assignment of cases into experimental and control groups, increases the
likelihood that the two groups being compared are equivalent.
 There is enough control over the evaluation to make certain that the experimental group
receives the treatment or intervention, while the control group does not.
Experimental Design
 A strength of Experimental Design is controlling for internal validity
 Internal validity deals with factors that could cause the results other than the measures that
were implemented
 Having a control group limits this threat to the program
Experimental Design cont’d
 The Maryland Scale of Scientific Methods (see Table 3.2) outlines the measures for
evaluation
 Developed for a review of literature on what works in prevention for the U.S.
Congress
 How closely a study adhered to the standards of a true experimental design
 Suggests that policy makers should only consider research that meets the gold
standard and that research funds should only be expended when an experimental
design is used
 Not many programs meet the experimental design goal
Other Threats to Evaluation
 Generalizability
 Are they applicable in other places, settings, and times?
 External Validity
 Wide range of potential problems inherent in trying to replicate the findings of any program
evaluation
 Difficult, if not impossible, to randomly assign communities to experimental and control
groups
 Matching cannot guarantee that the areas are comparable
 Mo way to isolate the experimental and control communities from all other influences
09.03.13
5
 Experimental Design evaluations often guide the project instead of the project being guiding
by the needs of the city
 Experimental Design (cont.)
Problems (cont.)
Underlying problem is that experimental designs fail to consider the context within which a
program or intervention operates
Too easy to jump to a conclusion that something does or does not work
Negative findings may be the result of factors such as poor program implementation,
misspecification of the appropriate target or causal mechanism underlying the problem, or
resistance by the target
Realistic Evaluation
 Rather than relying exclusively on experimental approaches, evaluation needs to observe the
phenomenon in its entirety.
 Two key ideas: mechanism and context
 Mechanism: understanding “what it is about a program which makes it work”
 By what process does an intervention impact an outcome measure such as crime or
fear of crime?
 Context: “the relationship between causal mechanisms and their effects is not fixed, but
contingent”
Summary
 Evaluation or analysis of crime prevention programs are often seen as meaningless or too cost
and are not completed
 A good evaluation is important for the growth and success of the program
 The remaining chapters will focus on different programs/designs and evaluation of program
success will be discussed for each
09.03.13