V&V CASE STUDIES
The Case of the Detached
Deterrent
This pitfall occurs when a deterrence measure fails
to be supported by a rationale for the program’s
deterrence effect, a threshold for what constitutes a
deterrence effect, and a grasp of the impact
detection capabilities have on the measure.
APPLICABLE
V&V CRITERIA

Valid
__ Complete
A detached deterrent is like snapping
your fingers to keep tigers away.
Who this case is about…
Take the case of the fictional Wildlife Management Program (WIMP). The program’s mission is to build and
maintain a healthy population of wild animals on federal lands. While the population levels of some animals
are robust, the population levels of others are anemic. In the former cases, WIMP allows limited hunting
periods to ensure population levels do not grow too high, but in the later cases, WIMP has instituted a strict
no-kill policy to allow population levels to grow. Wildlife officers patrol federal lands to ensure that hunters
do not violate the no-kill policy. Those who do are fined and have their kill seized. Repeat offenders are
subject to jail time.
Program management wanted to develop an outcome performance measure that captures the deterrent effect
that the program is having on illegal hunting. Managers wanted to use an outcome measure because it
provides a more direct indication of the impact that justifies the program’s budget. The following measure
was selected:
“The number of violations of WIMP’s no-kill policy.”
What went wrong…
After a few years, no discernable pattern in the data could be seen; values fluctuated between 50 and 80
violations per year. While some variation was expected as a result of the program's changing capacity to detect
violations (resources for patrols varied slightly from year to year), the amount of variation was higher than
could be explained. This unexpected variation caused program targets, which typically represented an
incremental change over the previous year, to be missed frequently. The program came under fire from external
stakeholders, who suggested that the measure results over the past few years failed to demonstrate a deterrence
impact. Internally, some wildlife officers expressed concerns over balancing the trade-off between detection
and deterrence. Detection involves simply identifying that a violation occurred. Deterrence requires that
officers pursue and apprehend violators and then complete paperwork to ensure the fines are imposed, which
takes time away from detection. The measure had become “detached” from the need to demonstrate a
deterrence impact and the need to detect the problem in the first place.
Why it’s a problem…
Deterrence measures can be an effective way to measure performance; however, their complicated
nature requires a sound rationale for the claimed deterrent impact as well as sufficient supporting data
to provide meaningful context for making a judgment of effectiveness.
Demonstrating impact is harder with deterrence measures because a positive impact means that
something is not happening. For this reason, deterrence measures need either empirical evidence, which is
extremely difficult and expensive to obtain, or a rationale for why those running the program believe it is
having a deterrent effect. Otherwise, a program’s claim that it is having a deterrent effect can ring hollow. It’s
like the man who constantly snaps his fingers. A woman walks by and asks, “Why are you snapping your

Consistent

Accurate
__ Timely
V&V CASE STUDIES
APPLICABLE
V&V CRITERIA
 Valid
__ Complete
 Consistent
 Accurate
__ Timely
fingers?” He says, “It keeps the tigers away.” She says, “But there aren’t any tigers in New York City!” To
which he replies, “Pretty effective, huh!” Of course, it’s absurd to infer a causal connection between fingersnapping and the absence of tigers, but deterrence measures can fall into such a trap by erroneously inferring a
causal connection between deterrence activity and the absence of something. The lack of a sound rationale
linking program activities to the outcome is a problem under the "valid" criterion of verification and validation
(V&V).
Complementing a sound rationale is an accounting of the numerous factors that can affect a measure. For a
deterrence measure, the most prominent of these is typically a program’s ability to detect what is being
deterred. If not properly accounted for, increases in detection capacity can be construed as erosion in deterrence
effectiveness—and decreases in detection capacity as increased deterrence effectiveness. Other factors (e.g.,
large increases in hunting fees) can also affect the measure. If these factors are not accounted for, using the
measure to assess program effectiveness can be difficult—a problem under the V&V’s “valid” criterion.
Interpreting the results of a deterrence measure requires some estimate of the counterfactual—what would have
happened if the program did not exist. For instance, given an estimated counterfactual of 50,000 hunting
violations per year versus 80 detected violations would represent a 99.84% reduction (an unqualified success).
However, if the counterfactual estimate was 100, the 80 violations would represent only a 20% reduction
(marginal success at best). The lack of a counterfactual makes an assessment of program effectiveness difficult
—a problem under the V&V’s “valid” criterion.
The trade-off between detection and deterrence creates conflicting incentives that different officers could
resolve in different ways. Some officers could feel pressure to maximize detection so that the program has a
better indication of the true level of violations (and the need to remedy it). Others may feel that it is more
important to catch and punish violators. Such inconsistent data collection and the accompanying errors create
problems under the V&V’s “consistency” and “accuracy” criteria.
The program’s solution…
Program staff started by developing a rationale for the program's deterrent impact. They described a twofold
effect. First, they were having a “cop-on-the-beat” impact. That is, the presence of Wildlife officers on federal
lands was sufficient to deter some people from illegally killing animals. Second, the frequency of being caught
and the severity of imposed penalties were sufficient to deter some hunters from illegally killing animals. The
deterrence literature is clear that if the possibility of being caught is too low and the penalties imposed are not
severe enough, a significant number of people will still commit illegal acts.
Program staff realized that they needed to collect additional data to support the measure. They began by
estimating the counterfactual. They looked at the number of hunters visiting federal land each year and then
conducted benchmarking research to estimate the fraction of this population who are likely to commit such
violations. The program tracked how often hunters visit federal lands and calculated the number of
opportunities that potential violators have to kill endangered animals. Staff also studied a range of other factors
such as how changes in economic conditions affected hunters’ willingness to travel to go hunting. Using these
data, the program estimated that the total number of violations in the absence of the program would be
upwards of 30,000 per year. Program management determined that an annual target of 300 or less, representing
a 99% reduction in violations, would provide significant value to taxpayers for the cost. The annual
fluctuations between 50 and 80 violations were now correctly interpreted as “noise” in the measure, caused by
a confluence of external factors.
Finally, the program established a sampling process for data collection whereby rigorous detection protocols
were implemented during randomly selected days, leaving the officers to concentrate on apprehension and
punishment the rest of the time.
Takeaway
The moral of the story: When using deterrence measures, be sure to develop a sound
rationale to support the claimed deterrence effect, maintain an understanding of the impact
detection has on the measure, estimate what a reasonable level of deterrence is, and
establish data collection procedures that minimize the trade-off between deterrence and
detection.