Practical, Affordable
Performance Measurement:
Why and How?
Shelley H. Metzenbaum
[email protected]
www.complianceconsortium.org
www.facilityreporting.org
Northeast Environmental Summit September 2005
Overview
• Why Measure?
• How to Measure and Use the
Measures?
• Examples
• Issues to Consider
• Workshop Tomorrow
Why Measure?
• Increase Accountability
• Learn to Achieve Better
Outcomes
• Strengthen Democracy
Measurement for Accountability
(Program Justification?)
• Ethical: Self-Dealing/Fraud (track $
flows to whom for what – accounting &
fiscal audits)
• Fiscal: Unallowed
spending/Waste/Efficiency
($
spent/$ allocated & goods or services
bought per input cost)
• Performance: Improve social
outcomes/reduce risk/more
cost-effective (better outcomes; better
outcomes at lowest cost)
Measurement to Learn so
Achieve Better Outcomes
• DETECTION: Detect problems needing
attention; progress deserving replicationwhat works and what doesn’t?
– Look for anomalies, unexpected direction,
speed ups, slow downs, etc.
– Use measurement to trigger focused follow up
• DIAGNOSTICS: Why? What is the reason for
variation? Is it a problem or lesson?
• ASSESSMENT: Did it seem to work? Replicate,
expand, or stop and fix?
Goals/Measures to
Strengthen Democracy
• Cannot Do Everything: Goals/
Targets Communicate:
– What You Will Do
– What You Won’t Do
• Are You Doing Right Thing?
Democracies can respond when
unhappy with agency choices
• How Well Are You Doing It?
Measurement
Start with Learning
Emphasis of Measurement
• Detection, Diagnostics,
Assessment
• Do this well …
Accountability & Democracy
will Rise, as well
• Practical, Affordable
Examples
DETECTION : Look for Pattern
Changes to Sense Problems
Why the Downturn ?
Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes
DETECTION: Volunteer Data -Look for Unexplained Variations
Description
Town
Riv
er
Mile
Central Street
Bridge
Discharge Pipe
@ Central
Street
2d Discharge
Pipe @ Central
Street
Mellen St.
Bridge
Rt. 126, N.
Main
Milford
3.5
1100
380
120
Milford
3.5
1470
2100
3400
Milford
3.5
<10
260
Belling
ham
Belling
ham
5.9
490
140
10
120
9.0
2/15/00 3/21/0 5/16/00
0
340
DETECTION : Look for Pattern
Changes to Find Progress
Recognize Progress !
Source: http://www.chesapeakebay.net/status/status_print.cfm?sid=101&print=yes
DETECT: Before & After Photos
DETECTION/DIAGNOSTICS:
Inspec. Data
Detect low comp. rate
Diagnose: Is it a problem?
“Scout &
Why is it happening?
Survey”
% Compliance by Violation Type
100
90
80
70
60
50
40
30
20
10
0
.
ri.
.
ra
ct
e
gs
ha
tin
se
d
C
lo
Po
s
A
is
le
Sp
ac
e
s
be
l
La
io
n
ct
sp
e
In
Tr
ai
ni
n
s
g
NH, CO
Diagnostics:
MA ERPASSESS:
Self-Audit Data
DETECT,
DIAGNOSE,
1997 & 1998
Self-Reported Practices Trends (97/98)
100%
90%
80%
Lower
70% than
Other
60%Areas:
Problem?
50%
40%
30%
20%
10%
0%
8
(10) Leak
Checks
Changes
over
Time:
Progress?
10a
(18-19)
Controls
13
18
22
35
(36i) Emergency
Procedures
36i
38
50
MA ERP
DETECTION and DIAGNOSTICS:
Mapping/Segmentation Highlights
Problem Areas and Progress
1996 DRY
WEATHER
BACTERIA
1996/97 WET WEATHER
BACTERIA
Swim Standard
Some Failures
Failed
DIAGNOSTICS/ASSESSMENT:
Slice & Dice by Categories to Identify
Cause of Problems, Progress
Detection: # Complaints
(Cleveland Air Pollution)
• General Environmental Mgt. – 64
• ISG – 52
• Ford Motor Company – 29
• Kirkwood Industries – 18
• Landfill – 14
• Wabash Alloys -12
DIAGNOSTICS: Complaint
Relevance?
Source: PA DEP
ASSESSMENT: Pictures
Consolidated Tailings Pile (CO)
Prior to 1989
After Reclamation
1999
Lots of Affordable Ways to
DETECT, DIAGNOSE, ASSESS
• Air and Water Quality Data
– Volunteer & govt collected data useful
– Water, Energy Supply – Boston Water and
Sewer Commission use data to cut water
use and costs
• Cameras – Texas detects problem air
plumes; Rutgers extension helps farmers
cut pesticides and costs
• Incident and Inspection Data:
– Repeat Violators - Accidents
– Nature of Non-Compliance – Complaints
– Fires
DETECTION, DIAGNOSTICS,
ASSESSMENT: Interim Outcome
Indicators Ideas Measure Awareness (OR)
Information
Attention
Comprehension
Beliefs/Attitudes
Motivation
Measure
Understanding
Measure
Attitudes
Measure Motivation
Behavior
Measure Behavioral Changes (ERP)
Newspaper Content Analysis,
Surveys, Observational
Measurement
70
60
50
Technology
Chemical Industry
Trade Press Content
Analysis (Hoffman 1999)
Regulation
Enforcement
40
Management
30
20
10
0
1962-1970
1971-1982
1983-1988
1989-1993
STRENGTHEN DEMOCRACY:
Specific Goals Clarify What
Govt Will Do and What It Won’t
• Where: Lower Charles River
• When: 10 years
• How much: Swimmable
• Who: All Sources
• Invite public attention
• Public can respond
Strengthen Democracy: Make
Goals, Progress, Problems Clear
Goal Not Met,
Past and Future
Strategy
Described
Goal Met;
Commitment
to New Goal
Described
Issues to Consider
• Data Accuracy: Match to need. Can
be lower for Detection; higher for
Rules and Enforcement. For
assessment, consider program $
• Project and Measurement Scale:
indiv. facility level work OK only if
have context/control. Count costs
and cost-effectiveness when project
level. Start with env. problem.
• Comp asstc and P2 activity counts
not useful indicator; may have
occasional diagnostic value
Tomorrow’s Workshop:
Brainstorming Your
Specific Measurement
Challenges Together
1:30 - 3:30
Shelley H. Metzenbaum
[email protected]
www.complianceconsortium.org
www.facilityreporting.org