From Outputs to Outcomes:
Measuring Results of Enforcement and
Compliance Activities
National Environmental Partnership Summit
May 10, 2006
Lisa Lund
Deputy Director
Office of Compliance
U.S. EPA
Evolution of Measurement in OECA

In the past, emphasis on activity counts
and direct results of activity (outputs) per
unit of time:



Number of inspections
Number of administrative orders, referrals,
etc.
Dollar value of penalties assessed, years of
incarceration, etc.
2
Evolution of Measurement in OECA


Today, emphasis on data for planning, managing
and evaluating
Need a combination of outputs and outcomes



Change in behavior of regulated entity as result of
compliance assistance
Number of pounds of pollutant reduced as result of
enforcement actions
Dollar value of health benefits achieved by pollutant
reductions
3
Outcome Measures - Definitions

Outcomes reflect the results of outputs

Intermediate outcome – measures progress
toward a final outcome


Intermediate outcome of enforcement action:
change in facility management practices
Final outcome – measures ultimate result the
program is designed to achieve

Final outcome of enforcement action: reduction in
number of people living in area where pollutant
standards were exceeded
4
Hierarchy of Indicators
5
Steps to Improve Outcome
Measurement




OECA measurement program review
Exploring use of hazard and exposure data
Seeking research on measuring deterrent effect
Engaging with international enforcement and
compliance programs

Moderated two international discussion forums
seeking input on best practices around the globe
6
Steps to Improve Outcome
Measurement



Environmental Assistance Network - developing
common measures to demonstrate results under
Goal 5 of EPA Strategic Plan
State Review Framework – ensuring consistency
among states in implementing enforcement and
compliance programs, improving data quality
within and among states
Element 13 – developing method for getting
outcome information on Elements 1-12
7
Environmental Assistance Network




Help EPA better plan, coordinate, and measure
its environmental assistance efforts
Achieve the maximum efficiency and
effectiveness in helping organizations improve
their environmental performance
Present a more organized, cohesive face to
external stakeholders
More effectively demonstrate results under Goal
5 of the Agency’s strategic plan
8
Common Measures for Goal 5





Entities reached by environmental assistance
(EA) activities
Entities reporting increased understanding as a
result of EA activities
Entities reporting changes in behavior as a result
of EA activities
Entities reporting a reduction in pollution/
environmental impact as a result of EA activities
Entities reporting reduced operating costs as a
result of EA activities
9
EA Network Sector Focus Area

Construction sector selected as the first EA
Network focus area. Sector activities may
include:



identifying metrics of common interest or gaps in
measurement
proposing and tracking standard cross-program
metrics
identifying where standard metrics may be applied
(e.g., in OECA’s commitment system supporting
national priorities) and promoting their use
10
State Review Framework

Tool to gauge state performance in
compliance assurance




Developed jointly by ECOS Compliance
Committee, EPA OECA, regions and states
Provides consistency in environmental protection,
consistent level of activities in states and a
consistent level of oversight by EPA regions
Based on 12 elements and a 13th optional element
Uses common set of output measures to
gauge performance
11
Element 13


Provides mechanism for state to share outcome
information related to Elements 1-12
Allows states to:



Seek recognition from EPA that innovative program is
important part of its core program
Seek reduction in core program activities in order to
conduct alternative/innovative program that leads to
improving compliance
Data must demonstrate outcomes that comport
with Agency goals
12
Challenges

Inconsistent data quality



Resource intensive


Complex and time-consuming; design and development require
careful consideration of numerous criteria
Scope of regulated universe makes development of
overall compliance rate infeasible


Effective outcome measures depend on complete and accurate
reporting by states and regions
Requires common set of measures
11 statutes x 28 programs x possibly hundreds of points of
compliance x thousands of regulated entities
Few environmental and human health models
appropriate for our use exist
13
Conclusion


Viability and future funding of our
programs depend on robust performance
measurement, using outcomes to the
greatest extent possible
We’ve made great progress, more work is
needed
14
Contact Information
Lisa C. Lund
Deputy Director
Office of Compliance
U.S. Environmental Protection Agency
(202) 564-2280
[email protected]
15