Why Data Matters
Building and Sustaining a
Business Case
NEAUC Conference
June 18, 2014
Presentation
• Evaluation versus Performance Measurement
• Types of Performance Measures
–
–
–
–
Inputs
Outputs
Outcomes
Impacts
• Data Sources
• Process
2
EVALUATION VERSUS
PERFORMANCE
MEASUREMENT
3
Comparison
Evaluation
Performance
Measurement
Periodic
Ongoing
In-Depth
Developmental
External
Internal
4
Evaluation
• What are the goals?
• How is my program performing compared to
goals or expectations?
• How does it compare to other programs?
• How can the program improve?
5
Performance Measurement
• How can I measure?
–
–
–
–
•
•
•
•
•
My organization’s efforts and inputs
Outcomes of those efforts
How we impacted clients
How we impacted utility
How has this changed over time?
How does my organization compare?
What are higher performers doing?
Are those actions related to results?
Can I implement those actions?
6
TYPES OF PERFORMANCE
MEASURES
7
Types of Measures
More difficult to obtain data
Inputs
Outputs
Outcomes
Impacts
More powerful information
8
Inputs
•
•
•
•
Staff hours
Equipment
Travel costs
Supplies
9
Input Example
Delivery Costs
Agency Job Cost
Percentiles
Year
# of
Jobs
Mean
Cost
2010
530
$2,744
$764
$1,297 $2,423 $4,105 $5,155
2011
762
$3,436
$605
$2,004 $3,435 $4,958 $5,871
2012
369
$4,324
$904
$2,622 $4,310 $6,039 $7,068
1,661 $3,413
$763
$1,679 $3,314 $4,912 $6,070
Total
P10
P25
P50
P75
P90
10
Input Example
Itemized Costs
UGI
Field Support
Administration
Inspections
No Measures
Rehab Pilot
Conservation Pilot
TOTAL
2010
$521,520
$114,529
$19,840
$4,768
$94,774
$0
$755,161
Electric
2011
2010
2011
$764,932 $119,334 $252,962
$167,622
$37,482 $35,671
$12,830
$7,385
$11,390
$4,075
$1,950
$1,600
$68,742
$0
$0
$50,000
$0
$0
$1,068,201 $166,151 $301,623
11
Outputs
•
•
•
•
•
Number of customers applied
Number of customers enrolled
Service delivered
Participant characteristics
Services coordinated with other programs
12
Output Example
Customers Served
Number
Referrals
51,868
Defaulted
10,166
Cancelled
17,006
Graduates
1,011
Moved
8,480
Re-certifications
8,512
New Enrollments
19,401
13
Output Example
Customer Characteristics
Federal
Poverty
Level
0-50%
51-100%
101-150%
Unknown
Total
2011
2012
2013
Number Percent Number Percent Number Percent
892
1,252
832
533
3,509
25%
36%
24%
15%
100%
860
1,079
725
450
3,114
28%
35%
23%
14%
100%
739
906
633
517
2,795
26%
32%
23%
19%
100%
14
Output Example
Customer Characteristics
2011
2012
2013
Number Percent Number Percent Number Percent
Child
2,133
62%
1,923
62%
1,749
64%
Elderly
369
11%
317
10%
325
12%
15
Output Example
Service Type
Phone
Mail
Office
Home
Total
Enrollments
731
14,748
1,141
10
16,630
Recertifications
257
4,663
299
-5,219
16
Output Example
Payment Type
Payment Type
Minimum Payment
Percent of Bill Payment
Percent of Income Payment
Annualized Average Payment
Agency Selected Payment
Enrollments
11%
54%
16%
7%
12%
17
Output Example
Grant Type
Electric
Oil
Natural Gas
Propane
Kerosene
Coal
Total
Number of
Grants
3,350
380
72
43
21
2
3,868
Percent
Payments
Percent
87%
10%
2%
1%
1%
<1%
100%
$783,576
$106,469
$19,418
$9,911
$5,982
$738
$926,094
85%
11%
2%
1%
1%
0%
100%
18
Output Example
Measures Installed
Measures
2011
2012
#
%
#
%
Audit
743
98%
352
95%
LIURP Intake
739
97%
352
95%
Repairs
675
89%
318
86%
Blower Door Test
587
77%
303
82%
Air Leakage Reduction
533
70%
276
75%
Door Work
526
69%
280
76%
Air Sealing
499
65%
282
76%
Efficiency Testing
446
59%
205
56%
HVAC Clean & Tune
395
52%
219
59%
Pipe Insulation
404
53%
213
58%
Energy Education
345
45%
262
71%
Window Repair/Replace 378
50%
185
50%
2011
#
%
Water Measure
327 43%
Attic Insulation
320 42%
Lighting
248 33%
Mileage
330 43%
Dryer Work
231 30%
Ventilation
160 21%
Wall Insulation
159 21%
Thermostat
100 13%
Insulation
96 13%
Header Insulation 140 18%
Attic Preparation 83 11%
Refrigerator
56 7%
Measures
2012
#
%
197 53%
204 55%
148 40%
217 59%
165 45%
113 31%
95 26%
100 27%
43 12%
68 18%
44 12%
20 5%
19
Outcomes
• Reduction in bill
• Reduction in energy burden
20
Outcome Example
Burden Reduction
2009 National LIHEAP Survey
PreLIHEAP
PostLIHEAP
≤ 5%
13%
37%
6% - 10%
32%
29%
11% - 15%
19%
17%
16% - 20%
15%
8%
21% - 25%
7%
4%
>25%
14%
7%
21
Outcome Example
Burden Reduction
2009 National LIHEAP Survey
Child
NonDisabled
Under 18 Vulnerable
All
Senior
Pre-LIHEAP
16%
14%
17%
16%
16%
Post-LIHEAP
11%
9%
11%
12%
10%
22
Outcome Example
Heat Restoration
2009 National LIHEAP Survey
Restored Heat Restored Heat Restored Heat
Due to Electric/ Due to Fuel Due to Broken
Equipment
Gas Shut Off Running Out
61%
69%
58%
23
Impacts
• Bill payment coverage rates increased
• Service terminations declined
• Energy usage declined
24
Impact Example
Customer Survey
Has the winter temperature improved, worsened, or stayed
the same since receiving weatherization services?
Change in Winter Temperature
Improved
Worsened
Stayed the Same
Did Not Notice a Difference
Total
Number of Respondents
19
1
0
8
28
25
Impact Example
Arrearage Impacts
Pilot Payment Program
ALL GRAD
Treatment Group
Net
Gross Change
#
Pre Post
Change
561 $276 $251 -$26** -$119**
1 – Graduated Discount
213 $257 $254
-$3
-$96**
2 – Discount &Audit
170 $332 $289
-$43*
-$136**
3 – Discount & Counseling
178
-$36** -$129**
$246 $210
26
Impact Example
Service Terminations
Service Terminations
Utility 1
Utility 2
Utility 3
Utility 4
Pre
5%
14%
15%
4%
Post
7%
8%
4%
2%
Gross Change
1%
-6%**
-10%**
-3%**
Net Change
-1%
-6%**
-10%**
0%
27
Impact Example
Usage Impacts
Usage
Obs.
Savings
Net Savings
Pre
Post
kWh/
ccf
%
kWh/
ccf
%
Electric
Baseload
62
14,108
13,475
633
4.5%
915
6.5%
Electric
Heating
43
27,808
26,343
1,465*
5.3%
1,295
4.7%
Gas
Heating
471
1,588
1,411
177**
11.2%
156**
9.8%
28
Impact Example
Major Measures
# Major
Measures
Usage (ccf)
Savings
Obs.
Pre
Post
ccf
%
0
58
1,575
1,408
167**
10.6%
1
198
1,555
1,475
80**
5.1%
2
254
1,543
1,380
162**
10.5%
3
167
1,594
1,368
226**
14.2%
4 or more
53
1,610
1,340
271**
16.8%
All
730
1,565
1,402
163**
10.4%
29
Impact Example
Cost-Effectiveness
Air Sealing/
Leakage Reduction
Attic Insulation
Wall Insulation
Misc. Insulation
Window Work
Thermostat
ccf
Saved
Mean
Cost
Payback
Years
$/ccf Saved
Lifetime
7.68
$509
13
$1.25
4.68
8.27
6.81
51
57
$915
$1,520
$1,190
$834
$142
21
12
15
16
2
$2.06
$1.16
$1.41
$1.58
$0.24
30
DATA SOURCES
31
Agency Records
•
•
•
•
Most accessible
Should be put in a database
May not be needed if good program database
Data
– Customers served
– Characteristics – income, poverty level, elderly,
children
– Services provided
32
Public Use Data
• Available for free download
• Characterize eligible population in service
territory
• Programming skills needed
• Data
– Number eligible
– Geography
– Characteristics – income, poverty level, elderly,
children, language
– Energy costs
33
Customer Survey
•
•
•
•
Real time feedback
Requires staff time
Document methodology
Data
– Customer characteristics
– Satisfaction
– Self-reported impacts
34
Program Database
•
•
•
•
Program manager – state or utility
Canned reports
Queries
Data
– Customers served
– Characteristics – income, poverty level, elderly,
children
– Services provided
35
Utility Data
•
•
•
•
Difficult to obtain
Easier for utility managed program
Requires software and programming skills
Data
–
–
–
–
–
Customer type – heating, water heating, baseload
Energy usage
Energy bills
Customer payments
Energy assistance
36
PROCESS
37
Process
•
•
•
•
•
Start with available data
Identify performance measures
Determine additional data sources
Collect additional data
Develop additional performance measures
38
SUMMARY
39
Summary
• Performance measurement overlaps with
evaluation
• Start with program goals
• Work with available data
• Identify ways to enhance data
• Measure performance over time
• Identify areas for improvement
• Impact measures require more data and analysis
40
Contact
Jackie Berger, Ph.D.
President and Co-Founder
APPRISE
32 Nassau Street, Suite 200
Princeton, NJ 08542
609-252-8009
[email protected]
www.appriseinc.org
41