Michael Li, Senior Policy Advisor
Office of Energy Efficiency and Renewable Energy
U.S. Department of Energy
What is Green Button?
Common-sense idea that electricity customers should be
able to download their own energy usage information in a
consumer- and computer-friendly format.
Source:
Who is implementing green button?
Utilities and electricity suppliers in 24 states across various regulatory regimes have
committed to provide 30 million US homes and businesses Green Button data.
Commitment to Green Button:
American Electric Power
Austin Energy
Baltimore Gas & Electric
CenterPoint Energy
Chattanooga EPB
Commonwealth Edison
Glendale Water and Power
National Grid
Oncor
PECO
Pepco Holdings
PPL Electric Utilities
Pacific Power
Rocky Mountain Power
Southern California Edison
Virginia Dominion Power
Utilities & Electricity Suppliers
with Green Button today:
• NSTAR
(almost 10 million homes)
• PG&E
NSTAR
PG&E
Reliant
SDG&E
TXU Energy
•
•
•
Reliant
SDG&E
TXU Energy
Green Button Facts
What it is:
• Energy usage information in a common XML format (NAESB ESPI data std)
• PG&E example: how much electricity (kWh) did one metered customer
consume every hour for the last year
Markets: Residential, commercial and industrial
Type: initially electricity, but the data standard is extensible to gas and water data
Timeliness: Not real time data, but 24 hour-old back office data.
Time interval:
• Again the data standard is extensible and could include any interval of data.
• Most will provide 15 minute interval or hourly data.
• However, some will provide monthly data, too.
Metering system:
• You don’t need to have AMI to participate. AMR works, too.
Transfer of data:
• Green Button, Download My Data – goes directly from the utility to the
customer; most will implement this version.
• Green Button Connect - automated data transfer from the utility to a third
party with customer authorization is the 2nd part of the data std; may be
implemented as early as this year in 1-2 states.
Helping to spur new innovation and
Green Button Apps
Green Button
Apps!
http://openei.org
http://appsforenergy.challenge.gov/
Apps for Energy Winners
Best Overall App Grand Prize: Leafully
Location: Seattle, Washington
Leafully, helps utility customers visualize their Green Button data, as a variety of units, such as
the amount of trees needed to offset an individual’s energy usage.
Best Overall App Second Prize: Melon
Location: Washington, DC
The app uses Green Button to evaluate the energy performance of commercial buildings.
Best Overall App Third Prize: VELObill
Location: New York, NY
VELObill app helps makes it easier for utility customers to view their energy usage, measure
whether it is high or low, and compare it to that of their peers.
Best Student App Grand Prize: wotz
Location: Irvine, CA
Best Student App Second Prize: Budget It Yourself
Location: Cleveland, OH
http://energy.gov/articles/first-winners-announced-apps-energy-competition-0
SEE Action Working Groups
Working Group Leadership
Susan Ackerman, OR PUC
Vaughn Clark, OK SEO
Todd Currier, WA SEO
Jennifer Easler, IA Consumer Advocate
Joshua Epel, CO PUC
Jim Gallagher, NY ISO
Bryan Garcia, CT Clean Energy Fund
Frank Murray, NY SEO
Pat Oshie, WA PUC
Phyllis Reha, MN PUC
Cheryl Roberto, OH PUC
Janet Streff, MN SEO
Keith Welks, PA Treasury
Malcolm Woolf, MD SEO
7
Evaluation, Measurement,
and Verification (EM&V) of
Residential Behavior-Based
Energy Efficiency Programs:
Issues and Recommendations
June 25, 2012
Michael Li US Department of Energy
Annika Todd Lawrence Berkeley National Lab
This information was developed as a product of the State and Local Energy Efficiency Action Network (SEE Action), facilitated by
the U.S. Department of Energy/U.S. Environmental Protection Agency. Content does not imply an endorsement by individuals or
organizations that are part of SEE Action working groups, or reflect the views, policies, or otherwise of the federal government.
Outline: Evaluation of
Behavior-Based Programs
• What is a behavior-based energy program?
• Why is evaluation of these programs hard?
• Why is designing a program as a “randomized
controlled trial” (RCT) so important?
www.seeaction.energy.gov
10
What is a behavior-based energy program?
• Programs that affect the way that consumers use energy
without using traditional methods, such as prices and rebates
• Instead, use simple psychological levers or information to
change behavior
• Example 1: Comparing your energy use with your neighbors
• Example 2: Providing real-time information about energy use
• Other examples:
• Competitions, rewards: Turning energy use into a game
• Education / Outreach: Information about energy behavior
• Display of feedback: Simplify / Framing
www.seeaction.energy.gov
11
What are the potential
benefits and concerns?
• Potential Benefits
• In theory, potentially cheap to implement and result in
significant energy savings  cost effective
• As a result, increasingly being adopted nationwide
• Potential Concerns
• In reality, these programs are relatively new and evidence
of energy saving effects is unclear
• Potential for unsubstantiated claims
www.seeaction.energy.gov
12
Why is evaluation crucially important?
 It is very important to measure effect of
these programs
• Need to gain information about how well different
types of programs work
• Are the estimates energy savings valid for utilities to
claim savings?
www.seeaction.energy.gov
13
Why is evaluation of these programs hard?
• Strong problems of “selection bias”: households who
join (choice, screening) are fundamentally different
Join
Population
Didn’t
Join
• Observed differences might be due to program; might
difference between groups
• Selection bias can skew the results of the evaluation
www.seeaction.energy.gov
14
Why is evaluation of these programs hard?
• Energy programs “selection bias”: households who
opt-in may be more energy conscious
Opt-in
Don’t opt-in
• Observed difference in energy use might be due to
the program; but might be difference between groups
www.seeaction.energy.gov
15
Why is evaluation of these programs hard?
• It may be more difficult to measure the impact of
behavior-based programs correctly (in contrast to
other programs such as appliance rebates)
– Impacts vary significantly between households
– Within a household, hard to disentangle changes in
overall energy usage between program, other factors
– Savings are relatively small: often 1-5%, so if an
evaluation is biased, large implications
www.seeaction.energy.gov
16
Why is evaluation of these programs hard?
 Bad evaluation could lead to bad policy
decisions
www.seeaction.energy.gov
17
SEE Action Report
• “Evaluation, Measurement, and Verification (EM&V)
of Residential Behavior-Based Energy Efficiency
Programs”
• Provides guidelines and best practices for
– Program design
– Program analysis and evaluation given design
– Provides rankings for different methods
• Target audiences:
– Senior managers responsible for overseeing and reviewing efficiency
program designs and evaluations
– Practitioners, evaluation professionals, and staff responsible for
reviewing efficiency program designs and evaluations
www.seeaction.energy.gov
18
Why is designing a program as a randomized
controlled trial (RCT) so important?
• Primary recommendation – a well designed, RCT
program results in:
– Transparent, straightforward analysis
– Robust, accurate, valid program impact estimates
– High degree of confidence in program effectiveness
• Why?
– RCT means that households are assigned to the
program randomly (as opposed to household choice or
screening criteria)
– Solves selection bias
www.seeaction.energy.gov
19
Why is designing a program as a randomized
controlled trial (RCT) so important?
• If RCTs are not feasible, recommendations for
acceptable “quasi-experimental” methods
– More opaque, difficult, complex analysis
– Quasi-experimental methods try to correct for
selection bias
– Lower degree of confidence in validity of savings
estimates
www.seeaction.energy.gov
20
Other Key Recommendations
• Problem: Potential conflicts of interest
– Recommendation: Third-party evaluator
transparently defines and implements program
evaluation, assignment to control and treatment
groups, data selection
• Problem: The same savings may be claimed by two
programs (e.g., a behavioral program & appliance
rebate program both claim savings from appliances)
– Recommendation: Estimate and account for this
“double counted savings” overlap to the extent
possible by comparing control to treatment groups
www.seeaction.energy.gov
21
Recommendations for the Future
• The hope is that in the future, we will have conclusive
evidence about the effectiveness of different types of
behavior-based programs
• Move away from RCTs
• We are not yet at this point…
www.seeaction.energy.gov
22
Questions?
• Main point: evaluation of behavior-based programs is
difficult, but if the program is designed in the right
way (using a RCT) then we can be confident that the
evaluation of the program’s energy savings is valid
• Many guidelines and technical recommendations in
the report:
– SEE Action website, www.seeaction.energy.gov
– Lawrence Berkeley National Lab website:
behavioranalytics.lbl.gov
Mike Li: [email protected]
Annika Todd: [email protected]
www.seeaction.energy.gov