Market design for a decarbonized electricity market
Anthony Papavasiliou1, Alex Papalexopoulos2, Shmuel Oren3
Abstract
This white paper sets forth a series of electricity market design proposals that are targeted towards
supporting the transition of European electricity markets to full de-carbonization. The proposed reforms
are necessitated by the large-scale integration of renewable energy resources in decarbonized power
systems, the required mobilization of distributed flexibility, and the need to properly value services in a
market for electric power which is characterized by near-zero marginal cost. We focus on four key
elements: (i) the central role of the real-time market and the need to signal scarcity in order to properly
remunerate ancillary services; (ii) the alignment between real-time market and forward market products,
in order to permit the back-propagation of ancillary service value to long-term investment signals; (iii) the
proper demarcation of transmission and distribution system operations and the development of organized
Distributed Energy Resources (DER) local markets in the distribution grid in order to permit the
mobilization of distributed flexibility of DER assets connected at the edge of the grid aimed at increasing
the penetration of renewable generation in the grid while ensuring grid security and resilience at the
lowest cost; and (iv) the mobilization of demand-side flexibility through aggregator business models that
draw on the successful experience of the information technology and telecommunications sectors. We
discuss the transition of the current market towards our envisioned prototype, and we illustrate the
effectiveness of our proposals through a series of case studies.
1. Context, Key Drivers, and Challenges
The steadiness with which the European Union has pursed its ambitious Roadmap 2050 policy objectives
[EC11] is an encouraging sign in the backdrop of a highly uncertain future for global energy policy. Despite
the upheaval of environmentally-minded policies in the United States which are imminent following the
Trump election, the damage that the Trump administration can inflict on the global renewable energy and
electricity market transformation agenda is expected to be limited as a result of major technological
innovations, global market forces and the general momentum of EU policy makers towards decarbonization and decentralization in electric power [B17]. The forward-looking policies of the European
Commission are targeting a transformation of legacy electric power systems and electricity markets to
systems that are increasingly decentralized and reliant on renewable energy supply and consumer
flexibility.
In this context, we identify the following major challenges for the decarbonized European electricity
markets of 2050: (i) Reliable investment signals in a system with near-zero marginal costs, where value is
shifting away from energy and into services. (ii) Efficient and reliable short-term operations in the presence
of highly variable, uncertain and limitedly controllable renewable power supply, through the
harmonization of the real-time market and short-term forward (day-ahead) markets. (iii) Active
management of distribution grid resources through a clear demarcation of the roles and responsibilities
of transmission system operators, distribution system operators, and aggregators and the development of
local DER organized markets. (iv) Mobilization of consumer flexibility at the retail and commercial sector.
1
Université catholique de Louvain, Belgium, [email protected]
ECCO international, USA, [email protected]
3
University of California at Berkeley, USA, [email protected]
2
These challenges are interdependent, and a holistic approach is required in confronting them. We propose
four pillars, one for attacking each of the four challenges, in section 2.
Challenge 1: Migration of value from energy to ancillary services. The introduction of renewable energy
resources in electric power systems is exerting downward pressure on energy prices, while increasing the
value of reserves. Fast-moving flexible resources that can respond to the increasing needs of the system
for ramp capability often exhibit high marginal costs. The large-scale integration of near-zero marginal cost
renewable resources is placing these flexible technologies (e.g. combined cycle gas turbines) under
financial strain and threatening their retirement from the market. Paradoxically, these are the resources
that are best suited for balancing in a future system with large-scale renewable energy penetration. This
paradoxical retirement, an artefact of the missing money problem [S03], can be addressed through the
correct valuation of reserve capacity in real-time markets, such that the real-time value of this reserve
capacity in moderating loss of load probability becomes consistent with system operator practices [H05,
H13]. In order for this valuation to be meaningful, it should reflect the uncertainty of net load changes, the
loss of load probability, minimum contingency reserve requirements, it should maintain consistency
between energy and reserve prices, and ideally it should reflect the co-optimization of energy and
reserves. Finally, it should provide a consistent representation of any locational differences in valuing
reserves. This correction in the valuation of reserve capacity can be embedded within an integrated
common European energy-only market, although the co-existence of this correction in scarcity pricing with
capacity remuneration mechanisms is not precluded. The design of a properly functioning real-time
market that simultaneously trades energy, ancillary services and transmission capacity is therefore the
first step towards generating healthy long-term investment signals, while safeguarding the harmonization
of the integrated European energy market.
Challenge 2: Harmonization of short-term operations. Once a properly functioning real-time market has
been put in place, it is necessary to design a consistent forward day-ahead market that trades the same
set of products and is based on the same modeling assumptions in order to preclude gaming opportunities
and operational inefficiencies. This implies a need for the pooling and coordination of resources across
zones in order to balance out the highly variable fluctuation of renewable supply. It is therefore necessary
to define a consistent set of reserve products across European zones, which is certainly feasible given the
recent efforts of Central European zones in harmonizing the definition of primary, secondary and tertiary
reserves [ER14]. This can be achieved through the simultaneous commitment of energy, reserves and
transmission capacity in the day-ahead time frame, which would enable the cross-zonal mobilization of
resources for the balancing of renewable supply unpredictability. Congestion management costs, which
have recently elevated substantially in a number of European systems [AP17], would be contained as a
result of such a reform, and the free-riding of networks through loop flows would be preempted through
the pricing and allocation of transmission capacity which is consistent with physical constraints.
Challenge 3: Active management of distribution. To a significant extent, renewable resources are being
sourced at the low-voltage distribution system. Germany, for example, which represents one of the largest
European systems, has integrated 70 GW of solar photovoltaic power and wind power by 2013, in a system
with an installed capacity of 192 GW. A significant portion of German solar installed capacity corresponds
to rooftop solar panels. At the same time, various storage technologies (such as Tesla power walls) have
experienced a remarkable breakthrough in the market. The potential electrification of the transportation
sector would correspond to a further paradigm shift in energy systems with profound implications on the
flexibility of distribution grid resources. The increasingly important role of distribution system resources
in the power system operations paradigm is creating a host of challenges in terms of reverse power flows,
reactive power management, voltage constraints, and the aggregation of enormous numbers of
distribution system resources in wholesale operations. The traditionally passive role of distribution system
operators has afforded little room for coordination with transmission system operators, and the shift
towards a system that actively dispatches distributed resources in a scalable and non-intrusive fashion will
require the clear definition of the roles and responsibilities of distribution system operators, transmission
system operators, and aggregators, as well as a clear definition of the coordination among these entities.
Further, the efficient treatment of these challenges will require the development of organized local
markets that will optimally dispatch these local assets at the edge of the distribution grid to provide for
the maximum penetration of renewable resources and secure the resilience of the grid at the lowest cost.
Challenge 4: Mobilization of consumer flexibility at the retail and commercial sector. Once the
demarcation of transmission and distribution system operations has been clearly defined, it is possible to
identify the value streams for demand-side flexibility and aggregate this flexibility into wholesale and
distribution market operations. The requisite flexibility for balancing the operation of highly renewable
systems can be sourced to a significant extent from consumer flexibility. The majority of this flexibility can
be found in the residential and commercial sector [G14]. Serendipitously, the most flexible consumption
tasks of residential consumers (such as wet appliances, i.e. dish washers, washing machines, and tumble
dryers) are the ones that are also the most energy-intensive. The economists’ first-best ideal of real-time
pricing has failed to mobilize flexibility in the commercial and residential sector, largely because consumers
are not willing to handle the financial risk and informational overload implied by real-time pricing. On the
other hand, direct load control is perceived as being too intrusive. Our proposal sets forth an approach
based on product differentiation that combines the best of price-based and quantity-based control,
inspired by the successful example of deregulation in telecommunications and information technology
[PBF13].
Vision. Maintaining the existing level of reliability implies additional operating costs, a significant amount
of discarded power, and ultimately a limit on the amount of renewable power that can be integrated within
reasonable cost [V05]. This is indicated in the left part of figure 1, which depicts the trade-off between
reliability of service and the level of renewable energy integration. The mobilization of distributed flexible
resources can mitigate this trade-off, but only to a certain extent, since beyond a certain threshold of
renewable energy integration the flexibility of distributed demand is not sufficient to absorb the random
fluctuations of renewable power. This is indicated by the decay of the blue curve in the left graph of figure
1. Our proposal builds upon the idea of designing power systems that deliver the highest possible quality
of service within a certain reasonable cost, rather than almost full reliability without regards to cost. The
flexibility of distributed demand is key in achieving the objective of economical large-scale renewable
energy integration, since flexible tasks can absorb stochastic and highly variable renewable supply without
experiencing a significant deterioration in quality of service. The vision of our proposal is indicated by the
blue curve in the right graph of figure 1, whereby the mobilization of demand-side flexibility results in
substantial levels of renewable energy integration by trading off an acceptable deterioration in quality
of service with a transition to sustainable long-term energy supply.
Figure 1: The vision of our proposal.
2. Proposal
Our proposal provides a holistic treatment of electricity market design. We first outline our proposal for
real-time market design, which is the cornerstone of a properly functioning market. Once the real-time
market has been clearly defined, earlier forward (e.g. intraday and day-ahead) markets follow the same
blueprint, whereas longer-term forward markets can follow a looser, more decentralized organization. The
introduction of distribution systems and organized distribution markets in this market design is then
tackled, followed by the definition of an aggregator business model that respects the desire of consumers
for privacy, simplicity and control.
Pillar 1: Real-time markets and scarcity pricing. The large-scale integration of renewable resources shifts
value away from energy markets and into markets for ancillary services, and especially reserve markets.
The central role of reserve markets in highly renewable systems implies that reserves need to be valued
accurately by the demand side. Due to the fundamental arbitrage relationship that links energy market
opportunity costs to reserve capacity prices, a consistent pricing of reserve capacity will result in an
adjustment of energy prices so as to accurately represent real-time scarcity. It is therefore important to
value reserve capacity in real-time markets in a way that is consistent with transmission system operations:
increments of reserve under tight conditions are more highly valued by system operators than increments
under comfortable system conditions, because they have a greater effect in reducing loss of load
probability under scarcity. This calls for a co-optimization framework of energy and reserve products in
addition to representing any locational differences in valuing reserves.
Scarcity pricing already exists in certain European systems, for example in the Belgian market the
imbalance price is corrected by a constant offset whenever the system is exceedingly long (above +120
MW) or short (below -120 MW). There is a sound economic theory [H05, H13] that can be developed in
order to support this form of scarcity pricing. The fundamental ingredient of such a theory requires
introducing a reserve capacity demand function. The introduction of operating reserve demand functions
is predicated on the simultaneous clearing of energy and reserves, and the trading of real-time reserve
capacity, which is currently absent in European energy markets. The resulting scarcity adder which is the
real-time price for reserve capacity also corrects the real-time price for reserve energy, and is a price signal
which rewards resources that support the system in real-time balancing while penalizing those resources
that cause real-time imbalances. The correction of energy prices under conditions of scarcity can occur
even if bids are mitigated due to regulatory concerns over the exercise of market power. The approach
respects the fundamental design of an energy-only market, thereby safeguarding the integration of the
common European energy-only market. At the same time, the design of such a demand function, coupled
with a consistent day-ahead market design (discussed in more detail in the next paragraph), ensures the
back-propagation of long-term investment signals that can support the expansion of much-needed flexible
capacity. There is nothing to preclude the coexistence of the proposed mechanism with capacity
remuneration schemes. However, the successful design and implementation of the proposed mechanism
would render any capacity remuneration scheme less critical and, therefore, would mitigate some of its
unintended consequences. Ultimately, the proposed design results in price jumps of lower amplitude, and
more predictable frequency, which provide a more reliable investment signal for investors and fewer
controversies among stakeholders.
Pillar 2: Alignment of real-time and day-ahead markets. The design of a harmonized real-time market
that simultaneously clears energy, reserve capacity and transmission capacity, needs to be accompanied
with a consistent day-ahead market design. Lack of consistency creates gaming opportunities, and
introduces operating inefficiencies. Under some conditions, as experience from the USA market indicates,
this lack of consistency between the markets can be detrimental to the market and more importantly to
the reliability of the system at a huge expense to consumers. Therefore, we propose a transition from the
existing day-ahead power exchange towards the simultaneous day-ahead auctioning of energy, reserves
and transmission capacity in the day-ahead time frame. The integration of reserves and energy in market
clearing allows for a more granular sizing of reserves, a more efficient commitment of thermal generators,
and a correct pricing of reserve capacity which becomes the main service offered by thermal generators
to the grid. The latter creates the opportunity for the introduction of improved scarcity pricing, as
discussed under pillar 1. The clearing of transmission with a proper representation of physical constraints
within the auction allows the mitigation of congestion management costs, the coordination of resources
from different areas to balance out local renewable supply fluctuations, and the seamless sharing of
reserve capacity.
Pillar 3: Coordination schemes for TSO-DSO operations. Although pillars 1 and 2 clarify how a wholesale
market for energy, reserves and transmission would operate, the corresponding market design still
conforms to a passive utilization of distributed resources. Under such a paradigm, distributed consumers
absorb power at will, and distributed solar resources inject power to the distribution system whenever it
is available. Instead, with the advent of distributed storage, either directly through distributed storage, or
indirectly through electric vehicle proliferation, the system can be handled more intelligently, thereby
deferring infrastructure upgrades4 and improving operational efficiency dramatically. Whereas the status
quo places all the intelligence in, and sources all flexibility from, the high-voltage transmission grid
resources, we envision a coordinated, active dispatch of transmission and distribution resources. This
vision presents a range of challenges, due to the fact that nonlinear distribution network constraints need
to be accounted for, and also due to the vast number of distributed resources.
The proper coordination of transmission and distribution resources requires a clear definition of how
transmission system operators will coordinate with distribution system operators in a highly renewable
energy system. The focus here is on being able to utilize distributed resources as reserve, while respecting
the constraints of the distribution network. A range of TSO-DSO coordination schemes can be envisioned,
4
For an overview of the benefits of intelligent distribution system control in terms of deferring investment in
distribution system infrastructure, see [PES17] for a range of case studies in California and New York.
each of which would need to be explored more carefully for its relative merits and disadvantages. Under
a fully coordinated dispatch of transmission and distribution level resources, the power flows on the
transmission level are represented through a linear model and the power flows on the distribution level
represented through a second order conic relaxation of the branch flow model for optimal power flow
[FL13]. This modeling approach properly accounts for the non-linearity of distribution grids, reactive power
flows, voltage constraints, and real power losses, features which cannot be ignored at the distribution
level, while preserving a computationally tractable model. The idea of the fully coordinated dispatch of
distributed resources with transmission resources is to have the transmission system operator operate
reserves at both the transmission as well as the distribution level while accounting for distribution level
constraints. Although this may appear as a daunting task due to the size of the problem and its nonlinearity, recent evolutions in decomposition algorithms render this vision possibly achievable with highly
distributed computing infrastructure [K+13]. From a market design point of view, this effectively
corresponds to the simultaneous trading of distribution network capacity, reactive power, and reserve
capacity in a simultaneous auction that is cleared by the system operator and which synthesizes one
aggregate distribution locational marginal price which accounts for the contribution of all of these scarce
resources to the formation of real power prices at individual distribution nodes [P17]. Alternative to the
fully coordinated dispatch of distributed resources with transmission resources framework, is the
development of local organized DER markets operated by DNOs or third parties which coordinate their
operations with wholesale markets. In this latter market architecture, DER assets can participate in
multiple markets at the same time with the same action and, given their versatility, they can develop
multiple revenue sources in the energy supply chain.
Pillar 4: priority service contracts. With a properly functioning short-term market in place, and a clear
definition of proactive distribution system operations, it is possible to determine new value streams for
demand-side flexibility, provided consumers are confronted with scalable aggregator business models. Our
proposed solution for mobilizing consumer flexibility is based on the premise that consumers perceive
electricity as a service, instead of a commodity that they are willing to purchase in a real-time market.
Inspired by the successful paradigm of other sectors, including telecommunications and information
technology, we propose a paradigm which combines the best of both price-based and quantity-based
control, while respecting the requirement of consumers for privacy, control, and simplicity. As in the case
of successful business models for telecommunications, consumers value simple and understandable
offerings. We propose an offering of electricity at various levels of reliability, which we argue consumers
can value accurately, as opposed to their valuation for increments of power in real time.
In practice, our proposal is implemented as follows [PBF13]: consumers set color tags on each plug in their
home, in order to prioritize the consumption of power in different devices. Inspired by traffic lights on
congested highways, we propose the following tagging of plugs: red indicates devices that have top priority
and cannot be interrupted by aggregators. This indicates to aggregators that they ‘cannot proceed’ with
interrupting devices. Orange indicates devices that can be interrupted, but only infrequently, thereby
indicating to aggregators that they can ‘proceed with caution’. Green indicates ‘proceed freely’, i.e. devices
that can be interrupted if there is a shortage in supply. Slices with lower reliability are priced lower, thereby
allowing consumers to pocket the benefits of their flexibility, something which is largely impossible under
existing retail tariffs. At the same time, consumers preserve control of their devices, because they can
decide how to color-tag devices throughout their home, thereby maintaining control on the assignment
of reliability within the household. Aggregators can collect this information over hundreds of thousands
of households, and apply stochastic distributed control strategies that allow for a rapid regulation of
aggregate residential and commercial consumption, which is suitable for a range of grid services, including
intertemporal energy price arbitrage and the provision of the full range of frequency control.
The challenge on the end of the aggregator is to design a menu with asymmetric information (i.e. without
knowing how individual consumers value power, thereby respecting privacy), while ensuring that the
reliability that a highly renewable system can afford is the reliability that the consumers are entitled to
through their reliability choices. More tangibly, prices that are too low will result in all consumers selecting
very high reliability, and this would be futile since this is the current paradigm from which we need to
evolve. On the other extreme, prices that are too high discourage consumers from enrolling to the menu
of the aggregator voluntarily, which clearly is also not desirable. There exists solid economic theory to
guide the optimal design of such reliability-differentiated menus, either through capacity-based tariffs
[CW87], or capacity and energy-based tariffs [C+86], while having the aggregator rely on aggregate
statistical information about the valuation of the population for power, a prerequisite which is clearly
realistic.
3. Evolution and Implementation
The proposed market evolution aims at improving the transition to a decarbonized energy system by (i)
improving the long-term security and reliability of flexible generation supply, (ii) improving market price
signals to encourage additional generation and/or transmission investment at the proper locations, (iii)
pricing all scarce resources such as transmission facilities and increasing transparency, (iv) improving
dispatch efficiencies to yield lower overall cost of power supply, and (v) mobilizing DERs at the edge of the
grid to provide various services to the grid at the lowest cost.
The current market evolution to a flow-based market coupling framework is a step in the right direction.
It allows for the relaxation of overly stringent restrictions on cross-border trade, and therefore enables a
more efficient use of the existing transmission system. This is especially the case in a meshed transmission
system, such as the one in continental Europe, where bilateral ATCs must be determined low enough for
the transmission system to withstand a “worst-case” congestion scenario on all trading directions. In
addition, with the flow-based approach, the capacity allocation follows the distribution of the power flows
and a better alignment of the commercial model with the physical system. However, even that framework
falls short of the challenges we face in transitioning to a low-carbon energy system. The massive
penetration of renewables has already created and will continue to create generation localisation issues
and congestion on transmission and distribution grids, due to a specific geography or other local
conditions.
As a practical implementation issue, the transition to a much smaller bidding zone configuration is an
important step in addressing these issues. Market power or liquidity issues from such a transition do not
create additional challenges (as some people content) as the creation of zonal hubs is an integral aspect
of such a transition. Eventually we need to include grid modelling in the market architecture clearing and
ensure the consistency of the day-ahead and real-time markets while allowing for the simultaneous
procurement of energy, reserves and transmission.
In this proposal we envision a gradual market evolution and implementation approach. The flow-based
market coupling framework could be the starting point. Transition to smaller bidding zones with inclusion
of grid modeling in market clearing should be the next task. Based on experience this is the most difficult
and time consuming task but given regulatory support it can be achieved within 3 years. This task will form
the foundation of reconciling the commercial model with the physical model and ensuring consistency of
the day-ahead and real-time markets. The simultaneous procurement of energy, reserves and
transmission should be the next task. Given our practical experience we argue that this task can be
achieved within a 2-year timeframe. The technical and design problems (such as data quality issues,
market processes, algorithmic and performance issues, etc.) are clearly understood and the key deciding
factor is again regulatory support and shareholder participation. Finally, the coordination of the
transmission and distribution systems, the development of the communication protocols between them,
the mobilization of DER assets at the edge of the grid, the development of the appropriate business models
for the DER aggregators along with the introduction of the organized local energy market in the
distribution level is the last task that can be implemented with a 5-year timeframe assuming active
participation of regulators and shareholders.
4. Case Studies
The following case studies illustrate the implementation of the four pillars of our proposed design. The
case study for scarcity pricing in real-time markets is focused on the Belgian market. The case study for
the alignment of real-time markets with day-ahead markets covers the entire CWE region. The case study
for TSO-DSO coordination is illustrated on a test system. Priority service pricing is illustrated for the Belgian
market.
Real-time markets and scarcity pricing: the Belgian case [PS17]
Belgian power production capacity connected to the regular grid amounts to 14765 MW. Between
September 2014 and mid-October 2014, four nuclear units in the Belgian system were retired from service
simultaneously due to technical malfunctions, amounting to a total unplanned outage of approximately
4000 MW. In light of these events and the paradoxical retirement and mothballing of flexible capacity in
Belgium, the Belgian Regulatory Commission for Electricity and Gas (CREG) issued an investigation about
whether adequate incentives are in place in order to attract investment in flexible power generation in
the country. In this case study we analyze how electricity prices in the Belgian market would be impacted
if scarcity price adders based on operating reserve demand curves were introduced in the market. We
specifically simulate the Belgian market from January 2013 until September 2014, and compute the profits
of CCGT units with and without scarcity pricing.
The goal of our study is to determine how the introduction of scarcity adders could impact the
remuneration of flexible plants in the Belgian electricity market. The computation of real-time scarcity
adders requires (i) knowledge of the marginal cost of the marginal unit in the real-time market, (ii)
quantifying the net load uncertainty which the system faces within a given time horizon in order to
compute the loss of load probability as a function of reserve, and (iii) quantifying the amount of reserve
that can be made available within the time horizon of reserve delivery.
The data that was provided for the study includes day-ahead and real-time prices of the Belgian market,
hourly production by fuel in both the day-ahead and real-time market, demand in the day-ahead and realtime market, the amount of activated reserve energy (instead, the amount of reserve capacity provided
by each unit or the amount of activated reserve energy by unit is not available), production capacity
available by fuel, and imports/exports in the day ahead and in real time over each interconnection.
The marginal cost of the marginal unit is estimated in our analysis by the real-time price. The net load
uncertainty within a fifteen-minute horizon is estimated based on the amount of activated reserve energy.
A normal distribution is fit to the net load uncertainty in order to obtain the loss of load probability
function. The major challenge was to estimate the amount of reserve capacity that is available in real time
within a time horizon of reserve delivery (7 or 15 minutes, depending on whether we are remunerating
secondary or tertiary reserve). This capacity depends on the ramp rates of the specific units that were
actually committed at each given hour of the study. This data was not available to us explicitly. We deduce
this information by using the data that was provided to us in order to build a bottom-up model of the
Belgian electricity market. The Belgian market model that we develop is validated against the data that
was provided to us by comparing its predictions to the actual day-ahead market-clearing price and marketclearing quantity of the Belgian market for the duration of the study. The day-ahead and real-time net
demand faced by thermal units over the duration of the study exhibit a mean absolute error of 172 MW,
which indicates that the day-ahead unit commitment decisions should be close predictors of the units that
actually operate in real time. Therefore, by being able to develop a model that closely emulates the
outcomes of the Belgian day-ahead electricity market we are able to deduce the individual units that were
actually on-line in real time over the duration of the study. This information is then adequate for inferring
the amount of reserve capacity that would be available within a time interval of reserve delivery (7 or 15
minutes), thus enabling us to estimate the scarcity price adder. Previous research on scarcity adders either
ignores individual unit ramp constraints [LB15] (which, we argue, may greatly influence the resulting
adder), or applies the analysis without previously calibrating the market model to actual market outcomes
[ZB13].
The methodology is explained in further detail in figure 2. The validation process of the Belgian market
model is indicated in the left part of the figure, while the simulation of the Belgian market in order to
determine price adders is indicated in the right part of the figure. Details about each part of the analysis
are provided in the appendix of [PS17].
Figure 2: A schematic diagram of the proposed methodology [PS17].
The Belgian market model that we develop is based on a unit commitment and dispatch model. Once our
market model is calibrated, we validate it by comparing its ability to explain observed market prices and
cleared quantities to competing approaches. A successful validation of our model against Belgian market
data would imply that the model could also be used for examining the impact of scarcity rice adders in a
prospective study for future conditions of the system. As indicated in figure 2, such an analysis would
require, as exogenous input, a forecast of real-time system demand and imports (which could be part of
the definition of the scenarios of a prospective study). Note that this is an open-loop analysis, i.e. we do
not account for how the expectation of introducing the scarcity price adder feeds back into the capacity
that is deployed in the market. A closed-loop analysis will be the subject of future investigation.
In order to estimate the profits of individual units, we use the historical energy and reserve prices in order
to estimate revenues and operating costs. We focus specifically on combined cycle gas turbine (CCGT)
units, whose economic viability is questioned despite the fact that these resources are best suited (in terms
of technical capabilities) for providing flexible reserve capacity to the system. The profits of CCGT units are
computed for historical prices as they occurred over the duration of the study, as well as for profits that
would have occurred if the scarcity price adder were applied to the energy price. We use scarcity adders
for the provision of reserves whose respective delivery times amount to 7 and 15 minutes, corresponding
to the delivery time of Belgian secondary and tertiary reserves, respectively.
Table 1: Profitability of CCGT units (January 2013 - September 2014) before and after adding scarcity price
adders, and average adder benefit.
Eleven CCGT units operate currently in the Belgian electricity market. The output of the market model that
we have developed permits a computation of CCGT profits. Table 1 presents the profitability of each unit
before and after the introduction of price adders5. These profits should be compared against the running
investment cost of a typical CCGT unit in order to ascertain the economic viability of CCGT resources. The
running investment cost of CCGT is estimated6 at 5.6 €/MWh. Profits that do not exceed 5.6 €/MWh in the
table are highlighted in bold font in order to indicate that the given unit is not economically viable. The
5
The reported profit accounts for a fixed operating and maintenance cost of 7.04 $/kW-year (EIA 2012 estimate) at
the average 2012 exchange rate of 0.778 €/$.
6
The estimate is based on an overnight cost of 676 $/kW (EIA 2012 estimate), the 2012 average exchange rate of
0.778 €/$, continuous discounting at rate of return of 8%, and an investment horizon of 25 years.
profit in the first column is computed as the profit over the entire duration of the study given historically
realized prices, normalized by the capacity of each unit and the number of hours in the study period. The
profit in the second column is computed in the same way, where prices have been adjusted according to
the scarcity adder. The final column represents the extra profit earned by each CCGT unit due to the
introduction of the adder, normalized by the total output of each unit.
Two notable conclusions can be drawn from the first two columns of table 1: (i) CCGT profits, as estimated
by the methodology set forth in the present paper, are not sufficient for ensuring the economic viability
of any CCGT unit. This observation is aligned with the existing policy debate, which has focused on the fact
that the existing market design is not sufficient for ensuring the economic viability of flexible resources,
although these resources are necessary for supporting the integration of renewable energy resources. (ii)
Adders, as computed in the study, could potentially render the majority (seven out of eleven) of CCGT
units economically viable. This confirms the fact that these resources add value to the system, although
paradoxically the existing market design is pushing these resources out of the market.
In the last column of table 1 we present the average adder benefit accrued by each CCGT unit. This adder
benefit is computed as the difference of the revenue earned by each unit before and after the introduction
of the adder, divided by the total production of each unit over the entire study period. The average adder
for the duration of the study amounts to 5.3 €/MWh. This is the average increase in revenues that can be
expected, for example, by base-load units that produce a constant output. By contrast, the adder benefit
presented in the last column of the table is effectively higher for all CCGT units, and amounts to up to 14.3
€/MWh for CCGT10. Whereas a capacity market would treat CCGT and base-load units identically, the
scarcity pricing mechanism rewards flexible units more handsomely by design. This effect is a result of the
positive correlation of the output of CCGT units with scarcity adders. Stated equivalently, flexible units are
able to increase their output under conditions of system scarcity, and are rewarded accordingly by the
scarcity pricing mechanism.
Alignment of real-time and day-ahead markets: the case of the Central Western European market [AP17]
For this case study, the network was populated using an industrial database of thermal generators which
includes technical and economic characteristics of 656 generating units in the Central Western European
(CWE) region. Our CWE model thus includes Austria, Belgium, France, Germany, Luxembourg, the
Netherlands, and Switzerland. Thermal generators are classified into five groups: 87 nuclear units (85G
W), 144 combined heat and power (CHP) units (40 GW), 272 slow units (conventional thermal units that
are neither nuclear nor CHP, and obey a minimum up and down time greater than 3 hours, totaling 99
GW), 126 fast units (conventional thermal units that are neither nuclear nor CHP, and obey a minimum up
and down time less than or equal to 3 hours, totaling 14 GW) and 27 aggregated small generators (10 GW).
We simulate the CWE region for a typical year in 2014. We used clustering to select 8 representative day
types of load for 2014, corresponding to one weekday and one weekend day for each season. We used
the forecast errors of 2013-2014 to generate samples of real-time renewable energy production.
We simulate system operation of the CWE system under three major policy designs: (i) MC Net Position:
zonal design without real-time coordination among zones, (ii) MC Free: zonal design with real-time
coordination among zones, and (iii) Deterministic UC: our proposed design of simultaneously clearing
energy, reserve capacity and transmission in the day-ahead market. Operations under these three designs
are modeled in two stages. The first stage takes place in the day ahead and determines the commitment
of the slow thermal generators based on a forecast for renewable energy supply. The second stage takes
place in real time and corresponds to the re-dispatch and balancing performed by the system operator
given the realization of multi-area renewable supply. The second stage must respect the commitment
determined for slow thermal generators in the first stage in all policies.
In summary, we report the following results: (i) The benefit of clearing energy, reserves and transmission
simultaneously in the day-ahead market (which is quantified by comparing Deterministic UC with MC Free)
amounts to 294 million € per year for the CWE region; (ii) The additional benefit of real-time coordination
in zonal markets (which is quantified by comparing MC Free with MC Net Position) amounts to 356 million
€ per year for the CWE region.
A centralized market can improve system performance in two ways: (i) by leading to more efficient
commitment of slow-moving generators which need to be committed in the day-ahead, thereby
preventing unscheduled flows in day-ahead markets, and (ii) by improving balancing strategies due to the
full coordination of balancing resources among multiple system operators in real time. The first type of
improvement is relevant for the European market. In contrast, the second type of improvement is
currently relevant for both the continental European system and the wide interconnections in US systems.
Figure 3. Left panel: Adjustment of zonal net position in real time with respect to the day-ahead net
position. A positive adjustment corresponds to a real-time net position that is larger than the day-ahead
net position, and vice versa. Net position adjustments of DE/AT/LX range between -6 GW and 5 GW.
The two effects described in the previous paragraph are illustrated in figures 3 and 4. Figure 3
demonstrates the benefits of sharing real-time balancing among multiple zones. In the case where
balancing resources are not shared, (MC Net Pos), each zone is ‘stuck’ with balancing its zonal imbalance
with its own resources, whereas a coordinated balancing strategy (MC Free) spreads this cost over the
system-wide supply function, which is clearly more efficient. Figure 4 demonstrates the benefits of
scheduling unit commitment in the day-ahead market while properly accounting for Kirchhoff’s laws within
the market clearing. Failing to account for Kirchhoff’s power flows results in the over-commitment of lowcost coal resources in Germany in the day-ahead time frame. This results in the violation of thermal limits
of transmission lines in northern Germany in real time. This congestion is relieved by re-dispatching coal
units downwards, while still incurring their minimum load cost, and by starting up fast-moving thermal
units, which is also expensive. A proper accounting of power flow constraints in the day-ahead scheduling
proposed by our design would preempt such an inefficient commitment in the day-ahead market.
Figure 4: Day-ahead schedule determined by MC Free for a spring weekday at the 17:00–18:00 interval.
Flows implied by the production schedule are infeasible for the real network since they overload lines in the
west of Germany. This infeasible schedule is altered in real time by re-dispatching all generators at D-100
down to their technical minimum and starting up fast units in the surrounding area in order to relieve
congestion.
Coordination schemes for TSO-DSO operations: A CIGRE test system [P17]
Given the vast number of flexible agents that are connected to the low-voltage grid, their scalable
mobilization should naturally rely on a distributed computing architecture. Due to the tight physical
constraints of power system operations, placing the computational intelligence at the lowest possible level
[C+16, K+13] is expected to present implementation challenges, because a minimum amount of
centralized control is an inevitable prerequisite of power system operations which, unlike for example
telecommunications infrastructure, cannot buffer power. Instead, our proposal explores a cloud-based
distributed computing approach whereby the system gathers information on the cloud, computes (in a
distributed fashion), and communicates back to individual devices via the cloud. Crucially, the distributed
nature our proposed algorithms is naturally poised for exploiting recent progress in multi-core processing,
high performance computing and cloud computing [POR15]. The concept is illustrated graphically in figure
5.
Figure 5: Our proposal explores a cloud-based distributed computing approach whereby the system gathers
information on the cloud, computes (in a distributed fashion), and communicates back to individual devices
via the cloud.
With such a computational infrastructure in place, we propose the following market clearing procedure:
(i) a transmission system market that trades real power and reserves, and (ii) low-voltage markets that
clear real power, reactive power and reserves, such that the reserves can be delivered by distributed
resources while respecting distribution line limits and voltage limits.
We present the results of such a market-clearing procedure on a 15-bus radial distribution network which
is based on a CIGRE case study and shown in figure 6. Two flexible resources are located at the root node
(node 0) and node 11, with a marginal cost of 50 €/MWh for the resource at the root, and 10 €/MWh for
the resource at location 11. The flexible resource at the root has unbounded capacity, the resource at node
11 has a capacity of 0.4 p.u. Voltage limits are set uniformly between 0.9 p.u. and 1.1 p.u. The detailed
description of the system is provided in [P17].
Figure 6. Left: The 15-bus CIGRE example. Grey nodes indicate locations of flexible resources. The arrows
indicate the direction of flow of real power for the optimal solution for both the case with and without line
capacity limits. Right table: The distribution locational marginal price in €/MWh (column 2) at each zone
(column 1), and its breakdown in root node marginal cost (column 3), contributions from losses (column 4),
contributions from voltage limits (column 5), and contributions from binding transmission limits (column
6).
The result of the coordinated dispatch of transmission and distribution resources results in marginal prices
that are differentiated by distribution node. We refer to these prices as distribution-LMPs (DLMPs). For
the 15-node case study of our example, these prices are shown in column 2 of figure 6. The table of figure
6 shows how these DLMPs are decomposed into contributions from root node marginal cost, real power
losses, voltage limits, and transmission limits.
The proposed coordinated dispatch of the distribution network results in an active utilization of distributed
resources while accounting for reactive power flow, while respecting voltage limits and line flow limits,
and by pushing real power losses over distribution lines as low as is economically optimal.
Priority service pricing: a case study of Belgium [MPC17], [PBF13]
In this case study, we apply priority service theory [CW87] to the Belgian residential electricity market.
Firstly, we calibrate aggregate demand functions for residential consumers and then use Monte Carlo
simulations of economic dispatch in order to obtain the valuation-reliability curve which describes the
optimal reliability that should be assigned to consumers of different valuation levels. Subsequently, we
develop price menus with a finite number of priority classes. The resulting menu with three reliability
offerings is shown in table 2. The second, third and fourth row correspond to increasing reliability at an
increasing price. For example, the second row corresponds to the ‘green’ color, with a reliability of 37%
with a monthly subscription fee of 5,4 € per kW, and so on.
Color
Reliability (%)
Green
Orange
Red
37
97
100
Monthly subscription fee
(€/kW)
5,4
48,9
66,4
Table 2: A reliability-based menu for the Belgian market.
We have further implemented and tested the proposed demand response approach into the ColorPower
software platform [PBF13]. In a system of devices managed using ColorPower, at any given time each
device (or device behavior mode) is either enabled, meaning that it can draw power freely, or disabled,
meaning that is has been shut off or placed in a lower power mode7. In order to prevent damage to
appliances and/or customer annoyance, devices must wait through a refractory period after switching
between disabled and enabled before they return to being flexible and can switch again (the length of this
period depends on device type). These combinations give four device states (e.g., EF is enabled and
flexible), through which each device moves according to a modified Markov model as shown in figure 7.
Devices move randomly from EF to DR with a certain probability per round which is determined by the
ColorPower architecture, and from DF to ER with a different well-defined probability. Devices move from
ER to EF by a randomized timeout, and likewise for the transition from DR to DF.
7
Note that an enabled device may not actually demand any power: for example, a toaster that is not currently in
use.
Figure 7: Modified Markov model for ColorPower device state switching: states with an “E” are enabled,
“D” are disabled, “F” are flexible, and “R” are refractory.
The control problem for a ColorPower system is shown by the block diagram in figure 8. The system
comprises a set of electrical devices, each of which apportions its demand between priority tiers, where
lower numbered tiers are intended to be shut off first (e.g., 1 for “green” pool pumps, 2 for “green” HVAC,
3 for “yellow” pool pumps, etc.), and where each tier has its own time constants (e.g., 𝑇𝐸𝐹,2 is the fixed
portion of the refractory timeout for Tier 2). The state 𝑠(𝑡, 𝑎) of each device 𝑎 at time 𝑡 is the magnitude
of power demand in each state for each priority tier (e.g., |𝐸𝐹1,𝑎 | is the number of watts of enabled and
flexible demand in tier 1 at device 𝑎). These values are summed over all reporting devices (some
implementations may use sampling for efficiency reasons) using a distributed algorithm and fed to a state
estimator to get an overall estimate 𝑠̂ (𝑡) of the true state 𝑠(𝑡) of total demand in each state for each tier
̂1,𝑎 | is the estimated total enabled and flexible demand in tier 1). This estimate is then broadcast
(e.g., |𝐸𝐹
to all devices, along with the demand goal 𝑔(𝑡) for the next total reduction in enabled demand over all
tiers. The controller at each device 𝑎 uses local information on state, color, and device type to set its
control vector 𝑐(𝑡, 𝑎), defined as the set of transition probabilities 𝑝𝑜𝑛,𝑖,𝑎 and 𝑝𝑜𝑓𝑓,𝑖,𝑎 for each tier 𝑖. Finally,
demands move through their states according to these transition probabilities, subject to exogenous
disturbances such as changes in demand due to customer override, changing environmental conditions,
imprecision in measurement, etc.
Figure 8: Block diagram of the ColorPower control architecture: the demand and priority tier states 𝑠(𝑡, 𝑎)
of each device 𝑎 at time 𝑡 are aggregated to produce an estimate 𝑠̂ (𝑡) of the total demand and priority tier
states over all devices. This is broadcast to each device, along with a demand goal 𝑔(𝑡), and each device
uses local information to set its control state 𝑐(𝑡, 𝑎) of transition probabilities. Loads then evolve according
to this control state, subject to exogenous disturbances.
Note that the aggregation and broadcast algorithms must be chosen carefully to ensure that the
communication requirements are lightweight enough to allow control rounds a few seconds long on lowcost hardware. The choice of algorithm depend on the network structure. In general, however, the system
must use aggregation and broadcast because it is not currently economically feasible to deploy reliable
communications for load control devices with enough bandwidth to poll individual state and deliver
individual instructions for millions of devices every few seconds. Aggregation reduces bandwidth
requirements by compressing the state of many devices into a single “collective state” message to a central
system, and while broadcast reduces bandwidth requirements by sharing a single “collective control”
message from the central system with many devices.
Simulations are executed on 10,000 controllable devices, each device consumes 1 kW of power (for a total
of 10 MW demand), devices are 20% green (low priority), 50% orange (medium priority) and 30% red (high
priority). The control signal sent to the demand response population can be updated every 10 seconds.
The results of the simulation test are illustrated in figure 9. When peak control is desired, the aggregate
demand remains below the quota, while individual loads are subjected stochastically to brief curtailments.
Post-event rush-in, a potentially severe problem for both traditional demand response and price signalbased control systems, is also managed gracefully due to the specific design of the modified Markov model
used for the control of the device.
Figure 9: Simulation results with 10,000 independently fluctuating power loads. Demand is shown as a
stacked graph, with Enabled demand on the bottom in saturated colors, Disabled demand at the top in
pale colors, and Refractory demand cross-hatched. The goal is the dashed blue line and the current total
Enabled demand is the solid magenta line. The plot illustrates a peak shaving case where a power quota,
the demand response target that may be provided from an externally-generated demand forecast, is used
as a guide for the demand to follow.
5. Conclusions
We present a proposal for a de-carbonized European market which is aimed at tackling the following
challenges: (i) the shifting of value from energy markets to ancillary services in a near-zero-marginal-cost
market; (ii) the lack of harmonization between regional markets and between time frames; (iii) the need
to mobilize distributed resources; (iv) the need to engage demand-side resources through scalable
aggregator business models. We tackle these challenges through a market design that is based on the
following pillars: (i) a well-designed real-time market with scarcity pricing that provides reliable long-run
investment signals for flexible resources; (ii) a harmonization of real-time and day-ahead markets that
allows sharing of balancing resources, efficient day-ahead scheduling and back-propagation of scarcity
signals; (iii) a coordinated operation of transmission and distribution network resources through a cloudbased distributed computing infrastructure; (iv) an aggregator business model based on priority service
pricing which respects the desire of consumers for simplicity, privacy, and control. We present a timeline
for the transition of the European market towards our proposal, and we illustrate the effectiveness of our
proposal through a number of case studies.
References
[A17] I. Arriaga et al., “Utility of the Future”, technical report, MIT Energy Institute, 2017.
[AP17] I. Aravena, A. Papavasiliou, “Renewable Energy Integration in Zonal Markets”, IEEE Transactions on
Power Systems, vol. 32, no. 2, pp. 1334-1349, March 2017.
[B17] G. Bade, “2017 Forecast: 10 trends shaping the electric utility industry”, Utility Dive technical report.
[C+16] M. Caramanis, E. Ntakou, W. Hogan, A. Chakrabortty, and J. Schoene, “Co-Optimization of Power
and Reserves in Dynamic T&D Power Markets with Non-Dispatchable Renewable Generation and
Distributed Energy Resources”. Proceedings of the IEEE invited paper, April, 2016.
[C+86] H. P. Chao, S. S. Oren, S. A. Smith, R. B. Wilson, “Multilevel Demand Subscription Pricing for Electric
Power”. Energy Economics, vol. 8, no. 4, pp. 199-217, 1986.
[CO16] Campaigne, Clay, and Shmuel S. Oren. "Firming renewable power with demand response: an endto-end aggregator business model." Journal of Regulatory Economics 50.1 (2016): 1-37.
[CW87] H.-P. Chao, R. Wilson, “Priority Service: Pricing, Investment and Market Organization”. The
American Economic Review, vol. 77, no. 5, pp. 899-916, December 1987.
[EC11] European Commission, “Energy Roadmap 2050”, 2011.
[ER14] 50Hertz Transmission GmbH, Amprion GmbH, Elia System Operator NV, TenneT TSO B.V., TenneT
TSO GmbH, and TransnetBW GmbH, “Potential cross-border balancing cooperation between the Belgian,
Dutch and German electricity Transmission System Operators,” report prepared by the Institute of Power
Systems and Power Economics and E-Bridge Consulting GmbH, October 2014.
[FL13] M. Farivar and S. H. Low, “Branch flow model: Relaxations and convexification - part I,” IEEE
Transactions on Power Systems, vol. 28, no. 3, pp. 2554–2564, 2013.
[G14] H. C. Gils, “Assessment of the Theoretical Demand Response Potential in Europe”. Energy, vol. 67,
pp. 1-18, 2014.
[H05] W. Hogan, “On an energy only electricity market design for resource adequacy.” Technical report,
Center for Business and Government, JFK School of Government, Harvard University, 2005.
[H13] W. Hogan, “Electricity scarcity pricing through operating reserves.” Economics of Energy and
Environmental Policy 2(2): 65–86. https://doi.org/10.5547/2160-5890.2.2.4, 2013.
[K+13] M. Kraning, E. Chu, J. Lavaei, and S. Boyd, “Dynamic network energy management via proximal
message passing”, Foundations and Trends in Optimization, vol. 1, no. 2, pp. 73–126, 2013.
[LB15] T. Levin, A. Botterud, “Electricity market design for generator revenue sufficiency with increased
variable
generation”,
Energy
Policy,
vol.
87,
pp.
392–406,
2015.
https://doi.org/10.1016/j.enpol.2015.09.012.
[MPC17] Y. Mou, A. Papavasiliou, P. Chevalier, “Application of Priority Service Pricing for Mobilizing
Residential Demand Response in Belgium”, IEEE conference on the European Electricity Market (EEM),
2017.
[P17] A. Papavasiliou, “Analysis of Distribution Locational Marginal Prices”, forthcoming in IEEE
Transactions on Smart Grid, 2017.
[PBF13] A. Papalexopoulos, J. Beal, S. Florek, “Precise Mass-Market Energy Demand Management through
Stochastic Distributed Computing”, IEEE Transactions on Smart Grid, vol. 4, no. 4, 2013.
[PES17] IEEE Power and Energy Magazine, vol. 15, no. 2, March/April 2017.
[PL14] Q. Peng, S.H. Low, “Distributed Algorithm for Optimal Power Flow on a Radial Network”, IEEE 53rd
Annual Conference on Decision and Control, December 2014.
[POR15] A. Papavasiliou, S. S. Oren, B. Rountree, “Applying High Performance Computing to TransmissionConstrained Stochastic Unit Commitment for Renewable Penetration”, IEEE Transactions on Power
Systems, vol. 30, no. 3, pp. 1690-1701, May 2015.
[PS17] A. Papavasiliou, Y. Smeers, “Remuneration of Flexibility using Operating Reserve Demand Curves:
A Case Study of Belgium”, forthcoming in the Energy Journal.
[S03] R. Shanker. “Comments on standard market design: resource adequacy requirements.” Federal
Energy Regulatory Commission, Docket RM01-12-000, 2003.
[V05] F. J. L.Van Hulle, “Large Scale Integration of Wind Energy in the European Power Supply: Analysis,
Issues and Recommendations”. European Wind Energy Association technical report, 2005.
[ZB13] Z. Zhou, A. Botterud. “Dynamic scheduling of operating reserves in co-optimized electricity markets
with wind power.” IEEE Transactions on Power Systems, vol. 29, no. 1, pp. 160–171, 2013.
https://doi.org/10.1109/TPWRS.2013.2281504.