Divide and conquer?
Assessing k-means clustering of demand data
in simulations of the British electricity system.
Richard Green†, Iain Staffell† and Nicholas Vasilakos†‡*
†
‡
Department of Economics,
University of Birmingham,
Birmingham, B15 2TT,
United Kingdom.
Norwich Business School,
University of East Anglia,
Norwich, NR4 7TJ,
United Kingdom.
Corresponding author: [email protected]
Abstract
We apply a k-means clustering algorithm to partition national electricity demand data for Great
Britain and obtain a set of representative demand profiles for each year over the period 1994-2005.
We then use a simulated dispatch model to assess the accuracy of these daily profiles against the
complete dataset on a year to year basis. We find that the use of data partitioning does not
compromise the accuracy of the simulations for most of the main variables considered, even with
significant intermittent wind generation. This technique yields significant gains in terms of
computational speed, allowing more complex Monte Carlo simulations to be performed.
JEL Codes: D43, L13, L94, Q41, Q42
Keywords: Demand data, k-means, clustering, data mining, computational efficiency, electricity
modelling.
*
This research is supported by the Research Councils‟ Energy Programme and by our industrial partners through the
Supergen Flexnet Consortium, grant number EP/E04011X/1. The views expressed are ours alone.
1
1
Introduction
Any model involves compromises. The modeller must often choose between a simple model that
neglects some key features of the real world or a more complex one that requires a long time to
solve. In the case of the electricity industry, the choice is frequently between a “merit-order stack”
model that simply meets each period‟s demand from the cheapest power stations, ranked in order of
increasing variable cost, and a more sophisticated “dispatch model” that takes into account the
temporal nature of demand and plant operation via the time-lag and cost of starting and stopping
plants. As the proportion of intermittent renewable generation increases in many countries such as
the UK, the demand to be met by conventional plant is likely to become more variable, making
these effects more important. Many of the input parameters into these models will be uncertain, and
repeated simulations might be required to represent this uncertainty. With a complex model
covering a full year plant dispatch it can easily become too costly, in terms of computing power and
time, to run sufficient simulations to capture this uncertainty with Monte Carlo or sensitivity
analysis.
This paper presents a way of identifying a small number of “typical” days that can represent
a year of operation, allowing a dispatch model to be run very fast and thus many times for
stochastic or other simulations. To do this we apply a k-means clustering algorithm, a method that
has been used widely in various disciplines to separate (often large) agglomerated datasets into a
number of smaller groups. Membership for each cluster is decided using a similarity measure – in
this case the Euclidean distance of each data point from the group mean. The objective of the
process is to allocate each observation between groups in such a way so that this distance is
minimised.
Clustering has been used to represent electricity demand data in previous studies, but this is
the first exercise that we are aware of that validates their use by comparing the results from
clustered data to those from a full year‟s simulation. We create and use clusters for twelve separate
years of data, thus allowing us to illustrate the reliability of this method, in terms of the differences
from those obtained from the original data, and how much they vary from year to year.
The data we use is partly drawn from Green and Vasilakos (2010), a simulation of wind
output and demand patterns for Great Britain in the year 2020. This data is based on hourly
national electricity demand and weather data from 1994 to 2005, hence giving twelve years
calibrated to the same overall level of demand and wind capacity. We combine this data with a
scenario for generation capacity and a dispatch model to calculate the costs and operating patterns
of meeting demand from day to day.
We run the model for each year of data with different numbers of clusters. We find that the
clustered data give a remarkably accurate simulation for many key variables, including the average
cost of electricity, carbon emissions and generator revenues. Other variables, which are more
dependent on extreme events, such as generator start-ups (often in response to a short-lived spike in
demand) are less accurately modelled, and so the method is not applicable to every problem that we
might wish to apply it to. Nonetheless, we find that using six or ten clusters to represent a year of
data gives a remarkable saving in computer time, and recommend that this method be considered to
allow the complexities of electricity dispatch to be represented in studies that require many repeated
simulations of an electricity market.
The paper starts with an overview of previous work in the field. We then introduce the data
that we use as an example in the following descriptions of the clustering technique and a means for
producing representative profiles from these clusters. We then explain how the validity of these
2
clustered profiles was tested using a generator dispatch model, and follow this with results from the
validation.
2
Previous work
Many problems in electricity are stochastic. Sometimes the law of large numbers can be used to
model a simpler deterministic equivalent; for example, the incidence of mechanical problems that
may make generating units unavailable can be represented by scaling down the available capacity of
every unit by a fixed proportion. Such techniques are not always appropriate, however: Borenstein
et al. (2002) point out that scaling available capacity in a deterministic manner fails to capture the
convexity of the marginal cost curve. Their simulations of competitive prices (as a counter-factual
to be compared with actual prices during the California electricity crisis) were based on 100
repeated model runs in which each unit had a probability of being completely unavailable. The
trade-off required to make the problem manageable was that they modelled each hour separately
and did not attempt to include the impact of start-up costs in their model. In using Monte Carlo
methods to estimate marginal costs, they follow researchers such as Baughman and Lee (1992),
who calculate electricity market spot prices and Mazumdar and Chrzan (1995), who estimate
generation costs.
Researchers in hydro-dominated systems have developed stochastic techniques for
optimising the use of water when future inflows are uncertain, as described by Førsund (2007).
These rely on backwards induction to find the optimal output pattern across (and responding to) a
range of possible future inflows. More recently, uncertainty over wind speeds has become an
important factor for countries with a high (or rising) proportion of wind generation. Müsgens and
Neuhoff (2006) model the additional costs imposed on the German electricity system when thermal
plants are committed on the basis of wind forecasts made 24 hours ahead rather than on the more
accurate forecasts available 4 hours before real time. Neuhoff et al. (2008) use repeated model runs
to find the optimal deployment patterns for wind and other generators across Great Britain in the
presence of transmission constraints.
Fuel prices differ from plant availability and the weather in that they are uncertain, rather
than risky, and so we should not expect past experience to give us the full probability distribution
for the future. Even so, stochastic models based upon some probability distribution may act as a
guide to possible outcomes or help illustrate the underlying forces at work. Green (2008) uses a
Monte Carlo analysis with variable fuel prices to compare the impact of carbon trading (with a
stochastic permit price related to the fuel prices) and a carbon tax on the profits of high- and lowcarbon generators. Once again, each period within a year is effectively modelled separately,
neglecting inter-temporal constraints to produce a tractable problem.
Roh et al. (2009) also model a combined generation-transmission expansion planning
problem with Monte Carlo methods, but introduce a scenario reduction technique to reduce the
computational burden of a large number of different cases. Scully et al. (1992) discuss a semiguided Monte Carlo technique that can be used with a full inter-temporal scheduling model and
greatly reduce the number of runs required to produce “reliable” production cost results. Our work
is in this vein, finding a way to combine a sophisticated system model with a limited set of input
data and still obtain reliable results.
Data partitioning methods of the kind we propose have been used widely in the general
energy economics literature, mostly as part of wider forecasting frameworks. A good number of
papers show how data clustering can be used to explain sources of retail and wholesale price
3
variation, or simply how these algorithms can be used to facilitate a more complex computational
framework. Most conclude that the use of clustering algorithms results in small forecasting errors
and a substantial increase in computational speed. For instance, Álvarez et al. (2007) apply two
separate clustering algorithms on time series data to forecast prices for the Spanish electricity
market. Their results indicate that both algorithms performed very well in identifying usable
patterns on the data and with small average errors.
Balachandra and Chandru (1999) use k-means partitioning to estimate 9 representative load
curves for the electricity system in Karnataka in 1994, which they then use to explain the sources of
variation in hourly demand. Although the aim of our paper is not to explain patterns of demand
variation per se, the clustering algorithm they have applied is very similar to ours. Their discussion
on the performance of the method also indicates that the use of clustering methods did not result to
substantial losses of forecasting accuracy (when compared with actual demand data for that year),
with 86% of the percentage error for the estimated hourly loads falling within the ±10% range.
The choice of a partitioning as a means to model electricity demand data is also discussed in
Marton et al. (2007) who use hierarchical clustering methods to construct representative load curves
for the province of Ontario, Canada. Gerbec et al. (2003) apply and compare two different
partitioning algorithms (fuzzy c-means and hierarchical clustering) on a sample of 288 measured
load profiles of eligible retail consumers obtained from Slovenian distribution companies. Both
methods produced similar results (although not identical, in terms of the optimal number of clusters
and group membership of individual observations), with the fuzzy c-means algorithm supporting 8
groups as opposed to 6 when the k-means algorithm was used.
3
Generating clustered demand data
The basis for our work is a set of national hourly demand and wind generation data, which
represents the British electricity system in 2020 and is drawn in large part from Green and
Vasilakos (2010).
That paper took 12 years of hourly wind speed data from the British
Atmospheric Data Centre, covering a number of weather stations around Great Britain. These data
were cleaned to interpolate missing observations (where just one or two consecutive observations
were missing) or to fill them in on the basis of regressions against the wind speeds at other stations
(where the gap was longer). The hourly wind speeds were then adjusted for the height of a typical
wind turbine, and a power curve was used to calculate the output (per MW of capacity) as a
function of the wind speed. The wind speed for each station was scaled up or down by a constant
(station-specific) percentage if this was required to give a plausible average load factor for the wind
turbines in that area – many weather stations are located in sites which are (locally) insufficiently
windy for a wind farm. We assumed that the wind speed at a suitable generation site was related to,
but higher than, the speed at the nearby weather station. We used a similar method to obtain
offshore wind outputs from coastal weather stations. The per-MW outputs were multiplied by a
total of 11 GW of onshore and 19 GW of offshore capacity, distributed around Great Britain in line
with British Wind Energy Association predictions.
The wind data for 1994–2005 were then combined with hourly demand data for the same
period, produced by combining the half-hourly figures given by National Grid. We assumed future
demand growth of 0.9% per year, giving a total demand in 2020 of 382 TWh in normal weather
conditions. The individual demand figures for each year in our data were scaled by the ratio of the
actual demand to the weather-adjusted demand for that year, giving a range across the twelve years
4
of 382.2 ± 2.6 TWh. This preserves the day-to-day and year-to-year variation in demand due to
weather conditions, ensuring that our hourly wind outputs are matched with a compatible demand
for electricity. We repeat our simulations with both the gross demand figures (effectively assuming
that there is no wind power at all) and the demand net of wind output.
Figure 1 summarises the two data sets, showing the general structure of demand throughout
the day and the range in demand levels seen over the 12 years. The net data (with wind subtracted)
shares the same intra-daily structure as the gross data, albeit with far greater variability between
days. A notable feature, and one which will present serious technical challenges for the GB system,
is that peak demand can coincide with a lull in wind, as can troughs in demand with full wind
output. Based on the estimated output from 30 GW of wind capacity, peak GB demand falls by just
2.3 GW from 67.4 GW to 65.1 GW net of wind, whereas the minimum level of demand falls from
24.0 GW down to just 1.5 GW.
70
70
(a)
Gross demand (GW)
50
40
30
20
10
0
(b)
60
Net demand (GW)
Percentiles:
85-95%
75-85%
65-75%
55-65%
45-55%
35-45%
25-35%
15-25%
5-15%
Max/Min
60
50
40
30
20
10
0
0
4
8
12
16
20
Hour of the day
0
4
8
12
16
20
Figure 1: The distribution of demand across the 4,383 days of (a) gross national demand, and (b) demand net of wind
output.
3.1 Clustering method
Data partitioning works by allocating individual observations into smaller groups in such a way that
(a) observations that exhibit some common feature (built-in within the algorithm) are grouped
together and (b) the groups that are created are dissimilar enough from each other to make them
distinct. To achieve this, a clustering algorithm applies a measure of dissimilarity to establish group
membership. The k-means algorithm used in this exercise employs as a dissimilarity measure the
Euclidean distance of each demand vector (containing 24 hourly demand points) to allocate daily
demand profiles to k groups. Membership for each group is decided as a solution to the
minimization problem:
(1)
where is a 24-dimensional demand vector and is the mean for group i to which this particular
vector has been allocated. In other words, all j vectors are allocated between the k groups in such a
way so that they are the closest possible to their group mean. We initialise the procedure by
defining a set of k demand vectors from the dataset as centroids for each cluster. The algorithm then
allocates the demand vectors between the clusters so that the objective function in (1) is minimised.
The new centroid is then recalculated for each group as the average of the values of all vector
5
members in each hour-dimension. The procedure repeats itself until convergence is achieved. A
different value of k would naturally produce a somewhat different set of clusters, and so we
experiment with a wide range of values.
3.2 Clustered demand data
This technique was applied to our data, matching both sets of demand (gross and net of wind) to 5,
6, 8, 10, 15 and 25 clusters. Figure 2 gives an example of how six clusters were distributed across
the twelve years of gross demand data. Even though the algorithm did not take explicit account of
any meta-data (such as ambient temperatures or dates), these clusters show strong correlations with
season and day of the week (including national holidays). Dividing this data into more clusters
simply increased the number of bands between seasons: 8 clusters had two grades of shoulder day,
10 had three grades, and so on. Beyond this we found that very small clusters of anomalous days
begin to form, e.g. for Christmas, New Year, and extreme weather events.
Winter weekend
Winter weekday
Shoulder weekend
Shoulder weekday
Summer weekend
Summer weekday
2004
2002
2000
1998
1996
1994
January February
March
April
May
June
July
August September October November December
Figure 2: Spectrogram showing the distribution of six clusters fitted to the gross data set.
The clusters are distributed similarly between years. The boundaries between seasons show
relatively little variation, and they correlate well with temperature records for the UK (Met Office,
2011). For example, March 1997 was unusually warm, and May 1996 was colder than average,
hence both were predominantly classified as having shoulder days.
The labels given in Figure 2 were manually assigned to the clusters based on our
observations. Figure 3 shows that a high proportion (82–94%) of the days within each cluster fall
into their „assigned‟ season; where, based on this data (from an electricity-consumption point of
view), we defined winter as running from the 8th of November to the 16th of March, and summer
from the 5th of May to the 27th of September.
6
Winter
Shoulder
Summer WD = Weekday WE = Weekend
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Winter
WD
WE
Shoulder
WD
WE
Summer
WD
WE
Figure 3: The seasonal distribution of days within in each cluster shows a good correlation across all twelve years.
The strong influence of stochastic output from wind turbines meant that clusters for net demand
data did not fall into such clear-cut patterns. Figure 4 shows the distribution of ten clusters; there is
still a visible distinction between summer (light) and winter (dark), but the clusters are also
determined by the pattern of wind output during the day (colours / hatching). From our
observations of the profiles within each cluster, we found that they converged on three broad
patterns of wind output: predominantly windy during the morning (A–C); predominantly windy
during the evening (D–G); or no overall pattern (H–J). The profiles only loosely adhere to these
categories, and so they were simply assigned letters rather than descriptive names in Figure 4.
A
B
C
D
E
F
G
H
I
J
2004
2002
2000
1998
1996
1994
January
February
March
April
May
June
July
August September October November December
Figure 4: The distribution of ten clusters fitted to the demand net of wind output.
3.3 Producing demand profiles
Once the individual days have been grouped into clusters, there are several methods for reducing
each cluster down to a single, representative profile. The main options are to average, selectively
filter, or randomly choose from the days within the cluster; each of which has its relative merits.
For example, averaging allows every single day to contribute to the final result, but incurs
the penalty of temporal smoothing. By selecting a single random day from each cluster, any
smoothing effects are eliminated, but other problems are incurred. The total level of demand over
the entire period will not be preserved, although this can be remedied by scaling the total demand of
each day to equal the average demand from all days within its cluster. The main drawback is the
sheer variation in end results that is obtained depending on which particular days were chosen.
We could instead choose the specific day that best represents the cluster, namely the one that
is closest to the cluster‟s mean (as defined by the same process that derives the clusters). However,
7
as the cluster mean is smoothed out, the day that is most similar to it will be an unusually smooth
day, and thus will poorly represent the entire data set.
A less extreme option is to filter out days from within each cluster, for example removing
those which deviate most from the cluster‟s average, or which have markedly different total levels
of demand. This also appears to offer no benefits, presumably because removing the influence of
even „spurious‟ data does not bring the resulting profile closer to the complete data set. The
principle of the first option – including all the data with no discrimination – was found to work best.
A basic implementation consists of calculating the average demand level from the first hour
of all days within the cluster and assigning that to hour 1 of the resulting profile, then averaging all
levels from the 2nd hours to give hour 2, and so on. This naturally preserves the total energy
demand over the entire day, but has a smoothing effect on the resulting profile. Extreme values at
both ends of the spectrum (e.g. unusually high and low demands at 4am) cancel each other out, as
do temporal differences.
Inter-temporal smoothing has a greater effect, as the rate of change of demand shows greater
swings than the levels of demand itself. For example, if the evening peak on one day occurs at
8pm, the change in demand from 7 to 8pm will be positive, and from 8 to 9 pm will be negative.
When this day is averaged with another where the evening peak occurs at 9pm, the ramp rates from
8 to 9pm will cancel each other out, resulting in a plateau rather than a spike. Ideally this situation
should not occur as the clusters were formed based on similarity in both demand and ramp rates,
however it is inevitable when small numbers of clusters are under consideration. This is reflected in
our actual data set, where morning and evening peaks get broadened out over time and lessened in
intensity, as demonstrated in Figure 5.
65
06/01/1995
21/02/2003
Average
Demand (GW)
60
55
50
45
12
14
16
18
Hour
20
22
24
Figure 5: Example of an evening peak being smoothed out by averaging two otherwise similar days.
A better alternative is therefore to average the days within each cluster based on both their levels of
demand and the changes between these levels. Since the mean of the changes between two adjacent
hours is equal to the change between their mean levels, the average changes provide no extra
information. Instead we consider a subset of the changes: either those which have the same sign as
the mean change in that hour (i.e. the dominant / mean change); or those which have the most
common sign in that hour (i.e. the common / median change). In other words, if the average profile
has an increase in demand between 7 and 8pm, the dominant profile takes the average of all positive
changes between those hours. It is possible that between those hours the majority of profiles have a
small negative change which is outweighed in magnitude by the minority of large positive changes.
8
In this case, the common profile would instead be determined by the average of the negative
changes, as illustrated in Figure 6.
In both cases this gives us 23 hour-to-hour changes between adjacent periods. Given these
changes, the demand in the first hour of the day is sufficient to determine the rest of the profile, and
this is chosen so that the mean demand over the profile equals the mean over the entire cluster. We
compare these options (the profiles built from the dominant and the commonest changes in demand)
with the profile given by the cluster mean and with equally weighted mixes of dominant/mean and
common/median.
65
65
(a)
(b)
60
Demand (GW)
Demand (GW)
60
55
50
06/01/1995
45
55
50
Level average
Dominant ramp
Common ramp
45
21/02/2003
18/02/2005
40
40
0
2
4
6
8
0
10 12 14 16 18 20 22 24
Hour
2
4
6
8
10 12 14 16 18 20 22 24
Hour
Figure 6: Three sample daily profiles (a), and the merged profiles that result from the three grouping methods (b).
The three sample profiles given in Figure 6a are similar throughout the day except between 16:00
and 17:00, which is highlighted. Between these hours, demand increases substantially on
06/01/1995, but decreases slightly on the other two days. This gives a marked difference in the
common and dominant merged profiles shown in Figure 6b: the dominant profile rises by 5.6 GW,
whereas the common profile falls by 0.2 GW. This highlights a drawback of using the dominant
changes, as the demand level rises for three consecutive hours, from 16:00 to 19:00 and peaks at a
higher level than any of the constituent days it is meant to represent. To balance this increase in
evening demand, the whole profile was normalised downwards (to maintain the correct overall level
of demand), meaning that it tracks about 1.5 GW lower than the average profile for the earlier part
of the day. In contrast, the profile built from the common ramps shows similar levels of demand to
the simple average, but sharpens the evening peak at 19:00. This is the method used in this paper.
Finally, when implementing the clusters it is important to assign a weighting factor, w, to
each of the resulting profiles, equal to the number of days that were contained within its cluster. All
results derived from that profile are then multiplied by its weighting factor, giving the same results
as if the simulation were run on w repetitions of the profile. This emphasises the importance of the
more „typical‟ clusters which contain the majority of the data (e.g. summer weekdays), and prevents
the outlying clusters (shoulder weekends, anomalous weather events) from dominating.
3.4 Clustered demand profiles
Figure 7 gives the profiles that were formed from the 6 gross clusters shown in Figure 2, produced
from the common ramp rates.
9
60
Demand (GW)
50
40
30
Winter weekend (457 days)
Winter weekday (1071 days)
Shoulder weekend (353 days)
Shoulder weekday (693 days)
Summer weekend (583 days)
Summer weekday (1226 days)
20
10
0
0
3
6
9
12
Hour
15
18
21
24
Figure 7: Six daily profiles that represent the gross UK electricity demand in 2020. The names we assigned each
profile are given along with their weighting factors, showing how many days they represent.
Demand (GW)
Similarly, Figure 8 gives the profiles that were formed from the 10 net clusters shown in Figure 4,
separated by the pattern of wind output. This categorisation of profiles was somewhat arbitrary, as
it was not possible to divide the clusters into clearly defined groups. Net demand on a winter‟s day
with relatively high wind output is indistinguishable from a summer‟s day with relatively little
wind. We therefore characterised the profiles based on the shape of the demand profiles, loosely
correlating to windy nights (left), windy afternoons (right), and „other‟ patterns (centre).
60
60
50
50
40
40
30
30
20
20
D (344 days)
E (378 days)
F (492 days)
G (288 days)
A (323 days)
B (328 days)
C (443 days)
10
H (300 days)
I (745 days)
J (742 days)
0
10
0
0
3
6
9
12 15 18 21 24
Hour
0
3
6
9
12 15 18 21 24
Hour
0
3
6
9
12 15 18 21 24
Hour
Figure 8: Ten daily profiles that represent UK electricity demand in 2020, net of 30 GW of wind output.
A seemingly negative impact of using the common or dominant ramp rates can be seen between the
last and first hours of profile B (i.e. hours 23 and 24 = 0 in Figure 8). The late evening demand is
on a steady downward trajectory, then drops abruptly from 38 to 20 GW, rising slightly to 22 GW
in hour 1. The method described in the previous section does not force the profiles to begin and end
on similar values, so if they are wrapped around as in Figure 8, two days of the same profile
following one after the other do not appear to be „natural‟. Especially large ramps can be seen in
profiles B and C, or ramps with the wrong apparent sign in profiles I and J.
An attempt to remove this bias was made by including the ramp rate from hour 23 to hour 0
in the optimisation process of producing the common and dominant ramp profiles. A constant bias
10
was added to all the ramp rates so as to alter the change between the end and beginning of the
profiles. For example, in profile B every ramp rate was reduced by 0.47 GW, so that the change
from hours 23 to 0 reduced from –16.7 GW to –5.5 GW. This however proved to be counterproductive, and greatly reduced the accuracy of the clusters in terms of the final results, and so the
anomalous end-of-day boundaries were retained.
4
Validation of the clusters
Having obtained sets of clusters, our aim is to determine whether a small number of clusters can be
used as an acceptable representation of a full year of data in a simple dispatch model that estimates
grid operation patterns and costs. Our simulations use a standard linear programming approach,
with a program derived from the MAGIC simulator in the GAMS model library (Day and Williams,
1982). The program aims to minimise the total cost of meeting the demand for electricity, subject
to constraints on the maximum and minimum generation from each unit in operation. The cost of
power includes a no-load cost for keeping each unit running, a cost per MWh of output and a start
cost, each of which depends on the type of power station considered. Each model run consists of
one or more complete days of data, and to avoid the possibility of starting or finishing the period
with an unrealistic amount of plant in operation, the solution is constrained to start and end with the
same stations in use (since the demand at midnight and at 11 pm is similar).
We treat each year of data as a separate test, which allows us to report distributions for our
results over the 12 years we have available. For each year, the base case consists of running the
model for the entire year as a single run. We then run the model for a small set of clusters, running
each day separately. We weight the results for each cluster by the number of days it represents to
obtain a value (average or total, as appropriate) for the year as a whole. The key results are the
demand for power (which will equal the base case, by construction), the output by plant type, the
average cost of electricity and carbon emissions. We also measure the marginal cost (which would
equal the market price in a perfectly competitive market), the number and cost of plant starts, and
the profits each type of plant would make.
We have taken generating capacity figures from National Grid‟s Seven Year Statement for
2010 and applied our own judgement for investment and plant retirements in the years to 2020.
Plant costs are largely based on Mott MacDonald‟s (2010) generation cost update for the
Department of Energy and Climate Change, which gives projected fuel prices and overall thermal
efficiencies for different types of plant. We use central prices of £20.40/MWh for gas, £9.52/MWh
of coal, and £30.00/tonne of CO2, based on the Budget 2011 announcement of the Carbon Price
Support scheme. We follow Troy (2008) and Meibom (2008) in deriving start-up costs, attempting
to capture both fuel costs (including carbon) and the cost of increased wear and tear from heating
and cooling the station. Table 1 gives the types of dispatchable plant considered; run-of-river and
pumped storage hydro were reduced to a single unit each due to having no lower limit on output,
and were constrained to optimise their outputs given 25 and 15 GWh of available storage which was
replenished at constant rates to give average load-factors of 0.415 and 0.150 respectively.
11
Unit
Total
Minimum
No-load
Incremental
Number
Start-up
capacity
capacity
stable
cost
cost
of units
cost (£)
(MW)
(GW)
generation
(£/hour)
(£/MWh)
Nuclear
500
20
10
75%
4,000,000
3510
7.39
CCGT (large)
750
28
21
50%
79,564
2226
48.42
CCGT (small)
350
22
7.7
50%
26,727
1039
53.64
Coal (large)
525
44
23.1
33%
198,528
3356
51.37
Coal (small)
150
20
3
33%
60,385
959
55.96
OCGT
30
70
2.1
10%
377
89
85.85
Oil
50
144
7.2
10%
596
320
116.81
Run-of-river hydro
1360
1
1.36
0%
0
0
0
Pumped hydro
2750
1
2.75
0%
0
0
0
Table 1: Parameters for the dispatchable generator types on the UK grid with central fuel prices and a £30/T carbon
price.
Generator type
5
Results
The initial series of tests consisted of 7,560 model runs:
 2 types of demand data (gross and net of wind);
 12 years of data;
 7 sizes of cluster (k = 1, 5, 6, 8, 10, 15, 25). The trivial case of k = 1 (i.e. no clustering of
the data) was included as a control experiment;
 15 clustering methods, including variations and combinations of those listed in Section 3.3;
 3 carbon prices (£10, £30, £50 / tonne CO2), to cover the range of roles for coal and CCGT
plant.
One of the complete year-long profiles could be processed in 19–22 minutes, whereas the average
for a cluster of 6 daily profiles was 19 seconds, rising to 26 seconds for a cluster of 10 days.
Because most of the calculations were for small groups of clustered data, it was possible to
complete these model runs in under a day of computing time, in spite of the number of trials
undertaken. Running the same number of trials using a full year‟s data would have taken
approximately two months. It should be noted that approximately 10 seconds of each model run
was spent reading and writing Microsoft Excel files, so with simpler file handling we expect
clustering could increase processing speed by a factor of over 100.
Of the various cluster sizes and methods, we found that 6 clusters gave the best trade-off of
speed and accuracy for the gross data, while 10 clusters were required to cope with the more
variable nature of the net data. In both cases, merging the clusters based on the most common ramp
rate yielded the best results, in terms of closeness to the true values and minimising variation in
closeness across different metrics, across years and carbon prices.
Figure 9 gives a comparison of the clustering method to the complete data sets for a
selection of metrics with the gross data. Each plot gives a fan chart, showing the range of results
from the 12 years of data against the number of clusters. The accuracy for cost metrics is very
good, with an average error of 0.4±0.2% for 6 clusters. For average and marginal carbon
intensities, annual outputs and revenue, these average errors increased to between 3 and 7%.
Remarkably, for these metrics, a single daily profile created from a whole year‟s observations gives
very similar results to those from 10 or even 25 clusters.
12
101%
101%
Average cost of electricity
(avg. £44.70/MWh)
105%
104%
103%
102%
101%
100%
99%
98%
97%
96%
95%
Marginal cost of electricity
(avg. £57.38/MWh)
100%
100%
99%
99%
1
5
6
8
10
15
25
1
110%
110%
105%
105%
100%
100%
95%
95%
Total annual CCGT output
(avg. 175.7 TWh)
90%
5
6
8
10
15
25
5
6
8
10
15
25
Annual CCGT revenue
(avg. £11.6 bn)
1
5
6
8
10
15
1
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
90%
1
Average carbon intensity
(avg. 405 kg/MWh)
25
5
6
8
10
15
CCGT Start-ups
(avg. 53 per plant per year)
1
5
6
8
10
15
Figure 9: Comparisons of the results from clustered data with different values of k (horizontal axis) to the values
obtained with the gross annual data sets for a selection of metrics.
A notable problem can be seen in the lower right plot of Figure 9; the simulation with clustered data
predicted few or no start-ups for CCGT and coal plants, compared to averages of 53 and 2 start-ups
per plant per year for the complete data sets. Because of this, the total cost of start-ups was
underestimated by an average of £47m per year, which accounted for more than half of the £80m
underestimate in the total cost of generation, or 0.3 percentage points of the total error. This
highlights an inherent drawback of clustering; it is not able to represent low frequency events such
as plant starts, outages, or peak requirements.
Figure 10 repeats the comparison given in Figure 9, showing the accuracy of the clustering
method when using the demand data net of wind. The additional stochastic variation introduced
with the wind demand meant that in most cases the clusters were less accurate for bulk properties
such as average costs and carbon emissions; however, the output, revenue and number of start-ups
for different plant types were more accurately modelled with the net demand than they had been
with the gross demand data.
13
25
25
102%
105%
104%
103%
102%
101%
100%
99%
98%
97%
96%
95%
Average cost of electricity
(avg. £41.55/MWh)
101%
100%
99%
98%
1
5
6
8
10
15
25
110%
100%
95%
Total annual CCGT output
(avg. 155.1 TWh)
90%
1
5
6
8
10
15
1
105%
104%
103%
102%
101%
100%
99%
98%
97%
96%
95%
105%
25
105%
104%
103%
102%
101%
100%
99%
98%
97%
96%
95%
Marginal cost of electricity
(avg. £55.37/MWh)
5
6
8
10
15
25
5
6
8
10
15
1
200%
180%
160%
140%
120%
100%
80%
60%
40%
20%
0%
Annual CCGT revenue
(avg. £10.1 bn)
1
Average carbon intensity
(avg. 318 kg/MWh)
25
5
6
8
10
15
CCGT start-ups
(avg. 120 per plant per year)
1
5
6
8
10
15
Figure 10: Comparisons of the results from clustered data with different values of k (horizontal axis) to the values
obtained with the net annual data sets for a selection of metrics.
The benefit of clustering, rather than creating a single profile from a whole year‟s data, is clearly
visible with the net demand data, given the step-change in results going from 1 up to 5 clusters.
With the gross data, this was only pronounced in a selection of metrics, for example the number of
OCGT start-ups and marginal carbon intensity. Overall, moving from 1 to 6 or more clusters
reduced the average error by a fifth with the gross data and by three-fifths with the net data,
averaged over all the metrics considered. We note that the errors for some variables continue to fall
by noticeable amounts as we move from 10 to 25 profiles, although this is not the case for the
marginal cost or revenue estimates.
Similarly, using the commonest ramp rate method of merging clusters into a single profile
was seen to reduce the average errors by two-fifths compared to using the average profile, and by
one-fifteenth compared to selecting six or ten random days from within each cluster.
In almost all of the metrics considered, the errors introduced through clustering were
determined to be normally distributed by the Jarque-Bera test at the 5% significance level, albeit
with means that were often non-zero. The only exceptions were in metrics that related to extreme
events, for example the number of CCGT start-ups shown in Figure 9.
6
Sensitivity to input data
When these simulations were repeated with carbon costs of £10 and £50 per tonne of CO 2, the
accuracy of the clusters improved in many respects. It can be calculated from Table 1 that a £30
carbon cost produces a relatively rare situation with our fuel price assumptions where the
incremental costs of coal and CCGT plants intersect. Between a carbon price of £24.15 and £34.92
per tonne, large CCGT plant have the lowest incremental cost, followed by large coal, then small
CCGT and small coal. This scenario results in larger errors in the ratio of coal to gas used in the
supply mix, as a minor change in the demand curve could result in substantial numbers of coal plant
starting up instead of CCGT, or vice versa.
14
25
25
As a test of the robustness of the clustering method, we repeated the tests of Section 5 with
carbon prices ranging from £10 to £50 per tonne, in steps of £1. A standard desktop computer
performed the tests using the optimum clustering method (common ramp) and number of clusters (6
for gross data, 10 for net); while a small parallel computer cluster was used to run the model with
the full annual data sets.
Figures 11 and 12 illustrate how the clusters compare to the full simulations with gross and
net demand respectively. The solid line with error bars depicts the results from the complete data
sets, while the shaded areas give the ten-percentile intervals for the clustered results.
600
£50
£45
£40
-£0.20
-£0.30
-£0.40
£15
£20 £25 £30 £35 £40
Carbon price (£ / tonne CO₂)
£45
500
450
400
350
Residuals
-£0.10
£10
Annual results, with 5–95% limits
300
£0.00
£35
Clustered results, with 5–95% limits
550
-£0.50
£50
£10
£15
£20 £25 £30 £35 £40
Carbon price (£ / tonne CO₂)
£45
30
20
10
0
-10
-20
-30
£50
Residuals
Average Carbon Intensity (kg/MWh)
Average Cost of Electricity (£/MWh)
£55
750
£0.20
700
£0.15
Annual coal profit (£bn)
650
600
550
500
450
400
Annual results, with 5–95% limits
£0.10
£0.05
£0.00
-£0.05
-£0.10
-£0.15
100
300
50
0
-50
£15
£20 £25 £30 £35 £40
Carbon price (£ / tonne CO₂)
£45
£0.08
-£0.20
£0.04
-100
£50
£0.00
-£0.04
£10
£15
£20 £25 £30 £35 £40
Carbon price (£ / tonne CO₂)
£45
-£0.08
£50
Figure 12: Comparisons of the results from clustered days and net annual data sets across a range of carbon prices.
15
Residuals
350
£10
Clustered results, with 5–95% limits
Residuals
Marginal Carbon Intensity (kg/MWh)
Figure 11: Comparisons of the results from clustered days and gross annual data sets across a range of carbon prices.
The deviation of the clusters from the true values is shown underneath each plot on a separate scale.
Both the average and marginal cost of electricity progress smoothly upwards with increasing
carbon cost, which is reliably replicated by the clusters in all cases. The clusters gradually drift
further away from the true values in absolute terms, but the average absolute errors are small,
reaching maximum values of £0.38–1.07 per MWh at £50 per tonne of carbon (relative errors of
0.7–1.6%). The clusters always under-predict costs, which is something that can easily be corrected
for.
The average and marginal carbon intensity display more complex patterns across the range
of carbon prices considered. Average intensity falls in stepped increments, with particularly strong
jumps occurring at £20, £25 and £35 per tonne due to the interaction between fuel prices and plant
efficiencies (given in Table 1). The clustered demand profiles are able to replicate the general
pattern of the average carbon intensity (Figure 11), but the locations of these step changes are
wrongly predicted. For a narrow range of carbon prices, the clustered results over-predict
emissions, instead of their more usual slight under-prediction. The marginal carbon intensity is
more reliably predicted throughout the whole range, however the absolute errors are of larger
magnitude (both for the gross and net data).
Output from CCGT and coal plant types follow the same pattern as the average carbon
intensity (with CCGT output going as the inverse of coal and carbon); and so these exhibit the same
structural errors between carbon prices of £15 and £25 per tonne. The frequency of plant start-ups
is not reliably modelled at any carbon price with the low number of clusters chosen. It remained at
zero for all trials with the gross data, and varied between 0 and 3 times the true value with the net
data.
Finally, operator profit took a relatively complicated form (as in Figure 12) but was
represented relatively well by the clusters. For both CCGT and coal the deviations increase
substantially at carbon costs above £35 per tonne.
Table 2 summarises the accuracy of the clusters over all carbon costs, giving the mean,
standard deviation and maximum relative error; defined as the difference between clustered and true
values relative to the true value, over the range of £10 to £50 per tonne of carbon. Some general
trends can be seen: costs and average carbon intensity are modelled with the greatest precision,
followed by marginal carbon and plant outputs, and again followed by generator revenues and
profits. The relative error in profits is influenced upwards by the fact that the true values (coming
from the annual data) were often around zero, e.g. for coal plants between £20 and £35 per tonne).
Metric
Average cost
Marginal cost
Average carbon
Marginal carbon
CCGT output
Coal output
CCGT profit
Coal profit
Average
relative error
–0.5 ± 0.2%
–0.8 ± 0.6%
–1.8 ± 2.0%
–3.5 ± 5.2%
–3.5 ± 4.8%
–7.0 ± 6.8%
–4.3 ± 5.3%
–7.4 ± 9.2%
Maximum
relative error
0.9%
1.9%
5.9%
15.1%
16.0%
19.2%
29.1%
32.3%
Table 2: Summary statistics for the deviations between clustered results and those from annual profiles.
What can be concluded from this exercise is that the way in which the clusters deviate – in
terms of systematic or random errors and in terms of the sign or magnitude of the error – cannot
(yet) be predicted in advance. This is an issue that we suggest researchers take into consideration
16
by testing the accuracy of clustering against the complete data set at selected points throughout their
range of input parameters before embarking on a large stochastic study with clustered data.
7
Conclusions
This paper has used the k-means clustering techniques to derive small sets of electricity demand
profiles that can be used in a simulation model to represent a full year of data. This allows a
complex model to be run many times to consider the impact of variations in other input parameters,
such as fuel prices. If a large time-series of operation has to be simulated, restrictions on computing
time might force researchers to use a simpler model, or consider fewer sets of the other variables.
We found it straightforward to obtain clusters from simulated electricity demand and wind
output data for Great Britain in 2020, and to derive appropriate daily profiles for gross demand and
demand net of wind. We found that six profiles gave an adequate representation of gross demand,
while ten were required for the more variable demand net of wind output. These clusters were able
to replicate the results of simulations based on the entire data set to within an accuracy of ±10% in
all cases, and to within ±1% when estimating the average or marginal costs of electricity.
The use of clustered data presents researchers with a trade-off: a reduction in the absolute
accuracy of results against the ability to conduct Monte Carlo or sensitivity studies with one
hundredth of the computing resource. The reduction in uncertainty gained by using more complex
models or broader sets of inputs outweighs that introduced by clustering; and provided that the
errors introduced by clustering are investigated and understood for the specific scenario, it should
be possible to qualitatively (if not quantitatively) correct for them. The clustering technique is
clearly ill-suited to modelling rare events and their consequences, but our work shows that it can be
used to good advantage for simulating normal operations.
8
References
Balachandra P. and V. Chandru (1999), “Modelling electricity demand with representative load
curves”, Energy, Vol. 24, No 3, pp 219-23
Baughman, M.L., and W.W. Lee (1992) “A Monte-Carlo Model for Calculating Spot Market Prices
of Electricity”, IEEE Transactions on Power Systems, vol. 7, is. 2, pp. 584-90
Borenstein, S., J. Bushnell and F. Wolak (2002) “Measuring Market Inefficiencies in California's
Restructured Wholesale Electricity Market” American Economic Review vol. 92, no. 5
(December), pp 1376-1405
Day, R.E. and H.P. Williams (1982) MAGIC: The design and use of an interactive modeling
language for mathematical programming. Tech. rep., Department Business Studies,
University of Edinburgh
Førsund, F.R. (2007) Hydropower Economics, New York, Springer
Gerbec, D. S. Gašperič And F. Gubina, (2003) Comparison of Different Classification Methods for the
Consumers' Load Profile Determination, 17th International Conference on Electricity
Distribution proceedings, Barcelona
Green, R.J. (2008) “Carbon Tax or Carbon Permits: The Impact on Generators‟ Risks,” Energy
Journal, vol. 29, no. 3, pp. 67-89
Green, R.J. and N. Vasilakos (2010) “Market Behaviour with Large Amounts of Intermittent
Generation” Energy Policy, vol. 38, no. 7, pp. 3211-3220
17
Martínez-Álvarez, F., A. Troncoso, J. C. Riquelme, and J. M. Riquelme (2007) “Partitioningclustering techniques applied to the electricity price time series” IDEAL'07 Proceedings of
the 8th international conference on Intelligent data engineering and automated learning,
Berlin, Springer-Verlag
Marton, C.H., A. Elkamel and T.A. Duever (2008), “An order-specific clustering algorithm for the
determination of representative demand curves”, Computers and Chemical Engineering, No
32, pp. 1365–1372
Mazumdar, M and L. Chrzan (1995) “Monte Carlo Stochastic Simulation of Electric Power
Generation System Production Costs Under Time-Dependent Constraints” Electric Power
Systems Research, vol. 35, no. 2, pp. 101-8
Meibom, P., R. Barth, H. Brand, B. Hasche, H. Ravn and C. Weber (2008) “All Island Grid Study”.
Wind Variability Management Studies Workstream 2B
Met Office. (2011) “Historic station data”. Cited June 2011; URL:
http://www.metoffice.gov.uk/climate/uk/stationdata/
Mott MacDonald (2010) UK Generation Costs Update, Brighton, Mott MacDonald
Müsgens, F., and K. Neuhoff (2006) Modelling Dynamic Constraints in Electricity Markets and the
Costs of Uncertain Wind Output, Electricity Policy Research Group working paper CWPE
06010 and EPRG 0514, University of Cambridge
Neuhoff, K., J. Cust and K. Keats (2008) “Implications of intermittency and transmission
constraints for renewables deployment”, in M. Grubb, T. Jamasb and M. Pollitt (eds.)
Delivering a Low Carbon Electricity System: Technologies, Economics and Policy,
Cambridge, Cambridge University Press
Roh, J.H. and M. Shahidehpour (2009) “Market-Based Generation and Transmission Planning With
Uncertainties” IEEE Transactions on Power Systems vol. 24, no. 3, pp. 1587-98
Scully A, A. Harpur, K.D. Le, J.T. Day, M.J. Malone, T.E. Mousseau, S. Hirakawa, A. Brennan,
C.J. Stansberry, T.M. Jackson, T.R.W. Russell and D.J. Anelli (1992) “Using a SemiGuided Monte-Carlo Method for Faster Simulation of Forced Outages of Generating-Units”
IEEE Transactions on Power Systems, vol. 7, no. 3, pp. 1313-21
Troy, N., E. Denny and M. O‟Malley (2008) “The Relationship Between Base-load Generation,
Start-up Costs and Generation Cycling”. Presented at the 28th USAEE/IAEE North
American Conference, New Orleans
18