Market Performance: a Hindcasting Perspective EPOC Jul 2014 Version 2.1 Grant Telfar, Meridian Energy July 2014 2 Introduction & Background • • Over recent years the NZ electricity industry has been buffeted by unprecedented levels of dissatisfaction from the wider public, policy makers, consumer groups, and regulators. Dissatisfaction has been particularly focused on market and regulatory structures and intensified over the 2008-13 period due to: − − − − − − • Winter 2008: the 4th ‘dry winter’ since 2001. Retail tariffs: up 70+% since 2001. ComCom: the Wolak report alleging $4.3B of excess profits since 2001. 2009 Ministerial Review: suggestion of poor retail competition. 2010 Electricity Industry Act: suggestion of overreliance on public conservation campaigns and mis-management of reservoir operation. 2012-3 the emergence of NZ Power, growing awareness of fuel poverty, and equity issues. Three years on from the implementation of the EIA there is a natural question as to how the ‘new’ wholesale market is performing: − Especially in the light of the extreme dry of 2012 • With market listing complete additional scrutiny of industry and company performance is likely: – • Hindcasting is an analytical approach that can be applied to examine how decision support tools, analysis, and key assumptions performed after the fact: – – • A priori we forecast a range of future outcomes. Ex post we observe actual market outcomes. Meridian has undertaken five hindcast exercises in recent years – the first in 2008 and most recently in early 2013: – • As well as a desire to understand and analyse key industry drivers and their impact. Hindcasting is seen by Meridian as part of good internal management. In early 2013, Meridian undertook an updated hindcasting exercise: – – – In particular an examination of the 2012 year. Seeking to isolate individual forecast assumptions, their impacts, and to test their relevance. Seeking to embed the hindcasting discipline within Meridian to compliment its existing selfexamination process and metrics. 3 Updating Meridian’s Hindcasting Perspective • • Meridian has undertaken five hindcasting exercises in recent years Some common conclusions were reached: − − − • – • • • This is the type of view that regulators typically apply on behalf of consumers and tax-payers. Once outcomes have been assessed against this metric then additional questions of appropriate commercial performance may be posed. A range of potential benchmarks can be suggested for examining the NZ power system. Traditionally in the NZ context stochastic reservoir and power system models are applied: – – – – Focusing on wholesale market outcomes. Covering the Jul-2009 to Dec-2012 period. As with previous hindcast exercises, we begin to answer the question of market performance (and Meridian’s ability to assess it) by first addressing the question of what is an appropriate metric to use in measuring the actions of the market. It is cleanest to start with a metric that judges behaviour from the perspective of what is best for NZ: – We focus here on the most recent of these – the 2012 hindcast exercise: − − • Market outcomes are largely determined by environmental factors – inflows and the unavoidable costs of generating and managing the system. The market is not a perfect reflection of a centrally controlled benchmark – but it is surprisingly close to one with reservoirs being managed in the best interests of NZ. A range of decision support models demonstrate a good ability to reflect market outcomes. • Spectra SDDP DOASA EMarket From a NZ inc perspective most of these models seek to balance the costs of excess thermal fuel burn against the costs of excess system shortage: – – In the face of significantly uncertain hydro inflows. To minimise the overall NZ fuel (offer) supply cost. 4 2012 Hindcasting • We configure a number of decision support models to reflect the fundamental underlying costs of electricity supply and demand in NZ: − − • Spectra: legacy ECNZ stochastic DP – 2 node LPcon: Meridian internal stochastic DP – 22 node Emarket: Energy Link market simulation tool – 2 node & 22 node configuration Implicit in the above assumptions, we are now only examining a key residual problem: the NZ hydro-thermal reservoir management problem: – Approach intended to be broadly consistent with the system short-run marginal cost (SRMC). NOT a re-litigation of the market’s operational decisions – rather we are using the models in the role of a proxy ‘regulatory benchmark’. – The decision support models used are: − − − • • – • Beginning in Jul09: 1. We update each model for events of Jul09-Dec12: − − − − − − − − Storage conditions as at Jul2009. Actual inflows and wind conditions. Market geothermal and co-generation output. Observed thermal and geothermal availability (planned & unplanned outages). Observed hydro and wind availability (planned & unplanned outages). Observed market HVDC configuration/outages. Observed market NI and SI demand. Assumed thermal SRMC offers and fuel costs. 2. 3. • All demand conditions, geothermal output, and plant outages are assumed to be known with perfect foresight – rather than via the application of generic planning assumptions. Whereas hydro and wind conditions are assumed to be unknowable. Thermal and hydro offer behaviours are assumed to be broadly cost reflective (a modified SRMC). Optimise the use of water in storage in the face of historical hydrological uncertainty – ie all weekly flows over the 1931-2012 period. Run the model to simulate the 42 month Jul09 to Dec12 period for all hydrological sequences. Examine the single inflow ‘sequence’ corresponding to the Jul09 to Dec12 period. Having done this we can now compare modelled benchmark results to actual market outcomes. 5 The Full Forecast Distribution 1,000 800 600 400 Dec12 Sep12 Apr12 Jun12 Jan12 Oct11 Jul11 Feb11 May11 Nov10 Aug10 May10 Mar10 Dec09 Sep09 - SI Hindcast SRMC Prices $250 $250 $200 $200 $150 $150 $100 $100 Dec12 Sep12 Jun12 Apr12 Jan12 Jul11 Oct11 May11 Feb11 Nov10 $Aug10 $May10 $50 Mar10 $50 Dec09 This holds true for both the ‘dry’ period of 2012 and the ‘wet’ period of 2009-11 Now we examine the single hydrological forecast consistent with what occurred on the day … 1,200 200 We can observe that prices and storage have oscillated within the extreme bounds of the feasible forecast produced (in this case by Spectra) beginning in Jul2009. – • 1,400 Jun09 • 1,600 Jun09 – We can view this via a familiar distributional perspective. These price and storage charts show a wide range of possible outcomes (in blue). Superimposed over the top of these charts are the actual market outcomes (in red) as occurred in reality. 1,800 Sep09 – – Pukaki Hindcast Storage stored energyy [GWh] Beginning in Jul09 each model forecasts a wide range of outcomes over the following 42 month period corresponding to 82 different historical hydrological sequences: baseload price [$/MWh] • 6 Comparison: Storage Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Sep10 Jun10 Mar10 Dec09 Dec10 3,750 3,500 3,250 3,000 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 0 Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Dec10 This broadly implies that there are no gross differences in the use of water between the models. Sep10 Of Hawea storage is not great. Of Taupo storage is adequate. Of the split between Tekapo and Pukaki is mixed but never fully satisfactory. PKI+TEK Storage LPcon PKI+TEK Storage EMarket-LP PKI+TEK Historical Average Hyd Seq Beginning: 2009-2010 Jun10 • • • 3,750 3,500 3,250 3,000 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 - Mar10 There are divergences between modelled outcomes but these are small (in general). Note that in all models (to differing degrees) the handling: 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 0 NZ Hydro Storage: LPcon, Spectra & EMarket Dec09 – • PKI+TEK Storage Market PKI+TEK Storage Spectra PKI+TEK Storage EMarket-SP Total NZ storage levels have show a very good level of alignment between the all modelled results: – Sep09 To some extent in terms of practical use storage outcomes in all models are configurable and if storage/risks implied here are considered unacceptable then outcomes can be altered. Sep09 – In the 2012 situation this is closer to market outcomes. Jun09 • Hyd Seq Beginning: 2009-2010 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 - Jun09 – There are some divergences between modelled outcomes – but these are not large. However late 2011 and winter 2012 show notable differences with LPcon holding the Southern lakes first lower and then higher than Spectra and Emarket: storage [GWh] – • Pukaki & Tekapo Hydro Storage: LPcon, Spectra & EMarket The storage levels from all models show good alignment over an extraordinary 42 month period that has traversed both an extreme dry period and an extended extreme wet period: storage [GWh] • ALL Storage Market ALL Storage LPcon ALL Storage Spectra ALL Storage EMarket-LP ALL Storage EMarket-SP NZ Historical Average 7 Comparison: Generation Meridian Hydro Generation: LPcon, Spectra & EMarket Meridian total market generation: 150 100 50 Eg higher levels of spill encountered in LPcon – at both Manapouri and Pukaki/Tekapo. Weekly Thermal Generation: LPcon, Spectra & EMarket - Market Huntly Average - 50 100 150 200 week generation [GWh] 250 CCGT OCGT - 50 100 150 200 week generation [GWh] 250 Market LPcon Dec12 Sep12 Jun12 Apr12 Jan12 Oct11 Jul11 May11 Feb11 Nov10 Aug10 Spectra EMarket-LPcon Sep12 Market 50 Jun12 Lpcon 100 Mar12 Lpcon 150 Dec11 Spectra 200 Sep11 Spectra 250 Jun11 Emarket-Lpcon 300 Mar11 Emarket-Lpcon 350 Dec10 Emarket-Spectra Hyd Seq Beginning: 2009-2010 Sep10 Emarket-Spectra LPcon EMarket-Spectra 400 generation [GWh] Meridian Weekly Generation: LPcon, Spectra & EMarket Market EMarket-LPcon Thermal Generation: LPcon, Spectra & EMarket Thermal volumes are high in LPcon & EMarket-LPcon reflecting higher levels of spill in all Southern catchments. Thermal volumes are particularly high in EMarket-Spectra due to Taupo spill effects (static reserves representation) – StdDev Man+Wind Spectra Jun10 – May10 Thermal generation trends are very similar between the models: Mar10 - Dec09 • 200 Jun09 • 250 Dec09 There are minor differences between modelled generation levels (highest in LPcon and all higher than market) driven by spill differences: 300 Sep09 – Reflected in the standard deviations of weekly generation. 350 Jun09 • Hyd Seq Beginning: 2009-2010 400 Sep09 Is delivered in a more volatile manner by LPcon and EMarket at the weekly level than suggested by Spectra – perhaps reflecting the LP nature of the underlying analytical engines, differences in tributary flow modelling, or both: generation [GWh] – Mar10 • EMarket-Spectra 8 Comparison: Market Prices $175 $150 $150 $125 $125 $100 Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Dec10 Hyd Seq Beginning: 2009-2010 $225 $225 $200 $200 $175 $175 $150 $150 $125 $125 $100 $100 BEN LPcon BEN EMarket-LP BEN2201 Market BEN EMarket-SP Sep12 Jun12 $Mar12 $25 $- Dec11 $50 $25 Sep11 $50 Jun11 $75 Mar11 $75 Dec10 A range of reasonable sensitivities can drive a movement in average prices of +/30%. NI Spectra SI Wholesale Market Prices: LPcon, Spectra & EMarket Sep10 – HLY2201 Market HLY EMarket-SP Jun10 Note that price outcomes show significantly more sensitivity to changes in models and/or assumptions than do physical outcomes: HLY LPcon HLY EMarket-LP Mar10 – Sep10 $- Jun10 $25 $- Mar10 $50 $25 Dec09 $50 Sep09 $75 Dec09 – Both the SI and the NI show good general alignment. There are clearly short lived spot market events that different models reflect quite different: eg Aug-2011 There are market events that none of the models reflect well: eg Nov-2010 The largest divergence is the second half of 2012 where SI Spectra and EMarket-Spectra prices increase to high levels. $100 $75 Sep09 Market $ 64 $ 54 $ 49 $ 91 $200 $175 Jun09 Spectra $ 62 $ 57 $ 51 $ 75 $225 $200 The general shape of modelled prices from all models track together reasonably tightly through high priced dry periods and low priced wet periods: – – • EMarketSpectra $ 68 $ 57 $ 54 $ 75 $225 Jun09 LPcon Jul09-Dec12 $ 64 FY2010 $ 63 FY2011 $ 53 FY2012 $ 80 EMarketLPcon $ 54 $ 48 $ 37 $ 72 Hyd Seq Beginning: 2009-2010 baseload price [$/MWh] NZ Average Price • NI Wholesale Market Prices: LPcon, Spectra & EMarket Average modelled NZ prices over the full 42 month period are similar, all in a $55-$65 range compared to $64/MWh in the market: baseload price [$/MWh] • SI Spectra 9 LPcon, Spectra, and EMarket Hindcast Jul2009 to Dec2012 HydroSeq:2009-2012 Comparison: Market Outcomes • High level outcomes over the Jul-2009 to Dec-2012 period are broadly similar between all of the models: – – – • • • – Generation There are variations in prices, spill, and thermal generation with physical outcomes in general being better aligned than pricing outcomes. Storage levels are similar overall - but over the crucial 2012 period LPcon held Southern reservoirs at higher levels than delivered by the other models (but lower during late 2011). Modelled generation revenues are similar – driven by modelled prices. Through a period of starkly different market conditions the consistency between the models is good. In broad terms this means that all the models are reflecting very similar inputs, market drivers, and underlying economic rationales. While the consistency of model outcomes – both between different models and within the same model but using different input assumptions is broadly good: – Prices Primal outcomes (physical – eg storage) are significantly more robust than dual (eg price, revenue) outcomes. Eg, changes to thermal offer/fuel price assumptions or shortage assumptions or ... may yield storage outcomes that are only modestly different but present price outcomes that vary much more significantly. Dem Res HVDC Spill Storage SI NI SI-S/FIR NI-S/FIR Pukaki Manapouri MEL Wind Taupo Waikaremoana TPD Tekapo Clutha TPL NI Hydro TPL SI Hydro Todd Hydro Other NI Hydro Other SI Hydro Geothermal Co-Gen Other Aux Huntly CCGTs OCGTs TOTAL Hydro TOTAL Thermal TOTAL Other NI SI S->N Pukaki Tekapo Clutha Manapouri Taupo Waitaki Avg Tekapo Avg Hawea Avg Taupo Avg LPcon $ 60.1 $ 66.7 $ 2.9 $ 16.2 22,821 16,563 3,837 14,591 2,049 4,733 3,260 12,541 4,941 3,662 1,077 531 841 14,295 4,899 10,015 10,047 20,755 2,062 87,611 32,863 33,046 0 25 6,192 1,921 274 1,149 1,710 321 1,101 545 219 373 EMarketLPcon $ 53.3 $ 54.7 $ 2.3 $ 5.8 23,849 16,977 3,862 14,707 1,972 4,599 3,560 12,237 4,743 3,253 925 1,033 737 17,064 4,605 5,016 10,468 20,129 2,024 88,594 32,620 30,547 7,009 1,140 270 1,198 390 321 1,220 507 215 401 EMarketSpectra $ 64.3 $ 69.7 $ $ 24,024 16,466 3,866 12,013 1,864 4,601 3,678 11,851 4,743 3,314 925 1,033 737 17,067 4,605 5,047 11,951 21,751 2,450 85,250 36,151 30,585 13 2 6,478 772 143 1,610 1,511 2,963 1,176 511 219 259 Spectra $ 61.4 $ 62.3 $ $ 23,430 16,698 3,869 14,258 1,998 4,629 3,488 12,840 5,339 3,571 1,111 799 790 17,628 4,899 6,537 9,158 20,370 2,027 88,951 31,555 32,932 6 80 7,100 1,397 508 934 320 701 1,071 550 125 468 Market $ 61.3 $ 65.4 $ 1.4 $ 3.9 22,552 16,575 3,863 14,230 1,935 4,762 3,581 12,441 5,276 3,712 1,140 480 537 17,628 4,899 6,537 8,656 21,619 2,324 87,220 32,599 32,927 - - 444 5,731 2,275 385 1,050 320 226 1,213 564 170 357 10 Conclusions • Market outcomes have broadly matched benchmark modelled results: − • − • Comfort should be taken from the fact that: – All models considered have matched high level market outcomes. Market outcomes through both an extreme dry period (2012) and a prolonged extreme wet period (2009-2010) have been largely driven by the unavoidable costs of generating and managing the system: − − • – – • Market results are still dominated by hydrology. Physical reservoir management is all about managing what inflows turn up, when, and with very limited storage capacity. • Market price outcomes are the result of balancing escalating thermal costs against too much reservoir spill in a fashion that ensures security of supply is not compromised. However care should be taken in drawing strong conclusions about the goodness of outcomes without an appropriate context and sensitivities: – − This is to be expected. However the gap between market outcomes and benchmark outcomes is not large. Imperfections should be considered in the light of the significant successes – particularly in market investment and the allocation of risk. Price outcomes in particular show a large sensitivity to changes in input assumptions. Three years on from the implementation of the EIA what conclusions can be drawn about the ‘new’ market arrangements: – The market has not been a perfect reflection of a centrally controlled benchmark: − − Reservoir operation (both physical and pricing) is being managed in a rational manner. Market outcomes are close to what could be achieved under central control. A range of different reservoir management tools demonstrate a good ability to shadow the market. Not a lot has changed – indeed similar behaviours are seen in aggregate to those observed in previous hindcast exercises: • • – – Thermal stations generate when they ‘should’. Reservoirs are being managed ‘appropriately’. However market prices do appear more volatile with short lived events pushing prices up quickly. There is some small evidence of the market holding Waitaki reservoirs higher than the benchmark during 2012. Additional Material EPOC Jul 2014 Version 1.0 Grant Telfar, Meridian Energy July 2014 12 Hindcasting at Meridian • Hindcasting is an analytical approach that can be applied to examine how decision support tools, analysis, and key assumptions performed after the fact: − − • We seek to answer the question of market performance by first addressing the question of what is an appropriate metric to use in measuring the actions of the market: − • A priori forecast vs ex post outcomes. Hindcasting is seen by Meridian as part of good internal management. Starting with a metric that judges behaviour from the perspective of what is best for NZ. Traditionally in the NZ context stochastic reservoir and power system models are applied: − − − − − Spectra SDDP DOASA Emarket ... • From a NZ inc perspective these models seek to balance the costs of excess thermal fuel burn against the costs of excess system shortage: – – • The decision support models used here: − − − • In the face of significant hydro uncertainty. To minimise the NZ fuel (offer) cost. Spectra: legacy ECNZ stochastic DDP LPcon: Meridian internal stochastic DDP EMarket: Energy Link market simulation tool – 2 node & 22 node configuration LPcon is an in-house Meridian hydrothermal power system model: − − − − − − − 2 stage optimisation/simulation. Weekly resolution and a 15 block LDC. Stochastic DDP creating water-values. Simple thermal offers/cost. DC load flow – 22 regions used. Dynamic risk and reserves. Diurnal wind characteristics. 13 Comparison: Pukaki and Tekapo Storage PKI+TEK Storage Market PKI+TEK Storage Spectra PKI+TEK Storage EMarket-SP Sep12 Jun12 Mar12 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 0 Dec11 Sep11 Jun11 Mar11 Dec10 Sep10 Jun10 Dec09 Sep09 Mar10 Hyd Seq Beginning: 2009-2010 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 Jun09 storage [GWh] Pukaki & Tekapo Hydro Storage: LPcon, Spectra & EMarket PKI+TEK Storage LPcon PKI+TEK Storage EMarket-LP PKI+TEK Historical Average 14 Comparison: NZ Storage (No Manapouri) 3,750 3,500 3,250 3,000 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 0 Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Dec10 Sep10 Jun10 Mar10 Dec09 Hyd Seq Beginning: 2009-2010 Sep09 3,750 3,500 3,250 3,000 2,750 2,500 2,250 2,000 1,750 1,500 1,250 1,000 750 500 250 - Jun09 storage [GWh] NZ Hydro Storage: LPcon, Spectra & EMarket ALL Storage Market ALL Storage LPcon ALL Storage Spectra ALL Storage EMarket-LP ALL Storage EMarket-SP NZ Historical Average 15 Comparison: Meridian Generation Meridian Hydro Generation: LPcon, Spectra & EMarket Hyd Seq Beginning: 2009-2010 400 350 250 200 150 100 50 Man+Wind Market LPcon Spectra EMarket-LPcon Dec12 Sep12 Jun12 Apr12 Jan12 Oct11 Jul11 May11 Feb11 Nov10 Aug10 May10 Mar10 Dec09 Sep09 - Jun09 generation [GWh] 300 EMarket-Spectra 16 Comparison: Thermal Generation Thermal Generation: LPcon, Spectra & EMarket Hyd Seq Beginning: 2009-2010 400 300 250 200 150 100 Market LPcon Spectra EMarket-LPcon Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Dec10 Sep10 Jun10 Mar10 Dec09 - Sep09 50 Jun09 generation [GWh] 350 EMarket-Spectra 17 Comparison: NI Market Prices NI Wholesale Market Prices: LPcon, Spectra & EMarket $225 $225 $200 $200 $175 $175 $150 $150 $125 $125 $100 $100 HLY LPcon HLY EMarket-LP HLY2201 Market HLY EMarket-SP Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 $- Mar11 $- Dec10 $25 Sep10 $25 Jun10 $50 Mar10 $50 Dec09 $75 Sep09 $75 Jun09 baseload price [$/MWh] Hyd Seq Beginning: 2009-2010 NI Spectra 18 Comparison: SI Market Prices SI Wholesale Market Prices: LPcon, Spectra & EMarket $225 $225 $200 $200 $175 $175 $150 $150 $125 $125 $100 $100 BEN LPcon BEN EMarket-LP BEN2201 Market BEN EMarket-SP Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 $- Mar11 $- Dec10 $25 Sep10 $25 Jun10 $50 Mar10 $50 Dec09 $75 Sep09 $75 Jun09 baseload price [$/MWh] Hyd Seq Beginning: 2009-2010 SI Spectra 19 Comparison: HVDC Transfers HVDC Transfers: LPcon, Spectra & EMarket Hyd Seq Beginning: 2009-2010 75 Sep12 Jun12 Mar12 Dec11 Sep11 Jun11 Mar11 Dec10 Sep10 Jun10 Mar10 Dec09 -25 Sep09 25 Jun09 energy transfer [GWh] 125 -75 Market LPcon Spectra EMarket-LPcon EMarket-Spectra 20 Conclusions • • • Market outcomes have broadly matched benchmark modelled results. Market outcomes through both an extreme dry period (2012) and a prolonged extreme wet period (2009-2010) have been largely • driven by the unavoidable costs of generating and managing the system. The market has not been a perfect reflection of a centrally controlled benchmark: − − • However the gap between market outcomes and benchmark outcomes is not large. Imperfections should be considered in the light of significant successes – particularly in market investment and the allocation of risk. Comfort should be taken from the fact that: − Reservoir operation (both physical and pricing) is being managed in a rational manner. • – – Market outcomes are close to what could be achieved under central control. A range of different reservoir management tools demonstrate a good ability to shadow the market. However care should be taken in drawing strong conclusions about the goodness of outcomes without an appropriate context and sensitivities: – Price outcomes in particular show a large sensitivity to changes in input assumptions. Three years on from EIA implementation not a lot seems to have changed – similar behaviours are seen to those observed in previous hindcast exercises: • • • Thermal stations generate when they ‘should’. Reservoirs are being managed ‘appropriately’. Market prices appear reasonable.
© Copyright 2026 Paperzz