A VARIATIONAL ENSEMBLE STREAMFLOW PREDICTION ASSESSMENT APPROACH
FOR QUANTIFYING STREAMFLOW FORECAST SKILL ELASTICITY
Andrew W. Wooda, Tom Hopsona, Andy Newmana, Jeff Arnoldb, Levi Brekkec, Martyn Clarka
ABSTRACT
Water resources decision-making commonly depends on monthly-to-seasonal streamflow
forecasts, among other kinds of information. The skill of operational model-based hydrologic
predictions depends on the ability to estimate a watershed’s initial moisture and energy
conditions, to forecast future weather and climate inputs, and on the quality of the hydrologic
model’s representation of watershed processes. We investigate the first two sources of
predictability in an idealized experiment using calibrated hydrologic simulation models for 424watersheds that span the continental US. Earlier work in this area outlined an ensemble-based
strategy for attributing streamflow forecast uncertainty between two endpoints representing zero
and perfect information about future forcings (ie, the National Weather Service ensemble
streamflow prediction, or ESP approach) and initial conditions. This study greatly expands this
approach to characterize the influence of varying levels of uncertainty in each area on
streamflow prediction uncertainty. Ensemble hindcasts are initialized on a monthly basis for
each basin’s periods of record, and forecasts of streamflow for the 1, 3 and 6 month periods
following the initialization are evaluated. Regional and seasonal variations in watershed
hydroclimatology largely determine the relative importance of initial conditions and boundary
forcings, leading to striking differences between rainfall driven and snowmelt driven watersheds.
We define the concept of flow forecast skill elasticities relative to skill in either predictability
source, and use this metric to characterize the regional, seasonal and predictand variations in
flow forecast skill dependencies. The resulting analysis provides insights on the relative benefits
of investments toward improving watershed monitoring (through modeling and measurement)
versus improved climate forecasting. Among other key findings, the results suggest that climate
forecast and initial condition skill improvements can be amplified in streamflow prediction skill,
which means that climate forecasts may have greater benefit for monthly-to-seasonal flow
forecasting than can be estimated from climate forecast skill alone.
a
NCAR Research Applications Laboratory
US Army Corps of Engineers, Institute for Water Resources
c
US Bureau of Reclamation Technical Services Center
b
1
1. INTRODUCTION
Every day, streamflow forecasts are used to support decisions by reservoir operators and water
managers who strive to balance a range of competing objectives. Locally, these might include
preventing floods by capturing water, maintaining cool stream temperatures for fish by releasing
water, or delivering water to irrigators through scheduled releases. At large regional scales,
decisions to store or release water may affect the available supply and water markets for large
US cities or growing regions situated hundreds of miles from the water’s headwater source, or
even have international treaty implications. The need for better streamflow forecasts – from
minutes to seasons – is perennially raised in studies related to water management (eg Raff et al,
2013), and large investments are being made in research, science and technology that are
relevant to streamflow prediction – e.g., in weather, and climate forecasting, in land surface
modeling, and in meteorological monitoring. Yet there is evidence that operational streamflow
forecast quality has not substantially improved in the last decade or more (Welles et al, 2005;
Pagano et al, 2004), thus there is a pressing need to understand where forecasting investments
have the greatest potential to benefit specific streamflow predictions made to support water
management.
Streamflow fluctuations are driven both by meteorological forcings to the contributing
watershed, and by runoff discharging from the watershed’s moisture stores, including soil
moisture (SM), groundwater, snowpack, and the channel network itself. Streamflow forecasts
thus are determined by two major contributing factors – watershed initial moisture conditions
and future weather (for short range forecasts) or climate (for monthly to seasonal forecasts). In
practice, streamflow prediction skill attributable to climate forecasts is low, while skill
attributable to initial conditions varies from non-existent to very high. The largest predictability
at seasonal scales arises where a winter snow accumulation and melt cycle precedes a dry
summer, causing a pronounced surface water hydrologic cycle. The relatively slow process
through which snowmelt raises soil moisture, generates runoff, and routes through a stream
network to produce streamflow yields usable forecast accuracy at lead times of up to six months.
The smallest seasonal streamflow predictability is found at the end of a climatologically dry
period and preceding a wetter one, such that initial hydrologic conditions (IHCs) provide little
contribution to future flows relative to future climate inputs. In this case, nearly all forecast skill
at monthly to seasonal (M2S) lead times derives from the skill of seasonal climate forecasts
(SCFs).
The predictability arising from IHCs has been exploited for nearly a century in seasonal
streamflow forecast practice (Wood and Schaake, 2008). Early efforts used multiple linear
regressions to link in situ observations of watershed conditions mostly in the form of snow water
equivalent (SWE) and accumulated precipitation to future streamflow. Since the 1970s,
hydrologic simulation models have also been used to capture IHC prediction skill, using
observed meteorology to accumulate (‘spin-up’ or ‘initialize’) moisture in the model’s state
variables, and estimates of future meteorology to evolve them into the future, generating
streamflow. Both approaches to prediction are practiced operationally – the former primarily in
the US National Resources Conservation Service and the latter primarily in the US National
Weather Service (NWS) River Forecast Centers (RFCs). Operational practice has long
emphasized IHC predictability, but made relatively few strides to incorporate potential SCF
predictability. In contrast, the land surface hydrology research community has been motivated
by advances in seasonal climate prediction over the last two decades to investigate the potential
2
for SCFs to advance flow prediction, and also tried to quantify the relative influence of each
predictability source. Wood et al (2005) used ensemble hindcasts from a single hydrology
model to quantify the benefits of SCFs relative to IHCs for western US streamflow forecast
locations. Others have demonstrated the influence of both sources of skill on streamflow
predictability in model-based regional to continental scale studies (e.g., Maurer and Lettenmaier,
2003; Berg and Mulroy, 2006; Mahanama and Koster, 2004; Mahanama et al., 2008). Koster et
al. (2010) and Mahanama et al. (2011) recently assessed the contributions of SM and SWE to
streamflow predictability across the conterminous United States, using a suite of hydrology
models to explore inter-model variations.
A number of recent efforts in this vein have employed an uncertainty attribution framework that
contrasts the streamflow forecast variance arising from an ensemble of IHCs, with the forecast
variance arising from an ensemble of boundary forcings. This framework followed earlier
efforts with similar motivations in the atmospheric sciences: e.g., Collins and Allen (2002) used
such a framework to comparing the magnitudes of each type of predictability and conversely the
potential for errors in each source to diminish forecast skill. Wood and Lettenmaier (2008;
hereafter WL08) adopted these concepts, albeit in a limited fashion, focusing on seasonal
streamflow forecast predictability and using climatological variance as a basis for perturbing
IHCs and SCFs. The climatological variance is estimated using historical simulation forcings
and model states: the IHC ensemble contains moisture (soil, snow, canopy, routed flow)
variables on the day of year of the forecast initialization, from all years, and the SCF ensemble
contains historical weather sequences from that day of year forward through the forecast period,
from all years. This framework contrasted forecast uncertainty from two hindcasting ‘end-point’
experiments, one in which ‘perfect’ IHCs are combined with climatological SCFs, and one in
which climatological IHCs are paired with ‘perfect’ SCFs. The former is identical to the
traditional NWS Ensemble Streamflow Prediction (ESP) forecasting approach, while the latter
was termed ‘reverse-ESP’ (WL08).
Though useful for illustrating variations in the behavior of different watersheds (WL08 focused
on just two), the reverse-ESP component was presented as ‘patently artificial’, because realistic
IHC uncertainty is almost always and usually much less than climatological uncertainty.
Operational forecasters can readily distinguish dry watershed conditions from wet ones,
particularly where accumulated watershed moisture is visible in the form of snow. Limitations
notwithstanding, the WL08 predictability construct has since been applied in a number of
predictability studies. Shukla and Lettenmaier (2011) replicated the WL08 analysis for 48
hydrologic units spanning CONUS, and extended the analysis further to a global study region in
Shukla et al (2013), which demonstrated that in general, IHC contributions to seasonal
hydrologic predictability are highest in the arid and snow-dominated climates. Li et al. (2009)
applied the construct to the southeastern US, and in addition explored changes to IHC influence
when climate model forecasts are used instead of the ESP climatology. Paiva et al. (2012)
focused the technique on the Amazon River basin, concluding that the importance of IHCs
indicated potential value for data assimilation. Singla et al. (2012) used a similar framework
(though with different terminology) to study soil moisture and flow predictability in France,
determining that the value of SCFs varied regionally, depending on the influence of climate on
IHCs.
The better to understand hydrologic predictability, and in particular to highlight combinations of
uncertainty that relate more closely to forecasting in practice, this study extends the WL08
3
experimental design to consider different fractions of uncertainty for IHCs and SCFs, spanning a
range from perfect knowledge to climatological uncertainty. A practical goal of this approach is
to provide realistic insights into the potential efficacy of efforts to improve streamflow forecast
accuracy, for different types of monthly to seasonal streamflow forecasts.
2. MODELS, DATA AND METHODS
We pursue the goals of the study through assessment of retrospective predictions (‘hindcasts’)
using a CONUS-wide watershed dataset, supporting demonstration and analysis for a broad
range of hydroclimatic settings. We apply a variational ensemble streamflow prediction
assessment (VESPA) approach that expands upon prior methods to provide a more
comprehensive depiction of hydrologic predictability, and we define the concept of forecast skill
elasticity – the change in streamflow forecast skill due to changes in predictability source skill.
The following subsections describe the models, data, methods and metrics used in the study.
2.1 A CONUS-­‐wide ‘large-­‐sample’ hydrologic modeling dataset We use a CONUS-wide platform of lumped Sacramento Soil Moisture Accounting (SacSMA)
and Snow17 watershed models, forcings and observed streamflow described in Newman et al
(2014). The watershed locations in this ‘large-sample’ modeling collection comprise a subset of
the USGS HCDN-2009 gage network (Lins 2012), that meet the following criteria: 1) have at
least 20 years of complete flow data between 1990-2009 and were active as of 2009, 2) are a
GAGES-II (Falcone, 2011) reference gage, c) have less than 5 percent imperviousness as
measured by the National Land Cover Database (NLCD-2006), and d) passed a manual survey of
human impacts in the basin by local Water Science Center evaluators. Because these basins have
minimal human influence, they are almost exclusively smaller, headwater-type basins, ranging
from wet, warm basins in the southeastern US to hot and dry basins in the southeastern US, to
wet cool basins in the Pacific Northwest and dry cold basins in the intermountain western US.
Daily forcings for the period 1980-2010 were generated for each basin by areally averaging the
daily, 1 km gridded Daymet (Thornton et al. 2012) meteorological dataset over the watershed
drainage areas, generating the input forcings – precipitation and temperature – required to drive
the daily simulation models used in this study. Potential Evapotranspiration (PET), another
required forcing, was derived from Daymet solar radiation via the Priestly Taylor method
(Priestly and Taylor 1972). Daily streamflow data for the associated gage was compiled from
the United States Geological Survey National Water Information System.
The Snow17 and SacSMA model parameters were calibrated to match simulated and observed
daily streamflow for all basins using the Shuffled Complex Evolution global optimization routine
(Duan et al, 1992, 1993), with the single objective of minimizing daily root mean squared error
(RMSE). From the 671 basins in Newman et al (2014), we selected 424 basins (Figure 1) by
choosing up to 30 basins in each of 18 HCDN regions that had the highest Nash Sutcliff
Efficiency (NSE) scores in validation, and excluding any basins with an NSE lower than 0.5.
Not all regions had 30 sites: the Souris-Red-Rainy and Rio Grande regions had fewer than 10 –
8 and 7 respectively.
Two locations, also plotted in Figure 1, were selected for illustrating streamflow predictability
contrasts. The Chattooga River Near Clayton GA (USGS 002177000) is a rain-driven watershed
with contributing drainage area of 530 km2 and a mean basin elevation 2495 feet above MSL.
4
The mean channel slope is 37.20 feet/mile, and river length is 45.80 miles. Less than 2 percent
of contributing drainage area covered by storage, and approximately 96% is forested. Annual
precipitation is 69.00 inches, with rainfall intensities reaching 5.25 inches/day on average once
every two years. January minimum temperature is 30.00 degrees Fahrenheit, and snowfall is
rare. The Chattooga River upstream flows southward along the South Carolina/Georgia border
from its northernmost headwaters in North Carolina, joining the Tallulah River downstream of
the gage to form the major inflows to Lake Tugalo, held back by the Tugalo Dam.
Figure 1. The 424 study locations, with the color and relative size of the plotting symbol
indicating drainage area. The two locations used to illustrate detailed watershed results are
denoted by numbered squares: 1 – Chattooga R; 2 – Crystal River. HCDN regions are also
plotted.
The Crystal River Above Avalanche Creek Near Redstone CO (USGS 009081600) is a
snowmelt driven basin of 428 km2, flowing westward from headwaters in Rocky Mountains,
west of the Continental Divide, before turning north to join the Roaring Fork River, a tributary to
the upper Colorado River. Relative to the Chattooga River, it is steeper (channel slope is 140.00
feet/mile), higher (mean elevation 10200 feet above MSL), shorter (length is 31.00 miles), and
drier, with an annual precipitation of 33.00 inches and rainfall intensities of 1.40 inches/day
expected on average once each two years. The drainage area is 60% forested, containing 2%
storage and lakes, and an average January minimum temperature of 1.00 degrees Fahrenheit
leads precipitation to fall as snow between November and April, reaching SWE of approximately
20 inches at the nearest SNOTEL sites, McClure Pass and North Lost Trail.
The observed precipitation and simulated monthly hydrologic water balances of the two study
basins are shown in Figure 2. The Chattooga River drainage receives a relatively uniform
precipitation input throughout the year and negligible amounts fall as snow. The seasonal cycle
of evapotranspiration, strongest in the summer, creates a moderate seasonal cycle in soil
moisture and runoff, leading to the lowest streamflows occurring in summer. Note, the
5
minimum average SM has been subtracted in the plotted SM values in Figure 2: 26 cm
occurring in August and 78 cm occurring in April for the Chattooga and Crystal River locations,
respectively. The Crystal River drainage receives moderately higher precipitation in the winter
and spring than in the summer, but precipitation falls as snow in the months of October through
April, on average. April sees a mixture of rain, snowfall and snowmelt, spurring rises in soil
moisture and runoff as SWE declines. High evaporation in summer months reduces SM and
runoff, which continues to decline through the fall and into the next winter snow accumulation
period. The Chattooga and Crystal River models were well calibrated, exhibiting daily
streamflow calibration (validation) Nash Sutcliff Efficiency values of 0.895 (0.780) and 0.901
(0.858), respectively.
Figure 2 -- Mean observed precipitation (P) and simulated water balance variables – active soil
moisture (SM), snow water equivalent (SWE), and runoff (RO) for the two study basins. Model
SM is reduced by the lowest mean monthly value so that the plotted values shown only the active
range.
2.2 Variational Ensemble Streamflow Prediction Assessment Approach and Application We extend the WL08 experimental design from the two end-points, ESP and reverse-ESP, to
investigate combinations of different fractions of uncertainty for IHCs and SCFs, spanning the
range from perfect knowledge to climatological uncertainty. Termed VESPA (variational
ensemble streamflow prediction assessment), the approach is illustrated in Figure 3.
6
In contrast to the ESP and reverse-ESP techniques (Fig. 3a,b) that combine a single ‘perfect’
IHC or SCF with an ensemble for the alternate source of predictability, VESPA uses each
member of an ensemble of IHCs to initialize an ensemble of SCFs. Each ensemble is scaled to a
variance between zero and the climatological variance taken from a continuous retrospective
simulation dataset (Fig 3c). The ensemble of IHCs is created by blending the IHC of the forecast
initialization date (eg, 1983 May 1) with the IHC of each member in turn of the climatological
simulation for the same day of year (eg, May 1, from each of the years 1980-2010 in turn). The
blended IHCs are a simple weighted average of the IHC constituent moisture variables Mihc_yr
and those from the blending year, Mblend_yr, according a specified weight (wihc) that varies from
zero to one:
M = Mihc_yr * (1-wihc) + Mblend_yr * wihc
where the state vector S for the Snow17/SACSMA models contains seven carryover moisture
variables (the soil water components – UZTWC, UZFWC, LZTWC, LZFSC, LZFPC, ADIMC –
and snow water equivalent). Channel input (tci) from SACSMA to the routing model (the unit
hydrograph method) were also blended with this approach, including inputs several days prior to
the forecast date, back to the time of concentration of the watershed.
Figure 3. Schematic illustrating hydrologic state evolution in the VESPA framework. The end
points of (a) ESP and (b) reverse-ESP concept pair perfect and climatological knowledge in
IHCs and SCFs, where climatology for IHCs and climate forcings is defined from (c) continuous
retrospective hydrologic simulations. The VESPA approach (d) combines intermediate blends
(depicted by the black dashed lines) of climatological IHC variance and SCF variance (ranges
depicted by purple arrows), exploring combinations of variance between zero (perfect
7
knowledge) and full climatology (no knowledge). For clarity, only three combinations of the
SCF ensemble with IHC members are drawn.
The climate forecast ensemble blends are constructed in a similar fashion. In each retrospective
forecast, the year of historical forcings following the forecast initialization date represents the
perfect climate forecast, Cfcst_yr. The ensemble SCF is created by blending this year with each
other historical forecast year in turn. The weighted average of the two is calculated on a monthly
basis: for each member of the climate forecast ensemble, C, the daily sequences that make up
Cfcst_yr are adjusted so that the monthly totals (for precipitation) and averages (for temperature)
equal weighted averages of Cfcst_yr and each blending year, Cblend_yr taken from the historical
climatology. If the weight (wscf) equals one, Cfcst_yr is unchanged, a perfect climate forecast. If
wscf equals zero, F has the daily pattern of Cfcst_yr with the monthly climate of Cblend_yr, and the
entire ensemble of C has the monthly spread of an ESP forecast. The blending is expressed by:
C = Ffcst_yr*(1-wscf) * Cblend_yr*wscf
In each case, the weights indicate the amount of uncertainty present in each predictability source:
zero signifies zero uncertainty, or perfect knowledge, whereas one signifies complete
uncertainty, as estimated by historical climatology. The square of the weight is proportional to
the fraction of climatological variance present in the uncertainty. ESP is represented by wihc=0
and wscf=1, and reverse-ESP is represented by wihc=1 and wscf=0.
In this study, we assess combinations of 9 weight values, w = (0, 0.05, 0.10, 0.25, 0.50, 0.75,
0.90, 0.95, 1.0), leading to 81 different blends of IHC and SCF uncertainty, or 81 hindcasts per
location. Given w, the percent variance explained v = (100, 99.8, 99.0, 93.8, 75.0, 43.8, 19.0,
9.8, 0). We draw from a 30-year retrospective climatology period, 1981-2010. All of the nonzero weight combinations are run with 30*30 or 900 members, although the members in which
the blend year matched the forecast year or initial condition year were later removed during
evaluation. Twelve forecast initialization dates are used – the first day of each calendar month,
for a retrospective forecast period of 30 years. The total number of 30-year long simulations
required for the study totaled approximately 370 million, not including simulations performed
during method development. The runs were executed on the NCAR-Wyoming Yellowstone
supercomputer.
2.3 Evaluation Metrics and Skill Elasticities We evaluate three predictands – forecasts of mean streamflow for 1-month, 3-month and 6month periods. The study objective is to discriminate IHC and SCF influences on prediction
skill and uncertainty, thus the forecasts are verified against simulated streamflow forced by
meteorological observations. Consequently, the forecast errors depend solely on the SCF and
IHC errors, and do not include modeling error. For readability, we refer to the simulated
observed streamflow simply as ‘observations’ or ‘observed flow’. We use one primary metric to
define forecast skill, the squared correlation of forecast ensemble medians with observations, r2,
which estimates the percentage of predictor variance explained by the forecast.
The assessment of streamflow forecast skill resulting from multiple combinations of IHC and
SCF skill allows the calculation of gradients in forecast skill relative to IHC and SCF skill,
which we term streamflow forecast skill elasticities. These gradients vary for each predictability
source, depending on the skill of the other source. The elasticity (E) is defined in this study as
the unit change in flow forecast skill for a unit change in predictability source skill. We
8
calculate E in piecewise-linear fashion over the intervals formed by the IHC and SCF variance
weights.
This calculation is illustrated through focusing on the weight combination that is of interest in
this study. The Results section reports elasticities for a combination weight point that
realistically represents the current state of M2S forecasting knowledge. Model-based estimates
of initial watershed conditions have relatively low error, and we use the interval of v between
93.8% and 75.0% (or about 6-25% uncertainty, mean 16%). Operational climate forecasts have
nearly climatological uncertainty, and we use the interval of v between 0% and 9.8% (or about
90-100% uncertainty, mean 95%). The streamflow forecast skill elasticities of interest for IHCs
and SCFs, respectively, are calculated from the skill slopes across these intervals, i.e.,:
!(! !
!! ,!
!!" )!!(! !
𝐸!"# 𝑣!"# = 14 , 𝑣!"# = 95 = !(! !!"# !! ,!!"# !!" )!!(! !!"#
!"#
!"#
!!" ,!!"# !!" ) !"# !!" ,!!"# !!"
)
and
𝐸!"# 𝑣!"# = 14 , 𝑣!"# = 95 =
!(! !!"# !! ,!!"# !!" )!!(! !!"# !! ,!!"# !!"" ) !(! !!"# !! ,!!"# !!" )!!(! !!"# !! ,!!"# !!"" )
where S() is a skill metric, such as r2. Elasticities are calculated separately for each predictand
(1-month, 3-month and 6-month flows) and for each forecast initialization date. An elasticity of
zero means that improvements or losses in skill of a predictability source have no influence on
the skill of the flow forecast, whereas positive elasticities indicate that improving a predictability
source will improve the flow forecast skill.
3. RESULTS
The application of the VESPA approach is presented first via a comparison of results for the two
focus watersheds, followed by aggregated regional analyses.
3.1 Predictability variations in two watersheds On any given day of the year, a watershed experiences season-specific land surface conditions
and also expects season-specific forecast period climate conditions, thus the climatology of a
forecast can vary markedly with initialization season and forecast period. Figures 4 and 5
illustrate such variations, showing 6-month hindcasts made with different levels of uncertainty
for the Chattooga and Crystal River study locations, respectively. The top plot in each figure
shows the combination of 100% climatological uncertainty for both IHCs and SCFs, revealing
broad spread and a pattern that repeats each year due to a lack of any information specific to a
given year. The next two plots show reverse-ESP and ESP forecasts and the bottom two plots
show two combinations of partial reductions in IHC and SCF uncertainty – a high reduction
leading to 25% uncertainty, and a low reduction leading to 81% uncertainty.
For the Chattooga River, the elimination of SCF uncertainty in reverse-ESP dramatically reduces
the spread and improves the skill of the forecasts, even while the IHCs contain no information,
for all forecast initialization months. In contrast, the ESP hindcasts in which IHC uncertainty is
eliminated retain ample spread and show less improvement in the hindcast correspondence with
observations. These effects are also evident in the bottom two plots, where high reductions in
SCF uncertainty (to 25%) are more effective at improving forecast skill than low ones with high
9
Ensemble 6mo Flow Forecast
002177000 CHATTOOGA RIVER NEAR CLAYTON GA
flow (cfs)
250
200
obs flow
IHC 100% SCF 100%
150
100
50
0
1981
1982
1983
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
flow (cfs)
250
200
IHC 100% SCF 0%
150
100
50
0
1981
1982
1983
flow (cfs)
250
200
IHC 0% SCF 100%
150
100
50
0
1981
1982
1983
flow (cfs)
250
200
IHC 81% SCF 25%
150
100
50
0
1981
1982
1983
flow (cfs)
250
200
IHC 25% SCF 81%
150
100
50
0
1981
1982
1983
Figure 4. Time-series of 6-month ensemble flow forecasts initialized each month, compared to
simulated observations, for varying levels of IHC and SCF climatological uncertainty, for the
Chattooga River location. Gray box and whisker symbols show each hindcast distribution
minimum, maximum and quartiles. The symbols are plotted in the month of initialization – e.g.,
the first data in each year represent a January-June hindcast and observation.
10
Ensemble 6mo Flow Forecast
009081600 CRYSTAL RIVER AB AVALANCHE CRK NEAR REDSTONE CO
300
obs flow
flow (cfs)
IHC 100% SCF 100%
200
100
0
1981
1982
1983
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
1984
1985
1986
1987
1988
300
flow (cfs)
IHC 100% SCF 0%
200
100
0
1981
1982
1983
300
flow (cfs)
IHC 0% SCF 100%
200
100
0
1981
1982
1983
300
flow (cfs)
IHC 81% SCF 25%
200
100
0
1981
1982
1983
300
flow (cfs)
IHC 25% SCF 81%
200
100
0
1981
1982
1983
Figure 5. Same as Figure 4 but for the Crystal River location.
reductions to IHC uncertainty. The Crystal River (Figure 5) hindcasts have greater seasonal
climatological variation in both mean and spread (top plot), with large uncertainty in high flow
months and narrow uncertainty in low flow months. The reverse ESP and ESP hindcasts show
greater seasonal variations in uncertainty influences than were found for the Chattooga River.
11
Reducing SCF uncertainty to zero most improves the hindcasts (ie, reduces spread and median
error) in fall and winter, whereas reducing IHC uncertainty most improves the hindcasts in
spring and summer. It is evident that the uncertainty of a December 6-month forecast is almost
entirely controlled by SCF skill, whereas IHC skill almost entirely determines June hindcast
uncertainty. The final two plots show these effects less clearly, due to the blending of
uncertainty contributions.
Further insight into the performance of streamflow hindcasts illustrated in Figures 6 and 7 can be
gained by plotting the skill of the forecast for all 81 30-year hindcasts, versus the skill (or
conversely uncertainty) in the two predictability sources, IHCs and SCFs. Figures 6 and 7 show
contours of the streamflow forecast skill for the two locations for all 12 initialization dates, for
the 6-month and 1-month predictands, respectively. In these figures, approximate values for skill
of the ESP (E), reverse-ESP (rE), perfect forecast (P), and climatological (C) prediction are
found in the top left, bottom right, bottom left, and top right of each box, respectively. In Figure
6, the rainfall-driven Chattooga River watershed (left) shows relatively uniform predictability
gradients throughout the year, with slightly greater predictability in fall than in summer. For all
forecast initialization months, the skill gradient slope indicates that SCF skill has a stronger
influence than IHC skill, though IHC skill also affects streamflow forecast skill.
Skill of Mean 6mo Runoff Forecast
Crystal River Ab Avalance Crk Nr Redstone CO
Skill of Mean 6mo Runoff Forecast
Chattooga River Nr Clayton GA
Oct 1
Dec 1
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.0
P
rE
Jan 1
Feb 1
Mar 1
1.0
0.8
0.6
0.4
0.2
0.0
Apr 1
May 1
Jun 1
1.0
0.8
0.6
0.4
0.2
Jul 1
Aug 1
0.0
1.0
Dec 1
1.0
P
Feb 1
Mar 1
1.0
rE
Jan 1
0.9
0.9
0.8 0.8
0.8
0.6 0.7
0.7
0.4
0.0
1.0
0.6
0.6
0.2
0.5
0.4
0.5
Apr 1
May 1
Jun 1
0.4
0.8 0.3
0.3
0.6 0.2
0.2
0.4
0.1
0.1
0.2
0.0
0.0
Jul 1
Sep 1
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2
0.0
Nov 1
C
0.2
0.0
0.0
E
R2
C
SCF Uncertainty (Fraction of Climo Variance)
SCF Uncertainty (Fraction of Climo Variance)
E
R2
Nov 1
Oct 1
1.0
Aug 1
Sep 1
0.0
0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0
IHC Uncertainty (Fraction of Climo Variance)
IHC Uncertainty (Fraction of Climo Variance)
Figure 6. Six-month streamflow ensemble median forecast skill (R2) for two locations, for
initializations on the first day of each month, versus uncertainty in the two predictability sources
(fraction of climatological variance).
12
The controls on snowmelt-driven Crystal River 6-month streamflow forecast skill vary by
month, according to the hydrologic cycle depicted in Figure 2. The water year begins with IHC
uncertainty controlling streamflow forecast skill because runoff from October-March is largely
baseflow-driven. The climate variability of the late fall and winter begin to affect runoff in
April, during snowmelt. As April and the following months enter the forecast period, the skill
gradients in the plots first flatten, indicating that SCFs are critical, and gradually steepen as the
accumulating snowpack transfers predictability from the climate to the land surface. From
March through August, predictability is dominated by IHC skill and uncertainty. In September,
SCFs regain importance because climate directly affects runoff, until snow again begins to build.
The 1-month streamflow forecast skill gradients (Figure 7) show similar contrasts between
locations, and for the Crystal River site, between forecast months. The Chattooga River
gradients are steeper than for the 6-month forecasts because IHCs exert greater control over
runoff. In the Crystal River watershed, SCFs are no longer influential in the fall months because
the snowmelt runoff that is linked to SCF variability is not within the forecast period. IHCs
dominate 1-month forecast skill in the fall and late summer. April and March show a strong SCF
control (the horizontal skill gradients), indicating that climate variability within the snowmelt
period can drive runoff directly. SCFs and IHCs have mixed influence in September and
October, when declining SM variability balances climate variability in determining runoff (see
Figure 2).
13
Figure 7. Same as Figure 6, but showing 1-month streamflow forecast skill.
Another way to use the multiple skill combinations of the variational ensemble assessment to
understand contributions to streamflow forecast skill is to examine the effects of adding skill in
one predictability source, given fixed uncertainty in the other. In Figure 8, we show the effects
for each calendar month initialization of augmenting skill in each source from a baseline of
climatological uncertainty, for increments of 19%, 44%, 75%, and 100% skill (i.e., variance
reduction in either IHCs or SCFs). For the Chattooga River location, adding SCF skill has
relatively more benefit than adding IHC skill as the prediction period lengthens from 1 month to
6 months (read top to bottom). The variations month to month reflect sampling uncertainty
(N=30 for each hindcast) given the modest seasonal cycle of the watershed, although the results
suggest a relatively stronger IHC (and lower SCF) influence for fall month seasonal forecasts.
Streamflow Forecast Skill Given Single Source Skill
1.0
Init Condition
Clim Forecast
0.8
skill (R2)
0.8
Crystal R
1 month flow forecast
0.6
0.6
0.4
0.4
0.2
0.2
0.0
skill (R2)
Chattooga R
1.0
0.0
O N D J F M A M J
J A S
O N D J F M A M J
J A S
0.8
0.8
skill (R2)
1.0
0.6
0.6
0.4
0.4
0.2
0.2
0.0
skill (R2)
3 month flow forecast
1.0
0.0
O N D J F M A M J
J A S
O N D J F M A M J
J A S
0.8
0.8
skill (R2)
1.0
0.6
0.6
0.4
0.4
0.2
0.2
0.0
skill (R2)
6 month flow forecast
1.0
0.0
O N D J F M A M J
InitMonth
J A S
O N D J F M A M J
InitMonth
J A S
Figure 8. Streamflow forecast skill sensitivity to skill in each predictability source, initial
condition (IHC) and climate forecast (SCF), for different initialization months. The black thick
line shows the climatological skill, each of the thin colored dashed lines show the addition of
14
19%, 44% and 75% skill (lowest to highest), and the thick colored line shows the addition of
100% skill, for each predictability source separately (assuming 0% skill in the alternate source).
For the Crystal River (right), IHCs dominate predictability during most forecast initialization
months, and for the 1- and 3-month forecasts, can lead to perfect skill despite a lack of
information in the climate forecasts, for snow accumulation months when soil moisture controls
runoff. SCFs are important for forecast periods that include the spring months of high runoff
variability (mostly March, April and May).
3.2 Regional comparisons The results for the two focus locations gave an example of the regional difference in the relative
contributions of IHC and SCF skill to flow forecast skill. The concept of streamflow forecast
skill elasticities (Section 3.1.3), the unit change in flow forecast skill per unit change in a
predictor skill, provides a useful metric for inter-comparing skill sensitivities and ultimately for
discussing the value of improvements in a predictability source for a given location, initialization
date and predictand. Figure 9 shows elasticities for all study locations, for one forecast date
(November 1) and predictand (3-month streamflow) to illustrate the regional variation in skill
dependence in more detail. The elasticities are calculated around the skill combination point that
best approximates skill levels in an ESP forecast: 16% IHC uncertainty and 95% SCF
uncertainty (assuming some climate information informs the ESP, such as from the ENSO state).
Despite much site-by-site variation, three broad areas emerge. Flow forecast skill for the west
coast of the US, a maritime climate regime that experiences a climatologically wet season, with
rainfall-driven runoff in November-January, with little to moderate snow accumulation, is
sensitive to SCF but not IHC skill. The intermountain western US and northern plains regions,
which experience sub-freezing winter temperatures and a prominent snow cycle, in contrast,
indicates a strong role for IHC skill but not SCF skill. Lastly, the eastern US, a more humid
regime with rainfall-driven runoff and variable winter snow in the north, shows a mixture of skill
influences over 3-month streamflow forecasts. As one can surmise from the more detailed
results of the two study locations, these sensitivities vary by forecast start date and predictand,
and also (as can be gathered from the skill gradients in Figures 6-7) depending on the
‘predictability point’ at which the elasticities are calculated.
It is not practical to show similar results for the remaining 35 combinations of forecast date and
predictands. We instead summarize regional findings for monthly variations in skill elasticities
in Figures 10-12, which are for 1-month, 3-month and 6-month streamflow forecasts,
respectively. We use the same predictability combination point so that the results describe, in
effect, potential skill improvement benefits relative to current practice. As might be expected,
the generally higher IHC elasticities in Figure 10 compared to the generally higher SCF
elasticities in Figure 12 confirm that progressively longer forecast periods translate the
streamflow prediction challenge from a mostly IHC-driven problem toward an SCF-driven one.
Even for the shorter 1-month predictand in Figure 10, though, several regions exhibit higher SCF
elasticities. The Ohio, Tennessee and Lower Mississippi regions may have faster-responding
soils that drain quickly and reduce the runoff persistence from soil moisture, and may also
experience high intensity tropically-fed storms that overwhelm runoff regulation by land surface
storages.
For 3-month streamflow forecasts, eastern regions (such as the Mid-Atlantic) show SCF skill
rising above ICF skill in importance, while in the western US, the hydroclimatic patterns of
15
either wet and dry seasons, or cold and warm seasons, lead to seasonal reversals in the dominant
influences on streamflow prediction skill. During the dry California and Northwest summers, for
instance, SCF skill has little value for improving streamflow predictions, versus high value for
forecasts leading into or beginning in the wetter fall and winter months. In the Rio Grande River
basin, like the Colorado, the snow cycle leads to three distinct predictability regimes: the late
summer to winter high IHC predictability as runoff is driven by soil moisture variability; the late
spring snowmelt runoff regime that is driven by a mixture of climate variability, and late spring
to summer regime that is driven by snowmelt and rising soil moisture.
16
Figure 9. Skill elasticities for 3-month streamflow forecast initialized November 1, relative to
SCF and IHC skill.
Figure 10. Regionally averaged streamflow forecast skill elasticities for 1-month predictions on
the first day of every month. Elasticities are shown for a baseline capability with a combination
of IHC and SCF skill levels of approximately 16% and 95%, respectively.
17
Figure 11. Same as Figure 10, but for 3-month flow predictions.
Figure 12. Same as Figure 10, but for 6-month flow predictions.
18
For six-month streamflow predictions (Figure 12), SCFs become the dominant streamflow
predictability source in the Northwest, California (until spring), the Midwestern regions and
much of the eastern US. The Great Basin, upper Colorado and Souris-Red-Rainy regions sustain
strong spring through end-of-summer IHC influence, in part due to the snow cycle. A dip in the
summer through early fall SCF predictability in the Ohio, Upper Mississippi, Tennessee and both
Atlantic regions appears to be related to the depletion of soil moisture during summer and the
fate of precipitation towards soil moisture recharge during fall. [I need to reanalyze this]
4. DISCUSSION AND CONCLUSIONS
We apply a model-based variational ensemble streamflow prediction assessment (VESPA)
approach to investigate the major sources of streamflow predictability across a wide range of
different CONUS streamflow locations, forecast dates and periods. The approach is motivated
by interest in quantifying the relative value of attempting to advance monthly to seasonal
streamflow forecasting through investing in improvements to the major sources of predictability
-- our ability to estimate current watershed conditions and to upgrade and apply watershed-scale
climate predictions. To this end, we define the metric of a forecast skill elasticity – ie, the unit
change in flow forecast skill perunit change in a predictability source skill – to facilitate
translation of source skill capability improvements to potential benefits for streamflow prediction
skill. For a specific forecast to support a specific reservoir operation (e.g., an Idaho watershed
June 1 3-month runoff forecast to inform summer irrigation water allocations) VESPA and skill
elasticities can suggest both where efforts should be devoted and also how much improvement
might be expected, given the current state of the practice and realistic potential improvements in
predictability sources.
Many of the skill sensitivity results in this paper are recognizable to those engaged in the practice
of hydrologic prediction, or in academic research into land surface predictability. For instance,
the dominant contribution of IHCs (particularly SWE) to long lead predictability in the western
US is a foundation of operational seasonal forecasting, and a reason why climate forecasting,
with both weaker skill and weaker flow forecast skill elasticity in many months, has been
relatively undeveloped. The link between dry or sub-freezing climate periods, which restrict
variation in hydrologic fluxes, and higher streamflow predictability corroborates earlier findings
(Wood and Lettenmaier, 2008; Shukla et al, 2011; among others), which also linked wet climate
periods with lower predictability.
The quantification of skill elasticities adds a useful dimension to these insights. A key finding is
that in many western US locations, the elasticities are greater than one for both IHCs and SCFs.
For SCFs, this result argues that the low levels of skill for precipitation forecasting can provide
usable improvements in seasonal streamflow prediction. This finding contradicts a perception
that has been common in operational flow forecasting, where even some regions of high climate
predictability and potential management value (such as the hydropower-rich Northwest), have
not developed SCF-based flow predictions. The reasons for this amplification of skill between
IHCs or SCFs and streamflow are not clear, but may relate to sub-unity runoff ratios, nonlinearities or threshold behavior in the forcing-runoff response, and multi-season formation of
snow or soil moisture anomalies, which could allow a commensurate accumulation of skill.
Lastly, the variation of elasticities by month, region, and predictand has an important implication
for climate prediction evaluation and services from the water-sector perspective. The best
climate forecast across a range of these dimensions (e.g., for all 3-month seasons and regions) is
19
not likely to be the best climate forecast for any particular season, location, prediction, and
decision use. In the age of multiple climate forecast options (such as the National Multi-model
Ensemble; ref), users would benefit from determining their choice or weighting of climate
forecast sources with specific predictands in mind. Generic measures of aggregate skill from
forecast producers may have limited utility in helping users design their SCF forecasting
application.
One goal of this paper is to demonstrate VESPA as a strategy for understanding skill
dependencies, and we make a number of choices that could be handled differently in other
studies.
We use a single skill metric (R2), relatively simple calibrated models
(Snow17/SACSMA), a linear scaling approach to blending IHCs and SCFs, a limited number of
predictands (3) and forecast dates (12), and a specific set of skill combinations (81) that may not
resolve skill gradients optimally for many purposes. A second objective of the paper, however,
is to provide a broad-based analysis of skill elasticities across CONUS, sufficient to provide
insights into hydroclimatically-driven variations in skill sensitivity at the watershed level. The
resulting hydrologic hindcasting dataset is by far the largest ever generated. Although only
summary level insights from the comprehensive study could be included here, the two focus
watersheds illustrate the more detailed assessment that has been created for all 424-watersheds.
This paper’s attributions of flow forecast uncertainty in the model-based experiment lays a
foundation for future work that includes the components of model uncertainty and total
streamflow prediction uncertainty. These can be estimated through validation of retrospective
simulated streamflow, and through verification of streamflow hindcast predictions, against
observed streamflows. Combining these components with independent estimates of climate
prediction skill will allow the hard-to-measure predictability component – watershed condition
uncertainty – to be inferred. From this larger perspective, this paper describes only part of a full
decomposition of streamflow prediction skill and uncertainty, the end goal of which is to support
the design of research and development toward improving streamflow forecasting and ultimately,
water management.
ACKNOWLEDGEMENTS
The research and analysis reported herein were supported by the US Bureau of Reclamation and
the US Army Corps of Engineers [add details of programs/grant #s]
REFERENCES
Berg, A. A. and K.A. Mulroy, 2006. Streamflow predictability in the Saskatchewan/Nelson
River basin given macroscale estimates of the initial soil moisture status, Hydrological
Sciences 51(4): 642-654.
Collins, M. and M.R. Allen, 2002, Assessing the Relative Roles of Initial and Boundary
Conditions in Interannual to Decadal Climate Predictability, J. Climate 15:21, 3104-3109,
10.1175/1520-0442(2002)015.
Duan, Q., S. Sorooshian, and V. K. Gupta (1992). Effective and efficient global optimization for
conceptual rainfall-runoff models. Water Resources Research, 28(4): 1015-1031.
20
Duan, Q., V. K. Gupta, and S. Sorooshian (1993). A shuffled complex evolution approach for
effective and efficient optimization. Journal of Optimization Theory and Applications, 76(3):
501-521.
Falcone, J. A. (2011). GAGES-II: Geospatial Attributes of Gages for Evaluating Streamflow.
Digital spatial data set 2011. Available online at:
http://water.usgs.gov/GIS/metadata/usgswrd/XML/gagesII_Sept2011.xml
Gustafsson, N., Kallen, E. and S. Thorsteinsson, 1998. Sensitivity of forecast errors to initial and
lateral boundary conditions. Tellus, Series A (Dynamic Meteorology and Oceanography),
50A:2, 167-85.
Koster, R. D., Mahanama, S. P. P., Livneh, B., Lettenmaier, D. P., and Reichle, R. H.: Skill in
streamflow forecasts derived from large-scale estimates of soil moisture and snow, Nat.
Geosci., 3, 613–616, 2010.
Koster, R.D., and M.J. Suarez, 2004: Suggestions in the Observational Record of Land–
Atmosphere Feedback Operating at Seasonal Time Scales. J. Hydrometeor., 5, 567–572.
Li, H., Luo, L., Wood, E. F., and Schaake, J.: The role of initial conditions and forcing
uncertainties in seasonal hydrologic forecasting, J. Geophys. Res., 114, D04114,
doi:10.1029/2008JD010969, 2009.
Lins, H. F. (2012). USGS Hydro-Climatic Data Network 2009 (HCDN-2009), U. S. Geological
Survey, Fact Sheet 2012-3047.
Mahanama, S. P. P., and R. D. Koster, 2003. Intercomparison of soil moisture memory in two
land surface models, J Hydrometeor. 4: 1134{1146.
Mahanama, S.P.P., R.D. Koster, R. H. Reichle, and L. Zubair, 2008. The role of Soil Moisture
Initialization in Subseasonal and Seasonal Streamflow Predictability: A case study in Sri
Lanka, Adv. Wat. Resources (in review).
Mahanama, S. P., Livneh, B., Koster, R. D., Lettenmaier, D. P., and Reichle, R. H.: Soil
Moisture, Snow, and Seasonal Streamflow Forecasts in the United States, e-View,
doi:10.1175/JHM-D-11- 046.1, 2011 Maurer, E. P., and D. P. Lettenmaier, 2003.
Predictability of seasonal runoff in the Mississippi River basin, J Geophysical Res. 108 (D16)
8607, doi:10.1029/2002JD002555.
Newman, A and many authors, 2014. Development of a large-sample watershed-scale
hydrometeorological dataset for the contiguous USA: Dataset characteristics and assessment
of regional variability in hydrologic model performance. Hydrol. Earth System Sciences (in
review).
Pagano, T., D. Garen, and S. Sorooshian (2004) Evaluation of official western U.S. seasonal
water supply outlooks, 1922–2002. J. Hydrometeorology 5: 896–909.
Paiva, R. C. D., Collischonn, W., Bonnet, M. P., and de Gonc ̧alves, L. G. G.: On the sources of
hydrological prediction uncer- tainty in the Amazon, Hydrol. Earth Syst. Sci., 16, 3127–3137,
doi:10.5194/hess-16-3127-2012, 2012.
Priestly, C. H. B. and R. J. Taylor (1972). On the assessment of surface heat flux and evaporation
using large-scale parameters. Monthly Weather Review, 100:81-82.
21
Singla, S, Ce ́ron, J.-P., Martin, E., Regimbeau, F., De ́que ́, M., Ha- bets, F., and Vidal, J.-P.:
Predictability of soil moisture and river flows over France for the spring season, Hydrol. Earth
Syst. Sci., 16, 201–216, doi:10.5194/hess-16-201-2012, 2012.
Shukla, S and Lettenmaier, DP, 2011: Seasonal hydrologic predic- tion in the United States:
understanding the role of initial hy- drologic conditions and seasonal climate forecast skill,
Hydrol. Earth Syst. Sci., 15, 3529–3538, doi:10.5194/hess-15-3529- 2011.
Shukla S, J Sheffield, EF Wood, and DP Lettenmaier1 2013. On the sources of global land
surface hydrologic predictability, Hydrol. Earth Syst. Sci., 17, 2781–2796, 2013 www.hydrolearth-syst-sci.net/17/2781/2013/ doi:10.5194/hess-17-2781-2013
Thornton, P. E., M. M. Thornton, B. W. Mayer, N. Wilhelmi, Y. Wei, and R. B. Cook (2012).
Daymet: Daily surface weather on a 1 km grid for North America, 1980-2012. Acquired
online (http://daymet.ornl.gov/) on 15/07/2013 from Oak Ridge National Laboratory
Distributed Active Archive Center, Oak Ridge, Tennessee, USA.
http://doi:10.3334/ORNLDAAC/Daymet_V2
Wood, A.W. and J.C. Schaake, 2008, Correcting errors in streamflow forecast ensemble mean
and spread, J. Hydrometeorology 9:1, 132-148.
Wood, A. W. and Lettenmaier, D. P.: An ensemble approach for attribution of hydrologic
prediction uncertainty, Geophys. Res. Lett., 35, L14401, doi:10.1029/2008GL034648, 2008.
Wood, A.W., A. Kumar and D.P. Lettenmaier, 2005. A retrospective assessment of NCEP
climate model-based ensemble hydrologic forecasting in the western U.S., J Geophys Res
2005; 110: 4105, doi:10,1029/2004JD004508.
Welles, E., S. Sorooshian, G. Carter, and B. Olsen (2007) Hydrologic verification: A call for
action and collaboration. Bull. Amer. Met. Society 88: 503–511.
22