Assessment of the Importance of Input Variables on Yield of Urban Water Supply Systems –
Use of Morris Method of Sensitivity Analysis
D. M. King* and B. J. C. Perera*
* School of Engineering and Science, Victoria University, PO BOX 14428, Melbourne, VIC 8001,
Melbourne, Australia
(email: [email protected]; [email protected])
Abstract
This paper presents a sensitivity analysis (SA) of the input variables used in the estimation of
yield, considering multiple hydroclimatic scenarios of system inflow, rainfall, evaporation and
demand. The Barwon urban water supply system in Australia was considered as the case study,
whilst the Morris method was used for the SA. Input variables of the simulation model of the
Barwon system were divided into two categories for use in the SA analysis: the climate dependant
variables (i.e. system inflow, rainfall, evaporation and demand) which were used to generate
various climate scenarios, and the system policy variables which were assumed to have knowledge
deficiency in relation to their optimum values. Twenty climate scenarios of various lengths were
selected from historic data record of the climate dependant variables. SA was performed on the
system policy variables over each of the twenty climate scenarios. The security of supply
thresholds were found to be the most important input variables, followed by the upper restriction
rule curve.
Keywords
Morris method, Sensitivity Analysis, Uncertainty, Variability, Yield estimation, Barwon water
supply system
INTRODUCTION
Balancing water demand with available water supply is a challenging task for water authorities
throughout the world. Growing population in urban cities and the effect of climate change have put
immense pressure on the management and operation of urban water supply systems. Such pressures
have been exerted on most Australian water supply systems, resulting in long restriction periods,
and in some cases the introduction of permanent water saving measures (Byrnes et al., 2006). A
supply/demand shortfall from an urban water supply system can be addressed by decreasing
demand via water saving measures and schemes, and education, or by increasing water supply
through optimisation of system operation, and/or augmentation with additional water sources. All of
these methods and many operational processes rely heavily on the yield estimate of the water
supply system.
Yield plays a central role in the planning and management of water resources and water supply
systems. It represents the performance of the physical system and the optimum operation and
management of the system. Yield is defined in this study and throughout much of the Australian
water industry as the maximum average annual volume of water that can be supplied from the water
supply system subject to climate variability, operating rules, demand pattern and adopted level of
service (or security criteria) (Erlanger and Neal, 2005; DSE, 2011). Operating rules include water
restriction rules, storage targets and environmental flow rules. The security of supply criteria are
defined by thresholds that safeguard against system failure, such as minimum storage threshold and
reliability of supply threshold. If one or more security criteria threshold is violated, the system is
deemed to have failed in the definition of system yield. Yield is simply the maximum allowable
average annual demand that the system can supply without adverse, long term drawdown resulting
in system failure.
In Australia, the yield of a water supply system is typically estimated by simulating the water
supply system using a computational model of the physical system considering the entire length of
available climate data. This is done by simulating the water supply system in a computational model
and iteratively increasing the average annual demand until system failure. The REALM (REsource
ALlocation Model) simulation software package is used extensively in the water supply industry in
Australia to simulate water supply systems (Perera et al., 2003; Perera et al., 2005).
Sensitivity Analysis (SA) is the study of the relationship between inputs and outputs of a
computational model (Saltelli et al., 2000). SA can measure which of the model’s input variables
carry the most importance to the model output, i.e. to which variables are the model and hence the
model output, most sensitive to variation. Commentary regarding SA theory, and applications from
hydrology and water resources, and various other scientific fields can be found in Saltelli et al.
(2000; 2004), Ratto et al. (2007), King (2009), and references therein. In an SA, input variables are
perturbed and the output observed. The greater the change in the model output, the greater the
sensitivity of the model to the input variable, or, the more important the input variable to the model.
The Morris Method of SA (Morris, 1991) is a qualitative technique that allows efficiently ranking
of input variables according to their importance (Campolongo et al., 2000; 2007).
The current paper follows a proof-of-concept case study by King and Perera (2010) which used the
Morris Method, the Fourier Amplitude Sensitivity Test (FAST) and the extended FAST methods of
SA to study the importance of input variables used in the estimation of yield of a simple case study
(VU and DSE, 2008). This case study dealt with an urban water supply system with two reservoirs
and a single demand centre. The yield was estimated using REALM, considering a single climate
sequence consisting of 28 years of monthly climate data; consisting of streamflow, evaporation,
rainfall and climate index. The SA framework adopted in King and Perera (2010) considered 26
input variables identified in the definition of yield, assuming they were subject to only data
measurement and handling errors. Results of the study showed that the yield was highly sensitive to
the streamflow variable, with the supply of reliability threshold, the upper restriction rule curve and
the consecutive months threshold of subsequent importance. Although the SA procedure was
successful, it was found that the methodology and SA framework used in King and Perera (2010)
needed improvement.
The main recommendation drawn from the case study presented by King and Perera (2010) was to
separate the input variables used in the estimation of yield into climate dependant variables and
system policy variables, and perform SA on the system policy variables using several climate
scenarios. The current study follows this recommendation, considering the Barwon urban water
supply system in Australia as the case study. The sources of uncertainty and variation are newly
considered, with the two groups of variables related elegantly to two sources of yield variation. The
climate dependant variables (i.e. streamflow, rainfall, evaporation and demand) are subject to
natural variability due to their inherent climate uncertainty. The system policy variables (i.e.
security of supply thresholds, restriction rule curves, and target storage curves) are controlled by
water authorities and in this study were deemed to be subject to knowledge deficiency in regards to
their optimum values. Whereas knowledge deficiency can be measured and refined, natural
variability will always be present and largely cannot be reduced. Since natural variability cannot be
reduced, only the knowledge deficiency of the system policy variables can be assessed. See King
(2009) and King and Perera (in press) for details.
The SA framework adopted in this paper is to consider multiple climate scenarios of various
simulation lengths over which the importance of the system policy variables are determined using
SA. This SA framework allows the effect of climate variability and simulation length on the
importance of input variables to be explicitly assessed whilst testing climate dependant variables via
the climate scenarios. The framework can then be used to determine if climate variability and/or
simulation length affects the importance of the controllable system policy variables used in the
estimation of yield. The Morris method of SA (Morris, 1991) was selected for use in this paper as it
is a reliable and efficient SA technique, shown to be successful when applied to a water supply
system planning model (King and Perera, 2010).
The Barwon urban water supply system is described first in this paper. It is followed by a further
description of the SA framework together with a brief introduction to the Morris method, including
sensitivity measures. The results of the SA of input variables used in the estimation of yield is then
described, with conclusions of the paper are presented in the final section.
BARWON URBAN WATER SUPPLY SYSTEM
The Barwon urban water supply system in Australia is owned and managed by Barwon Water
Corporation and considered as the case study in this paper. It is situated on a regional and coastal
area in south east Australia, 60 km south west of Melbourne. It supplies over approximately 43,000
ML a year to 285,000 permanent residents in an 8,100 km2 area around Greater Geelong. The
Barwon system headworks consists of six major reservoirs, 5,000 km of pipes, six water treatment
plants and nine water reclamation plants. Water is sourced from two rivers and their catchments,
and a number of groundwater sources.
Figure 1. Headworks Schematic of the Barwon Water system
A REALM model of the Barwon urban water supply system was available from Barwon Water
Corporation for this study (SKM, 2006). Input variables for the Barwon system REALM model
include streamflow (to model inflow into the system), evaporation and rainfall (to model the losses
and gains from storages), demand pattern and system policy variables (that dictate the system
behaviour). Geelong is the only demand centre considered in this study. Environmental flows and
other demands were assumed compulsory and excluded from the scope of this study. Fourteen
system policy variables were used in the operation of the Barwon system. These are two supply
security thresholds, a variable describing target storage curves and 11 variables used to model the
four-level demand restriction policy. A conceptual schematic of the Barwon system REALM model
is shown in Figure 1.
Barwon Water Corporation considers two supply security thresholds in managing the Barwon water
supply system, and its system model: the reliability of supply and the minimum level of storage.
The reliability of supply is the ratio of the number of time steps not subject to water consumption
restrictions to the total number of time steps considered in the planning period. A typical value for
this is 95%, which allows the system to have water restrictions of up to 5% of the planning period.
The minimum storage level threshold is triggered by a low volume of total system storage,
considering the six main reservoirs of the Barwon system. If either of these criteria is violated, the
system is deemed to have failed under the given conditions in the Barwon Water Corporation
planning studies. The estimation of yield is therefore directly linked to these SA variables which
can fail individually or at the same time.
A four-level demand restriction policy is implemented for the Barwon system, denoted as
Restriction Rule Curves (RRC). Consisting of upper and lower rule curves, three intermediate
restriction zones (with definitions of relative positions and percentage restrictable levels) and a base
demand curve, it is used to restrict the outdoor water demand during low storage volume periods.
See King (2009) and King and Perera (in press) for further details and nominal values of the RRC
used in this study, including the values for the upper, lower, and intermediate zone curves, and the
percentage restrictable levels in each zone.
Target storage curves are defined by a set of five-point curves for all months of the year, indicating
the preferred distribution of individual storage volumes for various total system storage volumes.
The curves impose inter-storage transfers to distribute water in the system so to supply the required
demands at various demand points. See King (2009) and King and Perera (in press) for details.
METHODS AND TECHNIQUES
As stated earlier, the Morris method of SA was used in this study. The Morris method (Morris
1991) is a specialised randomised One-At-a-Time (OAT) SA design that is an efficient and reliable
technique to identify and rank important variables (Morris, 1991; Campolongo et al., 2000; 2007).
The assumption of an OAT SA design is that if variables are changed by the same relative amount,
the variable that causes the largest variation in the output is the most important. Figure 2a shows
that in a traditional OAT SA design each variable is tested individually, by changing the variable in
question between a pair of model simulations. The pairs of model simulations can be considered as
experiments. The standardised effect of a positive or negative Δ change (or step) of an input
variable can be calculated using Equation (1). This is also known as the Elementary Effect (EE –
Morris, 1991). Traditional OAT SA design requires 2k model simulations to determine an EE for
each of the k input variables.
EEi (x)  [ y( x1 , x2 ,..., xi 1, xi  , xi 1,..., xk )  y( x)] / 
where
Δ
p
(1)
is the magnitude of a step, a multiple of 1/(p-1)
is the number of ‘levels’, or values, over which the variables can be
sampled. This is also known as the resolution of sampling
Figure 2.The Region of Experimentation, Ω.
(a) Individual EEs for a Three Variable Model. Six Simulations Required.
(b) Trajectory EEs for a Three Variable Model. Four Simulations Required.
The Morris Method uses the same principle of changing each variable one at a time by Δ but as
shown in Figure 2b, the experiments are arranged so as to create a trajectory through the variable
space. Since the experiments share simulation points in a trajectory, the Morris Method requires
k+1 model simulations to calculate one EE for each of the k input variables. Several trajectories (r)
are constructed providing r EEs for each of the k input variable.
Morris (1991) proposed two measures, namely the mean (μ) and standard deviation (σ) of the set of
EEs for each input variable, while Campolongo et al. (2007) introduce a third index, μ*, which
gives extra information. They are calculated using:
r
i 
i 
 EE
n
n 1
(2)
r
2
1 r
EEn  i 


r n1
(3)
r
 
*
i
 EE
n
n 1
r
(4)
The sensitivity index μi, calculated using Equation (2), assesses the sensitivity strength between the
i-th input variable and the output response due to all first- and higher-order effects that are
associated with that variable (Campolongo and Braddock, 1999). When μi is high in contrast to
other variables, the output has a high sensitivity to this input variable. Conversely, a variable with a
low μi value has small sensitivity associated to it as the same Δ change causes a relatively low
change in output. Equation (3) is used to determine the spread (variance) of the finite distribution of
the EEi values, which is denoted by σi. It indicates possible interactions with other variables and/or
that the variable has a non-linear effect on the output (Campolongo and Braddock, 1999).
Campolongo et al. (2007) discuss the use of μi*, the mean of the finite distribution of absolute
values of the EEi, as given in Equation (4). From here onwards, the subscript i is left out from the
discussion in this paper for easy reading.
The index μ* provides a ‘true’ importance measure, free of any non-monotonic input to output
behaviour that could be present in μ. That is μ* provides the overall sensitivity of the i-th input
variable void of any cancelling out effects that may be contained in μ. The µ and µ* are the
accepted sensitivity measures of the Morris method. However, due to the assumed linear input to
output relationship, the sparse sampling and low number of model simulations, the rankings on µ*
are more important indicator of sensitivity than the quantitative measures (Braddock and Schreider,
2006).
Weekly historic climate data (streamflow, rainfall, evaporation and demand) for a 77 year period
beginning on 1st January 1927 was available. A 20 year moving window of total streamflow
volumes (of the system) was calculated. Five sequences were selected with a range of total system
inflow volumes. For each streamflow sequence selected, the same period of the remaining climate
dependant variables (i.e. rainfall, evaporation and demand) were also selected to complete each
scenario and maintain cross correlations. The same procedure was used to select five 40 year and
five 60 year scenarios. Two additional 20 year scenarios were selected that have a similar total
streamflow volume as scenario 2, but at significantly different positions in the historic sequence.
These are labelled scenario 2b and 2c. The rank, total streamflow volumes and the starting time step
of the selected 20, 40 and 60 year scenarios are given in Table 1. Each scenario is identified through
the starting time step indicated by Year.Week, i.e. 1945.42 represents the 42nd week of 1945.
Table 1. Selected Scenarios
20 Year Scenarios
Total
Starting
Rank Streamflow
Year.Week
(Ml)
1 2990 3,665,400 1945.42
2 2242 3,351,200 1960.39
3 1495 3,137,500 1967.45
4 747 2,925,500 1936.29
5
1
2,487,400 1927.01
2b 2243 3,355,900 1952.37
2c 2244 3,353,700 1943.33
Rank
1951
1464
976
488
1
40 Year Scenarios
Total
Streamflow Year.Week
(Ml)
6,730,400 1951.07
6,600,700 1949.17
6,430,200 1957.50
6,160,200 1931.35
5,839,500 1964.26
Rank
911
684
456
228
1
60 Year Scenarios
Total
Streamflow Year.Week
(Ml)
9,509,100 1939.08
9,384,300 1941.22
9,306,700 1940.03
9,228,000 1930.32
9,152,000 1928.25
As stated earlier, the 14 system policy variables consist of two security of supply thresholds, a
target storage curves variable and 11 variables that control the RRC policy. The target storage
curves variable is a discrete representation of possible storage behaviour. The 11 RRC variables
model the RRC policy by changing the position and curvature of the upper, intermediate, lower and
base demand curves, and the percentage of demand restrictable in each stage. The upper and lower
RRC curvature variables modify the shape of the trigger levels over the 12 months of the year, the
upper and lower RRC position variables change the whole curve positions against the total system
storage, and the three relative positions determine the positions of the intermediate curves between
the upper and lower curves (creating four restriction levels). The three percentage restrictable
variables model the outdoor demand restricted in levels 1 to 3, with level 4 being 100% outdoor use
restricted. The base demand curve models indoor water use which is not restricted. See King (2009)
and King and Perera (in press) for their nominal values, and permutation ranges and strategies.
RESULTS AND DISCUSSION
A 50 trajectory, eight level (p = 8), Δ = 4 Morris design was used on each of the 20 climate
scenarios. Fifty trajectories were deemed to be sufficient to ensure that convergence of SA indices
were satisfied, whilst p and Δ were selected to provide a range of possible sampling points and a
wide variable perturbation. The work done in King and Perera (2010) showed that little was gained
by averaging results from experiments using different number of levels and different Δ values. The
extensive µ, µ* and σ measures of the 20 experiments are omitted from this paper for brevity,
however, they are summarised by importance ranking in Table 2. Importance ranks are based on µ*
which is not subject to cancelling out that can occur in µ. Rank 1 refers to the variable with the
greatest µ* value, i.e. the model and its output are most sensitive to that variable. Table 2 shows the
variable ranks based on the µ* and σ indices for all climate scenarios with colour shading that
highlight the top six ranked variables.
Table 2. Variable ranks from µ* and σ for all climate scenarios.
The supply reliability and the minimum storage thresholds were the two most important input
variables in all scenarios over all simulation lengths (except 20 year scenario 1). The importance of
these thresholds is expected as they directly influence the performance of the system by defining
system failure. The importance measure rankings (µ*) and the non-linearity/interaction rankings (σ)
of the top six variable are generally the same, however in different orders. These variables are: the
two threshold variables already mentioned, the two upper RRC variables, the target storage curve
variable and the base demand. The remaining variables do not show notable µ* ranking patterns in
Table 2 and consistently have a low importance measure (see King and Perera, in press), showing
negligible importance.
The µ* rankings of the two threshold variables reflect which is critical within each climate scenario,
i.e. which threshold drives system failure. Supply reliability is critical when the simulation length is
20 years, except for scenario 3. Observation of the 20 year scenario 3 streamflow sequence shows a
large and severe drought is present in the sequence which produces a rapid system drawdown,
triggering the minimum storage threshold. The minimum storage threshold is the critical threshold
for all 40, 60 and 77 year scenarios, except for the 40 year scenario 4; which does not trigger
minimum storage threshold due to the absence of a significant storage drawdown from the relatively
constant streamflow of that scenario.
The pattern described in the previous paragraph shows that simulation length is a key driver to the
type of failure of the system. The reliability of supply criteria causes system failure at a certain
percentage of the simulation length, whilst the minimum storage level threshold must be violated
within fewer time steps to be the reason for system failure. For a short simulation length a severe
drought is required to cause such an acute system drawdown for minimum storage level threshold to
cause system failure. This is the case in the 20 year Scenario 3. Longer simulation lengths are more
likely to have a sufficient drought present in the climate sequence and there are more time-steps for
drawdown to occur before the reliability of supply threshold is met and violated.
When the supply reliability threshold has µ* ranked 1 (i.e. supply reliability threshold is the critical
security of supply criteria), the two upper RRC variables (i.e. curvature and position) are ranked 3
and 4, or higher. They also have a µ* that is considerably higher than the other variables (King and
Perera, in press). This relationship is due to the position of the upper RRC defining when supply
reliability threshold is triggered. Conversely, when the minimum storage threshold is critical, the
remaining input variables showed non-conclusive importance.
While the security of supply thresholds are most important variables, their high σ rankings show
that they also have the largest spread of EE. That is, the Δ perturbations of the minimum storage
threshold and the supply reliability threshold cause the widest variation in the yield estimate. This
can be due to them having non-linear behaviour or due to interaction with other variables.
Comparison of the SA results of the 20 year scenarios 2, 2b and 2c and also the three 77 year
scenarios highlight the effect of climate variability on the importance of input variables. The three
20 year scenarios consist of different sub-sequences from the 77 year historic record with
approximately equal total streamflow volume, while the three 77 year scenarios that are simply
shuffled sequence and therefore the same total streamflow volume. Table 2 shows that the supply
reliability threshold is the critical security criteria for these three 20 year scenarios while the
minimum storage volume threshold is critical for the 77 year scenarios. This result is due to the
effect of the simulation length as discussed above. Due to the qualitative, ranking nature of the
Morris method no conclusions can be made on these two sets of scenarios.
SUMMARY AND CONCLUSIONS
This paper presents a sensitivity analysis used to identify the importance of input variables used in
the estimation of yield of the Barwon urban water supply system in Australia. Input variables were
considered to be subject to natural variability (climate dependent variables) or knowledge
deficiency (system policy variables). Twenty climate scenarios of four simulation lengths were
selected from the available historic data. The scenarios were selected based on total streamflow
entering the Barwon system or generated using a shuffling (or recycling) approach. The Morris
method of sensitivity analysis was used on the 14 system policy variables over the 20 hydroclimatic
scenarios to uncover system and yield estimate behaviour.
The results showed the two security criteria thresholds, the reliability of supply threshold and the
minimum storage threshold, to be the most important variables, with the upper RRC variables also
show notable importance. When the supply reliability threshold was the most important variable,
the upper RRC curvature and position variables were also important. However, when the minimum
storage threshold was the most important, the remaining input variables showed non-conclusive
importance. It was also found that for a 20 year simulation length the supply reliability threshold
was the most important variable, while the minimum storage threshold was the most important
under the 40, 60 and 77 year scenarios. As the simulation length increases, a drought is more likely
to be captured in the climate scenario making the minimum storage threshold critical. This causes
the minimum storage threshold variable to be highly important, which in turn makes the RRC
variables and the target storage curve variables not important.
REFERENCES
Braddock, R.D. & Schreider, S.Y. 2006, “Application of the Morris Algorithm for Sensitivity
Analysis of the REALM Model for the Goulburn Irrigation System”, Environmental Modeling and
Assessment, Vol. 11, No. 4, pp. 297-313.
Byrnes, J., Crase, L., Dollery, B., 2006. Regulation versus Pricing in Urban Water Policy: The Case
of the Australian National Water Initiative. Australian Journal of Agricultural and Resource
Economics. 50(3), 437-449.
Campolongo, F. and Braddock, R.D. 1999, “The use of Graph Theory in the Sensitivity Analysis of
the Model Output: A Second Order Screening Method”, Reliability Engineering and System Safety,
64(1), pp. 1-12.
Campolongo, F., Cariboni, J., Saltelli, A. 2007. An Effective Screening Design for Sensitivity
Analysis of Large Models. Environmental Modelling and Software. 22(10), 1509-1518.
Campolongo, F., Kleijnen, J., Andres, T., 2000. Screening Methods, in: Saltelli, A., Chan, K., Scott,
E.M. (Eds.), Sensitivity Analysis. Wiley, Chichester, England, pp. 65-80.
Department of Sustainability and Environment (DSE), 2011. Guidelines for the development of a
water supply demand strategy. Melbourne, Australia, 95p.
Erlanger, P. D., Neal B., 2005. Framework for urban water resource planning. Water Services
Association of Australia Occasional Paper 14, 32 p.
King, D.M., 2009. On the Importance of Input Variables and Climate Variability to the Yield of
Urban Water Supply Systems, Thesis (PhD). Victoria University, Melbourne, Australia, 236 p.
King, D.M., Perera B.J.C., 2010. Sensitivity Analysis of Yield Estimate of Urban Water Supply
Systems. Australian Journal of Water Resources. 14(2), 141-155.
Morris, M.D., 1991. Factorial Sampling Plans for Preliminary Computational Experiments.
Technometrics. 33(2), 161-174.
Perera, B.J.C. and James B. 2003, “A Generalised Water Supply Simulation Computer Software
Package”, Hydrology Journal, Institute of Engineers (India), Vol. 26(1-2), pp. 67-83.
Perera, B.J.C., James, B. and Kularathna, M.D.U. 2005, “Computer software tool REALM for
Sustainable Water Allocation and Management”, Journal of Environmental Management, 77(4), pp
291-300.
Ratto, M., Young, .P.C., Romanowicz, R., Pappenberger, F., Saltelli, A., Pagano, A., 2007.
Uncertainty, Sensitivity Analysis and the Role of data Based Mechanistic Modeling in Hydrology.
Hyrdology and Earth System Sciences. 11(4), 1249-1266.
Saltelli, A, Chan, K., Scott, E.M. (Eds.), 2000. Sensitivity Analysis. Wiley, Chichester, UK.
Saltelli, A., Tarantola, S., Campolongo, F., Ratto, M., 2004. Sensitivity Analysis in Practice. Wiley,
New York, USA.
Sinclair Knights Merz (SKM), 2006, “Central Region Sustainable Water Strategy, Configuration of
a Basin Wide REALM Model for the Barwon/ Moorabool Basin”. Prepared for Department of
Sustainability and Environment by Sinclair Knights Merz, Malvern, Australia, 335 p.
Victoria University and Department of Sustainability and Environment (VU and DSE), 2008.
REALM: Getting Started Manual. <http://www.water.vic.gov.au/monitoring/surface-watermodelling/realm> (last accessed 26.04.2012).