PERFORMANCE ASSESSMENT OF A
BAYESIAN FORECASTING SYSTEM (BFS) FOR
REALTIME FLOOD FORECASTING
Biondi D. , De Luca D.L.
Laboratory of Cartography and Hydrogeological Modeling,
University of Calabria (Italy)
GOALS
Properties of BFS under different hypothesis
Adeguate verification tools for probabilistic
forecasts
→ robust analysis of forecasting
capability
Interaction among different sources of error
GOALS
Properties of BFS under different hypothesis
Adeguate verification tools for probabilistic
forecasts
→ robust analysis of forecasting
capability
Interaction among different sources of error
Few Studies
Stochastic model for rainfall prediction
Effect of more intense rainfall events
Journal of Hydrology , Volume 479, 4 February 2013, Pages 51-63
BFS ARCHITECTURE
Deterministic
Hydrological Model
Probabilistic Quantitative
Precipitation Forecast
(PQPF)
Precipitation
Uncertainty
Processor
(PUP)
Hydrologic
Uncertainty
Processor
(HUP)
 s n 
 n  yn | sn , y0 , 
Integrator
INT
Some refs:
Krzysztofowicz,1999
Krzysztofowicz & Kelly, 2000
Kelly & Krzysztofowicz, 2000
Krzysztofowicz & Herr, 2001
Krzysztofowicz, 2001
Krzysztofowicz, 2002
Krzysztofowicz & Maranzano, 2004
Maranzano & Krzysztofowicz, 2004
Krzysztofowicz & Maranzano, 2004
Other
input
n  yn | y0 
Prior
y0
PRECIPITATION UNCERTAINTY PROCESSOR (PUP)
PQPF –PRAISE stochastic model (Prediction of Rainfall Amount Inside a Storm
Event, Sirangelo et al., 2007)
W total precipitation amount on T
t1
s3
t2
 n sn   PSn  sn 
t3
 n sn 
T=1 hour
 i 1, 2, ..., M
wi, j 
 j 1, 2, ..., T t 
Perfect
Deterministic
Hydrological Model
(RISE)
s n sn 0
PQPF
 1  a   a 'n sn  sn  sn 0
Other
input
a
t0
s2
a  PW  0
PUP
PRAISE
 s n 
Integrator
INT
1-a
s1
M simulations for W
Sn evaluation of Yn at tn
sn0
Sn
HYDROLOGIC UNCERTAINTY PROCESSOR (HUP)
Deterministic
Hydrological Model
Off-line estimation
other
input
sn
Observed rainfall data (perfect input)
HUP
Prior
y0
 n  yn | sn , y0 , 
T
y0
t2
t1
s1
y1 ?
t1
s2
t2
Integrator
INT
t3
y2 ?
s3
Posterior
Likelihood
Prior
f n sn yn , y0 g n  yn y0 
n  yn sn , y0  
k n sn y0 
y3 ?
t3
Precipitation dependent processor (Krzysztofowicz & Herr, 2001)
t0
n=1
n=2
n=3
W=0  V=0
n0 yn sn , y0 ,V  0
W>0  V=1
n1 yn sn , y0 ,V  1
INTEGRATORE (INT)
 i 1, 2, ..., M
wi , j 
 j 1, 2, ..., T t 
a  PW  0
PQPF
(PRAISE)
Deterministic
Hydrological Model
(RISE)
s n sn 0
sn
PUP
  PV  1
Altri
input
Prior
HUP
 s n 
y0
 n  yn | sn , y0 , 
Integrator
INT
n  yn | y0 
 00  y0   1  a 
n  yn | y0  
  n 0  y n | sn 0 , y0  
 0  y0 
 01  y0   a  

   n1  yn | sn , y0    n' sn  dsn
 0  y0  s
n0
CASE STUDY AND DATA DESCRIPTION
Area
29.2 km2
DTM resolution
30 m
Max Altitude
1015 m
Min Altitude
75 m
Mean Altitude
291 m
1. Database
November – March
heights forecast with1991
2.Rainfall
Precipitation
t =- 2005
20 min
River discharges
2000 – 2005
3. Precipitation forecasting period T = 1 hour
t = 20 min
4. V = 0 ↔ 0 < W < 2 mm and V = 1 ↔ W ≥ 2 mm
5. probabilistic forecasts of river discharge in 1-h steps
Selected flood events
ID
Date
1
2
3
4
5
6
7
8
9
28/12/2000
24/12/2000
01/01/2003
07/01/2003
28/01/2004
28/02/2004
14/02/2005
26/02/2005
06/03/2005
Qmax (m3/s)
17.75
14.32
19.57
16.00
28.78
18.35
21.25
20.40
18.55
HYPOTHESES ABOUT BFS
a) Perfect hydrological model
 n (.) is provided by PUP, and estimated on the
basisi on observed discharge y0, and probability a e , evaluated on-line. (UD)
(4)
  y  1  a   01 y0  a '
n  yn | y0   00 0

  n  yn 
 0  y0 
 0  y0 
prior distribution Gn(·|y0) (PD),
b) Non-informative PQPF for discharge forecasting
with climati prior probability ’

y n  sn 0

 00  y0  1   '
 01  y0  '
Gn  yn | y0  
 Gn 0  yn | y0  
 Gn1  yn | y0 
 0'  y0 
 0'  y0 
c) Perfect PQPF: The BFS provides a predictive distribution equal to the posterior
distribution (HD) , provided by HUP.

1

1

1


Q
(

(
y
))

A
Q
(

(
s
))

D
Q
(

(
y
))

B
nv
nv
n
nv
n
nv
0
v
0
nv



(
y
|
s
,
y
)

Q
nv
n
n
0

T
nv


d) Total predictive distributon (TD), which takes into account both sources of
uncertainty;
s


 00  y0   1    
1 a
a
n  yn  
  n 0  yn | sn 0 , y0  
   n 0  y n | sn , y 0  
 n' sn dsn  
1    s
1   
 0  y0 




 01  y0   
a '





   y | s , y   s ds 
 0  y0   s n1 n n 0  n n n 
n1
n0
n1
EVALUATION TOOLS FOR FORECAST VERIFICATION
(MURPHY, 1993)
• Calibration: probabilistic correctness of the forecasts or the statistical
consistency between the observed time series and its predictive distribution
(Reliability)
• Sharpness: the spread of the predictive distribution of the forecasts (attribute
only for forecasts)
• Accuracy: Agreement between an observed value and a representative index,
assumed as the best deterministic prediction (expected value, median, etc.)
• CRPS: Continuous Ranked Probability Score (Brown, 1974;Hersbach, 2000)

CRPS 
   y
n
0
n | y0   H  yn  yobs, n  dy n
2
y
H
PROBABILISTIC CALIBRATION
Empirical cumulative distribution
Negative Bias
Positive Bias
Under dispersion
Over dispersion
zi=yobs,i)
zi=yobs,i) should be i.i.d. U[0,1]
Nz

z  1 2
i 1
zit
R
 i / Nz
Nz
z  1
Nz
 1(0,1) zit / N z
i 1
1 if zit  0 or zit  1
1( 0,1) zit   
0 otherwise
Laio & Tamea, 2007; Renard et al., 2010
RESULTS – CALIBRATION
a)
b)
1
Ri/Nz
1
Ri/Nz
UD
UD
PD
0.8
PD
0.8
TD
TD
HD
HD
0.6
0.6
0.4
0.4
5 % Kolmogorov bands
0.2
0
0.2
0
0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
1
zi t
zi t
n = 1 hour
n = 2 hours
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
n
 st
Z
Z
UD
PD
HD
TD
UD
PD
HD
TD
UD
PD
HD
TD
1 hour
8.69
1.60
1.33
1.22
0.50
0.72
0.74
0.84
0.46
1.00
1.00
1.00
2 hours
4.31
-1.24
-0.21
0.25
0.61
0.62
0.83
0.84
0.47
1.00
0.97
1.00
- UD: significant positive bias (many observed values with a predictive probability
very small or null) a very narrow distribution
- PD: a marked underestimation associated to forecasted values
- HD: an overestimation for forecasts (due to the adopted hydrological model)
- TD: a slight overestimation associated to predicted values
RESULTS - SHARPNESS
Box-plot for InterQuartile Range (IQR)
10
10
Forecast interval width (m3/s)
b) 12
Forecast interval width (m3/s)
a) 12
8
6
75 %
4
25 %
2
0
PD
HD
n = 1 hour
2.
3.
4.
5.
6
4
2
0
UD
1.
8
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
TD
UD
PD
HD
TD
n = 2 hours
TD provides the worst results, with respect to the other BFS hypotheses, and median value of
IQR is the highest.
HD is characterized by an higher sharpness (only Hydrological Uncertainty is considered)
compared with TD
UD provides the narrowest bands.
For n = 1 hour, PD has a median IQR similar to UD but the bands are wider.
For n = 2 hours UD has a median IQR less than n=1 hour, unlike other BFS hypotheses. This is due
to the presence of sn0, which implies truncated probability distributions with generally small
values of CV.
RESULTS - SHARPNESS
.
Event #2 uncertainty bands for n=1 hour
UD
50%
25%
20
22:40
22.20
22:00
21.40
21:20
21.00
20:40
18.20
22:40
22.20
22:00
21.40
21:20
0
21.00
0
20:40
5
20.20
5
20:00
10
19.40
10
20.20
15
20:00
15
5%
19.40
m3/s
5%
19:20
observed
discharge
75%
25%
18:40
95%
30
25
50%
18.20
m3/s
20
b)
19:20
75%
25
observed
discharge
19.00
95%
18:40
30
19.00
a)
PD
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
HD
20
22:40
22.20
22:00
21.40
21:20
21.00
20:40
20.20
20:00
22:40
22.20
22:00
21.40
21:20
21.00
0
20:40
0
20.20
5
20:00
5
19.40
10
19:20
5%
15
10
18:40
25%
19.40
15
50%
19:20
5%
observed
discharge
19.00
25%
18.20
m3/s
20
95%
75%
25
50%
m3/s
25
d)
30
18:40
75%
TD
observed
discharge
18.20
95%
19.00
c)
30
(Biondi & De Luca, 2011)
RESULTS - ACCURACY
a)
b)
35
35
Inside Interquartile
UD
30
30
25
Inside Interquartile
PD
Outside Interquartile
Outside Interquartile
25
50% quantile (m3/s)
50% quantile (m3/s)
n = 1 hour
20
15
10
20
15
10
18%
5
40%
5
RMSE=7.7
0
0
0
c)
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
d)
Observed discharge (m3/s)
Observed discharge (m3/s)
35
Inside Interquartile
Figura
1.
Diagrammi
QQ-plots
per
a)
n=1
ora
e
b)
n=2
ore
Inside Interquartile
35
HD
30
TD
Outside Interquartile
30
25
50% quantile (m3/s)
50% quantile (m3/s)
RMSE=6.2
20
15
10
44%
5
Outside Interquartile
25
20
15
10
51%
RMSE=5.4
5
RMSE=5.2
0
0
0
5
10
15
20
25
Observed discharge (m3/s)
30
35
0
5
10
15
20
Observed discharge
25
(m3/s)
30
35
RESULTS - ACCURACY
a)
b)
35
35
Inside Interquartile
UD
30
30
25
Inside Interquartile
PD
Outside Interquartile
Outside Interquartile
25
50% quantile (m3/s)
50% quantile (m3/s)
n = 2 hours
20
15
10
20
15
10
11%
5
36%
5
RMSE=7.4
0
0
0
c)
5
10
15
20
25
30
35
0
5
10
15
20
25
30
35
d)
Observed discharge (m3/s)
Observed discharge (m3/s)
35
Inside Interquartile
Figura
1.
Diagrammi
QQ-plots
per
a)
n=1
ora
e
b)
n=2
ore
Inside Interquartile
35
HD
30
TD
Outside Interquartile
30
25
50% quantile (m3/s)
50% quantile (m3/s)
RMSE=7.1
20
15
10
36%
5
Outside Interquartile
25
20
15
10
38%
RMSE=7.9
5
RMSE=6.1
0
0
0
5
10
15
20
25
Observed discharge (m3/s)
30
35
0
5
10
15
20
Observed discharge
25
(m3/s)
30
35
RESULTS - CRPS
CRPS
n
UD
PD
HD
TD
1 ora
5.27
3.43
2.63
2.82
2 ore
5.00
4.48
3.54
4.29
F(y)
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore
n=2
n=1
Sn0
Sn0
yobs
y
CONCLUSIONI
CONCLUSIONS
Calibration
TD: best results
PD – HD: negative and positive bias, respectively
UD: marked overprediction (due to sn0)
Sharpness
UD: the best results (due to sn0)
TD: the worst results
Accuracy and CRPS
TD and HD: the best results (TD is the best for a real time prediction)
Crucial role of HUP
Similar results beyond T
Importance of performing a comprehensive analysis by using different
validation tools (in order to avoid misleading conclusions)
Figura 1. Diagrammi QQ-plots per a) n=1 ora e b) n=2 ore