COSMO-LEPS Verification
Chiara Marsigli
ARPA-SMR
Available italian stations
Verification on station points
Bi-linear interpolation (4 nearest points)
X
X
X
X
X
X
X
X
X
Verification on super-boxes
Average value – maximum value – frequency
PRED.
OBS.
X
X
X
X
X
X
X
X
X
OVERLAPPING BOXES
X
X
X
X
COSMO-LEPS
vs
observations
station points
weighted
ROC area
Nov 02 – Dec 02 – Jan 03
Need for LM
verification
at these fc ranges
COSMO-LEPS
vs
observations
station points
not weighted
ROC area
w
+48 h
nw
w
+120 h
nw
COSMO-LEPS
vs
observations
station points
weighted
Brier Skill Score
COSMO-LEPS
vs
observations
station points
not weighted
Brier Skill Score
Brier Skill Score
Weighting procedure
It is possible to decide (in real-time)
if it is better to weight or not to weight?
Dependence from ensemble spread?
Flow dependence?
Brier Skill Score
station points
average values on super-boxes
Brier Skill Score
station points
average values on super-boxes
Brier Skill Score
Nov 02 only
Brier Skill Score
contingency
table
Forecast
Observed
Yes
No
Yes
a
b
No
c
d
ROC area
a
Hit Rate (H) =
ac
b
False Alarm Rate (F) = b  d
A contingency table can be built
each probability
class
probability
class can
becan
defined
as the For
1.for
A contingency
table
for(aeach
probability
class
be built.
% of ensemble elements which actually forecast a given event). For the k-th probability class:
the k-th probability class:
M
Hk =
H
Fk =
F
i
i k
M
i
i k
1. Hit rates are plotted against the corresponding false alarm rates
to generate the ROC Curve.
2. The area under the ROC Curve (ROC area) is a probabilistic
index
The area under the ROC curve
is used as a statistic measure of forecast usefulness
Brier Skill Score
Brier Score
1 n
2
BS    f k  ok 
n k 1
•o i
= 1 if the event occurs
= 0 if the event does not occur
•fi is the probability of occurrence
according to the forecast system (e.g.
the fraction of ensemble members
forecasting the event)
•BS can take on values in the range
[0,1], a perfect forecast having BS = 0
Brier Skill Score
BS  BS
BSS  cli
BS cli
BS cli  o 1  o 
The forecast system has predictive skill if
BSS is positive, a perfect system having
BSS = 1.
o = total frequency of the event
(sample climatology)