Evaluation statistics of
cloud fraction and
water content
Robin Hogan
Ewan O’Connor
Damian Wilson
Malcolm Brooks
Overview
• Cloudnet level 3 data
• A solution to the problem of evaluating high cloud?
• Summary of errors in model cloud fraction and
water content climatologies over Europe
–
–
–
–
–
–
ECMWF model
KNMI Regional atmospheric climate model (RACMO)
Met Office mesoscale and global
SMHI Rossby Centre atmospheric model (RCA)
Meteo France ARPEGE model
DWD Lokal Modell
• Forecast skill
Cloudnet level 3 data
• Level 3 files summarise the comparison of a
observations and model over a certain period:
–
–
–
–
Long-term mean of a quantity versus height
Separation into “freq. of occurrence” and “amount when present”
PDFs in height ranges 0-3 km, 3-7 km, 7-12 km and 12-18 km
Skill scores versus height for different thresholds
• Separate level-3 files/quicklooks are produced for
–
–
–
–
–
–
Each variable: cloud fraction, LWC, IWC, high cloud fraction
Each site: 4 European, 4 ARM (so far)
Each model: 7 so far, plus persistence/climatology forecasts
Each month and each year
Different forecast lead times (Met Office meso and DWD only)
In principle: different model resolutions / parameterisations
• Over 5000 files so far!
Cloud fraction
Observations
Met Office
Mesoscale
Model
ECMWF
Global Model
Meteo-France
ARPEGE Model
KNMI
RACMO Model
Swedish RCA
model
What can we do about high cloud?
• All models see more cirrus than observed
– We use the known radar sensitivity to remove clouds from model
that we would not expect to detect (affecting heights > 7 km)
– Does not usually remove enough cloud to bring into agreement
• Are all models wrong?
– Or does radar miss more IWC than it thinks due to small
particles?
ARM Nauru 8 Nov 2003
Radar
35 GHz
MMCR
Lidar
Night-time
Merged
ceilometer
and
micropulse
lidar
October 2003: Normal processing
No periods
when rain rate
> 8 mm/h
Large
difference
between
observations
and ECMWF
model,
whether model
is modified for
radar
sensitivity or
not
…only periods of high lidar
sensitivity
Consider only
night-time and
periods when
lidar is
unobscured by
liquid cloud,
rain or melting
ice
Liquid clouds
removed from
comparison
Cloud fraction
OK but peak 2
km too high
One month later
Model grossly
overestimates
high cloud
fraction
To evaluate high
clouds in models:
need a high
sensitivity lidar
and appropriate
sampling of data
(both model and
observations)
ECMWF
cloud
fraction
• Cabauw 2002: • Chilbolton 2004
– Amount when
(and all midpresent is good
latitude sites
– Mean cloud
2003-2005):
fraction and
frequency of
occurrence too
high in the
boundary layer
– Need to treat
snow as cloud in
the model
– Boundary layer
cloud fraction much
more accurate
– Still need to treat
snow as cloud
Chilbolton 2004: LWC
ECMWF water content
• Mean LWC and IWC accurate
to observational uncertainties
• Freq. of occurrence too high;
amount when present too low
• Inconsistent with cloud frac.?
• PDF shows occurrence of low
values is too high
Chilbolton 2004: IWC
RACMO
• Cloud fraction errors similar to
ECMWF before 2003
• Water content errors (mean,
frequency of occurrency) much
as ECMWF
• Lower IWC in high cirrus
Met Office mesoscale
cloud fraction
Cabauw 2004
• Mean amount when present too
low through most of atmosphere
• Largely due to inability of model
to simulate 100% cloud fraction,
as shown by the PDFs
• Error in high cloud needs to be
checked using high sensitivity
lidar
Met Office global
cloud fraction
Cabauw 2004
• Observations show greater
frequency of cloud with increased
gridbox size; opposite in model
• PDF error unchanged
Met Office mesoscale water content
Chilbolton 2004: LWC
Chilbolton 2004: IWC
• Liquid occurrence • Mean IWC
very good
very good
• Boundary layer • Frequency of
perhaps too low
ice cloud
occurrence
• Mean LWC
too high
underestimated
above 3 km
above 3 km
• PDFs much
• Similar to
better than
previous result
ECMWF!
found for
occurrence of
supercooled
layers
Met Office global water content
Chilbolton 2004: LWC
Chilbolton 2004: IWC
• Mean LWC
similar but
frequency
of
occurrence
much lower
• IWC
generally
higher
SMHI Rossby Centre model
Palaiseau 2004
• Amount when present
reasonable but frequency of
occurrence and overall mean
much too high
• Similar picture for LWC/IWC:
mean overestimated due to
cloud too often
Meteo France
cloud
fraction
• After Apr ‘03
• Before Apr ‘03
Cabauw 2002
Cabauw 2004
– Amount
when
present far too
low
– High values
rarely
predicted
– Amount when
present very good
(better than Met
Office & ECMWF)
– Mean cloud
fraction much
better
– Amazingly, worse
agreement with
synoptic obs of
cloud cover!
Chilbolton 2004: LWC
Meteo Fr. water content
• Boundary-layer LWC too low
• Frequency of supercooled liquid
much too high
– Need to change the T-dependent
ice/liquid ratio
• PDF of LWC and IWC too narrow
• Mean IWC too low in mid-levels
Chilbolton 2004: IWC
DWD cloud
fraction
• Cloud fraction
generally very
good
Chilbolton 2004
– But frequency
of occurrence
always
overestimated
by 20-30%
• PDFs
particularly
well simulated
DWD water
content
Chilbolton 2004
• Frequency of
liquid cloud
occurrence
too high
• LWC in
supercooled
clouds too
high
• Frequency of
ice cloud
occurrence OK
• Mean IWC and
mean amount
when present
(in-cloud IWC)
are both
underestimated
below 7 km
Equitable threat score
• Measure of skill of forecasting
cloud fraction>0.05
• Persistence and climatology
shown for comparison
• Lower skill in summer
convective events
Skill versus lead time
• Unsurprisingly UK model most accurate in UK,
German model most accurate in Germany!
• Typically 500-mb
geopotential
height used in
operational
forecast
verification
• Cloud fraction a
more challenging
test: more rapid
loss of skill with
time