Metrics for Evaluating Measurement Systems:
“Optimal” Design of Observing Systems
based on quantitative assessment
Yuanfu Xie, Zoltan Toth, Bob Atlas
and Lars-Peter Riishojgaard
Acknowledgment:
Dr. Alexander E. MacDonald
Betsy Weatherhead
Observing systems:
Important and Costly
•
A long list of observing systems includes,
satellite, radar, RAOB, ACARs and so on for
applications of weather, ocean, environment,
energy, and so on;
•
They are usually expensive in development,
deployment and maintenance; they have
different limitations or impact in applications
•
A good design could lead great saving while
maximizing their socio-economic impact;
•
Cost examples:
NPOESS RAOB
– Development:
– Deployment:
– Maintenance:
0.9 Billion ???
???
150 x 2 x 200
???
50 x 2 x 200
total in 5 years:
1 Billion
0.15 Billion
(Jim Yoe at NWS/NCEP)
Optimal Design of Observing Systems:
an overhaul design for ocean, weather and environment
collaboration across the enterprise
• Situational awareness/nowcasting;
• Numerical Prediction ;
• Climate trend;
A do-as-we-go
approach is
history!
It is what
we are doing!
Building Observing Systems: present
• Subjective:
– Expert option & suggestion;
– Lack of quantitative assessment of data need & impact;
– Less consideration of overall socio-economic impact;
• Not systematic:
– Overlook existing systems;
– Redundant and ineffective;
• Disintegrated:
– Disconnected from data assimilation, forecast and
applications;
– Insufficient focus on deployment and maintenance.
Optimal Design: Metrics (NWP)
A multi-objective optimization problem
considering socio-economic impact, life and
property loss, ……
Minimize cost (development, deployment, maintenance) × weightc
−impact (socio, economic, environment)
× weighti
+damage (populations, locations, industries)
× weightd
+probability(false alarm) + … …
Subject to death
≤ 0
spending ≤ budget
technology ≥ maturity
The weight estimations involve a multi-disciplinary team work in economy,
meteorology, sociology, management, emergency and so on
Observing System Simulation Experiment
for optimal design
Multi-objective
optimization
Quantitative Assessment of
Observation needs and impacts
OSSE
Decision
Making
What obs
systems
Observations
Better
use obs
Data
assimilation
costs
Better
forecasts
Numerical
forecasts
Applications:
Public
Private
Academia
benefit
OSSE:
A synthetic world with known “truth”
Operation system
Tools available for quantitative
assessment
Obs count in millions
• OSSE:
– Good for evaluating future observing
systems before they are deployed;
– Unique for supporting an overhaul optimal
observing system design.
– It has been used in some observation
system design and correctly predicted the
impact before system deployment (Atlas
1985 and 1997).
air
cra
ft
su
rfa
ce
qk
sw
nd
ds
ss
mi
sp
bs
tw
in
sa
rao
a
ms
u
su
s
b
es
s_
am
go
eo
-10
hir
-5
air
s
a
0
su
– Good for existing systems;
– No economic consideration;
5
su
• Adjoint sensitivity:
10
am
– Good for evaluating existing systems;
15
am
• OSE:
-15
-20
Sensitivity
Adjoint sensitivity
From NASA/GMAO
Review of OSSE Status
•
Global OSSE with fvGCM 0.5 deg NR for Lidar on hurricane tracks;
•
Global quick OSSE with fvGCM 0.25 deg NR for wind profile on hurricane Ivan
track;
•
Regional quick OSSE with MM5 NR for HIRAD on hurricane surface wind analysis;
•
Regional quick OSSE with WRF-ARW NR for AIRS, Lidar on hurricane intensity;
•
NCEP global OSSE on wind lidar;
•
NOAA joint global OSSE with ECMWF NR on:
– UAS, WISDOM data impact on hurricane tracks;
– Lidar wind data impact;
•
•
Regional OSSE on hurricane intensity;
……
Previous OSSE
• Evaluated observation data impact:
– Different meteorological parameters’ impact:
wind, temperature and moisture data;
– Wind data at different altitudes;
– Wind lidar etc;
– UAS and WISDOM data impact;
• Improving assimilation methods:
– Scatterometer data;
– Satellite surface wind speed;
OSSE example
• Impact experiment for a 4-telescope hybrid Wind Lidar carried out by JCSDA for NASA,
using the NCEP operational Global Forecast System
• Control experiment (all routine observations assimilated) in black
• Perturbation experiment (Control + Wind Lidar data) in
• Impact of Wind Lidar data on forecast skill; statistically significant in both hemispheres
NH
SH
Current OSSE
Capability & Improvements
OSSE must be done correctly to ensure its realism and accuracy:
• Current global OSSE system uses an ECMWF T511 13 month forecast as its
nature run; Higher resolution nature run is needed (under development).
• Regional and local applications require improvement of NR; regional OSSE
imbedded in the global OSSE is developed for hurricane intensity study.
• Calibration is a complex and time-consuming process; A nature run close
to reality or analysis would save time in not only calibration but also
synthetic observations, observation error estimation and ensemble
forecasts.
• Targeting observation scheme has to be considered for optimal
observation system design;
An example of WISDOM OSSE
One improvement
may find better
launch sites through
targeted observation
schemes.
Conclusion Remark
• Now it is time for a quantitative assessment
for optimal observing systems!