The Cloud Hunter’s Problem
An Automated Decision Algorithm to Improve the Productivity
of Scientific Data Collection in Stochastic Environments
Arthur A. Small, III
Venti Risk Management
Presentation at AMS Enterprise
11 August 2010
c
2010
Arthur A. Small, III
Acknowledgments
Based on joint work with Jason B. Stefik and Johannes Verlinde
(Penn State) and Nathaniel C. Johnson (U Hawaii).
Small gratefully acknowledges support from the U.S. National Science
Foundation under the Human and Social Dynamics and Decision Making Under
Uncertainty Programs, grant award number NSF SES-0729413 and cooperative
agreement NSF SES-0345840.
The authors are grateful for the cooperation extended by investigators of the
RACORO campaign of the Atmospheric Radiation Measurement Program
sponsored by the U.S. Department of Energy.
Outline
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
The Client’s Problem
I
Goal: Observations of boundary layer clouds
I
Forecasts of conditions are imperfect
I
Flight time is limited, costly
I
Daily decision: Fly tomorrow or not?
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
Current Heuristic Decision Process
Time-consuming. Stressful. Expensive.
Photo: U.S. Department of Energy’s Atmospheric Radiation Measurement Program
Our Approach: An Automated Decision Algorithm
Probabilistic forecasts
+
Optimization
→
Automated decision recommendations
Creating a probabilistic forecasting system
Challenge: translate off-the-shelf forecasting products into
probability distributions over events of interest to this
decision-maker.
Strategy: Use GFS Model, plus custom statistical post-processing.
Problems to overcome:
I
The GFS model is an imperfect predictor of RHP.
I
RHP patterns are high-dimensional.
I
RHP patterns are imperfect indicators of the presence of
boundary-layer clouds.
Must address both sources of uncertainty.
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Algorithm in Brief
Step 1: Use GFS numerical weather prediction model to forecast next
day’s relative humidity profile (RHP) at site.
Step 2: Match forecast RHP to “closest” of 24 RHP clusters.
Step 3: Based on weather model’s track record, map forecast RHP
cluster to a conditional probability distribution over set of
realized RHP clusters.
Step 4: Compute probability of success (weighted average).
Step 5: Use dynamic programming to estimate opportunity cost of
using up a flight.
Step 6: Recommend: Fly if and only if expected benefits exceed
expected costs.
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Step 1: Use GFS model to forecast RHP
Figure: A relative humidity profile.
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
Step 2: Map GFS forecast to canonical RHP
Figure: The SOM grid for relative humidity profiles at the Lamont site.
Step 3: Convert forecast signal into
conditional probability distribution over set
of canonical RHPs realizations
Figure: The performance of the GFS model: empirical distribution of RHP clusters,
conditioned on a forecast that RHP will fall into Cluster 1.
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Step 4: Compute probability of success.
Compute probability of success as weighted average:
P(clouds|forecast) =
X
P(cluster |forecast) · P(clouds|cluster )
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
Step 5: Dynamic programming procedure
Figure: Dynamic programming procedure.
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Step 6: Decision: Apply cost-benefit test
Combining expressions, we have an optimizing decision rule as a
function of realized SOM forecast signal s:
Fly if and only if
pd|s̃ ≥ V (d − 1, f ) − V (d − 1, f − 1).
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
(1)
Expected results, pre-season
Figure: RACORO application: Expected number of successful flights, as function of
days and flights remaining, conditioned on use of Cloud Hunter algorithm.
Results
Figure: The SOM grid for relative humidity profiles at the Lamont site.
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Benefits of automated decision procedure
I
Increased efficiency: Same resources in, more data out.
I
Less waste: Of 68 flights budgeted, algorithm used 67,
heuristic procedure used 59.
I
Less stress: Automated decision system saves valuable human
capital, imposes less strain on scientists and flight crews,
improves ability to plan.
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Manifesto: “Not better forecasts — better decisions.”
Probabilistic forecasts offer tremendous latent value in intelligent
application to problems in decision making and risk management.
Currently, most of this value is thrown away: human beings
generally can’t process probabilistic information.
Strategy: Feed probabilistic forecasts into application-specific
decision modules.
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem
History: Airline Yield Management
”We were a vibrant, profitable company from 1981 to 1985, and
then we tipped right over into losing $50 million a month.
“We were still the same company. What changed was American’s
ability to do widespread yield management in every one of our
markets.
“We had been profitable from the day we started until American
came at us with Ultimate Super Savers.
“That was the end of our run because they were able to
under-price us at will and surreptitiously.”
– Donald Burr, CEO, PeopleExpress
Introduction
The Algorithm
Application to RACORO 2009
Conclusions
Epilogue: Implications for Work of the Enterprise
Thanks!
Arthur A. Small, III
[email protected]
Arthur Small, Venti Risk Management
The Cloud Hunter’s Problem