Are forecas*ng methods too complex?
Kesten C. Green*, University of South Australia
J. Sco; Armstrong*, University of Pennsylvania *Ehrenberg-­‐Bass Ins:tute at University of South Australia
Interna*onal Symposium on Forecas*ng
Riverside, California
24 June 2015
Slides available at ForPrin.com
G&A ISF 2015 – Complex-­‐V13
Thursday, 23 July 15
1
On the value of complex forecas*ng methods
Analysts have long assumed that complex methods are needed to deal with complex problems. As long ago as 1985 (Armstrong, pp.225-­‐232), a review of the literature found that complexity tends to:
•reduce accuracy and understanding, and
•increase costs and mistakes
Despite the evidence, forecasters con*nue to use ever more complex methods. We reviewed this issue by organizing a JBR Special Issue on Simple versus complex forecas:ng. We report on our paper in that issue reviewing the evidence. 2
Thursday, 23 July 15
2
See HANDOUT
3
Thursday, 23 July 15
3
Review of experimental research
Reviewed published research from all areas of forecas*ng, including the Special Issue papers… 1.defined simplicity in forecas*ng
2.iden*fied studies with evidence on compara*ve accuracy
3.assessed direc*onal and effect size evidence
4
Thursday, 23 July 15
4
Simple forecas*ng defined
Simplicity in forecas*ng requires that are all of the following are understood by clients. 1. forecas*ng method,
2. representa*on of cumula*ve knowledge, 3. rela*onships in models,
4. rela*onships among models, forecasts, and decisions
5
Thursday, 23 July 15
5
Findings
Found 32 papers with 97 comparisons: a) None of the papers found that complexity helped accuracy
b) Complexity increased error by 27% on average across papers
“Simple versus complex forecas*ng: The evidence” was published in JBR in 2015. Due out in print in August.
6
Thursday, 23 July 15
6
Summary of evidence on accuracy of forecasts from complex vs. simple methods
7
Thursday, 23 July 15
7
Simplicity: A checklist
Score the following on a 0-­‐to-­‐10 scale
Use of prior knowledge in forecas:ng models
1.Do you know what prior knowledge about the situa*on was used?
2.Do you know how prior knowledge about the situa*on was used?
3.How simply is prior knowledge represented?
Nature of the rela:onships among the model elements (Non-­‐linear… Mul*plica*ve… Addi*ve… Single?)
Nature of the rela:onships among models, forecasts, and decisions (Weak… Strong?)
Explaining the forecas:ng process
I am confident that I could explain… to the decision maker
1.the forecas*ng methods
2.how prior knowledge about the situa*on is represented in the forecas*ng models
3.the nature of the rela*onships among the model elements
4.how the models, forecasts, and decisions are related to each other
8
Thursday, 23 July 15
8
Demonstra*on: Effect of simplicity vs. complexity in climate forecas*ng
IPCC warming alarmists do not forecast, they create “scenarios” via computer simula*ons
1.Scenarios are:
a. Stories about “what happened in the future”
b. Biased, so they do not provide valid forecasts (Gregory & Duran, 2001).
2.The stories are based on expert judgments. According to prior research, expert judgments about what will happen in complex, uncertain situa:ons are useless:
a. Seer-­‐sucker Theory
b. Tetlock’s 20-­‐year experiment
Thursday, 23 July 15
9
9
IPCC process of crea*ng climate scenarios via computer simula*ons is enormously complex
1. Judgments are made on what variables to include (e.g. CO2), and exclude (e.g. Sun); 2. Judgments are made on the values of many parameters, and their (nonlinear) rela*onships
3. Around 50,000 grid squares are modeled
4. Grid square models interact
5. Models are adjusted to produce expected outputs
6. Budget for computer simula*ons is enormous
10
Thursday, 23 July 15
10
Green, Armstrong, & Soon (2009) no-­‐change model is sophis0catedly simple
1. Based on an examina*on of diverse long temperature histories…
2. No long-­‐term trend
3. Reversals on all *me scales
4. Correlated with solar cycles and varia*ons in ac*vity 5. Weakly correlated with CO2…
but temperature changes precede CO2 changes, and high CO2 levels have been associated with ice ages.
11
Thursday, 23 July 15
11
Simple Forecas*ng Checklist ra*ngs:
IPCC projec*ons vs. no-­‐change forecasts 12
Thursday, 23 July 15
12
Evidence on accuracy of IPCC projec*ons vs. no-­‐change forecasts using Hadley data Tests of forecasts over the 1851-­‐1975 forecas*ng period yielded 58 forecasts for horizons of 91 to 100 years. The errors of these IPCC forecasts were 12.6 *mes larger than those from the easily understood no-­‐change model (Green, Armstrong, & Soon 2009).
Chart from Forecas*ng Global Climate Change (2014)
13
Thursday, 23 July 15
13
Evidence on accuracy of IPCC projec*ons vs. no-­‐
change forecasts using Loehle & McCulloch (2008) data From Forecas*ng Global Climate Change (2014)
14
Thursday, 23 July 15
14
Valida*on over different *me-­‐periods; data: Similar results
From Forecas*ng Global Climate Change (2014)
15
Thursday, 23 July 15
15
The Complexity Penalty
In the late 1970, a research review found that complex forecas*ng methods harmed accuracy.
In the late 1990s, the Forecas*ng Principles Project developed 139 principles for forecas*ng. All were simple to understand.
Our recent systema*c review failed to find a single study showing that a complex method was more accurate than a simpler method.
16
Thursday, 23 July 15
16
Two Ques*ons for you
What percentage of papers at this conference propose complex methods?
Why does that happen?
17
Thursday, 23 July 15
17
Seduced by complexity
Some evidence suggests that the popularity of complexity may be due to incen*ves: (1)researchers are rewarded for publishing in highly ranked journals, which favor complexity;
(2)forecasters can use complex methods to provide forecasts that support decision-­‐makers’ plans; and (3)forecasters’ clients may be impressed and reassured by incomprehensibility. 18
Thursday, 23 July 15
18
Conclusions
1. Those who prefer their forecasts to be accurate should accept forecasts only from simple evidence-­‐based procedures.
2. Alarming IPCC temperature projec*ons are based on procedures that are too complex to be trusted.
3. Rate the simplicity of forecas*ng procedures used for problems you are interested in using the ques*onnaire at… simple-­‐forecas*ng.com. 19
Thursday, 23 July 15
19