Evaluation of Timber Committee
Forecasts
Goal: Improve response rate and
forecast accuracy
• What is the response rate
• What is the accuracy of the forecasts
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Responses 2002-2004
2002
Total Replies
Countries not Albania
replying to TCQ Armenia
Azerbaijan
Belarus
Bulgaria
Denmark
Georgia
Iceland
Israel
Kazakhstan
Kyrgysztan
Luxembourg
Malta
Slovenia
Tajkistan
Ukraine
Uzbekistan
2003
32
2004
31
Armenia
Azerbaijan
Belarus
Bosnia
Bulgaria
Denmark
Georgia
Greece
Iceland
Israel
Kazakhstan
Kyrgysztan
Liechtenstein
Luxembourg
Malta
Moldova
Slovenia
Tajkistan
Turkmenistan
Ukraine
Uzbekistan
Contact
31
Albania
Armenia
Azerbaijan
Belarus
Bosnia
Denmark
Georgia
Greece
Iceland
Israel
Kazakhstan
Kyrgysztan
Latvia
Liechtenstein
Luxembourg
Malta
Moldova
Spain
Tajkistan
Turkmenistan
Ukraine
Uzbekistan
Dida
Kuzmenkov
Stenin
Hviid
Papaevangelou
I. Koval
Arvids Ozols
Felix Naescher
Erasmy
Anthony Mifsud
Tomaz Remic
J. M. Solano
Zibtsev
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Response Completeness- forecast for current year
Country
Albania
Austria
Belgium
Bulgaria
Canada
Croatia
Cyprus
Czech Republic
Estonia
Finland
France
Germany
Greece
Hungary
Ireland
Italy
Latvia
Lithuania
Macedonia
Netherlands
Norway
Poland
Portugal
Romania
Russian Federation
Serbia and Montenegro
Slovakia
Slovenia
Spain
Sweden
Switzerland
Turkey
United Kingdom
United States
2002
61%
79%
2003
64%
76%
91%
89%
100%
94%
100%
100%
95%
82%
73%
3%
100%
85%
85%
100%
100%
80%
88%
45%
95%
100%
100%
89%
100%
82%
80%
100%
94%
100%
100%
100%
100%
94%
100%
83%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
80%
100%
100%
100%
76%
89%
98%
95%
100%
100%
97%
100%
76%
2004
89%
89%
68%
100%
100%
98%
100%
100%
100%
100%
73%
100%
79%
85%
100%
59%
98%
98%
100%
100%
100%
100%
100%
100%
79%
100%
100%
91%
100%
100%
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Completeness of Response
20
Number of Countries
18
16
14
12
2002
10
2003
2004
8
6
4
2
0
100%
>=90%
>=80%
>=70%
<70%
Questionnaire Completion rate
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Responses to specific item – Sawn
softwood and OSB
Forecast
TC
Meeting
for
2004
2003
2002
2005
2004
2004
2003
2003
2002
Number of Countries replying to questions on
Production
Import
Export
Sawn Softwood
OSB
OSB
31
24
24
24
31
24
26
26
29
27
26
27
30
29
29
29
28
21
21
21
30
23
23
23
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Forecast Accuracy
Constraints
• One set of forecasts (current and next year) to
limit initial work
• Data sufficiently far back so that forecast years
would have more or less final data – October
2002
• One product – sawn softwood
• 31 replies to this item (although not every
country forecast all 6 possible data points)
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Summary Statistics
Forecast Errors (forecast % change minus actual % change)
Median
Average
Standard Deviation
Production
2002/2001
2003/2002
-0.00595
0.00355
-0.03197
0.00852
0.12619
0.12754
Exports
2002/2001
2003/2002
-0.04919
0.00935
0.04374
0.07183
1.21599
0.32429
Imports
2002/2001
2003/2002
-0.04564
-0.00568
0.69130
-0.03180
3.78351
0.16305
Bear in mind
• More detailed statistics available in attached
paper (excluding dropped outliers)
• This is only for one product, one set of
forecasts
• No weighting by size of country
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Forecast Error Distribution - Exports
20
18
16
Frequency - # of countries
14
12
10
8
6
4
2
0
<=-45%
-40%
-30%
Source: UNECE Timber Comittee Forecasts, Oct 2002
-20%
-10%
0%
10%
Forecast % Change minus Actual % Change
20%
30%
2002 / 2001
40%
>=45%
2003 / 2002
Forecast Error Distribution - Imports
20
18
16
Frequency - # of countries
14
12
10
8
6
4
2
0
<=-45%
-40%
-30%
Source: UNECE Timber Comittee Forecasts, Oct 2002
-20%
-10%
0%
10%
Forecast % Change minus Actual % Change
20%
30%
2002 / 2001
40%
>=45%
2003 / 2002
Forecast Error Distribution - Production
20
18
16
Frequency - # of countries
14
12
10
8
6
4
2
0
<=-45%
-40%
-30%
Source: UNECE Timber Comittee Forecasts, Oct 2002
-20%
-10%
0%
10%
Forecast % Change minus Actual % Change
20%
30%
2002 / 2001
40%
>=45%
2003 / 2002
Significant errors
What is a “significant” error?
• When the direction of the forecast is opposite
to the actual outcome
• We used a cutoff of forecast >1% and actual
<-1% or forecast <-1% and actual >1%
Production
Exports
Imports
2002/2001 2003/2002 2002/2001 2003/2002 2002/2001 2003/2002
wrong direction
7
3
11
5
8
7
forecast missing
0
1
2
5
2
6
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
“No change” forecasts
What is a “no change” forecast?
• Where the forecast % change is between
+0.5% and –0.5%
• Such a forecast could be the outcome of a
careful process of consultation, estimation and
modeling or simply rounding/repeating an
earlier figure
Production
Exports
Imports
2002/2001 2003/2002 2002/2001 2003/2002 2002/2001 2003/2002
number of countries with
actual no change
number of countries with
"no change" forecast
number of countries
predicting no change right
2
3
1
2
1
1
6
10
1
13
3
12
2
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
1
F
A
O
Are the forecasts better than guessing
results of EAD analysis
• 5 of the 6 series analysed showed a close to
normal distribution (the exception was
2002/2001 exports)
• The forecasts were better than random
guessing in more than 80% of the cases
(>86% for most series)
• Forecasts appear to be more conservative than
warranted by random outcomes
• There is still room for improvement
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O
Future steps
•
•
•
•
•
Analytical
Cover earlier years (1997, 1987) to compare
current forecast success with other periods
Cover other products to see if trend in sawn
softwood is representative (one roundwood,
one panel)
Accomplishing goals
Reaching out to find more respondents
Feedback to show how successful forecasts
are, encourage forecasters to “risk”
Training / networking to share knowledge and
“best practice”
Working Party on Forest Economics and Statistics
27th Session, 22-24 March 2005, Geneva
F
A
O