Economic Forecast Validation: Evaluating the Calibration of Models

π‘₯
𝑦
πœ‡
ο‚·
ο‚·
ο‚·
ο‚·
ο‚·
𝜎
π‘₯
𝑦
π‘₯
𝐹(π‘₯) ≔ 𝑃[𝑋 ≀ π‘₯]
π‘₯1 , … , π‘₯𝑛
𝑋=
π‘₯
𝑋
π‘₯
𝐹
π‘₯1 , … , π‘₯𝑁
𝐷𝑁 ≔ sup |𝐹 (𝑁) (π‘₯) βˆ’ 𝐹(π‘₯)|
π‘₯
𝐹
(𝑁)
π‘₯
𝑦
𝑁
𝐹
(𝑁) (π‘₯)
1
≔ βˆ‘ 𝐼(π‘₯𝑖 ≀ π‘₯)
𝑁
𝑖=1
𝐹1 , … , 𝐹𝑁
π‘₯̃𝑖 ≔ 𝐹𝑖 (π‘₯𝑖 )
π‘₯𝑖
𝐹𝑖
π‘˜
𝑖
π‘₯Μƒ1 , … , π‘₯̃𝑁
𝒙
𝑭
𝑧1 ≔ 𝐹1 (π‘₯1 )
𝑧2 ≔ 𝐹2 (π‘₯2 |π‘₯1 )
…
π‘§π‘˜ = πΉπ‘˜ (π‘₯π‘˜ |π‘₯1 , … , π‘₯π‘˜βˆ’1 )
(π‘₯1 , … , π‘₯π‘˜ )
[0,1]π‘˜
𝒙
𝒙
𝐹𝑖
π‘₯𝑖
𝑭
π‘₯1 , … , π‘₯π‘–βˆ’1
𝒛 ≔ (𝑧1 , … , π‘§π‘˜ )
𝑁
π‘­πŸ , … , 𝑭𝑡
π’™πŸ , … , 𝒙𝑡
π‘˜
π‘˜
π‘˜
[0,1]π‘˜
𝒛
𝑧1 , … , π‘§π‘˜
[0,1]
𝑁
π‘˜
π‘βˆ—π‘˜
[0,1]π‘˜
π‘˜
𝑁
π‘˜
π’™πŸ
π’™πŸ
π‘₯Μƒ1, … , π‘₯̃𝑁
π‘₯2
π‘₯1
π‘˜!
𝑁
Μ‚ =
𝐡𝑆
1
βˆ‘(𝑝𝑖 βˆ’ 𝐼𝑖 )2
𝑁
𝑖=1
𝑁
𝑝𝑖
𝐼𝑖
𝐡𝑆 = 0
𝑝𝑖 = 𝐼𝑖
π‘ž
𝐸[𝐡𝑆] = π‘ž βˆ— (𝑝 βˆ’ 1)2 + (1 βˆ’ π‘ž) βˆ— 𝑝2 = 𝑝2 βˆ’ 2π‘π‘ž + π‘ž
𝑝
𝑝=π‘ž
2
2
𝐡𝑆 = 𝐸𝑝 [(𝑝 βˆ’ 𝑓(𝑝)) ] βˆ’ 𝐸𝑝 [(𝑠 βˆ’ 𝑓(𝑝)) ] + 𝑠(1 βˆ’ 𝑠)
𝑓(𝑝)
𝑝
𝑝 𝑠
ο‚·
ο‚·
π‘‰π‘Žπ‘Ÿπ‘ [𝑓(𝑝)]
Μ‚
𝐡𝑆
𝑠
𝑁
𝑖
ο‚·
𝑠
𝑠 = 0.5
𝑠
𝑦1 < 𝑦2 < β‹― < 𝑦𝐾
𝑝1 : = 𝑃[π‘₯ ≀ 𝑦1 ], … , 𝑝𝐾 ≔ 𝑃[π‘₯ ≀ 𝑦𝐾 ]
1
2
𝑅𝑃𝑆 = 𝐸 [ (π‘π‘˜ βˆ’ 𝐼(π‘₯ ≀ π‘¦π‘˜ )) ]
𝐾
𝐢𝑅𝑃𝑆 = 𝐸 [∫(𝐹(π‘₯) βˆ’ 𝐼(π‘₯0 ≀ π‘₯))2 𝑑π‘₯]
𝐹
π‘₯0
𝑁
1
Μ‚ = βˆ‘ ∫(𝐹𝑖 (π‘₯) βˆ’ 𝐼(π‘₯𝑖 ≀ π‘₯))2 𝑑π‘₯
𝐢𝑅𝑃𝑆
𝑁
𝑖=1
2
𝐿
1
Prob(X < x)
I (x_0 < x)
0.8
0.6
0.4
0.2
0
0%
𝐹(π‘₯) = 0
π‘₯ < 0 𝐹(π‘₯) = (1 βˆ’ π‘ž)
2%
0≀π‘₯<1
2
(π‘ž βˆ’ 𝐼{π‘₯0 =1} )
4%
6%
π‘₯0 = 1
𝐹(π‘₯) = 1
1≀π‘₯
8%
π‘₯0 = 0
π‘ž
π‘₯
𝐢𝑅𝑃𝑆𝑆 = 1 βˆ’
π‘₯𝑖
𝐢𝑅𝑃𝑆
πΆπ‘…π‘ƒπ‘†π‘Ÿπ‘’π‘“π‘’π‘Ÿπ‘’π‘›π‘π‘’
(0.5) βˆ— (0.9 + 0.1) = 0.5
0.1
𝐸[(0.5 βˆ’ πΌπ»π‘’π‘Žπ‘‘π‘  )2 ] = (0.5) βˆ— (0.5 βˆ’ 1)2 + (0.5) βˆ— (0.5 βˆ’ 0)2 = 0.25
0.9
𝑖 π‘‘β„Ž
0.5 βˆ— 𝐸[(0.9 βˆ’ πΌπ»π‘’π‘Žπ‘‘π‘  )2 |𝑅𝑒𝑑 π‘π‘œπ‘–π‘›] + 0.5 βˆ— 𝐸[(0.1 βˆ’ πΌπ»π‘’π‘Žπ‘‘π‘  )2 |πΊπ‘Ÿπ‘’π‘’π‘› π‘π‘œπ‘–π‘›]
= 0.5 βˆ— [(0.9) βˆ— (0.9 βˆ’ 1)2 + (0.1) βˆ— (0.9 βˆ’ 0)2 ] + 0.5 βˆ— [(0.1) βˆ— (0.1 βˆ’ 1)2 + (0.9) βˆ— (0.1 βˆ’ 0)2 ] = 0.09
1βˆ’
0.09
0.25
= 0.64
𝐢𝑅𝑃𝑆𝑆 = 64%
ο‚·
ο‚·
ο‚·
ο‚·
ο‚·
ο‚·
ο‚·
π‘₯1 , … , π‘₯8
𝑧1 , … , 𝑧8
𝑧
𝑖
π‘₯1 , π‘₯2 , … , π‘₯π‘–βˆ’1
K-S Statistic: βˆšπ‘ βˆ— 𝐷𝑁
Median
Min
Max
1.3300
0.5857
2.3373
CRPSS vs. Through-the-Cycle
(best possible = 100%)
7.7%
-4.5%
17.4%
𝛼
0.2
0.1
0.05
0.01
𝑦=π‘₯
π‘₯𝑖
Ξ¦βˆ’1 (𝑧)
K-S Stat
1.07
1.22
1.36
1.63
CRPSS vs. Stationary
(best possible = 100%)
3.8%
-25.0%
18.9%
3
3
Pooled
Pooled
2
y=x
2
y=x
1
1
0
0
-3
-2
-1
0
1
2
-3
3
-2
-1
0
-1
-1
-2
-2
-3
-3
3
Pooled
2
y=x
1
0
-3
-2
-1
0
-1
-2
-3
1
2
3
1
2
3
2.5
CAD
y=x
-2.5
2.5
USD
2
1.5
-1.5
Correlation
USD
CAD
AUD
GBP
HKD
CHF
JPY
EUR
2
CAD|USD
1.5
y=x
1
1
0.5
0.5
0
-0.5
-0.5
0.5
1.5
2.5
-2.5
-1.5
0
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-2.5
-2.5
USD
100%
59%
61%
62%
60%
61%
45%
61%
CAD
59%
100%
63%
61%
60%
61%
43%
60%
AUD
61%
63%
100%
64%
62%
64%
41%
63%
GBP
62%
61%
64%
100%
61%
63%
44%
61%
HKD
60%
60%
62%
61%
100%
61%
47%
60%
CHF
61%
61%
64%
63%
61%
100%
44%
62%
JPY
45%
43%
41%
44%
47%
44%
100%
47%
0.5
EUR
61%
60%
63%
61%
60%
62%
47%
100%
1.5
2.5
Correlation
USD
CAD
AUD
GBP
HKD
CHF
JPY
EUR
USD
100%
96%
84%
90%
75%
77%
58%
73%
CAD
96%
100%
81%
93%
69%
79%
64%
74%
AUD
84%
81%
100%
83%
75%
75%
55%
67%
GBP
90%
93%
83%
100%
65%
82%
67%
76%
HKD
75%
69%
75%
65%
100%
66%
35%
64%
CHF
77%
79%
75%
82%
66%
100%
50%
80%
JPY
58%
64%
55%
67%
35%
50%
100%
67%
EUR
73%
74%
67%
76%
64%
80%
67%
100%
2.5
USD
CAD|USD
y=x
2
1.5
1
0.5
-2.5
-1.5
0
-0.5
-0.5
0.5
1.5
2.5
-1
-1.5
-2
-2.5
< 10βˆ’24
< 0.005
140
120
100
80
60
HKD Outlier
40
20
0.02%
0.13%
0.25%
0.36%
0.48%
0.59%
0.71%
0.82%
0.94%
1.05%
1.17%
1.28%
1.40%
1.51%
1.63%
0
2.5
Pooled_Q1
Pooled_Q2
Pooled_Q3
Pooled_Q4
y=x
-2.5
-1.5
2.5
Pooled_Q1
2
Pooled_Q2
1.5
Pooled_Q3
1
Pooled_Q4
y=x
0.5
0
-0.5
-0.5
0.5
1.5
2.5
-2.5
-1.5
2
1.5
1
0.5
0
-0.5
-0.5
-1
-1
-1.5
-1.5
-2
-2
-2.5
-2.5
0.5
1.5
2.5
ο‚·
ο‚·
ο‚·
𝑦𝑖 = log(π‘₯𝑖 )
𝑋1 , … , 𝑋𝑁
π‘Œπ‘–
𝑖
π‘₯1 , … , π‘₯𝑁
π‘Œπ‘– = log(𝑋𝑖 )
πœ‡ = (πœ‡1 , … , πœ‡π‘ )𝑇
Ξ£ = (πΆπ‘œπ‘£(π‘Œπ‘– , π‘Œπ‘— ))
1≀𝑖,𝑗≀𝑁
π‘Œπ‘˜+1
(𝑦 βˆ’ πœ‡)π‘˜
π‘Œ1 = 𝑦1 , … , π‘Œπ‘˜ = π‘¦π‘˜
𝑦1 βˆ’ πœ‡1
𝑦2 βˆ’ πœ‡2
(𝑦 βˆ’ πœ‡)π‘˜ = [ … ]
π‘¦π‘˜ βˆ’ πœ‡π‘˜
π‘Œπ‘˜+1
Ξ£
Ξ£ = 𝐿𝐿𝑇
𝐿
πΏβˆ’1 (π‘Œ βˆ’ πœ‡)
𝑁(0, 𝐼𝑁 )
𝐿(π‘˜) : = (𝐿𝑖𝑗 )1≀𝑖,π‘—β‰€π‘˜
π‘˜π‘₯π‘˜
𝐿
πΏπ‘˜+1 : = [πΏπ‘˜+1,1 πΏπ‘˜+1,2 … πΏπ‘˜+1,π‘˜ ]
π‘˜
(π‘˜ + 1)π‘‘β„Ž
𝐿
π‘Œπ‘˜+1 = πœ‡π‘˜+1 + πΏπ‘˜+1 [𝐿(π‘˜) ]βˆ’1 (𝑦 βˆ’ πœ‡)π‘˜ + πΏπ‘˜+1,π‘˜+1 π‘π‘˜+1
π‘π‘˜+1
𝑁(0,1)
𝐸[π‘Œπ‘˜+1 |π‘Œ1 = 𝑦1 , … , π‘Œπ‘˜ = π‘¦π‘˜ ] = πœ‡π‘˜+1 + πΏπ‘˜+1 [𝐿(π‘˜) ]βˆ’1 (𝑦 βˆ’ πœ‡)π‘˜
π‘‰π‘Žπ‘Ÿ[π‘Œπ‘˜+1 |π‘Œ1 = 𝑦1 , … , π‘Œπ‘˜ = π‘¦π‘˜ ] = 𝐿2π‘˜+1,π‘˜+1
π‘Œπ‘˜+1