Bootstrapping time series models
Hongyi Li & G.S. Maddala'
1 - INTRODUCTION
1. Introduction
The purposes of this paper are:
1. To provide a survey of bootstrap procedures
applied to time series econometric models
Econometrics
the application of statistical and mathematical
theories in economics for the purpose of testing
hypotheses and forecasting future trends
2. To present some guidelines for empirical
researchers in this area
1. Introduction
• Most of the inferential procedures available in the analysis of time
series data are asymptotic
• Although analytic small sample results are available in a few cases,
there is currently, no widely applicable and easily accessible method
that can be used to make small sample inferences. Methods like
Edgeworth expansions involve a lot of algebra and are also applicable
in very special cases.
• The bootstrap technique introduced by Efron (1979) could possibly be
a potential alternative in estimation and inference from time series
models in finite samples. However, in time series regressions, the
standard bootstrap resampling method designed for independent and
identically distributed (IID) errors is not applicable because in most
situations the assumption of IID errors is violated.
2 - GENERAL GUIDELINES FOR
USING THE BOOTSTRAP APPROACH
2. General guidelines for using the bootstrap approach
The basic bootstrap
approach consists of
drawing repeated samples
(with replacement)
Time series models
the simplest one that is valid for
IID observations
IID assumption is not satisfied
The method needs to be
modified
2. General guidelines for using the bootstrap approach
• Estimating Standard Errors
Using BS distribution to Estimate SE
Small Sample Size
Using asymptotic distribution to
Estimate SE
BUT
The “Small Sample Size” distribution
Must be NORMAL
Better
Estimation of SE
2. General guidelines for using the bootstrap approach
Using BS distribution to Estimate SE
Even if
Same
Using asymptotic distribution to
Estimate SE
• Confidence Interval statements
Using BS distribution to Estimate CI
+
BS distribution is skewed
Different CI
Using asymptotic distribution to
Estimate CI
2. General guidelines for using the bootstrap approach
Methods to Construct Confidence intervals
Coverage error
1. Using the asymptotic distribution of 𝜽
O(n-1/2)
2. The percentile method
3. Bias-corrected (BC) percentile interval
4. Accelerated bias-corrected (BCα) percentile interval
O(n-1)
5. The percentile-t (or bootstrap-t) method
6. Beran’s pivotal method (iterative, BS
)
O(n-3)
Coverage Error
The difference between the actual
coverage and nominal coverage
2. General guidelines for using the bootstrap approach
Methods to Construct Confidence intervals
1.
Use the asymptotic distribution of 𝜽
The two-sided confidence interval is:
𝜽 ± zα·SE(𝜽)
zα - the (100 - α) percentile from the
standard normal distribution
SE(𝜽) - the asymptotic
standard error of 𝜽
Symmetric Interval
Coverage error = O(n-1/2)
2. General guidelines for using the bootstrap approach
Methods to Construct Confidence intervals
2. The percentile method
Use the bootstrap distribution of 𝜽 *
The two-sided (100 - 2·α) confidence
interval for θ is
(𝜽 – z*1-α , 𝜽 + z*α)
Two-sided equal-tailed interval
(often non-symmetric)
The nominal coverage of this
interval is (100 - 2α)
Coverage error = O(n-1/2)
z*α - the 100α percentile of the
distribution of 𝜽∗ - 𝜽
2. General guidelines for using the bootstrap approach
Hypothesis Testing
𝑯𝟎 : 𝜽 = 𝜽𝟎
𝑯𝟏 : 𝜽 ≠ 𝜽𝟎
Reminder:
𝜽𝟎
- The parameter’s value under the null hypothesis
𝜽
- An estimation of the parameter’s value using the original samples
𝜽∗
- An estimation of the parameter’s value using the BS resamples
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?
Use the BS distribution of:
𝜽∗ − 𝜽
BUT NOT:
𝜽∗ − 𝜽𝟎
𝜽∗ − 𝜽
Use the properly studentized statistic:
𝜽∗
−𝜽
𝝈∗
𝝈
BUT NOT:
𝝈∗ - the estimate of 𝝈 from the BS sample
𝜽∗ − 𝜽
2. General guidelines for using the bootstrap approach
Hypothesis Testing
• Two important issues concerning hypothesis testing using
bootstrap methods relate to the questions about:
a) What test statistic to bootstrap?
b) How to generate the bootstrap samples?
The Model we’ll discuss is simple regression:
𝒚 = 𝜷𝒙 + 𝜺,
𝜺~𝒊𝒊𝒅(𝟎, 𝝈𝟐 )
𝜷, 𝝈 are OLS estimators for
𝜺 - the OLS residuals
𝜺∗ − the BS residuals, obtained by resampling 𝜺
2. General guidelines for using the bootstrap approach
Methods of BS Samples Generation
Consider two sampling schemes for the generation of the bootstrap samples:
𝑺𝟏 : 𝒚∗ = 𝜷𝒙 + 𝜺∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
Both use 𝜀 ∗
For each sampling scheme, consider two test t-statistics:
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝝈∗
𝝈∗
2. General guidelines for using the bootstrap approach
4 versions of the t-statistic can be defined:
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙 + 𝜺∗
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝝈∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝝈∗
2. General guidelines for using the bootstrap approach
4 versions of the t-statistic can be defined:
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙 + 𝜺∗
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
Hall & Wilson
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝝈∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
Giersbergen & Kiviet
Based on Monte-Carlo study of an AR(1) model
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝝈∗
2. General guidelines for using the bootstrap approach
4 versions of the t-statistic can be defined:
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙 + 𝜺∗
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝝈∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
Giersbergen & Kiviet
Based on Monte-Carlo study of an AR(1) model
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝝈∗
2. General guidelines for using the bootstrap approach
Giersbergen & Kiviet
Based on Monte-Carlo study of an AR(1) model
𝑻𝟏 : 𝑻 𝜷 = 𝜷∗ − 𝜷
𝝈∗
𝑺𝟏 : 𝒚∗ = 𝜷𝒙 + 𝜺∗
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
Equivalent in non-dynamic models
In dynamic models
𝑺𝟐
𝑻𝟐
is better
𝝈∗
3 - STRUCTURED TIME SERIES MODELS:
THE RECURSIVE BS
3. Structured Time Series Models: The Recursive BS
ARMA Models
ARMA – AutoRegressive + Moving Average
ARMA(p,q) models with known p and q
3. Structured Time Series Models: The Recursive BS
ARMA Models
Consider the stationary AR(p) model
𝑝
𝑦𝑡 =
𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 ,
𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1
Given data on n + p observations (𝑦1−𝑝 , … , 𝑦0 , 𝑦1 , … , 𝑦𝑛 )
Our objective is to get:
• confidence intervals for the parameters 𝑎𝑖 or
• some smooth function h(𝑎1 , 𝑎2 , . . . , 𝑎𝑝 ) of the parameters 𝑎𝑖
3. Structured Time Series Models: The Recursive BS
The stationary AR(p) model
𝑝
𝑦𝑡 =
𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 ,
𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1
1 – Estimate (𝑎1 , 𝑎2 , . . . , 𝑎𝑝 ) by OLS based on n observations (𝑦1 , … , 𝑦𝑛 )
We get (𝑎1 ,𝑎2 , . . . ,𝑎𝑝 ) and the least squares residuals 𝑒𝑡
2 – Define the centered and scaled residuals 𝑒𝑡 = 𝑒𝑡 −
1
𝑛
𝑒𝑡
𝑛
𝑛−𝑝
Bickel & Freedman – residuals tend to be smaller than the true errors
3 – Resample 𝑒𝑡 with replacement to get the BS residuals 𝒆𝒕 ∗
1 2
3. Structured Time Series Models: The Recursive BS
The stationary AR(p) model
𝑝
𝑦𝑡 =
𝑎𝑖 𝑦𝑡−𝑖 + 𝑒𝑡 ,
𝑒𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑖=1
𝑝
4 – Construct the BS sample
∗
𝑎𝑖 𝑦𝑡−𝑖
+ 𝑒𝑡 ∗
𝑦𝑡∗ =
𝑖=1
(done recursively using 𝑦𝑡∗ = 𝑦𝑡 for 𝑡 = 1 − 𝑝, … , 0 in each BS iteration)
Bose – the LS estimates 𝑎𝑖 can be BS-ed with accuracy 𝑜
1
𝑛
−2
- Little o
1
improving the normal approximation error of 𝑂 𝑛
−2
- Big o
3. Structured Time Series Models: The Recursive BS
Stationary AR(p) model
Stationary
AR(1) Model
Unstable
Explosive
3. Structured Time Series Models: The Recursive BS
The AR(1) Model
𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡
𝑦0 = 0
𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
−∞ < 𝛽 < ∞
3. Structured Time Series Models: The Recursive BS
The AR(1) Model
𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡
𝑦0 = 0
𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
−∞ < 𝜷 < ∞
𝛽 <1
𝑦𝑡 is stationary
𝛽 =1
𝑦𝑡 is unstable
𝛽 >1
𝑦𝑡 is explosive
3. Structured Time Series Models: The Recursive BS
The AR(1) Model
𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡
𝑦0 = 0
𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
−∞ < 𝛽 < ∞
Rubin(1950) – the OLS estimator β of β is consistent in the range
−∞, ∞
However, the asymptotic distributions of 𝜷 in the different ranges are different:
• In the stationary case 𝛽 < 1 - the asymptotic distribution is normal
• For the explosive case 𝛽 > 1 - Anderson (1959) - the limiting distribution is a
Cauchy distribution
3. Structured Time Series Models: The Recursive BS
The AR(1) Model with intercept
𝑦𝑡 = 𝛼 + 𝛽𝑦𝑡−1 + 𝑢𝑡 ,
𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
Since the distribution of the OLS estimator of 𝛽 of 𝛽 is invariant to α and 𝜎 2
Set α = 0 and 𝜎 2 = 1 and get
𝑦𝑡 = 𝛽𝑦𝑡−1 + 𝑢𝑡 , 𝑢𝑡 ~𝑖𝑖𝑑(0,1)
Dufour (1990) & Andrews (1993) – developed exact inference
procedures for the AR(1) parameter but these depend on the normality
assumption of the errors
While, the BS methods which are robust to distributional assumptions of the
errors, hold promise
3. Structured Time Series Models: The Recursive BS
The AR(1) Model with intercept
𝑢𝑡 ~𝑖𝑖𝑑(0, 𝜎 2 )
𝑦𝑡 = 𝛼 + 𝛽𝑦𝑡−1 + 𝑢𝑡 ,
The procedure for the generation of the BS samples is the recursive procedure
𝑝
∗
𝑎𝑖 𝑦𝑡−𝑖
+ 𝑒𝑡 ∗
𝑦𝑡∗ =
𝑖=1
(𝑎1 , 𝑎2 , . . . , 𝑎𝑝 )
The sampling scheme
𝑺𝟐 : 𝒚∗ = 𝜷𝟎 𝒙 + 𝜺∗
𝑦0 is given
𝑝=1
𝑦(𝑡=1,…,𝑛) are recursively BS calculated
𝛽0
The t-statistic
𝑻𝟐 : 𝑻 𝜷𝟎 = 𝜷∗ − 𝜷𝟎
𝝈∗
Rayner (1990) – the use of the student-t approximation is not satisfactory, particularly for high values of 𝛽
the bootstrap-t performs very well in samples of sizes 5-10, even when mixtures of normal
distributions are used for the errors
4 - GENERAL ERROR STRUCTURES - THE
MOVING BLOCK BOOTSTRAP (MBB)
4. General error structures – The MBB
The Moving Block Bootstrap
Background
Application of the residual based bootstrap methods is straightforward if the
error distribution is specified to be an ARMA(p,q) process with known p and q
However, if the structure of serial correlation is not tractable or is misspecified,
the residual based methods will give inconsistent estimates
Carlstein (1986) – first discussed the idea of bootstrapping blocks of observations rather
than the individual observations. The blocks are nonoverlapping
Künsch (1989) and Singh (1992) – independently introduced a more general BS
procedure, the moving block BS (MBB) which is applicable to stationary time series data. In
this method the blocks of observations are overlapping.
4. General error structures – The MBB
The Moving Block Bootstrap
Divide the data of n observations into blocks of length l and select b of these blocks
(with repeats allowed) by resampling with replacement all the possible blocks
𝑦1
𝑦𝑛−1
𝑦2
l
l
For simplicity assume 𝑛 = 𝑏𝑙
l
l
In the Carlstein procedure:
𝑦𝑛
𝑛
𝑙
= 𝑏 blocks
In the Künsch procedure: 𝑛 − 𝑙 + 1 blocks
The 𝑘 𝑡ℎ block is 𝐿𝑘 = 𝑥𝑘 , … , 𝑥𝑘+𝑙−1
𝑘 = 1,2, … , (𝑛 − 𝑙 + 1)
For example with n = 6 and I = 3 suppose the data are: xt = {3,6,7,2,1,5).
The blocks according to Carlstein are {(3,6,7), (2,1,5)). The blocks according
to Kiinsch are {(3,6,7), (6,7,2), (7,2, l), (2,1,5)).
4. General error structures – The MBB
The Moving Block Bootstrap
EXAMPLE
𝑥𝑡 =
The blocks in the
Carlstein procedure are:
3
6
7
2
1
5
3
6
7
2
1
5
The blocks in the Künsch
procedure are:
3
6
7
6
7
2
7
2
1
2
1
5
Draw a sample of two blocks with replacement in each case
Suppose, the first draw gave
3
6
Than, The probability of missing all of
Carlstein: 50%
(WLOG)
7
2
1
5
is:
Künsch: 25%
Higher probability of missing entire blocks in the Carlstein scheme (non overlapping blocks)
Carlstein scheme is not popular and not often used
4. General error structures – The MBB
The Moving Block Bootstrap
A comparison of four different block bootstrap methods
Boris Radovanov & Aleksandra Marcikić
MBB – Moving block bootstrap
NBB – Non-overlapping block
bootstrap
SBB – Stationary block bootstrap
SS - Subsampling
4. General error structures – The MBB
Problems with MBB
There are some important problems worth noting about the MBB procedure
1.
The pseudo time series
generated by the moving
block method is not
stationary, even if the original
series {𝑥𝑡 ) is stationary
Politis and Romano (1994)
A stationary bootstrap method
The suggested method involves sampling blocks of
random length, where the length of each block has a
geometric distribution. They show that the pseudo
time series generated by the stationary bootstrap
method is indeed stationary.
4. General error structures – The MBB
Problems with MBB
There are some important problems worth noting about the MBB procedure
2.
The mean 𝑥𝑛∗ of the moving block bootstrap is biased in the sense that:
𝐸 𝑥𝑛∗ |𝑥1 , 𝑥2 , … , 𝑥𝑛 − 𝑥𝑛 ≠ 0
3.
The MBB estimator of the variance of 𝑛 ∙ 𝑥𝑛 is also biased
Davidson and Hall (1993)
This creates problems in using the percentile-t method with the MBB
the usual estimator:
𝑛
2
𝜎 =𝑛
−1
𝑛
𝑥𝑖 − 𝑥𝑛
𝑖=1
Should be modified to:
2
𝜎 2 = 𝑛−1
𝑖−1 𝑛−𝑘
𝑥𝑖 − 𝑥𝑛
𝑖=1
2
+
𝑥𝑖 − 𝑥𝑛 𝑥𝑖+𝑘 − 𝑥𝑛
𝑘=1 𝑖=1
With this modification the bootstrap-t can improve substantially on
the normal approximation
4. General error structures – The MBB
Optimal Length of Blocks
Several rules that have been suggested are based on different criteria. However,
the rules are useful as rough guides to selecting the optimal sized blocks
1.
2.
Carlstein’s
non-overlapping blocks
<
Künsch’s
moving blocks
Politis and Romano‘s stationary bootstrap method
• The average length of a block is 𝑝1, where p is the parameter of the
geometric distribution
• The application of stationary bootstrap is less sensitive to the choice of
p than the application of moving block bootstrap is to the choice of I
4. General error structures – The MBB
Optimal Length of Blocks
3. Carlstein’s rules for non-overlapping blocks
Interested in minimizing the MSE of the block bootstrap estimate
of the variance of a general statistic 𝑡(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
VARIANCE
As the block size increases
BIAS
As the dependency among
the 𝑥𝑖 gets stronger
A longer block size is
needed
4. General error structures – The MBB
Optimal Length of Blocks
3. Carlstein’s rules for non-overlapping blocks
Interested in minimizing the MSE of the block bootstrap estimate
of the variance of a general statistic 𝑡(𝑥1 , 𝑥2 , … , 𝑥𝑛 )
VARIANCE
As the block size increases
BIAS
As the dependency among
the 𝑥𝑖 gets stronger
A longer block size is
needed
The optimal block size 𝑙 ∗ for the AR(1) model 𝑥𝑡 = 𝜑𝑥𝑡−1 + 𝜀𝑡 is:
𝒍∗ = 𝟐𝝋 𝟏 − 𝝋𝟐
𝟐 𝟑 𝟐 𝟑
𝒏
4. General error structures – The MBB
Optimal Length of Blocks
4. Hall and Horowitz’s rules
For Carlstein – They’re interested in minimizing the MSE of the variance
For Künsch – They’re interested in minimizing the MSE of the bias
They argue that the rules are similar both for Künsch’s moving block scheme and
For Carlstein’s non-overlapping block scheme
Carlstein’s Model
𝒍 = 𝒏𝟏 𝟑 𝝆−𝟐/𝟑
𝜌=
𝛾 0 +2
∞
𝑗=1 𝑗
∞
𝑗=1 𝛾(𝑗)
∙ 𝛾(𝑗)
Künsch’s Model
𝒍 = 𝟑/𝟐 𝟏/𝟑 𝒏𝟏 𝟑 𝝆−𝟐/𝟑
γ(𝑗) –the covariance of 𝑥𝑡 at lag j
4. General error structures – The MBB
Optimal Length of Blocks
4. Hall and Horowitz’s rules
For the AR(1) process
𝑥𝑡 = 𝜑𝑥𝑡−1 + 𝜀𝑡
𝜌 = 1 − 𝜑2
For the MA(1) process
𝑥𝑡 = 𝜀𝑡 + 𝜃𝜀𝑡−1
𝜑
Computed 𝒍∗ for n=200
𝜌 = 1 + 𝜑2
𝜑