Chapter 4
Properties of the Least Squares
Estimators
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 1
Chapter Contents
 4.1 The Least Squares Estimators as Random
Variables
 4.2 The Sampling Properites of the Least Squares
Estimators
 4.3 The Gauss-Markov Theorem
 4.4 The Probability Distributions of the Least
Squares Estimators
 4.5 Estimating the Variance of the Error Term
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 2
4.1
The Least Squares Estimators as
Random Variables
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 3
4.1
The Least Squares
Estimators as
Random Variables
We repeat assumptions SR1-SR6 for easy refference
•
•
•
•
•
SR1. yt=β1+β2xt+et
SR2. E(et)=0⇔E(yt)=β1+β2xt
SR3. var(et)=σ2=var(yt)
SR4. cov(ei,ej)=cov(yi,yj)=0
SR5. xt is not random and must take at least two different
values.
• SR6. et~N(0,σ2)⇔yt~N[(β1+β2xt),σ2](optional)
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 4
4.1
The Least Squares
Estimators as
Random Variables
 In this Chapter, based on assumptions SR1-SR6,
we investigate the statistical properties of the least
squares estimators, which are procedures for
obtaining estimates of the unknown parameters β1
and β2 in the simple linear regression model.
 In this context b1 and b2 are random variables. The
properties of the least squares estimation
procedures we establish in this chapter do not
depend on any particular sample of data collection
or analysis.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 5
4.1
The Least Squares
Estimators as
Random Variables
 After the data are collected, the least squares
estimates are calculated numbers, such as
b2=0.1283, from the previous chapter. In “postdata” analysis, nonrandom quantities such as this
have no statistical properties. Their reliability and
usefulness are assessed in terms of the properties
of the procedures by which they were obtained.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 6
4.1
The Least Squares
Estimators as
Random Variables
 Two questions we will investigate in Chapter 4
1. If the least squares estimators b1 and b2 are random
variables, then what are their means, variances, covariances,
and probability distributions?
2. The least squares principle is only one way of using the
data to obtain estimates of β1 and β2. How do the least squares
estimators compare alternative estimators compare with other
rules that might be used, and how can we compare alternative
estimators?
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 7
4.2
The Sampling Properties of the Least
Squares Estimators
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 8
4.2
The Sampling
Properties of the
Least Squares
Estimators
 The least squares b1 and b2 estimators are random
variables and that have probability distributions
that we can study prior to the collection of and
data.
 These “pre-data” characteristics of b1 and b2 are
called sampling properties, because the
randomness of the estimators is brought on by
sampling from a population.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 10
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
We will show that if our model assumptions hold,
then E(b2) = β2, which means that the estimator is
unbiased. We can find the expected value of b2
using the fact that the expected value of a sum is the
sum of the expected values:
E(b2)=E(β2+∑wtet)=E(β2+w1e1+w2e2+…+wTeT)
=E(β2)+E(w1e1)+E(w2e2)+…+E(wTeT)
=E(β2)+∑E(wtet)
=β2+∑wtE(et)
=β2
(4.2.3)
using E(et)=0 and ∑wtE(et)=0
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
 The property of unbiasedness is about the average
values of b1 and b2 if many samples of the same
size are drawn from the same population.
 If we took the averages od estimates from many
samples, these averages would approach the true
parameter values β1 and β2.
 Unbiasedness does not say that an estimate from
any one samples is close to the true parameter
value, and thus we can not say that an estimate is
unbiased. We can sat that the least squares
estimation procedure is unbiased.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.1
The Expected
Values of
b1 And b2
Table 4.1 Least Squares Estimates from 10 Random Samples
of Size T=40
n
1
2
3
4
5
6
7
8
9
10
Undergraduated Econometrics
b1
51.1314
61.2045
40.7882
80.1396
31.0110
54.3099
69.6749
71.1541
18.8290
36.1433
Chapter 4: Properties of the Least Squares Estimators
b2
0.1442
0.1286
0.1417
0.0886
0.1669
0.1086
0.1003
0.1009
0.1758
0.1626
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.2
The Variances
And Covariance of
b1 And b2
 Given the expected values, or means, of b1 and b2 ,
the variance of b2 is defined as
var(b2)=E[b2-E(b2)]2
It measures the spread of b2 the probability
distribution of .
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.2
The Variances
And Covariance of
b1 And b2
 In figure 4.1 are graphs of two possible probability
distributions of b2, f1(b2) and f2(b2), that have the
same mean value but different variances.
Figure 4.1
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.2
The Variances
And Covariance of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.2
The Variances
And Covariance of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.2
The Variances
And Covariance of
b1 And b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
Figure 2.11 The influence of variation in the explanatory variable x on
precision of estimation (a) Low x variation, low precision (b) High x
variation, high precision
4.2.2
The Variances
And Covariance of
b1 And b2
Undergraduated Econometrics
2
The variance of b2 is defined as var( b2 )  Eb2  E (b2 )
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.2
The Sampling
Properties of the
Least Squares
Estimators
4.2.3
Linear Estimators
 The least squares estimator b2 is a weighted sum of the
observation yt, b2= ∑ wtyt (see 4.2.8). In mathematics
weighted sums like this are called linear combinations of
the yt; consequently, statisticians call estimators like b2,
that are linear combinations of an observable random
variable, linear estimators.
 We can describe b2 as a linear, unbiased estimator of β2,
with a variance given in (4.2.10), Similarly, b1 can be
described as a linear, unbiased estimator of β1, with a
variance given in (4.2.10)
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.3
The Gauss-Markov Theorem
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 8
4.3
The Gauss-Markov
Theorem
GAUSS-MARKOV THEOREM
Under the assumptions SR1-SR5 of the linear
regression model, the estimators b1 and b2 have the
smallest variance of all linear and unbiased
estimators of b1 and b2. They are the Best Linear
Unbiased Estimators (BLUE) of b1 and b2
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.3
The Gauss-Markov
Theorem
MAJOR POINTS ABOUT THE GAUSS-MARKOV THEOREM
1. The estimators b1 and b2 are “best” when compared to similar
estimators, those which are linear and unbiased. The Theorem does
not say that b1 and b2 are the best of all possible estimators.
2. The estimators b1 and b2 are best within their class because they
have the minimum variance. When comparing two linear and
unbiased estimators, we always want to use the one with the
smaller variance, since that estimation rule gives us the higher
probability of obtaining an estimate that is close to the true
parameter value.
3. In order for the Gauss-Markov Theorem to hold, assumptions SR1SR5 must be true. If any of these assumptions are not true, then b1
and b2 are not the best linear unbiased estimators of β1 and β2.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.3
The Gauss-Markov
Theorem
MAJOR POINTS ABOUT THE GAUSS-MARKOV THEOREM
4.
The Gauss-Markov Theorem does not depend on the assumption
of normality (assumption SR6).
5.
In the simple linear regression model, if we want to use a linear
and unbiased estimator, then we have to do no more searching.
The estimators b1 and b2 are the ones to use. This explains why
we are studying these estimators and why they are so widely used
in research, not only in economics but in all social and physical
sciences as well.
6.
The Gauss-Markov theorem applies to the least squares
estimators. It does not apply to the least squares estimates from a
single sample.
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.4
The Probability Distributions of the
Least Squares
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 8
4.4
The Probability
Distributions of the
Least Squares
If we make the normality assumption (assumption
SR6 about the error term) then the least squares
estimators are normally distributed:

σ 2  xi2 

b1 ~ N  β1 ,
2
 N  x  x  
i




σ2

b2 ~ N  β 2 ,
2
  x  x  
i


Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
（4.4.1）
Page 9
4.4
The Probability
Distributions of the
Least Squares
If assumptions SR1-SR5 hold, and if the sample
size N is sufficiently large, then the least squares
estimators have a distribution that approximates the
normal distributions shown in Eq. 2.17 and Eq. 2.18
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the Variance of the Error
Term
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 8
4.5
Estimating the
Variance of the
Error Term
The variance of the random error ei is:
var(ei )  σ 2  E[ei  E (ei )]2  E (ei ) 2
(4.5.1)
if the assumption E(ei) = 0 is correct.
Since the “expectation” is an average value we might
consider estimating σ2 as the average of the
squared errors:
σ̂ 2 
2
e
i
(4.5.2)
N
where the error terms are ei  yi  β1  β 2 xi
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the
Variance of the
Error Term
The least squares residuals are obtained by replacing
the unknown parameters by their least squares
estimates:
ê𝑡 =yt-b1-b2xt
(4.5.3)
There is a simple modification that produces an
unbiased estimator, and that is:
so that:
Undergraduated Econometrics
 
E σ̂ 2  σ 2
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the
Variance of the
Error Term
4.5.1
Estimating the
Variance and
Covariance of the
Least Squares
Estimators
Replace the unknown error variance σ2 in (4.2.40)
by its estimator to obtain:
(4.5.6)
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the
Variance of the
Error Term
Table 4.2 Least Squares Residuals for Food Expenditure Data
4.5.2
The Estimated
Variance and
Covariance for the
Food Expenditure
Example
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the
Variance of the
Error Term
4.5.2
The Estimated
Variance and
Covariance for the
Food Expenditure
Example
The estimated variances, covariance
corresponding standard errors are
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
and
4.5
Estimating the
Variance of the
Error Term
Table 4.3 EViews Regression Output
4.5.3
Sample Computer
Output
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9
4.5
Estimating the
Variance of the
Error Term
Table 4.4 SAS Regression Output
4.5.3
Sample Computer
Output
Undergraduated Econometrics
Chapter 4: Properties of the Least Squares Estimators
Page 9