STA301 – Statistics and Probability
Lecture No 34:
• Desirable Qualities of a Good Point Estimator:

Efficiency
Methods of Point Estimation:

The Method of Moments

The Method of Least Squares

The Method of Maximum Likelihood
•
Interval Estimation:

Confidence Interval for 
The students will recall that, in the last lecture, we presented the basic definition of a point estimator, and considered
some desirable qualities of a good point estimator.
Three main qualities of a good point estimator are:
DESIRABLE QUALITIES OF A GOOD POINT ESTIMATOR:
•



unbiasedness
consistency
efficiency
In the last lecture, we discussed in some detail the concepts of unbiasedness and consistency. Why is consistency
considered a desirable quality?
In order to obtain an idea regarding the answer to this question, consider the following:
As a sample is only a part of the population, it is obvious that the larger the sample size, the more
representative we expect it to be of the population from which it has been drawn. In agreement with the above
argument, we will expect our estimator to be close to the corresponding parameter if the sample size is large. Hence,
we will naturally be happy if the probability of our estimator being close to the parameter increases with an increase in
the sample size.As such, consistency is a desirable property.
Another important desirable quality of a good point estimator is EFFICIENCY:
EFFICIENCY:
An unbiased estimator is defined to be efficient if the variance of its sampling distribution is smaller than that of the
sampling distribution of any other unbiased estimator of the same parameter
In other words, suppose that there are two unbiased estimators T1 and T2 of the same parameter.
Then, the estimator T1 will be said to be more efficient than T2 if Var (T1) < Var (T2).
In the following diagram, since Var (T1) < Var (T2),
hence T1 is more efficient than T2 :
Sampling
Distribution of T1
Sampling
Distribution of T2
The relative efficiency of T1 compared to T2 (where both T1 and T2 are unbiased estimators) is given by the ratio
Ef 
Var T2 
.
Var T1 
And, if we multiply the above expression by 100, we obtain the relative efficiency in percentage form. It thus provides
a criterion for comparing different unbiased estimators of a parameter. Both the sample mean and the sample median
Virtual University of Pakistan
Page 272
STA301 – Statistics and Probability
for a population that has a normal distribution, are unbiased and consistent estimators of  but the variance of the
sampling distribution of sample means is smaller than the variance of the sampling distribution of sample medians.
Hence, the sample mean is more efficient than the sample median as an estimator of .The sample mean may therefore
be preferred as an estimator.
Next, we consider various methods of point estimation. A point estimator of a parameter can be obtained by several
methods.
We shall be presenting a brief account of the following three methods:
METHODS OF POINTESTIMATION:

The Method of Moments

The Method of Least Squares

The Method of Maximum Likelihood
These methods give estimates which may differ as the methods are based on different theories of estimation.
We begin with the Method of Moments:
THE METHOD OF MOMENTS
The method of moments which is due to Karl Pearson (1857-1936), consists of calculating a few moments of
the sample values and equating them to the corresponding moments of a population, thus getting as many equations as
are needed to solve for the unknown parameters.
The procedure is described below:
Let X1, X2, …, Xn be a random sample of size n from a population. Then the rth sample moment about zero is
 X ir
m' r 
, r  1,2,...
n
and the corresponding rth population moment is
We then match these moments and get as many equations as we
 'r .
need to solve for the unknown parameters.
The following examples illustrate the method:
EXAMPLE-1:
Let X be uniformly distributed on the interval (0, ). Find an estimator of  by the method of moments.
SOLUTION
The probability density function of the given uniform distribution is
f x  
1

, 0  x 
Since the uniform distribution has only one parameter, (i.e. ), therefore, in order to find the maximum likelihood
estimator of  by the method of moments, we need to consider only one equation.
The first sample moment about zero is
 Xi
.
And, the first population momentnabout zero is
m'1 

2

 1
1 x

'1   x.f x dx   x. dx    
 2  2
0 moments, we obtain:
0 
Matching these
0
 Xi 
 or   2X.
n
2
Hence, the moment estimator of  is equal to
i.e.
ˆ  2X.
2X
In other words, the moment estimator of  is just twice the sample mean.
An Interesting Observation:
It should be noted that, for the above uniform distribution, the mean is given by
Virtual University of Pakistan
Page 273
STA301 – Statistics and Probability


2
.
(This is so due to the absolute symmetry of the uniform distribution around the value
Now,


implies that

.)
2
  2 .
2
In other words, if we wish to have the exact value of , all we need to do is to multiply the population mean  by 2.
Generally, it is not possible to determine, and all we can do is to draw a sample from the probability distribution, and
compute the sample mean X.
Hence, naturally, the equation will be replaced by the equation
(As 2 x provides an estimate of , hence a ‘hat’ is placed on top of .)
It is interesting to note
that 2
isxexactly the same quantity as what we obtained as an estimate of  by the method
of moments!(The result obtained by the method of moments coincides with what we obtain through simple logic
EXAMPLE-2:
Let X1, X2, …, Xn be a random sample of size n from a normal population with parameters  and 2. Find
these parameters by the method of moments.
SOLUTION
Here
we
need
two
equations
as
there
are
two
unknown
parameters,
 and 2.
The first two sample moments about zero are
m'1 
1
1
X i  X and m' 2   X i2 .

n
n
The corresponding two moments of a normal distribution are
1 =  and 2 = 2 + 2.
= 2 – 12
= 2 – 2 )
To get the desired estimators by the method of moments, we match them.
Thus, we have :
(

2
1
1
X i and  2   2   X i2

n
n
Solving the above equations simultaneously, we obtain:
1
 X i  X, and
n
2
 X i  X 2  1 X  X 2  S2 .
ˆ 2 
 i
n
n
ˆ 
as the moment estimators for  and 2.
A shortcoming of this method is that the moment estimators are, in general, inefficient.
Next, we consider the Method of Least Squares:
The method of Least Squares, which is due to Gauss (1777-1855) and Markov (1856-1922), is based on the theory of
linear estimation.
It is regarded as one of the important methods of point estimation.
THE METHOD OFLEAST SQUARES:
An estimator found by minimizing the sum of squared deviations of the sample values from some function
that has been hypothesized as a fit for the data, is called the least squares estimator.
The method of least-squares has already been discussed in connection with regression analysis that was presented in
Lecture No. 15.
You will recall that, when fitting a straight line y = a+bx to real data, ‘a’ and ‘b’ were determined by minimizing the
sum of squared deviations between the fitted line and the data-points.
Virtual University of Pakistan
Page 274
STA301 – Statistics and Probability
The y-intercept and the slope of the fitted line i.e. ‘a’ and ‘b’ are least-square estimates (respectively) of the y-intercept
and the slope of the TRUE line that would have been obtained by considering the entire population of data-points, and
not just a sample.
Next, we consider the Method of Maximum Likelihood:
The method of maximum likelihood is regarded as the MOST important method of estimation, and is the most widely
used method. This method was introduced in 1922 by Sir Ronald A. Fisher (1890-1962).The mathematical technique of
finding Maximum Likelihood Estimators is a bit advanced, and involves the concept of the Likelihood Function.
In this course, we consider only the overall logic of this method:
RATIONALE OF THE METHOD OF MAXIMUM LIKELIHOOD (ML):
“To consider every possible value that the parameter might have, and for each value, compute the probability
that the given sample would have occurred if that were the true value of the parameter.
That value of the parameter for which the probability of a given sample is greatest, is chosen as an estimate.”
An estimate obtained by this method is called the maximum likelihood estimate (MLE).
It should be noted that the method of maximum likelihood is applicable to both discrete and continuous random
variables.
Let us consider a few examples:
EXAMPLES OF MLE’s IN CASE OF DISCRETE DISTRIBUTIONS
Example-1:
For the Poisson distribution given by
e -  x
, x  0,1,2, ......,
x!
the MLE of  is X (the sample mean).
P(X  x) 
Example-2:
For the geometric distribution given by
the MLE of p is
Hence, the MLE of p is equal to the reciprocal of the mean.
Example-3:
For the Bernoulli distribution given by
P(X  x)  p x q1 x , x  0,1 ,
the MLE of p is
(the sample mean).
EXAMPLES OF MLE’s IN CASE OF CONTINUOUS DISTRIBUTIONS
Example-1:
For the exponential distribution given by
f x   e x , x  0,   0 ,
1
the MLE of  is
.
X
(the reciprocal of the sample mean).
Example-2:
For the normal distribution with parameters  and 2,
the joint ML estimators of  and 2 are the sample mean
and the sample variance S2(which is not an unbiased estimator of 2).
As indicated many times earlier, the normal distribution is encountered frequently in practice, and, in this regard, it is
both interesting and important to note that, in the case of this frequently encountered distribution, the simplest formulae
(i.e. the sample mean and the sample variance) fulfill the criteria of the relatively advanced method of maximum
likelihood estimation !The last example among the five presented above (the one on the normal distribution) points to
another important fact --- and that is :
The Maximum Likelihood Estimators are consistent and efficient but not necessarily unbiased. (As we know, S2 is not
an unbiased estimator of 2.)
Let us apply this concept to a real-life example:
EXAMPLE:
It is well-known that human weight is an approximately normally distributed variable.
Virtual University of Pakistan
Page 275
STA301 – Statistics and Probability
Suppose that we are interested in estimating the mean and the variance of the weights of adult males in one
particular province of a country.
A random sample of 15 adult males from this particular population yields the following weights (in pounds):
131.5
136.9
133.8
130.1
133.9
135.2
129.6
134.4
130.5
134.2
131.6
136.7
135.8
134.5
132.7
Find the maximum likelihood estimates for 1 =  and 2 = 2.
SOLUTION:
The above data is that of a random sample of size 15 from N(, 2).
It has been mathematically proved that the joint maximum likelihood estimators of  and 2 areX and S2.
We compute these quantities for this particular sample, and obtain
X = 133.43, and S2 = 5.10 .
These are the Maximum Likelihood Estimates of the mean and variance of the population of weights in this
particular example.
Having discussed the concept of point estimation in some detail, we now begin the discussion of the concept of interval
estimation:
As stated earlier, whenever a single quantity computed from the sample acts as an estimate of a population parameter,
we call that quantity a point estimate e.g. the sample mean is a point estimate of the population mean .
The limitation of point estimation is that we have no way of ascertaining how close our point estimate is to the true
value (the parameter).
For example, we know that is an unbiased estimator of  i.e. if we had taken all possible samples of a particular size
from the population and calculated the mean of each sample, then the mean of the sample means
would have been
equal to the population mean (), but in an actual survey we will be selecting only one sample from the population and
will calculate its mean .
We will have no way of ascertaining how close this particular is to .
Whereas a point estimate is a single value that acts as an estimate of the population parameter, interval estimation is a
procedure of estimating the unknown parameter which specifies a range of values within which the parameter is
expected to lie.
A confidence interval is an interval computed from the sample observations x1, x2….xn, with a statement of how
confident we are that the interval does contain the population parameter.
We develop the concept of interval estimation with the help of the example of the Ministry of Transport test to which
all cars, irrespective of age, have to be submitted :
EXAMPLE:
Let us examine the case of an annual Ministry of Transport test to which all cars, irrespective of age, have to
be submitted. The test looks for faulty breaks, steering, lights and suspension, and it is discovered after the first year
that approximately the same number of cars has 0, 1, 2, 3, or 4 faults. You will recall that when we drew all possible
samples of size 2 from this uniformly distributed population, the sampling distribution of X was triangular:
Sampling Distribution ofX for n = 2
P x 
5/25
4/25
3/25
2/25
1/25
0
X
0.
0
0.
5
1.
0
1.
5
2.
0
2.
5
3.
0
3.
5
4.
0
But when we considered what happened to the shape of the sampling distribution with if the sample size is increased,
we found that it was somewhat like a normal distribution:
Virtual University of Pakistan
Page 276
STA301 – Statistics and Probability
Sampling Distribution ofX for n = 3
P x 
20/125
16/125
12/125
8/125
4/125
X
0
0. 0. 0. 1. 1. 1. 2. 2. 2. 3. 3. 3. 4.
00 33 67 00 33 67 00 33 67 00 33 67 00
And, when we increased the sample size to 4, the sampling distribution resembled a normal distribution even more
closely :
Sampling Distribution ofX for n = 4
P x 
100/625
80/62
5
60/62
5
40/62
5
20/62
5
0
X
0. 0. 0. 0. 1. 1. 1. 1. 2. 2. 2. 2. 3. 3. 3. 3. 4.
00 25 50 75 00 25 50 75 00 25 50 75 00 25 50 75 00
It is clear from the above discussion that as larger samples are taken, the shape of the sampling distribution of X
undergoes discernible changes.
In all three cases the line charts are symmetrical, but as the sample size increases, the overall configuration
changed from a triangular distribution to a bell-shaped distribution.
x
In other words, for large samples, we are dealing with a normal sampling distribution of
.In other words:
When sampling from an infinite population such that the sample size n is large,X is normally distributed with mean 
and variance  2
n
i.e. X is
 2
.
N   ,
n 

Hence, the standardized version of X i.e.
X 
Z

n
Virtual University of Pakistan
Page 277
STA301 – Statistics and Probability
is normally distributed with mean 0 and variance 1 i.e.
Z is N(0, 1).
Now, for the standard normal distribution, we have:
For the standard normal distribution, we have:
0.4750
0.0250
0.4750
0.0250
Z
-1.96
0
1.96
The above is equivalent to
P(-1.96 < Z < 1.96)
= 0.4750 + 0.4750 = 0.95
0.95
0.025
0.025
Z
-1.96
0
1.96
In other words:




X 

P  1.96 
 1.96   0.95



n


The above can be re-written as:


 
P  1.96
 X    1.96
 0.95
n
n 

Or


 
P  X  1.96
     X  1.96
  0.95
n
n


or


 
P X  1.96
   X  1.96
 0.95
n
n 

Virtual University of Pakistan
Page 278
STA301 – Statistics and Probability
or


 
P X  1.96
   X  1.96
 0.95
n
n 

The above equation yields the 95% confidence interval for  :
The 95% confidence interval for  is


 X  1.96  , X  1.96   .

n
n 

In other words, the 95% C.I. for  is given by
X  1.96

n
In a real-life situation, the population standard deviation is usually not known and hence it has to be estimated.
It can be mathematically proved that the quantity
 X  X 
s 
2
2
n 1
is an unbiased estimator of 2 (the population variance).
(just as the sample mean is an unbiased estimator of ).
In this situation, the 95% Confidence Interval for  is given by:
s
s 

P X  1.96
   X  1.96
  95%
n
n

The points
X  1.96
s
s
and X  1.96
n
n
are called the lower and upper limits of the 95% confidence interval.
Virtual University of Pakistan
Page 279