Ch. 3.5 Estimation – Continuous Case
• Example. Suppose the proportion of impurity X in an iron ore
specimen can be modelled with the probability density function
fX (x) = (α + 1)xα,
0≤x≤1
where α is an unknown parameter.
• Before measurements are taken, and α is estimated, it is impossible
to know which probability density function best models the impurity
measurements.
1
2
Likelihood
• An impurity measurement has been taken: x = .1348.
• The density is highest at values of x that are most probable.
• Of the 3 densities pictured, the most likely corresponds to α = −.62.
3
Likelihood
• We haven’t graphed all of the possible densities. Which one is really
the most likely to have resulted in x = .1348?
• i.e. Which density gives the largest value of fX (.1348)?
• Note that fX (.1348) is a function of α:
fX (.1348) = (α + 1).1348α
• This function measures the likelihood of the observed measurement:
L(α) = (α + 1).1348α
4
5
Likelihood
• We can estimate α by maximizing this likelihood.
• The maximum occurs near α = −.5.
• By setting the derivative of the likelihood to 0 and solving for α you
can obtain the maximum likelihood estimate.
6
Likelihood
• Another Example.
• Consider a uniform random variable X on the interval (0, θ], where
θ > 0.
7
1.0
Some U(0, theta) Densities
0.0
0.2
0.4
f (x)
0.6
0.8
θ=2
θ = 2.5
θ = 3.5
θ = 4.2
−1
0
1
2
3
4
5
x
8
Likelihood
• Suppose x = 3.4.
• What is the most likely value of θ to have resulted in that observation?
• We can see from the plot that any density function for which θ is less
than 3.4 will give 0 density at that point. The one that gives maximum
probability density among the ones graphed is the one for which
θ = 3.5.
• You can imagine that if we plotted the density function corresponding
to θ = 3.4, it would put slightly higher density on 3.4.
• The maximum likelihood estimator of θ is 3.4.
9
• The density function f (x) = 1/θI(0<x≤θ) takes different values for
different values of θ.
• For fixed x (i.e. x = 3.4), this is really a function of θ only
1
L(θ) = I(θ≥3.4)
θ
• L(θ) is the likelihood function of θ, given the data.
10
Likelihood
• We should get better parameter estimates if we have more than one
measurement.
• Iron Ore Example (Cont’d) A second independent impurity
measurement is .0381.
• The joint density function for 2 independent impurity measurements is
f (x1, x2) = fX (x1)fX (x2)
= (α + 1)2(x1x2)α
When this is evaluated at the 2 measurements, we have
L(α) = (α + 1)2(.00514)α
Again, we should choose the value of α that maximizes this.
11
• Code for plot below:
2
1
0
like (x)
3
like <- function(alpha) (alpha + 1)ˆ2*(.00514)ˆalpha
curve(like, -1, .5)
−1.0
−0.5
0.0
0.5
x
12
Likelihood
• It is equivalent to maximize the likelihood of L(α):
log L(α) = 2 log(α + 1) + α log(.00514)
b = −.62, so
The maximizer, found by differentiating and solving, is α
we might write
fX (x) = .38x−.62.
13
• Code for plot below:
−1
−2
−3
loglike (x)
0
1
loglike <- function(alpha) 2*log(alpha + 1) +
alpha*log(.00514)
curve(loglike, -1, .5)
−1.0
−0.5
0.0
0.5
x
14
Likelihood
• Suppose we have two independent uniform (0, θ] observations X1
and X2.
• The joint density is
1
f (x1, x2) = 2 I(0<x1≤θ)I(0<x2≤θ)
θ
• If x1 = 3.4 and x2 = 4.2, then the likelihood for θ becomes
1
L(θ) = 2 I(3.4≤θ)I(4.2≤θ)
θ
1
= 2 I(4.2≤θ)
θ
• This is maximized at 4.2. Thus, the maximum likelihood estimate is
θb = 4.2.
15
Likelihood
• A better estimate can be obtained with a larger sample. Here is a
sample of 10 independent measurements:
0.1348 0.0020 0.0216 0.4593 0.0177
0.0381 0.3911 0.6264 0.0286 0.0002
• The joint density evaluated at the measurements is
f (x1, x2, . . . , x10) = (α + 1)10(2.53 × 10−15)α
Again, this is a function of the unknown parameter α. We can take
logs of this likelihood function:
log L(α) = 10 log(α + 1) − 33.6α
• Maximizing this, we see that
b = −.702
α
16
Likelihood
• The likelihood function for independent measurements x1, . . . , xn
coming from a population modelled by a density f (y) is
L(θ) = f (x1)f (x2) · · · f (xn)
• θ denotes parameter(s) to be estimated.
17
Likelihood
• A final example.
• The likelihood function for n independent uniform (0, θ]
measurements x1, . . . , xn is
1
L(θ) = n I(θ≥max(x))
θ
• Thus, the maximum likelihood estimator for θ is the maximum value of
the sample. (The maximum order statistic.)
18