Problem Sheet VI
1. Let X1 , X2 , . . . be i.i.d. U(0, M ) random variables. Derive M̂ , the maximum likelihood estimator
of M . What can you say about the bias as n → ∞? Use Chebyshev’s inequality to prove that it is
consistent.
Hint 1: Use the notes.
Hint 2: Use the triangle inequality |x + y| ≤ |x| + |y|.
Hint 3: A more user friendly version of Chebyshev’s inequality is
P [|X − E X| ≥ ] ≤
var(X)
.
2
2. Let X1 , . . . Xn be i.i.d. Poisson(λ), that is
P(Xi = k) =
e−λ λk
,
k!
k = 0, 1, 2, . . . ,
i = 1, . . . , n.
Find the maximum likelihood estimator λ̂ of λ. Is it unbiased? Is it efficient? Is it consistent?
3. Let X1 , . . . , Xn be i.i.d. with p.d.f.
1
f (x; θ) = (1 + θx),
2
x ∈ (−1, 1), θ ∈ (−1, 1).
Find a consistent estimator of θ and show it is consistent.
4. Let X1 , . . . , Xn be i.i.d. Bernoulli random variables with P(X1 = 1) = p. Find the maximum
likelihood estimator p̂M LE of p. Show that it is consistent, efficient and sufficient.
1
Solutions VI
1. We have
1
1{X1 , . . . , Xn < M }
Mn
1
= n 1{max Xi < M }.
i
M
L(M ; x1 , . . . , xn ) =
This is clearly decreasing up for all M > maxi xi and vanishes otherwise, so the maximum occurs
at M̂n := maxi Xi . From the notes and lectures we know that
E M̂n =
so it is biased, and
n
M,
n+1
n
1
bias(M̂n ) = M
− 1 = −M
,
n+1
n+1
which vanishes as n → ∞.
Also we have the variance from the notes which we found to be
var(M̂n ) = M 2
n
.
(n + 2)(n + 1)2
Now to prove consistency, we cannot use Chebyshev’s inequality directly because of the bias. Let
> 0 be given. We have to show that
h
i
P |M̂n − M | ≥ → 0,
as n → ∞.
Then
h
i
h
i
P |M̂n − M | ≥ = P |M̂n − E M̂n + E M̂n − M | ≥ and applying the triangle inequality
h
i
≤ P |M̂n − E M̂n | + | E M̂n − M | ≥ h
i
= P |M̂n − E M̂n | + |Bias(M̂n )| ≥ but we know Bias(M̂n ) = −M/(n + 1) and thus
M
≥
n+1
M
= P |M̂n − E Mn | ≥ −
n+1
= P |M̂n − E M̂n | +
and applying Chebyshev’s inequality
≤
var(M̂n )
−
M
n+1
2
n
= M2
(n + 2)(n + 1)2 −
as n → ∞.
2
M
n+1
2 → 0
2. We compute
L(λ; x1 , . . . xn ) = e−nλ λnx̄ /(x1 ! · · · xn !)
l(λ; x1 , . . . , xn ) = −nλ + nx̄ log(λ) + constants
∂
nx̄
l(λ; x1 , . . . , xn ) = −n +
;
∂λ
λ
∂2
nx̄
l(λ; x1 , . . . , xn ) = − 2 .
2
∂λ
λ
Equating the first derivative to 0 we find
λ̂ = x̄.
This is of course unbiased, since E Xi = λ, and its variance is
λ
var(X1 )
= ,
n
n
var(λ̂) =
since the variance is also λ.
To see why the variance is indeed λ
var(X) =
=
=
=
∞
X
k2
k=0
∞
X
k
k=1
∞
X
λk e−λ
k!
λk e−λ
(k − 1)!
(k − 1 + 1)
k=1
∞
X
(k − 1)
k=2
∞
X
λk e−λ
(k − 1)!
∞
X
λk e−λ
λk e−λ
+
(k − 1)! k=1 (k − 1)!
∞
X
λk e−λ
λk e−λ
+
k
=
(k − 2)! k=0
k!
k=2
= λ2
∞
X
λk e−λ
k=0
k!
+ EX
= λ2 + λ,
var(X) = λ2 + λ − λ2 = λ.
To find Fischer’s information we use the second derivative formula
"
#
n
∂2
X
l(λ; X) = n E 2 = .
In (λ) = −n E
2
∂λ
λ
λ
It is clear the estimator is efficient.
Consistency follows easily from the law of large numbers.
3. Here maximum likelihood can run into problems. However a quick calculation shows that
1 1
EX =
x(1 + θx)dx
2 −1
Z
1 1 2
=
θx d x
2 −1
Z
3
θ h x3 i1
dx
2 3 −1
θ2
θ
=
dx = .
23
3
=
Therefore θ̂ = 3X̄ is consistent, since X̄ is consistent for θ/3.
4. The likelihood is
L(p; x1 , . . . , xn ) =
l(p; x1 , . . . , xn ) =
∂
l(p; x1 , . . . , xn ) =
∂p
∂2
∂p2
n
Y
i=1
n
X
pxi (1 − p)1−xi
xi log p + (1 − xi ) log(1 − p);
i=1
n X
i=1
l(p; x1 , . . . , xn ) = −
xi 1 − xi
;
−
p
1−p
n X
xi
i=1
1 − xi
+
.
p2 (1 − p)2
To find the MLE we set
n
X
∂
xi 1 − xi
l(p; x1 , . . . , xn ) =
−
= 0,
∂p
p
1−p
i=1
where we find that
nx̄
n − nx̄
=
,
p
p
which is solved by p̂ = x̄.
This is of course consistent for p by the LLN since
X̄ =
X1 + · · · + Xn
,
n
and E[X1 ] = p.
To see it is efficient
"
#
∂2
In (θ) = −n E
l(p; X)
∂p2
Xi
1 − Xi
= nE 2 +
p
(1 − p)2
E Xi 1 − E Xi
=n
+
p2
(1 − p)2
1
1
1−p+p
n
=n
+
=n
=
.
p (1 − p)
p(1 − p)
p(1 − p)
On the other hand the variance is
var(X̄) =
var(X1 )
p(1 − p)
1
=
=
,
n
n
In (θ)
so it is efficient.
We know efficient estimators are sufficient so we are done.
4