Probability and statistics
May 10, 2016
Lecturer: Prof. Dr. Mete SONER
Coordinator: Yilin WANG
Solution Series 9
Q1. Let α, β > 0, the p.d.f. of a beta distribution with parameters α and β is
(
Γ(α+β) α−1
x (1 − x)β−1 for 0 < x < 1,
f (x|α, β) = Γ(α)Γ(β)
0
otherwise.
Suppose that X1 , · · · , Xn form a random sample from the Bernoulli distribution with parameter θ, which is unknown (0 < θ < 1). Suppose also that the prior distribution of θ is the
beta distribution with parameters α > 0 and β > 0. Show that the posterior distribution
of
Pn
x
and
θ given that
X
=
x
(i
=
1,
.
.
.
,
n)
is
the
beta
distribution
with
parameters
α
+
i
i
i=1 i
P
β + n − ni=1 xi .
In particular the family of beta distributions is a conjugate family of prior distributions for
samples from a Bernoulli distribution. If the prior distribution of θ is a beta distribution,
then the posterior distribution at each stage of sampling will also be a beta distribution,
regardless of the observed values in the sample.
Solution:
First we calculate the joint p.f. of X1 , · · · , Xn , θ:
fX1 ,··· ,Xn ,θ (x1 , · · · , xn , x) = xy (1 − x)n−y
=
Γ(α + β) α−1
x (1 − x)β−1
Γ(α)Γ(β)
Γ(α + β) α+y−1
x
(1 − x)β+n−y−1
Γ(α)Γ(β)
where y = x1 + · · · + xn . So that the marginal p.f. of X1 , · · · , Xn at (x1 , · · · , xn ) is
Z 1
Γ(α + β) α+y−1
Γ(α + β) Γ(α + y)Γ(β + n − y)
x
(1 − x)β+n−y−1 dx =
.
Γ(α)Γ(β)
Γ(α + β + n)
0 Γ(α)Γ(β)
Thus the conditional p.d.f. of θ given x1 , · · · , xn is
fθ|x1 ,··· ,xn (x) =
Γ(α + β + n)
xα+y−1 (1 − x)β+n−y−1 ,
Γ(α + y)Γ(β + n − y)
which is the Beta distribution with parameters α + y and β + n − y.
1
Probability and statistics
May 10, 2016
Q2. Let ξ(θ) be defined as follows: for constants α > 0 and β > 0:
( α
β
θ−(α+1) e−β/θ for θ > 0,
ξ(θ) = Γ(α)
0
for θ ≤ 0.
A distribution with this p.d.f. is called an inverse gamma distribution.
(a) Verify that ξ(θ) is actually a p.d.f..
(b) Consider the family of probability distributions that can be represented by a p.d.f. ξ(θ)
having the given form for all possible pairs of constants α > 0 and β > 0. Show that this
family is a conjugate family of prior distributions for samples from a normal distribution
with a known value of the mean µ and an unknown value of the variance θ.
Solution:
(a) The integral of ξ is
∞
β α −(α+1) −β/θ
θ
e
dθ
Γ(α)
0
Z 0
βα
β
=
− 2 β −(α+1) y α+1 e−y dy
Γ(α) ∞ y
Z ∞
1
y α−1 e−y dy
=
Γ(α) 0
=1
Z
(b) As we did before, let Θ be r.v. following the inverse gamma distribution with parameter
α and β, let X be a random variable such that the conditional p.d.f. given Θ = θ is
1
2
e−(x−µ) /2θ .
2πθ
The conditional p.d.f. of Θ given X = x is
fX|θ (x) = √
ξ(θ)fX|θ (x)
fΘ|x (θ) = R ∞
ξ(θ)fX|θ (x)dθ
0
ξ(θ)fX|θ (x)
= βα R ∞
√
θ−(α+1/2)−1 e−(β+(x−µ)2 /2)/θ dθ
Γ(α) 2π 0
=
2
βα
√ θ −(α+1/2)−1 e−(β+(x−µ) /2)/θ
Γ(α) 2π
Γ(α+1/2)
βα
√
Γ(α) 2π [β+(x−µ)2 /2]α+1/2
0
[β 0 ]α −α0 −1 −β 0 /θ
=
θ
e
.
Γ(α0 )
Where we set α0 = α + 1/2 and β 0 = β + (x − µ)2 /2. The conditional distribution of
Θ given X = x is an inverse Gamma distribution with parameter α0 and β 0 . Thus the
family of inverse Gamma distribution is a family of prior distributions for samples from
a normal distribution with a known mean µ and an unknown value of the variance.
2
Probability and statistics
May 10, 2016
Q3. Suppose that the number of defects in a 1200-foot roll of magnetic recording tape has a
Poisson distribution for which the value of the mean θ is unknown, and the prior distribution
of θ is the gamma distribution with parameters α = 3 and β = 1. When five rolls of this tape
are selected at random and inspected, the numbers of defects found on the rolls are 2, 2, 6, 0
and 3.
(a) What is the posterior distribution of θ?
(b) If the squared error loss function is used, what is the Bayes estimate of θ?
Solution:
(a) Recall that X follows the Poisson distribution with mean θ then
P(X = k) = e−θ
θk
.
k!
And θ has the p.d.f.
fθ (x) =
x3−1 e−x
= x2 e−x /2.
Γ(3)
The conditional p.d.f. of θ is obtained as before:
x2 e−x /2 · e−5x x2+2+6+0+3 /(2!2!6!0!3!)
fθ|2,2,6,0,3 (x) = R ∞ 2 −s
s e /2 · e−5s s2+2+6+0+3 /(2!2!6!0!3!)ds
0
e−6x x15
= R ∞ −6s 15
e s ds
0
Γ(16)
= e−6x x15 16 := ξ(x|2, 2, 6, 0, 3)
6
Remark also that the posterior distribution of θ is the Gamma distribution with parameter (α = 16, β = 6).
(b) The squared error loss function is
L(θ, a) = (θ − a)2 .
The Bayes estimator of θ is the a which minimizes
Z
E[L(θ, a)|observation] = L(x, a)ξ(x|observation)dx,
where ξ(θ|observation) is the posterior p.d.f. of θ given the observation.
We have computed the k-th moments of Gamma distribution X with parameter α and
β in previous series, in particular
E(X) =
α
,
β
3
E(X 2 ) =
α(α + 1)
.
β2
Probability and statistics
May 10, 2016
Hence
E[(θ − a)2 |2, 2, 6, 0, 3] = E[θ2 |2, 2, 6, 0, 3] − 2aE[θ|2, 2, 6, 0, 3] + a2
17 × 16
16
=
− 2a + a2
2
6
6
2
8
4
= a−
+ .
3
9
The Bayes estimate for the observation 2, 2, 6, 0, 3 is 8/3.
Q4. Let c > 0 and consider the loss function
(
c|θ − a|
L(θ, a) =
|θ − a|
if θ < a,
if θ ≥ a.
Assume that θ has a continuous distribution.
(a) Let a ≤ q be two real numbers, show that
E[L(θ, a) − L(θ, q)] ≥ (q − a)[P(θ ≥ q) − cP(θ ≤ q)],
and
E[L(θ, a) − L(θ, q)] ≤ (q − a)[P(a ≤ θ) − cP(θ ≤ a)].
(b) Prove that a Bayes estimator of θ will be any 1/(1 + c) quantile of the posterior distribution of θ.
Solution: Let θ follows a continuous distribution,
(a) let a ≤ q,
E[L(θ, a) − L(θ, q)] =E[(L(θ, a) − L(θ, q))(1θ≤a + 1a≤θ≤q + 1q≤θ )]
=E[c(a − θ − q + θ)1θ≤a ] + E[(θ − a − c(q − θ))1a≤θ≤q ]
+ E[(θ − a − θ + q)1q≤θ ]
=c(a − q)P(θ ≤ a) + (1 + c)E(θ1a≤θ≤q ) − (a + cq)P(a ≤ θ ≤ q)
+ (q − a)P(q ≤ θ)
≥(q − a)[P(q ≤ θ) − cP(θ ≤ a)] + a(1 + c)P(a ≤ θ ≤ q)
− (a + cq)P(a ≤ θ ≤ q)
=(q − a)[P(q ≤ θ) − cP(θ ≤ q)].
Similarly we have
E[L(θ, a) − L(θ, q)] ≤ (q − a)[P(a ≤ θ) − cP(θ ≤ a)].
4
Probability and statistics
May 10, 2016
(b) Let q be a 1/(1 + c)-quantile of θ. Then
P(θ ≤ q) = 1/(1 + c) and P(θ ≥ q) = c/(1 + c).
For all a ≤ q, by the first inequality,
E(L(θ, a) − L(θ, q)) ≥ 0.
For all a ≥ q, by the second inequality,
E(L(θ, a) − L(θ, q)) ≥ (q − a)[P(q ≤ θ) − cP(θ ≤ q)] = 0.
Thus q minimizes E[L(θ, a)] among all a ∈ R, is the Bayes estimator with loss function
L.
5