AMS 570
Professor Wei Zhu
*Review: The derivative and integral of some important functions one should
remember.
d k
( x )  kx k 1
dx
d x
(e )  e x
dx
d
1
(ln x) 
dx
x

b

b
a
a
x k dx  (
1 k 1 x  b
1
x )

(b k 1  a k 1 )
x  a k 1
k 1
e x dx  e x
xb
xa
 eb  e a
*The Chain Rule
d
g[ f ( x)]  g '[ f ( x)]  f '( x)
dx
d x2
x2
e

e
 2x
For example:
dx
*The Product Rule
d
[ g ( x)  f ( x)]  g '( x) f ( x)  g ( x) f '( x)
dx
*Review: MGF, its second function: The m.g.f. will also generate the moments
Moment:
1st (population) moment: E ( X )   x  f ( x)dx
2nd (population) moment: E ( X 2 )   x 2  f ( x)dx
…
Kth (population) moment: E ( X k )   x k  f ( x )dx
1
Take the Kth derivative of the M X (t ) with respect to t, and the set t = 0, we obtain the
Kth moment of X as follows:
d
M X (t )
 E( X )
t 0
dt
d2
M X (t )
 E( X 2 )
2
t 0
dt
...
dk
M X (t )
 E( X k )
k
t 0
dt
Note: The above general rules can be easily proven using calculus.
Example: When X ~ N ( ,  2 ) , we want to verify the above equations for k=1 & k=2.
1
 t   2t 2
d
M X (t )  (e 2 )  (    2t ) (using the chain rule)
dt
So when t=0
d
M X (t )
   E( X )
t 0
dt
d2
M X (t )
dt 2
d d
 [ M X (t )]
dt dt
1
 t   2t 2
d
 [(e 2 )  (    2t )]
dt
 (e
1
2
 t   2t 2
)  (    t )  (e
2
2
(using the result of the Product Rule)
1
2
 t   2t 2
)  2
And
d2
M X (t )
t0
dt 2
 2  2
Considering  2  Var ( X )  E ( X 2 )   2
 E( X 2 )
*** Joint distribution, and independence
Definition. The joint moment generating function of two random variables X and Y
is defined as 𝑀𝑋,𝑌 (𝑡1 , 𝑡2 ) = 𝐸(𝑒 𝑡1𝑋+𝑡2𝑌 )
2
Theorem. Two random variables X and Y are independent ⇔ (if and only if)
𝑓𝑋,𝑌 (𝑥, 𝑦) = 𝑓𝑋 (𝑥)𝑓𝑌 (𝑦) ⇔ 𝑀𝑋,𝑌 (𝑡1 , 𝑡2 ) = 𝑀𝑋 (𝑡1 ) 𝑀𝑌 (𝑡2 )
Definition. The covariance of two random variables X and Y is defined as
𝐶𝑂𝑉(𝑋, 𝑌) = 𝐸[(𝑋 − 𝜇𝑋 )(𝑌 − 𝜇𝑌 )].
Theorem. If two random variables X and Y are independent, then we have
𝐶𝑂𝑉(𝑋, 𝑌) = 0. (*Note: However, 𝐶𝑂𝑉(𝑋, 𝑌) = 0 does not necessarily mean that X
and Y are independent.)
Exercise
𝑄1. 𝐿𝑒𝑡 𝑋1 & 𝑋2 𝑏𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑁(𝜇, 𝜎 2 ). Prove that 𝑋1 + 𝑋2 and 𝑋1 − 𝑋2 are
independent in two approaches: (1) pdf, (2) mgf
Solution:
(1) The pdf approach: Let
Then we have
𝑊 = 𝑋1 + 𝑋2
𝑍 = 𝑋1 − 𝑋2
𝑊+𝑍
2
𝑊−𝑍
𝑋2 =
2
This is a 1-1 transformation between (X1, X2) and (W, Z).
𝑋1 =
Define the Jacobian J of the transformation as:
𝜕𝑥1 𝜕𝑥1
1 1
𝐽 = | 𝜕𝑤 𝜕𝑧 | = |2 2 | = −1/2
1
1
𝜕𝑥2 𝜕𝑥2
−
2
2
𝜕𝑤 𝜕𝑧
Then the joint distribution of the new random variables is given by:
𝑓𝑊,𝑍 (𝑤, 𝑧) = 𝑓𝑋1,𝑋2 (
𝑤+𝑧 𝑤−𝑧
,
) |𝐽|
2
2
Since 𝑋1 and 𝑋2 are independent, we have:
𝑓𝑋1 ,𝑋2 (𝑥1 , 𝑥2 ) = 𝑓𝑋1 (𝑥1 )𝑓𝑋2 (𝑥2 )
3
Thus we find:
𝑓𝑊,𝑍 (𝑤, 𝑧) = 𝑓𝑋1 (
=(
2
1
)
𝑤+𝑧
𝑤−𝑧
) 𝑓𝑋2 (
) |𝐽|
2
2
2
2
1 𝑤+𝑧
1 𝑤−𝑧
− 2(
−𝜇) − 2 (
−𝜇)
2
2
2σ
2σ
e
∗
1
2
√2𝜋𝜎 2
1
1
1 1
− 2 (𝑤−2𝜇)2 − 2 (𝑧)2
4σ
4σ
= (
)e
2 2𝜋𝜎 2
1
1
1
1
− 2 (𝑤−2𝜇)2
− 2𝑧 2
4σ
4σ
=
e
e
√2𝜋 × 2𝜎 2
√2𝜋 × 2𝜎 2
= 𝑓𝑍 (𝑧)𝑓𝑊 (𝑤)
So Z and W are independent. Furthermore we see that 𝑋1 + 𝑋2 ~𝑁(2𝜇, 2𝜎 2 ) and 𝑋1 −
𝑋2 ~𝑁(0, 2𝜎 2 )
(2)
The mgf approach: Now we p𝑟𝑜𝑣𝑒 𝑡ℎ𝑎𝑡
𝑴𝑿𝟏+𝑿𝟐 ,𝑿𝟏 −𝑿𝟐 (𝒕𝟏 , 𝒕𝟐 ) = 𝑴𝑿𝟏 +𝑿𝟐 (𝒕𝟏 )𝑴𝑿𝟏−𝑿𝟐 (𝒕𝟐 )
𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏:
𝐿𝑒𝑡 𝑊 = 𝑋1 + 𝑋2 𝑎𝑛𝑑 𝑍 = 𝑋1 − 𝑋2
𝑊 𝑎𝑛𝑑 𝑍 𝑎𝑟𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡 𝑖𝑓 𝑎𝑛𝑑 𝑜𝑛𝑙𝑦 𝑖𝑓 𝑀𝑊,𝑍 (𝑡1 , 𝑡2 ) = 𝑀𝑊 (𝑡1 )𝑀𝑍 (𝑡2 )
𝑆𝑢𝑏𝑠𝑡𝑖𝑡𝑢𝑡𝑒 𝑓𝑜𝑟 𝑊 & 𝑍
𝑀𝑊,Z (𝑡1 , 𝑡2 ) = 𝐸(𝑒 𝑡1𝑊+𝑡2 𝑍 ) ⇒
𝐸(𝑒 𝑡1(𝑋1 +𝑋2 )+𝑡2(𝑋1 −𝑋2 ) )
= 𝐸(𝑒 (𝑡1+𝑡2 )𝑋1 +(𝑡1 −𝑡2 )𝑋2 )
𝐸(𝑒 (𝑡1+𝑡2)𝑋1 +(𝑡1−𝑡2)𝑋2 )
= 𝐸(𝑒 (𝑡1+𝑡2 )𝑋1 )𝐸(𝑒 (𝑡1−𝑡2)𝑋2 ) 𝑏𝑒𝑐𝑎𝑢𝑠𝑒 𝑋1 & 𝑋2 𝑎𝑟𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑡
∗ 𝐸(𝑒
(𝑡1 +𝑡2 )𝑋1
∗ 𝐸(𝑒
(𝑡1 −𝑡2 )𝑋2
∴ 𝐸(𝑒
)=
1
)+ (𝑡
)2 2
𝑒 𝜇(𝑡1+𝑡2 2 1 +𝑡2 𝜎
)=
1
𝜇(𝑡1 −𝑡2 )+ (𝑡1 −𝑡2 )2 𝜎 2
2
𝑒
(𝑡1 +𝑡2 )𝑋1 +(𝑡1 −𝑡2 )𝑋2
)=𝑒
𝜇𝑡1 +𝜇𝑡2 +
=𝑒
𝜇𝑡1 +𝜇𝑡2 +
𝜎 2 𝑡12 2
𝜎 2 𝑡22
+𝜎 𝑡1 𝑡2 +
2
2
=𝑒
𝜇𝑡1 −𝜇𝑡2 +
𝜎 2 𝑡12 2
𝜎 2 𝑡22
−𝜎 𝑡1 𝑡2 +
2
2
𝜎 2 𝑡12 2
𝜎 2 𝑡22
𝜎 2 𝑡12 2
𝜎 2 𝑡22
+𝜎 𝑡1 𝑡2 +
𝜇𝑡1 −𝜇𝑡2 +
−𝜎 𝑡1 𝑡2 +
2
2 𝑒
2
2
4
𝜎 2 𝑡12
𝜎 2 𝑡22
𝜎 2 𝑡12
2
= exp ( 𝜇𝑡1 + 𝜇𝑡2 +
+ 𝜎 𝑡1 𝑡2 +
+ 𝜇𝑡1 − 𝜇𝑡2 +
− 𝜎 2 𝑡1 𝑡2
2
2
2
𝜎 2 𝑡22
+
)
2
1
1
= exp (2𝜇𝑡1 + (2𝜎 2 )𝑡12 + (2𝜎 2 )𝑡22 )
2
2
1
= 𝑒 2𝜇𝑡1+2(2𝜎
2 )𝑡 2
1
1
𝑒 0𝑡2+2(2𝜎
2 )𝑡 2
2
= 𝑀𝑋1 +𝑋2 (𝑡1 )𝑀𝑋1−𝑋2 (𝑡2 )
= 𝑀𝑊 (𝑡1 )𝑀𝑍 (𝑡2 )
∴ 𝑀𝑊,𝑍 (𝑡1 , 𝑡2 ) = 𝑀𝑊 (𝑡1 )𝑀𝑍 (𝑡2 )
∴ 𝑊 = 𝑋1 + 𝑋2 𝑎𝑛𝑑 𝑍 = 𝑋1 − 𝑋2 are independent
Q2. 𝐿𝑒𝑡 𝑍 𝑏𝑒 𝑁(0,1), (1) please derive the covariance between 𝑍 𝑎𝑛𝑑 𝑍 2 ; (2) Are
𝑍 𝑎𝑛𝑑 𝑍 2 independent?
𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏:
(1) 𝑐𝑜𝑣(𝑍, 𝑍 2 ) = 𝐸 [(𝑍 − 𝐸(𝑍)) (𝑍 2 − 𝐸(𝑍 2 ))] = 𝐸[(𝑍 − 0)(𝑍 2 − 1)] = 𝐸[𝑍 3 ] =
𝜕3
𝜕𝑡3
[𝑒𝑥𝑝(12𝑡 2 )]|𝑡=0 = 0
(2) 𝑍 𝑎𝑛𝑑 𝑍 2 are not independent. You can do it in many ways, using (a) pdf, (b)
cdf/probabilities, or (c) mgf.
For example, 𝑃(𝑍 > 1, 𝑍 2 > 1) = 𝑃(𝑍 > 1) ≠ 𝑃(𝑍 > 1) ∗ 𝑃(𝑍 2 > 1) = 2[𝑃(𝑍 > 1)]2
Q3. Let X ~ N (3, 4) , please calculate P(1  X  3)
𝑺𝒐𝒍𝒖𝒕𝒊𝒐𝒏:
P (1  X  3)
1 3 X  3 3  3
 P(


)
2
2
2
 P (1  Z  0)
 P (0  Z  1)
 0.3413
5
𝑁𝑜𝑡𝑒: 𝐼𝑛 𝑡ℎ𝑒 𝑎𝑏𝑜𝑣𝑒, 𝑍~𝑁(0,1)
Q4. Students A and B plans to meet at the SAC Seawolf market between 12noon and
1pm tomorrow. The one who arrives first will wait for the other for 30 minutes, and
then leave. What is the chance that the two friends will be able to meet at SAC
during their appointed time period assuming that each one will arrive at SAC
independently, each at a random time between 12noon and 1pm?
Solution: 3/4 as illustrated by the following figure.
Let X and Y denote the arrival time (in hour where 0 indicates 12noon and 1
indicates 1pm) of A and B respectively. Our assumption implies that X and Y each
follows Uniform[0,1] distribution, and furthermore, they are independent, Thus the
joint pdf of X and Y follows the uniform distribution with f(x,y) = 1, if (x,y) is in the
square region, and f(x,y) = 0, outside the square region. A and B will be able to meet
iff |X − Y| ≤ 1/2 as represented by the red area M bounded by the lines: X − Y =
−1/2 and X − Y = +1/2.
That is:
1
1
1
3
P(A & B will meet) = P (|X − Y| ≤ ) = ∬M 1𝑑𝑥𝑑𝑦 = 1 − − =
2
8
8
4
*Definitions: population correlation & sample correlation
6
Definition: The population correlation coefficient ρ is defined as:
𝑐𝑜𝑣(𝑋, 𝑌)
𝜌=
√𝑣𝑎𝑟(𝑋) ∗ 𝑣𝑎𝑟(𝑌)
Definition: Let (X1 , Y1), …, (Xn , Yn) be a random sample from a given bivariate
population, then the sample correlation coefficient r is defined as:
𝑟=
∑(𝑋𝑖 − 𝑋̅)(𝑌𝑖 − 𝑌̅)
√[∑(𝑋𝑖 − 𝑋̅)2 ][∑(𝑌𝑖 − 𝑌̅)2 ]
*Definition: Bivariate Normal Random Variable
( X , Y ) ~ BN (x ,  x2 ;  y ,  y2 ;  ) where  is the correlation between X &Y
The joint p.d.f. of ( X , Y ) is
f X ,Y x, y  

1

exp 
2 1  2
1  2


 x  
X

  X
  y  Y
  
  Y



2

 x    2
 x   X  y   Y   y   Y
1

X




  

exp 

2

2


2



2 1   
2 X  Y 1  
X
X
Y



  Y



where    x   ,    y   . Then X and Y are said to have the bivariate
normal distribution. The joint moment generating function for X and Y is



2
1
2 X  Y


2

 x  X
  2  

 X
 y   Y

  Y


 ,
 

Exercise: Please derive the mgf of the bivariate normal distribution.
Q5. Let X and Y be random variables with joint pdf
f X ,Y x, y  
1





 ,
 


1


M t1 , t 2   exp t1 X  t 2 Y  t12 X2  2 t1t 2 X  Y  t 22 Y2  .
2


(a) Find the marginal pdf’s of X and Y;
(b) Prove that X and Y are independent if and only if ρ = 0.
(Here ρ is indeed, the <population> correlation coefficient between X and Y.)
(c) Find the distribution of  X  Y  .
(d) Find the conditional pdf of f(x|y), and f(y|x)
7
Solution:
(a)
The moment generating function of X can be given by
1
𝑀𝑋 (𝑡) = 𝑀(𝑡, 0) = 𝑒𝑥𝑝 [𝜇𝑋 𝑡 + 𝜎𝑋2 𝑡 2 ].
2
Similarly, the moment generating function of Y can be given by
1
𝑀𝑌 (𝑡) = 𝑀(0, 𝑡) = 𝑒𝑥𝑝 [𝜇𝑌 𝑡 + 𝜎𝑌2 𝑡 2 ].
2
Thus, X and Y are both marginally normal distributed, i.e.,
𝑋~𝑁(𝜇𝑋 , 𝜎𝑋2 ), and 𝑌~𝑁(𝜇𝑌 , 𝜎𝑌2 ).
The pdf of X is
𝑓𝑋 (𝑥) =
The pdf of Y is
𝑓𝑌 (𝑦) =
1
√2𝜋𝜎𝑋
1
√2𝜋𝜎𝑌
𝑒𝑥𝑝 [−
𝑒𝑥𝑝 [−
(𝑥 − 𝜇𝑋 )2
2𝜎𝑋2
(𝑦 − 𝜇𝑌 )2
2𝜎𝑌2
].
].
(b)
If

  0 , then M (t1 , t2 )  exp   X t1  Y t2 

1 2 2

 X t1   Y2t22    M (t1 , 0)  M (0, t2 )

2

Therefore, X and Y are independent.
If X and Y are independent, then
1


M (t1 , t2 )  M (t1 , 0)  M (0, t2 )  exp   X t1  Y t2   X2 t12   Y2t22  
2


1


 exp   X t1  Y t2   X2 t12  2  X  Y t1t2   Y2t22  
2


Therefore,   0
(c)
M X Y (t )  E et ( X Y )   E etX tY 
8
1
Recall that M (t1 , t2 )  E e
t X  t2Y
t1  t2  t
in
 , therefore we can obtain M X Y (t ) by setting
M (t1 , t2 )
That is,
1


M X Y (t )  M (t , t )  exp   X t  Y t   X2 t 2  2  X  Y t 2   Y2t 2  
2


1


 exp   X  Y  t   X2  2  X  Y   Y2  t 2 
2


 X  Y ~ N (    X  Y ,  2   X2  2  X  Y   Y2 )
(d)
The conditional distribution of X given Y=y is given by
𝑓(𝑥|𝑦) =
𝑓(𝑥, 𝑦)
1
=
𝑒𝑥𝑝 −
𝑓(𝑦)
√2𝜋𝜎𝑋 √1 − 𝜌2
2
𝜎
(𝑥 − 𝜇𝑋 − 𝜎𝑋 𝜌(𝑦 − 𝜇𝑌 ))
𝑌
{
Similarly, we have the conditional distribution of Y given X=x is
𝑓(𝑦|𝑥) =
𝑓(𝑥, 𝑦)
1
=
𝑒𝑥𝑝 −
2
𝑓(𝑥)
√1
2𝜋𝜎
−
𝜌
√
𝑌
}
2
𝜎
(𝑦 − 𝜇𝑌 − 𝜎𝑌 𝜌(𝑥 − 𝜇𝑋 ))
𝑋
2(1 − 𝜌2 )𝜎𝑌2
{
Therefore:
.
2(1 − 𝜌2 )𝜎𝑋2
.
}
𝜎𝑋
(𝑦 − 𝜇𝑌 ), (1 − 𝜌2 )𝜎𝑋2 )
𝜎𝑌
𝜎𝑌
𝑌|𝑋 = 𝑥 ~ 𝑁 (𝜇𝑌 + 𝜌 (𝑥 − 𝜇𝑋 ), (1 − 𝜌2 )𝜎𝑌2 )
𝜎𝑋
𝑋|𝑌 = 𝑦 ~ 𝑁 (𝜇𝑋 + 𝜌
Q6: 𝑍~𝑁(0,1), and the value of random variable W depends on a coin (fair coin) flip:
𝑍, 𝑖𝑓 ℎ𝑒𝑎𝑑
𝑊={
, find the distribution of W. Is the joint distribution of Z and W a
−𝑍, 𝑖𝑓 𝑡𝑎𝑖𝑙
bivariate normal?
Answer:
𝐹𝑊 (𝑤) = 𝑃(𝑊 ≤ 𝑤)
9
So 𝑊~𝑁(0,1).
= 𝑃(𝑊 ≤ 𝑤|𝐻)𝑃(𝐻) + 𝑃(𝑊 ≤ 𝑤|𝑇)𝑃(𝑇)
= 𝑃(𝑍 ≤ 𝑤)𝑃(𝐻) + 𝑃(−𝑍 ≤ 𝑤)𝑃(𝑇)
1
1
= 𝐹𝑍 (𝑤) × + 𝑃(𝑍 ≥ −𝑤) ×
2
2
1
1
= 𝐹𝑍 (𝑤) × + 𝑃(𝑍 ≤ 𝑤) ×
2
2
1
1
= 𝐹𝑍 (𝑤) × + 𝐹𝑍 (𝑤) ×
2
2
= 𝐹𝑍 (𝑤)
The joint distribution of Z and W is not normal. This can be shown by deriving the
joint mgf of Z and W and compare it with the joint mgf of bivariate normal.
𝑀(𝑡1 , 𝑡2 ) = 𝐸(𝑒 𝑡1𝑍+𝑡2𝑊 )
= 𝐸(𝐸(𝑒 𝑡1𝑍+𝑡2 𝑊 |𝑐𝑜𝑖𝑛))
= 𝐸(𝑒 𝑡1𝑍+𝑡2𝑊 |𝐻)𝑃(𝐻) + 𝐸(𝑒 𝑡1𝑍+𝑡2𝑊 |𝑇)𝑃(𝑇)
1
1
= 𝐸(𝑒 (𝑡1+𝑡2)𝑍 ) × + 𝐸(𝑒 (𝑡1−𝑡2)𝑍 ) ×
2
2
(𝑡1 −𝑡2 )2
1 (𝑡1+𝑡2 )2
= (𝑒 2 + 𝑒 2 )
2
This is not the joint mgf of bivariate normal.
Alternatively, you can also derive the following probability:
P(W + Z = 0) = P(W + Z = 0|H)P(H) + P(W + Z = 0|T)P(T)
1
1
1
1 1
= P(2Z = 0) × + P(−Z + Z = 0) × = 0 × + 1 × =
2
2
2
2 2
This shows that W+Z can not possibility follow a univariate normal distribution –
which in turns shows that W and Z can not possible follow a bivariate normal
distribution.
10