MTH 202 : Probability and Statistics
Lecture 3 & 4
8, 13 January, 2013
Conditional Probability and Independence
2.1 : Conditional Probability
The king comes from a family of two children. What is the probability
that the other child is his sister?
The sample space for this problem perhaps can be modeled as Ω =
{(B, B), (B, G), (G, B), (G, G)}, where ”B” represents a boy, ”G” represents a girl and the positions in the order represents the first and
second born child respectively. Assigning equal probability to each of
the occurrences, i.e., assigning 1/4 to each of them.
However, the moment the information that ”the king comes...” is given,
the possibility of (G, G) is gone from the sample space. At this moment
the new space is Ω = {(B, B), (B, G), (G, B)} and the probability that
the king would have a sister (elder or younger) is seen to be 2/3.
At many places while defining probability, it often happens that certain
information are brought which make the sample space shorter, changing
the given probability assignment.
Definition 2.1.1 : Let (Ω, S, P ) denote a probability space and A, B
are two given events from S with P B > 0. Then the conditional
probability P (A|B) of occurrence of A given B is defined by :
P (A ∩ B)
PB
In above example, suppose U denote the event ”one child is a girl”
and V denote ”one child is the king”. Then we have from the above
formula :
P (U ∩ V )
2/4
=
= 2/3
P (U |V ) =
PV
3/4
Example 2.1.2 : Find the probability of a toss of two dies of different
colors will result in a number less than 5 given that the toss resulted
in an odd number.
P (A|B) :=
1
2
Solution : Let A denote the event of ”less than 5” and B denote that
the toss resulted in an odd number. We see that :
A = {(1, 1), (1, 2), (1, 3), (2, 1), (2, 2), (3, 1)}
Assuming all thirty six elementary possibilities are uniformly assigned
to 1/36. Now A ∩ B = {(1, 2), (2, 1)}. Also
B = {(1, 2), (1, 4), (1, 6), (2, 1), (2, 3), (2, 5), . . .
Then we have P B = 1/2 and hence
2/36
P (A ∩ B)
=
= 1/9
P (A|B) =
PB
1/2
Theorem 2.1.3 : Let (Ω, S, P ) denote a probability space and let
U ∈ S with P U > 0. Define PU (A) := P (A|U ). Then PU define a
probability on (Ω, S).
Proof : Exercise
From the definition of conditional probability we see that P (A ∩ B) =
P A.P (B|A) if P A > 0. Similarly P (A ∩ B) = P B.P (A|B) assuming
P B > 0. These together implies :
Property 2.1.4 :
P A.P (B|A) = P B.P (A|B), assuming P A > 0, P B > 0
Theorem 2.1.5 : (Multiplication Rule) Let (Ω, S, P ) denote a
probability space and A1 , A2 , . . . , An ∈ S with P (∩n−1
i=1 Ai ) > 0. Then
P (∩ni=1 Ai ) = P A1 .P (A2 |A1 ).P (A3 |A1 ∩ A2 ). . . . .P (An | ∩n−1
i=1 Ai )
Proof : We will use induction on n. For n = 1, it is obviously true
and n = 2 is merely the definition as stated. Suppose we have proved
it for n = 1, 2, . . . , N (N < n), and we need to establish it for N + 1.
Hence we have :
N −1
P (∩N
i=1 Ai ) = P A1 .P (A2 |A1 ).P (A3 |A1 ∩ A2 ). . . . .P (AN | ∩i=1 Ai )
Let B := ∩N
i=1 Ai . Then
P (B ∩ AN +1 ) = P B.P (AN +1 |B)
n−1
since by hypothesis P B ≥ P (∩i=1
Ai ) > 0. But this is just the same as
+1
N
P (∩N
i=1 Ai ) = P A1 .P (A2 |A1 ).P (A3 |A1 ∩ A2 ). . . . .P (AN +1 | ∩i=1 Ai )
Hence the rule is established for N + 1 and by induction the statement
follows.
Exercise 2.1.6 : Consider a hand of five cards in a game of poker. If
the cards are dealt at random, there are 52
possible hands of five cards
5
3
each. Let A = {at least three cards of spades}, B = {all five cards of spades.
Then find the probability P (B|A).
Solution : See [RS], Section 1.5, Example 3, Page 30.
Figure 1. Five cards of spades
Theorem 2.1.7 : (Total Probability Rule) Let (Ω, S, P ) be a
probability space. {Ej }∞
j=1 be a countable collection of mutually disjoint events from S (i.e., Ei ∩ Ek = ∅ if i 6= k) such that Ω = ∪∞
j=0 Ej .
Suppose that P (Ej ) > 0 for all j ∈ N. Then for all B ∈ S we have
PB =
∞
X
P (Ej )P (B|Ej )
j=1
∞
Proof : We have B = ∪∞
j=0 (B ∩ Ej ). Since {B ∩ Ej }j=1 are mutually
disjoint events, we have from the definition of probability that
PB =
∞
X
P (B ∩ Ej )
j=1
Since P (Ej ) > 0, we have that P (B ∩ Ej ) = P (Ej )P (B|Ej ) for all
j ∈ N.
4
Note that a similar rule is possible while the sets Ej are finitely many.
We will apply this in the next exercise.
Exercise 2.1.8 : Each of N + 1 identical urns marked 0, 1, 2, . . . , N
contains N balls. The k-th urn contains k black and N − k white balls,
k = 0, 1, 2, . . . , N . An urn is chosen at random, and n random drawing
are made from it, the ball drawn is always being replaced. If all the
draws result in black balls, find the probability that the (n + 1)-th
draw will also produce a black ball. How does this probability behave
as N → ∞?
Solution : Let Ai be the event of choosing i-th urn, B be the event of
drawing black balls in n draws and C be the event of getting a black
ball in the (n + 1)-th draw.
Since all the events A0 , A1 , . . . , AN are disjoint, we have from the total
probability rule that
P B = P (A0 ).P (B|A0 ) + P (A1 ).P (B|A1 ) + . . . + P (AN ).P (B|AN )
P
1
i n
= N
i=0 N +1 .( N )
Now we will have to compute
P (B ∩ C)
PB
But B ∩C represents the event that all n+1 draws result
balls.
PN in black
i n+1
1
A similar calculation as above give that P (B ∩ C) = i=0 N +1 .( N ) .
Hence
PN 1
PN i n+1
i n+1
(N )
i=0 N +1 .( N )
P (C|B) = PN 1
= Pi=0
N
i n
i n
i=0 N +1 .( N )
i=0 ( N )
P (C|B) =
To see what happen when N → ∞ we note that, from the definition of
Riemann integral
Z 1
N
X
1
1
i
n
=
x dx = lim
.( )n
N
→∞
n+1
N +1 N
0
i=0
Similarly,
1
=
n+2
Z
0
1
xn+1 dx = lim
N →∞
N
X
i=0
1
i
.( )n+1
N +1 N
1
n+2
Since
6= 0, using algebra of limits we have that P (C|B) →
N → ∞.
As a consequence of total probability rule, we will deduce :
n+2
n+1
as
5
Theorem 2.1.9 : (Bayes Rule) Let (Ω, S, P ) be a probability space.
{Ej }∞
j=1 be a countable collection of mutually disjoint events from S
such that Ω = ∪∞
j=0 Ej . Suppose that P (Ej ) > 0 for all j ∈ N. Then
for all B ∈ S with P B > 0 we have :
P (Ej )P (B|Ej )
P (Ej |B) = P∞
i=1 P (Ej )P (B|Ej )
Proof : We know that P (B ∩ Ej ) = P BP (Ej |B) = P (Ej )P (B|Ej ).
Hence
P (Ej )P (B|Ej )
P (Ej |B) =
PB
The result now follows from the total probability rule for P B.
Exercise 2.1.10 : Five percent of patients suffering from a certain
disease are selected to undergo a new treatment that is believed to
increase the recovery rate from 30% to 50%. A person is randomly
selected from these patients after the completion of the treatment and
is found to have recovered. What is the probability that the patient
received the new treatment.
Solution : Let A denote the event of patient chosen has received new
treatment and B denote the event of patient chosen has recovered. If
Ω denote the set of all patients, then A ∪ Ac = Ω. Let P (B|Ac ) =: p
(i.e., this is the probability of recovery of a patient chosen without new
treatment). According to the information P (B|A) = p(1 + α) where
3 1
α ∈ [ 10
, 2 ].
Now using Bayes Rule :
P (B|A).P (A)
P (A|B) =
P (B|A).P (A) + P (B|Ac ).P (Ac )
5
p(1 + α). 100
1+α
=
5
95 =
20 + α
p(1 + α). 100 + p. 100
1+α
The graph of f (α) = 20+α is increasing in the interval [3/10, 1/2] since
19
0
f 0 (α) = (20+α)
2 (you can easily verify that f (α) > 0 in this interval).
Hence P (A|B) is approximately between f (3/10) ≈ 0.064 to f (5/10) ≈
0.073.
6
2.2 : Independence of events
Let A and B be two events with P A > 0, P B > 0. It might happen
sometime that P (A|B) = P A. It would essentially mean that the probability of the occurrence of the event A would not affect by occurrence
of the event B.
Example 2.2.1 : Let’s think of tossing an unbiased die three times.
Let A denote of getting 6 at the first toss and B denote of getting 1 at
the second toss. It is likely to say that the occurrence of the events A
and B are independent of each other. Now the events would look like :
A = {(6, i, j) : 1 ≤ i, j ≤ 6}, B = {(i, 1, j) : 1 ≤ i, j ≤ 6}
Assuming that the die is fair, we can assign equal probability to all
1
the events, i.e., P (a, b, c) = 6×36
for all a, b, c ∈ {1, 2, . . . , 6}. From
36
above description of A and B, we see that P A = 6×36
= 16 . Similarly,
P B = 16 . Also
1
A ∩ B = {(6, 1, i) : 1 ≤ i ≤ 6} ⇒ P (A ∩ B) =
36
We now see that :
P (A ∩ B)
1
P (A|B) =
= = PA
PB
6
which was kind of expected. We would formally say this as :
Definition 2.2.2 : Let (Ω, S, P ) be a probability space. Two events
A, B ∈ S are said to be independent if P (A ∩ B) = P A.P B.
Using the definition of conditional probability it turns out that :
Theorem 2.2.3 : Let (Ω, S, P ) be a probability space. If A, B ∈ S
are independent events, then
P (A|B) = P A if P B > 0
P (B|A) = P B if P A > 0
It is interesting to note that if A and B are independent, then so are :
(i) A and B c , (ii) Ac and B, (iii) Ac and B c . (Why?)
Suggested exercise : Read the section ”1.6 Independence of events”
from [RS]
7
References :
[RS] An Introduction to Probability and Statistics, V.K. Rohatgi and
A.K. Saleh, Second Edition, Wiley Students Edition.