1. Sample Space and Probability Part II: Independence; Examples ECE 302 Spring 2012 Purdue University, School of ECE Prof. Ilya Pollak Independence: DefiniHon Two events A and B are called independent if P(B∩A) = P(A)P(B) Ilya Pollak
Example •  I toss a fair coin here; somebody tosses another
fair coin in the next room. •  {My coin comes up H} and {his coin comes up H}
are reasonably modeled as independent
events: P( my toss H ) × P( his toss H ) = 1/2 × 1/2 = 1/4 = P( both tosses H ) Ilya Pollak
Independence: Some IntuiHon •  Two events are independent if the occurrence of one does not provide any informaHon as to how likely the other one is to occur. –  E.g., the events {this lecture will last somewhere between 48 and 52 minutes} and {President Obama will be reelected in 2012} can safely assumed to be independent; –  the events {U.S. unemployment will be under 6% in October 2012} and {President Obama will be reelected in 2012} are NOT independent. Ilya Pollak
More formally, •  Independence: P(B∩A) = P(A)P(B) •  CondiHonal probability: P(B∩A) = P(A)P(B|A) •  Thus, if P(A)>0, then independence is equivalent to P(B) = P(B|A) -­‐-­‐-­‐ i.e., when A and B are independent, knowing that A occurred does not change the probability of B. •  Similarly, from P(B∩A) = P(B)P(A|B) we have that, if P(B)>0, then independence is equivalent to P(A) = P(A|B). Ilya Pollak
Example •  In prisoner’s dilemma, the guard’s answer
provided no new informaHon regarding the
probability of A’s release: –  P(A released) = 2/3 before guard’s response –  P(A released | guards response) = 2/3 sHll •  The events {A released} and {guard says B} are
independent. •  The events {A released} and {guard says C} are
independent. Ilya Pollak
Independent ≠ mutually exclusive!! •  Two mutually exclusive (or disjoint) events cannot
both occur. •  If A and B are mutually exclusive and both have
nonzero probability, then P(B∩A) = P(Ø) = 0 but P(A)P(B) > 0, and therefore P(B∩A) ≠ P(A)P(B), so A and B are dependent! •  E.g., {A Republican will win the 2012 US PresidenHal
elecHon} and {President Obama will win the 2012
US PresidenHal elecHon} are mutually exclusive
events. They are not independent! Ilya Pollak
CondiHoning may affect
independence! • 
• 
• 
• 
• 
A and B are independent: P(B∩A) = P(A)P(B). Event C occurs. Are A and B condiHonally independent, given C? In other words, is P(B∩A|C) = P(A|C)P(B|C) ? Answer: Not necessarily! Ilya Pollak
Example 1.20 •  A BriHsh Airways flight from Paris to London and
a United flight from Indianapolis to Chicago. •  Event A={Paris-­‐London flight departs on Hme}. •  Event B={Indy-­‐Chicago flight departs on Hme}. •  Event C={Exactly one of them departs on Hme}. •  A and B are reasonably modeled as independent. •  A and B are condiHonally dependent given C: P(A∩B|C)=0 whereas P(A|C)P(B|C)>0. Ilya Pollak
False-­‐PosiHve Puzzle Revisited • 
• 
• 
• 
• 
A crime commiled on an island, populaHon 5000 A priori, all are equally likely to have commiled it Based on a forensic test, a suspect is arrested Apart from the test, no other evidence Accuracy of the test is 99.9%, i.e., P(test is posiHve | suspect is guilty) = 0.999 and P(test is negaHve | suspect is innocent) = 0.999 •  P(guilty | +) ≈ 0.167. •  Can we improve accuracy by tesHng again? Ilya Pollak
Improving accuracy through repeated
tesHng •  If the errors of repeated tests are the same, no use
repeaHng. –  E.g., suppose that a blood test produces an error if and only if
presented with a blood sample of some rare geneHc makeup. Then retesHng the same person using the same test would
either always produce an error or always be correct. •  If the errors of repeated tests are independent or close
to independent, accuracy is improved. –  E.g., suppose that a blood tests produces an error if and only
if the blood sample is contaminated. Then collecHng and
tesHng two different samples on two different days may
reasonably be assumed to lead to independence of errors. Ilya Pollak
False-­‐PosiHve Puzzle with Two
CondiHonally Independent Tests •  Let i+ be the event “ Test i posiHve” for i=1,2, and let
i-­‐ be the event “ Test i negaHve.” •  Suppose accuracy of both tests is 99.9%, i.e., P(i+ | guilty) = 0.999 and P(i-­‐ | innocent) = 0.999 •  Suppose the tests are condiHonally independent
given guilt or innocence. •  What is P(guilty | 1+ and 2+)? Ilya Pollak
False-­‐PosiHve Puzzle with CondiHonally
Independent Tests: SoluHon •  P(1+ and 2+ and guilty) = P(1+ and 2+ | guilty) P(guilty) = P(1+ | guilty) P(2+ | guilty) P(guilty) using condiHonal
independence of the test results given guilt Ilya Pollak
False-­‐PosiHve Puzzle with CondiHonally
Independent Tests: SoluHon •  P(1+ and 2+ and guilty) = P(1+ and 2+ | guilty) P(guilty) = P(1+ | guilty) P(2+ | guilty) P(guilty) = 0.9992·∙0.0002 =
1.996002·∙10-­‐4 •  P(1+ and 2+ and innocent) = P(1+ and 2+ | innocent) P(innocent) =
P(1+ | innocent) P(2+ | innocent) P(innocent) using condiHonal
independence of test results given innocence Ilya Pollak
False-­‐PosiHve Puzzle with CondiHonally
Independent Tests: SoluHon •  P(1+ and 2+ and guilty) = P(1+ and 2+ | guilty) P(guilty) = P(1+ | guilty) P(2+ | guilty) P(guilty) = 0.9992·∙0.0002 =
1.996002·∙10-­‐4 •  P(1+ and 2+ and innocent) = P(1+ and 2+ | innocent) P(innocent) =
P(1+ | innocent) P(2+ | innocent) P(innocent) = 0.0012·∙0.9998 =
0.009998·∙10-­‐4 •  P(1+ and 2+) = P(1+ and 2+ and guilty) + P(1+ and 2+ and innocent) =
2.006·∙10-­‐4 •  P(guilty | 1+ and 2+) = P(1+ and 2+ and guilty) / P(1+ and 2+) =
1.996002·∙10-­‐4/(2.006·∙10-­‐4) ≈ 0.995 – much more accurate than the
0.167 probability obtained from a single test. Ilya Pollak
Independence of several events Ilya Pollak
Example 1.22 Two independent flights, Paris-­‐London and Indy-­‐Chicago. Each has on-­‐Hme departure probability 1/2. A={Paris-­‐London on Hme} B={Indy-­‐Chicago on Hme} C={Exactly one on Hme} D={Both on Hme} P(A)=1/2; P(B) = 1/2; P(A∩B) = 1/4 = P(A)P(B) P(C) = P[(A∩Bc) U (Ac∩B)] = P(A∩Bc) + P(Ac∩B) = (1/2)·∙(1/2) +
(1/2)·∙(1/2) = 1/2 •  P(A∩C) = P(A∩Bc) = 1/4 = P(A)P(C) A,B, and C are pairwise •  P(B∩C) = P(Ac∩B) = 1/4 = P(B)P(C) independent •  P(A∩B∩C) = 0 ≠ P(A)P(B)P(C) A,B, and C are not independent • 
• 
• 
• 
• 
• 
• 
• 
Ilya Pollak
Example Two independent flights, Paris-­‐London and Indy-­‐Chicago. Each has on-­‐Hme departure probability 1/2. A={Paris-­‐London on Hme} B={Indy-­‐Chicago on Hme} C={Exactly one on Hme} D={Both on Hme} P(A∩D∩Ø) = 0 = P(A)P(D)P(Ø), yet this is not enough for
independence •  P(D)=1/4 •  P(A∩D) = P(D) = 1/4 ≠ P(A)P(D), therefore A, D, and Ø are not
independent! • 
• 
• 
• 
• 
• 
• 
Ilya Pollak
Example 1.24: Network ConnecHvity •  Computer network connects two nodes, A and B,
through several intermediate nodes. •  For every pair of directly connected nodes, there
is a given probability that the link is working. •  The link failures are independent. 0.8
P(link A -­‐> C working) = 0.9
E C 0.9
0.95
A F 0.75
D 0.85
B 0.95
P(there is a working path A -­‐> B) = ?
Ilya Pollak
Example: Network ConnecHvity P(path C -­‐> B down) = P(path CEB down)P(path CFB down) = [1-­‐P(CEB ok)] ·∙ [1-­‐P(CFB ok)]
= [1-­‐P(CE ok)P(EB ok)] ·∙ [1-­‐P(CF ok)P(FB ok)] = (1 -­‐ 0.72)(1 -­‐ 0.8075) = 0.0539
P(path C -­‐> B ok) = 1 -­‐ P(path C -­‐> B down) = 0.9461
P(A -­‐> C -­‐> B ok) = P(A -­‐> C ok) P(C -­‐> B ok) = 0.9 · 0.9461 = 0.85149 P(A -­‐> D -­‐> B ok) = P(A -­‐> D ok) P(D -­‐> B ok) = 0.75 · 0.95 = 0.7125 P(path A -­‐> B down) = P(path ACB down)P(path ADB down) = [1-­‐P(ACB ok)] ·∙ [1-­‐P(ADB ok)]
= (1 -­‐ 0.85149)(1 -­‐ 0.7125) ≈ 0.043
P(path A -­‐> B ok) = 1 -­‐ P(path A -­‐> B down) ≈ 0.957 0.8
C P(link A -­‐> C working) = 0.9
E 0.9
0.95
A F 0.75
D 0.85
B 0.95
Ilya Pollak
Example: Network ConnecHvity Used the following two rules repeatedly:
p1
p2
p3
pm
P(series subsystem succeeds) = p1 p2 … pm
p1
p2
P(parallel subsystem fails) = (1-­‐p1)(1-­‐p2) … (1-­‐pm)
pm
Ilya Pollak
Example: CommunicaHon through
a noisy channel (p. 60) transmit
0
receive
P(receive 0 | transmit 0) = 1 – ε0
0
ε0
Errors in different symbol
transmissions are independent
ε1
1
1 – ε1
1
Suppose P(0 transmitted) = p. What is P(symbol is correctly received)?
P(correctly received) = P( received 1, sent 1) + P(received 0, sent 0)
= P( received 1 | sent 1) P(sent 1) + P(received 0 | sent 0) P(sent 0)
= (1 – ε1)(1 – p) + (1 – ε0)p
Ilya Pollak
Example: CommunicaHon through
a noisy channel 0
1 – ε0
0
ε0
ε1
1
1
1 – ε1
Suppose 1011 is transmitted. What is the probability that it is received correctly?
(1 – ε1)(1 – ε0)(1 – ε1)(1 – ε1) = (1 – ε1)3 (1 – ε0)
Ilya Pollak
Example: CommunicaHon through
a noisy channel 0
1 – ε0
0
ε0
ε1
1
1
1 – ε1
Each symbol is transmitted three times, and the received symbol is decoded by majority
rule. What is the probability that a zero is correctly decoded?
P( 0 correctly received) = P( 000 received | 000 sent ) + P( 001 received | 000 sent )
+ P( 010 received | 000 sent ) + P( 100 received | 000 sent )
= (1 – ε0)3 + 3 (1 – ε0)2 ε0
Ilya Pollak
Example: CommunicaHon through
a noisy channel 0
1 – ε0
0
ε0
ε1
1
1
1 – ε1
Each symbol is transmitted three times, and the received symbol is decoded by majority
rule. What is the probability that a zero is correctly decoded?
P( 0 correctly received) = P( 000 received | 000 sent ) + P( 001 received | 000 sent )
+ P( 010 received | 000 sent ) + P( 100 received | 000 sent )
= (1 – ε0)3 + 3 (1 – ε0)2 ε0
Using the same encoding strategy, P( 0 sent |101 received) = ?
Ilya Pollak
Example: CommunicaHon through
a noisy channel 0
1 – ε0
0
ε0
ε1
1
1
1 – ε1
Each symbol is transmitted three times, and the received symbol is decoded by majority
rule. What is the probability that a zero is correctly decoded?
P( 0 correctly received) = P( 000 received | 000 sent ) + P( 001 received | 000 sent )
+ P( 010 received | 000 sent ) + P( 100 received | 000 sent )
= (1 – ε0)3 + 3 (1 – ε0)2 ε0
P(0 sent | 101 received ) = ?
Approach: since P( 101 received | 0 sent) = ε02 (1 – ε0) is easy to compute, use Bayes rule.
Ilya Pollak
Example: CommunicaHon through
a noisy channel 0
1 – ε0
0
ε0
ε1
1
1
1 – ε1
P( 0 sent | received 101) = ?
Approach: since P( 101 received | 0 sent) = ε02 (1 – ε0) is easy to compute, use Bayes rule.
P( 0 sent | 101 received ) = P( 101 received | 0 sent) P( 0 sent ) / P( 101 received )
= ε02 (1 – ε0) p / P( 101 received )
To compute the denominator, use the total probability theorem:
P( 101 received ) = P( 101 received | 0 sent) P( 0 sent ) + P( 101 received | 1 sent ) P( 1 sent )
= ε02 (1 – ε0) p + ε1 (1 – ε1) 2 (1 – p)
Answer: P( 0 sent | 101 received ) = ε02 (1 – ε0) p / [ε02 (1 – ε0) p + ε1 (1 – ε1) 2 (1 – p)]
Ilya Pollak
Example: An unfair tetrahedral die •  Ayer S. Ross, A First Course in Probability, 6th
Edi8on, PrenHce-­‐Hall, 2002, pages 87-­‐88. Ilya Pollak
Example: An unfair tetrahedral die • 
• 
• 
P(1) = 0.1, P(2) = 0.2, P(3) = 0.3, P(4) = 0.4. Rolled many Hmes, independently. What’s the probability that a 4 is obtained before the first Hme that a 1 is
obtained? First roll
Second roll
4 before 1
P(4) = 0.4
P(2 or 3) = 0.5
P(1) = 0.1
P(4) = 0.4
P(2 or 3) = 0.5
P(1) = 0.1
1 before 4
Ilya Pollak
Example: An unfair tetrahedral die •  P(4 before 1) = 4/5 = 0.4/(0.1 + 0.4) = P(4)/[P(1) + P(4)] =
P(4 | 1 or 4). •  This is true for any disjoint events A and B with P(A) ≠ 0
and P(B) ≠ 0. •  In independent repeHHons of the experiment, –  P(A occurs before B) = P(A)/[P(A) + P(B)] = P(A | A or B) •  This can be shown as in the die example. •  Another way of showing this is to condiHon everything on
the outcome of the first repeHHon of the experiment. Ilya Pollak
Example: An unfair tetrahedral die P(A before B) = P(A before B | A on 1st trial)P(A on 1st trial)
+ P(A before B | B on 1st trial)P(B on 1st trial)
+ P(A before B | A c ∩ B c on 1st trial)P(A c ∩ B c on 1st trial)
P(A before B | A on 1st trial) = 1
P(A before B | B on 1st trial) = 0
P(A before B | A c ∩ B c on 1st trial) = P(A before B). This is because
if neither A nor B occurs on the first trial, the remainder of the experiment looks
exactly like it did before the first trial: we perform independent trials until
A or B occurs. Since trials are independent, the probability of "A before B"
during the remaining trials is not affected by the non-occurrence of A or B on
the first trial.
Ilya Pollak
Example: An unfair tetrahedral die P(A before B) = P(A before B | A on 1st trial)P(A on 1st trial)
+ P(A before B | B on 1st trial)P(B on 1st trial)
+ P(A before B | A c ∩ B c on 1st trial)P(A c ∩ B c on 1st trial)
P(A before B | A on 1st trial) = 1
P(A before B | B on 1st trial) = 0
P(A before B | A c ∩ B c on 1st trial) = P(A before B).
P(A before B) = 1⋅ P(A on 1st trial) + 0 ⋅ P(B on 1st trial)
+ P(A before B) ⋅ P(A c ∩ B c on 1st trial)
P(A on 1st trial)
P(A)
P(A)
P(A before B) =
=
=
= P(A | A ∪ B)
c
c
1 − P(A ∩ B on 1st trial) P(A ∪ B) P(A ∪ B)
Ilya Pollak
An aside on geometric series Ilya Pollak