Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Random Experiment
TMS-062: Lecture 2
Probability
Sergei Zuyev
Experiment – any repetitive process which yields an observable
outcome – an observation - usually called the value.
Random experiment – the one with unpredictable outcome.
Sample space (SS = Ω) the set of all possible outcomes of
the experiment. These (e. g. a list) are called elementary
events
An event is a subset of the Ω, i. e. a collection of outcomes.
Event E occurs in experiment if the outcome ω ∈ E.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Operations on events
Example 1. Tossing a coin: Ω = {H, T }
Example 2. Tossing two distinguishable coins:
Ω = {HH, HT , TH, TT }
Event ‘The coins show the same face’={HH, TT } ⊂ Ω
As events are subsets of Ω set operations are applicable to
them. Consider Example 2: tossing two distinguishable coins:
Ω = {HH, HT , TH, TT }.
Event not A = A = {ω 6∈ A}, i. e. event ‘A does not occur’.
Ω
Example 3. Tossing two indistinguishable coins:
Ω = {HH, HT , TT }
Example 4. Microsoft corp. share price at randomly selected
day in the past: Ω = [0, ∞)
A
not A = A
Example. Event A =‘The coins show the same
face’= {HH, TT }, event ‘The coins show different
faces’={HT , TH} = Ω \ A.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Event A and B: at a single realisation ω of experiment both A
and B occur, i. e. ω ∈ A and ω ∈ B implying ω ∈ A ∩ B = AB (∩
is usually omitted).
Event A or B: at a single realisation ω either A or B or both
occur, i. e. ω ∈ A ∪ B.
Ω
Ω
A
A
B
B
A or B = A ∪ B
A and B = AB
Example. Event B =‘The first coin shows Head’= {HH, HT }
AND A =‘The coins show the same face’ is
AB = {HH, TT } ∩ {HH, HT } = {HH}.
Sergei Zuyev
Example. Event B =‘The first coin shows Head’= {HH, HT }
OR A =‘The coins show the same face’ is
A ∪ B = {HH, TT } ∪ {HH, HT } = {HH, HT , TT }.
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Probability
Event A but not B: at a single realisation ω event A occurs but B
does not, i. e. ω ∈ A \ B = AB.
Ω
A
B
A\B
Sergei Zuyev
TMS-062: Lecture 2 Probability
Despite unpredictability of the outcome in a given random
experiment, there is often an intrinsic stability in following
sense:
in a long run of repetitions the relative frequencies of
occurrence of events stabilise about constant values: these
values are taken as probability of events, for an event E its
probability is denoted by P(E).
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Law of Large Numbers
Probability Space
Example 1 (cont’d) Relative frequency of ‘Tail’ in 500 simulated
symmetric coin tosses:
Definition
Probability space is Ω with defined on it probability P.
Example 1 (cont’d) Probability space of one toss of a fair coin:
Ω
P
Sergei Zuyev
TMS-062: Lecture 2 Probability
H
1/2
Sergei Zuyev
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
T
1/2
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Properties of probability
Is there anything wrong with the following probability space
Ω
P
H
0.53
T
0.47
NO! e. g. your coin can be bent. Whether or not your probability
model corresponds to your experiment is a different question!
Definition and consequences:
1
0 ≤ P(E) ≤ 1
2
P(∅) = 0 equiv. ‘∅ is impossible event’ – never occurs.
3
P(Ω) = 1 equiv. ‘Ω is certain event’ – always occurs.
4
If A1 and A2 cannot occur simultaneously, i. e. A1 ∩ A2 = ∅
(disjoint events), then P(A1 ∪ A2 ) = P(A1 ) + P(A2 ).
5
Sergei Zuyev
TMS-062: Lecture 2 Probability
In general, if E = ∪∞
Ai where A1 , A2 , . . . are all pairwise
i=1P
disjoint, then P(E) = ∞
i=1 P(Ai ).
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Discrete Probability Space
Discrete symmetric experiment
Important implication: if Ω is discrete Ω = {ω1 , ω2 , . . . }, then
X
P(E) =
P(ω)
ω∈E
and thus probability of events is defined by probability of all the
elementary events.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
If Ω has finite number N of elementary events ωi which are
known to be P
equally likely to occur then P(ωi ) = const. But since
1 = P(Ω) = N
i=1 P(ωi ) then P(ωi ) = 1/N and thus for any
event
|E|
No. of ωi ∈ E
=
P(E) =
No. of ωi ∈ Ω
N
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Example 2 (cont’d) Two distinguishable symmetric coins:
Ω
P
HH
1/4
HT
1/4
TH
1/4
TT
1/4
Prob. of ‘there is at least one Head’= P{HH, HT , TH}
= P(HH) + P(HT ) + P(TH) = 1/4 + 1/4 + 1/4 = 3/4 which is
the no. of elementary events in the event over the total no. of
outcomes as all are equiprobable.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Example 5. Two symmetric dice are thrown. What is Prob. of
event E that the sum of numbers showing is greater than 10?
Elementary events are pairs (i, j), where 0 ≤ i, j ≤ 6 and they
all equiprobable. So N = |Ω| = 6 × 6 = 36 and there are 3
elementary events implying E = {(5, 6), (6, 5), (6, 6)}. Thus
P(E) = 3/36 = 1/12 ≈ 8.33%.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Indistinguishable coins
Further Aspects of Probability
Example 3 (cont’d) Two indistinguishable symmetric coins:
Ω
P
HH
1/4
HT
1/2
TT
1/4
Consider event not A = A = {ω 6∈ A}. By properties 3) and 4) of
probability, P(A) = 1 − P(A).
Ω
Can be easily obtained from Example 2. It is wrong to assume
that the outcomes here are equiprobable!
A
not A = A
NB. It is a common approach: make similar indistinguishable
objects distinguishable! This usually simplifies description of
the probability space.
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Probability of Union
Event A or B: at a single realisation ω either A or B or both
occur, i. e. ω ∈ A ∪ B.
Example. 65% of students in a class are male. What is prob.
that a randomly chosen student is female?
Ω
P{female} = 1 − P{male} = 1 − 0.65 = 0.35 .
A
B
A or B = A ∪ B
P(A ∪ B) = P(A) + P(B) − P(AB)
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sergei Zuyev
TMS-062: Lecture 2 Probability
(1)
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Odds
Indeed, by property 4 of prob., one has
P(A ∪ B) = P(A \ B) + P(B \ A) + P(AB)
Along with probability there are also considered Odds for event
and Odds against event:
But
P(A) = P(A \ B) + P(AB)
OF (E) =
P(E)
P(E)
=
1 − P(E)
P(E)
OA (E) =
1
P(E)
=
P(E)
OF (E)
P(B) = P(B \ A) + P(AB)
Thus P(A) + P(B) counts P(AB) twice, implying (1).
Example: Card drawn at random from pack. What is prob. that
it’s a Spade (S) or an Ace (A)?
yielding
P(E) =
P(S ∪ A) = P(S) + P(A) − P(SA)
= 13/52 + 4/52 − 1/52 = 16/52 = 4/13 .
Sergei Zuyev
1
OF (E)
=
.
1 + OA (E)
1 + OF (E)
E. g., “2:1 against” means P(success) = 1/(1 + 2) = 1/3,
“3:1 on” (or “3:1 for”) means P(success) = 3/(1 + 3) = 3/4.
TMS-062: Lecture 2 Probability
Sergei Zuyev
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Conditional Probability
It is probability of A given that B also occurs:
P(A
B) =
P(AB)
P(B)
provided P(B) > 0 .
In equiprob. case we have
(2)
If we know B occurs we only need to consider the outcomes in
B, i. e. sample space is reduced to B and A can occur only if
the outcome is in B.
Ω
P(A
B) =
No. outcomes in AB
No. outcomes in B
thus P(A B) represents the relative frequency of A when B
occurs.
Example: Knowing that a Dame is drawn from a 52-card pack,
what is prob. that it is the Dame of spades?
P(S
D) = P(SD)/P(D) = (1/52)/(4/52) = 1/4 .
A
B
P(A B)
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Independence
Example: Tossing two symmetric coins. Events H1 = {First coin
shows Head} and T2 = {Second coin shows Tail} are
independent. Indeed
Two events A and B are called independent, if
P(AB) = P(A)P(B) .
P(H1 T2 ) = 1/4 = 1/2 · 1/2 = P(H1 ) · P(T2 )
Events A1 , . . . , An are mutually independent, if for any
sub-system of events Ai1 , . . . , Aik one has
If events A and B are independent and P(B) > 0 then
P(Ai1 . . . Aik ) = P(Ai1 ) · · · · · P(Aik ) .
P(A
NB. There are examples of events which are pairwise
independent but not mutually independent!
Sergei Zuyev
B) =
P(AB)
P(A) P(B)
=
= P(A)
P(B)
P(B)
as expected.
TMS-062: Lecture 2 Probability
Sergei Zuyev
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Total probability
Ω
E1
E2
E3
Formula (2) implies the formula for joint probability:
P(AB) = P(A
B) P(B) .
A
(3)
Events E1 , . . . , En (cases) are said to form a partition of Ω,
if they are
1
2
mutually exclusive: Ei Ej = ∅ for any i 6= j, i. e. they cannot
occur simultaneously;
exhaustive: E1 ∪ · · · ∪ En = Ω, i. e. in any outcome at least
one (in fact, exactly one by 1.) of events Ei occurs.
Sergei Zuyev
TMS-062: Lecture 2 Probability
En
Ej
Then for any event A we have by (3)
P(A) = P(AE1 ) + P(AE2 ) + · · · + P(AEn )
= P(A
E1 ) P(E1 ) + · · · + P(A
Sergei Zuyev
En ) P(En )
TMS-062: Lecture 2 Probability
(4)
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Tree Diagram
Example: In a particular population 55% are male (M) and 10%
of males are color blind (C). It is also known that 5% of female
(F ) are color blind. An individual is chosen randomly. What is
prob. that it is colour blind? (the same as: what is proportion of
colour blind people?)
Events M: the chosen ind. is male, and F : female, form a
partition of Ω. Hence by the Total Probability formula (4)
M)P(M) + P(C
P(C) = P(C
The same is obtained by summing ‘C’-branches in the following
Tree diagram:
P(MC ) = 0.55 · 0.1
C
0.1
0.9
M
0.55 C
0.45 C
F )P(F )
0.05
0.95
F
= 0.1 · 0.55 + 0.05 · 0.45 = 0.0775 = 7.75%
C
Sergei Zuyev
TMS-062: Lecture 2 Probability
Sergei Zuyev
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
P(MC ) = 0.55 · 0.9
P(FC ) = 0.45 · 0.05
P(F C ) = 0.45 · 0.95
TMS-062: Lecture 2 Probability
Sample Space and Events
Probability
Conditional Probability and Independence
Total Probability and Bayes Formula
Bayes Formula
Bayes formula (or Rule) derives prob. P(Ei
forming the partition in terms of P(A
P(Ei
A) =
Example: What is the proportion of males among all colour
blinds?
A) of cases
Ei ):
P(M
P(A Ei ) P(Ei )
P(AEi )
=
P(A)
P(A)
P(A
=
P(A
Sergei Zuyev
P(C
(5)
M) P(M)
M) P(M) + P(C
=
Ei ) P(Ei )
E1 )P(E1 ) + · · · + P(A
P(C
C) =
F )P(F )
0.1 · 0.55
= 0.71 = 71% .
0.1 · 0.55 + 0.05 · 0.45
En )P(En )
TMS-062: Lecture 2 Probability
Sergei Zuyev
TMS-062: Lecture 2 Probability