Outline
•
•
•
•
•
Review of Probability Theory
M. Sami Fadali
Professor of EE
University of Nevada, Reno
Probability definitions.
Axiomatic definition of probability.
Joint probability.
Marginal probability.
Bayes theorem.
1
2
Classical (Intuitive) Definition of
Probability
Random Signals
• Some physical signals (noise) cannot be
expressed as an explicit mathematical
formula.
• Noise: typically unwanted and to eliminate
it we need to understand it quantitatively.
• Must be described in probabilistic terms.
• We need probability theory.
• Assume all outcomes are equally likely
number of ways event can occur
total number of possible outcomes
3
4
Probability of A = 4 of a kind
Ex. 1.1 Poker Game
Each player gets 5 cards.
number of possible 5-card hands
= number of 52 items taken 5 at a time
• 4 of a kind leaves 48 cards  48 hands
• 13 possible 4 of a kind hands
5
Probability of B = Straight Flush
Problems: Classical Approach
• Allow ace to be counted as high or low
(1,2,…,5), (2,3,…,6),…, (9,10,…,13), (10,11,…,1)
 10 possible hands for each suit  NB = 40
P( B) 

6
NB
N
40
1

 1.54 10 5
2,598,960 64,974
7
• Cannot handle outcomes that are not
equally likely.
• Not practical in cases where there is a large
(finite) number of outputs.
• Cannot handle an (uncountably) infinite
number of outcomes without ambiguity.
8
Relative Frequency Definition of
Probability
• Perform an experiment
times (
• Count the number of times
that
Problems: Relative Frequency
large)
occurs
• Experiments cannot be performed an
infinite (very large) number of times.
• Assumption that the relative frequency
approaches a constant limit (in practice the
ratio hovers around a constant value).
9
Axiomatic Definition of Probability
• Accept a set of axioms based on experience
• Derive a complete theory based on axioms
• Random Experiment: experiment with
nondeteministic outputs.
• Probability Space:
1- Sample space S.
2- Event space E.
3- Probability measure P(E).
11
10
Sample Space S
• Set of outcomes of random experiment
(elementary events )
• Elements: finite, countably infinite, or
infinite.
• Assume elements of sample space are
disjoint.
12
Event
Sample Space Examples
• Event E = subset of sample space (subject
to constraints)
• Example:
S = {Si, i = 1, 2, 3, 4,…, 15}
E1 = {Si, i = 1,3,4}
E2 = {Si, i = even}
• Events need not be disjoint.
Experiment. Single draw from 52-card deck
52 possible outcomes52 elements in S
Experiment. (Vector Outcome) Throw of two
fair dice observe number on each die
66=36 possible outcomes
36 elements in S, each a 2-tuple
(1,1),…,(1,6), (2,1),…,(2,6),…,(6,6)
13
Event Space E
14
Probability Measure
• Set of subsets of S called events.
• Space: The union, intersection,
complement of any event is an event.
• Discrete set S: event space = power set of
S (set of all subsets).
• Continuous Case: event =open interval
(more complicated)
15
• Assign a number in [0,1] to each event in
the event space.
• Satisfies the axioms of probability.
16
Axioms of Probability
Probability Space (S, E, P)
1- Sample space S.
2- Event space E.
3- Probability measure P(E).
• Discrete Problem: finite or countably
infinite outcomes
• Complications for continuous case with
uncountably infinite outcomes.
disjoint (mutually exclusive)
finite or countably infinite
17
Set Operations in Probability
Theory
Union
AB
Intersection
A B
18
Example: Throw of two Dice
Complement
Ac
P  A  B   P  A  P  B   P  A  B 
19
• Sample Space
S = {2,3,4,…,11,12} 11 elements
• Event Set = {S, ,{2},…,{12},{2,3},…,
{2,12},….}
= Power Set of S
• A1 = 7 11  P(A1) = P(7) + P(11)
=6/36 + 2/36 = 2/9
Add for mutually exclusive outcomes.
20
Marginal Probability
Joint Probability
Written in margin outside m by n probability array
• Probability that both events A and B occur
P(AB)
Two-dice Example:
A= even number
B= number less than 5
AB={2, 4}
P(AB) =1/36 + 3/36 = 1/9
21
Joint and Marginal Probabilities
22
Example 1.11: Toss two coins
Event
B1

Bn
Marginal
Probabilities
A1
P(A1B1)

P(A1Bn)
P(A1)
Event
Heads 2
Tails 2
Marginal
Probabilities
:
:
:
:
:
Heads 1
1/4
1/4
1/2
Am
P(AmB1)

P(AmBn)
P(Am)
Tails 1
1/4
1/4
1/2
P(B1)

P(Bn)
Sum=1
Marginal
Probabilities
1/2
1/2
1
Marginal
Probabilities
23
24
Bayes’ Theorem
Conditional Probability
mutually exclusive events
• Probability of event given that event has
occurred (i.e. = certain event)
• Not defined for impossible.
• Definition agrees with relative frequency notion
25
Example: Two Dice
26
Independence
Q. What is the probability of an even number
given that we have a number less than 5?
Occurrence of one event does not affect the
likelihood of the other
= even number
= number less than 5
27
28
Examples: Independence
Examples: Dependence
Draw two cards from a deck.
• Roll die twice.
• Toss two coins.
Q. What is the probability of an Ace for the second
card?
• Draw card from a deck with replacement.
• Draw a card with replacement
• Note: For no replacement, second draw
depends on the outcome of the first.
• Draw a card deck with no replacement
29
References
1. Brown & Hwang, Introduction to Random Signals
and Applied Kalman Filtering, Wiley, NY, 2012.
2. Stark & Woods, Probability and Random Processes,
Prentice Hall, Upper Saddle River, NJ, 2002.
3. R. M. Gray & L. D. Davisson, Random Processes: A
mathematical Approach for Engineers, Prentice Hall,
Englewood Cliffs, NJ, 1986.
31
30