Probability
Tim Marks
University of California San Diego
Tim Marks, Dept. of Computer Science and Engineering
Probability
• Foundations of probability theory were developed
around 1654 by Pascal and Fermat.
– To answer a question from a gambling French
nobleman, the Chevalier de Méré, concerning games
of chance
• We will discuss probability in the context of a
much more noble and important problem:
Listening to music on my iPod
Tim Marks, Dept. of Computer Science and Engineering
1
My iTunes library (100 songs)
Number
of songs
40
10
50
Genre
Decade
Classical
C
Folk
F
Rock
R
20
Seventies
S
8
2
10
50
Eighties
E
15
5
30
30
Nineties
N
17
3
10
Tim Marks, Dept. of Computer Science and Engineering
Random experiment
• Experiment:
– A song is randomly selected from my iTunes library
– (There are 100 songs in my iTunes library)
• Outcomes:
– There are 100 possible outcomes
• o1 = Song 1 is selected
• o2 = Song 2 is selected
• o100 = Song 100 is selected
• Event
– An event is a set of outcomes
• Example: C is the event “a classical song is selected”
C = {o2, o5, o17, …}
! The set of all classical song outcomes
Tim Marks, Dept. of Computer Science and Engineering
2
Events
• Every event A must have
0 ! P(A) ! 1
• Examples of events:
C = a Classical song is selected
F = a Folk song is selected
R = a Rock song is selected
S = a song from the Seventies is selected
E = a song from the Eighties is selected
N = a song from the Nineties is selected
• Examples
– P(F) = ?
– P(N) = ?
Tim Marks, Dept. of Computer Science and Engineering
Two special Events
• The certain event
– The universal set (the entire sample space) of outcomes
" = {o1, o2, o3 , …, o100}
– P(") = 1 ! Probability that when an outcome occurs
(when a song is selected from the iPod), it
will be one of the outcomes in the sample
space (one of the songs on the playlist).
• The impossible event
– The empty set of outcomes (the complement of the set ")
ø = {}
– P(ø) = 0
! Probability that when an outcome occurs
(when a song is selected from the iPod), it
will not be one of the outcomes in the sample
space (one of the songs on the playlist).
Tim Marks, Dept. of Computer Science and Engineering
3
Events formed from other Events
• Intersection
– The event “A and B” = A # B
P(A and B) = P(A # B) = P(A, B)
• Examples:
40
10
50
Genre
Decade
Classical
C
Folk
F
Rock
R
20
Seventies
S
8
2
10
50
Eighties
E
15
5
30
30
Nineties
N
17
3
10
– P(R, E) = P(R # E) = ?
– P(C, F) = ?
Tim Marks, Dept. of Computer Science and Engineering
Events formed from other Events
• Union
– The event “A or B” = A $ B
P(A or B) = P(A $ B)
P(A $ B) = P(A) + P(B) – P(A, B)
• Examples:
40
10
50
Genre
Decade
Classical
C
Folk
F
Rock
R
20
Seventies
S
8
2
10
50
Eighties
E
15
5
30
30
Nineties
N
17
3
10
– P(F $ S) =
P(F) + P(S) – P(F, S)
=?
– P(F $ R) = ?
– P(R $ E) = ?
Tim Marks, Dept. of Computer Science and Engineering
4
Conditional probability
• The probability of A given B
P(A | B) =
P(A, B)
P(B)
• Examples:
P(R, S)
P(S)
P(R | S) =
40
10
50
Genre
Decade
Classical
C
Folk
F
Rock
R
20
Seventies
S
8
2
10
50
Eighties
E
15
5
30
30
Nineties
N
17
3
10
=?
– P(F | N) = ?
Tim Marks, Dept. of Computer Science and Engineering
Statistical Independence
• We say that A and B are independent if
P(A, B) = P(A)P(B)
• Examples:
40
10
50
Genre
Decade
Classical
C
Folk
F
Rock
R
20
Seventies
S
8
2
10
50
Eighties
E
15
5
30
30
Nineties
N
17
3
10
– S and C are independent
– E and R are not independent
– Are S and F independent?
– Are C and R independent?
Tim Marks, Dept. of Computer Science and Engineering
5
Equivalent conditions for
Independence
P(A, B) = P(A)P(B)
%
!
P(A,B)
= P(A)
P(B)
% P(A | B) = P(A)
i.e., knowing whether B happened gives you
no information about whether A happened.
» Review examples from last slide with this interpretation
By similar reasoning,
% P(B | A) = P(B)
Tim Marks, Dept. of Computer Science and Engineering
Bayes’ Rule (a.k.a. Bayes’ Theorem)
• Reverend Thomas Bayes (1702–1761)
• Suppose we know P(A | B).
– What if we want to know P(B | A) ?
– Can infer P(B | A) from our knowledge of:
• P(A | B)
• P(B)
! Likelihood
! Prior
• Derivation of Bayes’ Rule (on board)
• Bayes’ rule:
P(A | B)P(B)
P(B | A) =
P(A)
• Example:
!
P(E | R) =
P(R | E)P(E)
=?
P(R)
Tim Marks, Dept. of Computer Science and Engineering
!
6