© 2009 Winton
1
Empirical and Discrete Distributions
© 2009 Winton
2
E i i l Distributions
Empirical
Di t ib ti
• An empirical distribution is one for which each
possible event is assigned a probability derived
f
from
experimental
i
l observation
b
i
– It is assumed that the events are independent and the
sum of the probabilities is 1
• An empirical distribution may represent either a
continuous or a discrete distribution
– If it represents a discrete distribution, then sampling is
done “on step”
p
– If it represents a continuous distribution, then sampling
is done via “interpolation”
© 2009 Winton
3
Discrete vs. Continuous Sampling
• The way the empirical table is described usually
d
determines
i
if an empirical
i i l distribution
di ib i is
i to be
b
handled discretely or continuously
discrete description
value probability
10
.1
20
.15
35
.4
40
.3
60
.05
continuous description
value
probability
0 – 10.1
10 – 20.15
20 – 35.4
35 – 40.3
40 – 60.05
© 2009 Winton
4
Empirical Distribution Linear Interpolation
rsample
discrete case continuous case
value prob
value
prob
10 .1
0 – 10- .1
20 .15
10 – 20- .15
35 .4
4
20 – 3535 .44
40 .3
35 – 40- .3
60 .05
40 – 60- .05
60
50
40
30
20
10
x
0
0
.55
1
• To use linear interpolation for continuous sampling, the
discrete points on the end of each step need to be
connected by line segments
– This is represented in the graph by the green line segments
– The steps are represented in blue
© 2009 Winton
5
Sampling On Step
rsample
discrete case continuous case
value prob
value
prob
10 .1
0 – 10- .1
20 .15
10 – 20- .15
35 .4
4
20 – 3535 .44
40 .3
35 – 40- .3
60 .05
40 – 60- .05
60
50
40
30
20
10
x
0
0
.55
1
• In the discrete case, sampling on step is accomplished by accumulating
probabilities from the original table
– For x = 0.4, accumulate probabilities until the cumulative probability
exceeds 0.4
– rsample is the event value at the point this happens
• the cumulative probability 0.1+0.15+0.4 is the first to exceed 0.4,
– the rsample value is 35
© 2009 Winton
6
Sampling By Linear Interpolation
rsample
discrete case continuous case
value prob
value
prob
10 .1
0 – 10- .1
20 .15
10 – 20- .15
35 .4
4
20 – 3535 .44
40 .3
35 – 40- .3
60 .05
40 – 60- .05
60
50
40
30
20
10
x
0
0
.55
1
• In the continuous case, the end points of the probability accumulation are needed
– For x = 0.4, the values are x=0.25 and x=0.65 which represent the points
(.25,20) and (.65,35) on the graph
– From basic college algebra, the slope of the line segment is
(35-20)/(.65-.25) = 15/.4 = 37.5
– slope = 37.5 = (35-rsample)/(.65-.4) so
rsample = 35 - (37.5.25) = 35 – 9.375 = 25.625
© 2009 Winton
Discrete Distributions
• Historical perspective behind the names used with discrete
distributions
– James Bernoulli (1654-1705) was a Swiss mathematician whose book
Ars Conjectandi
j
(published pposthumously
(p
y in 1713)) was the first
significant book on probability
• It gathered together the ideas on counting, and among other things
provided a proof of the binomial theorem
– Siméon-Denis Poisson (1781-1840) was a professor of mathematics at
the Faculté des Sciences whose 1837 text Recherchés sur la probabilité
des jugements en matière criminelle et en matière civile introduced the
discrete distribution now called the Poisson distribution
• Keep in mind that scholars such as these evolved their theories
with the objective of providing sophisticated abstract models of
p
real-world phenomena
– An effort which, among other things, gave birth to the calculus as a
major modeling tool
7
© 2009 Winton
I. Bernoulli Distribution
Bernoulli events, trials, and processes
• Bernoulli event
– One for which the probability the event occurs is p and
the probability the event does not occur is 11-p
p
• The event is has two possible outcomes (usually viewed as
success or failure) occurring with probability p and 1-p,
respectively
• Bernoulli trial
– An instantiation of a Bernoulli event
• Bernoulli process
– A sequence of Bernoulli trials where probability p of
success remains the same from trial to trial
– This means that for n trials,, the pprobabilityy of n
consecutive successes is pn
8
© 2009 Winton
9
Bernoulli Distribution
• Bernoulli distribution
– Given by the pair of probabilities of a Bernoulli event
• Too simple to be interesting in isolation
– Implicitly used in “yes-no” decision processes where the
choice occurs with the same pprobabilityy from trial to trial
• e.g., the customer chooses to go down aisle 1 with
probability p
– It can be cast in the same kind of mathematical notation
used to describe more complex distributions
© 2009 Winton
10
Bernoulli Distribution pdf
pz((1-p)
p)1-z for z = 0,1
,
p(z) =
0
p(z)
otherwise
1

Note: this seemingly
bizarre description is
chosen since it mimics
the case of the
binomial distribution
when n=1 
p

1-p
0
1
z
• The expected value of the distribution is given by
E(X)
( )  (1
( -p))  0  p 1  p
• The standard deviation is given by
σ  (1 - p)(0 - p) 2  p(1 - p) 2  p  (1-p)
• Notational overkill, but useful for understanding other distributions
© 2009 Winton
11
Sampling
• Sampling from a discrete distribution,
distribution requires a function that
corresponds to the distribution function of a continuous distribution f
x
given by
F(x)
( )   f(z)dz
( )

• This is given by the mass function F(x) of the distribution, which is
the stepp function obtained from the cumulative (discrete)
(
) distribution
x
given by the sequence of partial sums
p(z)

0
• For
F th
the B
Bernoulli
lli distribution,
di t ib ti F(x)
F( ) has
h the
th construction
t ti
0
for -  x < 0
F(x)
( ) = 1-pp for 0  x < 1
1
for x  1
– A non-decreasing function (an so can be sampled on step)
© 2009 Winton
12
Graph of F(x) and Sampling Function
F(x)
• Distribution function F(x)
1

p

1-p
0
• Sampling function
l
– for random value x drawn from [0,1), rsample
0 if 0  x < 1-p
rsample =
1
1 if 1-pp  x < 1
– demonstrates that sampling from a
discrete distribution, even one as
simple as the Bernoulli distribution,
can be viewed
ie ed in the same manner as
for continuous distributions
0
z
1
1-p
1
x
© 2009 Winton
13
II Bi
II.
Binomial
i l Distribution
Di t ib ti
• Th
The B
Bernoulli
lli distribution
di ib i represents the
h success or failure
f il
of a single Bernoulli trial
• The Binomial Distribution represents the number of
successes and failures in n independent Bernoulli trials for
some given value of n
– For example, if a manufactured item is defective with probability
p, then the binomial distribution represents the number of
successes and failures in a lot of n items
• Unlike a barrel of apples, where one bad apple can cause
others to be bad
– Sampling
p g from this distribution ggives a count of the number of
defective items in a sample lot
– Another example is the number of heads obtained in tossing a coin
n times
© 2009 Winton
14
Binomial
o a Theorem
eo e (1)
( )
• The binomial distribution gets its name from the
bi
binomial
i l theorem
h
n
 n  k n -k
n
n!
n
a  b      a b where   
0 k
 k  k!(n - k)!
• It is worth pointing out that if a = b = 1,
1 this becomes
n
n
n
n
1 1  2   
0 k
• If S is a set of size nn, the number of k element subsets
of S is given by
n
n!
  
k!(n
( - k)!  k 
© 2009 Winton
15
Binomial
o a Theorem
eo e (2)
( )
• This formula is the result of a simple counting
analysis:
l i there
h are
n!
n  (n - 1)  ...  (n - k  1) 
(n - k)!
ordered ways to select k elements from n
– n ways to
t choose
h
the
th 1st item,
it
(n-1)
( 1) th
the 2nd , and
d so on
– Any given selection is a permutation of its k elements, so
the underlying subset is counted k! times
– Dividing by k! eliminates the duplicates
• C
Consequence: 2n counts
t the
th total
t t l number
b off subsets
b t off
an n-element set
© 2009 Winton
16
Binomial Distribution pdf (1)
• For n independent
p
Bernoulli trials the ppdf of the
binomial distribution is given by
p(z) =
n z
  p (1  p) n  z for z  0, 1, ..., n
z
0 otherwise
h
i
• By the binomial theorem
n
 p(z)  (p  (1 - p))
0
verifying that p(z) is a pdf
n
1
© 2009 Winton
17
Binomial Distribution pdf (2)
• Wh
When choosing
h i z items
it
from
f
among n items
it
with
ith
probability p for an item being defective, the term
n z
  p (1  p) n  z
z
represents the probability that z are defective (and
concomitantly that (n
(n-z)
z) are not defective)
© 2009 Winton
18
E
Expected
t d Value
V l andd Variance
V i
• E(X) = np for a binomial distribution on n items
where probability of success is p
term in common
• The calculation is accomplished by
n
n z
n!
n -z
E(X)   p(z i )  z i     p (1  p)  z  
p z (1  p) n -z  z
1
0 z
1 z! (n - z)!
n
n
 n - 1 z -1
 n - 1 z -1
n -z
 p (1 - p)  np  np   
 p (1 - p) n -z  np(p  1 - p) n -1  np
  
1  z -1
1  z -1
n
n
present in every summand
apply the binomial theorem to this
((note that n-z = ((n-1)) - ((z-1)) )
• It can be similarly shown that the standard
deviation is npp  (1  p)
© 2009 Winton
19
Graph of pdf and Mass Function F(x)
• The binomial distribution with
n=10 and p=0.7 appears as
f ll
follows:
(
(mean
= np = 7)
• Its
It corresponding
di mass function
f ti
F(z) is given by
© 2009 Winton
20
Sampling
Sa
p g Function
u c o
rsample 10
9
8
7
6
5
4
3
2
1
0
0
1
x
• A ttypical
i l sampling
li tactic
t ti is
i to
t accumulate
l t the
th sum
rsample
( )
 p(z)
0
increasing rsample until the sum's value exceeds the random value
between 0 and 1 drawn for x
– The final rsample summation limit is the sample value
– In contrast to a continuous pdf described by some formula, the
function
u ct o for
o a finite
te discrete
d sc ete pd
pdf has
as to be given
g ve in its
ts relational
e at o a
form by a table of pairs, which in turn mandates the kind of
"search" algorithm approach used above to obtain rsample
© 2009 Winton
21
III Poisson Distribution
III.
(values z = 0, 1, 2, . . .)
• With a little work it can be shown that the Poisson
di ib i is
distribution
i the
h limiting
li i i case off the
h binomial
bi
i l
distribution using p= /n  0 as n  
• The expected value E(X) = 
• The standard deviation is λ
z
λ
• The pdf is given by
λ e
p(z)) 
p(
z!
© 2009 Winton
III. Poisson Distribution
history and use
• This distribution dates back to Poisson's 1837 text
regarding
di civil
i il andd criminal
i i l matters, in
i effect
ff scotching
hi the
h
tale that its first use was for modeling deaths from the
kicks of horses in the Prussian army
• In addition to modeling the number of arrivals over some
interval of time the distribution has also been used to
model the number of defects on a manufactured article
– Recall the relationship to the exponential distribution; a Poisson
process has exponentially distributed interarrival times
• In general the Poisson distribution is used for situations
where
h the
th probability
b bilit off an eventt occurring
i is
i very small,
ll
but the number of trials is very large (so the event is
expected to actually occur a few times)
– Less cumbersome than the binomial distribution
22
© 2009 Winton
23
Graph
p of pdf
p and Sampling
p g Function
p(z)
• With  = 2
mean
0.3
0.25
0.2
rsample
0.15
9
0.1
8
0.05
7
0
z
0 1
6
5
4
3
2
1
0
x
0
1
2 3
4 5 6
7 8 9 10 . . .
© 2009 Winton
24
IV Geometric Distribution
IV.
• The geometric distribution gets its name from the geometric series:

1
for r  1,  r n 
,
1- r
0

r
n
n

r

,
0
2
(1 - r)

 (n  1)  r n 
0
1
(1 - r) 2
various
i
flavors
fl
off the
h geometric
i series
i
• The pdf for the geometric distribution is given by
p(z) =
(1  p) z-1  p for z  1,
1 2,
2 ...
otherwise
• The geometric distribution is the discrete analog of the exponential
distribution
– Like the exponential distribution, it is "memoryless"; i.e.,
P(X > a+b | X > a) = P(X > b)
– The geometric distribution is the only discrete distribution with this
property just as the exponential distribution is the only continuous
one behaving in this manner
© 2009 Winton
25
Memoryless
e o y ess Property
ope y
• Being memoryless is characterized by
P(X > a+b | X > a) = P(X > b)
• The interpretation is that if we haven’t had an arrival in “a” seconds, the
probability of an arrival in the next “b” seconds is the same as if the “a”
seconds had not elapsed
– Only the geometric and exponential distributions have this property
• This is NOT a property associated with independent events,
events since among
other things there are three events involved: X > a+b, X > a, and X > b
– Recall that A and B are independent if P(A ∩ B) = P(A)P(B)
– So long as P(B) is non-zero, P(A | B) = P(A ∩ B )/P(B)
• If effect, if A and B are independent events and B has already
occurred then by this equation P(A) is unchanged; i.e.,
occurred,
i e as one would
expect P(A|B) = P(A) when A and B are independent
– The events X > a + b and X > a are certainly NOT independent since
independence would infer P(X > a+b | X > a) = P(X > a+b) – NOT!
© 2009 Winton
26
Expected Value and Standard
Deviation
• With expected value is given by

1
1
E(X)   z(1 - p)  p  p 

2
(1 - 1  p)
p
1
z -1
– by applying the 3rd form of the geometric series
• The standard deviation is ggiven byy
1 p
p
© 2009 Winton
27
Graph
• A plot of the geometric distribution with p = 0.3
0 3 is given by
p(z)
0 35
0.35
0.3
mean
0 25
0.25
0.2
0 15
0.15
0.1
0 05
0.05
z
0
1
2
3
4
5
6
7
8
9
10
© 2009 Winton
28
Utili ti
Utilization
• A ttypical
i l use off the
th geometric
t i distribution
di t ib ti is
i for
f modeling
d li
the number of failures before the first success in a
sequence of independent Bernoulli trials
• This is the scenario for making sales
– Suppose
pp
that the probability
p
y of makingg a sale is 0.3
– Then
• p(1) = 0.3 is the probability of success on the 1st try
• p(2) = (1-p)p = 0.70.3 = 0.21 which is the probability of
failing on the 1st try (with probability 1-p) and succeeding on
y p)
the 2nd ((with pprobability
• p(3) = (1-p)(1-p)p = 0.15 is the probability that the sale takes 3
tries, and so forth
– A random
d
sample
l from
f
the
h di
distribution
ib i represents the
h number
b off
attempts needed to make the sale