APPENDIX B
Poisson process and derivation of Bellman equations
1. Poisson process
Let us first define the exponential distribution.
Definition B.1. A continuous random variable X is said to have an exponential distribution with parameter λ > 0 if its cumulative distribution function (cdf) is given by:
F (t) = P {X < t} = 1 − e−λt
for all t ≥ 0. This implies that the probability density function (pdf) is equal to:
f (t) = λe−λt
for all t ≥ 0. The exponential distribution has mean
Z+∞
1
E [X] =
t f(t)dt =
λ
0
and variance
£ ¤
1
V [X] = E X 2 − (E [X])2 = 2
λ
It is straightforward to show the following proposition:
Proposition B.1. The exponential distribution with parameter (or rate) λ > 0 has the
following properties:
(i) Independence:
P {X > t + t0 } = P {X > t} P {X > t0 }
(ii) Memoryless
P {X > t + t0 | X > t} = P {X > t0 }
∀t, t0 ≥ 0
Observe that the exponential distribution is the unique distribution possessing the memoryless property. Following Ross (1996), let us now define a counting process (and its properties) and then a Poisson process.
395
396
B. POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS
Definition B.2. Let {Xn , n ≥ 1} be a sequence of random variables representing the
inter-event times. Define S0 = 0, Sn = X1 + ... + Xn . Then Sn is the time of occurrence of
the nth event. Define
S(t) = max {n ≥ 0 | Sn ≤ t} , t ≥ 0
Thus S(t) represents the total number of events that have occurred up to time t. A stochastic
process {S(t), t ≥ 0} is called a counting process.
Definition B.3. A counting process {S(t), t ≥ 0} is said to possess:
(a) independent increments if the number of events which occur in disjoint intervals are
independent, i.e. for 0 ≤ t1 ≤ ... ≤ tn , the increments S(t1 ), S(t2 ) − S(t1 ), ..., S(tn ) −
S(tn−1 ) are independent random variables.
(b) stationary increments if the distribution of the number of events which occur in any
interval of time only depends the length of the time interval, i.e. the distribution of
X(t + t0 ) − X(t0 ) is independent of t0 .
A Poisson process is frequently used as a model for counting events occurring one at a
time. Let us now define formally a Poisson process.
Definition B.4. The counting process {S(t), t ≥ 0} is said to be a Poisson process
having rate λ > 0, if:
(i) The process has stationary and independent increments;
(ii) The number of events in any interval of length t is Poisson distributed with mean
λt. That is, for all t ≥ 0,
P {S(t) = n} = e−λt
(λt)n
,
n!
n = 0, 1, ...
(B.1)
We have the following alternative definition:1
Definition B.5. The counting process {S(t), t ≥ 0} with S(0) = 0 is said to be a Poisson
process having rate λ > 0, if:
(i) The process has stationary and independent increments;
1A
function f : R → R is said to be an o(h) function if
lim
h→0
f (h)
=0
h
1. POISSON PROCESS
397
(ii) P {S(h) = 1} ≡ P {S(t + h) − S(t) = 1} = λh + o(h);
(iii) P {S(h) ≥ 2} ≡ P {S(t + h) − S(t) ≥ 2} = o(h).
This definition helps us understand how “randomness in time” can be interpreted. Condition (i) implies that what happens under non-overlapping time intervals is independent.
Condition (ii) states that rate λ is constant over time while condition (iii) says that two (or
more) events cannot occur at the same time. Ross (1996) shows that these two definitions
(i.e. definitions B.4 and B.5) are equivalent.
Example B.1. Consider a person who receives a lot of “junk” mails in his/her mailbox.
Assume that the number of “junk” mails follows a Poisson process having rate 2 per hour.
B.1.1. What is the probability that no “junk” mail arrives between 9 am and 11 am?
This probability is defined by P {S(11) − S(9) = 0}. Using Definition B.4 and in partic-
ular equation (B.1), we obtain (for λ = 2):
0
−2t (2t)
P {S(11) − S(9) = 0} = e
0!
= e−2×2 ∼
= 0.0183.
Observe that, because of time homogeneity, this probability is the same as the probability
that no “junk” mail arrives between 2:00 pm and 4:00 pm or any other times as long as there
are two hours difference.
B.1.2. What is the expected number of “junk” mails that arrive during 8 hours?
The expected number of “junk” mails that arrive during 8 hours is:
E [S(8)] = λ × 8 = 16.
B.1.3. What is the probability that one “junk” mail arrives between 1:00 pm and 2:00
pm and two “junk” mails arrive between 1:30 pm and 2:30 pm?
Using time homogeneity and using hours as units of time, let us start at t = 0 (which
corresponds to 1:00 pm here) so that 2:00 pm corresponds to t = 1 (here, 1 hour corresponds
to 1 unit of time). Then, 1:30 pm corresponds to t = 0.5 and 2:30 pm to t = 1.5. Thus,
the probability that one “junk” mail arrives between 1:00 pm and 2:00 pm and two “junk”
mails arrive between 1:30 pm and 2:30 pm can be written as:
P {S(1) = 1, S(1.5) − S(0.5) = 2} .
398
B. POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS
Using the fact that increments over (0, 0.5], (0.5, 1], and (1, 1.5] are independent, we can
write the following:
P {S(1) = 1, S(1.5) − S(0.5) = 2}
1
P
P {S(0.5) = n, S(1) − S(0.5) = 1 − n, S(1.5) − S(1) = 1 + n} .
=
n=0
Let us explain this last equation. We divide time in three intervals: (0, 0.5], (0.5, 1], and
(1, 1.5], that is between 1:00 pm and 1:30 pm, 1:30 pm and 2:00 pm, 2:00 pm and 2:30 pm.
Then, in order to have 1 “junk” mail between t = 0 (i.e. 1:00 pm) and t = 1 (i.e. 2:00 pm),
it has to be that either no “junk” mail arrives between t = 0 and t = 0.5 (i.e. 1:30 pm) and
1 arrives between t = 0.5 and t = 1 or 1 “junk” mail arrives between t = 0 and t = 0.5 and 0
arrives between t = 0.5 and t = 1. Similarly, in order to have 2 “junk” mails between t = 0.5
and t = 1.5 (i.e. 2:30 pm), it has to be that either no “junk” mail arrives between t = 0.5
and t = 1 and 2 arrive between t = 1 and t = 1.5 or 2 “junk” mails arrive between t = 0.5
and t = 1 and 0 arrives between t = 1 and t = 1.5 or 1 “junk” mail arrives between t = 0.5
and t = 1 and 1 arrives between t = 1 and t = 1.5. By combining all these possibilities, it
has to be that:
P {S(1) = 1, S(1.5) − S(0.5) = 2}
= P {S(0.5) = 0, S(1) − S(0.5) = 1, S(1.5) − S(1) = 1}
+P {S(0.5) = 1, S(1) − S(0.5) = 0, S(1.5) − S(1) = 2} .
In other words, either no “junk” mail arrives between t = 0 and t = 0.5 and 1 “junk” mail
arrives both between t = 0.5 and t = 1 and between t = 1 and t = 1.5, or 1 “junk” mail
arrives between t = 0 and t = 0.5, 0 “junk” mail between t = 0.5 and t = 1 and 2 between
t = 1 and t = 1.5.
1. POISSON PROCESS
399
We have:
1
P
n=0
=
1
P
n=0
=
1
P
P {S(0.5) = n, S(1) − S(0.5) = 1 − n, S(1.5) − S(1) = 1 + n}
P {S(0.5) = n} × P {S(1) − S(0.5) = 1 − n} × P {S(1.5) − S(1) = 1 + n}
e−1
n=0
1
P
(1)n −1 (1)1−n −1 (1)1+n
e
e
n!
(1 − n)!
(1 + n)!
1
n=0 n! (1 − n)! (1 + n)!
µ
¶
1
−3
1+
= e
= 1.5e−3 ' 0.0747.
2
= e−3
Consider a Poisson process {S(t), t ≥ 0} and denote the time of the first event by T1 .
Moreover, for n > 1, let Tn denote the elapsed time between the (n − 1)th and the nth
event. The sequence {Tn , n = 1, 2, ...} is called the sequence of interarrival times. We can
now determine the distribution of the Tn . We have the following result:
Proposition B.2. Tn , n = 1, 2, are independent identically distributed (i.i.d.) exponential random variables having mean 1/λ. This means, in particular, that
P {T1 > t} = P {S(t) = 0} = e−λt
and
P {T2 > t | T1 = t0 } = e−λt
etc. Furthermore, the arrival time of the nth event, also called the waiting time until the nth
event and denoted by Σn , is given by
Σn =
n
P
Ti ,
i=1
n≥1
Σn has a gamma distribution with parameters n and λ. That is, the probability density of
Σn is:
fΣn (t) = λe−λt
(λt)n−1
(n − 1)!
400
B. POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS
2. An intuitive way of deriving the Bellman equations
Let us first derive the expected lifetime-utility of an unemployed worker IU . This analysis
is valid for both search-matching (Part 1) and efficiency wage (Part 2) models. During a small
interval of time dt, the unemployed worker obtains WU and during this time, he/she may
find a job and enjoys an expected lifetime-utility level of IE at time t + dt. The probability
that he/she finds a job is: adt + o(dt) with lim o(dt)/dt = 0. If he/she does not find a job,
dt→0
then he/she enjoys utility IU at time t + dt. Observe that, during this small interval of time
dt, the probability that the unemployed worker leaves unemployment and then straightaway
loses his/her new job is negligible with respect to dt (since it is a term in (dt)2 ). We have:2
IU (t) = WU (t)dt +
1
[adt IL (t + dt) + (1 − adt)IU (t + dt)] .
1 + rdt
This is equivalent to
(1 + rdt) IU (t) = (1 + rdt) WU (t)dt + adt IL (t + dt) + (1 − adt)IU (t + dt)
⇔ rIU (t)dt = WU (t)dt + rWU (t)(dt)2 + adt [IL (t + dt) − IU (t + dt)] + IU (t + dt) − IU (t).
Now, by dividing everything by dt, we obtain (observe that (dt)2 is negligible compared to
dt)
rIU (t) = WU (t) + a [IL (t + dt) − IU (t + dt)] +
IU (t + dt) − IU (t)
.
dt
By taking the limit when dt → 0, we get
·
rIU (t) = WU (t) + a [IL (t) − IU (t)] + IU (t)
since
·
IU (t) ≡
dIU (t)
IU (t + dt) − IU (t)
= lim
.
dt→0
dt
dt
·
In steady-state, IU (t) = 0 and IU (t) = IU and IL (t) = IL . The steady-state lifetime
expected utility of an unemployed worker is given by:
rIU = WU + a (IL − IU ) .
2We
could have discounted in a different way,
IU (t) =
1
[WU (t)dt + adtIE (t + dt) + (1 − adt)IU (t + dt)]
1 + rdt
i.e. before starting the period. In this case, it is easily verified that the Bellman equation would have been
exactly the same.
3. A FORMAL WAY OF DERIVING THE BELLMAN EQUATIONS
401
Let us now determine the expected lifetime-utility of a non-shirking employed worker IL .
We have:
IL (t) = WLN S (t)dt +
£
¤
1
δdt IU (t + dt) + (1 − δdt)ILN S (t + dt) .
1 + rdt
By replicating the same analysis as for the unemployed, we easily obtain for employed
workers:
rIL = WLNS − δ (IE − IU ) .
Let us now focus more specifically on efficiency wage models (Part 2). Determining the
case of shirking is more complicated since shirking workers can lose their jobs either by an
exogenous shock δ or by being caught shirking at rate m. We have
ILS (t) = WLS (t)dt +
£
¤
1
δdt IU (t + dt) + mdt IU (t + dt) + (1 − (δ + m) dt) ILS (t + dt) .
1 + rdt
By replicating the same analysis, we obtain:
¡
¢
rILS = WLS − (δ + m) ILS − IU .
(B.2)
3. A formal way of deriving the Bellman equations
As stated above, changes in employment status are assumed to be governed by a Poisson
process with two states: employed or unemployed. Consider a Poisson process (as defined
by Definition B.4 or Definition B.5). As shown in Proposition B.2, the key feature of these
stochastic processes is that the duration time spent in each state is a random variable with
exponential distribution. More precisely, if we denote the (random) unemployment and
employment duration times by Ta and Tδ , then
F (Ta ) = P {Ta < t} = 1 − e−a Ta
F (Tδ ) = P {Tδ < t} = 1 − e−δ Tδ .
This implies that the probability densities are given by:
f (Ta ) = a e−a Ta
(B.3)
f(Tδ ) = δ e−δ Tδ .
(B.4)
402
B. POISSON PROCESS AND DERIVATION OF BELLMAN EQUATIONS
As a result, the average time spent in each state is equal to (see Definition B.1):
E [Ta ] =
=
Z
+∞
Ta f(Ta )dTa
0
Z
+∞
a Ta e−a Ta dTa
0
=
Z
+∞
0
= −
=
1
a
£
¤+∞
e−a Ta dTa − Ta e−a Ta 0
¤+∞
1 £ −a Ta ¤+∞ £
e
− Ta e−a Ta 0
0
a
since
lim Ta e−a Ta = lim Ta e−a Ta = 0.
Ta →+∞
Ta →0
Similarly,
E [Tδ ] =
Z
+∞
0
1
Tδ f(Tδ )dTδ = .
δ
As above, let us first determine the expected lifetime-utility of an unemployed worker, IU .
It is given by:
IU = ETa
∙Z
Ta
−rt
WU e
−rTa
dt + e
0
¸
IL .
(B.5)
IU is thus the discounted value at time t = 0. The unemployed worker remains unemployed
during a random period of time Ta . During this period, he/she earns wU discounted at rate
r. Then, after this period Ta , he/she becomes employed and obtains an expected utility of
IL discounted at rate r starting at time t = Ta .
By developing (B.5), we obtain:
IU =
+∞
Z ∙Z
0
Ta
0
+∞
¸
Z
−rt
WU e dt f (Ta )dTa + e−rTa IE f(Ta )dTa .
0
Since
Z
0
Ta
−rt
WU e
dt = WU
∙
¸
1 − e−rTa
,
r
3. A FORMAL WAY OF DERIVING THE BELLMAN EQUATIONS
403
and using (B.3), it can be rewritten as:
IU
WU
= a
r
+∞
Z
0
¡
¢
1 − e−rTa e−a Ta dTa + aIL
+∞
Z
e−rTa e−a Ta dTa
0
⎤
⎡+∞
+∞
Z
Z
¢
WU ⎣ ¡ −a Ta
−(r+a)Ta
= a
−e
e
dTa ⎦ + a IL e−(r+a)Ta dTa
r
0
=
0
a
WU
+
IL .
r+a r+a
We finally obtain:
rIU = WU + a (IL − IU ) .
(B.6)
Similarly, we can calculate the expected lifetime-utility of an employed worker IL (non shirking in the efficiency wage model). It is:
∙Z
NS
IL = ETδ
Tδ
WLNS e−rt dt
−rTδ
+e
0
By making exactly the same analysis, one easily obtains:
¸
IU .
rILNS = WLNS − δ (IL − IU ) .
(B.7)
(B.8)
For the efficiency wage model, we need to consider the case of a shirker. It is more complicated
since, when employed, a shirker can lose his/her job because either he/has been caught
shirking or the job has been destroyed. Denote by Tm the (random) length of time until the
next control of shirking occurs. This implies that Ta is still the (random) unemployment
duration time whereas min(Tδ , Tm ) is now the employment duration time for a shirker. Since
we know (see, for example, Kulkarni, 1995, ch. 5) that min(Tδ , Tm ) is a random variable
characterized by an exponential distribution of parameter δ + m, i.e.
F (min(Tδ , Tm )) = P [min(Tδ , Tm ) < t] = 1 − e−(δ+m) min(Tδ ,Tm )
then the expected lifetime-utility of a shirker ILS is equal to:
#
"Z
min(Tδ ,Tm )
ILS = E
WLS e−rt dt + e−r min(Tδ ,Tm ) IU .
0
By doing exactly the same kind of manipulations as above, we obtain (B.2).