In the Name of God, the Compassionate, the Merciful
Stochastic Processes
Department of Computer Engineering
Sharif University of Technology
Fall 2009
Homework #4
Due : Final Exam Date
1) Consider a production line where each manufactured item may be defective with probability p
∈ (0, 1). The following inspection plan is proposed with a view to detecting defective items
without checking every single one. It has 2 phases: In phase A, the probability of inspecting an
article is r ∈ (0, 1). In phase B, all the articles are inspected. One switches from phase A to
phase B as soon as a defective item is detected. One switches from phase B to phase A as soon
as a sequence of N successive acceptable items has been found. Find the long-run proportion of
items inspected. (Hint : Try to take states carefully such that long-run proportion of visiting a
set of states equals to the long-run proportion of items inspected).
2) Consider a Markov process with the transition probability diagram shown below.
a. Under what conditions on the a's is the process recurrent?
b. Under what conditions on the a's is the process positive recurrent?
c. Find the equilibrium distribution in terms of a's.
[ ]
0 1 0
3) a. Consider a Markov chain with transition matrix P= 0 0 1 and initial distribution
1 0 0
0 , if X n=1
0=1 /3, 1/3, 1/3 . For each n, Y n=
. Show that Y 0 ,Y 1 , ... is not a
1 , otherwise
markov chain.
{
}
b. Let  X 0 , X 1 , ... be a Markov chain with transition matrix P. Define Y 0 ,Y 1 , ... by
defining Y n= X 2n . Is Y 0 ,Y 1 , ... a Markov chain? If so, find its transition matrix (in
terms of P).
4) Two players, A and B, play the game of matching pennies: at each time n, each player has a
penny and must secretly turn the penny to heads or tails. The players then reveal their choices
simultaneously. If the pennies match (both heads or both tails), Player A wins the penny. If the
pennies do not match (one heads and one tails), Player B wins the penny. Suppose the players
have between them a total of 5 pennies. If at any time one player has all of the pennies, to keep
the game going, he gives one back to the other player and the game will continue.
a. Show that this game can be formulated as a Markov chain.
b. Is the chain irreducible? Is it aperiodic?
c. If Player A starts with 3 pennies and Player B with 2, what is the probability that A will
lose his pennies first? (Hint : Set a(i) as the probability that A will lose first starting in
state i).
5) Poisson arrivals with rate λ join a queue in front of two parallel servers A and B, with
exponentially distributed service times with rates μA and μB, respectively. A customer that upon
arrival finds the system empty is allocated to the server that has been idle for the longest time.
Otherwise, the head of the queue takes the first free server.
a. Define the states describing the system as a Markov chain and draw the statetransition diagram.
b. Find the stationary distribution.
6) Consider an M/M/1 queue that can accommodate at most K customers in the system (queued
or in service), and suppose that a customer that arrives and finds the system full is not lost, but
stored in an external queue with infinite space as shown in the figure.
The transition of a customer from the external to the server queue is instantaneous. Therefore, a
customer that arrives when the number of customers in the server queue is less than K enters
instantaneously the server queue. Similarly, when a customer departs from the server queue, the
customer at the head of the external queue moves to the server queue instantaneously. This twoqueue system can be modeled as a two dimensional Markov chain with states (i, k) where 0 ≤ i
≤∞ and 0 ≤ k ≤ K.
a. Draw the state transition diagram of the two-dimensional chain.
b. Find the (steady-state) probability p(i, k) that there are i customers in the external queue
and k customers in the server queue.
c. Find the average number of customers in the server queue, and the average number of
customers in the external queue.
d. Find the average total time that a customer spends in the two-queue system.
7) Poisson arrivals with rate λ join a queue in front of two parallel servers denoted as A and B,
with exponentially distributed service times with rates μA and μB, respectively. When the system
is empty, arrivals go into server A with probability α and into B with probability 1-α. Otherwise,
the head of the queue takes the first free server.
a. Define the states describing the system as a Markov chain, draw the state-transition
diagram and setup up the balance equations.
b. Find the stationary distribution.
c. In terms of the probabilities in part b, find (i) the average number of customers in the
system, and (ii) the average number of idle servers.
d. In terms of the probabilities in part b, what is the probability that an arbitrary arrival
will get serviced in server A?
8) The state transition diagram for a continuous time Markov chain is given below where λ is
the arrival rate and μ is the service rate :
Solve this system to obtain its equilibrium state probabilities stating the condition (if any) under
which such a solution will exist.
[Note that the state probabilities will have to be found for all states 0, 1, 1', 2, 2' ........].
You should find p0 and pn' explicitly and at least give expressions which may be evaluated for
pn.]
9) In an M/M/1 queue, once the system becomes empty, the server does not start serving again
until the number of jobs in the system becomes 3. Otherwise, the system behaves normally with
an average arrival rate of λ and an average service rate of µ.
a. Draw a State Transition Diagram for the system.
b. Find the state probabilities of the system under equilibrium conditions and use these
to give the probability of finding k users in the system for k=0,1,2,3....... Expressing
your results in terms of λ, µ or ρ=λ/µ.
c. Does the server work more or less here than in a normal M/M/1 queue? (Give a
quantitative answer based on probability values.)
10) Consider the open network of infinite capacity, single-server queues with exponential
servers as shown in the below figure. Note that the external arrivals may come either from point
A or from point B with the rates as indicated in the figure. The service rates of the queues are
also indicated in the figure.
Use ρ=λ/µ in answering the following parts.
a. What is the joint state distribution P(n1,n2,n3,n4) of the number in each queue?
b. What is the mean number in the system?
11) Consider a M/-/1/2 queue where the service facility is modeled as shown below. After
finishing service at Stage 1, the job either exits the system with probability 0.5 or moves to
Stage 2 with probability 0.5, as shown. Note that a job entering the service facility always gets
served at Stage 1 first. Assume that the arrival rate to the queue is λ.
a. Draw an appropriate state transition diagram for the system.
b. Write the balance equations and solve these for the state probabilities using ρ=λ/µ
c. What is the mean queuing delay seen by an arrival entering the system?
(Hint: You may try to model the whole system using a Continuous Markov Model).
12) Consider the closed queuing network of single server queues with exponentially distributed
service times, as shown in the figure below
The average service rates of Q1, Q2 and Q3 are respectively µ1 = 1, µ2 = 2, and µ3 = 2. The system
has a total user population of 4. Obtain the following:
a. The state probability distribution for the queuing network
b. The actual throughput of each queue
The mean number in each queue