Reliability Engineering
Markov Model
State Space Method
• Example: parallel structure of two
components
• Possible System States: 0 (both components
in failed state); 1 (component 1 functioning,
component 2 in failed state); 2 (component
2 functioning, component 1 in failed state);
3 (both components functioning).
State Space Diagram
3
2
1
0
Markov Processes
• The event X (t )  j means that the system at time t is in
state j and j = 0,1, 2, …,r.
• The probability of this event is denoted by
Pj (t )  Pr  X (t )  j
• The transitions between the states may be described by a
stochastic process  X (t ); t  0
• A stochastic process satisfying the Markov property is
called the Markov process.
Markov Property
Given that a system is in state i at time t, i.e.
X(t)=i, the future states X(t+v) do not
depends on the previous states X(u), u<t.
PrX (t  v)  j | X (t )  i; X (u )  x(u );0  u  t
 PrX (t  v)  j | X (t )  i
For all possible x(u) and 0≦u<t.
Stationary Transition
Probability
PrX (t  v)  j | X (t )  i  Pij (v)
t , v  0; i, j  0,1,, r
A Markov process with stationary transition
probabilities is often called a process with
no memory.
Properties of Transition
Probabilities
Pij (t )  0 t  0
r
 P (t )  1
j 0
t  0
ij
r
Pij (t  v)   Pik (t ) Pkj (v)
k 0
Chapman-Kolmogorov equation
t , v  0
Transition Rate
In the same way as the failure rate is defined,
the transition rate from state i to state j can
be defined as:
PrX (t  t )  j | X (t )  i
aij  lim
t 0
t
Pij (t ) 
 lim
 Pij (0)
t 0
t
Derivation of State Equation (1)
• From Chapman-Kolmogorov equation
r
Pij (t  t )   Pik (t ) Pkj (t )
k 0
r
 Pij (t ) Pjj ( t )   Pik (t ) Pkj ( t )
k 0
k j
• Substitute
r
Pjj (t )  1   Pjk (t )
k 0
k j
Derivation of State Equation (2)
r
r
k 0
k 0
k j
k j
Pij (t  t )  Pij (t )   Pij (t ) Pjk (t )   Pik (t ) Pkj (t )
After dividing by Δt, letting Δt→0, we get the
state equations.
State Equations
dPij (t )
dt
r
r
k 0
k 0
k j
k j
  Pij (t ) a jk   Pik (t )akj
Pi (0)  1
Pk (0)  0
 k i
Simplified State Equations
Since the initial state is known, the state
equations can be simplified by omitting the
first index i
r
r
k 0
k 0
k j
k j
P j (t )   Pj (t ) a jk   Pk (t )  akj
Pi (0)  1
Pk (0)  0
k  i
State Equations in Matrix
Notation
r
Let a jj   a jk  the sum of departure rates from state j
k 0
k j
Then
where
P (t )  A  P(t )
 a00
 a
 01
A   a02

 
 a0 r
a10
a20

 a11
a21

a12
 a22 



a1r
a2 r

aro 
ar1 
ar 2 

 
 arr 
 P0 (t ) 
 P (t ) 
 1 
P (t )   P2 (t )





 Pr (t ) 
Additional Properties
• Notice that the sums of the columns of the
transition rate matrix add up to zero. Since this
implies that the matrix is singular, the following
additional constraint must be imposed
r
 P (t )  1
j 0
j
• The mean staying time in state j
1
E Tj 
j  0,1,r
a jj
 
Alternative Solution
P (t )  e P(0)
At

k
t
=P(0)  A
k!
k 0
k
where A  I
0
This equation is often computationally convenient way of
approximating P(t).
Example
• Consider a single component with two
states: 1 (the component is working) and 0
(the component is in a failed state). Thus,
a10  
a01  
• The state equations:
 P0 (t )      P0 (t )

 


P
(
t
)



P
(
t
)
  1 
 1  
P1 (0)  1 P0 (0)  0
Example
Since P1 (t )  P0 (t )  1
It can be derived that
A(t )  P1 (t ) 





e (    ) t
Irreducible Markov Process
• A state j is said to be reachable from state i if for
some t>0 the transition rate aij  0
• The process is said to be irreducible if every state is
reachable from every other state.
• For an irreducible Markov process, the following
limits always exist and are independent of the initial
state of the process.
lim Pj (t )  Pj
t 
for j  0,1, 2,
,r
Asymptotic Probabilities
  a00
a
 01
0  A  P   a02


 a0 r
a10
a20
 a11
a21
a12
 a22
a1r
a2 r
aro   P0 



ar1 P1
 
ar 2   P2 
 
 
arr   Pr 
Frequency of Departure from
State j to State k
The unconditional probability of a departure
from state j to state k in the time interval (t,
t+Δt] is
Pr X (t  t )  k    X (t )  j 
 PrX (t  t )  k | X (t )  j PrX (t )  j
 Pjk (t )  Pj (t )
The frequency of departure is defined as

dep
jk
(t )  lim
t 0
Pr  X (t  t )  k 
t
 X (t )  j 
 a jk  Pj (t )
Frequency of Departure from
State j at Steady State
At steady state
 dep
jk  a jk Pj
The total frequency of departure
 dep
j


 r

   a jk  Pj  a jj P j
 k 0 
 k j 
Frequency of Arrival to State j
at Steady State
The frequency of arrival from state k to state j
at the steady state
arr
 kj  akj Pk
The total frequency of arrivals to state j
 arr
j


 r

   akj Pk   a jj Pj
 k 0

 k j

(from state
equations
at steady state)
Visit Frequency
The visit frequency  j to state j is defined as
the expected number of visits to state j per
unit time.
 j 
At steady state!
dep
j

arr
j
 a jj Pj
Mean Duration of a Visit
The total departure rate from state j
r
a jj   a jk
k 0
k j
Since the departure rate is constant, the
duration of a stay in state j should be
exponentially distributed with parameter a jj
Thus, the mean duration of stay is
1
j 
a jj
A Useful Relation
The mean proportion of time the system is
spending in state j ( Pj )
Pj   j j
A special case is the formula for unavailability
under corrective maintenance policy
Q  
System Availability
Let S={1, 2, …, r} be the set of all possible
states of a system. Let B denote the subset
of states in which the system is functioning.
Let F=S-B denote the states in which the
system is failed. Then, the average (or
long-term) system availability and
unavailability are
As   Pj
jB
1  As   Pj
jF