Calhoun: The NPS Institutional Archive
DSpace Repository
Faculty and Researchers
Faculty and Researchers Collection
1975-09
A flow-graph formula for the stationary
distribution of a Markov Chain
Shubert, Bruno O.
IEEE Transactions on Systems, Man, and Cybernetics, September 1975.
http://hdl.handle.net/10945/41310
Downloaded from NPS Archive: Calhoun
565
CORRESPONDENCE
to avoid any risk of damaging the experimental system we did
not make any attempt to optimize the speed. Excluding probing,
we believe that in a production environment the entire operation
could be speeded up by a factor of three, reducing the time
required to a maximum of 20 s. Similarly, the mechanical probing
could have been completed in a maximum of 15 s.
IV. EXTENSIONS
[9] M. Ejiri, T. Uno, H. Yoda, T. Goto, and K. Takeyasu, "A prototype
intelligent robot that assembles objects from plane drawings," IEEE
Trans. Comput., vol. C-21, pp. 161-170, Feb. 1972.
[10] A.
P. Ambler, al.,Int."AJoint
versatile
computer controlled pp. 298Conf. Artificial Intelligence, assembly
system," in Proc.et3rd
316, Aug. 1973.
[I11] R. Bolles and R. Paul, "The use of sensory feedback in a programmable assembly system," Stanford Artificial Intelligence Laboratory,
Stanford, Calif., Memo AIM-220, Oct. 1973.
[12] S. S. M. Wang, "Theoretical study of parts orientation in a manipulator's hand," in Proc. IEEE Symp. Automatic Control, Milwaukee,
Wis., pp. 226-234, Mar. 1974.
[13] H. Yoda, S. Ikeda, and M. Ejiri, "A new attempt of selecting objects
When a TV camera is added to our system, it will become
using a hand-eye system," Hitachi Rev., vol. 22, Sept. 1973.
R. Birk, "A computer-controlled rotating-belt hand for object
possible to scan a part in the orienting box and provide silhouette [14] J.orientation,"
IEEE Trans. Syst., Man, Cybern., vol. SMC-4, pp.
image data to the computer program. Each probe would then
186-191, Mar. 1974.
W. B. Heginbotham, P. W. Kitchin, and A. Pugh, "Visual feedback
consist of taking one picture element and comparing it to a [15] applied
to programmable assembly machines," in Proc. 2nd Int. Symp.
Industrial Robots, lIT Research Institute, Chicago, Ill., pp. 77-88,
threshold to establish the presence or absence of the part. This
1972.
seqencofofbinary
inay tests
tets may
ay be
b contrasted
cotrased with
ith the
he twowo- [16]
16]May
shortshor
sequence
G. Boothroyd
and A. H. Redford, "Statistical distributions of natural
resting aspects of parts for automatic handling," Manufacturing Eng.
dimensional image processing used by Heginbotham [15].
Trans., Society of Manufacturing Automation, vol. 1, 1972.
Unlike mechanical probing, visual probing would be purely [17] D.
E. Knuth, The Art of Computer Programming, vol. 1. Reading,
Mass.: Addison-Wesley, 1968, pp. 402-404.
two-dimensional, but for the overwhelming majority of mechanical parts, two-dimensional probing is sufficient to determine
a unique three-dimensional orientation, primarily because the
orienting box so severely limits the number of possible orientations. The great advantage of video is that it should reduce to
much less than 1 s the time required for probing.
Once visual probing is achieved, other extensions of the method A Flow-Graph Formula for the Stationary Distribution of a
will be concerned with finding ways in which the programming
Markov Chain
burden can be significantly diminished. One such extension
BRUNO O. SHUBERT
would be to automate the choice of probe points. The system
would repeatedly drop the same part in the box and scan the
Abstract-It is shown that a stationary distribution of a regular
resulting image until eventually all the possible orientations were Markov chain can be obtained directly from its transition graph. The
physically enumerated. A program could then compute set technique is similar to signal flow-graph methods, however, it uses trees
intersections of various combinations of the silhouette images to of the graph rather than loops. The proof is direct and simple.
obtain effective probe points.
It is hoped that ultimately a sufficiently sophisticated data
Let P = [pij] be a transition probability matrix of a regular
representation of complex part shapes can be developed so that (see [2]) Markov chain and let P = (,ul, - * ,,,) be its stationary
it will be possible to automate the choice of probe points com- distribution. The transition graph g = (V,A) of this chain is a
pletely, without any need for physically enumerating the stable directed graph (digraph, see [1]), whose set of vertices V cororientations. When such a representation becomes available, it responds to the set of states and whose set of arcs A c V x V
might also be used as a basis for automatically generating the is defined by the relation
programs which reorient the part into the desired position.
c A,
if and only if i # j, pi > 0.
ACKNOWLEDGMENT
Thus the one-step transition from a state to itself is disregarded
Indispensable assistance was provided by all the members of in this graph.
the Automation Research Group and several people from the
If (i,]) E A call the vertex j a successor of the vertex i. A
Central Scientific Service Department. In particular, we would sequence (il.
,im) of distinct vertices, where each ik+ 1 is a
like to single out George Folchi, who made the first orienting successor of ik, k = 1, *, m - 1, is called a path, if i1 is also a
box, and Wayne Book, who performed the initial feasibility successor of i4, it is a cycle.
tests. Helpful suggestions were made by Peter Will and John
Consider now a subgraph f = (V,B), B c A such that
Griffith.
1) each vertex has at most one successor;
REFERENCES
2) f has no cycles;
3) f is maximal, i.e., no further arcs can be added without
[1] P. M. Will and D. D. Grossman, "An experimental system for computer controlled mechanical assembly," IBM T. J. Watson Research
violating 1) or 2).
(i)
-
Center, Yorktown Heights, N.Y., Res. Rep. RC-4922, July 1974, to
be published.
[2] Industrial Robots-A Survey. Bedford, England: International
Fluidics Services Ltd., 1972.
[3] Proc. 1st National Symp. Industrial Robots, IIT Research Institute,
Chicago, 1ll., Apr. 1970. Proc. 2nd Int. Symp. Industrial Robots,
lIT Research Institute, Chicago, Ill., May 1972. Proc. 3rd Int. Symp.
Industrial Robots, Zurich, May 1973.
[4] Automated Assembly, 18 vol. series of monographs, Institution of
Production Engineers, London, 1970.
[5] W. V. Tipping, An Introduction to Mechanical Assembly. London:
Business Books, 1969.
[6] H. A. Ernst, "MH-1, a computer operated mechanical hand," Ph.D.
dissertation, Dep. Elec. Eng., MIT, Cambridge, Mass., Dec. 1961.
[7] T. Goto, K. Takeyasu, T. Inoyama, and R. Shimomura, "Compact
packaging by robot with tactile sensors," in Proc. 2nd Int. Symp.
Industrial Robots, IIT Research Institute, Chicago, Ill., May 1972.
[8] J. Feldman, et al., "'The use of vision and manipulation to solve the
"instant insanity" puzzle," presented at 2nd Int. Joint Conf. Artificial
Intelligence, The British Computer Society, Sept. 1971.
Clearly, such a subgraph is a spanning tree of g with all the arcs
directed toward a single vertex. It may be appropriately called a
confluence with the particular vertex being its sink.
Proposition: Let
(iO be the class of all confluences with sink i
and let, for each confluence f = (V,B),
n()=
Pn
fl
p3
(L,J) eB
Manuscript received November 7, 1974; revised March 6, 1975. This
work was supported by Research Foundation Funds of the Naval Postgraduate School.
The author is with the Department of Operations Research and Administrative Services, Naval Postgraduate School, Monterey, Calif. 93940.
566
IEEE TRANSACTIONS ON SYSTEMS, MAN, AND CYBERNETICS, SEPTEMBER 1975
Then the stationary probabilities .l *, u,, are given by
p(f)
i
f em
(1)
F
P
/
where C > 0 is a normalizing constant determined from Iu1 +
.+Iln
=
1.
Proof: Let h = (V,E) be a subgraph of g (not necessarily
a confluence), let
p(h) = 7 Pij
(i,j)eE
and h ± (i,j) denote a subgraph obtained from h by adding or
removing arc (i,j).
=
If pi = Ife.,D p(f), then it is easily seen that, for]1,j
,\
n
i EPii= E
n
p(h)
ipi =
heSj
ihShj
i=l
i_j
where
il
(j,i): f eF'j, iu V - {j}}
{f + (i,j): f Di, i V j}}.
S = {f+
Rj
=
heRj
-
E
E
p(h)
(2)
.2
P
0
0
.8
.2 .1 0 .7
.4 .6
0
0
I
Now if h e Sj, let us say h = f + (j,i), then h must contain a
.
s .3
single cycle (i,- *,k,j). Hence f' = h - (k,j) c Dk since the
Fig. 1.
vertex k could not have had any other successor than j. Thus
h = f' + (k,j) E Ri. Conversely, if h E Rj, let us say h = f +
(i,j), then by removing the arc (j,k) with k being the successor of
j we obtain a confluence with sink j. Hence h e Sj. However,
Sj = Rj implies that the left sides of the (2) are equal. This, in
turn, is equivalent to p = pP since the matrix P is stochastic,
and the proof is complete.
.024 4
.040 +
.008 *
.028 =.100
Comment: Notice that the formula (1) makes lit proportional
to the sum of the products of the off-diagonal entries of P, each
product contains exactly n - 1 different entries, and no two
products contain the same set of entries. Notice also that if the
transition graph g is complete (all Pij > 0), then the sum consists of exactly n"'2 terms. All this is hardly surprising since the
*
stationary distribution p, being the solution of a system of n + 1
096
.160 = .256
.048 +
.168 = .216
linear equations p = pP, 1'l + ... + An = 1, must be proportional to the vector of principal cofactors of the matrix
P - I. This is because, for a stochastic matrix P, each offdiagonal cofactor of P - I is equal to the principal cofactor in
.oo +.288 +.256 +.216 =.NO
the same row, the latter being calculated by (1). On the other
hand, the theoretically interesting feature of the formula (1) is
.064
.224 = .288
+
that it relates the stationary probabilities directly to the topological structure of the transition graph. Thus it is akin to the
uL= (.100X.860, .256/.860, .216/.860, .288/.860)
signal flow-graph techniques of Mason and Coates [1]. As for
Fig. 2.
the computational aspect of the formula (1), it clearly rests upon
the problem of identifying all the confluences of a given transition be computed from stationary probabilities of a modified chain
graph. Although techniques for generating all spanning trees (see [2, sec. 6.2]). Finally, if the chain has several ergodic sets of
of a given graph (and thereby identifying all those that are states, each set can be treated separately as a regular chain.
confluences of a digraph) are available (see [1 ]) we suspect that
'
E
C matrix
i a faand transition
for larger n the computational requirements may soon become tion
probability
graph shown in Fig. 1.
excessive. However, for transition graphs with predominantly The confluences, together with the corresponding products p(f),
tree-like structure, the formula offers a definite advantage over and the computation of the stationary distribution are shown in
conventional methods of solving the system p = pP.
Fig. 2. The sinks are circled.
It may perhaps be worth mentioning that the method could
also be adapted to Markov chains that are not regular. If the
ACKNOWLEDGMENT
Thauorwsetoxpssisprcaintoheefes
chain contains transient states, then each confluence whose sink
corresponds to such a state fails to span all the vertices of g. o hi epu omns
Hence setting p(f) = 0 for these confluences (1) still applies.
If the chain is cyclic then, of course, a stationary distribution
REFERENCES
does not exist. However, the solution of p = pP is then the [1] D. E. Johnson and J. R. Johnson, Graph Theory. New York: Ronald,
Cesaro limit of occupation probabilities and (1) produces those.
1972.
Kemeny and J. L. Snell, Finite Markov Chains. New York:
...
If the chain is absorbing, then the absorption probabilities
can ~[2] J.VanG.Nostrand,
Reinhold, 1960.
-