J. Appl. Prob. 43, 127–140 (2006)
Printed in Israel
© Applied Probability Trust 2006
GAMBLING TEAMS AND WAITING TIMES FOR
PATTERNS IN TWO-STATE MARKOV CHAINS
JOSEPH GLAZ,∗ University of Connecticut
MARTIN KULLDORFF,∗∗ Harvard Medical School and Harvard Pilgrim Health Care
VLADIMIR POZDNYAKOV,∗ ∗∗∗ University of Connecticut
J. MICHAEL STEELE,∗∗∗∗ University of Pennsylvania
Abstract
Methods using gambling teams and martingales are developed and applied to find
formulae for the expected value and the generating function of the waiting time to
observation of an element of a finite collection of patterns in a sequence generated by a
two-state Markov chain of first, or higher, order.
Keywords: Gambling; team; waiting time; pattern; success run; failure run; Markov
chain; martingale; stopping time; generating function
2000 Mathematics Subject Classification: Primary 60J10
Secondary 60G42
1. Introduction
How long must we observe a stochastic process with values from a finite alphabet until we
see a realization of a pattern that belongs to a specified collection, C, of possible patterns?
For independent processes this is an old question; in some special cases it was even considered
by Feller (1968, pp. 326–328). Nevertheless, in the context of more general processes, or even
for Markov chains, there are many natural problems that have not been fully addressed. The
main goal here is to show how some progress can be made by further developing the martingale
methods that were introduced by Li (1980) and Gerber and Li (1981) in their investigation of
independent sequences. Their key observation was that information on the occurrence times of
patterns can be obtained from the values assumed by a specially constructed auxiliary martingale
at a certain well-chosen time.
In the case of Markov chains of first, or higher, order, this observation is still useful, but to
utilize it requires a rather more elaborate plan for the construction of the auxiliary martingale.
This construction depends in turn on several general devices that seem more broadly useful;
these include ‘teams of gamblers,’ ‘watching then betting,’ ‘reward matching,’ and a couple of
other devices that will be described shortly.
Before giving that description, we should note that pattern matching has been studied using
many other techniques. The combinatorial methods of Guibas and Odlyzko (1981a), (1981b)
Received 4 April 2003; revision received 14 November 2005.
∗ Postal address: Department of Statistics, University of Connecticut, 215 Glenbrook Road, U-4120, Storrs, CT
06269-4120, USA.
∗∗ Postal address: Department of Ambulatory Care and Prevention, Harvard Medical School, 133 Brookline Avenue,
Boston, MA 02215-3920, USA.
∗∗∗ Email address: [email protected]
∗∗∗∗ Postal address: Department of Statistics, Wharton School, Huntsman Hall 447, University of Pennsylvania,
Philadelphia, PA 19104, USA.
127
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
128
J. GLAZ ET AL.
are particularly effective, and there are numerous treatments of pattern matching problems using
probabilistic techniques, such as Benevento (1984), Biggins and Cannings (1987a), (1987b),
Blom and Thorburn (1982), Breen et al. (1985), Chryssaphinou and Papastavridis (1990), Han
and Hirano (2003), Pozdnyakov et al. (2005), Pozdnyakov and Kulldorff (2006), Robin and
Daudin (1999), Stefanov (2003), and Uchida (1998). One of the more general techniques is the
Markov chain embedding method introduced by Fu (1986), which has been further developed by
Antzoulakos (2001), Fu (2001), Fu and Chang (2002), and Fu and Koutras (1994). The approach
of Stefanov (2000) and Stefanov and Pakes (1997) also uses Markov chain embedding, though
their method differs substantially from that of Fu. Only a few investigations have considered
waiting time problems for higher-order Markov chains, and these have all focused on specific
waiting times such as the ‘sooner or later’ problem studied by Aki et al. (1996).
2. Expected waiting time until a pattern is observed
We take {Zn , n ≥ 1} to be a Markov chain with two states, S and F , which may model
‘success’ and ‘failure.’ We suppose the chain to have the initial distribution with P(Z1 = S) =
pS and P(Z1 = F ) = pF , and the transition matrix
pSS pFS
,
pSF pFF
where pSF is shorthand for P(Zn+1 = F | Zn = S). We then consider a collection, C, of
finite sequences Ai , 1 ≤ i ≤ K, from the two-letter alphabet {S, F }. If τAi denotes the time
elapsed before the pattern Ai is first observed as a completed run in the series Z1 , Z2 , . . . , then
the random variable of main interest here is τC = min{τA1 , . . . , τAK }, the time at which we
first observe a pattern from C. Throughout our discussion we assume that no pattern from C
contains another pattern from C as an initial segment. Naturally, this assumption entails no
loss of generality.
2.1. A run of ‘failures’ under a Markov model
To illustrate the construction, we first consider the rather easy case in which K = 1 and
the pattern A1 is a run of r consecutive F s. Thus, we will compute the expected value of
τ = τA1 , the time of the first completion of a run of r ‘failures’ under our two-state Markov
model. This example can be treated in several ways, and offers a useful benchmark for more
challenging examples.
We consider a casino where gamblers may bet in successive rounds on the output of our
given two-state Markov chain, and assume that the casino is fair in a sense that we will soon
make precise. We then consider a sequence of gamblers, one of whom arrives just before each
new round of betting. Thus, gambler n + 1 arrives in time to observe the result of the nth trial,
Zn , and we assume that he bets a dollar on the event that the next trial yields an F . If Zn+1 = S,
he loses his dollar and leaves the game. If he is lucky, and Zn+1 = F , then he wins 1/pSF
when Zn = S and wins 1/pFF when Zn = F . This is the sense in which the casino is fair; the
expected return on a one-dollar bet is one dollar.
After this gambler collects his money, he then bets his entire capital on the event that
Zn+2 = F . Again, if Zn+2 = S then the gambler leaves the game with nothing, while if
Zn+2 = F then the gambler wins this round and his capital is increased by a factor 1/pFF .
Successive rounds proceed in the same way, with a new gambler arriving at each new round
and with the gamblers from earlier periods either continuing to win or else going broke and
leaving.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
129
We need to be precise about how this process ends. If gambler n + 1 begins by observing
Zn = S, then he bets until he either goes broke or observes r successive F s, and if he begins
by observing Zn = F , then he bets until he either goes broke or observes r − 1 successive F s.
Once some gambler stops without going broke, all of the gambling stops.
Finally, we let Xn denote the casino’s net gain at the conclusion of round n. Since each bet
is fair and since the bet sizes depend only on the previous observations, the sequence Xn is a
martingale with respect to the σ -field generated by {Zn , n ≥ 1}. Now, to obtain a relation that
will determine E[τ ], we just need to consider the casino’s net gain when the gambling stops,
Xτ . By calculating E[Xτ ] in two ways we will then obtain the expected value of the time τ .
At time τ , many gamblers are likely to have lost all their money; only those who entered
the game after round number τ − r − 1 have any money. We thus face two different ending
scenarios. First, it could happen that we have a block (denoted by F (r) ) of r instances of F
that occurs at the very beginning of the sequence {Zn , n ≥ 1}. Second, it could happen that
we end with SF (r) , an S followed by a block of r instances of F . Obviously we do not need to
consider the possibility of ending with FF (r) = F (r+1) , since by definition F (r) cannot occur
before time τ .
When we sum the wins and losses of all of the gamblers, we then find that the value of the
stopped martingale Xτ is exactly given by
⎧
1
1
1
⎪
⎪
in the first scenario,
⎪
⎨τ − 1 − p r−1 − p r−2 − · · · − pFF
FF
FF
Xτ =
1
1
1
1
⎪
⎪
⎪
− r−1 − r−2 − · · · −
in the second scenario,
⎩τ − 1 −
r−1
p
pSF pFF
pFF
pFF
FF
which can be written more succinctly as
⎧
r−1
⎪
1 − pFF
⎪
⎪
τ
−
1
−
⎪
⎨
r−1
pFF
(1 − pFF )
Xτ =
r−1
⎪
1 − pFF
1
⎪
⎪
−
τ
−
1
−
⎪
⎩
r−1
r−1
pFF
(1 − pFF )
pSF pFF
in the first scenario,
in the second scenario.
Since E[τ ] < ∞ and the increments of Xn are bounded, the optional stopping theorem for
martingales (see, e.g. Williams (1991, p. 100)) tells us that 0 = E[X1 ] = E[Xτ ]. From this
identity and the formula for Xτ , algebraic simplification gives us
r−1
r−1 1 − pFF
1 − pFF
1
r−1
.
+ (1 − pF pFF
)
+
E[τ ] = 1 + pF
r−1
r−1
1 − pFF
pSF pFF
pFS pFF
2.2. Second step: a single pattern
We now consider the more subtle case of a single (nonrun) pattern A of length r, and for
specificity we assume that the pattern begins with an F ; hence, A = FB with B ∈ {S, F }r−1 .
As before, we consider a sequence of gamblers, but this time we need to consider three different
ending scenarios:
1. A occurs at the beginning of the sequence {Zn , n ≥ 1}.
2. The pattern SA occurs.
3. The pattern FA occurs.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
130
J. GLAZ ET AL.
The probability, p1 , of the first scenario is trivial to compute, but we then face a problem.
We do not know the probability of the pattern SA appearing earlier than the pattern FA, so the
probabilities of the second and third ending scenarios are not readily computable. To circumvent
this problem, we introduce two teams of gamblers.
2.3. Rules for the gambling teams
1. A gambler from the first team who arrives before round n watches the result of the nth
trial and then bets y1 dollars on the first letter in the sequence A. If he wins he then bets all of
his capital on the next letter in the sequence A, and continues in this way until he either loses his
capital or observes all of the letters of A. Such players are called ‘straightforward gamblers.’
2. The gamblers of the second team make use of the information that they gather. On the
one hand, if gambler n + 1 observes that Zn = S just before he begins his play, then he bets
like a straightforward gambler, except that he begins by wagering y2 dollars on the first letter of
pattern A. On the other hand, if he observes that Zn = F when he first arrives, then he wagers
y2 dollars on the first letter of the pattern B. He then continues to wager on the successive
letters of B until either he loses or he observes B. Such players are called ‘smart gamblers.’
The two gambling teams continue their betting until one team stops. At that time, all
gambling stops and we consider the wins and losses. Only those gamblers who entered the
game after the time τ − r − 1 will have any money, and the amount they have will depend on
the ending scenario. If we let Wij yj denote the amount of money that team j ∈ {1, 2} wins in
scenario i ∈ {1, 2, 3}, then the values Wij are easy to compute and, in terms of these values, the
stopped martingale Xτ , which represents the casino’s net gain, is given by
⎧
⎪
⎨(y1 + y2 )(τ − 1) − y1 W11 − y2 W12 in the first scenario,
Xτ = (y1 + y2 )(τ − 1) − y1 W21 − y2 W22 in the second scenario,
⎪
⎩
(y1 + y2 )(τ − 1) − y1 W31 − y2 W32 in the third scenario.
Now, if we take (y1∗ , y2∗ ) to be a solution to the system
y1∗ W21 + y2∗ W22 = 1,
y1∗ W31 + y2∗ W32 = 1,
we see that, with these bet sizes, we have a very simple formula for Xτ :
⎧
∗
∗
∗
∗
⎪
⎨(y1 + y2 )(τ − 1) − y1 W11 − y2 W12 in the first scenario,
Xτ = (y1∗ + y2∗ )(τ − 1) − 1
in the second scenario,
⎪
⎩ ∗
in the third scenario.
(y1 + y2∗ )(τ − 1) − 1
The optional stopping theorem then gives us
0 = (y1∗ + y2∗ )(E[τ ] − 1) − p1 (y1∗ W11 + y2∗ W12 ) − (1 − p1 ),
where p1 is the probability of scenario one occurring. We therefore find that
E[τ ] = 1 +
p1 (y1∗ W11 + y2∗ W12 ) + 1 − p1
.
y1∗ + y2∗
(1)
Formula (1) is more explicit than it may at first appear. In the typical case, the calculation
of p1 , {Wij , 1 ≤ i ≤ 3, 1 ≤ j ≤ 2}, and {yj∗ , 1 ≤ j ≤ 2} is genuinely routine, as can be seen
in the following example.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
131
2.4. Example: waiting time until FSF is observed
Here, our straightforward gamblers bet y1 dollars on FSF without regard for the preceding
observation, while the smart gamblers bet y2 dollars on FSF if they observed S before placing
their first bet, and bet y2 dollars on SF if they observed F . The three ending scenarios are
now the occurrence of FSF at the beginning (scenario one), the occurrence of SFSF at the end
(scenario two), or the occurrence of FFSF at the end (scenario three). The 3 × 2 ‘profit matrix’
{Wij } is then given by
⎛
1
pSF
⎜
⎜
⎜
1
1
⎜
+
⎜
⎜ pSF pFS pSF
pSF
⎜
⎝
1
1
+
pFF pFS pSF
pSF
1
1
+
pFS pSF
pSF
⎞
⎟
⎟
1
1
1 ⎟
⎟
+
+
⎟,
pSF pFS pSF
pFS pSF
pSF ⎟
⎟
⎠
1
1
+
pFS pSF
pSF
and the bet sizes y1∗ and y2∗ are determined by the relations
y1∗
1
1
1
1
1
∗
= 1,
+
+ y2
+
+
pFS pSF
pSF
pSF pFS pSF
pSF pFS pSF
pSF
1
1
1
1
y1∗
+ y2∗
= 1,
+
+
pFF pFS pSF
pSF
pFS pSF
pSF
which can be solved to obtain
y1∗ =
pFF pFS pSF
pFS + pSF + pFS pSF
and
y2∗ =
pFS pSF (pSF − pFF )
.
pFS + pSF + pFS pSF
The probability, p1 , of the first scenario is just pF pFS pSF , so after substitution and simplification
the general formula (1) yields
E[τF SF ] = 1 +
pS
1
1
+ 2 +
,
pSF
pFS pSF
pSF
which is as explicit as could be desired.
3. Expected time until observing one of many patterns
We now consider a collection, C = {Ai , 1 ≤ i ≤ K}, of K strings of possibly varying lengths
from the two-letter alphabet {S, F }, and we take on the task of computing the expected value
of τC = min{τA1 , . . . , τAK }, the time at which any one of the patterns in C is first observed.
The method we propose is analogous to the two-team method we used in the previous section,
although many teams are now needed. The real challenge is the construction of the list of
appropriate ending scenarios, which now requires some algorithmic techniques.
3.1. Listing the ending scenarios
Given the collection C = {Ai }1≤i≤K , we first consider the set sequence transformation
C = {Ai }1≤i≤K → {SAi , F Ai }1≤i≤K = {Bi }1≤i≤2K = C ,
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
132
J. GLAZ ET AL.
which doubles the cardinality of C. We then delete from C each pattern B that can occur only
after the stopping time τC . The resulting collection, C , is called the ‘final list.’ We denote the
elements of C by Ci , 1 ≤ i ≤ K , and note that K ≤ K ≤ 2K.
To illustrate the construction, suppose that the initial collection is C = {FSF, FF}.
The ‘doubling step’ (from C to C ) gives us C = {SFSF, FFSF, SFF, FFF}. Since FFS
and FFF cannot occur before τC , these are eliminated from C and the final list is simply
C = {SFSF, SFF}. Similarly, if the initial collection is C = {FS, SSS}, then the final list is
C = {SFS, FFS}.
Now, before we describe the ending scenarios, we need one further notion. If patterns C
and C in the final list C satisfy C = SA and C = FA for some pattern A ∈ C, then we say
that C and C are ‘matched.’ Also, if C and C are matched and C = SA and C = FA, then we
say that C and C are ‘generated’ by A. Finally, even though there are many ending scenarios,
they may be classified into three basic kinds.
1. There are K scenarios in which we observe an element of C to be an initial segment of
the Markov sequence {Zn , n ≥ 1}.
2. There is a scenario for each unmatched pattern from C . We denote the number of these
by L.
3. There is pair of scenarios for each matched pattern from C . We denote the number of
these by 2M.
3.2. From the listing to the teams
For each scenario associated with the unmatched pattern Cj , we introduce one team of
straightforward gamblers who bet yj dollars on the pattern Ai that generated Cj . For each pair
of scenarios associated with the matched patterns Cp and Cm , which were generated by pattern
Ak , say, we introduce two teams. One team bets yp dollars on Ak in the straightforward way,
and the other bets ym dollars on Ak in the smart way. If Wij yj , i = 1, 2, . . . , K + L + 2M,
j = 1, 2, . . . , L + 2M, denotes the amount of money that the j th team wins in the j th scenario,
then the stopped martingale Xτ is given by the sum
Xτ =
L+2M
yj (τ − 1) − S(y1 , . . . , yL+2M ),
j =1
where
S(y1 , . . . , yL+2M ) =
K+L+2M
i=1
1Ei
L+2M
yj Wij .
j =1
Here, 1Ei is the indicator function for the event, Ei , that the ith scenario occurs.
∗
If (y1∗ , . . . , yL+2M
) is a solution to the linear system
∗
WK+1,L+2M
y1∗ WK+1,1 + · · · + yL+2M
..
.
= 1,
.. ..
. .
(2)
∗
y1∗ WK+L+2M,1 + · · · + yL+2M
WK+L+2M,L+2M = 1,
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
133
then we have
∗
S(y1∗ , . . . , yL+2M
)=
⎧
L+2M
⎪
⎪
⎨
y∗W
⎪
⎪
⎩1,
j =1
j
ij
in scenario i ∈ {1, 2, . . . , K}
in scenario i > K.
By the optional stopping theorem, we have
0 = E[X1 ] = E[XτC ] =
L+2M
j =1
yj∗ (E[τC ] − 1) −
K
pi
L+2M
i=1
j =1
K
yj∗ Wij − 1 −
pi ,
i=1
where pi is the probability that Ai is an initial segment of {Zn , n ≥ 1}. We can now solve this
equation to obtain a formula for E[τC ], which we summarize as a theorem as follows.
∗
Theorem 1. If (y1∗ , y2∗ , . . . , yL+2M
) solves the linear system (2), then
K
E[τC ] = 1 +
i=1 pi
L+2M
j =1
yj∗ Wij + (1 −
L+2M ∗
j =1 yj
K
i=1 pi )
.
(3)
As before, this formula is more explicit than it may at first appear. For example, all of the
required terms can be computed straightforwardly in problems of reasonable size.
3.3. Computation of the profit matrix
Formula (3) requires the computation of the profit matrix {Wij }. We will first show how
this can be done in general. The method will then be applied to a specific example to confirm
that (3) may be rewritten in terms of the basic model parameters.
Consider a scenario that ends with pattern C = c1 c2 · · · cm ∈ C . The team of straightforward gamblers who begin by betting one dollar and continue by betting on the successive terms
of the pattern A = a1 a2 · · · ap will, by time τC , have won an amount
min{m−1,p}
i=1
δist (A, C),
where
δist (A, C)
=
⎧
⎨
1
pc a pa a · · · pai−1 ai
⎩ m−i 1 1 2
0
if a1 = cm−i+1 , a2 = cm−i+2 , . . . , ai = cm ,
otherwise.
Similarly, in the same time, the team of smart gamblers will have won an amount
min{m−1,p}
i=1
δism1 (A, C) +
min{m−1,p−1}
i=1
δism2 (A, C),
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
134
J. GLAZ ET AL.
where
δism1 (A, C) =
⎧
⎪
⎪
⎨
1
pcm−i a1 pa1 a2 · · · pai−1 ai
⎪
⎪
⎩0
⎧
1
⎨
sm2
δi (A, C) = pa1 a2 pa2 a3 · · · pai ai+1
⎩
0
if a1 = cm−i+1 , a2 = cm−i+2 , . . . , ai = cm ,
and cm−i = a1 ,
otherwise,
if a1 = cm−i , a2 = cm−i+1 , . . . , ai+1 = cm ,
otherwise.
3.4. Explicit determination of E[τC ]
To illustrate the use of (3), we consider the collection C = {SS, FSF}. After doubling and
elimination we obtain the final list {FSS, SFSF, FFSF}. We then need to work out the set of
scenarios: we have two scenarios in which either C1 = SS or C2 = FSF occurs as an initial
segment of {Zn , n ≥ 1}; we have the unmatched scenario C3 = FSS, associated with the
pattern SS; and we have a pair of matched scenarios, C4 = SFSF and C5 = FFSF, which are
associated with the pattern FSF. The profit matrix {Wij } is then given by
⎛
⎞
1
pSS
⎜
⎜
⎜
⎜
0
⎜
⎜
⎜ 1
1
⎜
+
⎜
⎜ pFS pSS
pSS
⎜
⎜
⎜
0
⎜
⎜
⎜
⎝
0
0
1
pSF
0
1
1
+
pSF pFS pSF
pSF
1
1
+
pFF pFS pSF
pSF
0
⎟
⎟
⎟
⎟
⎟
⎟
⎟
⎟
0
⎟,
⎟
⎟
1
1
1 ⎟
⎟
+
+
⎟
pSF ⎟
pSF pFS pSF
pFS pSF
⎟
⎠
1
1
+
pFS pSF
pSF
1
1
+
pFS pSF
pSF
and, after solving the corresponding linear system, we find that the appropriate initial team bets
are given by
y1∗ =
pFS pSS
,
1 + pFS
y2∗ =
pFF pFS pSF
,
pFS + pSF + pFS pSF
y3∗ =
pFS pSF (pSF − pFF )
.
pFS + pSF + pFS pSF
The probabilities, p1 and p2 , that SS and FSF are initial segments of the process {Zn , n ≥ 1}
are respectively given by pS pSS and pF pFS pSF . Thus, (3) leads to the pleasantly succinct
result
1 − pS pSS
.
E[τC ] = 2 + pS pSF +
pFS
4. Generating functions for pattern waiting times
To find the generating function of the waiting time τC , we need to introduce the same
scenarios and the same betting teams, but we need to make some changes to the design of the
initial bets. A gambler from the j th team who arrives at moment k − 1 and who begins his
betting in round k will now begin with a bet of size yj α k , where 0 < α < 1. If α τC Wij (α)yj
denotes the total winnings of the j th team when the game ends in the j th scenario, then we call
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
135
Wij (α) the ‘α-profit matrix.’ As before, the α-profit matrix does not depend on τC , and can be
computed if we know the ending scenario.
If Xn again denotes the casino’s net gain at time n, then
XτC =
L+2M
α 2 − αα τ yj − S(α, y1 , . . . , yL+2M ),
1−α
j =1
where
S(α, y1 , . . . , yL+2M ) =
K+L+2M
1Ei
i=1
K+2M
α τC yj Wij (α).
j =1
As before, 1Ei is the indicator function for the event, Ei , that the j th scenario occurs.
∗
If (y1∗ , . . . , yL+2M
) is a solution to the linear system
∗
WK+1,L+2M (α) = 1,
y1∗ WK+1,1 (α) + · · · + yL+2M
..
.. ..
.
. .
(4)
∗
y1∗ WK+L+2M,1 (α) + · · · + yL+2M
WK+L+2M,L+2M (α) = 1,
then we might hope to mimic our earlier calculation of E[XτC ], but unfortunately we run into
trouble since E[α τC 1E1 ] may not equal p1 E[α τC ].
Nevertheless, if the ith, i ≤ K, scenario occurs, then we know the value of τC exactly. It
is equal to |Ai |, the length of the ith sequence. Therefore, we have a formula for the stopped
martingale:
L+2M
α 2 − αα τ ∗
∗
XτC =
yj − α τ − I (α, y1∗ , . . . , yL+2M
).
1−α
j =1
∗
Here, I (α, y1∗ , . . . , yL+2M
) is defined by setting
⎧
L+2M
⎪
⎪
∗
⎨α |Ai |
y
W
(α)
−
1
in scenario i ∈ {1, 2, . . . , K}
j ij
∗
I (α, y1∗ , . . . , yL+2M
)=
j
=1
⎪
⎪
⎩0,
in scenario i > K.
From this formula and the optional stopping theorem, we then find the anticipated formula for
the moment generating function of τC .
∗
Theorem 2. If (y1∗ , . . . , yL+2M
) is a solution to the linear system (4), then
τC
E[α ] =
[α 2 /(1 − α)]
L+2M
K
|Ai | [ L+2M
i=1 pi α
j =1
L+2M ∗
1 + [α/(1 − α)] j =i yj
j =1
yj∗ −
yj∗ Wij − 1]
.
(5)
4.1. Computation of the α-profit matrix
As before, we need to know how to compute the profit matrix before (5) may be properly
regarded as an explicit formula. This is only a little more difficult than before. First, assume
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
136
J. GLAZ ET AL.
that a scenario ends with the pattern C = c1 c2 · · · cm . The team of straightforward gamblers
who bet a dollar on pattern A = a1 a2 · · · ap will, by time τ , win an amount
min{m−1,p}
δist (A, C)
,
α i−1
i=1
while, in the same time, the team of smart gamblers will win an amount
min{m−1,p}
δism1 (A, C)
α i−1
i=1
+
min{m−1,p−1}
δism2 (A, C)
.
α i−1
i=1
These formulae provide almost everything we need, but before we can be completely explicit,
we need to focus on a concrete example.
4.2. A generating function example
Consider the waiting time to observation of the three-letter pattern FSF in the random
sequence {Zn , n ≥ 1} produced by the Markov model. In this case, the α-profit matrix {Wij (α)}
is given by
⎞
⎛
1
α −1
1
+
⎟
⎜
pSF
pFS pSF
pSF
⎟
⎜
⎟
⎜
−2
−2
−1
⎟
⎜
α
α
1
α
1
⎟,
⎜
+
+
⎜p p p + p
pSF pFS pSF
pFS pSF
pSF ⎟
SF
⎟
⎜ SF FS SF
⎟
⎜
−2
−1
⎠
⎝
α
α
1
1
+
+
pSF
pFF pFS pSF
pSF
pFS pSF
and, by solving the associated linear system, we obtain
y1∗ =
α 2 pFF pFS pSF
,
1 − αpFF + αpSF + α 2 pFS pSF
y2∗ =
α 2 pFS pSF (pSF − pFF )
.
1 − αpFF + αpSF + α 2 pFS pSF
The general moment generating representation (5) then gives us the simple formula
E[α τC ] =
1 − α(pSS
α 3 pFS pSF (pF + α(pS − pSS ))
.
+ pFF − α(pFF − pSF (1 − pFS (1 − αpSS ))))
Naturally, such a formula provides complete information on the distribution of τC , and to obtain
an explicit formula for P(τC = k) we can rewrite the rational function (5) in its partial fraction
expansion.
5. Higher-order Markov chains
In this work, we have applied the gambling team method only to two-state chains.
For reasons that will be explained later, this limitation is not easily lifted. Nevertheless, there
are more complex chains to which the team method applies, and it is instructive to consider
one of these. Specifically, we briefly consider how the gambling team method may be applied
to second-order two-state chains. Here we obviously need to avoid the naive representation of
such chains as first-order chains with four states.
In the team approach to a second-order model, the gamblers need to observe two rounds of
betting before they place their first bets, and, consequently, we need to consider a larger number
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
137
of final scenarios. Moreover, for each pattern A = a1 a2 · · · ap we will need to consider up to
seven termination cases, including three ‘initial’ cases, which are associated with the patterns
A, SA, and FA, and four ‘later’ cases, which are associated with the patterns SSA, SFA, FSA,
and FFA.
As before, our main objective is to count accurately all the ending scenarios and create a
matched number of gambling teams. However, for a second-order chain there is an additional
difficulty that must be addressed. Specifically, two cases must be considered separately: (i) there
are no runs in the initial list C and (ii) there are runs in C.
5.1. Case one: there are no runs in C
First we need to replace the earlier doubling step with an analogous quadrupling step. Now,
given the collection of patterns C = {Ai }1≤i≤K , we consider the set sequence transformation
C = {Ai }1≤i≤K → {SSAi , SFAi , FSAi , FFAi }1≤i≤K = {Bi }1≤i≤4K = C .
We then delete from C each scenario that can occur only after the stopping time τC , and we
take the collection C that remains to be our ‘final list’ of ending scenarios.
Each pattern from the collection C leads us to four – or perhaps fewer – ending scenarios.
For each pattern from C we consider a sequence of gamblers who belong to teams of different
types. As before, these gamblers arrive sequentially and observe the game before placing any
bets, according to the following rules.
1. A new gambler from the ‘type-I’ team arrives two rounds before he begins to bet. He watches
these rounds and then bets on the successive letters of pattern A, with complete indifference to
what he may have seen in the first two rounds he observed.
2. A new gambler from the ‘type-II’ team also arrives two rounds before he begins to gamble,
but is influenced by what he sees. If this gambler observes the pattern Sa1 in these two rounds,
then he bets on the sequence a2 a3 · · · ap . If he observes anything other than Sa1 , then he places
his bets according to the sequence A.
3. Similarly, a new gambler from the ‘type-III’ team watches two rounds, and if he observes
the pattern F a1 then he bets according to the sequence a2 a3 · · · ap . If he observes anything
other than F a1 , then he places his bets according to A.
4. Finally, a gambler from the ‘type-IV’ team watches two rounds, and if he observes a1 a2 then
he bets according to the sequence a3 a4 · · · ap . Otherwise, he bets according to the sequence A.
Each pattern A from the initial list C leads to zero, one, two, three, or four scenarios in the
final list C . Thus, instead of there being only matched and unmatched patterns, the patterns
in final list C are now of four kinds: ‘unmatched,’ ‘double-matched,’ ‘triple-matched,’ and
‘quadruple-matched.’
To see how this works, consider the initial collection C = {FSFF, FFSF}. First, note that
we have five initial cases, namely FSFF, FFSF, SFSFF, SFFSF, and FFFSF. Pattern FFSFF
cannot occur before τC ; therefore, the scenario associated with this pattern is eliminated from
the list of initial cases. Second, in the final list C we have five patterns: the double-matched
patterns SSFSFF and FSFSFF generated by FSFF, and the triple-matched patterns SSFFSF,
SFFFSF, and FFFFSF generated by FFSF. Thus, we need to introduce five teams: type-I and
type-II teams that bet on FSFF, and type-I, type-II, and type-III teams that bet on FFSF.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
138
J. GLAZ ET AL.
5.2. Case two: special treatment of runs
If the initial list C contains a run, then there may be a problem with the straightforward
application of the method described above. The difficulty is that if we observe the game only
until the moment τC , then there is no difference in behavior between teams of different types
that place their bets on the run.
To illustrate the problem, let us consider the initial list C = {F (r) }. The straightforward usage
of the above algorithm tells us that we must introduce two initial cases, F (r) and SF (r) . The
final list C contains two double-matched patterns, SSF (r) and FSF (r) . Therefore, according
to the algorithm, we need two (type-I and type-II) teams that respectively bet y1 and y2 dollars
on F (r) . However, since before time τC there is no difference in gambling between these two
teams, we in effect have only one team, which bets y1 + y2 dollars on the run. Thus, the number
of free parameters does not match the number of ending scenarios.
However, a simple modification of the gambling method easily solves the problem. Before
time τC , the run F (r) can only occur as an initial segment of the sequence {Zn , n ≥ 1}, or, later,
in the pattern SF (r) . Thus, if the initial list C contains runs (obviously we can have only one or
two runs in C, namely F (r) or S (p) or both), then we first need to substitute the runs F (r) and
S (p) in C by SF (r) and FS (p) , respectively, to obtain a different collection C̃. The collection C̃
contains no runs; therefore, we can proceed as before. After application of the quadrupling and
elimination processes to the list C̃ we will obtain the final list of ending scenarios C̃ , and for
this list we will be able to create a matched number of gambling teams. Since we are interested
in τC , not τC̃ , the elimination process has to be based on τC , and the runs must be included in
the list of initial cases.
For instance, if C = {F (r) } then we must consider four initial cases, F (r) , SF (r) , SSF (r) ,
and FSF (r) , and four later cases, in which the game ends with SSSF (r) , SFSF (r) , FSSF (r) , or
FFSF (r) . For this collection, it can be shown that each of the four teams (type I, II, III, and IV)
that bet on SF (r) bets in its own way.
5.3. Final step
After attending to this bookkeeping, we can now calculate the expected observation times in
a way that parallels our earlier calculation. Since we have matched the number of (non-initial)
ending scenarios and the number of teams, we can choose the size of the initial bet for each
team in a way that makes all the expressions for the stopped martingale equal to 1, however the
game may end.
Let us summarize this as a theorem. Assume that ultimately we have P initial cases and Q
later cases. Let Wij yj , i = 1, 2, . . . , P + Q, j = 1, 2, . . . , Q, denote the amount of money
won by the j th team that bets yj dollars in the ith scenario. Finally, let pi , i = 1, 2, . . . , P ,
be the probability that the ith initial scenario occurs.
∗ ) solves the linear system
Theorem 3. If (y1∗ , y2∗ , . . . , yQ
∗
WP +1,Q = 1,
y1∗ WP +1,1 + · · · + yQ
..
.
.. ..
. .
∗
y1∗ WP +Q,1 + · · · + yQ
WP +Q,Q = 1,
then
P
E[τC ] = 2 +
i=1 pi
Q
∗
j =1 yj Wij + (1 −
Q
∗
j =1 yj
P
i=1 pi )
.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
Gambling teams and waiting times
139
6. Concluding remarks
The method of gambling teams deals quite effectively with the waiting time problems of twostate chains, but for N -state chains it is much less effective. The problem is that we typically
find that the number of ending scenarios is larger than the number of teams, meaning that there
are too few free parameters to achieve the required matching.
We might think to reduce the waiting time problems for an N -state chain by encoding the
states {1, 2, . . . , N} as sequences of 0s and 1s, but this idea typically fails, since the natural
encodings do not lead to a waiting time problem for a homogeneous two-state Markov chain on
{0, 1}. Ironically, for many of the pattern problems associated with N -state Markov chains, the
method of gambling teams is ineffective when N ≥ 3, even though it is typically the method
of choice for the corresponding problems in a two-state chain.
A possible computational advantage of the martingale method over the Markov chain
embedding method (see, e.g. Antzoulakos (2001), Fu (2001), and Fu and Chang (2002)) is
the size of the matrices involved in the calculations. The size of the profit matrix depends only
on the number of patterns, K, while the size of the transition matrix of the embedded Markov
chain depends also on the length of patterns in C. For example, if C contains K patterns,
each of which has a length that is approximately N , and K is much smaller than N , then the
dimensions of the transition matrix in the Markov chain embedding method are approximately
(K × N) × (K × N ). For large N , this can cause technical problems. The size of profit matrix
is at most 2K × 2K.
References
Aki, S., Balakrishnan, N. and Mohanty, S. G. (1996). Sooner and later waiting time problems and failure runs in
higher order Markov dependent trials. Ann. Inst. Statist. Math. 48, 773–787.
Antzoulakos, D. (2001). Waiting times for patterns in a sequence of multistate trials. J. Appl. Prob. 38, 508–518.
Benevento, R. V. (1984). The occurrence of sequence patterns in ergodic Markov chains. Stoch. Process. Appl. 17,
369–373.
Biggins, J. D. and Cannings, C. (1987a). Formulas for mean restriction-fragment lengths and related quantities.
Amer. J. Hum. Genet. 41, 258–265.
Biggins, J. D. and Cannings, C. (1987b). Markov renewal processes, counters and repeated sequences in Markov
chains. Adv. Appl. Prob. 19, 521–545.
Blom, G. and Thorburn, D. (1982). How many random digits are required until given sequences are obtained?
J. Appl. Prob. 19, 518–531.
Breen, S., Waterman, M. and Zhang, N. (1985). Renewal theory for several patterns. J. Appl. Prob. 22, 228–234.
Chryssaphinou, O. and Papastavridis, S. (1990). The occurrence of a sequence of patterns in repeated dependent
experiments. Theory Prob. Appl. 35, 145–152.
Feller, W. (1968). An Introduction to Probability Theory and Its Applications, Vol. 1, 3rd edn. John Wiley, New York.
Fu, J. C. (1986). Reliability of consecutive-k-out-of-n: F systems with (k − 1)-step Markov dependence. IEEE Trans.
Reliab. 35, 602–606.
Fu, J. C. (2001). Distribution of the scan statistics for a sequence of bistate trials. J. Appl. Prob. 38, 908–916.
Fu, J. C. and Chang, Y. (2002). On probability generating functions for waiting time distribution of compound patterns
in a sequence of multistate trials. J. Appl. Prob. 39, 70–80.
Fu, J. C. and Koutras, M. V. (1994). Distribution theory of runs: a Markov chain approach. J. Amer. Statist. Assoc.
78, 168–175.
Gerber, H. and Li, S. (1981). The occurrence of sequence patterns in repeated experiments and hitting times in a
Markov chain. Stoch. Process. Appl. 11, 101–108.
Guibas, L. and Odlyzko, A. (1981a). Periods of strings. J. Combinatorial Theory A 30, 19–42.
Guibas, L. and Odlyzko, A. (1981b). String overlaps, pattern matching and nontransitive games. J. Combinatorial
Theory A 30, 183–208.
Han, Q. and Hirano, K. (2003). Sooner and later waiting time problems for patterns in Markov dependent trials.
J. Appl. Prob. 40, 73–86.
Li, S. (1980). A martingale approach to the study of occurrence of sequence patterns in repeated experiments. Ann.
Prob. 8, 1171–1176.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418
140
J. GLAZ ET AL.
Pozdnyakov, V. and Kulldorff, M. (2006). Waiting times for patterns and a method of gambling teams. Amer. Math.
Monthly 113, 134–143.
Pozdnyakov, V., Glaz, J., Kulldorff, M. and Steele, J. M. (2005). A martingale approach to scan statistics. Ann.
Inst. Statist. Math. 57, 21–37.
Robin, S. and Daudin, J.-J. (1999). Exact distribution of word occurrences in a random sequence of letters. J. Appl.
Prob. 36, 179–193.
Stefanov, V. T. (2000). On some waiting time problems. J. Appl. Prob. 37, 756–764.
Stefanov, V. T. (2003). The intersite distances between pattern occurrences in strings generated by general discreteand continuous-time models: an algorithmic approach. J. Appl. Prob. 40, 881–892.
Stefanov, V. T. and Pakes, A. G. (1997). Explicit distributional results in pattern formation. Ann. Appl. Prob. 7,
666–678.
Uchida, M. (1998). On generating functions of waiting time problems for sequence patterns of discrete random
variables. Ann. Inst. Statist. Math. 50, 655–671.
Williams, D. (1991). Probability with Martingales. Cambridge University Press.
Downloaded from https://www.cambridge.org/core. IP address: 88.99.165.207, on 31 Jul 2017 at 18:29:25, subject to the Cambridge Core terms of use,
available at https://www.cambridge.org/core/terms. https://doi.org/10.1017/S0021900200001418