Submitted to Operations Research
manuscript (Please, provide the manuscript number!)
Indifference-Zone-Free Selection of the Best
Weiwei Fan
Department of Management Science, School of Management, University of Science and Technology of China, Hefei, China,
[email protected]
L. Jeff Hong
Department of Economics and Finance and Department of Management Sciences, College of Business, City University of
Hong Kong, Kowloon Tong, Hong Kong, jeff[email protected]
Barry L. Nelson
Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, Illinois, 60208,
[email protected]
Many procedures have been proposed in the literature to select the simulated alternative with the best
mean performance from a finite set of alternatives. Among these procedures, frequentist procedures are typically designed under either the subset-selection (SS) formulation or the indifference-zone (IZ) formulation.
Both formulations may encounter problems when the goal is to select the unique best alternative for any
configuration of the means. In particular, SS procedures may return a subset that contains more than one
alternative, and IZ procedures hinge upon the relationship between the chosen IZ parameter and the true
mean differences that is unknown to decision makers a priori. In this paper, we propose a new formulation
that guarantees to select the unique best alternative with a user-specified probability of correct selection
(PCS), as long as the means of alternatives are unique, and we design a class of fully sequential procedures
under this formulation. These procedures are parameterized by the PCS value only, and their continuation
boundaries are determined based on the Law of the Iterated Logarithm. Furthermore, we show that users
can add a stopping criterion to these procedures to convert them into IZ procedures, and we argue that these
procedures have several advantages over existing IZ procedures. Lastly, we conduct an extensive numerical
study to show the performance of our procedures and compare their performance to existing procedures.
1. Introduction
Decision makers often face the problem of selecting the best from a finite set of alternatives, where
the best refers to the alternative with the largest (or smallest) mean performance. For instance, an
inventory manager may need to select an inventory policy to minimize the total mean cost, while a
financial investor may want to choose an investment strategy to maximize the mean payoff. In many
practical situations, the mean performances of these alternatives are not explicitly available and
can only be evaluated by running simulation experiments. This is known as the selection-of-thebest problem. To solve this problem, a decision procedure is often needed to determine the proper
sample sizes of all alternatives and to decide which alternative to select. There is a large body of
1
2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
literature on designing such procedures, see Bechhofer et al. (1995), Kim and Nelson (2006a) and
Branke et al. (2007) for introductions and overviews.
Among these procedures in the literature, there are frequentist and Bayesian approaches. Frequentist procedures (e.g., Rinott 1978, Kim and Nelson 2001) allocate simulation observations to
different alternatives to achieve a pre-specified probability of correct selection (PCS) even for the
least favorable configuration of the means. They are typically conservative (i.e., requiring more
samples than necessary) for an average case. Bayesian procedures (e.g., Chick and Inoue 2001,
Chick and Frazier 2012) often allocate a finite computing budget to different alternatives to either
maximize the posterior (Bayesian) PCS or minimize the opportunity cost. They often require fewer
observations than frequentist procedures, but typically do not provide a guaranteed (frequentist)
PCS. An exception is Frazier (2014) who recently proposes a Bayes-inspired procedure to achieve
a pre-specified frequentist PCS. In this paper we take the frequentist viewpoint and our goal is to
design procedures that deliver a user-specified PCS.
The first frequentist selection-of-the-best formulations were designed in the 1950s, when observations were often collected through physical experiments, e.g., agricultural experiments or clinical
trials. Each of these experiments may take quite a long time (e.g., weeks to months) to conduct,
and it only makes sense to conduct these experiments in batches. Therefore, the stage-wise procedures that select the best at the end of the last stage were prevalent. For these procedures, the
critical issue is to determine the appropriate sample size for each alternative so that the best may
be selected. Notice that the means of the best and the second-best alternatives may be arbitrarily
close. Therefore, guaranteeing to find the unique best for all configurations of the means could
require an arbitrarily large number of observations, which is not practical. To avoid this difficulty,
various formulations were proposed to soften the original goal of selecting the unique best alternative. One is the subset-selection (SS) formulation of Gupta (1956, 1965), which guarantees to select
a random-sized subset of the alternatives that contains the best (instead of selecting the unique
best). The procedures of Panchapakesan et al. (1971) and Sullivan and Wilson (1989) are also
designed under this formulation. The other formulation is the indifference-zone (IZ) formulation
of Bechhofer (1954), which guarantees to select the unique best alternative when the difference
between the means of the best and the second-best is at least δ, where δ > 0 is called an IZ parameter. The procedures proposed by Rinott (1978), Clark and Yang (1986) and many others are also
designed under this formulation.
Starting in the 1990s there was a paradigm change in selection-of-the-best problems. Under this
new paradigm, observations of alternatives are often obtained through computer simulation experiments. Each of these experiments takes a fraction of a second to a few minutes and they are often
generated sequentially from either a single processor or a small number of parallel processors. This
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
3
sequential nature of computer simulation experiments motivates the use of sequential procedures
that date back to Paulson (1964) under the IZ formulation. The major milestone of this stream of
research was the paper of Kim and Nelson (2001) that completely adapts fully sequential procedures
to computer simulation experiments. These fully sequential procedures typically specify a continuation region. They then approximate the sum of differences between two competing alternatives
as a Brownian motion (BM) process with drift, and make an elimination decision when the process exits the continuation region. Different from stage-wise procedures, fully sequential procedures
gradually gather information (i.e., simulation observations) on the unknown differences between
the means of competing alternatives and eliminate the inferior alternatives once enough statistical
evidence is available. Therefore, they tend to be more efficient, i.e., need fewer observations, than
their stage-wise counterparts.
Even though many fully sequential frequentist procedures have been proposed since the work of
Kim and Nelson (2001), e.g., the procedures of Batur and Kim (2006), Hong and Nelson (2005,
2007), Pichitlamken et al. (2006), Hong (2006), Kim and Nelson (2006b) and many others, none of
these questioned the necessity of the IZ formulation. Unlike stage-wise procedures where information on the means of the competing alternatives are unknown, fully sequential procedures gather
this information as they progress. This information may free us from having to provide an IZ
parameter and help us to select the unique best alternative for any configuration of the means.
In this paper we propose a new formulation that selects the best alternative with a user-specified
PCS, as long as all the alternatives have unique means. Under this formulation, the means of any
pair of alternatives are not equal and can be arbitrarily close. Therefore, the key to designing such
a procedure is to construct a continuation region where, with the specified PCS, no elimination
decision will be made if the mean difference between these two alternatives is zero, while a correct
elimination decision will be made if it is not. To achieve this, we need to construct a continuation
region so that a BM with no drift will stay inside while a BM with drift will exit from the correct
side. Notice that BM with and without drift differ in the rates at which they grow. In particular, a
BM with positive drift approaches infinity at the rate O(t), while a BM without drift approaches
√
infinity at a rate bounded by O( t log log t ) due to the Law of the Iterated Logarithm (Durrett
2010). In light of this, any continuation region, formed by boundaries which grow at a rate between
√
O( t log log t ) and O(t), can achieve our objective.
To further determine the design parameters of such continuation regions, we need to be able
to evaluate the first-exit probability of a BM from the given continuation region. However, it
is generally difficult or even impossible to explicitly analyze the first-exit-time distribution of a
BM (see Durbin 1985). To solve the problem, we consider two approaches. In the first approach,
we obtain the design parameter by numerically solving a one-dimensional stochastic root-finding
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
4
problem, which can be solved efficiently with the use of an importance sampling scheme. In the
second approach, we obtain the design parameter by evaluating the first-exit probability using an
asymptotic result and we prove the validity of the procedures as PCS goes to 1. It is worthwhile
noting that the asymptotic regime of letting PCS go to 1 is a classical one and it dates back to
Perng (1969) and Dudewicz (1969).
Beyond selecting the unique best alternative, our procedures can also be used as sequential
subset-selection procedures. In particular, at any time before the procedures terminate, the surviving subset of alternatives has retained the best alternative with the same probability guarantee.
One shortcoming of our procedures is that they may not stop when two or more alternatives
have the same means. To avoid this situation, we propose a stopping criterion for our procedures.
The stopping criterion requires the user to specify an error tolerance ε. We show that, with the
same PCS, the selected alternative is either the best or within ε of the best. This indicates that
our procedures provide both the correct selection guarantee and good selection guarantee, which
is the goal of Ni et al. (2014). If one treats ε as an IZ parameter, then our procedures become
IZ procedures. However, unlike IZ procedures, the average sample size of our procedures is typically insensitive to the setting of the error tolerance, thus one may set it very small for practical
implementations.
Our research is closely related to the literature on best-arm identification in stochastic multiarmed bandit (MAB) problems. (Readers interested in the MAB problems may refer to Bubeck
and Cesa-Bianchi (2012) for a comprehensive overview). The best-arm identification procedures
seek the same goal as our procedures do, i.e., selecting the unique best arm with a pre-determined
probability. In contrast, our procedures and these best-arm identification procedures differ in at
least three aspects. Firstly, they employ different elimination mechanisms. In particular, the bestarm identification procedures (e.g. Karnin et al. 2013, Jamieson and Nowak 2014) sequentially
construct a confidence bound for the mean reward of each arm based on the estimated sample
mean, and make eliminations if there are non-overlapping confidence bounds. Secondly, the bestarm identification procedures assume the cumulant generating function of each reward is bounded
by a known convex function, while our procedures only require a finite joint moment generating
function of rewards in a neighborhood of zero vector (see Theorem 2) and it need not be known.
Lastly, our procedures are able to allow dependence among the rewards of different arms and thus
common random numbers can be used to speed up the selection of the best.
The rest of our paper is organized as follows. In Section 2, we introduce a new selection-ofthe-best formulation. In Section 3, we design a class of fully sequential IZ-free procedures under
the new formulation when observations from all alternatives are jointly normally distributed. In
Section 4, we relax the normality assumption to design the corresponding sequential procedures
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
5
that are asymptotically valid. In Section 5 we analyze the asymptotic efficiency of our procedures
and discuss a stopping criterion in Section 6. In Section 7 we conduct a comprehensive numerical
study to understand our procedures and compare them to existing procedures, and we conclude in
Section 8.
2. A New Selection-of-the-Best Formulation
Suppose there are k (≥ 2) competing alternatives at the beginning of the selection process, and
the goal is to select the alternative that has the largest mean performance. For i = 1, 2, . . . , k,
denote µi as the unknown mean performance of alternative i and we evaluate it through computer
simulation. Without loss of generality, assume µ1 > µ2 > · · · > µk , implying that the best alternative
is unique and alternative 1 is the best. It is worthwhile noting that we make no assumption on
the difference between µ1 and µ2 because there is often no such knowledge in practice. Further,
let 1 − α (0 < α ≤ 1 − 1/k) denote the user-specified PCS. In other words, users target at selecting
alternative 1 with probability at least 1 − α.
To achieve this target, there are two different frequentist formulations in the literature. They
are the SS and IZ formulations. As stated in Section 1, both of them may encounter difficulties.
In light of this, we propose a new selection-of-the-best formulation in this section. Under the new
formulation, procedures guarantee to select the best alternative with a user-specified PCS value
1 − α, as long as their means are unique, i.e., µ1 > µ2 > · · · > µk . Equivalently, procedures under
the new formulation guarantee to satisfy
P{select alternative 1} ≥ 1 − α.
(1)
Therefore, to use the procedures under the new formulation, one only needs to specify the PCS
value 1 − α. Compared to the IZ formulation, the new formulation frees users from having to specify
an IZ parameter, and compared to the SS formulation, it selects the unique best alternative instead
of a random subset, but can select a subset if procedures are terminated early.
In the following two sections, we design sequential procedures under the new formulation for
cases where observations are jointly normally distributed as well as cases where observations are
generally distributed, and prove that these procedures can deliver the required statistical guarantee,
i.e., Equation (1). We emphasize that we use sequential procedures not only because they are more
efficient but also because stage-wise procedures cannot achieve the necessary statistical guarantee.
3. Procedures with Normally Distributed Observations
Let Xir , r = 1, 2, . . . , denote the rth independent observation from alternative i, for i = 1, 2, . . . , k.
In this section, we consider the case where (X1r , X2r , . . . , Xkr )(r = 1, 2, . . .) are jointly normally
6
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
distributed with unknown mean vector (µ1 , µ2 , . . . , µk ) and unknown positive definite covariance
matrix. The assumption of possible dependence among observations from different alternatives
enables the use of common random numbers (CRNs). The task of this section is to design a
sequential procedure that deliver the statistical guarantee stated in (1).
Existing sequential procedures, such as the KN procedure, often decompose the selection-of-thebest problem into pairwise comparisons. In each pairwise comparison, they then approximate the
partial-sum difference process between two competing alternatives by a Brownian motion with drift
and design a continuation region to eliminate the inferior alternative based on the first-exit time of
the Brownian motion from it. In this paper, we adopt the same mechanism to design a sequential
procedure under the new formulation and, consequently, we need to address the following two
issues: (a) constructing a Brownian approximation to the partial-sum difference process, and (b)
designing a proper continuation region.
Under the normality assumption, it is straightforward to construct a Brownian approximation.
Consider a pairwise comparison between alternatives i and j, where i ̸= j. Let X̄i (n) denote the
sample mean calculated from the first n observations from alternative i and define
2
tij (n) = n/σij
, and Zij (tij (n)) = tij (n)[X̄i (n) − X̄j (n)],
(2)
2
where σij
= Var[Xi1 − Xj1 ]. Throughout this paper, we define B(·) as a Brownian motion (BM)
with unit variance and no drift, and define B∆ (·) as a BM with unit variance and drift ∆. Theorem
1 of Hong (2006) shows that the random sequences {Zij (tij (n)) : n ≥ 1} and {Bµi −µj (tij (n)) : n ≥ 1}
have the same joint distribution. Therefore, {Zij (tij (n)) : n ≥ 1} can be viewed as a BM with drift
µi − µj observed at the discrete time points {tij (1), tij (2), . . .}.
Let (−gc (t), gc (t)) denote the continuation region used in the procedure, where c is a constant
determined based on the number of alternatives k, the PCS value 1 − α and the common first-stage
sample size n0 . As discussed in Section 1, the continuation region is expected to satisfy that, with
a pre-specified PCS, a BM with no drift stays inside, while a BM with drift exits from the correct
side. Notice that the former grows to infinity at a rate slower than that of the latter. In particular,
the Law of the Iterated Logarithm (Durrett 2010) states that, the rate of the former is bounded
√
by O( t log log t ) while the rate of the latter is O(t). The rate difference motivates the use of a
boundary function that increases at rates in-between to construct the targeted continuation region.
In light of this, boundaries of interest are categorized into the following set,
{
}
gc (t)
gc (t)
1
G = gc (t) ∈ C [0, ∞) : lim
= 0, and lim inf √
∈ (0, ∞] ,
t→∞
t→∞
t
t log log t
where C 1 [0, ∞) denotes the set of functions whose first derivatives are continuous on [0, ∞). Furthermore, for the ease of theoretical analysis and practical implementation, we choose gc (t) from
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
7
G such that gc (t) is increasing in t and gc (t)/t is decreasing in t for any fixed c. For instance, we
√
√
can choose 2t log(c + log(t + 1)) (c ≥ 1), [c + log(t + 1)](t + 1) (c > 0) or ctβ (1/2 < β < 1, c > 0).
The determination of the design parameter c is deferred to Section 3.1.
3.1. The Procedure
In this subsection, we propose a sequential procedure designed based on the key idea stated above.
Particularly, the procedure uses a first stage to estimate the unknown variances and determine a
proper continuation region, and then collects observations sequentially to eliminate inferior alternatives until only one is left.
Procedure 1 (Sequential Procedure for Normally Distributed Observations)
Setup. Select the PCS 1 − α (0 < α ≤ 1 − 1/k), and a common first-stage sample size n0 ≥ 2.
Choose a boundary gc (t) from G and calculate the constant c that is the unique root to the following
equation,
}]
[ {
(
)
α
n0 − 1
Y
=
E P B(T ) ≤ −gc
T
, T < ∞Y
,
(3)
Y
n0 − 1
k−1
}
{
(
)
where T = inf t : |B(t)| ≥ gc n0Y−1 t n0Y−1 and Y is a chi-squared distributed random variable with
n0 − 1 degrees of freedom that is independent of B(·). Please refer to Appendix B on how to find
the root c of Equation (3).
Initialization. Let I = {1, 2, . . . , k } be the set of alternatives in contention. For each i ∈ I, simulate n0 observations Xi1 , Xi2 , . . . , Xin0 from alternative i and calculate its sample mean as
0
1 ∑
Xil .
n0 l=1
n
X̄i (n0 ) =
For any i, j ∈ I with i ̸= j, calculate the sample variance of their difference as
0
[
]2
1 ∑
=
Xir − Xjr − (X̄i (n0 ) − X̄j (n0 )) .
n0 − 1 r=1
n
2
Sij
Set n = n0 .
Screening. Let I old = I and for any i, j ∈ I with i ̸= j, let
2
τij (n) = n/Sij
Let
and
Zij (τij (n)) = τij (n)[X̄i (n) − X̄j (n)].
{
}
I = I old \ i ∈ I old : Zij (τij (n)) ≤ −gc (τij (n)) for some other j ∈ I old .
Stopping Rule. If |I | = 1, stop and select the alternative whose index is in I as the best. Otherwise,
take one additional observation from each alternative i ∈ I, set n = n + 1 and go to Screening.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
8
Remark 1. Procedure 1 can work with many choices of boundaries (i.e., G ). In the numerical
√
studies reported in Section 7, we use a specific boundary gc (t) = [c + log(t + 1)](t + 1) as an
instance.
Procedure 1, like those of Kim and Nelson (2001) and Hong (2006), is essentially a sequential
screening procedure and works as follows. It includes all the alternatives into I at the Initiation
step and eliminates the inferior alternatives from I over the Screening steps until only one alternative is left. At each Screening step, it considers all the possible pairwise comparisons between
the surviving alternatives and eliminates the inferior alternatives based on the first-exit time of the
partial-sum difference process from the given continuation region. Take the comparison between
alternatives i and j for an instance. When the partial-sum difference process Zij (τij (·)) exits the
continuation region from below (above), Procedure 1 eliminates alternative i (j), otherwise it eliminates neither of them and continues collecting observations (see Figure 1). Once an alternative is
eliminated, it will not be considered again.
Z ij(τ ij(n))
eliminate alternative j
g (τ (n))
c ij
τ (n)
ij
eliminate alternative i
c
Figure 1
(τ (n))
ij
Continuation region of our sequential procedures. (The x-axis is well defined because τij (n) increases
linearly with n.)
Even though Procedure 1 is designed to select the unique best alternative, users can stop Procedure 1 at any time before termination and it still guarantees to have retained the best alternative.
This result is summarized as the following Lemma 1.
Lemma 1. Define N as the number of replications when Procedure 1 terminates and only one
alternative is left. Let I(n) for n ≤ N denote the set of surviving alternatives in Procedure 1 after
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
9
n replications. If Procedure 1 delivers the statistical guarantee as stated in Equation (1), then we
have, for any n ∈ Z + ,
P{1 ∈ I(n ∧ N )} ≥ 1 − α,
where x ∧ y denotes the smaller of x and y.
Lemma 1 shows that, if there is a maximal stopping time n, Procedure 1 guarantees with at
least probability 1 − α to either select the best alternative before n or retain the best alternative
in a subset when it is stopped at n. Therefore, Procedure 1 can be utilized as a sequential subsetselection procedure when users have a sampling budget on screening. Besides, Procedure 1 tends
to be more efficient than the existing subset-selection procedures (e.g., Gupta 1956) that use equal
allocation of observations among alternatives, especially when the number of alternatives is large.
This arises because Procedure 1 tends to allocate fewer observations to the significantly inferior
alternatives.
3.2. Statistical Validity
In this subsection, we establish the statistical validity of Procedure 1, i.e., it delivers the statistical
guarantee stated in Equation (1). To achieve this goal, we need several lemmas.
In each pairwise comparison, an incorrect-selection event occurs if the better alternative is eliminated, or equivalently, the partial-sum difference process exits the continuation region from the
wrong direction. Notice that we only collect observations at discrete time points in the procedure.
Hence, the partial-sum process corresponds to observing a BM at discrete time points. The following
lemma states that, under very general conditions, the procedure designed for the continuous-time
BM provides an upper bound on the probability of incorrect selection (PICS) for the discrete-time
BM.
Lemma 2. Let B∆ (t) denote a continuous-time Brownian motion with drift ∆ > 0. A discrete-time
process is obtained by observing B∆ (t) at a (possibly) random, increasing sequence of times {ti : i =
1, 2, . . .} taking values in a given countable set. The value of ti depends on B∆ (t) only through its
values in the period [0, ti−1 ]. Define Td = min{ti : |B∆ (ti )| ≥ g(ti )} and Tc = inf {t : |B∆ (t)| ≥ g(t)},
where g(t) ∈ G . Then we have
(1) Tc ≤ Td < ∞, w.p.1,
(2) P{B∆ (Td ) ≤ −g(Td )} ≤ P{B∆ (Tc ) ≤ −g(Tc )}. (Jennison et al. 1980)
The drift of the BM refers to the difference of two competing means, which is unknown in our
setting. In the following lemma, we show that PICS of the BM with the unknown drift is bounded
by its counterpart without drift. This statement is valid for general stochastic processes, not just
for BM.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
10
Lemma 3. Let Z(·) denote a (discrete-time or continuous-time) stochastic process. (Assume that
Z(·) has right-continuous sample paths if Z(·) is a continuous-time stochastic process.) Let Z∆ (t) =
Z(t) + ∆t, where ∆ > 0. Further, define T and T∆ as the stopping times that Z(·) and Z∆ (·) first
exit the region (−g(t), g(t)), where g(t) is a continuous and non-negative function. Then, we have
that
P{Z∆ (T∆ ) ≤ −g(T∆ ), T∆ < ∞} ≤ P{Z(T ) ≤ −g(T ), T < ∞}.
Now we can establish the statistical validity of Procedure 1 and summarize it as Theorem 1.
Theorem 1. Suppose that there are k alternatives and we are interested in selecting the alternative
with the largest mean. Assume that (X1r , X2r , . . . , Xkr ), r = 1, 2, . . ., are independent and identically
distributed (i.i.d.) and jointly normally distributed with unknown mean vector (µ1 , µ2 , . . . , µk ) and
unknown positive definite covariance matrix. If µ1 > µ2 > . . . > µk , Procedure 1 stops in finite time
with probability 1 (w.p.1) and selects the best alternative (i.e., alternative 1) with probability at
least 1 − α, i.e., P{select alternative 1} ≥ 1 − α.
Remark 2. Theorem 1 only requires the covariance matrix to be positive definite. Therefore, it
allows the use of CRNs that introduces positive correlation among observations from different
alternatives to make the comparisons sharper.
Proof. For any pair of alternatives i and j with i ̸= j, since µi − µj ̸= 0, we see that the elimination
between them occurs in finite time w.p.1 by part (1) of Lemma 2. Similarly, we have all the pairwise
eliminations occur in finite time w.p.1. Therefore, Procedure 1 stops in finite time w.p.1.
When Procedure 1 stops, define an incorrect-selection (ICS) event happens when alternative 1
is eliminated by some other alternative. Then, we have
{k
}
∪
P{ICS} = P
{alternative 1 is eliminated by alternative i}
i=2
≤
k
∑
P{alternative 1 is eliminated by alternative i}.
(4)
i=2
The inequality in (4) arises from the Bonferroni inequality. For any i = 2, 3, . . . , k, we denote ICSi as
the incorrect-selection event that alternative 1 is eliminated by alternative i. To prove the statistical
validity, we view each incorrect-selection event ICSi separately. In particular, we have
P{ICSi } = P {Z1i (τ1i (Ni )) ≤ −gc (τ1i (Ni )), Ni < ∞} ,
{
}
2
2
and it follows
where Ni = min n ≥ n0 : Z1i (τ1i (n)) ≥ gc (τ1i (n)), n ∈ Z + . Let Y = (n0 − 1)S1i
/σ1i
that
{
( 2
) 2
}
σ1i
S1i
P{ICSi } = P Z1i (t1i (Ni )) ≤ −gc
t
(N
)
,
N
<
∞
i
i
2 1i
2
S1i
σ1i
}]
[ {
(
)
n0 − 1
Y
= E P Z1i (t1i (Ni )) ≤ −gc
t1i (Ni )
, Ni < ∞Y
.
Y
n0 − 1
(5)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
11
Under the normality assumption, we have shown {Z1i (t1i (n)) : n ∈ Z} can be viewed as a BM with
drift µ1 − µi observed at the discrete time points {t1i (n) : n ≥ n0 }. By Basu’s theorem (Basu 1955),
we see Y follows a chi-squared distribution with n0 − 1 degrees of freedom and is independent of
Z1i (t1i (n0 )) = t1i (n0 )[X̄1 (n0 ) − X̄i (n0 )]. Further, Y is a function of X11 − Xi1 , X12 − Xi2 , . . . , X1n0 −
Xin0 and thus is independent of X1,n0 +1 − Xi,n0 +1 , X1,n0 +2 − Xi,n0 +2 , . . .. Therefore, Y is independent
of Z1i (t1i (n)) for any n ≥ n0 . This implies that, conditioning on Y , {Z1i (t1i (n)) : n ≥ n0 } can still
be viewed as a BM with drift µ1 − µi observed at the discrete time points {t1i (n) : n ≥ n0 }. Then,
we get that
}]
)
[ {
(
Y
n0 − 1
Ti
, Ti < ∞Y
by Lemma 2
(5) ≤ E P Bµ1 −µi (Ti ) ≤ −gc
Y
n0 − 1
)
}]
[ {
(
n0 − 1
Y
≤ E P B(T ) ≤ −gc
T
, T < ∞Y
by Lemma 3
Y
n0 − 1
α
=
,
k−1
}
{
}
{
(
(
)
)
where Ti = inf t : |Bµ1 −µi (t)| ≥ gc n0Y−1 t n0Y−1 and T = inf t : |B(t)| ≥ gc n0Y−1 t n0Y−1 . The last
equation holds due to the choice of the design parameter c (see Equation (3)). Furthermore, combining with (4), we have that P{ICS} ≤ α. Therefore, the conclusion of this theorem holds.
2
In Theorem 1, we established the statistical validity of Procedure 1 for the case when all the
means are unique, i.e., µ1 > µ2 > . . . > µk . More generally, the best alternative may be unique and
there might be potential ties amongst the non-optimal alternatives, i.e., µ1 > µ2 ≥ . . . ≥ µk . For
this case, we show in the following proposition that Procedure 1 can provide a statistical guarantee
that is similar to but weaker than that of Theorem 1.
Proposition 1. Under the same assumptions as in Theorem 1, if µ1 > µ2 ≥ . . . ≥ µk ,
Procedure 1 terminates and selects alternative 1 with probability at least 1 − α, i.e.,
P{termiante and select alternative 1} ≥ 1 − α.
Proof. For any sample path, part (1) of Lemma 2 implies that Procedure 1 does not terminate in
finite time only if all the remaining alternatives have the same mean. Because the best alternative
is unique, it suffices to state that the procedure does not terminate in finite time only if alternative
1 is eliminated. Besides, we have
{
P{alternative 1 is eliminated} = P
k
∪
}
{alernative 1 is elimianted by alternative i}
i=2
≤
k
∑
i=2
P{alernative 1 is elimianted by alternative i}.
(6)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
12
The inequality above arises from the Bonferroni inequality. Notice that (6) is the same as (4),
which is shown to be no larger than α in the proof of Theorem 1. Therefore, we have
P{Procedure 1 does not terminate in finite time} ≤ P{alternative 1 is eliminated} ≤ α,
and
P{terminate and select alterantive 1} = 1 − P{alternative 1 is eliminated} ≥ 1 − α.
2
Therefore, the proposition holds.
If there are tied means among alternatives other than the best, it is possible that Procedure 1
may not terminate in finite time. Proposition 1 proves that the probability is at most α. Therefore,
in practical situations when users are unsure whether the means are unique, the Procedure with a
stopping criterion is recommended to use and the detailed discussion is deferred to Section 6.
4. Procedures with Generally Distributed Observations
In practice, observations are rarely jointly normally distributed, and may even follow different distributions across different alternatives. Therefore, we are motivated to relax the normality assumption
and extend our procedure to solve general selection-of-the-best problems.
When observations are generally distributed, it appears difficult to design procedures that deliver
a finite-sample statistical validity as we do in Section 3. Therefore, we propose a new procedure that
delivers an asymptotic statistical validity, i.e., the procedure satisfies Equation (1) in a meaningful
limit.
Recall that, to design a sequential procedure, we face two issues: (a) constructing a Brownian
approximation to the partial-sum difference process, and (b) designing a proper continuation region.
For the case of generally distributed observations, neither of these two issues are trivial.
Consider a pairwise comparison between alternatives 1 and i with i ̸= 1. As in Equation (2),
let Z1i (·) denote the partial-sum difference process on a set of time points {t1i (n) : n = 1, 2, . . .}.
Under the normality assumption, we have shown in Section 3 that {Z1i (t1i (n)) : n = 1, 2, . . .} can be
viewed as a Brownian motion process observed at discrete time points. However, this will not be
true for a general case. Therefore, we consider a standardized version of {Z1i (t1i (n)) : n = 1, 2, . . .}
with a scaling parameter M , that is
⌊M
∑s⌋
Z1i (t1i (⌊M s⌋)) − t1i (M s)(µ1 − µi )
√
C1i (M, t1i (s)) :=
=
M
l=1
(X1l − Xil ) − M s(µ1 − µi )
√
, for s > 0(7)
2
M σ1i
where ⌊x⌋ denotes the largest integer that is no bigger than x. Then we approximate (7) by a
BM in the limit using the following Functional Central Limit Theorem (see, e.g., Theorem 4.3.2 in
Whitt 2002a).
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
13
Lemma 4 (Functional Central Limit Theorem). Suppose that {Xi : i ≥ 1} is a sequence of
i.i.d. random variables with mean µ and finite variance σ 2 . Then,
∑⌊nt⌋
i=1 Xi − ntµ d
√
→ B(t), as n → ∞, in D[0, ∞),
nσ
d
where → denotes convergence in distribution and D is defined as the Skorohod space.
According to Lemma 4 and the self-similarity scaling property of a BM, we have that
−1
C1i (M, t1i (s)) → σ1i
B(s) = B(t1i (s)), as M → ∞, in D[0, ∞),
d
d
(8)
d
where X = Y denotes that X and Y are identically distributed. Even though C1i (M, t1i (s)) is
defined for a general distribution, Lemma 4 guarantees its convergence to a known BM in the limit.
We call the parameter M the scaling parameter because it determines how the partial-sum
process Zi1 (t1i (n)) is scaled and how fast the scaled process Ci1 (M, t1i (s)) converges to a BM. To
ensure the convergence in (8), we only need M to grow to infinity.
We address the second issue by choosing one boundary gc (t) from the set G , similar to that
in Section 3. To determine its design parameter c, we need to evaluate the first-exit probability,
which shrinks to zero in the asymptotic regime. In this situation, it becomes impractical to find
c numerically as in Appendix B. Fortunately, there is a body of literature on asymptotic approximations of this type of first-exit probabilities. For instance, Jennen and Lerche (1981) provide the
asymptotic approximations for a list of boundaries. In the following lemma, we present one of these
√
approximations where the boundary is gc (t) = [c + log(t + 1)](t + 1). In this lemma, we need to
use the concept of asymptotic equivalence. In particular, we say f (c) is asymptotically equivalent
c→∞
to g(c) with respect to c, denoted as, f (c) ∼ g(c), if limc→∞ f (c)/g(c) = 1.
√
Lemma 5 (Jennen and Lerche (1981)). Let gc (t) = [c + log(t + 1)](t + 1) (c ≥ 0), a nonnegative continuous function, and let B(t) be a Brownian motion without drift. Then P{B(T ) ≤
c→∞ 1 −c/2
e
,
2
−gc (T ), T < ∞} ∼
√
where T = inf {t : |B(t)| ≥ gc (t)}.
Lemma 5 provides a closed-form approximation to the first-exit probability when gc (t) =
(c + log(t + 1))(t + 1). This allows us to choose the value of c more easily compared to the root-
finding problem (3) in Section 3, although this value is only an approximation.
4.1. The Procedure
In this subsection, we propose a sequential procedure for generally distributed observations, i.e.,
Procedure 2.
Procedure 2 (Sequential Procedure for Generally Distributed Observations)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
14
Setup. Select the PCS 1 − α (0 < α ≤ 1 − 1/k), and a common first-stage sample size n0 ≥ 2.
√
Choose the boundary as gc (t) = [c + log(t + 1)](t + 1) and set c = −2 log(2α/(k − 1)).
Initialization. Let I = {1, 2, . . . , k } be the set of alternatives in contention. For all i = 1, 2, . . . , k,
simulate n0 observations Xi1 , Xi2 , . . . , Xin0 from alternative i and calculate its sample mean as
0
1 ∑
X̄i (n0 ) =
Xil .
n0 l=1
n
For any i, j with i ̸= j, calculate the sample variance of their difference,
2
Sij
(n0 )
n0
[
]2
1 ∑
Xir − Xjr − (X̄i (n0 ) − X̄j (n0 )) .
=
n0 − 1 r=1
Set n = n0 .
Update. If n > n0 , calculate the sample mean for each alternative i ∈ I, and for any pair of
2
alternatives i, j ∈ I with i ̸= j, update the sample variance Sij
(n) of their difference.
Screening. Let I old = I and
2
τij (n) = n/Sij
(n),
Define
and
Zij (τij (n)) = τij (n)[X̄i (n) − X̄j (n)].
{
}
I = I old \ i ∈ I old : Zij (τij (n)) ≤ −gc (τij (n)) for some other j ∈ I old .
Stopping Rule. If |I | = 1, stop and select the alternative whose index is in I as the best. Otherwise,
take one additional observation Xi,n+1 from each alternative i for i ∈ I, set n = n + 1 and go to
Update.
Remark 3. Similar to Procedure 1, Procedure 2 can also work with many choices of boundaries
√
(i.e., G ). As an instance, Procedure 2 uses a specific boundary gc (t) = [c + log(t + 1)](t + 1) and
the following results for Procedure 2 are shown for this specific boundary. Nevertheless, these
results can be easily extended to other boundaries in G .
Procedure 2 differs from Procedure 1 in two aspects. First, Procedure 2 determines the value of
c using an asymptotic result while Procedure 1 does it by solving a stochastic root-finding problem
(i.e., Equation (3)). Second, Procedure 2 updates the sample variances sequentially as more and
more observations are generated. This enables us to establish the asymptotic validity of Procedure
2 based on the strong consistency of the variance estimators.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
15
4.2. Statistical Validity
Procedure 2 is only parameterized by the PCS value 1 − α. Therefore, we consider an asymptotic
regime, that is, PCS goes to 1 (or α → 0). Under this asymptotic regime, a procedure is asymptotically valid if its actual probability of incorrect selection (PICS) not only converges to 0 as α
converges to 0 but also converges at least as fast as α does, i.e., lim sup P{ICS}/α ≤ 1.
α→0
To show the asymptotic validity of Procedure 2, we need Lemma 6. It shows that the first-exit
time goes to infinity as α → 0 (or equivalently, c → ∞) with probability 1 (w.p.1). Based on this
lemma, we can prove that the variance estimators at the exit time converge to the true values as
α → 0. This allows us to treat the unknown variances essentially as their true values when analyzing
the asymptotic statistical validity of Procedure 2.
Lemma 6. Let {Yi : i = 1, 2, . . .} denote i.i.d. random variables with E[Y1 ] = µ, and define Z(n) =
∑n
+
i=1 Yi , for n = 1, 2, . . .. Let Nc = min{n ∈ Z : |Z(n)| ≥ gc (n)}, where gc (t) is defined in Procedure
2. Then Nc → ∞, w.p.1, as c → ∞.
Now we establish the asymptotic validity of Procedure 2 and summarize it in the following
theorem. The proof of this theorem is included in Appendix A.
Theorem 2. Suppose that there are k alternatives and we are interested in selecting the alternative
with the largest mean. Assume that (X1r , X2r , . . . , Xkr ), r = 1, 2, . . ., are independent and identically
jointly distributed with unknown mean vector (µ1 , µ2 , . . . , µk ), and their moment generating function
exists in a neighborhood of (0, 0, . . . , 0) ∈ Rk . Further, let the first-stage sample size n0 be a function
of α that satisfies n0 → ∞ as α → 0. If µ1 > µ2 > . . . > µk , Procedure 2 stops in finite time w.p.1
and satisfies lim sup P{alternative 1 is eliminated }/α ≤ 1.
α→0
Remark 4. To ensure the strong consistency of sequentially updated variance estimators, in addition to Lemma 6, we put some technical conditions on the first-stage sample size n0 , as listed
in Theorem 2. Although the theoretical condition facilitates the asymptotic proof, it does not
prescribe a specific setting of n0 in practice.
Similar to Proposition 1, we establish Proposition 2 for Procedure 2 in the case when the best
alternative is unique but the means are not all unique.
Proposition 2. Under the same assumptions as in Theorem 2, if µ1 > µ2 ≥ . . . ≥ µk , Procedure
2 terminates and eliminates alternative 1 with probability at most α in an asymptotic regime, i.e.,
lim sup P{terminate and eliminate alternative 1}/α ≤ 1.
α→0
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
16
5. Asymptotic Efficiency
When choosing a frequentist selection-of-the-best procedure, one not only requires the procedure to
deliver the guaranteed PCS, but also wants the procedure to be efficient, i.e., using as few observations as possible to select the best. To understand the efficiency of our procedures, we consider the
the general case when the observations are jointly generally distributed. In this situation, we focus
on our asymptotic procedure (i.e., Procedure 2) and study its average sample size asymptotically.
In Procedure 2, the strong consistency of variance estimators leads to an asymptotic analysis
that is similar to the known-variances case. Therefore, for simplicity, we assume that the variances
are known in this section. In Section 5.1 we provide an analytic expression for the asymptotic
average sample size required by Procedure 2, and in Section 5.2 we compare it to those of existing
IZ procedures.
5.1. Asymptotic Average Sample Sizes
As a building block, we consider a pairwise comparison between alternatives 1 and i with mean
difference µ1 − µi . Lemma 6 implies that the sample size required to detect them grows to infinity
in the asymptotic regime. In this case, their partial-sum difference process behaves more and more
like a line with slope µ1 − µi . Therefore, a heuristic method to calculate the average sample size
is to find the intersection point between the boundary and the line (see Figure 2). The method
is called the mean path approximation. Perng (1969) provides a rigorous proof of this heuristic
method for Paulson’s procedure and the proof can be extended easily to sequential IZ procedures
with a triangular continuation region (such as the KN procedure). The result is summarized in
Theorem 3.
(µ1 − µi )t
g(t)
Ni
−g(t)
Figure 2
Heuristic Mean Path Approximation Method
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
17
Theorem 3 (Perng (1969)). Suppose that there are k alternatives and we are interested in
selecting the alternative with the largest mean. Assume that (X1r , X2r , . . . , Xkr ), r = 1, 2, . . ., are
independent and identically jointly distributed with unknown mean vector (µ1 , µ2 , . . . , µk ) and a
positive definite covariance matrix. Further, assume µ1 − δ ≥ µ2 ≥ . . . ≥ µk , where δ > 0 is the IZ
parameter. Let (−aij + λij t, aij − λij t) (0 < λij < δ, 0 ≤ t ≤ aij /λij ) be the triangular region used
for a pairwise comparison between alternatives i and j. Define Ni as the sample size required to
distinguish µ1 and µi , for i = 2, 3, . . . , k, and define 1 − α as the probability of correct selection.
Then,
α→0
E[Ni ] ∼
2
where σ1i
= Var[X11 − Xi1 ].
2
a1i σ1i
, for all i = 2, 3, . . . , k,
µ1 − µi + λ1i
In particular, the KN procedure often chooses λij = δ/2 and aij = − 1δ log(2α/(k − 1)). Based
on Theorem 3, it is straightforward to derive a theoretical expression for the asymptotic average
sample size of the KN procedure, i.e.,
α→0
EKN [Ni ] ∼
2
f 2 σ1i
, for i = 2, 3, . . . , k,
(µ1 − µi + δ/2)δ
(9)
where f 2 = − log(2α/(k − 1)).
However, a rigorous proof has not been given so far for the sequential procedures with a general
boundary, such as the one used in Procedure 2. In next theorem, we accomplish the task and show
this heuristic method is also valid for Procedure 2 under mild conditions. The proof is included in
the appendix.
Theorem 4. Suppose that there are k alternatives and we are interested in selecting the alternative
with the largest mean. Assume that (X1r , X2r , . . . , Xkr ), r = 1, 2, . . ., are independent and identically
jointly distributed with unknown mean vector (µ1 , µ2 , . . . , µk ), and their moment generating function
exists in the neighborhood of (0, 0, . . . , 0) ∈ Rk . Let (−gc (t), gc (t)) denote the continuation region
√
in Procedure 2, where gc (t) = [c + log(t + 1)](t + 1). Define Ni as the sample size required to
distinguish µ1 and µi , for i = 2, 3, . . . , k, and define 1 − α as the probability of correct selection. If
µ1 > µ2 > . . . > µk , then, in Procedure 2, we have
2
c σ1i
α→0
, for all i = 2, 3, . . . , k,
E[Ni ] ∼
(µ1 − µi )2
2
= Var[X11 − Xi1 ].
where σ1i
(10)
Remark 5. Although the theorem is proven when the boundary function is gc (t) =
√
[c + log(t + 1)](t + 1), the proof can be easily extended to other boundaries.
Theorem 4 provides a theoretical foundation to evaluate the asymptotic average sample size
using the mean path approximation. For the comparison between alternatives 1 and i, it shows
that the asymptotic average sample size is inversely proportional to (µ1 − µi )2 , i.e., the square of
their true mean difference.
18
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
5.2. Comparisons between Procedure 2 and IZ Procedures
In this subsection we compare the asymptotic average sample sizes of Procedure 2 to the KN++
procedure. There are two reasons why we choose the KN++ procedure as the benchmark. First, the
KN++ procedure is the most similar procedure to Procedure 2. Both allow unknown and unequal
variances and CRNs, and both break the comparisons into pairwise comparisons and approximate
the partial sums of differences by Brownian motions. Second, the KN++ procedure, as the most
efficient one in the family of the KN procedures, has been well studied and compared in the
literature (Branke et al. (2007), Wang and Kim (2013)) and implemented by commercial simulation
software. Notice that it is not our intention to argue that the KN++ procedure is the most efficient
procedure in the literature. Indeed, it is not. When there are a large number of alternatives with
equal unknown variance, the BIZ procedure of Frazier (2014) is more efficient because it avoids the
use of Bonferroni inequality to break the comparisons into pairs, which causes inefficiency for the
KN++ procedure and our procedures as well. Therefore, when the number of alternatives is large,
we also expect the BIZ procedure to outperform ours. However, the objective of this comparison
is to show that the avoidance of setting IZ parameters may give an advantage to our procedures.
Even though the comparison is done between the KN++ procedure and Procedure 2, we believe
that the insights obtained here hold robustly for all IZ procedures.
Since both the KN++ procedure and Procedure 2 decompose the selection-of-the-best problem
into pairwise comparisons, we compare their asymptotic average sample sizes for the case of two
alternatives (i.e., k = 2). Let ∆ denote the difference of their means and set the variance of their
difference to 1. It is easy to calculate the asymptotic average sample size required by Procedure
2 based on Theorem 4. Meanwhile, given any specified IZ parameter δ, the asymptotic average
sample size of the KN++ procedure can be also evaluated according to Theorem 3.
To compare the asymptotic average sample sizes, we vary ∆/δ, which indicates how the chosen
IZ parameter relates to the true mean difference. In Figure 3, we plot the ratios of sample sizes of
√
the KN++ procedure and Procedure 2 with boundary [c + log(t + 1)](t + 1) for different values of
∆/δ. From Figure 3, we find that the KN++ Procedure needs about one fifth samples of Procedure
2 when ∆/δ = 1. However, when ∆/δ is above 5, Procedure 2 begins to out-perform the KN++
procedure. Particularly, as ∆/δ becomes larger and larger than 5, Procedure 2 requires fewer and
fewer samples than the KN++ procedure.
Even though Procedure 2 appears to require more samples when ∆/δ is smaller than 5, the
disadvantage may vanish when the number of alternatives is large. In this situation, there are often
many alternatives whose means are significantly smaller than the best. This may result in large
values of ∆/δ for many pairwise comparisons and thus increase the total number of observations
required by the KN++ procedure relative to Procedure 2.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
Figure 3
Table 1
19
Ratios of asymptotic average sample sizes using the KN++ procedure and Procedure 2 (with the bound√
ary gc (t) = [c + log(t + 1)](t + 1) ) when ∆/δ varies.
Asymptotic average sample sizes of the Rinott’s procedure, the KN++ procedure and Procedure 2 to select from
two random variables whose difference is of mean ∆ and variance σ 2 .
Procedures
Asymptotic Sample Size
Rinott’s Procedure
KN++ Procedure
Procedure 2
h2 σ 2
δ2
f 2 σ2
(∆−δ/2)δ
cσ 2
∆2
Note: h2 is the Rinott’s constant (see Rinott 1978).
To close this section, we summarize the asymptotic average samples sizes required by Procedure
2, the KN++ procedure as well as the Rinott’s procedure, and list them in Table 1. For simplicity,
Table 1 considers a pairwise comparison where ∆ and σ 2 are the mean and variance of their
difference. We have two findings from the table and they are consistent with the findings from
Figure 3. First, both the KN++ and Rinott’s procedures are sensitive to the choice of δ. In particular,
the asymptotic sample sizes of the KN++ and Rinott’s procedures are inversely proportional to δ
and its square. Second, the asymptotic sample size of Procedure 2 is immune to the choice of δ
because Procedure 2 is IZ-free. Therefore, Procedure 2 is relatively most efficient if δ is chosen too
conservatively.
6. A Stopping Criterion
One drawback of Procedures 1 and 2 is that the continuation regions are open on the right-hand
side, which may potentially cause the procedures to run very long time before they stop or maybe
never stop. As a remedy, part (1) of Lemma 2 shows that these procedures can stop in finite time
w.p.1 in each pairwise comparison as long as the two alternatives have different means.
20
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
However, there might be two or more alternatives that have the same means. Then, there is
a positive probability that our procedures may not terminate (seen from Propositions 1 and 2 ),
resulting in an infinite sample size. To avoid this problem, we design truncated procedures, which
force the original procedures to stop when a stopping criterion is satisfied, even if there is more
than one alternative in contention.
Beyond avoiding an infinite sample size, users may have practical reasons for designing such
stopping criteria. For instance, they may have a practically significant difference below which difference in mean performances is not meaningful. This could arise if the resolution of the simulation
model itself is limited (e.g., nearest $10, 000), or if the precision of the software itself is finite (e.g.,
5 decimal digits is the default in Matlab). We refer to this limit as the error tolerance ε. Therefore,
the revised goal is to select an alternative that is within ε of the best, and to design a stopping
criterion that terminates once this goal is achieved.
Now we illustrate how to design a stopping criterion that corresponds to the error tolerance ε.
Denote Iε as the set of alternatives whose means are within ε of the best and the goal is to select
an alternative that is in Iε . Intuitively, we should choose a termination time T ∗ (ε) such that only
alternatives in Iε can survive at T ∗ (ε). In particular, we design T ∗ (ε) as follows:
Stopping Criterion Determine a termination time T ∗ (ε) that is a root to T ε − gc (T ) = 0.
Basically, the choice of T ∗ (ε) above is driven by the mean path approximation (see, Theorem
4). This method shows that the asymptotic average time required to eliminate all the alternatives
outside Iε is bounded by the value which is a root of T ε − gc (T ) = 0. In light of this, we determine
T ∗ (ε) as stated in Stopping Criterion above.
Provided the stopping criterion, we design a truncated procedure, i.e. Procedure 3, that is the
same as Procedure 2 except adding a stopping rule as follows: if |I | > 1 and τij (n) ≥ T ∗ (ε) for all
i, j ∈ I with i ̸= j, let i∗ = arg maxi∈I X̄i (n) and return i∗ . In the following theorem, we show
that Procedure 3 achieves the goal of selecting an alternative within ε of the best, with at least
the user-specified PCS in the limit.
Theorem 5. Assume that (X1r , X2r , . . . , Xkr ), r = 1, 2, . . ., are independent and identically jointly
distributed with unknown mean vector (µ1 , µ2 , . . . , µk ) and their moment generating function exists
in a neighborhood of (0, 0, . . . , 0) ∈ Rk . Further, let the first-stage sample size n0 be a function of
α that satisfies n0 → ∞ as α → 0. Without loss of the generality, assume that µ1 ≥ µ2 ≥ . . . ≥ µk .
Let ε denote the tolerance error and 1 − α denote the user-specified PCS. Let i∗ be the alternative
selected by Procedure 3. Then, we have, lim sup P{µi∗ < µ1 − ε}/α ≤ 1.
α→0
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
21
Remark 6. We refer to “correct selection” as selecting the best alternative and refer to “good
selection” as selecting any alternative within ε of the best. (The definitions are consistent with those
in Ni et al. (2014).) Theorem 5 states that Procedure 3 provides a good selection guarantee when
µ1 ≤ µ2 + ε. Furthermore, when µ1 > µ2 + ε, Procedure 3 provides a correct selection guarantee
because {µi∗ ≥ µ1 − ε} implies i∗ = 1. Therefore, Procedure 3 can provide both correct selection
and good selection guarantees.
If one treats ε as an IZ parameter, Procedure 3 turns into an IZ procedure because the selected
alternative is within ε of the best (shown by Theorem 5). Nevertheless, we want to emphasize that
the concept of an error tolerance is different from an IZ parameter. When the difference between
two alternatives is larger than ε, the average sample size required by the existing IZ procedures
depend critically on the choice of ε. In other words, a conservative ε may lead to excessive samples.
In contrast, a conservative ε will in general not affect the required sample size as our procedures
are expected to stop long before they reach the maximal sample size (due to Theorem 4). Another
way to see this is that the continuation regions of our procedures are not a function of ε. Therefore,
an error tolerance can be set very conservatively.
7. Numerical Experiments
In this section, we test the performance of our procedures through extensive numerical experiments
and compare them with existing procedures. Firstly, we establish the small-sample performance of
Procedures 1 and 2. Secondly, we consider the selection-of-the-best problem with various numbers of
alternatives. In this experiment, we compare the efficiency of Procedure 2 with the KN++ procedure
under three different configurations of variances. Thirdly, we test Procedure 3 that incorporates an
error tolerance and compare it with IZ procedures under different configurations of means. Lastly,
we test our procedures by addressing a realistic problem.
For simplicity, CRNs are not used in the experiments because using CRNs to increase their
efficiency is not our focus.
7.1. Small-Sample Performance
To get rid of the effect of using the Bonferroni inequality, we consider the case of only two alternatives. Assume that their observations are normally distributed with mean (µ1 , µ2 ) and variance
(5, 5). Further, assume alternative 1 is the better alternative which we would like to select. In this
experiment, we implement Procedures 1 and 2 to select alternative 1 with a desired PCS 0.95. For
different combinations of (µ1 , µ2 ) and the first-stage sample size n0 , we report the approximated
values of c, the estimated PCS and the average sample size with 95% confidence interval across
10,000 microreplications in Table 2 .
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
22
Table 2
The estimated PCS (Est. PCS) and the average sample sizes (Avg. SSize) with 95% confidence intervals
for Procedures 1 and 2 when observations are normally distributed.
µ1 − µ2
0.05
0.20
n0
10
15
20
25
30
35
40
10
15
20
25
30
35
40
Procedure 1
c
Est. PCS
Avg. SSize
7.358
0.97
(1.149 ± 0.019) × 105
5.870
0.97
(9.905 ± 0.160) × 104
5.163
0.97
(9.181 ± 0.148) × 104
5.040
0.97
(9.124 ± 0.142) × 104
4.671
0.97
(8.710 ± 0.138) × 104
4.514
0.97
(8.509 ± 0.136) × 104
4.415
0.98
(8.411 ± 0.133) × 104
7.358
0.99
(5.849 ± 0.097) × 103
5.870
0.99
(4.971 ± 0.083) × 103
5.163
0.99
(4.526 ± 0.078) × 103
5.040
0.99
(4.427 ± 0.074) × 103
4.671
0.99
(4.173 ± 0.070) × 103
4.514
0.99
(4.181 ± 0.070) × 103
4.415
0.99
(4.077 ± 0.068) × 103
Procedure 2
Est. PCS
Avg. SSize
0.96
(8.126 ± 0.125) × 104
0.97
(8.298 ± 0.127) × 104
0.97
(8.393 ± 0.127) × 104
4.605
0.97
(8.518 ± 0.126) × 104
0.98
(8.536 ± 0.126) × 104
0.98
(8.500 ± 0.125) × 104
0.98
(8.492 ± 0.125) × 104
0.98
(3.952 ± 0.067) × 103
0.99
(4.020 ± 0.065) × 103
0.99
(4.070 ± 0.067) × 103
4.605
0.99
(4.057 ± 0.066) × 103
0.99
(4.120 ± 0.066) × 103
0.99
(4.108 ± 0.067) × 103
0.99
(4.124 ± 0.060) × 103
c
From Table 2, we obtain several findings. First, both procedures deliver a larger PCS than the
desired PCS, and their delivered PCS decreases slightly as the true mean difference ∆ decreases
from 0.20 to 0.05. This arises because both procedures are designed for the case when the mean
difference is zero. Second, the average sample size of Procedure 1 depends on the setting of n0 .
In particular, for a given desired PCS, a smaller n0 leads to a larger sample size. This occurs
because Procedure 1 explicitly accounts for the (n0 − 1) degrees of freedom of the variance estimator
through using different values of c. Third, Procedure 1 often needs more samples than Procedure
2 does, but their difference tends to be smaller as n0 increases. Specially, as n0 increases to 40,
Procedure 1 needs fewer observations than Procedure 2 does. To understand this, as n0 increases,
the randomness coming from variance estimators fades away and the corresponding value of c for
Procedure 1 tends to be smaller and even less conservative (or smaller) than that of Procedure 2,
which is obtained by asymptotic approximation.
Table 2 shows that both procedures can be used to select the best in this experiment. Therefore,
users may face the choice between Procedures 1 and 2, and our suggestions are as follows. When
observations are normally distributed, Procedure 1 would be preferred unless it is significantly
less efficient than Procedure 2, because Procedure 1 guarantees PCS in small samples. Otherwise,
Procedure 2 would be preferred because it guarantees PCS although in an asymptotic regime while
Procedure 1 can not. Besides, Procedure 2 is more easily used because of its closed-form expressions
for the design parameter c and achieves a higher efficiency because it updates variance estimators
sequentially.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
23
7.2. Large-scale Selection of the Best
In practice, the means of alternatives are often spread out when the number of alternatives is
large. In light of this, we consider a monotone decreasing configuration of means (µ1 , µ2 , . . . , µk )
where µi = 1.5 − 0.5i, and an equal-variance configuration where σi2 = 10 for all i. Our target is to
select alternative 1 with a desired PCS 0.95. To show the performance of our procedure to solve
large-scale selections of the best, we vary the number of alternatives as k = 20, 50, 100, 500.
To solve this problem, we use Procedure 2 for simplicity of the simulation study because its
design parameter c is more easily determined. As a comparison, we implement the KN procedure
by choosing an IZ parameter δ. It is worthwhile emphasizing that δ is chosen to enable the implementation of the KN procedure and we do not refer it to the smallest difference users care to detect
in this experiment. Further, to make a fair comparison, we compare Procedure 2 with the KN++
procedure because both of them update variance estimators sequentially. In this experiment, we
set n0 to 10.
Table 3
KN++
k
The estimated PCS and average sample sizes with 95% confidence interval for Procedure 2 and the
procedure under equal configuration of variances when all observations are normally distributed.
Procedure 2
0.99
20
2.816 × 103
±0.082 × 103
1.00
50
3.588 × 103
±0.089 × 103
1.00
100
4.388 × 103
±0.089 × 103
0.99
500
9.138 × 103
±0.102 × 103
1/4
0.70
2.639 × 102
±0.020 × 102
0.72
5.904 × 102
±0.023 × 102
0.76
1.112 × 103
±0.003 × 103
0.79
5.164 × 103
±0.003 × 103
1/2
0.91
5.503 × 102
±0.064 × 102
0.94
9.619 × 102
±0.075 × 102
0.95
1.540 × 103
±0.008 × 103
0.97
5.721 × 103
±0.009 × 103
KN++ procedure: (µ1 − µ2 )/δ
1
2
1.00
1.00
1.371 × 103
3.247 × 103
3
±0.016 × 10 ±0.029 × 103
0.99
1.00
2.014 × 103
4.454 × 103
±0.018 × 103 ±0.033 × 103
0.99
1.00
2.710 × 103
5.506 × 103
±0.020 × 103 ±0.034 × 103
1.00
1.00
7.217 × 103
1.077 × 104
±0.021 × 103 ±0.004 × 104
4
1.00
7.045 × 103
±0.050 × 103
1.00
9.779 × 103
±0.054 × 103
1.00
1.179 × 104
±0.006 × 104
1.00
1.877 × 104
±0.006 × 104
8
1.00
1.470 × 104
±0.007 × 104
1.00
2.070 × 104
±0.008 × 104
1.00
2.514 × 104
±0.009 × 104
1.00
3.628 × 104
±0.009 × 104
To compare Procedure 2 with the KN++ procedure, we consider different settings of δ where
(µ1 − µ2 )/δ = 1/4, 1/2, 1, 2, 4, 8. In Table 3, we report the estimated PCS and the average sample
sizes with 95% confidence intervals based on 1000 macroreplications of each procedure. From the
table, we obtain the following three findings. Firstly, when users select δ that is larger than µ1 − µ2
in the KN++ procedure, they may suffer from a lower PCS than they require, seen from the third
and fourth columns. Secondly, for any fixed number of alternatives (see each row in Table 3),
Procedure 2 demands fewer observations than the KN++ procedure, when δ is much smaller than
the true difference µ1 − µ2 . Furthermore, the gap of their sample sizes becomes larger as δ becomes
smaller. Thirdly, from the second and fifth columns, we can conclude that, although Procedure 2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
24
needs more observations than the KN++ procedure when δ is exactly µ1 − µ2 , the relative difference
between their average sample sizes tends to be narrower as the number of alternatives k increases.
This implies the average sample size required by Procedure 2 grows more slowly with k than the
KN++ procedure.
Furthermore, Procedure 2 is shown to be asymptotically valid when the observations are generally
distributed. To show its small-sample performance for this general case, we repeat the experiment
above using the same parameters except that all the observations are exponentially distributed.
The results are reported in Tables 4 and they are similar to that of the normally distributed case.
Table 4
The estimated PCS and the average sample sizes with 95% confidence intervals of Procedure 2 and the
KN++ procedure when all observations are exponentially distributed.
k
Procedure 2
0.99
20
2.902 × 103
±0.084 × 103
0.99
50
3.610 × 103
±0.088 × 103
1.00
100
4.396 × 103
±0.092 × 103
1.00
500
9.095 × 103
±0.104 × 103
1/4
0.71
2.740 × 102
±0.022 × 102
0.74
5.972 × 102
±0.026 × 102
0.74
1.119 × 103
±0.003 × 103
0.79
5.167 × 103
±0.003 × 103
1/2
0.90
5.470 × 102
±0.067 × 102
0.95
9.430 × 102
±0.074 × 102
0.95
1.525 × 103
±0.008 × 103
1.00
5.699 × 103
±0.010 × 103
KN++ procedure: (µ1 − µ2 )/δ
1
2
0.99
1.00
1.344 × 103
3.143 × 103
3
±0.015 × 10 ±0.028 × 103
1.00
1.00
1.932 × 103
4.215 × 103
±0.017 × 103 ±0.030 × 103
1.00
1.00
2.627 × 103
5.306 × 103
±0.018 × 103 ±0.031 × 103
1.00
1.00
7.114 × 103
1.051 × 104
±0.018 × 103 ±0.004 × 104
4
1.00
6.927 × 103
±0.046 × 103
1.00
9.316 × 103
±0.052 × 103
1.00
1.124 × 104
±0.006 × 104
1.00
1.806 × 104
±0.006 × 104
8
1.00
1.463 × 104
±0.007 × 104
1.00
2.020 × 104
±0.008 × 104
1.00
2.414 × 104
±0.009 × 104
1.00
3.483 × 104
±0.009 × 104
Last, we further investigate the efficiency of Procedure 2 and the KN++ Procedure in equal,
increasing and decreasing configurations of variances where σi2 = 10, 10 × (0.95 + 0.05i) and
10/(0.95 + 0.05i) for i = 1, 2, . . . , k, respectively. To avoid reporting the similar results, in the following Table 5 we only the report the results when observations are normally distributed. From
this table, we find that the same conclusions seen from Tables 3 and 4 are valid.
7.3. Comparisons with IZ Procedures
In this subsection, we consider a selection-of-the-best problem with an error tolerance ε and solve
this problem using Procedure 3. If one treats ε as an IZ parameter, IZ procedures can also be used
to solve this problem. In this experiment we choose the (two-stage) Rinott’s procedure and the
(fully sequential) KN++ procedures as representatives of IZ procedures, and compare Procedure 3
with them in terms of the efficiency.
We consider two different configurations of means. The first one is the slippage configuration (SC)
where µ1 = 0.5, µ2 = µ3 = . . . = µ50 = 0. The second one is the monotone decreasing configuration
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
Table 5
25
The estimated PCS and average sample sizes with 95% confidence interval for Procedure 2 and the
KN++ procedure under different configurations of variances when all observations are normally distributed.
k
Variances
Decreasing
20
Equal
Increasing
Decreasing
50
Equal
Increasing
Decreasing
100
Equal
Increasing
Procedure 2
0.99
2.780 × 103
±0.081 × 103
0.99
2.816 × 103
±0.082 × 103
0.99
2.916 × 103
±0.085 × 103
0.99
3.551 × 103
±0.086 × 103
1.00
3.588 × 103
±0.089 × 103
0.99
3.635 × 103
±0.090 × 103
1.00
4.351 × 103
±0.093 × 103
1.00
4.388 × 103
±0.089 × 103
0.99
4.444 × 103
±0.093 × 103
1
0.99
1.326 × 103
±0.016 × 103
1.00
1.371 × 103
±0.016 × 103
0.99
1.473 × 103
±0.017 × 103
1.00
1.881 × 103
±0.017 × 103
0.99
2.014 × 103
±0.018 × 103
0.99
2.147 × 103
±0.018 × 103
1.00
2.592 × 103
±0.019 × 103
0.99
2.710 × 103
±0.020 × 103
0.99
2.934 × 103
±0.020 × 103
KN++ procedure: (µ1 − µ2 )/δ
2
4
1.00
1.00
3.092 × 103
6.720 × 103
3
±0.028 × 10 ±0.049 × 103
1.00
1.00
3.247 × 103
7.045 × 103
±0.029 × 103 ±0.050 × 103
1.00
1.00
3.423 × 103
7.437 × 103
±0.031 × 103 ±0.049 × 103
1.00
1.00
4.120 × 103
8.970 × 103
±0.033 × 103 ±0.052 × 103
1.00
1.00
4.454 × 103
9.779 × 103
±0.033 × 103 ±0.054 × 103
1.00
1.00
4.958 × 103
1.085 × 104
±0.034 × 103 ±0.005 × 104
1.00
1.00
5.061 × 103
1.060 × 104
±0.033 × 103 ±0.006 × 104
1.00
1.00
5.506 × 103
1.179 × 104
±0.034 × 103 ±0.006 × 104
1.00
1.00
6.350 × 103
1.412 × 104
±0.036 × 103 ±0.006 × 104
8
1.00
1.411 × 104
±0.007 × 104
1.00
1.470 × 104
±0.007 × 104
1.00
1.536 × 104
±0.008 × 104
1.00
1.910 × 104
±0.008 × 104
1.00
2.070 × 104
±0.008 × 104
1.00
2.268 × 104
±0.002 × 104
1.00
2.238 × 104
±0.009 × 104
1.00
2.514 × 104
±0.009 × 104
1.00
3.000 × 104
±0.009 × 104
of means (MDM) where µi = 1 − 0.5i for i = 1, 2, . . . , 50. For both configurations of means, we use
the equal-variance configuration where σi2 = 10, for all i = 1, 2, . . . , 50. For each configuration, we
select the best using three different procedures and report their average sample sizes based on 1000
macroreplications of each procedure in Table 6. We set PCS to 0.95 and n0 to 10.
From Table 6, we obtain the following insights. Firstly, the last column shows that the sample size
required by Procedure 3 appears stable as the ratio (µ1 − µ2 )/ε varies under MDM. It follows from
that, under MDM, Procedure 3 tends to terminate early before reaching the stopping criterion.
Secondly, Procedure 3 has a wider confidence interval under SC than the other two procedures.
This occurs because Procedure 3 either stops long before reaching the stopping criterion or stops
when the stopping criterion is met. This implies that Procedure 3 effectively avoids infinite samples.
Thirdly, from the first two columns, we find that the sample size required by the Rinott’s procedure
increases nearly 4 times as (µ1 − µ2 )/ε increases from 1 to 2, and increases nearly 25 times as
(µ1 − µ2 )/ε increases from 1 to 5. Similar results hold for the KN++ procedure although it increases
more slowly. Therefore, the Rinott’s procedure is the most sensitive to the choice of ε, then the
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
26
Table 6
Average sample sizes with 95% confidence intervals for the Rinott’s procedure, the KN++ procedure
and Procedure 3.
(µ1 − µ2 )/ε
1
2
5
(µ1 − µ2 )/ε
1
2
5
Normally Distributed Observations
SC
MDM
Rinott
KN++ Procedure 3
Rinott
KN++
6.071 × 104
1.576 × 104
4.781 × 104
6.045 × 104
2.002 × 103
±0.035 × 104 ±0.061 × 104 ±0.025 × 104
±0.025 × 104 ±0.018 × 103
5
4
4
2.421 × 10
4.022 × 10
6.430 × 10
2.426 × 105
4.425 × 103
5
4
4
5
±0.010 × 10 ±0.042 × 10 ±0.217 × 10
±0.010 × 10 ±0.033 × 103
6
5
4
1.515 × 10
1.129 × 10
8.857 × 10
1.518 × 106
1.254 × 104
6
5
4
6
±0.006 × 10 ±0.008 × 10 ±1.354 × 10
±0.006 × 10 ±0.006 × 104
Exponentially Distributed Observations
SC
MDM
Rinott
KN++ Procedure 3
Rinott
KN++
4
4
4
4
6.071 × 10
1.602 × 10
4.879 × 10
6.046 × 10
1.933 × 103
4
4
4
4
±0.046 × 10 ±0.015 × 10 ±0.058 × 10
±0.048 × 10 ±0.017 × 103
5
4
4
2.440 × 10
4.020 × 10
6.351 × 10
2.426 × 105
4.223 × 103
±0.019 × 105 ±0.035 × 104 ±0.148 × 104
±0.019 × 105 ±0.030 × 103
1.518 × 106
1.131 × 105
7.685 × 104
1.516 × 106
1.194 × 104
6
5
4
6
±0.012 × 10 ±0.007 × 10 ±0.985 × 10
±0.012 × 10 ±0.006 × 104
Procedure 3
3.119 × 103
±0.053 × 103
3.633 × 103
±0.091 × 103
3.615 × 103
±0.086 × 103
Procedure 3
3.117 × 103
±0.050 × 103
3.583 × 103
±0.087 × 103
3.646 × 103
±0.089 × 103
KN++ procedure and Procedure 3 is the least sensitive to the choice of ε. The results in Tables 6
are consistent with the theoretical results listed in Table 1. Lastly, the similar results appear for
both the normally and exponentially distributed cases.
7.4. Ambulance Allocation in Emergency Medical Services System
An Emergency Medical Services (EMS) system is an important component of public safety that
attempts to allocate scarce resources as critical events occur. In the EMS system, one critical
resource allocation challenge is deploying several ambulances to best serve requested emergency
calls, for instance, keeping response time small. The response time for a call is the elapsed time
from when the call is received to when an ambulance arrives at the scene. Here we measure the
performance as the fraction of requested calls received that have response times of 8 minutes or
less. In this experiment, we consider an example similar to that in Ni et al. (2012).
Consider a city of size 15 miles by 15 miles. Suppose there are 4 ambulances with traveling speed
24 miles per hour, 9 ambulance bases and 2 hospitals in the city. Distances from point to point
are measured by the Manhattan metric. See Figure 4 for a map of this city. When a requested call
arrives, the EMS system will dispatch the call to the nearest free ambulance. After the dispatched
ambulance arrives at the scene, the time spent there is exponentially distributed with mean 12
minutes before carrying the patient to the nearest hospital. Once the patient reaches the hospital,
the ambulance travels back to its home base to wait for the next call. We assume the call location
is uniformly distributed in the city. Our target is allocating these 4 ambulances to possible bases
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
27
to maximize the performance, i.e., the portion of calls with response time no more than 8 minutes.
In this experiment, we assume that more than one ambulance can be allocated to the same base.
15
miles
12
9
6
3
3
6
9
12
15
miles
Figure 4
Map for the city. (∗ stands for a possible ambulance base and ◦ stands for a hospital.)
To select the optimal policy in this example, we are faced with at least three challenges. First,
there are 495 alternative policies in this problem, which is not a small number for the classic
selection procedures. Second, the observations from each alternative policy are generally distributed
and with unknown variances. Last and the most important, the ideal case is to set the IZ parameter
to zero and select the best policy in this problem because when human life is concerned no difference
is so small that we are indifferent. In other words, when the IZ parameter is set as 0.01, it implies
that 0.01 × 100% of the citizens are in a dangerous situation because it may be fatal if their calls
are not handled in time. To circumvents these challenges, we use Procedure 2 that is IZ-free. In
our experiment, we set the PCS to 0.99 and replicate the selection process 1000 times. The result
shows that the optimal policy is selected as the best with probability 1.000 and the average total
sample size required is 2.10 × 104 , seen from Table 7.
One may argue that the IZ procedures (e.g. the KN++ procedure) can also be utilized to select the
optimal policy by choosing a conservative IZ parameter. To compare the performances of Procedure
3 and the KN++ procedure, we run the following experiments when ε is set as 0.01 and 0.001
respectively, and the results are listed in Table 7. Via Monte Carlo simulation, we evaluated the
performance of Policy 1, 2, 3, 4 as 0.8189, 0.8162, 0.8151 and 0.8144 because they are all selected
as the best in our experiments, and Policy 1 is the optimal policy.
From Table 7, we obtain the following conclusions. First, both procedures select a policy within
δ of the best with a probability greater than 0.99. Second, when δ is 0.01, Procedure 3 achieves
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
28
a larger PCS, although it costs about 1.5 times more samples than the KN++ procedure. As δ
decreases to 0.001, the sample size required by the KN++ procedure increases faster than that of
Procedure 3, implying that the KN++ procedure is more sensitive to the setting of δ.
Comparisons of our procedures and the KN++ procedure under different settings of the IZ parameter.
Table 7
δ
Good Selection
0.01
Policy
Policy
Policy
Policy
1
2
3
4
0.001
Policy 1
0
Policy 1
Procedure 3
Selection Prob. Avg. SSize
1.00
0.00
1.859 × 104
0.00
±0.013 × 104
0.00
2.102 × 104
1.00
±0.022 × 104
2.097 × 104
1.00
±0.021 × 104
KN++ Procedure
Selection Prob. Avg. SSize
0.93
0.04
7.943 × 103
0.02
±0.027 × 103
0.01
1.107 × 105
1.00
±0.003 × 105
NA
NA
8. Conclusion
In this paper, we propose a new frequentist selection-of-the-best formulation which selects the best
alternative with a user-specified probability of correct selection whenever the alternatives have
unique means. Under this formulation, we design a class of sequential procedures based on the Law
of the Iterated Logarithm. Among these procedures, the design parameters are obtained by solving
numerically a one-dimensional root-finding problem or using asymptotic approximating results.
We call them Procedures 1 and 2, and they are shown to be statistically valid in a finite regime
and an asymptotic regime, respectively. The merit of Procedures 1 and 2 is that they free users
from having to specify an IZ parameter. In addition, we add a stopping criterion to turn them into
an IZ procedure (Procedure 3). The numerical results show that our procedures are less sensitive
to the setting of IZ parameter and more efficient than the KN procedure when there are a large
number of alternatives with many significantly inferior ones.
Acknowledgement
The authors would like to thank Professor Shane G. Henderson from Cornell University for his
helpful discussions. A preliminary version of this paper (Fan and Hong 2014) was published in the
Proceedings of the 2014 Winter Simulation Conference.
Appendix A: Proofs of Lemmas and Theorems
Proof of Lemma 1
Proof. For any n ∈ Z + , 1 ∈ I(n ∧ N ) is equivalent to the event where either alternative 1 is
selected as the best eventually or it is eliminated after n, i.e.,
P{1 ∈ I(n ∧ N )} = P{select alternative 1} + P{eliminate alternative 1 after n}
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
29
≥ P{select alternative 1}
≥ 1 − α.
The second inequality holds due to Equation (1). Therefore, the conclusion is drawn.
2
Proof of Lemma 2
Proof. For statement (1), it is straightforward to deduce that Tc ≤ Td and thus it suffices to prove
that Td < ∞ with probability 1 (w.p.1). Define T̃d = min{t ∈ {ti : i = 1, 2, . . .} : B∆ (t) ≥ g(t)}. One
can easily check that T̃d ≥ Td and thus only need to prove T̃d < ∞ w.p.1. Letting B(t) = B∆ (t) − ∆t,
we know that lim B(ti )/ti = 0, w.p.1 (e.g., Karatzas and Shreve (1991), 2.9.3). And, g(t) ∈ G
i→∞
implies that g(t) = o(t). Therefore, we have
B∆ (ti ) − g(ti ) B(ti )
g(ti )
=
+∆−
→ ∆,
ti
ti
ti
a.s.
which implies lim (B∆ (ti ) − g(ti )) = ∞, w.p.1, and hence T̃d < ∞, w.p.1.
i→∞
Given statement (1), statement (2) has been shown in Jennison et al. (1980).
2
Proof of Lemma 3
Proof. Denote Z(t, ω) as a sample path from the stochastic process Z(t) and let Z∆ (t, ω) =
Z(t, ω) + ∆t. Further, let T (ω) = inf {t : |Z(t, ω)| ≥ g(t)} and T∆ (ω) = inf {t : |Z∆ (t, ω)| ≥ g(t)}. To
justify this lemma, it suffices to show that,
{ω : Z∆ (T∆ (ω), ω) ≤ −g(T∆ (ω)), T∆ (ω) < ∞} ⊆ {ω : Z(T (ω), ω) ≤ −g(T (ω)), T (ω) < ∞}. (11)
In the rest of this proof, we show (11) by contradiction. For any sample path ω, when
Z∆ (T∆ (ω), ω) ≤ −g(T∆ (ω)) holds with T∆ (w) < ∞, we readily obtain that
Z(T∆ (ω), ω) = Z∆ (T∆ (ω)) − T∆ (ω)∆ ≤ −g(T∆ (ω)),
which implies that Z(·, ω) exits the continuation region no later than T∆ (ω), i.e.,
T (ω) ≤ T∆ (ω) < ∞.
(12)
Suppose that {Z(T (ω), ω) ≤ −g(T (ω)), T (ω) < ∞} is not true, i.e., Z(T (ω), w) ≥ g(T (ω)) with
T (ω) < ∞. Then we can derive
Z∆ (T (ω), ω) = Z(T (ω), ω) + T (ω)∆ ≥ g(T (ω))
(13)
which implies that Z∆ (·, ω) exits from the continuation region no later than T (ω), i.e.,
T∆ (ω) ≤ T (ω) < ∞.
(14)
From inequalities (12) and (14), we have T∆ (ω) = T (ω) < ∞. Plugging T∆ (ω) = T (ω) into inequality
(13) yields Z∆ (T∆ (ω), ω) ≥ g(T∆ (ω)), which contradicts from Z∆ (T∆ (ω), ω) ≤ −g(T∆ (ω)). Therefore, this lemma is drawn.
2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
30
Proof of Lemma 6
Proof. Denote Z(·, ω) as a sample path from the stochastic process Z(·) and Nc (ω) = min{n ∈ Z + :
|Z(n, ω)| ≥ gc (n)}. We show this lemma by contradiction. Let A = {ω : limc→∞ Nc (ω) ̸= ∞}. If the
conclusion is not true, then P{A} > 0. Notice that, for any sample path ω, Nc (ω) is non-decreasing
in c since gc (·) is increasing in c, and Nc (ω) only takes integer values. Therefore, A = ∪∞
n=1 An with
An = {ω : limc→∞ Nc (ω) = n}. Since P{A} > 0, there exists some constant integer N ∗ such that
P{AN ∗ } > 0. Further, for any ω ∈ AN ∗ , there exists a constant c∗ (ω) such that Nc (w) ≡ N ∗ for any
c ≥ c∗ (ω). Therefore, for any ω ∈ AN ∗ and c ≥ c∗ (ω),
|Z(N ∗ , ω)| = |Z(Nc (ω), ω)| ≥ gc (Nc (ω)) = gc (N ∗ ).
Letting c → ∞, we have |Z(N ∗ , ω)| = ∞ for any ω ∈ AN ∗ . Therefore, P{ω : |Z(N ∗ , ω)| = ∞} ≥
P{AN ∗ } > 0, contradicting with E[Z(N ∗ )] = N ∗ µ < ∞.
2
Proof of Theorem 2
Proof. For any pair of alternatives i and j with µi ̸= µj , we can follow the same logic (i.e. using
the Law of Iterated Logarithm) for proving part (1) of Lemma 2 and further show the elimination
between alternatives i and j occurs in finite time w.p.1. Then, noticing that the alternatives have
unique means, we conclude that all the eliminations occur in finite time w.p.1, implying that
Procedure 2 stops in finite time w.p.1.
Given the procedure stops in finite time, we define an incorrect-selection event happens when
alternative 1 is eliminated by some other alternative before termination of the procedure. Hence,
we can study the probability of each pairwise incorrect-selection (ICS) event separately and bound
the total probability of ICS using the Bonferroni inequality, i.e.,
P{ICS} ≤
k
∑
P{ICSi },
(15)
i=2
where ICSi denotes the incorrect-selection event that alternative 1 is eliminated by alternative i,
for i = 2, 3, . . . , k. To show the conclusion, we only need to show that
lim sup P{ICSi }/(α/(k − 1)) ≤ 1, for all i ̸= 1.
α→0
In particular, by Procedure 2, we have
P{ICSi } = P {Z1i (τ1i (Ni )) ≤ −gc (τ1i (Ni )), Ni < ∞} .
(16)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
31
where Ni = min{n ≥ n0 : |Z1i (τ1i (n))| ≥ gc (τ1i (n))}. For the ease of theoretical analysis, throughout
this proof, we consider the case of single-sided boundary instead of the case of double-sided boundary and their connection can be built as follows. Let Ni− = min{n ≥ n0 : Z1i (τ1i (n)) ≤ −gc (τ1i (n))},
then we get that Ni− ≥ Ni and
{
(
)
}
P{ICSi } ≤ P Z1i (τ1i (Ni− )) ≤ −gc τ1i (Ni− ) , Ni− < ∞
(
)
}
{
= P Z1i (t1i (Ni− )) ≤ −gc t1i (Ni− )Y −1 (Ni− ) Y (Ni− ), Ni− < ∞
2
2
(Yi (n) = S1i
(n)/σ1i
).
(17)
First, we would like to show that in the asymptotic regime α → 0, (17) is upper bounded by the
corresponding probability that replaces the variance estimator with the true variance, or equivalently,
lim sup
α→0
(17)
≤ lim sup
α/(k − 1)
α→0
}
{
P Z1i (t1i (Ñi− )) ≤ −gc (t1i (Ñi− )), Ñi− < ∞
α/(k − 1)
,
(18)
where Ñi− = min{n ≥ n0 : Z1i (t1i (n)) ≤ −gc (t1i (n))}. This is ensured by the strong consistency of
variance estimators and the detailed clarification is as follows. Due to Lemma 6 and the fact that
Ni− ≥ Ni , we have Ni− → ∞ as α → 0. Further, since Y (n) → 1 w.p.1, for any small number ϵ, there
exists some n0 (< Ni− ) large enough, such that
P {|Y (n) − 1| ≤ ϵ for all n ≥ n0 } ≥ 1 − ϵ.
Also notice that the function h(t, y) := gc (t/y)y is an increasing function of y, as long as c > 1.
Actually, when c > 1,
∂
1
log h(t, y) = −
∂y
2
(
t
t
+
y(t + y)(c + log(t/y + 1)) y(t + y)
)
+
1
t
1
>−
+ > 0.
y
y(t + y) y
Therefore, we obtain
}
{
(1−ϵ)
(1−ϵ)
(1−ϵ)
(17) ≤ P Z1i (t1i (Ni
)) ≤ −h(t1i (Ni
), 1 − ϵ), Ni
< ∞; |Y (n) − 1| ≤ ϵ for all n ≥ n0
+P {|Y (n) − 1| > ϵ for some n ≥ n0 }
{
}
(1−ϵ)
(1−ϵ)
(1−ϵ)
≤ P Z1i (t1i (Ni
)) ≤ −h(t1i (Ni
), 1 − ϵ), Ni
< ∞ + ϵ,
(y)
where Ni
= min{n ≥ n0 : Z1i (t1i (n)) ≤ −h(t1i (n), y)}. Now set ϵ = o(α) and let α → 0, we obtain
{
}
(1)
(1)
(1)
P
Z
(t
(N
))
≤
−
h(t
(N
),
1),
N
<
∞
1i
1i
1i
i
i
i
(17)
lim sup
≤ lim sup
α/(k
− 1)
α/(k − 1)
α→0
α→0
{
}
P Z1i (t1i (Ñi− )) ≤ −gc (t1i (Ñi− )), Ñi− < ∞
= lim sup
.
(19)
α/(k − 1)
α→0
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
32
In other words, (18) holds.
−2
Furthermore, noticing that Z1i (t1i (n)) = σ1i
∑n
j=1 (X1j
− Xij ), we define a standardized version
of Z1i (t1i (·)) with a scaling parameter M (α), that is
∑⌊M (α)s⌋
−2
σ1i
[X1j − Xij − (µ1 − µi )]
j=1
√
C(M (α), t1i (s)) =
, for s ≥ 0,
M (α)
(20)
where we let M (α) → ∞ as α → 0. Based on Equation (8), we get that C(M (α), t1i (s)) → B(t1i (s)),
as M (α) → ∞. Plugging C(M (α), ·) into (18), we get that
{
}
(18) ≤ lim sup P Z1i (t1i (Ñi0− )) − t1i (Ñi0− )(µ1 − µi ) ≤ −gc (t1i (Ñi0− )), Ñi0− < ∞ /(α/(k − 1))
α→0
)
(
 (

(
))
0−


0−
)
g
t
(
Ñ
c
1i
i
Ñi
= lim sup P C M (α), t1i
≤− √
, Ñi0− < ∞ /(α/(k − 1))


M (α)
α→0
M (α)
}
{
(
) √
, lim sup P C(M (α), Td− ) ≤ −gc M (α)Td− / M (α), Td− < ∞ /(α/(k − 1))
α→0
= lim sup P{Td− < ∞}/(α/(k − 1)),
(21)
α→0
where Ñi0− = min{n ≥ n0 : Z1i (t1i (n)) − t1i (n)(µ1 − µi ) ≤ −gc (t1i (n)) } and Td− = t1i (Ñi0− /M (α)).
The inequality above follows from Lemma 3.
Next, we would like to show that (21) is upper bounded by the probability that considers a
continuous-time Brownian motion instead of C(M (α), ·) when
M (α) → ∞, c3 /M (α) → 0, as α → 0.
(22)
In the rest of the proof, for the simplicity of notation, we write M instead of M (α).
To show this result, we start with studying the asymptotic cumulative distribution function
P{Td− ≤ t} for any fixed t > 0. Because X11 − Xi1 has a finite moment generating function in the
neighborhood of zero, then by Theorem 2.2.4 in Whitt (2002b), there exist positive constants C1 ,
C2 , λ and a BM B̃(·) depending on the distribution of Xi1 and X11 , such that for every M ≥ 1,
{
}
(
)
C2 exp(−c)
−1
2
P max n[X̄1 (n) − X̄i (n) − (µ1 − µi )]/σ1i − B̃(n) ≥ C1 log s + λ
c − log M σ1i ≤
.
2
1≤n≤s
M σ1i
(23)
√
2
2
t)/ M σ1i
Define B(·) = B̃(M σ1i
. It is easy to see B(·) is still a BM by the self-similarity scaling
√
2
2
2
)]/ M σ1i
u + λ−1 (c − log M σ1i
property of BM. Let ψM (u) = [C1 log M σ1i
, then for any fixed t > 0,
P{Td− ≤ t}
=P{Td− ≤ t, |C(M, q) − B(q)| ≥ ψM (q), ∃ q = t1i (n/M ) ≤ t, n ∈ Z + }
+ P{Td− ≤ t, |C(M, q) − B(q)| ≤ ψM (q), ∀ q = t1i (n/M ) ≤ t, n ∈ Z + }
(24)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
33
We bound the two probability terms in (24) separately. For the first term in (24), by the definition
of C(M, ·) in (20), we have
the first term in (24)
≤ P{|C(M, q) − B(q)| ≥ ψM (q), ∃ q = t1i (n/M ) ≤ t, n ∈ Z + }
2
⌊M σ1i
t⌋
≤
∑
P {|C(M, t1i (s/M )) − B(t1i (s/M ))| ≥ ψM (t1i (s/M ))}
s=1
{
}
n[X̄1 (n) − X̄i (n) − (µ1 − µi )]/σ1i
B̃(n)
√
√ ≥ ψM (t1i (s/M ))
P max ≤
−
1≤n≤s
σ
M
σ
1i
1i M
s=1
≤ C2 t exp(−c) by (23).
2
⌊M σ1i
t⌋
∑
(25)
To bound the second term in (24), we define a new single lower boundary −L(u), where L(u) =
√
gc (M u)/ M − ψM (u). Then, {Td− ≤ t, |C(M, q) − B(q)| ≤ ψM (q), ∀ q = t1i (n/M ) ≤ t, n ∈ Z + }
implies that B(·) crosses the single lower boundary −L(u) before t, that is
−
the second term of (24) ≤ P(Td,L
≤ t) ≤ P(TL− ≤ t),
(26)
−
where Td,L
= min{q = t1i (n/M ) : B(q) ≤ −L(q), n ∈ Z + } and TL− = inf {u : B(u) ≤ −L(u)} denote a
discrete and continuous stopping time for the one-sided lower boundary, respectively. The second
−
inequality arises from the fact that Td,L
≥ TL− . Actually, we can always find a large enough M that
satisfies the conditions in (22) to ensure a well-defined boundary L(u), i.e., L(u) > 0 for 0 < u ≤ t.
√
To see it, when M satisfies the conditions in (22), we have ψM (u) → 0 and gc (M u)/ M → ∞ as
α → 0, for any fixed u.
Combining (25) and (26), we obtain an upper bound for P{Td− ≤ t}, that is
P{Td− ≤ t} ≤ P(TL− ≤ t) + C2 t exp(−c).
(27)
Lastly, we calculate the asymptotic expression for P(TL− ≤ t) based on Theorem 4 in Jennen and
Lerche (1981). This theorem provides the asymptotic probability density of TL− , that is
(
)
′
L2 (u)
α→0 L(u) − uL (u)
√
fT − (u) ∼
exp −
, uniformly in (0, t) when α is sufficiently close to 0.
L
2u
2πu3/2
√
Let g̃c,M (u) = gc (M u)/ M and Tg̃−c,M = inf {t : B(t) ≤ −g̃c,M (t)}. Then we derive that
]
[
(
)
′
2
fT − (u) α→0
(u)
2g̃c,M (u)ψM (u) − ψM
ψM (u) − uψM (u)
L
exp
∼ 1+
.
′
fT − (u)
2u
g̃c,M (u) − ug̃c,M (u)
g̃c,M
Since c3 /M → 0 as α → 0, we get that g̃c,M (u)ψM (u) = o(1) as α → 0. Besides, by the definition
′
(u) → 0 as M → ∞ for any
of g̃c,M (u) and ψM (u), we know that g̃c,M (u) → ∞, ψM (u) → 0 and ψM
fixed u. Therefore,
α→0
fT − (u) ∼ fT −
L
g̃c,M
(u), uniformly on u ∈ (0, t].
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
34
from which we have
P{TL−
∫
t
α→0
< t} ∼
fT −
(u)du
√
√
( 2
)
∫ Mt
gc (M u) / M − ugc′ (M u) M
gc (M u)
α→0
√
∼
exp −
du
2M u
2πu3/2
0
( 2
)
∫ Mt
gc (v) − vgc′ (v)
gc (v)
√
=
exp −
dv
3/2
2v
2πv
0
[
(√
)]
α→0
∼ exp(−c/2) 1 − Φ
(c + log(M t + 1))/(M t) ,
0
g̃c,M
(28)
where Φ(·) denotes the cumulative distribution function of a standard normal random variable.
The second asymptotics arises from Equation (19) in Jennen and Lerche (1981). Plugging (28) into
(27) leads to
[
(√
)]
lim sup P{Td− ≤ t}/(α/(k − 1)) ≤ lim sup 2 1 − Φ
[c + log(M t + 1)]/(M t) .
α→0
(29)
α→0
Further, letting t → ∞ in (29) and combining it with (17), (18), (21), we have (16) holds and then
2
our conclusion is drawn.
Proof of Theorem 4
Proof. The proof contains two parts. Part one, we justify the mean path approximation, i.e.,
α→0
E[Ni ] ∼ λi , where λi denotes the intersection point between the boundary gc (t1i (λ)) and (µ1 −
µi )t1i (λ). Part two, we show λi can be further asymptotically rewritten as an explicit form, as
shown in the theorem.
To prove the first part, it suffices to prove the following two statements: (1) lim sup E[Ni ]/λi ≤ 1;
α→0
(2) lim inf E[Ni ]/λi ≥ 1, for all i = 2, 3, . . . , k. The proof can be summarized as follows.
α→0
To prove statement (1), we construct an upper linear support for the current boundary and show
the average sample size is λi asymptotically under this linear boundary. Then we claim that our
procedure has a smaller asymptotic average sample size than the one using the linear boundary.
As for statement (2), we show that it holds for a subset of the event space, thus it holds for the
whole event space.
We start with statement (1). Since gc (t) is concave, gc′ (t1i (λi )) t + t1i (λi )(µ1 − µi − gc′ (t1i (λi ))
builds an upper tangent line to gc (t) at the point (t1i (λi ), (µ1 − µi )t1i (λi )), i.e., gc (t) ≤ gc′ (t1i (λi ))t +
t1i (λi )(µ1 − µi − gc′ (t1i (λi ))) for any t. Let Ñi denote the first exit time of Z1i (t1i (·)) from the tangent
line, i.e.,
Ñi = min{n : Z1i (t1i (n)) ≥ gc′ (t1i (λi ))t1i (n) + t1i (λi )(µ1 − µi − gc′ (t1i (λi )))}.
It is easy to derive that
Ni = min{n : |Z1i (t1i (n))| ≥ gc (t1i (n))} ≤ Ñi .
(30)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
35
One can check that the limit of gc′ (t1i (λi )) exists by noticing c, λi → ∞ as α → 0, and then due to
Lemma 2 in Siegmund (1967), we get that lim sup E[Ñi ]/λi ≤ 1. Further, by (30), we see that
α→0
lim sup E[Ni ]/λi ≤ lim sup E[Ñi ]/λi ≤ 1.
α→0
(31)
α→0
Now we move to justify the statement (2). Let (Ω, G , P) be the underlying probability space and
let Ω0 = {ω : alternative 1 eliminates alternative i}. By the strong law of large numbers, we have
X̄1 (n) − X̄i (n) converges to µ1 − µi almost surely on Ω0 . By Egoroff’s theorem (Egoroff 1911), for
every ϵ ∈ (0, µ1 − µi ) there exist disjoint subsets A and B, such that A ∪ B = Ω0 , P(B) < ϵ and
X̄1 (n) − X̄i (n) converges uniformly on A. Hence, there exists a positive integer n(ϵ), s.t. for every
ω ∈ A and n > n(ϵ), we have
0 < µ1 − µi − ϵ ≤ X̄1 (n) − X̄i (n) ≤ µ1 − µi + ϵ.
Define A∗ = {ω : ω ∈ A and Ni ≤ n(ϵ)}. Then, on the set A − A∗ , we have
0 < µ1 − µi − ϵ ≤ X̄1 (Ni ) − X̄i (Ni ) ≤ µ1 − µi + ϵ.
and hence
gc (t1i (Ni )) ≤ |t1i (Ni )[X̄1 (Ni ) − X̄i (Ni )]| = t1i (Ni )[X̄1 (Ni ) − X̄i (Ni )] ≤ t1i (Ni )(µ1 − µi + ϵ).
This implies that Ni ≥ λϵi on A − A∗ , where λϵi is such that gc (t1i (λϵi )) = t1i (λϵi )(µ1 − µi + ϵ).
Therefore, we can derive that
lim inf E[Ni ]/λi ≥ lim inf E[Ni I(w ∈ A − A∗ )]/λi ≥ lim P(A − A∗ )λϵi /λi .
α→0
α→0
α→0
According to Lemma 6, we know that Ni → ∞ as α → 0 w.p.1. Hence, we have lim P(A∗ ) = 0, and
α→0
lim P(A − A∗ ) = lim [P(A) − P(A∗ )] = P(A) = 1 − P(B) ≥ 1 − ϵ.
α→0
α→0
(32)
In other words, lim inf E[Ni ]/λi ≥ (1 − ϵ)λϵi /λi . Letting ϵ → 0, we have the second statement is true,
α→0
i.e., lim inf α→0 E[Ni ]/λi ≥ 1. Combining with Inequality (31), we get
α→0
E[Ni ] ∼ λi .
(33)
Now we move to the second part. Since λi satisfies the equation (µ1 − µi )t1i (λ) =
√
[c + log(t1i (λ) + 1)](t1i (λ) + 1), we can derive c = (µ1 − µi )2 t21i (λi )/(λi +1) − log(t1i (λi )+1). Then,
c
t1i (λi )
log(t1i (λi ) + 1)
=
−
.
2
t1i (λi )(µ1 − µi )
t1i (λi ) + 1 t1i (λi )(µ1 − µi )2
(34)
α→0
As α → 0, we have t1i (λ) → ∞. Consequently, (34) ∼ 1 and in other words,
2
c
cσ1i
α→0
=
∼ 1.
2
2
t1i (λi )(µ1 − µi )
λi (µ1 − µi )
Combining (33) and (35) results in the conclusion.
(35)
2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
36
Proof of Theorem 5
Proof. If µ1 − µk ≤ ε, then the conclusion is trivially valid. Otherwise, there exists some integer
v such that µ1 − µv ≤ ε and µ1 − µv+1 > ε. To avoid the confusion, we define alternative i kills
alternative j(̸= i) if alternative i eliminates alternative j before T ∗ (ε) or alternative i has a larger
sample mean at T ∗ (ε), and define alternative i eliminates alternative j if the partial-sum process
of their mean difference exits the designed continuation region from above. (The definition of
“elimination” is consistent with the rest of this paper.) Then
P{µ1 − µi∗ ≤ ε}
≥ P {alt. v + 1, v + 2, . . . , k are killed}
{ v
}
k
∩
∩
≥P
{alt. 1 is not eliminated by alt. j before T ∗ (ε)}
{alt. 1 kills alt. i}
j=2
≥ 1−
v
∑
i=v+1
P {alt. 1 is eliminated by alt. j before T ∗ (ε)} −
j=2
k
∑
P {alt. i kills alt. 1} .
(36)
i=v+1
The last inequality arises from Bonferroni inequality. Throughout this proof, we define the
following notations. For all l ̸= 1, define two stopping times Nl and Nl0 as Nl = min{n :
0
0
Z1l (τ1l (n)) ≥ gc (τ1l (n))} and Nl0 = min{n : Z1l
(τ1l (n)) ≥ gc (τ1l (n))}, where Z1l
(τ1l (n)) =
Z1l (τ1l (n)) − τ1l (n)(µ1 − µl ). Then, for j = 2, 3, . . . , v, we have
P{alt. 1 is eliminated by alt. j before T ∗ (ε)}
= P{Z1j (τ1j (Nj )) ≤ −gc (τ1j (Nj )), τ1j (Nj ) ≤ T ∗ (ε) < ∞}
≤ P{Z1j (τ1j (Nj )) ≤ −gc (τ1j (Nj )), τ1j (Nj ) < ∞}
0
≤ P{Z1j
(τ1j (Nj0 )) ≤ −gc (τ1j (Nj0 )), τ1j (Nj0 ) < ∞}
α→0
∼ α/(k − 1).
(37)
The first inequality holds because the condition {τ1j (Nj ) ≤ T ∗ (ε)} is omitted and the second
inequality holds due to Lemma 3. The asymptotic equivalence is seen by the proof of Theorem 2.
For i = v + 1, . . . , k, alternative i kills alternative 1 if alternative 1 is eliminated by alternative i
before T ∗ (ε) or alternative 1 has a smaller sample mean at T ∗ (ε). In other words,
P{alt. i kills alt. 1}
=P{Z1i (τ1i (Ni )) ≤ −gc (τ1i (Ni )), τ1i (Ni ) ≤ T ∗ (ε) < ∞} + P{Z1i (T ∗ (ε)) < 0, T ∗ (ε) < τ1i (Ni )}. (38)
Now we study these two terms in (38) separately. For the first term, we have,
{ 0
}
the first term in (38) ≤ P Z1i
(τ1i (Ni0 )) ≤ −gc (τ1i (Ni0 )), τ1i (Ni0 ) < ∞, τ1i (Ni ) ≤ T ∗ (ε) < ∞ , (39)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
37
which follows from Lemma 3. As for the second term in (38), we have that,
the second term in (38)
{ 0 ∗
}
= P Z1i
(T (ε)) ≤ −T ∗ (ε)(µ1 − µi ), T ∗ (ε) < τ1i (Ni )
{ 0 ∗
}
≤ P Z1i
(T (ε)) ≤ −ε T ∗ (ε), T ∗ (ε) < τ1i (Ni )
(since µ1 − µi > ε)
}
{ 0 ∗
(by the definition of T ∗ (ε)).
= P Z1i (T (ε)) ≤ −gc (T ∗ (ε)), T ∗ (ε) < τ1i (Ni )
(40)
0
In the probability statement above, the first statement implies that Z1i
(·) first exits the contin-
uation region no later than T ∗ (ε) (i.e., τ1i (Ni0 ) ≤ T ∗ (ε) < ∞) and the second statement further
0
0
claims that Z1i
(·) first exits the region from below. Otherwise, if Z1i
(τ1i (Ni0 )) ≥ gc (τ1i (Ni0 )), then
0
Z1i (τ1i (Ni0 )) = Z1i
(τ1i (Ni0 )) + τ1i (Ni0 )(µ1 − µi ) ≥ gc (τ1i (Ni0 )), implying that τ1i (Ni ) ≤ τ1i (Ni0 ) ≤
T ∗ (ε). This contradicts with T ∗ (ε) < τ1i (Ni ). Therefore, we validate this claim and then get that
the second term in (38) is upper bounded by
{ 0
}
P Z1i
(τ1i (Ni0 )) ≤ −gc (τ1i (Ni0 )), τ1i (Ni0 ) < ∞, T ∗ (ε) < τ1i (Ni ) .
Plugging (39) and (41) into (38), we obtain that
{ 0
} α→0
P{alt. i eliminates alt. 1} ≤ P Z1i
(τ1i (Ni0 )) ≤ −gc (τ1i (Ni0 )), τ1i (Ni0 ) < ∞ ∼
(41)
α
.
k−1
(42)
The asymptotic equivalence can be derived based on the proof of Theorem 2. Combining (36) with
2
(37) and (42), the conclusion is drawn.
Proof of Proposition 2
Proof. For any sample path, part (1) of Lemma 2 implies that Procedure 2 does not terminate in
finite time only if all the remaining alternatives have the unique mean. Because the best alternative
is unique, it suffices to state that the procedure does not terminate in finite time only if alternative
1 is eliminated. Further, we have
{
P{eliminate alternative 1} = P
k
∪
}
{alernative 1 is elimianted by alternative i}
i=2
≤
k
∑
P{alernative 1 is elimianted by alternative i}.
(43)
i=2
The inequality above arises from the Bonferroni inequality. Notice that (43) is the same as (15)
which is shown to be at most α asymptotically in the proof of Theorem 2. Therefore, we have
lim sup P{do not terminate in finite time}/α ≤ lim sup P{eliminate alternative 1}/α ≤ 1,
α→0
α→0
and
lim sup P{terminate and eliminate alterantive 1}/α ≤ 1.
α→0
Therefore, the proposition holds.
2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
38
Appendix B: Simulation Based Algorithm for Finding c in Equation (3)
In this appendix, we design a simulation-based algorithm to find c(> 0) by solving Equation (3),
i.e.,
}]
)
[ {
(
Y
n0 − 1
α
T
, T < ∞Y
,
β , E P B(T ) ≤ −gc
=
Y
n0 − 1
k−1
{
}
(
)
where gc (t) ∈ G , T = inf t : |B(t)| ≥ gc n0Y−1 t n0Y−1 , B(·) denotes a Brownian motion with unit
variance and no drift, and Y is a chi-squared random variable with n0 − 1 degrees of freedom
that is independent of B(·). Solving (3) can be treated as a stochastic root-finding problem. To
make the problem well defined, it is worthwhile mentioning that Equation (3) is solvable. This
is guaranteed because the boundary function gc (t) ∈ G satisfies a certain lower bound condition
√
(i.e., it approaches to the infinity at a rate no slower than O( t log log t )). If this condition is not
satisfied, by the Law of the Iterated Logarithm, the left-hand side of Equation (3) is always 1/2
and thus (3) may have no solution.
(
{
)
}
To solve Equation (3), the key issue is to evaluate P B(T ) ≤ −gc n0y−1 T n0y−1 , T < ∞ for any
c > 0 when Y is observed as y. Once the probability has been evaluated, Equation (3) can be
addressed using linear interpolation, bisection method, Newton’s method and so on. Because gc (·)
can be a general boundary from G , it is difficult to obtain an explicit expression for this probability
and we have to estimate it via Monte Carlo simulation.
To estimate the probability, we generate a sample path from B(·) under the probability measure
P and check whether this sample path exits the boundary in finite time. However, this may be
infeasible because P{T = ∞} > 0. When {T = ∞} occurs for some sample path, the simulation
process fails to terminate in finite time. We overcome this difficulty by using a truncation method, in
(
{
)
}
which we estimate P B(T ) ≤ −gc n0y−1 T n0y−1 , T ≤ N (y) . The truncating point N (y) is chosen
to bound the truncation error below a pre-determined level η ∈ (0, 1), i.e.,
{
(
)
}
n0 − 1
y
T
, N (y) < T < ∞ ≤ η.
P B(T ) ≤ −gc
y
n0 − 1
(44)
Let f (t) denote the probability density function of the first exit time T and define hc (t; y) =
(
)
gc n0y−1 t n0y−1 . Since gc (t) is a concave function in t, it follows by Equation (9) in Jennen (1985)
that
( 2
)
hc (t; y) − th′c (t; y)
hc (t; y)
√
f (t) ≤ 2
exp −
, for any t > 0.
2t
2πt3/2
This implies that a proper N (y) can be determined by letting
( 2
)
∫
hc (t; y) − th′c (t; y)
h (t; y)
1 ∞
√
2
exp − c
dt ≤ η.
2 N (y)
2t
2πt3/2
(45)
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
Define ξ = (n0 − 1)/y, then the left-hand-side expression in (45) is simplified as
)
( 2
∫ ∞
hc (t; y)
hc (t; y) − th′c (t; y)
√
dt
exp −
2t
2πt3/2
N (y)
( 2
)
∫ ∞
gc (ξt) − (ξt)gc′ (ξt)
g (ξt)
√
=
exp − c 2
dt
2ξ t
2πt3/2 ξ
N (y)
( 2 )
∫ ∞
gc (t) − tgc′ (t)
g (t)
√
=
exp − c
dt
√
2ξt
2πt3/2 ξ
ξN (y)
( 2 ) (
) ∫ ∞
( 2 )
∫ ∞
1
gc (t)
gc (t)
gc (t)
g (t)
√
√
=−
exp −
d √
+
exp − c
dt
3/2
2ξt
2ξt
2πξ
t
ξN (y)
ξN (y) 2 2πξ t
)
(
( 2 )
∫ ∞
g (t)
g (t)
gc (ξN (y))
√
√ c 3/2 exp − c
−1+
=Φ
dt,
2ξt
2
2πξ
t
ξ N (y)
ξN (y)
39
(46)
where Φ(·) denotes the cumulative distribution function of a standard normal distribution. Notice
√
that we can always make N (y) large enough so that gc (t)/ t is monotone over [ξN (y), ∞) and
hence the changing of variables in (46) is valid.
Further, the integration term in (46) can be rewritten as
(
)
∫ ∞ √
[c + log(t + 1)](t + 1)
[c + log(t + 1)](t + 1)
√
exp −
dt
2ξt
2 2πξ t3/2
ξN (y)
)
(
∫ ∞ √
[c + log(t + 1)](t + 1)
c + log(t + 1)
√
≤
dt
exp
−
2ξ
2 2πξ t3/2
ξN (y)
(
)3/2 ∫ ∞ √
)
(
[c + log(t + 1)](t + 1)
1
c + log(t + 1)
√
≤ 1+
dt
exp −
ξN (y)
2ξ
2 2πξ (t + 1)3/2
ξN (y)
(
)3/2 ∫ ∞ √
(
) (
)
c + log(t + 1)
1
c + log(t + 1)
c + log(t + 1)
√
= 1+
exp −
d
ξN (y)
2ξ
2ξ
2πξ −1
ξN (y)
)3/2 ∫ ∞
(
ξ
1
=√
x1/2 e−x dx
1+
ξN (y)
π
[c+log(ξN (y)+1)]/(2ξ)
(
)3/2
ξ
1
=
1+
Γ ([c + log(ξN (y) + 1)]/(2ξ); 3/2) ,
2
ξN (y)
(47)
where Γ( · ; 3/2) denotes the upper incomplete gamma function with index 3/2. Based on Equations
(46) and (47), we have the truncation error
P{B(T ) ≤ −gc (T ), N (y) < T < ∞}
(
)
(
)3/2
gc (ξN (y))
ξ
1
√
≤Φ
−1+
Γ ([c + log(ξN (y) + 1)]/(2ξ); 3/2) .
1+
2
ξN (y)
ξ N (y)
(48)
Since the right-hand side in (48) decreases to 0 as the N (y) increases to the infinity, we can always
find a N (y) to meet inequality (44) for any η ∈ (0, 1) and y > 0 by solving (48) = η. It follows that
}]
[ {
(
)
n0 − 1
Y
E P B(T ) ≤ −gc
T
, N (y) < T < ∞Y
≤η
(49)
Y
n0 − 1
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
40
This implies a biased estimator
{
(
}
)
n
n0 − 1
1∑
yi
I B(Ti ) ≤ −gc
, Ti ≤ N (yi )
p̂ =
Ti
n i=1
yi
n0 − 1
(50)
{
(
)
}
with the bias less than η for the probability P B(T ) ≤ −gc n0y−1 T n0y−1 , T < ∞ , where y1 , y2 , . . .
are i.i.d. observations from Y .
Further, we implement an importance-sampling (IS) technique, which leads to a new estimator
with a smaller variance and thus a higher efficiency of estimation. Let B(t) denote a standard
Brownian motion process under P and define a new probability measure Pµ by
Lµ (t) =
dPµ
= exp(µB(t) − tµ2 /2), where µ ̸= 0.
dP
By Girsanov theorem, B̃(·) = B(t) − µt is a Brownian motion process without drift µ and unit
variance under the probability measure Pµ . Therefore, we get a new IS estimator
{
(
)
}
n
1∑
n0 − 1
yi
p̂IS =
I B̃µ(yi ) (Ti ) ≤ −gc
Ti
, Ti ≤ N (yi ) L−1
µ(yi ) (Ti )
n i=1
yi
n0 − 1
where y1 , y2 , . . . are i.i.d. observations from Y .
Next, we would like to choose the drift µ(·) so that the new estimator has a lower variance.
{
}
Define I , I B̃µ(Y ) (T ) ≤ −gc (ξ(Y )T ) /ξ(Y ), T ≤ N (Y ) L−1
µ(Y ) (T ) where ξ(y) = (n0 − 1)/y. Then
Eµ(y) [I 2 |Y = y]
[ {
}
]
(T
)
=Eµ(y) I B̃µ(y) (T ) ≤ −gc (ξ(y)T ) /ξ(y), T ≤ N (y) L−2
µ(y)
}
]
[ {
(T
)
|
Y
=Eµ(Y ) I B̃µ(Y ) (T ) ≤ −gc (ξ(Y )T ) /ξ(Y ), T ≤ N (Y ) L−2
µ(Y )
[ {
}
]
(T
)
=Eµ(y) I B̃µ(y) (T ) ≤ −gc (ξ(y)T ) /ξ(y), T ≤ N (y) L−2
µ(y)
[ {
}
(
)]
gc (ξ(y)T )
gc (ξ(y)T )
2
=Eµ(y) I B̃µ(y) (T ) ≤ −
, T ≤ N (y) exp −2µ(y)
+ T µ (y)
ξ(y)
ξ(y)
[ {
}
(
)]
gc (ξ(y)T )
gc (ξ(y)N (y))
2
≤Eµ(Y ) I B̃µ(Y ) (T ) ≤ −
, T ≤ N (y) exp −2µ(y)
+ N (y)µ (y) . (51)
ξ(y)
ξ(y)
The last inequality holds because gc (t)/t is decreasing. In the right-hand side of (51), we choose
µ∗ (y) = −gc (ξ(y)N (y))/(ξ(y)N (y)) that minimizes L−2
µ(y) (T ). It follows that
[ {
}
( 2
)]
gc (ξ(y)T )
g (ξ(y)N (y))
Eµ(y) [I |Y = y] ≤ Eµ(y) I B̃µ(y) (T ) ≤ −
, T ≤ N (y) exp − c 2
ξ(y)
ξ (y)N (y)
[ {
}]
gc (ξ(y)T )
≤ E I B̃(T ) ≤ −
, T ≤ N (y)
ξ(y)
= E[p̂2 |Y = y],
(52)
2
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
41
where the second inequality holds because of Lemma 3. Now we show that the IS estimator p̂IS has
a smaller variance the plain estimator p̂. Notice that
Var[p̂IS ] = Var[Eµ(Y ) [p̂IS |Y ]] + E[Var[p̂IS |Y ]]
= Var[Eµ(Y ) [p̂IS |Y ]] + E[Eµ(Y ) [p̂2IS |Y ]] − E[E2 [p̂IS |Y ]]
≤ Var[E[p̂|Y ]] + E[E[p̂2 |Y ]] − E[E2 [p̂|Y ]]
= Var[p̂].
This implies that the IS estimator has a smaller variance.
Using the method described, we estimate the left-hand side probability of Equation (3) for
various values of n0 and c and report them in Table 8. For any fixed α/(k − 1), users can find its
corresponding c if α/(k − 1) exists in Table 8. Otherwise, users can start with finding two constants
in the table such that α/(k − 1) lies between them and then approximate c using the interpolation
method.
References
Basu, D. 1955 On statistics independent of a complete sufficient statistic. Sankhyā: The Indian Journal of
Statistics 377–380.
Batur, D., S. H. Kim. 2006. Fully sequential selection procedures with a parabolic boundary. IIE Transactions
38(9) 749–764.
Bechhofer, R. E. 1954. A single-sample multiple decision procedure for ranking means of normal populations
with known variances. The Annals of Mathematical Statistics 16–39.
Bechhofer, R. E., T. J. Santner, D. M. Goldsman. 1995. Design and analysis of experiments for statistical
selection, screening, and multiple comparisons. New York: John Wiley & Sons.
Branke, J., S. E. Chick, C. Schmidt. 2007. Selecting a selection procedure. Management Science 53(12)
1916–1932.
Bubeck, S., N. Cesa-Bianchi. 2012. Regret analysis of stochastic and nonstochastic multi-armed bandit
problems. Foundations and Trends in Machine Learning 5(1) 1–122.
Chick, S. E., K. Inoue. 2001. New two-stage and sequential procedures for selecting the best simulated
system. Operations Research 49(5) 732–743.
Chick, S. E., P. I. Frazier. 2012. Sequential sampling with economics of selection procedures. Management
Science 58(3) 550–569.
Clark, G. M., W. Yang. 1986. A bonferroni selection procedure when using commom random numbers with
unknown variances. Proceedings of the 1986 Winter Simulation Conference (WSC) 313–315.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
42
Dudewicz, E. J. 1969. An approximation to the sample size in selection problems. Annals of Mathematical
Statistics 40(2) 492–497.
Durbin, J. 1985. The first-passage density of a continuous gaussian process to a general boundary. Journal
of Applied Probability 99–122.
Durrett, R. 2010. Probability: Theory and Examples, vol. 3. Cambridge university press.
Egoroff, D. T. 1911. Sur les suites de fonctions mesurables. CR Acad. Sci. Paris 152 244–246.
Fan, W., L. J. Hong. 2014. A frequentist selection-of-the-best procedure without indifference zone. Proceedings of the 2014 Winter Simulation Conference (WSC) 3737–3748.
Frazier, P. I. 2014. A fully sequential elimination procedure for indifference-zone ranking and selection with
tight bounds on probability of correct selection. Operations Research 62(4) 926–942.
Gupta, S. S. 1956. On a decision rule for a problem in ranking means. Ph.D. thesis, University of North
Carolina. Chapel Hill.
Gupta, S. S. 1965. On some multiple decision (selection and ranking) rules. Technometrics 7(2) 225–245.
Hong, L. J. 2006. Fully sequential indifference-zone selection procedures with variance-dependent sampling.
Naval Research Logistics 53(5) 464–476.
Hong, L. J., B. L. Nelson. 2005. The tradeoff between sampling and switching: New sequential procedures
for indifference-zone selection. IIE Transactions 37(7) 623–634.
Hong, L. J., B. L. Nelson. 2007. Selecting the best system when systems are revealed sequentially. IIE
Transactions 39(7) 723–734.
Jamieson, K., R. Nowak. 2014. Best-arm identification algorithms for multi-armed bandits in the fixed
confidence setting. The 48th Annual Conference on Information Sciences and Systems (CISS) IEEE,
1–6.
Jennen, C., H. R. Lerche. 1981. First exit densities of brownian motion through one-sided moving boundaries.
Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 55(2) 133–148.
Jennen, C. 1985. Second-order approximations to the density, mean and variance of Brownian first-exit
times. The Annals of Probability 13(1) 126–144.
Jennison, C., I. M. Johnston, B. W. Turnbull. 1980. Asymptotically optimal procedures for sequential
adaptive selection of the best of several normal means. Technical Report, Dept. of ORIE, Cornell Univ.,
Ithaca, New York .
Karatzas, I. A., S. E. Shreve. 1991. Brownian Motion and Stochastic Calculus, vol. 113. Springer, New York.
Karnin, Z., K. Tomer, S. Oren, 2013. Almost optimal exploration in multi-armed bandits. Proceedings of
the 30th International Conference on Machine Learning (ICML-13) 1238–1246.
Kim, S. H., B. L. Nelson. 2001. A fully sequential procedure for indifference-zone selection in simulation.
ACM Transactions on Modeling and Computer Simulation (TOMACS) 11(3) 251–273.
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
43
Kim, S. H., B. L. Nelson. 2006a. Selecting the best system. Handbooks in Operations Research and Management Science: Simulation 13 501–534. Elsevier.
Kim, S. H., B. L. Nelson. 2006b. On the asymptotic validity of fully sequential selection procedures for
steady-state simulation. Operations Research 54(3) 475–488.
Ni, E. C., S. R. Hunter, S. G. Henderson, H. Topaloglu. 2012. Exploring bounds on ambulance deployment
policy performance. Proceedings of the 2012 Winter Simulation Conference (WSC) 1–12.
Ni, E. C., S. G. Henderson, S. R. Hunter. 2014. A comparison of two parallel ranking and selection procedures.
Proceedings of the 2014 Winter Simulation Conference (WSC) 3761–3772.
Panchapakesan, S., S. S. Gupta, J. Yackel. 1971. On a subset selection procedure for the most probable
event in a multinomial distribution. Statistical Decision Theory and Related Topics 275–298.
Paulson, E. 1964. A sequential procedure for selecting the population with the largest mean from k normal
populations. The Annals of Mathematical Statistics 35(1) 174–180.
Perng, S. K. 1969. A comparison of the asymptotic expected sample sizes of two sequential procedures for
ranking problem. The Annals of Mathematical Statistics 40(6) 2198–2202.
Pichitlamken, J., B. L. Nelson, L. J. Hong. 2006. A sequential procedure for neighborhood selection-of-thebest in optimization via simulation. European Journal of Operational Research 173(1) 283–298.
Rinott, Y. 1978. On two-stage selection procedures and related probability-inequalities. Communications in
Statistics-Theory and methods 7(8) 799–811.
Siegmund, D. 1967. Some one-sided stopping rules. The Annals of Mathematical Statistics 38(6) 1641–1646.
Sullivan, D. W., J. R. Wilson. 1989. Restricted subset selection procedures for simulation. Operations
Research 37(1) 52–71.
Wang, H., S. H. Kim. 2013. Reducing the conservativeness of fully sequential indifference-zone procedures.
IEEE Transactions on Automatic Control 58(6) 1613–1619.
Whitt, W. 2002a. Stochastic-process limits: an introduction to stochastic-process limits and their application
to queues. Springer, New York.
Whitt, W. 2002b. Internet supplement to stochastic-process limits (2002). URL http://www. columbia. edu/˜
ww2040/supplement.html .
Fan, Hong and Nelson: Indifference-Zone-Free Selection of the Best
Article submitted to Operations Research; manuscript no. (Please, provide the manuscript number!)
44
Table 8
Table of β for different values of c and n0 when gc (t) =
c
4.0
4.2
4.4
4.6
4.8
5.0
5.2
5.4
5.6
5.8
6.0
6.2
6.4
6.6
6.8
7.0
7.2
7.4
7.6
7.8
8.0
8.2
8.4
8.6
8.8
9.0
9.2
9.4
9.6
9.8
10.0
10.2
10.4
10.6
10.8
11.0
11.2
11.4
11.6
11.8
12.0
12.2
12.4
12.6
12.8
13.0
13.2
13.4
13.6
13.8
14.0
14.2
14.4
14.6
14.8
15.0
10
0.1042
0.0990
0.0945
0.0905
0.0862
0.0822
0.0786
0.0750
0.0718
0.0691
0.0657
0.0632
0.0607
0.0581
0.0556
0.0538
0.0519
0.0495
0.0479
0.0458
0.0443
0.0424
0.0410
0.0393
0.0382
0.0369
0.0357
0.0341
0.0330
0.0319
0.0309
0.0295
0.0289
0.0280
0.0269
0.0259
0.0252
0.0243
0.0236
0.0228
0.0221
0.0215
0.0207
0.0199
0.0197
0.0190
0.0184
0.0178
0.0173
0.0168
0.0163
0.0158
0.0153
0.0150
0.0145
0.0141
15
0.0798
0.0801
0.0750
0.0711
0.0668
0.0631
0.0596
0.0560
0.0534
0.0508
0.0485
0.0456
0.0431
0.0410
0.0392
0.0368
0.0352
0.0335
0.0315
0.0301
0.0291
0.0271
0.0261
0.0249
0.0235
0.0224
0.0217
0.0204
0.0195
0.0186
0.0180
0.0172
0.0164
0.0156
0.0150
0.0141
0.0137
0.0129
0.0125
0.0121
0.0114
0.0110
0.0105
0.0101
0.0098
0.0094
0.0089
0.0086
0.0083
0.0079
0.0077
0.0073
0.0070
0.0067
0.0065
0.0061
20
0.0744
0.0697
0.0652
0.0613
0.0572
0.0538
0.0506
0.0476
0.0442
0.0417
0.0392
0.0367
0.0348
0.0327
0.0305
0.0287
0.0271
0.0257
0.0249
0.0227
0.0215
0.0202
0.0191
0.0182
0.0174
0.0164
0.0152
0.0145
0.0137
0.0131
0.0124
0.0117
0.0113
0.0107
0.0099
0.0096
0.0090
0.0042
0.0082
0.0077
0.0073
0.0070
0.0066
0.0064
0.0059
0.0056
0.0053
0.0051
0.0048
0.0047
0.0045
0.0042
0.0041
0.0038
0.0037
0.0035
n0
25
0.0686
0.0639
0.0594
0.0552
0.0515
0.0479
0.0450
0.0419
0.0389
0.0367
0.0340
0.0320
0.0298
0.0279
0.0261
0.0246
0.0230
0.0214
0.0200
0.0188
0.0179
0.0167
0.0155
0.0145
0.0140
0.0130
0.0121
0.0115
0.0107
0.0101
0.0095
0.0091
0.0085
0.0081
0.0074
0.0071
0.0066
0.0064
0.0060
0.0056
0.0054
0.0050
0.0047
0.0043
0.0042
0.0040
0.0038
0.0036
0.0033
0.0032
0.0031
0.0029
0.0027
0.0026
0.0025
0.0023
30
0.0647
0.0594
0.0552
0.0512
0.0478
0.0442
0.0411
0.0380
0.0359
0.0330
0.0306
0.0287
0.0266
0.0248
0.0231
0.0216
0.0201
0.0189
0.0175
0.0163
0.0152
0.0142
0.0134
0.0124
0.0115
0.0110
0.0102
0.0094
0.0089
0.0085
0.0078
0.0074
0.0069
0.0066
0.0059
0.0058
0.0054
0.0049
0.0047
0.0044
0.0041
0.0039
0.0037
0.0033
0.0032
0.0030
0.0029
0.0027
0.0026
0.0024
0.0023
0.0021
0.0020
0.0020
0.0018
0.0017
√
[c + log(t + 1)](t + 1)
35
0.0613
0.0569
0.0524
0.0482
0.0449
0.0412
0.0383
0.0354
0.0328
0.0305
0.0282
0.0267
0.0245
0.0227
0.0211
0.0198
0.0182
0.0169
0.0158
0.0145
0.0135
0.0127
0.0118
0.0111
0.0101
0.0094
0.0088
0.0083
0.0077
0.0072
0.0067
0.0063
0.0058
0.0054
0.0050
0.0047
0.0044
0.0041
0.0039
0.0036
0.0035
0.0032
0.0030
0.0027
0.0026
0.0025
0.0023
0.0021
0.0020
0.0018
0.0019
0.0017
0.0016
0.0014
0.0014
0.0013
40
0.0593
0.0542
0.0503
0.0463
0.0428
0.0393
0.0366
0.0338
0.0313
0.0289
0.0264
0.0248
0.0226
0.0212
0.0196
0.0181
0.0166
0.0157
0.0145
0.0134
0.0124
0.0116
0.0107
0.0099
0.0093
0.0085
0.0079
0.0074
0.0069
0.0064
0.0058
0.0054
0.0052
0.0047
0.0045
0.0042
0.0038
0.0036
0.0033
0.0031
0.0029
0.0027
0.0025
0.0024
0.0023
0.0021
0.0019
0.0018
0.0016
0.0016
0.0015
0.0014
0.0012
0.0012
0.0011
0.0010
Note: All the estimated values reported in this table are within 0.0001 of the true values with 95% confidence level.