O RTHO M ADS: A deterministic M ADS instance
with orthogonal directions ∗
Mark A. Abramson † Charles Audet ‡ J.E. Dennis Jr.
Sébastien Le Digabel ¶
§
4th April 2008
Abstract
The purpose of this paper is to introduce a new way of choosing directions for
the Mesh Adaptive Direct Search (M ADS) class of algorithms. The advantages of
this new O RTHO M ADS instantiation of M ADS are that the polling directions are
chosen deterministically, ensuring that the results of a given run are repeatable,
and that they are orthogonal to each other, therefore the convex cones of missed
directions at each iteration are minimal in size.
The convergence results for O RTHO M ADS follow directly from those already
published for M ADS, and they hold deterministically, rather than with probability
one, as for LT M ADS, the first M ADS instance. The initial numerical results are
quite good for both smooth and nonsmooth, and constrained and unconstrained
problems considered here.
Keywords: Mesh Adaptive Direct Search algorithms (M ADS), deterministic, orthogonal directions, constrained optimization, nonlinear programming.
∗
Work of the second author was supported by F CAR grant N C 72792 and N SERC grant 239436-05.
The third author was supported by L ANL 94895-001-04 34, and the first three authors were supported by
A FOSR FA9550-07-1-0302, the Boeing Company, ExxonMobil Upstream Research Company.
†
Air Force Institute of Technology,
Department of Mathematics and Statistics, 2950 Hobson Way, Bldg 641, Wright Patterson A FB, Ohio 45433 USA,
www.afit.edu/en/ENC/Faculty/MAbramson/abramson.html, [email protected].
‡
GERAD and Département de mathématiques et de génie industriel, École Polytechnique de Montréal,
C.P. 6079, Succ. Centre-ville, Montréal, Québec H3C 3A7 Canada, www.gerad.ca/Charles.Audet,
[email protected].
§
Computational and Applied Mathematics Department, Rice University - MS 134, 6100 South Main
Street, Houston, Texas 77005-1892 USA, www.caam.rice.edu/∼dennis, [email protected].
¶
GERAD and Département de mathématiques et de génie industriel, Ecole Polytechnique de Montréal,
C.P. 6079, Succ. Centre-ville, Montréal, Québec H3C 3A7 Canada, [email protected].
1
Introduction
This paper considers optimization problems of the form
min f (x) ,
x∈Ω
where f : Ω ⊂ Rn → R ∪ {∞} is typically evaluated through a black-box computer
simulation with no available derivatives, and Ω is a set of feasible points also defined
by black-box nonlinear constraint, or even Boolean, functions. Because no exploitable
information on the nature of f or Ω exists, we consider direct search methods which only
use functions evaluations to drive their search.
Mesh Adaptive Direct Search (M ADS) is introduced in [4] as a SEARCH / POLL derivative-free class of methods with strong convergence properties. It extends the Generalized Pattern Search (G PS) method of [18]. The constraints are treated by the extreme
barrier approach, which simply rejects points outside Ω by setting their objective function value to ∞. The first instance of this class of methods is called LT M ADS.
LT M ADS behaves well in practice, but it has drawbacks that we wish to correct in
this paper. First, there is a probabilistic component to the choice of polling directions.
For each new mesh size, a random direction is chosen to a current mesh point. That
direction is completed somewhat randomly to a positive spanning set of directions from
the current iterate to other current mesh points. The resulting algorithm is shown to
have Clarke stationary point convergence with probability one. However, it has been
observed [10] that this way of choosing polling directions can lead to undesirably large
angles between some of the members of the LT M ADS polling set at a given iteration.
The purpose of this paper is to introduce a new variant of M ADS, which we call O R THO M ADS , that uses an orthogonal positive spanning set of polling directions and thus
avoids large angles between polling directions. In Figure 3, we show some experiments
in which the O RTHO M ADS directions do seem better distributed than the LT M ADS.
We show that O RTHO M ADS shares the same theoretical convergence results as LTM ADS, except that the convergence is not qualified by being of probability one. In the
tests given here, O RTHO M ADS performs generally better than LT M ADS.
O RTHO M ADS is detailed in Section 2, where we show a deterministic way to construct a polling set on the current mesh of orthogonal polling directions (the O RTHO M ADS directions). Section 2 also gives the convergence results, based on those in [4].
Finally, we present numerical results in Section 3 and some concluding remarks in Section 4.
Notation: Throughout the text, k · k denotes the `2 norm, ei ∈ Rn is the ith coordinate
vector, and e ∈ Rn is the vector whose components are all equal to 1. Bε (x) denotes the
open ball of radius ε around x.
1
2
The O RTHO M ADS algorithm
The O RTHO M ADS algorithm is described in this section. We will not give details for the
M ADS class of algorithms and its LT M ADS instantiation, since they are available in [4].
Each M ADS iteration k is separated into two steps, the SEARCH and the POLL, where
the objective function f and the test for feasibility are evaluated at finitely many trial
points. These trial points lie on the mesh Mk defined by
nD
} ⊂ Rn ,
Mk = {x + ∆m
k Dz : x ∈ Vk , z ∈ N
where Vk ⊂ Rn is the set of all evaluated points by the start of the iteration, ∆m
k ∈ R+
n×nD
is the mesh size parameter at iteration k, and D is a matrix in R
composed of nD
directions in Rn . This paper focuses on the POLL step which is characterized by the set
of trial points
Pk = {xk + ∆m
k d : d ∈ Dk } ⊂ Mk ,
where xk is the POLL center at iteration k and Dk is the set of POLL directions, which
have to form a positive spanning set and to be constructed so that POLL trial points lie
on the mesh Mk . In G PS, a related method, the directions contained in Dk are always
chosen among the columns of D. Therefore, in G PS, there is only the same finite number
of possibilities for selecting the directions in every Dk .
The differences between LT M ADS and O RTHO M ADS lie in the way to generate the
directions in Dk : With LT M ADS, Dk is randomly generated and directions are not necessarily orthogonal, possibly leading to large angles between directions and large unexplored convex cones of directions at a given step. However, the union of all normalized
LT M ADS directions over all iterations k is dense in the unit sphere with probability one.
O RTHO M ADS introduces a new way to generate the POLL directions Dk . This new
method is deterministic and generates orthogonal directions, which together with their
negatives form Dk , and such that the union of all normalized O RTHO M ADS directions
over all iterations is dense in the unit sphere. Furthermore, the components of these
directions are integer, so that POLL points lie on the mesh defined with D = [In − In ],
where In is the identity matrix in dimension n. The orthogonality of the O RTHO M ADS
directions offers a better distribution of the POLL trial points in the search space, and the
advantage of determinism is that numerical results are now easily reproducible. Because
of the random component of LT M ADS, we felt that numerical experiments had to be
performed on series of several runs to show the reader the variations in the results.
At each iteration of O RTHO M ADS, the main steps for the construction of these directions are as follows. First, the pseudo-random Halton sequence produces one vector
in [0, 1]n (Subsection 2.1). Second, this vector is scaled and rounded to an appropriate
length (Subsection 2.2). The resulting direction is called the adjusted Halton direction.
Third, the Householder transformation is then applied to the adjusted Halton direction,
producing n orthogonal and integer vectors, forming a basis for Rn (Subsection 2.3).
2
Finally, the basis is completed to a positive basis formed by 2n O RTHO M ADS POLL
directions Dk , by including in Dk the basis and its negatives (Subsection 2.4). Figure 1
summarizes these steps, and will be referred to throughout the section.
−H
t,` e2
1 1 Ht,` e2
−Ht,` e1
1
1 qt,` ut
1
1
√ (2ut − e)
8
k2ut − ek
Halton
Ht,` e1
Adjusted Halton
P OLL directions
scaling and rounding
Householder
Figure 1: Example with n = 2 and (t, `) = (6, 3). The Halton direction is ut =
(3/8, 2/9)T , the adjusted Halton direction qt,` = (−1, −2)T with αt,` = 2 and the set of
POLL directions Dk = [Ht,` − Ht,` ] with Ht,` e1 = (3, −4)T and Ht,` e2 = (−4, −3)T .
p
Every POLL direction d ∈ Dk satisfies ∆m
k kdk = 5/64 < ∆k = 1/8.
In this section we show that the O RTHO M ADS directions meet all the conditions
detailed in [2, 4], so that O RTHO M ADS is a valid M ADS instance and thus inherits all of
its convergence properties.
2.1
The Halton sequence ut
Halton [11] introduced a deterministic family of sequences that grow dense in the hypercube [0, 1]n . We consider the simplest sequence of this family, whose tth element
is
ut = (ut,p1 , ut,p2 , . . . , ut,pn )T ∈ [0, 1]n
where p1 = 2, p2 = 3, p3 = 5 and pj is the j th prime number, and ut,p is the radicalinverse function in base p. More precisely,
ut,p =
∞
X
at,r,p
r=0
3
p1+r
,
where the at,r,p ∈ Z+ are the unique coefficients of the base p expansion of t:
t =
∞
X
at,r,p pr .
r=0
Table 1 describes the first five elements of ut for n = 4 (for example, u5,3 = 1 × 3−2 +
2 × 3−1 = 79 ). Our specific sequence of ut vectors is from this point addressed as the
sequence of Halton directions.
Table 1: The sequence of Halton directions for n = 4 and t = 0, 1, . . . , 6.
t
0
1
2
3
4
5
6
t in base
2 3
5
0 0
0
1 1
1
10 2
2
11 10
3
100 11
4
101 12 10
110 20 11
ut
7
0
1
2
3
4
5
6
ut,2
0
1/2
1/4
3/4
1/8
5/8
3/8
ut,3
ut,5 ut,7
0
0
0
1/3 1/5 1/7
2/3 2/5 2/7
1/9 3/5 3/7
4/9 4/5 4/7
7/9 1/25 5/7
2/9 6/25 6/7
In order to remove the linear correlation of the last columns of ut , it is proposed
in [15] to exclude initial points of the Halton sequence. In the present work, we start the
sequence at t = n + 1.
The following properties will be used in Subsection 2.2:
|2ut,pi
2ut − e = 0 ⇔ n = t = 1
− 1| = |2ut,pj − 1| ⇔ t = 0 .
(1)
(2)
Property (2) follows from the fact that ut,pi and ut,pj can be written as reduced fractions
with denominators that are powers of different prime numbers pi and pj .
The next result shows that the union of all the directions in the sequence of Halton
is dense in [0, 1]n , i.e. any direction v ∈ [0, 1]n is an accumulation point of the sequence
{ut }∞
t=1 .
n
Proposition 2.1 The Halton sequence {ut }∞
t=1 is dense in [0, 1] .
Proof. It suffices to show that for any vector v ∈ [0, 1]n and any ε > 0, there exists an
integer t such that kut − vk < ε. A construction of such an integer t involves solving a
system of n Diophantine equations, and existence of a solution is ensured by the Chinese
Remainder Theorem [8], and by the fact that prime numbers are used in the definition of
ut . We refer the reader to [11] for a detailed proof.
4
2.2
The adjusted Halton direction qt,`
The directions in Dk used in the POLL step of M ADS cannot be arbitrarily chosen, they
must satisfy precise requirements. The Halton directions ut do not satisfy these requirements and the first steps toward generating a satisfactory set Dk are to translate, scale
and round ut .
These operations depend on another integer parameter, `, which is related to the
m
mesh size parameter ∆m
k (this relationship with ∆k is unimportant at this point and will
be detailed in Subsection 2.4). The parameter ` is used to transform the direction ut
into the adjusted Halton direction qt,` ∈ Zn , a direction whose norm is close to 2|`|/2 .
qt,`
will be constructed so that it is close to
Furthermore, the normalized direction kqt,`
k
2ut −e
. We already observed in (1) that 2ut − e = 0 is possible only if n = 1 and t = 1,
k2ut −ek
and our algorithm never uses t = 1 (we begin our Halton sequence at t = n + 1, see
Subsection 2.4).
In order to define qt,` , we first introduce the following sequence of functions:
n
1
2ut − e
1
n
qt (α) = round α
∈ Z ∩ −α − , α +
k2ut − ek
2
2
where α ∈ R+ is a scaling factor, and ut is the tth Halton direction. The function qt (·) is
a monotone non-decreasing step function on R+ . Let αt,` be a scalar such that kqt (αt,` )k
is as close as possible to 2|`|/2 , without exceeding it:
αt,` ∈ argmax kqt (α)k
α∈R+
kqt (α)k ≤ 2|`|/2 .
s.t.
(3)
Problem (3) can easily be solved using a bisection method. The adjusted Halton direction
qt,` is defined to be equal to qt (αt,` ), and the following Lemma ensures that qt,` is a
nonzero integer vector:
Lemma 2.2 If t 6= 0, the adjusted Halton direction satisfies kqt,` k ≥ 1.
Proof. From (2), if t 6= 0 and α =
k2ut −ek
,
2k2ut −ek∞
then kqt (α)k = 1 ≤ 2|`|/2 for all `.
The following lemma gives a lower bound on the value of αt,` . It will be used later
to justify that αt,` grows large with `.
2|`|/2 1
Lemma 2.3 The optimal solution of Problem (3) satisfies αt,` ≥ √ − .
2
n
Proof. Let αt,` be an optimal solution of Problem (3) and set qt,` = qt (αt,` ). Then every
5
feasible solution α to Problem (3) satisfies
2
α(2u
−
e)
t
2
kqt (α)k = round k2ut − ek 2
n
X
α(2uit − 1)
=
round
k2ut − ek
i=1
2
2
n X
1
1
≤
α+
=n α+
.
2
2
i=1
|`|/2
Define β = 2√n − 12 . Then β is feasible for Problem (3), since kqt (β)k2 ≤ n(β + 21 )2 =
2|`| ; therefore, αt,` ≥ β.
Table 2 shows elements of the sequences ut and qt,` for n = 4 and eight pairs (t, `)
whose values are compatible with the O RTHO M ADS algorithm presented in Subsection 2.4. The values of αt,` and the square norm kqt,` k2 are also reported. One can
|`|/2
also notice that αt,` often differs from
2 . In the example illustrated in Figure 1,
α
(t, `) = (6, 3) and qt (α) = round √481
(−9, −20)T . An optimal solution of (3) is
√
√
√
αt,` = 2 and satisfies kqt,` k = 5 < 8 = 2|`|/2 < kqt (2|`|/2 )k = k(−1, −3)T k = 10.
Table 2: The sequence of Halton directions ut and the adjusted Halton directions qt,` for
n = 4 and eight pairs (t, `).
(t, `)
(5, 0)
(6, 1)
(7, 2)
(8, 3)
(9, 4)
(10, 5)
(11, 6)
(12, 7)
ut
αt,`
qt,`
ut,2
ut,3
ut,5
ut,7
5/8
7/9 1/25
5/7 1.0 0 0 −1
3/8
2/9 6/25
6/7 1.0 0 −1 0
7/8
5/9 11/25 1/49 1.0 1 0 0
1/16
8/9 16/25 8/49 2.5 −2 1 1
9/16 1/27 21/25 15/49 4.0 0 −3 2
5/16 10/27 2/25 22/49 5.5 −2 −1 −5
13/16 19/27 7/25 29/49 7.7 5 4 −4
3/16 4/27 12/25 36/49 11.0 −7 −7 0
kqt,` k2
0
1
−1
−1
−1
−1
2
5
1
2
2
7
14
31
61
123
The following proposition gives a property of the scaling and rounding operations,
which transform a vector v into q = round (αv/kuk). The property states that the
directions v/kvk and q/kqk are arbitrarily close for sufficient large values of α:
6
√
√
2 n
n
+
and
Proposition 2.4 Let v 6= 0 be a vector in R . For any ε > 0, if α >
ε
2
q
v
v < ε .
q = round α
6= 0, then −
kvk
kqk kvk 2
√
√
Proof. Consider ε > 0 and α > 2 n/ε + n/2. The vector q may be expressed as
v
q = α
+ δ, where δ = (δ1 , δ2 , . . . , δn )T and |δi | < 1/2 for all i = 1, 2, . . . , n. It
kvk
follows that
q
α
v
v
δ
kqk − kvk = kqk − 1 kvk + kqk α
δ v
≤ kqk − 1 kvk + kqk |α − kqk| kδk
=
+
.
kqk
kqk
n
kvk
kvk
The norm of q can be bounded with α
− kδk ≤ kqk ≤ α
+ kδk and therefore
kvk
√kvk
√
√
√
|α − kqk| ≤ kδk. Furthermore, α > 2 n/ε + n/2 > n/2 and kδk < n/2 implies
that α satisfies 0 < α − kδk. It follows that
√
q
v
2kδk
2kδk
n
ε
kqk − kvk ≤ kqk ≤ α − kδk < α − √n/2 < 2 .
2.3
Construction of an orthogonal integer basis
This subsection gives a way to transform a sequence of directions into a sequence of
orthogonal bases. Given an integer nonzero vector q ∈ Zn , we apply the (symmetric)
scaled Householder transformation [12] to construct an orthogonal basis for Rn composed of integer vectors:
H = kqk2 (In − 2vv T ), where v =
q
.
kqk
Proposition 2.5 The columns of H form an integer orthogonal basis for Rn .
Proof. First, the columns of H are mutually orthogonal, since v T v = 1 and
H T H = kqk4 (In − 2vv T )T (In − 2vv T )
= kqk4 (In − 2vv T − 2vv T + 4vv T vv T ) = kqk4 In .
7
(4)
Second, by dividing the previous equation by kqk4 and applying symmetry, we reveal
1
−1
the inverse of H as H −1 = kqk
exists, the columns of H form a basis in
4 H. Since H
n
R . Finally, the entries of
H = kqk2 In − 2kqk2
q qT
= kqk2 In − 2qq T
kqk kqk
are integer, since q and kqk2 are integer.
The next proposition shows that the Householder transformation applied to a dense
set of normalized directions produces a dense set of normalized directions:
Proposition 2.6 For t = 1, 2, . . ., let vt = kqqtt k and Ht = kqt k2 (In − 2vt vtT ). If {vt }∞
t=1
th
is dense
sphere, then the normalized sequence composed of the i columns
n on theounit
∞
Ht ei
of Ht , kHt ei k
is dense on the unit sphere.
t=1
Proof. Let w ∈ Rn with kwk = 1 be an arbitrarily unit vector, ε > 0 be some small
positive number, and i ∈ {1, 2, . . . , n} be the index of a column. For n > 1 (n = 1 is
trivial), we need to show that there exists an index t ∈ N such that the ith column of Ht ,
Ht ei , satisfies
Ht ei
kHt ei k − w < ε.
p
Ht ei
First, observe that kHt ei k = eTi HtT Ht ei = kqt k2 , and therefore kH
= ei − 2vt vtT ei .
t ei k
Now, define the vector
(
√ 1
(ei − w)
if wi < 1,
2(1−wi )
d=
ei+1 (the sum i + 1 is modulo n) otherwise.
Observe that if wi = 1 then the vector d satisfies kdk = 1 and 2di d = 0 = ei − w, and if
wi < 1 then
s
√
1
kdk =
dT d =
(ei − w)T (ei − w) = 1,
2(1 − wi )
2di d =
1
(ei − w)i (ei − w) = ei − w.
(1 − wi )
By assumption, {vt }∞
t=1 is dense on the unit sphere, and therefore there exists some
index t such that vt = d + δ, where δ ∈ Rn is small enough to satisfy kδi (d + δ) + di δk <
8
ε/2. The proof may be completed as follows:
Ht ei
T
kHt ei k − w = ei − 2vt vt ei − w
= ei − 2(d + δ)(d + δ)T ei − w
= kei − 2(di + δi )(d + δ) − wk
= kei − 2di d − w − 2(δi (d + δ) + di δ)k
= kei − (ei − w) − w − 2(δi (d + δ) + di δ)k
= 2 kδi (d + δ) + di δk < ε.
In Figure 1, the Householder transformation
is applied
to qt,` = (−1, −2)T and
3 −4
produces the integer orthogonal basis Ht,` =
.
−4 −3
2.4
The O RTHO M ADS instance of M ADS
The new O RTHO M ADS instance of M ADS can be now defined by combining the components introduced in Subsections 2.1–2.3. The POLL set Pk used by O RTHO M ADS at
iteration k is entirely determined by the values of the pair tk and `k . The tth
k element of
the Halton sequence utk is used to create the adjusted Halton direction qtk ,`k whose norm
is as close as possible to 2|`k |/2 . The Householder transformation on qtk ,`k produces an
orthogonal integer basis Htk ,`k , and the norm of each column is close to 2|`k | .
The LT M ADS and O RTHO M ADS algorithms are identical except for the construction
of the set Pk and the POLL directions Dk . The set of directions D = [In − In ] defining
the mesh Mk and the mesh update parameters τ = 4, w− = −1 and w+ = 1 are the same
p
for both algorithms. The mesh size parameter ∆m
k and the POLL size parameter ∆k are
still defined with the integer `k , except that it is allowed to be negative. This extension is
not specific to O RTHO M ADS and can be applied in LT M ADS as well: at each iteration
k, the POLL and mesh size parameters are entirely defined by the value of `k :
−`
4 k if `k > 0
p
−`k
m
∆k = 2
and ∆k =
(5)
1 otherwise.
p
At iteration k = 0, `k is set to 1 and ∆m
0 = ∆0 = 1. The mesh and POLL size parameters
p
m |`k |
always satisfy ∆m
= ∆pk .
k ≤ ∆k and ∆k 2
In the update step of iteration k, if no new incumbent is found, the iteration is said
to be unsuccessful and `k+1 ← `k + 1. Otherwise, the iteration is a success and `k+1 ←
`k − 1. The M ADS algorithm generates POLL trial points at a distance of order ∆pk from
m
the POLL center, on a mesh Mk of size ∆m
k . At an unsuccessful iteration, ∆k is reduced
9
[0] Initializations
x0 ∈ Ω, `0 ← 0, k ← 0
[1] Iteration k
(optional)
evaluate f on Sk ⊂ Mk
POLL (optional is the SEARCH was successful)
if the POLL size is the smallest one so far
(i.e., if ∆pk = min{∆pj : j = 0, 1, . . . , k})
tk ← `k + n + 1
else (i.e., smaller POLL sizes were considered)
tk ← max{tj : j = 0, 1, . . . , k − 1}
compute utk , qtk ,`k , Htk ,`k , and Dk = [Htk ,`k − Htk ,`k ]
evaluate f on Pk ⊂ Mk
SEARCH
[2] Updates
if the iteration is successful
xk+1 ← xs ∈ Sk or xp ∈ Pk
`k+1 ← `k − 1
else (iteration failure)
xk+1 ← xk
`k+1 ← `k + 1
k ←k+1
goto [1] if no stopping condition is met
Figure 2: The O RTHO M ADS algorithm.
faster than ∆pk and the number of possible POLL trial points increases, allowing more
flexibility in the choice of the POLL directions Dk .
Figure 2 describes our algorithm. The POLL directions Dk depend entirely on the
two integers tk and `k . These integers are chosen to ensure that there will be a sequence
of unsuccessful iterations for which the mesh size parameter goes to zero, and such that
the directions used in that subsequence will be the tail of the entire Halton sequence.
In order to accomplish that goal, we keep track of the value of the smallest POLL size
parameter visited so far. At every iteration where ∆pk is equal to that value, we set
tk = `k + n + 1. A consequence of this way of fixing tk is that the set of ordered indices
U := {k1 , k2 , . . .} = {k : iteration k is unsuccessful, and ∆pk ≤ ∆pj ∀j = 0, 1, . . . k}
satisfies (tk1 , `k1 ) = (n + 1, 0), (tk2 , `k2 ) = (n + 2, 1), . . . , (tki , `ki ) = (n + i, i − 1), and
the set of Halton directions {utk }k∈U is precisely {ut }∞
t=n+1 .
At the other iterations, those for which smaller POLL sizes were previously considered, we just keep increasing tk so that a new Halton direction is used. Examples of
10
Table 3: Example of O RTHO M ADS iterations for n = 4. Iterations k ∈ {4, 5, 8} correspond to failed iterations with consecutive Halton elements tk = 5, 6 and 7 satisfying
tk = `k + n + 1.
k Succ/Fail (tk , `k ) ∆m
∆pk kDk ei k
k
0
S
(5 , 0) 1
1
1
1
S
(6 , −1) 1
2
2
2
F
(7 , −2) 1
4
4
3
F
(8 , −1) 1
2
2
4
F
(5 , 0) 1
1
1
5
F
(6 , 1) 1/4 1/2
2
6
S
(7 , 2) 1/16 1/4
4
7
F
(9 , 1) 1/4 1/2
2
8
F
(7 , 2) 1/16 1/4
5
9
S
(8 , 3) 1/64 1/8
8
pairs (tk , `k ) can be seen in Table 3. The boldface entries are those where the POLL size
parameter is the smallest one so far. In this example, the first three indices of U would
be {4, 5, 8}.
As in LT M ADS, the basis Htk ,`k is completed to a maximal positive basis composed
of 2n directions,
Dk = [Htk ,`k − Htk ,`k ],
the set of POLL directions. A minimal positive basis with n + 1 directions is not considered in order to keep orthogonal directions. Table 4 illustrates O RTHO M ADS bases
Htk ,`k , with possible pairs (tk , `k ).
Notice that any direction Dk ei (1 ≤ i ≤ 2n) satisfies kDk ei k = kqt,` k2 ≤ (2|`|/2 )2 =
2|`| and kDk ei k ≤ 2|`| . Therefore, the POLL trial point xk + ∆m
k Dk ei is at an euclidean
p
m |`|
distance of at most ∆k 2 = ∆k from the POLL center. This distance is comparable to
that used in LT M ADS, where the POLL trial points are exactly at a distance ∆pk (using
the `∞ norm) from the POLL center.
We conclude this section with the following propositions that show that O RTHO M ADS
has the same convergence properties as in [4] with no need for a probabilistic argument.
n
o∞
qt,`
with ` = t − n − 1 is dense
Proposition 2.7 The set of normalized directions kqt,`
k
t=1
in the unit sphere.
Proof. Let ε > 0 and d ∈ Rn with kdk = 1. Proposition 2.1 states that the Halton
sequence {ut }∞
in the unitcube [0, 1]n .Therefore, there exists an index t such
t=1 is dense
√
√
|t−n−1|/2
2ut −e
that 2 √n − 12 > 2 ε n + 2n and k2u
− d ≤ 2ε .
t −ek
11
Table 4: A sequence of O RTHO M ADS bases corresponding to seven consecutive failed
iterations. Pairs (tk , `k ) correspond to consecutive Halton elements t = 5, 6, . . . , 12 with
tk = `k + n + 1.
(tk , `k )
kHtk ,`k ei k
(5, 0)
1
Htk ,`k

1
0
0
0





0
1
0
0
0
−1

0

0

0
2
0
0
0
0
0
0
0
0
0
2
0
0
2
0
0
0
0
2
0
0
0
2
2
−1
0
4
0
4
4
5
−2
4
−2
5
−4
2
2

(6, 1)
2






(7, 2)
2






(8, 3)
7
(tk , `k )
kHtk ,`k ei k






1

0

2

0

0

2

0

0

0

−4

2

2

5
12
(9, 4)
14
Htk ,`k

14
0
0
0





0
−4
12
0
12
6

−6 

4

(10, 5)
31





−4 −2 −10 29

11 −40 40 −20

−40 29
32 −16 

40 32
29
16 


−20 −16
25 −98

(11, 6)
61
(12, 7)
123
−6
4
−4 −20
12

−4


29 −10 −2 
 −4


 −20 −10 −19 −10 



0
23

16
0

 −98


0

25
0
0
123
70
70
0
53

70

70 

0 

73
|`|/2
√
√
Lemma 2.3 ensures that that αt,` ≥ 2√n − 12 > 2 ε n + 2n . Combining this last
inequality with Proposition 2.4 gives
qt,`
qt,`
2ut − e
2u
−
e
t
≤ +
−
d
−
−
d
kqt,` k
kqt,` k k2ut − ek k2ut − ek
ε ε
<
+ = ε.
2 2
This allows us to state our main result:
Theorem 2.8 O RTHO M ADS is a valid M ADS instance.
Proof. In order to show that O RTHO M ADS is a valid M ADS instance we need to show
that the POLL directions satisfy the following four properties [2, 4]:
• Any direction Dk ei (1 ≤ i ≤ 2n) can be written as a non-negative integer combination of the directions of D: This is the case by construction.
• The distance from the POLL center xk to a POLL trial point (in `∞ norm) has to be
bounded above by ∆pk : This is also the case by construction because we ensured
m
that kDk ei k ≤ 2|`k | for all i in {1, 2, ..., 2n} and k∆m
k Dk ei k∞ ≤ k∆k Dk ei k ≤
p
|`k |
∆m
= ∆k .
k 2
• Limits (as defined in [7]) of convergent subsequences of the normalized sets Dk =
{d/kdk : d ∈ Dk } are positive spanning sets. This can be shown the same way as
since, for O RTHO M ADS and with
in [2] where the proof for LT M ADS is detailed,
Htk ,`k = {d/kdk : d ∈ Htk ,`k }, det Htk ,`k = −1.
• The set of normalized directions used over all failed iterations is dense in the unit
sphere: The strategy chosen for the values of tk and `k ensures that there exists a
sequence of failed iterations corresponding to consecutive values of tk . These iterations k ∈ U can be chosen to correspond to large values of `k because, from [4],
m
−`k
lim ∆m
for `k ≥ 0. For k ∈ U , the sets of directions
k = 0, and ∆k = 4
k∈U
k→∞
{Dk }k∈U are constructed from consecutive directions qtk ,`k , which are dense in
the unit sphere after normalization (Proposition 2.7). Then, from n
Proposition
o 2.6
Dk ei
and since Dk = [Htk ,`k − Htk ,`k ], the set of normalized directions kDk ei k
is
k∈U
also dense in the unit sphere for all i = 1, 2, . . . , 2n.
13
3
Numerical Tests
In this section, O RTHO M ADS is compared to its predecessor LT M ADS [4] and to the
G PS method [18], on 45 problems from the literature. In the M ADS algorithms, the
theory supports handling constraints by the extreme barrier approach: Points outside Ω
are simply ignored and f is not evaluated. For G PS, the extreme barrier approach is
supported by the theory only for a finite number of linear constraints [13]. Still, for
comparison, we apply two different approaches: the extreme barrier (G PS -EB), and the
filter method described in [3] (G PS -F ILTER), which has stronger theoretical support.
Because of its random behavior, 30 instances of LT M ADS are performed for each
problem. G PS and O RTHO M ADS are scored by comparing them against the 30 LT M ADS
instances. A score of s for G PS or O RTHO M ADS means that this instance gave a value
of f at least as good as s of the 30 LT M ADS instances, with a relative precision of 1%.
The worst score is 0 and the perfect score corresponds to 30. We consider that a bad
instance has a score less than 10, an acceptable instance is between 10 and 19, and a
good instance has a score greater than or equal to 20.
p
The integer `k (see (5)), defining the mesh and POLL size parameters ∆m
k and ∆k
at iteration k, is allowed to be negative for both LT M ADS and O RTHO M ADS. Maximal positive bases (2n directions) are used in the three methods, as is the opportunistic
strategy (the POLL is interrupted at the first success), and the optimist strategy: after a
successful point has been found, a SEARCH point is generated further along the same
direction. No other SEARCH is performed. The stopping criteria is satisfied when the
p
POLL size parameter ∆k drops below 1E-12 or when the number of function evaluations
reaches 1000n.
The methods are tested on 45 problems divided into 4 groups: our choice of smooth
and nonsmooth unconstrained problems is the same as in [10] and [9], respectively, with
21 smooth problems from the C UTEr test set [16] and 13 nonsmooth problems from [14],
which is a compilation of nonsmooth problems from the literature. We also tested on 9
constrained problems from [5, 6, 14], and in addition, we added two problems from [1]
that correspond to real applications.
All results and problem descriptions are summarized in Tables 5–9, where f (x∗ )
corresponds to the best known minimal value of f , value to the final value of f for each
method, and evals to the number of function evaluations that each method performed.
Tables 5 and 6 show results on the 21 unconstrained smooth problems from C UTEr.
O RTHO M ADS has a perfect score on 17 of these problems. Table 7 displays results on
the 13 unconstrained nonsmooth problems, where O RTHO M ADS achieves good scores
on 7 problems. Table 8 shows results for the 9 constrained problems. The same number
of problems (4) is considered good and bad for O RTHO M ADS. Finally, Table 9 presents
results for the two real applications, and O RTHO M ADS has perfect scores on both of
them.
Table 10 summarizes the results. The first observation is that both M ADS instances
14
Table 5: C UTEr unconstrained smooth problems (1 of 2). A score of s for a method
indicates that the final f value is at least as good as s of the 30 LT M ADS runs (with a
relative error of 1%).
Problem
n f (x∗ )
A RWHEAD
10 0.00
A RWHEAD
20 0.00
B DQRTIC
10 11.9
B DQRTIC
20 35.4
B IGGS 6
6 0.00
B ROWNAL 10
10 0.00
B ROWNAL 20
20 0.00
P ENALTY 1
10 7.09E-5
P ENALTY 1
20 1.58E-4
P ENALTY 2
10 0.294E-3
P ENALTY 2
20 0.829E-2
LT M ADS×30
G PS
worst
best
evals
evals
evals
score
value
value
value
6128
650
1039
30
0.00
0.00
0.00
20000
1285
2079
30
0.00
0.00
0.00
6320
4497
3510
30
18.3
18.3
18.3
20000
17884
17074
30
58.3
58.3
58.3
831
570
764
30
2.06
2.06
2.06
10000
10000
10000
1
0.07
0.00
0.06
20000
20000
20000
3
0.34
0.00
0.16
10000
10000
10000
0
7.09E-5 7.09E-5 8.82E-5
20000
20000
20000
0
1.58E-4 1.58E-4 1.88E-4
10000
10000
10000
30
1.280E-3 1.241E-3 1.243E-3
20000
20000
20000
0
1.080E-2 1.078E-2 1.152E-2
. . . continued on Table 6
15
O RTHO M ADS
evals
score
value
660
30
0.00
1320
30
0.00
5763
30
18.3
20000
30
58.3
713
30
2.06
10000
3
0.04
20000
20
0.01
10000
30
7.09E-5
20000
30
1.58E-4
10000
30
1.250E-3
20000
30
1.079E-2
Table 6: C UTEr unconstrained smooth problems (2 of 2). A score of s for a method
indicates that the final f value is at least as good as s of the 30 LT M ADS runs (with a
relative error of 1%).
Problem
n f (x∗ )
P OWELLSG
12 0.00
P OWELLSG
20 0.00
S ROSENBR
10 0.00
S ROSENBR
20 0.00
T RIDIA
10 0.00
T RIDIA
20 0.00
VARDIM
10 0.00
VARDIM
20 0.00
W OODS
12 0.00
W OODS
20 0.00
LT M ADS×30
G PS
worst
best
evals evals evals
score
value value value
12000 12000 10093
30
0.00 0.00 0.00
20000 20000 20000
30
0.00 0.00 0.00
10000 10000 10000
30
6.31 0.00 0.00
20000 20000 20000
30
16.52 0.48 0.00
10000 7591 9317
30
0.00 0.00 0.00
20000 20000 20000
30
0.00 0.00 0.00
1000 8163 10000
0
0.00 0.00 4.01
20000 20000 20000
0
0.00 0.00 110.59
10951 7327 8433
30
104.91 104.91 104.91
20000 19181 20000
30
174.84 174.84 174.84
average scores 20.2
16
O RTHO M ADS
evals
score
value
12000
30
0.00
20000
30
0.00
10000
25
0.06
1958
30
0.00
10000
30
0.00
20000
30
0.00
10000
30
0.00
20000
0
110.84
9479
30
104.91
20000
30
174.84
26.6
Table 7: Results for unconstrained nonsmooth problems from [14]. A score of s for a
method indicates that the final f value is at least as good as s of the 30 LT M ADS runs
(with a relative error of 1%).
LT M ADS×30
G PS
worst
best
n f (x∗ )
evals evals evals
score
value value value
E L ATTAR
456
1795
2392
16
6 0.560
8.021 0.563 1.714
E VD 61
490
3280
920
5
6 0.0349
1.6001 0.0417 0.5443
F ILTER
1293
1761
1132
7
9 0.00619 0.00971 0.00797 0.00950
G OFFIN
50000 50000 24097
30
50 0.00
1.10
0.06
0.00
H S 78
403
1026
819
13
5 −2.92
10.00 −2.88
0.00
L1H ILB
17953 50000
8738
0
50 0.00
1.84
0.04
3.95
M X H ILB
11523 20377
9384
0
50 0.00
0.280 0.003 0.976
O SBORNE 2
2046
5414
1660
0
11 0.0480
0.1703 0.0549 0.2799
P BC 1
1211
1127
677
2
5 0.0223
0.4146 0.0343 0.3845
P OLAK 2
1449
1742
1327
30
10 54.6
54.6
54.6
54.6
S HOR
1345
882
787
0
5 22.6
22.9
22.6
23.5
W ONG 1
1161
2109
1100
30
7 681
699
693
697
W ONG 2
5403
5403
1871
0
10 24.3
31.4
24.8
47.4
average scores 10.2
Problem
17
O RTHO M ADS
evals
score
value
3984
20
1.504
4224
22
0.0709
1332
9
0.00935
16842
30
0.00
405
13
0.00
50000
14
0.22
20755
3
0.197
4555
20
0.1089
1291
17
0.1602
949
30
54.6
1087
30
22.8
1823
30
693
4977
0
32.8
18.3
Table 8: Results for constrained problems. A score of s for a method indicates that the
final f value is at least as good as s of the 30 LT M ADS runs (with a relative error of 1%).
LT M ADS×30
G PS -F ILTER
G PS -E B
worst
best
n m f (x∗ )
evals evals evals
evals
score
score
value value value
value
C RESCENT 10 [5] 1279 4473
2152
1172
0
0
10 2 −9.00
−8.26 −8.95 −6.19
−2.32
D ISK 10 [5]
1909 2626
2322
1143
0
0
10 1 −17.3
−17.2 −17.3 −13.0
−10.0
B250 [6]
60000 60000 15412
27773
0
0
60 1 7.95
15.41
7.99 1142.01
1116.03
B500 [6]
16705 15359 11189
18912
0
0
60 1 104
557
104
1235
1254
G2 [6]
3880 6461
2056
2689
4
29
10 2 −0.728
−0.181 −0.728 −0.221
−0.706
G2 [6]
10877 15722
6376
6551
3
29
20 2 −0.804
−0.203 −0.736 −0.241
−0.721
H S 114 [14]
1506 2135
1756
1756
0
0
9 6 −1769
−1012 −1312 −968
−968
M AD 6 [14]
1122 1542
1378
1378
22
22
5 7 0.102
0.113 0.102 0.103
0.103
P ENTAGON [14]
859 2525
601
601
0
0
6 15 −1.86
−1.60 −1.86
0.00
0.00
average scores 3.2
8.9
Problem
18
O RTHO M ADS
evals
score
value
5497
30
−8.97
2359
30
−17.2
60000
0
16.92
29858
6
277
5414
22
−0.561
20000
29
−0.711
1661
4
−1016
1671
7
0.108
980
19
−1.81
16.3
Table 9: Results for real applications. A score of s for a method indicates that the final
f value is at least as good as s of the 30 LT M ADS runs (with a relative error of 1%).
Displayed z values for problem S TY are divided by 107 .
LT M ADS×30
G PS -F ILTER
worst
best
n m f (x∗ ) evals evals evals
score
value value value
M DO [1]
1767 1715 2719
0
10 10 −3964 −2530 −3964 −1386
S TY [1]
1590 1189 2073
22
8 11 −3.35 −2.88 −3.29 −3.11
average scores 11.0
Problem
G PS -E B
O RTHO M ADS
evals
evals
score
score
value
value
2048
1212
0
30
−1386
−3964
2113
1214
0
30
−2.82
−3.27
0.0
30.0
outperform G PS. For 25 problems out of 45, O RTHO M ADS found the same solution as
the best of 30 LT M ADS runs. The new method solved 32 problems out of 45 problems
efficiently enough that, for these problems, the single run of O RTHO M ADS was better
than two thirds of the 30 LT M ADS runs. For 4 problems, the two methods performed
equally well, and for 9 problems, at least two thirds of the LT M ADS runs gave a better
solution than the one produced by O RTHO M ADS.
Table 10: Summary for the G PS and O RTHO M ADS performances. F, E and O correspond
respectively to G PS -F ILTER, G PS -E B, and O RTHO M ADS. A bad instance has a score
between 0 and 9, an acceptable (acc.) instance a score between 10 and 19, a good
instance a score higher than 20 and a perfect (perf.) instance has a score of 30.
problems
smooth
nonsmooth
constrained
real appli.
total or avg
average
# of
# of bad # of acc.
scores (on 30) problems instances instances
F
E O
F E O F E O
20.2 20.2 26.6
21
7 7 2 0 0 0
10.2 10.2 18.3
13
8 8 3 2 2 3
3.2 8.9 16.3
9
8 6 4 0 0 1
11.0 0.0 30.0
2
1 2 0 0 0 0
13.5 14.2 22.3
45
24 23 9 2 2 4
19
# of good
instances
F E O
14 14 19
3 3 7
1 3 4
1 0 2
19 20 32
# of perf.
instances
F E O
14 14 17
3 3 4
0 0 2
0 0 2
17 17 25
Figure 3 illustrates the spread of the directions for both LT M ADS and O RTHO M ADS.
Rosenbrock’s function [17] with n = 2 and n = 3 was used with 2000 and 3000 evaluations, respectively. In the two-dimensional case, all the normalized directions used to
generate POLL trial points are directly represented on the top two subfigures. It is clear
that O RTHO M ADS directions are well distributed on the unit circle. This is not the case
with LT M ADS because half the directions correspond to either ±e1 or ±e2 . For n = 3,
the two plots on the bottom represent the standard angles of the normalized directions
in spherical coordinates. There again it can be seen that O RTHO M ADS directions have
a better distribution than those of LT M ADS, since at least two thirds of the LT M ADS
directions possess some null coordinates. On the subfigure using LT M ADS with n = 3,
the horizontal bar at Φ = π/2 corresponds to the set of directions where z = 0. The
vertical bars at θ = ±π/2 correspond to directions with x = 0, and the one at θ = 0 and
θ = π correspond to directions with y = 0.
LTMads n=2
y
OrthoMads n=2
1
1
0.5
0.5
y
0
−0.5
−1
−1
"
0
−0.5
−0.5
0
0.5
−1
−1
1
0.5
LTMads n=3
OrthoMads n=3
3
2.5
2.5
2
"
1.5
1.5
1
0.5
0.5
0
0
2
1
2
1
−2
0
x
3
0
−0.5
x
−2
!
0
2
!
Figure 3: LT M ADS and O RTHO M ADS normalized
function with n = 2 and n = 3.
20
POLL
directions on the Rosenbrock
4
Discussion
This paper introduced O RTHO M ADS, an alternative instantiation of the M ADS class of
algorithms. The advantages of O RTHO M ADS over the original LT M ADS are that the
M ADS directions are chosen deterministically, and that those directions are orthogonal
to each other. Moreover, O RTHO M ADS inherits all of the M ADS convergence properties,
without probabilistic arguments, and without additional parameters.
Intensive tests on 45 problems from the literature showed that both M ADS instances
outperform the G PS algorithm, and that O RTHO M ADS is at least as competitive as LTM ADS, with a better distribution of the POLL directions.
Acknowledgements
We would like to thank Ana Custódio and Luis Vicente for helpful remarks about missed
cones of directions in single iterations of LT M ADS. These remarks and similar observations of our own helped motivate us to discover O RTHO M ADS. We also wish to thank
Andrew Booker for suggesting quasi-Monte Carlo methods which started us in directions that led to our use of Halton sequences.
References
[1] C. Audet, V. Béchard, and S. Le Digabel. Nonsmooth optimization through Mesh Adaptive Direct Search and Variable Neighborhood Search. To appear in Journal of Global
Optimization, D OI: 10.1007/s10898-007-9234-1, 2007.
[2] C. Audet, A. L. Custódio, and J. E. Dennis, Jr. Erratum : Mesh Adaptive Direct Search
Algorithms for Constrained Optimization. S IAM Journal on Optimization, 18(4):1501–
1503, 2008.
[3] C. Audet and J. E. Dennis, Jr. A Pattern Search Filter Method for Nonlinear Programming
without Derivatives. S IAM Journal on Optimization, 14(4):980–1010, 2004.
[4] C. Audet and J. E. Dennis, Jr. Mesh Adaptive Direct Search Algorithms for Constrained
Optimization. S IAM Journal on Optimization, 17(1):188–217, 2006.
[5] C. Audet and J. E. Dennis, Jr. A M ADS Algorithm with a Progressive Barrier for DerivativeFree Nonlinear Programming. Technical Report G-2007-37, Les Cahiers du GERAD, May
2007.
[6] C. Audet, J. E. Dennis, Jr., and S. Le Digabel. Parallel Space Decomposition of the Mesh
Adaptive Direct Search Algorithm. Technical Report G-2007-81, Les Cahiers du GERAD,
November 2007.
21
[7] I. D. Coope and C. J. Price. Frame-Based Methods for Unconstrained Optimization. Journal of Optimization Theory and Applications, 107(2):261–274, 2000.
[8] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms,
Second Edition, pages 873–876. M IT Press and McGraw-Hill, 2001.
[9] A. L. Custódio, J. E. Dennis, Jr., and L. N. Vicente. Using Simplex Gradients of Nonsmooth
Functions in Direct Search Methods. To appear in I MA Journal of Numerical Analysis,
2007.
[10] A. L. Custódio and L. N. Vicente. Using Sampling and Simplex Derivatives in Pattern
Search Methods. S IAM Journal on Optimization, 18:537–555, 2007.
[11] J. H. Halton. On the Efficiency of Certain Quasi-Random Sequences of Points in Evaluating
Multi-Dimensional Integrals. Numerische Mathematik, 2(1):84–90, 1960.
[12] A. S. Householder. Unitary Triangularization of a Nonsymmetric Matrix. Journal of the
Association for Computing Machinery (ACM), 5(4):339–342, 1958.
[13] R. M. Lewis and V. Torczon. Pattern Search Methods for Linearly Constrained Minimization. S IAM Journal on Optimization, 10(3):917–941, 2000.
[14] L. Lukšan and J. Vlček. Test Problems for Nonsmooth Unconstrained and Linearly Constrained Optimization. Technical Report V-798, I CS A S C R, February 2000.
[15] J. Matoušek. On the L2 -Discrepancy for Anchored Boxes.
14(4):527–556, 1998.
Journal of Complexity,
[16] D. Orban N. I. M. Gould and Ph. L. Toint. C UTEr (and S IF D EC): a Constrained and Unconstrained Testing Environment, revisited. ACM Transactions on Mathematical Software,
29(4):373–394, 2003.
[17] H. H. Rosenbrock. An Automatic Method for Finding the Greatest or Least Value of a
Function. The Computer Journal, 3(3):175–184, October 1960.
[18] V. Torczon. On the Convergence of Pattern Search Algorithms. S IAM Journal on Optimization, 7(1):1–25, February 1997.
22