Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Solving Minimax Problems using an Heuristic
Pattern Search Algorithm
Isabel A.C.P. Espírito Santo and Edite M.G.P. Fernandes
University of Minho, Braga, PORTUGAL
{iapinho;emgpf}@dps.uminho.pt
ICCAM 2008
Ghent, Belgium
July 7 - 11, 2008
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
1 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Outline
1
Introduction
2
Pattern search method - Hooke and Jeeves
3
Heuristic Pattern Search
4
Numerical Experiments and Conclusions
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
2 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Outline
1
Introduction
2
Pattern search method - Hooke and Jeeves
3
Heuristic Pattern Search
4
Numerical Experiments and Conclusions
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
3 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
The Minimax Optimization Problem
The bound constrained problems to be addressed are:
min f (x)
x∈Ω
where
f (x) = max Fj (x),
j=1,...,m
Fj : IRn → IR, j = 1, . . . , m are continuously differentiable
functions
Ω = {x ∈ IRn : l ≤ x ≤ u} is a closed set.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
4 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Motivation
Minimax problems appear in many engineering areas:
optimal control,
engineering design,
discrete optimization,
Chebyshev approximation,
game theory,
computer-aided design,
circuit design.
See references in Laskari, Parsopoulos & Vrahatis (2002) and Xu
(2001).
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
5 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Motivation
Inequality constrained optimization problem
minx o(x)
subject to gj (x) ≥ 0, j = 1, . . . , m
can be transformed into a minimax problem
min max Fj (x)
x
j=1,...,m
where
F1 (x) = o(x)
Fj (x) = o(x) − αj gj (x), αj > 0,
j = 2, . . . , m.
For sufficiently large αj , the minimizer of the minimax problem
coincides with solution of inequality problem.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
6 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Optimality conditions
Stationary point (Xu, 2001)
x∗ is a stationary point to the minimax problem, if there exist
elements λ∗j ≥ 0, j = 1, . . . , m such that
Pm ∗
Pm ∗
∗
j=1 λj ∇Fj (x ) = 0 and
j=1 λj = 1,
and
λ∗j = 0 if Fj (x∗ ) < max{F1 (x∗ ), . . . , Fm (x∗ )}.
Theorem (Xu, 2001)
If x∗ is a local minimum of minimax problem, then it is a stationary
point. Conversely, if f (x) is convex and x∗ is a stationary point,
then it is a global minimum to the minimax problem.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
7 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Difficulties
Minimax problems are difficult to solve through traditional
gradient based algorithms.
First derivatives of f (x) are discontinuous
at points where f (x) = Fj (x)
for two or more values of j in the set {1, . . . , m},
even if all the functions Fj (x) have continuous first derivatives.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
8 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Tools of Heuristic Pattern Search Method
Our proposal:
1
uses a derivative-free method, known as pattern search
method, as outline in Lewis & Torczon (1999);
2
is based on the Hooke and Jeeves moves - exploratory move +
pattern move - Hooke & Jeeves (1961);
3
and uses an heuristic move - a random descent walk - Hedar
& Fukushima (2004)
to obtain high accuracy solutions
⇒ the pattern move is followed by a random descent walk, when a
successful iterate is encountered.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
9 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Outline
1
Introduction
2
Pattern search method - Hooke and Jeeves
3
Heuristic Pattern Search
4
Numerical Experiments and Conclusions
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
10 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Hooke and Jeeves pattern search method
Let xk ∈ IRn be the iterate at iteration k;
Let ∆k be the step length;
The Hooke and Jeeves (HJ) method performs two types of moves:
the exploratory move carries out a coordinate search - a
search along the coordinate axes - about a selected iterate,
with a step size ∆k ;
when xk is a successful iterate, the pattern move - a
promising direction - is defined by xk − xk−1 .
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
11 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Hooke and Jeeves moves
When iterate xk is successful ⇒ pattern move, followed by an
exploratory move about xk + (xk − xk−1 ):
in IR2
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
12 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Hooke and Jeeves moves
When pattern move is unsuccessful ⇒ exploratory move about xk :
in IR2
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
13 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Outline
1
Introduction
2
Pattern search method - Hooke and Jeeves
3
Heuristic Pattern Search
4
Numerical Experiments and Conclusions
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
14 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Heuristic pattern move
Heuristic pattern search method performs two types of moves:
the exploratory move
is a coordinate search about a selected iterate, with a step size
∆k ;
a two-stage move
is a pattern move followed by an approximate descent
random search - defined by xk − xk−1 + dk - when xk is a
successful iterate.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
15 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Approximate descent random search
Let pk = xk + (xk − xk−1 )
based on two points y1 and y2 , randomly generated from the
neighborhood of pk ;
an approximate descent search for f at pk is
2
X
1
pk − y i
(∆fi )
,
kpk − yi k
j=1 |∆fj | i=1
dk = − P2
where ∆fj = f (pk ) − f (yj ).
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
16 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Heuristic descent move
When iterate xk is successful ⇒ a pattern move followed by a
descent random search dk at xk + (xk − xk−1 ):
in IR2
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
17 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Heuristic descent move
When pattern move is unsuccessful ⇒ exploratory move about xk :
in IR2
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
18 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Constraining for feasibility
To maintain iterate in Ω, a reflexion into the feasible region is
carried out - componentwise (for i = 1, . . . , n)

li + (li − xk,i ) if xk,i < li





xk,i
if li ≤ xk,i ≤ ui
xk,i =





ui − (xk,i − ui ) if xk,i > ui
If a component of an iterate still is out of the bounds then
xk,i = (li + ui )/2.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
19 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Outline
1
Introduction
2
Pattern search method - Hooke and Jeeves
3
Heuristic Pattern Search
4
Numerical Experiments and Conclusions
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
20 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Details
Algorithms coded in C programming language with AMPL interface
to read problems coded in AMPL:
22 benchmark bound constrained minimax problems - Lukšan &
Vlček (2000); Petalas, Parsopoulos & Vrahatis (2007)
16 minimax problems;
6 problems - mm10, mm12, mm2, mm20, mm3 and mm30 are
inequality constrained problems that were transformed into
minimax problems;
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
21 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Details
Termination conditions
The found solution has objective function value within 1% of the
optimal objective value (best known):
if |f ∗ | ≤ 10−12 then
|f (xk ) − f ∗ | ≤ 0.012 |1 + f (xk )|
else
|f (xk ) − f ∗ | ≤ 0.01 |f (xk )|
or
number of function evaluations ≤ 20000.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
22 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Details
Initialization of the step length
The step length ∆k is a vector - to handle variables of different
order.
The step length initialization (k = 0) is defined componentwise by
∆0,i = γ∆ x0,i , i = 1, . . . , n
(x0 published in literature)
Tested values:
γ∆ = 0.01 γ∆ = 0.1
γ∆ = 1
γ∆ = 10 γ∆ = 100 γ∆ = 1000
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
23 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Table of results - HJ pattern search / Heuristic pattern
search
P - Problem; f ∗ - best known solution in the literature;
HJ pattern search (deterministic method):

 N it − number of iterations to achieve the desired accuracy
N f e − number of objective function evaluations

solution - obtained solution according to termination conditions.







Heuristic pattern search (stochastic method) - each problem
was run 100 times:
AvN it − average number of iterations, over the 100 runs
AvN f e − average number of function evaluations, over the 100 runs
solution - best of the solutions found in the 100 runs
average - average of the solutions found in the 100 runs.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
24 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Tests with the heuristic pattern search algorithm
With the heuristic pattern search we solve
1
each problem 100 times, using the 6 values of γ∆ - total of
600 runs for each problem;
2
for each γ∆ , choose the best run (over the 100 runs) - with
solution closest to f ∗ ;
3
choose the γ∆ that gives solution closest to f ∗ ;
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
25 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Results #1 - Heuristic pattern search
P
mm1
mm10
mm11
mm12
mm13
mm14
mm15
mm16
mm17
mm2
mm20
f∗
γ∆
solution
AvN it
AvN f e
average
1.952225
7.20
2
-1.414214
0
0
3.59972
115.706
0.002636
-44
-40.10
0.01
1000
1
1
1
1
1000
0.01
0.01
0.1
0.1
1.957691
7.201657
2
-1.41395
0
0
3.600187
116.0436
0.002749
-43.8542
-43.9738
27
257
3
10
1
1
1079
1303
1709
171
129
237
1454
22
65
102
6
7972
19206
20005
1744
1296
1.964877
7.260439
2
-1.40479
0
0
3.675277
141.0696
0.053114
-43.6039
-40.3967
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
26 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Results #2 - Heuristic pattern search
P
mm21
mm22
mm3
mm30
mm4
mm5
mm51
mm6
mm7
mm8
mm9
f∗
γ∆
solution
AvN it
AvN f e
average
0.147e-7
3.88719
680.630
247
0
0
0
0
0.002016
0
-3
1000
0.01
1000
1000
0.1
0.01
0.1
1
0.01
1
100
0.467807
3.708345
684.6938
247.7281
8.26e-05
0.003266
3.71e-05
3.66e-07
0.005054
0
-2.99623
472
63
75
874
2824
864
800
825
2004
1
20
20021
1240
1983
14236
19472
20012
17128
9607
20005
6
130
0.495673
3.904433
686.9234
253.4493
0.01632
0.011521
46.17007
0.000474
0.06983
0
-2.98209
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
27 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Tests: HJ pattern search vs Heuristic pattern search
When comparing the HJ pattern search with the heuristic pattern
search we solve
1
2
each problem by pattern search algorithm, using the 6 values
of γ∆ ;
choose the best run - solution closest to f ∗ ;
3
use this γ∆ to solve each problem by heuristic pattern search
algorithm, 100 times;
4
choose the best run (over the 100 runs) - solution closest to
f ∗.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
28 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Comparison - minimax problems
Pattern search
P
mm1
mm11
mm13
mm14
mm15
mm16
mm17
mm21
mm22
mm4
mm5
mm51
mm6
mm7
mm8
mm9
Heuristic pattern search
γ∆
solution
N it
Nfe
solution
AvN it
AvN f e
10
1
1
1
1
0.1
1000
0.1
1000
100
100
100
1
1
1
1
1.95719
2
0
0
3.62122
116.73
0.084704
0.801571
3.92475
0.333333
1.60864
2.58774
0.12282
0.086806
0
-3
17
3
1
1
14
250
2236
585
84
4009
960
960
4995
2462
1
3
79
14
102
6
93
1847
20004
20026
1608
20001
20012
20012
20001
20006
6
14
1.957188
2
0
0
3.602588
116.3001
0.017755
0.475883
3.710802
0.000357
0.004962
4.9e-05
3.66e-07
0.007452
0
-2.99191
144
3
1
1
551
1292
1736
467
37
2762
843
546
825
1883
1
16
786
22
102
6
4204
19064
20004
20023
738
20003
20013
12543
9607
20005
6
133
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
29 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Comparison - Inequality constrained problems
Comparison with α = αj = 1 and α = αj = 10 (j = 2, . . . , m)
Pattern search
P
Heuristic pattern search
α
γ∆
solution
N it
Nfe
solution
AvN it
AvN f e
1
10
mm12 1
10
mm2
1
10
mm20 1
10
mm3
1
10
mm30 1
10
0.01
100
10
100
1000
1
1
0.01
0.1
1000
0.01
10
7.2576
7.34863
-1.41101
-1.40479
-43.5742
-43.6279
-52.3333
-39.7264
687.308
687.086
154.295
267.999
129
4013
9
17
11
14
2228
115
3
25
1490
1345
520
20004
43
84
57
119
20001
822
39
283
20005
20003
7.24468
7.213753
-1.40234
-1.40479
-53.3333
-43.5998
-40.1463
-43.9834
682.8735
684.6938
247.9039
248.8527
3263
222
13
17
2205
2024
1255
117
4
75
842
1044
16806
1151
85
108
20005
18230
11407
1171
67
1083
14444
16861
mm10
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
30 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Conclusions and Future Work
We presented a derivative-free pattern search search method
that incorporates an heuristic descent random walk, when a
successful iterate is found:
1
it improves solution accuracy;
2
it solves difficult non-differentiable problems - bound minimax
problems;
3
it is easy to implement.
Future development: extend heuristic pattern search to equality and
inequality constrained problems, using an augmented Lagrangian
function - penalty multiplier method.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
31 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
References
A.-R. Hedar and M. Fukushima, Heuristic pattern search and its hybridization
with simulated annealing for nonlinear global optimization, Optimization
Methods and Software, 19 (2004) 291–308.
R. Hooke and T. A. Jeeves, Direct search solution of numerical and statistical
problems, Journal on Associated Computation, 8 (1961) 212–229.
E. C. Laskari, K. E. Parsopoulos and M. N. Vrahatis, Particle swarm
optimization for minimax problems, Proceedings of IEEE 2002 Congress on
Evolutionary Computation, ISBN: 0-7803-7278-6, 1576–1581, 2002.
R. M. Lewis and V. Torczon, Pattern search algorithms for bound constrained
minimization, SIAM Journal on Optimization, 9 (1999) 1082–1099.
L. Lukšan and J. Vlček, Test problems for nonsmooth unconstrained and linearly
constrained optimization, TR 798, ICS, Academy of Science of the Czech
Republic, January 2000.
V. Torczon, On the convergence of pattern search algorithms, SIAM Journal on
Optimization, 7 (1997) 1–25.
S. Xu, Smoothing method for minimax problems, Computational Optimization
and Applications, 20 (2001) 267–279.
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
32 / 33
Introduction
Pattern search method - Hooke and Jeeves
Heuristic Pattern Search
Numerical Experiments and Conclusions
Thanks for your attention
Isabel A.C.P. Espírito Santo
[email protected]
Edite M.G.P. Fernandes
[email protected]
www.norg.uminho.pt/NSOS/
IACP Espírito Santo and EMGP Fernandes
Minimax Problems using an Heuristic Pattern Search
33 / 33