Nizhni Novgorod State University
Faculty of Computational Mathematics and Cybernetics
Searching Globally-Optimal
Decisions for Multidimensional
Optimization Problems
Sysoyev A.
Software Department
2007
Contents
• Optimization problem
• Global optimization technique
• Informational Statistical Approach
• Index method
• Parallelization
• Computing with arbitrary precision math
• Visualization
• Results
2
Optimization problem…
• Global optimization problem is stated as follows:
 f  y   min

 g j  y   0, j  1,..., k
 y  D  y ,..., y : y  a , b , 1  i  n
 1
 i  i i


n
• Criteria and constraints are continuous and
satisfy the Lipschitz condition
f  y1   f  y2   L  y1  y2
3
Optimization Problem…
• Given problem belongs to one class of timeconsuming problems
– Problem of integration
I 
  ( y )dy,
Q   y  D : gi ( y)  0, 1  i  m

Q
– The solution of the system of nonlinear equations
qi ( y)  0, 1  i  m
– Problem of multiextremal multidimensional problems
 *   ( y* )  min  ( y) : y  Q

– Problem of reconstruction of dependences, etc…
4
Optimization problem
Necessity of parallel computations
• Global optimization problems
can be:
– multidimensional
– multiextremal
– nonconvex and
multiply-connected
y*
– time-consuming at
function value computations.
So these problems are highly computational
intensive
5
Global Optimization Technique…
• To solve optimization problems various iterative schemes
can be applied
• An iteration:
– Choosing a current point yn
in the search domain
yn
– Calculating the problem functions
at the point yn
z n  f ( yn )
• Iterations are terminated by the given time limitations or
the accuracy
6
Global Optimization Technique…
• Naive method – the scanning technique
•
The number of iterations grows
exponentially with increasing of
problem dimension
N
T  T ()   (b j  a j ) /  
j 1
• Example
T  min  ( y t ) : 1  t  T , gi ( y t )  0, 1  i  m   *
7
Global Optimization Technique
• More efficient approach is to use iterative
procedures which generate iteration points with
nonuniform density in the search domain, viz.:
– More dense in locality
of function extreme
– Less dense otherwise
• Among such techniques – Informational
Statistical Approach
8
Informational Statistical Approach
Lipschitz condition
• We propose the limited variation Δy of the argument y
generates restricted variations Δ Fi(y)
Fi ( y)  Fi ( y)  Li
y  y , y, y  D, 1  i  s.
• In a wide set of problems arising in applications this
condition is satisfied as a consequence of limitations on
energy changes inherent to real systems to be modeled
• Such proposal shows a way to construct non-uniform
grids to solve the problems
9
Informational Statistical Approach
Successive scheme to generate nodes of the grid
• To utilize efficiently any information obtained at the time of
calculation we have to generate nodes of the grid in a
successive way
y k 1  Gk  y1 ,
, y k ; Z 1,
,Zk  , k  t
where the values Zl,1lk, being arguments of the
decision function Gk, are values of the vector-function
F(y), computed at the nodes yl,1lk, of the grid Yk.
• Such approach poses a problem of accumulating all
computed information as the data base
k   y1 ,
, yk ; Z 1,
, Z k
10
Informational Statistical Approach
Dimension reduction
• A possible way to cut down the complexity of formulating
the decision function Gk is to reduce multidimensional
problems
• It can be provided by using the continuous single-valued
correspondence of Peano type curve
…
1
0
• The Lipschitz condition is transformed to the Hölder condition
f
y  f y 
1
2
 L  y1  y2
f
 y  x   f  y  x 
2
1
 4 L
n   x2  x1 
1/ n
11
Informational Statistical Approach
Index scheme
• Successive scheme of function calculations:
-
g1  g1 ( xn )
g 2  g 2 ( xn )
- if g1  0 , then
- if g 2  0 , then…
…
- if g k  0 , then
-
f ( y ( xn ))
• Result – the value of the last calculated function
and its number (index)
12
Informational Statistical Approach
Scheme of iterations
• Choice of a current iteration point:
– Computing the interval characteristics
R0
x0
R1
x1
Rk
x2
xk
Rn-1
xk+1
xn-2
xn
– Choosing the interval with maximum characteristic
Rk  max
Ri
i  0, n 1
– Selecting the current point
xn  [ xk , xk 1 ]
13
Index method
A weak spot
• Close points in RN – can be not close on [0, 1]
0
1
14
Index method
Multiple mappings…
• A possible way for overcoming this disadvantage is using
the set of joint space-filling curves Yl ( x)  { y1 ( x), y2 ( x),..., yl ( x)}
…
1
0
…
1
0
…
1
0
…
0
0
1
…
1
15
Index method
Multiple mappings
• Multiple mappings produces the corresponding set of
univariate global optimization problems
i ( x)   ( yi ( x)), 1  i  l.
feasible point
0
1
2
…
L-1
L
L+1
unfeasible point
16
Parallelization
Approach
• Such information unity gives a possibility to solve the
function set l ( x)  {1 ( x), 2 ( x),..., l ( x)} concurrently
3) function calculations
2) mapping
1)
y ( xn )  xn , i  0, L
i
g j  g j ( y ( xn ))
f ( y ( xn ))  min
…
…
queue
0
1
Process 0
i-1
Process i
queue
i
L
L+1
Process L
17
Parallelization
Scheme of parallel computations…
• Choosing a point of a new iteration
18
Parallelization
Scheme of parallel computations…
• Each process informs all other ones
19
Parallelization
Scheme of parallel computations
• Each process sends a message including the function
value obtained to all other processes
20
Computing with arbitrary precision
math
• Problem
– To get the accuracy ε = (½)m in RN the accuracy (½)m·N
in [0, 1] should be applied
– If the C++ type double is used the maximum possible
accuracy is limited by the condition Nm < 52
(if m = 10 than the problem dimension have to N <= 5)
• Decision
– To use arbitrary precision numbers to represent points
in [0, 1]
21
Visualization…
22
Visualization…
23
Visualization
24
Results…
Coordinates of optimizer: x1 = 0.942, x2 = 0.946
Proc. х mapping
Iterations* Speed up
• 1 x 16
2000
• 2 х 16
1015
1.97
• 4 х 16
519
3.85
• 8 х 16
270
7.41
• 16 х 16
147
13.61
* - Iterations means maximum of the number of feasible
points calculated by processors
25
Results…
Coordinates of optimizer: x1 = 0.942, x2 = 0.946
Proc. х mapping
Iterations* Speed up
• 1 x 32
2000
• 2 х 32
1081
1.85
• 4 х 32
577
3.46
• 8 х 32
282
7.09
• 16 х 32
172
11.63
• 20 х 20
118
16.95
* - Iterations means maximum of the number of feasible
points calculated by processors
26
Results…
F ( x)    xi2  10 cos  2 xi   10  , x   x1 , x2 ,..., xN 
N
Rastrigin’s function
Proc. х N
• 1x8
• 8х8
• 16 х 8
• 1 х 10
• 8 х 10
• 16 х 10
i 1
D   xi   1.5, 1.5 , 1  i  N 
Iterations*
1000
147
69
1000
150
75
Speed up
6.80
14.49
6.67
13.33
* - Iterations means maximum of the number of feasible
points calculated by processors
27
Results…
Rastrigin’s function
F ( x)    xi2  10 cos  2 xi   10  , x   x1 , x2 ,..., xN 
N
i 1
D   xi   1.5, 1.5 , 1  i  N 
28
Results…
• Applied problems solved:
– The problem «Parameter identification in a model of
economics» (jointly with The Computing Center of
RAS, Moscow)
– The problem «Optimization of parameters of a
measurement system» (jointly with The Research
Institute «Burevestnik», Nizhni Novgorod)
– The optimization system developed is applied for
solving applied optimization problems in the Delft
University, Netherland
29
Results
• «Parameter identification in a model of
economics»
– 12 parameters in search domain
– 2 functional constraints
– The task was solved on MVS-1000 by scanning
technique with 8 points on each dimension. 500
processors was used. Solving time was about 5 hours
– The task was solved on single PC by Global Expert
system with 16 points on each dimension. Solving time
was about 2 minutes.
30
Questions,
Remarks,
Something to add…
31