5th World Congress on Industrial Process Tomography, Bergen, Norway
Intel® MKL Trust-Region Solvers in Mathematical Problems of
Geophysics
Nikolay I. Gorbenko, Nikita A. Shustrov, Alexander V. Avdeev
Intel Corporation R&D Lab, Novosibirsk, Russia, e-mail: [email protected]
ABSTRACT
Trust-Region (TR) algorithms are relatively new iterative algorithms for solving nonlinear optimization
problems. High efficiency of TR methods was demonstrated in a number of recent papers and books.
They have global convergence and local super convergence, which differ them from line search
methods, used quite often for solving Inverse Problems. TR techniques is implemented in a number
of well-known SW libraries such as IMSL, TAO, GALAHAD, LANCELOT, etc.
Let us explain the main difference of TR method from classical Newton one. Assume we have a
current guess of the solution for the optimization problem. An approximate model can be constructed
near the current point. Solution of the approximate model can be taken as the next iterate point. The
classical line search algorithms also solve approximate models to obtain search directions. However,
in TR algorithms, the approximation model is only “trusted” in the region near current iterate. This
seems reasonable, because for general nonlinear functions local approximate models (such as linear
and quadratic approximations) can only fit the original function locally. The region that the approximate
model is trusted is called “trust region”. The trust region is adjusted from iteration to iteration, i.e. if
computations indicate the approximation model fit the original problem quite well, the trust region can
be enlarged. Otherwise when the approximation works not good enough the trust region will be
reduced.
TR solvers developed and implemented by the authors are based on Intel MKL (Math Kernel Library).
There are versions of TR for Intel IA32, EM64T, IA64 platforms (Windows and Linux operating
systems). In addition to Fortran-interface, TR solvers include C-language interface for all functions and
routines. TR solvers are implemented with OpenMP support and can be used in multiprocessing
mode.
Keywords Nonlinear optimization, Trust-Region solvers, Intel MKL, Oil & Gas Problems
1 Trust-Region algorithm
Trust-Region algorithms allow to solve (Conn et al. 2000):
– nonlinear least squares problem (without and with bound constraints);
– system of nonlinear equations (without and with bound constraints);
– minimization of functional (without constraints).
Let us explain some details of TR approach through the example of a nonlinear least squares problem:
2
minn F ( x)  minn y  f ( x) 2 , y  R m , x  R n , f : R n  R m , m  n
xR
xR
(1)
where
is continuously differentiable function.
The step for TR method is the solution of the subproblem
2
2
2
2
minn F ( xk )  J ( xk )T d , where d
d R
that is the approximation of the goal function
Jacobian of
<k
in a neighborhood of current point х, and
(2)
-
.
*Trademark Information Intel, Itanium, Dual-Core Xeon and Xeon are trademarks of Intel Corporation in the U.S.
and other countries. Intel's trademarks may be used publicly with permission only from Intel. Fair use of Intel's trademarks in
advertising and promotion of Intel products requires proper acknowledgement.
*Other names and brands may be claimed as the property of others.
1
3rd World Congress on Industrial Process Tomography, Banff, Canada
TR pseudo code for solving a nonlinear least squares problem:
Step 1: Choose initial guess
x1  R n , 1  0
Step 2: Solution of (2) gives value of dk
If || F ( xk ) || 2 || F ( xk )  J ( xk ) d k || 2 then stop;
T
Computes rk 
|| F ( xk ) || 2  || F ( xk  d k ) || 2
|| F ( xk ) || 2  || F ( xk )  J ( xk )T d k || 2
Step 3: Choose
 xk  d k
xk 1  
 xk
if rk  0
else
Define
1 || d k || 2 if rk  1

 k 1  2 k
if 1  rk   2
max[  || d ||,  ]
if rk   2
2
k
k

Step 4: k = k +1, go to Step 2.
The main problem of TR method is to solve the subproblem (2).
The following lemma is well known (Conn et al. 2000):

Lemma: A vector d  R is the solution of the problem
n
minn ( g T d 
d R
where
1 T
d Bd ), d
2
2

g  R n , B  R nn is a symmetric matrix, and   0 , if and only if there exits 0 such
that ( B  

I )d    g and B   I is semi-definite, d 
2
  and  ( d
2
 )  0.
The case where B   I has zero eigenvalues is called “hard” case and d can be written by


d   ( B   I )  g  v , where v is a vector in the null space of B   I . Unless in the hard case
  is the unique solution of the following equation:
1
(B   I ) g


1
1
 0.

2
For solving this equation we use Newton’s method to calculate  .

Consider the TR usage for solving the nonlinear least square problem with simple bound
constraints.
Let min f ( x)  min
l  x u
l  x u
1
2
F ( x) . The first-order necessary conditions of vector-minimizer of (1) are
2
stated as
T
D( x) 2 f ( x)  D( x) 2 F ' ( x) F ( x)  0.
where D is the diagonal scaling matrix (Coleman, 1996):
2
(3)
5th World Congress on Industrial Process Tomography, Bergen, Norway
1
D( x)  diag ( v1
2
1
1
( x) , v2 2 ( x) ,  , vn 2 ( x) ),
(4)
with
 xi  u i
x  l
 i i
min( xi  u i , u i  xi )
vi  x   
 1
 1

 1
if
(f ( x) i  0 and
ui  
if
(f ( x) i  0 and
li  
if
(f ( x) i  0 and
u i   or l i  
if
(f ( x) i  0 and
ui  
if
(f ( x) i  0 and
l i  
if
(f ( x) i  0 and
u i  l i  
for i  1,  , n.
Let   {x  R : l  x  u}  R , and we write int( ) for the strict nonempty interior of  . To
update the current iterate we need the solution of an elliptical trust-region subproblem and
computation of a search step at each iteration.
Let xi  int( ) and trust-region size  k  0 be given. We consider the elliptical trust-region
n
n
subproblem:
min {mk ( p) : Dk p   k },
(5)
p
mk ( p) is quadratic model for f (x) at xki , i.e.
2
1
1
1
2
mk ( p)  Fk  Fk' p  Fk  FkT Fk' p  p T FkT Fk' p.
(6)
2
2
2
We let ptr ( k ) and pc ( k ) to the solution and Cauchy point of the trust-region subproblem (5),
where
respectively. Namely,
pc ( k ) is the minimizer of mk along the scaling steepest descent direction d k
subject to the satisfaction of the trust-region bound, i.e.
pc ( k )   k d k   k Dk2 FkT Fk' ,
(7)
where
 k  arg min {mk ( d k ) :  Dk d k   k }.
 0
The function
mk ( d k )
  f kT d k / Fk' d k
2
.
 , and the unconstrained minimizer has the form
is quadratic in
k
is given by
T
 k  min{
Dk1 Fk' Fk
2
k
'
k
'T
k
2
F D F Fk
The search step
2
,
k
T
Dk1 Fk' Fk
}.
(8)
p( k ) used in TR algorithm is defined by a linear combination of two vectors
ptr ( k ) and pc ( k ) :
_
_
p( k )  t p c ( k )  (1  t ) p tr ( k )
(9)
3
3rd World Congress on Industrial Process Tomography, Banff, Canada
_
where t  [0,1) is suitable scalar. The vector p tr (  k ) in (9) is equal to q, where
q  q1   q 2 is
given component-wise by
(l  xk )i ,

(q1 )i   ptr ( k )i ,
(u  x ) ,
k i

if
( xk  ptr ( k )) i  li
if
li  ( xk  ptr ( k )) i  ui
( xk  ptr ( k )) i  ui
if
(l  xk ) i ,

(q ) i   0,
(u  x ) ,
k i






( xk  ptr ( k )) i  li


(10)
if li  ( xk  ptr ( k )) i  u i 

if ( xk  ptr ( k )) i  u i

1
2
At the k-th iteration we consider the second degree polynomial r ( )  mk (q   q ) . On the basis
if
2
of the current information, r ( ) models the restriction of f to the points
q1   q 2 for  varying
from 0 to 1. Therefore, we compute the value  that minimizes this model and choose
fixed lower and upper bounds, 0  1   2  1 . This means that we set

  arg min r ( ) 



according
T
Fk Fk' q 2  (q1 ) T Fk' Fk' q 2
T
T
(q 2 ) T Fk' Fk' q 2
  max{1 , min{   ,  2 }} .
_
Now we consider the choose of the vector p c (  k ) . If the point
_
let
( xk  pc ( k ))  int( ) we simply
_
p c ( k )  pc ( k ) . Otherwise, p c (  k ) is a simple scaling of pc ( k ) obtained by a step-back
along it. In other words, we let
  min  ,
k
c
l  x u
k
i
where
 kc to be the stepsize along pc ( k ) to the boundary:

 (l  x k ) i (u  x k ) i 
,
max 

 
 p c ( k ) i pc ( k ) i 


k
i

if ( p c ( k )) i  0 

if ( p c ( k )) i  0 
_
The step p c (  k ) is given by
for some fixed
  0,1.
_

 pc ( k )
p c ( k )   k

 c p c ( k )
if
kc  1

othewise 

(11)
t in (9) is chosen in order to satisfy the following conditions
m (0)  mk ( p( k ))
(12)
 c ( p( k ))  k
 1 ,
_
mk (0)  mk ( p k ( k ))
for the given constant 1  0,1 and
mk ( p( k ))  mk ( ptr ( k ))
(13)
To force conditions (12), (13), we compute the scalar t in (9) according to the following strategy. If
The scalar
_
 c ( ptr ( k ))  1 , we simply set t  0 , i.e. we choose p( k )  ptr ( k )
4
as the candidate step.
5th World Congress on Industrial Process Tomography, Bergen, Norway
_
p( k ) so that x k  p c ( k ) is the point which lies on the segment from
Otherwise, we fix
_
_
x k  p tr ( k ) to xk  p c ( k ) and satisfies
 c ( p( k ))  1 .
In fact,
p( k ) is given by (9) setting t equal to the following scalar

t
zT u  w
u
(14)
2
where
_
_
_
u  Fk' ( p c ( k )  p tr ( k )),
z  ( Fk  Fk' p tr ( k )),
_

1 ' _
Fk ( p tr ( k )  1 Fk' ( p c ( k ) )) 2
2
2
A good agreement between the model function m k and objective function f is ensured for the
_
2
_
2
w  (( z T u ) 2  2 u ( Fk Fk' ( p tr ( k )  1 p c ( k )) 
2
T
1
following standard condition
 f ( p( k )) 
with the given constant
size
f ( xk )  f ( xk  p( k ))
 2
mk (0)  mk ( pk ( k ))
(15)
 2  (0,1) . If this condition is not met, we reject p( k )
and the trust-region
 k is adjusted be successive reduction so that p( k ) satisfies the accuracy requirement (15).
Algorithm of k-th iteration:
Let xk  int( ),  k  0,
1 ,  2 ,  1 ,  (0,1),  min  0,   (0,1),
_
be given
_
D k by (3). Set  k  min{  k ,  min } , set  k   k .
Compute the matrix
Repeat

Set  k   k .
ptr ( k ) for problem (5).
Compute the Cauchy step pc ( k ) by (7).
Find the solution
_
_
Compute p tr (  k ) and p c (  k ) by (9), (10), (11) respectively.
t such that satisfies (12), (13).

Set  k   1  k .
Find
2 Intel Trust-Region solvers performance
Intel MKL TR solvers are tested carefully on standard test sets for the unconstrained and constrained
optimization from CUTEr collection (CUTEr is testing environment for optimization and linear algebra
solvers. The package contains a collection of test problems, along with Fortran 77, Fortran 90/95 and
Matlab tools).
TR solvers are implemented with OpenMP support and can be used in multiprocessing mode (fig.1):
5
3rd World Congress on Industrial Process Tomography, Banff, Canada
Fig.1. Intel TR solvers scaling
TR solvers show excellent performance vs. VNI IMSL Fortran library v5.0 and TAO v1.8.1. On
benchmark tests TR give up to 10 times speed-up vs. IMSL (Fig.2) and up to 5 times speed-up vs.
TAO.
1000
IMSL
IMSL (with MKL)
MKL TR Solvers
Computation Time (sec)
Lower is better
100
10
1
100
200
400
600
800
1000
1200
1400
1600
0.1
0.01
Size
Fig.2. Intel TR solvers vs. IMSL
3 Intel Trust-Region solvers usage in mathematical problems of Geophysics
TR solvers were applied for solving the inverse problem of pore pressure evaluation (Madatov et al.
1995), namely – it was required to reconstruct some lythological values of strata (initial porosity,
constants of compaction, “source” term and specific surface) using real porosity and pore pressure
data.
The numerical solutions of this inverse problem (Lavrentiev, et al, 2003) were obtained with the
required precision and rapid convergence (see Fig.3, 4).
6
5th World Congress on Industrial Process Tomography, Bergen, Norway
Fig. 3. Inverse problem of pore pressure evaluation: Porosity curve reconstruction
Fig. 4. Inverse problem of pore pressure evaluation: Pore pressure curve reconstruction
4 REFERENCES
CONN A. R., GOULD N. I.M., TOINT P. L., (2000), Trust-Region Methods. SIAM Society for Industrial
& Applied Mathematics, Englewood Cliffs, New Jersey.
LAVRENTIEV M.M., AVDEEV A.V., LAVRENTIEV Jr. M.M., PRIIMENKO V.I., (2003) Inverse
Problems of Mathematical Physics. VSP Publ., The Netherlands, 272 p.
COLEMAN T.F., LI Y., (1996) An interior trust-region approach for minimization subject to bounds,
SIAM J. Optim. 6 (3) 418–445.
CUTEr, testing environment for optimization and linear algebra solvers, http://hsl.rl.ac.uk/cuter-www/
MADATOV A.G., HELLE H.B., and SEREDA V.I. (1995) , A modelling and inversion technique applied
to pore pressure evaluation, 57th EAEG Conference, Glasgow, May 29-June 2
INTEL MATH KERNEL LIBRARY 10.0, http://www.intel.com/cd/software/products/asmona/eng/perflib/mkl/index.htm
7