Optimization and feasibility problems with expensive
functions
Markus Kaiser1 , Alexander Thekale2 , Kathrin Klamroth1
and Philippe Toint3
1
University of Wuppertal, 2 University of Erlangen-Nuremberg, 3 University of Namur
16-th European Conference on Mathematics for Industry
Wuppertal, July 27, 2010
Outline
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem Formulation
Expensive Functions
General Expensive Feasibility and Optimization Problem
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Solution Methods
Basic Ideas
Outline of EFNES
Main Convergence Results
Numerical Results
Performance Profiles for EFNES
Numerical Results
EFNES
What are Expensive Functions?
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive functions are in general time-consuming or
cost-intensive black-boxes, e.g.
◮
simulations,
experiments or
◮
applications of algorithms to a testset.
◮
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
What are Expensive Functions?
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive functions are in general time-consuming or
cost-intensive black-boxes, e.g.
◮
simulations,
experiments or
◮
applications of algorithms to a testset.
◮
⇒ No derivative information available.
⇒ Avoid evaluations of expensive functions whenever possible.
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Expensive Feasibility and Optimization Problem
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
nonlinear feasibility
problem (EFP)
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
cI (x , u(x )) ≤ 0
cE (x , u(x )) = 0
x ∈ Rn
u:
cI , cE :
expensive function (sufficiently smooth)
twice continuously differentiable
EFNES
Convergence Results
Numerical Results
EFNES
Expensive Feasibility and Optimization Problem
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
nonlinear feasibility
problem (EFP)
nonlinear constrained optimization problem (EOP)
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
cI (x , u(x )) ≤ 0
min f (x , u(x ))
cE (x , u(x )) = 0
s.t. cI (x , u(x )) ≤ 0
x ∈ Rn
u:
f , cI , cE :
EFNES
Convergence Results
Numerical Results
EFNES
cE (x , u(x )) = 0
x ∈ Rn
expensive function (sufficiently smooth)
twice continuously differentiable
Reformulation: Sequence of Cheap Systems
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
nonlinear feasibility
problem (EFP)
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
cI (x , mku (x )) ≤ 0
cE (x , mku (x ))
=0
x ∈ Qk (xk , δk )
mku :
Qk :
model of the expensive function in iteration k
trust region Qk (xk , δk ) := {x ∈ Rn | kx − xk k ≤ δk }
EFNES
Convergence Results
Numerical Results
EFNES
Reformulation: Sequence of Cheap Systems
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
nonlinear feasibility
problem (EFP)
nonlinear constrained optimization problem (EOP)
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
cI (x , mku (x )) ≤ 0
cE (x , mku (x ))
=0
x ∈ Qk (xk , δk )
mku :
Qk :
min f (x , mku (x ))
s.t.
cI (x , mku (x ))
cE (x , mku (x ))
EFNES
Convergence Results
≤0
=0
x ∈ Qk (xk , δk )
model of the expensive function in iteration k
trust region Qk (xk , δk ) := {x ∈ Rn | kx − xk k ≤ δk }
Numerical Results
EFNES
Properties of the Model mku
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
◮
linear function
(n + 1 sample points X = {x̄0 , . . . , x̄n } ⊂ Qk , xk ∈ X )
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Optimization and
feasibility problems
with expensive
functions
Properties of the Model mku
Kaiser, Thekale,
Klamroth & Toint
◮
◮
linear function
(n + 1 sample points X = {x̄0 , . . . , x̄n } ⊂ Qk , xk ∈ X )
based on Lagrange basis polynomials
mku (x )
=
n
X
Li (x ) =
(
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
u(x̄i )Li (x )
i=0
with
Motivation and Problem
Formulation
EFNES
Convergence Results
Numerical Results
EFNES
1 if
x = x̄i
0 if
x ∈ X \ {x̄i }
Optimization and
feasibility problems
with expensive
functions
Properties of the Model mku
Kaiser, Thekale,
Klamroth & Toint
◮
◮
linear function
(n + 1 sample points X = {x̄0 , . . . , x̄n } ⊂ Qk , xk ∈ X )
based on Lagrange basis polynomials
mku (x )
=
n
X
Li (x ) =
◮
(
Basic Ideas
u(x̄i )Li (x )
max
Convergence Results
EFNES
1 if
x = x̄i
0 if
x ∈ X \ {x̄i }
max
EFNES
Numerical Results
Λ-poisedness of the sample points, i.e.
i=0,...,n
Expensive Functions
EFP and EOP
Solution methods
i=0
with
Motivation and Problem
Formulation
x∈Q(xk ,δk )
|Li (x )| ≤ Λ
can be verified and established if neccessary
Least-Squares Reformulation of (EFP)
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Find a (local) minimum of the unconstrained problem
Expensive Functions
EFP and EOP
Solution methods
minn
x∈R
f (x , u(x ))
=
1
2
2 kϑ(x , u(x ))k2
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
where
ϑ(x , u(x )) :=
cE (x , u(x ))
[cI (x , u(x ))]+
∈ Rp+q
is the vector of violations of the equations and inequalities.
Trust-Region Subproblem of (EFP)
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Find a (local) minimum of the constrained problem
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
min f (x , mku (x ))
x∈Rn
s.t. x ∈ Q(xk , δk )
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
where
EFNES
f (x , mku (x )) =
1
kϑ(x , mku (x ))k22
2
Q(xk , δk ) = {x ∈ Rn | kx − xk k∞ ≤ δk }
cE (x , mku (x ))
u
ϑ(x , mk (x )) :=
∈ Rp+q .
[cI (x , mku (x ))]+
Trust-Region Subproblem of (EFP)
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Find a (local) minimum of the constrained problem
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
min f (x , mku (x ))
x∈Rn
s.t. x ∈ Q(xk , δk )
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
where
EFNES
f (x , mku (x )) =
1
kϑ(x , mku (x ))k22
2
Q(xk , δk ) = {x ∈ Rn | kx − xk k∞ ≤ δk }
cE (x , mku (x ))
u
ϑ(x , mk (x )) :=
∈ Rp+q .
[cI (x , mku (x ))]+
Basic Ideas
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
◮
◮
expensive function
⇒ Λ-poised interpolation models
by Conn, Scheinberg, Vicente (2009)
⇒ conditional trust-region algorithm
by Conn, Gould, Toint (2000)
general nonlinear system of equations and inequalities
⇒ filter trust-region algorithm FILTRANE
by Gould, Toint (2007)
◮
general nonlinear constrained optimization problem
⇒ trust-region SQP-filter algorithm
by Fletcher, Gould, Leyffer, Toint, Wächter (2002)
◮
general nonlinear trust-region subproblem
⇒ subproblem solver needs to satisfy the
sufficient model decrease condition
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Basic Ideas
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
◮
◮
expensive function
⇒ Λ-poised interpolation models
by Conn, Scheinberg, Vicente (2009)
⇒ conditional trust-region algorithm
by Conn, Gould, Toint (2000)
general nonlinear system of equations and inequalities
⇒ filter trust-region algorithm FILTRANE
by Gould, Toint (2007)
◮
general nonlinear constrained optimization problem
⇒ trust-region SQP-filter algorithm
by Fletcher, Gould, Leyffer, Toint, Wächter (2002)
◮
general nonlinear trust-region subproblem
⇒ subproblem solver needs to satisfy the
sufficient model decrease condition
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Basic Ideas
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
◮
◮
expensive function
⇒ Λ-poised interpolation models
by Conn, Scheinberg, Vicente (2009)
⇒ conditional trust-region algorithm
by Conn, Gould, Toint (2000)
general nonlinear system of equations and inequalities
⇒ filter trust-region algorithm FILTRANE
by Gould, Toint (2007)
◮
general nonlinear constrained optimization problem
⇒ trust-region SQP-filter algorithm
by Fletcher, Gould, Leyffer, Toint, Wächter (2002)
◮
general nonlinear trust-region subproblem
⇒ subproblem solver needs to satisfy the
sufficient model decrease condition
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Basic Ideas
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
◮
◮
expensive function
⇒ Λ-poised interpolation models
by Conn, Scheinberg, Vicente (2009)
⇒ conditional trust-region algorithm
by Conn, Gould, Toint (2000)
general nonlinear system of equations and inequalities
⇒ filter trust-region algorithm FILTRANE
by Gould, Toint (2007)
◮
general nonlinear constrained optimization problem
⇒ trust-region SQP-filter algorithm
by Fletcher, Gould, Leyffer, Toint, Wächter (2002)
◮
general nonlinear trust-region subproblem
⇒ subproblem solver needs to satisfy the
sufficient model decrease condition
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Step 0: Initialization
Motivation and Problem
Formulation
Step 1: Stopping test
Step 2: Trial point determination
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Step 3: Acceptance of the trial point
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
Numerical Results
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Input initial values for parameters and compute mk0 and
c0 = c(x0 , u(x0 )).
Motivation and Problem
Formulation
Step 1: Stopping test
Solution methods
Expensive Functions
EFP and EOP
Basic Ideas
EFNES
Step 2: Trial point determination
Convergence Results
Numerical Results
EFNES
Step 3: Acceptance of the trial point
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
Stop if ϑ(xk , u(xk )) = 0 or k∇f (xk )k2 < ε for a valid mku .
If mku is not valid improve the model.
EFP and EOP
Solution methods
Basic Ideas
EFNES
Step 2: Trial point determination
Convergence Results
Numerical Results
Step 3: Acceptance of the trial point
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Try to compute xk+ which provides a sufficent model decrease.
If this is not possible improve the model and go to Step 1.
Step 3: Acceptance of the trial point
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
EFNES
Convergence Results
Numerical Results
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Step 3: Acceptance of the trial point
If the ratio of predicted (by mku ) and actual (requires evaluation of
u(xk+ )) improvement is "good enough" accept the trial point.
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
EFNES
Convergence Results
Numerical Results
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Step 3: Acceptance of the trial point
EFNES
Convergence Results
Numerical Results
Step 4: Model improvement
If neccessary improve mku (dependent on the quality of its prediction).
Consider arising alternatives for the trial point.
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Step 3: Acceptance of the trial point
EFNES
Convergence Results
Numerical Results
Step 4: Model improvement
Step 5: Selection of new iterate
Choose the best candidate among xk+ and the points generated during
the improvement of the model as final trial point xk∗ .
Step 6: Acceptance test
Step 7: Trust region update
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Step 3: Acceptance of the trial point
EFNES
Convergence Results
Numerical Results
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Decide if xk∗ is accepted as xk+1 dependent on the accuracy of the
prediction by the model in xk∗ and some additional criterion.
Step 7: Trust region update
EFNES
Outline of EFNES
Optimization and
feasibility problems
with expensive
functions
Step 0: Initialization
Kaiser, Thekale,
Klamroth & Toint
Step 1: Stopping test
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Step 2: Trial point determination
Solution methods
Basic Ideas
Step 3: Acceptance of the trial point
EFNES
Convergence Results
Numerical Results
Step 4: Model improvement
Step 5: Selection of new iterate
Step 6: Acceptance test
Step 7: Trust region update
Update the radius of the trust-region according to the accuracy of the
model if the trust-region was used in the current iteration.
EFNES
Assumptions 1 - 6
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive Functions
1. cI , cE , u twice continuously differentiable, bounded from
above
2. iterates remain in bounded domain
3. mku twice continuously differentiable
4. error bounds on model mku
5. check and guarantee model validity, whenever necessary
6. the solver for the TR-subproblem achieves a sufficient model
decrease
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Global Convergence of EFNES
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Theorem
Suppose that Assumptions 1-6 hold. Then either
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
k∇f (xk , u(xk ))k2 = 0
Basic Ideas
EFNES
Convergence Results
for some finite k or
Numerical Results
EFNES
lim k∇f (xk , u(xk ))k2 = 0.
k→∞
Global Convergence of EFNES
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Theorem
Suppose that Assumptions 1-6 hold. Then either
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
k∇f (xk , u(xk ))k2 = 0
Basic Ideas
EFNES
Convergence Results
for some finite k or
Numerical Results
EFNES
lim k∇f (xk , u(xk ))k2 = 0.
k→∞
Moreover, if infinitely many values are added to the filter, then we
have that
lim kϑ(xk , u(xk ))k2 = 0.
k→∞
Test Environment
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Testset:
◮
◮
54 small and medium sized test problems from the CUTEr
library
randomly selected parts are treated as expensive functions
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Test Environment
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Testset:
◮
54 small and medium sized test problems from the CUTEr
library
randomly selected parts are treated as expensive functions
Compared methods:
◮
◮
◮
◮
◮
Matlab implementation of EFNES
Matlabfunctions fmincon and patternsearch (from the
optimization- and direct-search toolbox, respectively)
TRESNEI (Filter-Trust-Region based solver for nonlinear
systems of (in-)equalities by Morini and Porcelli, 2009)
FILTRANE (part of the Galahad toolbox by Gould, Orban
and Toint, 2007)
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Performance Profile Comparing Computation
Time
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Performance Profile Comparing the Number of
Evaluations of the Expensive Function
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Convergence Results
Numerical Results
EFNES
Literature
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Andrew R. Conn, Nicholas I. M. Gould, and Philippe L. Toint.
Trust-Region Methods.
MPS-SIAM Series on Optimization. Society for Industrial Mathematics, 2000.
Motivation and Problem
Formulation
Expensive Functions
Andrew R. Conn, Katya Scheinberg, and Luís N. Vicente.
Introduction to Derivative-Free Optimization.
MPS-SIAM Series on Optimization. Society for Industrial Mathematics, 2009.
EFP and EOP
Solution methods
Basic Ideas
Roger Fletcher, Nicholas I. M. Gould, Sven Leyffer, Philippe L. Toint, and Andreas Wächter.
Global convergence of a trust-region SQP-filter algorithm for general nonlinear programming.
SIAM Journal on Optimization, 13(3):635–659, 2002.
Nicholas I. M. Gould and Philippe L. Toint.
FILTRANE, a Fortran 95 filter-trust-region package for solving nonlinear least-squares and nonlinear feasibility
problems.
ACM Transactions on Mathematical Software, 33(1):3–25, 2007.
Markus Kaiser, Kathrin Klamroth, and Alexander Thekale.
Test examples for nonlinear feasibility problems with expensive functions.
Technical Report, University of Wuppertal , BUW-AMNA-OPAP 10/09.
Markus Kaiser and Alexander Thekale.
Solving nonlinear feasibility problems with expensive functions.
Technical Report, University of Wuppertal , BUW-AMNA-OPAP 10/03.
Benedetta Morini and Margherita Porcelli.
TRESNEI, a Matlab trust-region solver for systems of nonlinear equalities and inequalities.
Technical report, Dipartimento di Energetica, Università di Firenze, 2009.
EFNES
Convergence Results
Numerical Results
EFNES
Optimization and
feasibility problems
with expensive
functions
Kaiser, Thekale,
Klamroth & Toint
Motivation and Problem
Formulation
Expensive Functions
EFP and EOP
Solution methods
Basic Ideas
EFNES
Thank you for your
attention!
Markus Kaiser
e-Mail: [email protected]
Convergence Results
Numerical Results
EFNES