Global Optimization
5. Constrained Minimization Conditions
Richárd Molnár-Szipai
Budapest University of Technology and Economics
2017 Spring
Richárd Molnár-Szipai
Global Optimization 5.
Constraints
General nonlinear programming problems:
min f
h1 (x) = 0
.
.
.
hm (x) = 0
(x)
g1 (x) ≤ 0
.
.
.
gp (x) ≤ 0
x ∈ Ω ⊂ Rn
The set constraint
x∈Ω
is not emphasized in the following.
Richárd Molnár-Szipai
Global Optimization 5.
Constraints
An inequality constraint
point
x
if
gi (x) = 0,
gi (x) ≤ 0
and
is said to be
inactive if gi (x) < 0.
active at a feasible
Inactive constraints have no inuence on the feasible neighborhood
of
x.
If we knew in advance which constraints are active at the optimum
x∗ ,
then we could ignore the inactive ones, and treat the active
ones with equality. Then
x∗
would be a local minimum.
This suggests that a big part of the theory can be derived by
considering equality constraints alone.
Richárd Molnár-Szipai
Global Optimization 5.
Constraints
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
The set of equality constraints
h1 (x) = 0, . . . , hm (x) = 0
dene a
hypersurface.
If the constraints are regular in a sense, then this hypersurface
will have dimension
If the functions
n − m.
hi ∈ C 1 ,
then the surface is said to be
Richárd Molnár-Szipai
Global Optimization 5.
smooth.
Tangent Plane
A point on this smooth surface has a
Formally, the tangent plane at
{x 0 (t ∗ ) : x(t)
x∗
tangent plane.
is dened as
is a curve on
S, x(t ∗ ) = x∗ }
In our case, under mild conditions, it will be equal to
M = {y : ∇hi (x∗ )> y = 0, ∀i}
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Denition
A point
x∗
satisfying the constraint
h(x∗ ) = 0
is said to be a
regular point of the constraint, if the gradient vectors
∇h1 (x∗ ), . . . , ∇hm (x∗ ) are linearly independent.
Example
h(x) = Ax + b,
independent of x.
If
x)
(∇hi (
is the
then regularity is equivalent to
i th
row of
r (A) = m,
A.)
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Theorem
At a regular point x∗ of the surface S dened by h(x) = 0 the
tangent plane is equal to
M = {y : ∇h(x∗ )> y = 0}
Proof
Let T be the tangent plane at x∗ .
It is clear that T ⊆ M (whether x∗ is regular or not).
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Proof
To prove M ⊂ T , take y ∈ M , we have to show a curve on S
passing through x∗ with derivative y.
Consider the equations
h(x∗ + t y + ∇h(x∗ )> u(t)) = 0
where for xed t we consider u(t) to be the unknown.
This is a nonlinear system of m equations and m unknowns,
parameterized continuously by t .
At t = 0, the solution is u(0) = 0.
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Proof
The Jacobian matrix with respect to u at t = 0 is
∇h(x∗ )∇h(x∗ )>
which is nonsingular if x∗ is a regular point.
By the implicit function theorem there is a continuously
dierentiable solution u(t) in some region −ε ≤ t ≤ ε.
The curve x(t) = x∗ + t y + ∇h(x∗ )> u(t) is thus a curve on S .
Richárd Molnár-Szipai
Global Optimization 5.
Tangent Plane
Proof.
The curve
x(t) = x∗ + t y + ∇h(x∗ )> u(t)
is thus a curve on
Dierentiate the system of equations with respect to
0=
We have
d
dt
t=0
∇h(x∗ )y = 0
∇h(x∗ )∇h(x∗ )>
And thus
h(x(t))
(as
t
at
t = 0:
= ∇h(x∗ ) y + ∇h(x∗ )> u0 (0)
y ∈ M ).
is nonsingular, therefore
u0 (0) = 0.
x0 (0) = y.
Richárd Molnár-Szipai
Global Optimization 5.
S.
Tangent Plane
Note that regularity is not a condition on the constraint surface,
but on how it is represented.
T
is independent of the representation,
M
is not.
Example
Let
h(x1 , x2 ) = x1 .
Then the constraint
h(x) = 0
is the
x2
axis, and
every point on the axis is regular.
Let
h(x1 , x2 ) = x12 .
Then the constraint
h(x) = 0 is again the x2
M = R2 .
axis, but no point on the axis is regular. Indeed,
Richárd Molnár-Szipai
Global Optimization 5.
First-Order Necessary Conditions (Equality Constraints)
Theorem
Let x∗ be a regular point of the constraints h(x) = 0 and a local
extremum point of f subject to these constraints. Then all y
satisfying
∇h(x∗ )> y = 0
must also satisfy
∇f (x∗ )> y = 0
Richárd Molnár-Szipai
Global Optimization 5.
First-Order Necessary Conditions (Equality Constraints)
Proof.
Let
y
be a vector in the tangent plane at
point, there is an
x(t)
y.
h(x(t)) = 0
smooth curve on
derivative
That is,
Since
x∗
and so
for
S
x∗ .
As
x∗
is a regular
passing through
x∗
with
−ε ≤ t ≤ ε, x(0) = x∗ , x0 (0) = y.
is a constrained local extremum of
d
f (x(t))
dt
t=0
Richárd Molnár-Szipai
f,
it also is along
= ∇f (x∗ )> y = 0
Global Optimization 5.
x,
First-Order Necessary Conditions (Equality Constraints)
Theorem
Let x∗ be a regular point of the constraints h(x) = 0 and a local
extremum point of f subject to these constraints. Then there is a
λ ∈ Rm such that
∇f (x∗ ) + λ> ∇h(x∗ ) = 0
Richárd Molnár-Szipai
Global Optimization 5.
First-Order Necessary Conditions (Equality Constraints)
Proof.
Consider the following linear program:
max ∇f
(x∗ )y
∇h(x∗ )y = 0
By the previous theorem, the optimum value (and the only possible
value) of this problem is 0.
By the duality theorem of linear programming, the dual problem is
feasible, thus
λ> ∇h(x∗ ) = ∇f (x∗ )
is solvable.
Richárd Molnár-Szipai
Global Optimization 5.
First-Order Necessary Conditions (Equality Constraints)
These
λ
coecients are called
Lagrange multipliers.
The following equation system has
n+m
n+m
variables (
x∗ , λ),
(generally nonlinear) equations:
∇f (x∗ ) + λ> ∇h(x∗ ) = 0
h(x∗ ) = 0
This can also be understood as
∇l(x, λ) = 0,
where
l(x, λ) = f (x) + λ> h(x)
is the
Lagrangian associated with the constrained problem.
Richárd Molnár-Szipai
Global Optimization 5.
and
Examples
Example
Consider the problem
min x1 x2
+ x2 x3 + x1 x3
x1 + x2 + x3 = 3
The Lagrangian is
l(x, λ) = x1 x2 + x2 x3 + x1 x3 + λ(x1 + x2 + x3 − 3)
Richárd Molnár-Szipai
Global Optimization 5.
Examples
Example
∇l(x, λ) = 0
gives the following system:
x2 + x3 + λ = 0
x1 + x3 + λ = 0
x1 + x2 + λ = 0
x1 + x2 + x3 = 3
Which can be solved to give
x1 = x2 = x3 = 1, λ = −2.
Richárd Molnár-Szipai
Global Optimization 5.
Examples Maximum volume
Example
Consider the problem
max xyz
xy + xz + yz = c/2
The Lagrangian is
l(x, λ) = xyz + λ(xy + xz + yz − c/2)
Richárd Molnár-Szipai
Global Optimization 5.
Examples Maximum volume
Example
∇l(x, λ) = 0
gives the following system:
yz + λy + λz = 0
xz + λx + λz = 0
xy + λx + λy = 0
xy + xz + yz = c/2
It is easily established that none of the variables can be 0.
Richárd Molnár-Szipai
Global Optimization 5.
Examples Maximum volume
Example
Multiply the rst equation by
x,
the second by
y,
and subtract:
λ(x − y )z = 0
which gives
x = y.
Similarly
y =z
can be derived from the second
and the third equations.
Then 3
· x 2 = c/2
gives
x =y =z =
Richárd Molnár-Szipai
p
c/6.
Global Optimization 5.
Examples Entropy
Example
Consider the problem
max −
n
X
pi log pi
i=1
n
X
pi = 1
i=1
n
X
x i pi = m
i=1
pi ≥ 0
Richárd Molnár-Szipai
Global Optimization 5.
Examples Entropy
Example
Ignoring the nonnegativity constraints, the Lagrangian is
n
X
(−pi log pi + λpi + µxi pi ) − λ − µm
i=1
Dierentiating by
pi
gives
− log pi − 1 + λ + µxi = 0
pi = e −1+λ+µxi
pi > 0,
and we must select
λ
and
µ
to satisfy the 2 equality
constraints.
Richárd Molnár-Szipai
Global Optimization 5.
Examples Portfolio design
Example
Consider the problem
min
n
X
wi σi,j wj
i,j=1
n
X
w i ri = r
i=1
n
X
wi = 1
i=1
(wi ≥ 0)
Richárd Molnár-Szipai
Global Optimization 5.
Examples Portfolio design
Example
Ignoring the nonnegativity constraints, the Lagrangian is
n
X
wi σi,j wj + λ
i,j=1
n
X
!
wi ri − r
+µ
i=1
n
X
i=1
Dierentiating gives a system of linear equations:
n
X
2wj σi,j + λri + µ = 0
j=1
n
X
wi ri = r
i=1
n
X
wi = 1
i=1
Richárd Molnár-Szipai
Global Optimization 5.
!
wi − 1
Examples
There are many more examples of nonlinear programming in
Physics (a solution can often be described as a process
minimizing some kind of energy)
Industrial operations (optimal control)
Business decision making (prots, consumer surplus)
Data analysis (average error)
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Throughout this section
f , h ∈ C 2.
Theorem
Suppose that x∗ is a local minimum of f subject to h(x) = 0 and
that x∗ is a regular point of these constraints. Then there is a
λ ∈ Rm such that
∇f (x∗ ) + λ> ∇h(x∗ ) = 0
and for all y ∈ M (the tangent plane at x∗ ):
y> L(x∗ )y = y> F (x∗ ) + λ> H(x∗ ) y ≥ 0
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Proof.
For an
x ∈ C2
curve on
S
with
x(0) = x∗
we have
d2
= x0 (0)> F (x∗ )x0 (0) + ∇f (x∗ )x00 (0)
0 ≤
f
(
x
(t))
dt 2
t=0
Dierentiating
λ> h(x(t)) = 0
twice gives
x0 (0)> λ> H(x∗ )x0 (0) + λ> ∇h(x∗ )x00 (0) = 0
Adding them gives the desired result.
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Theorem
Suppose there is a point x∗ satisfying h(x∗ ) = 0 and λ ∈ Rm such
that
∇f (x∗ ) + λ> ∇h(x∗ ) = 0
Suppose also that the matrix L(x∗ ) = F (x∗ ) + λ> H(x∗ ) is positive
denite on M = {y : ∇h(x∗ )y = 0}.
Then x∗ is a strict local minimum of f subject to h(x) = 0.
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Proof
Assuming x∗ is not a strict relative minimum point, there exists
yk → x∗ on S such that for each k , f (yk ) ≤ f (x∗ ).
We can write it in the form yk = x∗ + δk sk , where |sk | = 1, and
δk > 0.
Clearly δk → 0, and we might assume that sk → s∗ for some
|s∗ | = 1.
h(x )
= 0.
Note that ∇h(x∗ )s∗ = limk→∞ h(yk )−
δk
∗
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Proof
By Taylor's theorem, we have for each j
0
= hj (yk ) = hj (x∗ ) + δk ∇hj (x∗ )sk +
δk2
2
2
s>
k ∇ hj (ηj )sk
and
0
≥ f (yk ) − f (x∗ ) = δk ∇f (x∗ )sk +
δk2
2
2
s>
k ∇ f (η0 )sk
multiplying the hj equations by λj and adding them together with
the last inequality eliminates the rst-order terms.
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Proof.
And we get
0
≥
δk2
2

∇2 f (η0 ) +
s>
k
which leads to a contradiction as
Richárd Molnár-Szipai
n
X

λj ∇2 hj (ηj ) sk
j=1
k → ∞.
Global Optimization 5.
Second-order conditions
Example
Consider the problem
min x1 x2
+ x2 x3 + x1 x3
x1 + x2 + x3 = 3
We have seen that
x1 = x2 = x3 = 1, λ = −2
satisfy the rst-order
necessary conditions.
Richárd Molnár-Szipai
Global Optimization 5.
Second-order conditions
Example
The matrix
L = F + λH = F in

L=
this case is (independent of
0
1
1

1
0
1

1
1
0
which is neither positive nor negative denite.
The tangent space is
M = {y : y1 + y2 + y3 = 0}.
Richárd Molnár-Szipai
Global Optimization 5.
x)
Second-order conditions
Example
y:
With such
y> Ly = 2y1 y2 + 2y1 y3 + 2y2 y3 = y1 (y2 + y3 )+
+ y2 (y1 + y3 ) + y3 (y1 + y2 ) = −y12 − y22 − y32 ≤ 0
Therefore
L
is negative denite on
M,
our solution is a local
maximum.
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
L
restricted to the subspace
M
plays a similar role to
H
in the
unconstrained case.
In the unconstrained case the eigenvalues of
H
determine the
convergence of algorithms, so we will investigate the eigenvalues of
L
on
M.
We dene the restriction to
projection of
Ly
to
M
as
LM y
being the orthogonal
M.
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Using the usual denition,
such that
LM y = λy.
In terms of
L
y
it means that
a vector orthogonal to
is an eigenvector of
Ly
e1 , . . . , en−m
of
if there is a
can be written as the sum of
λy
M.
And let
LM , introduce an orthonormal
E be the matrix consisting of
these column vectors.
Richárd Molnár-Szipai
Global Optimization 5.
λ
and
M.
To get a matrix representation for
basis
LM ,
Eigenvalues in Tangent Subspace
Then any
of
L
y∈M
can be written as
Ez . LEz
represents the action
on such a vector, and projecting it back to
M
is
E > LEz
(in
the aforementioned basis of course).
This way, we can investigate the eigenvalues of
independent of the particular basis
Richárd Molnár-Szipai
E > LE ,
E.
Global Optimization 5.
which is
Eigenvalues in Tangent Subspace
Example
In the last example we arrived to
on the subspace

0
1
1

L=
1
0
1

1
1
0
M = {y : y1 + y2 + y3 = 0}.
An orthonormal basis is
1
e1 = √ (1, 0, −1)
2
1
e2 = √ (1, −2, 1)
6
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Example
This gives, upon expansion
>
E LE =
and hence
L
restricted to
M
−1
0
0
−1
acts like
Richárd Molnár-Szipai
−I .
Global Optimization 5.
Eigenvalues in Tangent Subspace
Example
Consider the problem
min / max x1
1
2
+ x22 + x2 x3 + 2x32
(x12 + x22 + x32 ) = 1
The rst-order necessary conditions are
1
2x2
+ λx1 = 0
+ x3 + λx2 = 0
x2 + 4x3 + λx3 = 0
x12 + x22 + x32 = 2
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Example
One solution to this is
(1, 0, 0)
with
λ = −1.
−1
0
0

0
1
1

0
1
3
The Lagrangian matrix here is

L=
and the tangent space is
M = {y : y1 = 0}.
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Example
An orthonormal basis of
M
is
(0, 1, 0)
and
1
1
1
3
(0, 0, 1),
with which we
get
>
E LE =
The eigenvalues of this matrix are:
LE − λI ) = (1 − λ)(3 − λ) − 1 = λ2 − 4λ + 2 = 0
√
gives λ = 2 ±
2, hence LM is positive denite.
det(E
which
>
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Another approach is that of
bordered Hessians.
Let us rst characterize all vectors orthogonal to
M = {y : ∇hy = 0}.
z
is orthogonal to
M,
if
z> y = 0
for all
y ∈ M.
It is not hard to show that it will happen if, and only if
for some
w∈
Rm . (z> y
=0
z = ∇h> w
is trivial, the other direction follows
from the duality theorem).
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
LM , it needs to satisfy two
Lx = λx + z, where z is orthogonal to M :
To characterize an eigenvector of
conditions:
x ∈ M,
and
∇hx = 0
Lx = λx + ∇h> w
which is a linear system of equations in
w
and
x.
It has a nonzero solution if and only if the determinant of the
matrix is zero:
det
0
∇h
>
−∇h
L − λI
Richárd Molnár-Szipai
= p(λ) = 0
Global Optimization 5.
Eigenvalues in Tangent Subspace
This polynomial
p(λ)
is of degree
n−m
Example
Taking the last example, we get

0
1
 −1 −1 − λ
p(λ) = det 

0
0
0
0
which has the same determinant:
Richárd Molnár-Szipai
1
0
0
0
0
−λ
1
1
3
−λ
(1 − λ)(3 − λ) − 1.
Global Optimization 5.




Eigenvalues in Tangent Subspace
Having the same characteristic polynomial, we can hope that tests
for positive deniteness can be extended as well.
Let the bordered Hessian be
B=
0
∇h
∇h> L
(Note the sign change in the lower left.)
Richárd Molnár-Szipai
Global Optimization 5.
Eigenvalues in Tangent Subspace
Theorem
The matrix L is positive denite on the subspace M if and only if
the last n − m principal minors of B have sign (−1)m .
Example

0
1
 −1 −1
B=
 0
0
0
The last 3
−2,
−1
0
0
0
0
0
1
1
1
3




principal minors (top left 3x3 and 4x4) are
which both have sign
(−1)1 ,
Richárd Molnár-Szipai
hence
L
−1
and
is positive denite on
Global Optimization 5.
M.
Sensitivity
The Lagrange multipliers associated with a constrained
minimization problem have an interpretation as prices, similar to
linear programming.
Suppose the problem
min f
(x)
h(x) = 0
has a solution at the point
x∗ ,
which is a regular point of the
constraints, with Lagrange multiplier
Richárd Molnár-Szipai
λ.
Global Optimization 5.
Sensitivity
Then for a small range of
c
near
0,
min f
the problem
(x)
h(x) = c
will have a solution point
The change of
f (x(c))
x(c)
near
0
near
x∗ .
can be characterized by
following theorem shows
Richárd Molnár-Szipai
Global Optimization 5.
λ,
as the
Sensitivity
Theorem
Let f , h ∈ C 2 , and consider the family of problems
min f
(x)
h(x) = c
Suppose that for c = 0 there is a local minimum x∗ that is a
regular point, with Lagrange multiplier λ, that satises the
second-order suciency conditions.
Then for every c in a region containing 0 there is an x(c),
depending continuously on c such that x(0) = x∗ and such that
x(c) is a local minimum of the corresponding problem.
Furthermore
∇c f (x(c))|c=0 = −λ>
Richárd Molnár-Szipai
Global Optimization 5.
Sensitivity
Proof
Consider the system of equations
∇f (x) + λ> ∇h(x) = 0
h(x) = c
x∗ , λ is a solution to this system when c = 0. The Jacobian of the
system there is
L(x∗ ) ∇h(x∗ )>
∇h(x∗ )
0
As x∗ is a regular point, and L(x∗ ) is positive denite on M , this
matrix is invertible. We can use the implicit function theorem to
show that there is a solution x(c), λ(c) to the system (in C 1 ).
Richárd Molnár-Szipai
Global Optimization 5.
Sensitivity
Proof
By the chain rule we have
∇c f (x(c))|c=0 = ∇x f (x∗ )∇c x(0)
and
∇c h(x(c))|c=0 = ∇x h(x∗ )∇c x(0)
As h(x(c)) = c, the second one equals I .
Richárd Molnár-Szipai
Global Optimization 5.
Sensitivity
Proof
We had
∇f (x) = −λ> ∇h(x)
multiplied by ∇c x(0) we get
∇c f (x(c))|c=0 = ∇x f (x∗ )∇c x(0) = −λ> ∇x h(x∗ )∇c x(0) = −λ>
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Now consider problems of the form
min f
(x)
h(x) = 0
g(x) ≤ 0
Denition
x∗ is regular if the gradient vectors of the equality
∇hi (x∗ ), and the active inequality constraints ∇gj (x∗ )
A feasible point
constraints
are linearly independent.
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Theorem (KarushKuhnTucker)
Let x∗ be a relative minimum point for the problem
min f
(x),
h(x) = 0,
g(x) ≤ 0
and suppose that it is a regular point.
Then there is λ ∈ Rm and µ ∈ Rp , µ ≥ 0 such that
∇f (x∗ ) + λ> ∇h(x∗ ) + µ> ∇g(x∗ ) = 0
µ> g(x∗ ) = 0
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Proof
Note that as µ ≥ 0 and g(x∗ ) ≤ 0, the second equality can only
hold, if for every index either µi = 0, or gi (x∗ ) = 0, also called
complementary slackness.
In other words, only active constraints can appear in the rst
equation.
Since x∗ is a relative minimum also if we restrict the active
constraints to zero, we know that there exist λ and µ that satisfy
the rst equation, without the sign requirements.
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Proof
Suppose that µk < 0 for some k ∈ J . Let S and M be the surface
and tangent plane dened by all the other active constraints at x∗ .
By regularity, there is y ∈ M such that ∇gk (x∗ )y < 0. Let x(t) be
a curve on S passing through x∗ at t = 0 with x0 (0) = y.
Then for small t ≥ 0, x(t) is feasible, and
df
(x(t))
= ∇f (x∗ )y < 0
dt
t=0
which contradicts the minimality of x∗ .
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Example
Consider the problem
2
min 2x1
+ 2x1 x2 + x22 − 10x1 − 10x2
x12 + x22 ≤ 5
3x1
+ x2 ≤ 6
The rst order necessary conditions, in addition to the constraints:
4x1
2x1
+ 2x2 − 10 + 2µ1 x1 + 3µ2 = 0
+ 2x2 − 10 + 2µ1 x2 + µ2 = 0
µ1 ≥ 0,
µ1 (x12
+ x22
µ2 ≥ 0
− 5) = 0
µ2 (3x1 + x2 − 6) = 0
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Example
To nd the solutions, we turn on and o various constraints.
For example, assume that the rst constraint is active, and the
second is inactive.
4x1
+ 2x2 − 10 + 2µ1 x1 = 0
2x1
+ 2x2 − 10 + 2µ1 x2 = 0
µ1 ≥ 0,
µ2 = 0
x12 + x22 = 5
which has the solution
3x1
x1 = 1, x2 = 2, µ1 = 1.
This satises
+ x2 ≤ 6.
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Theorem
Suppose that f , g , h ∈ C 2 and x∗ is a regular point of the problem.
If x∗ is a relative minimum point, then there is a λ, and µ ≥ 0 such
that
∇f (x∗ ) + λ> ∇h(x∗ ) + µ> ∇g(x∗ ) = 0
µ> g(x∗ ) = 0
and L(x∗ ) = F (x∗ ) + λ> H(x∗ ) + µ> G (x∗ ) is positive semidenite
on the tangent subspace of the active constraints at x∗ .
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Theorem
Let f , g , h ∈ C 2 . Suppose that x∗ is feasible, there exist λ, µ such
that
µ≥0
µ g(x∗ ) = 0
>
∇f (x∗ ) + λ> ∇h(x∗ ) + µ> ∇g(x∗ ) = 0
the Hessian L(x∗ ) is positive denite on the subspace
M 0 = {y : ∇h(x∗ )y = 0, ∇gj (x∗ )y = 0 for all j ∈ J}
where
J = {j : gj (x∗ ) = 0, µj > 0}
then x∗ is a local minimum.
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Theorem (Sensitivity)
Let f , h ∈ C 2 , and consider the family of problems
min f
(x),
h(x) = c,
g(x) ≤ d.
Suppose that for c = 0, d = 0 there is a local minimum x∗ that is
a regular point, with Lagrange multipliers λ, µ ≥ 0 that satises
the second-order suciency conditions. Assume that no active
inequality constraint is degenerate.
Then for every c, d in a region containing 0 there is a solution
x(c, d), depending continuously on c, d such that x(0, 0) = x∗ and
such that x(c, d) is a local minimum of the corresponding problem.
Richárd Molnár-Szipai
Global Optimization 5.
Inequality constraints
Theorem (Sensitivity (cont'd))
Furthermore
∇c f (x(c, d))|0,0 = −λ>
∇d f (x(c, d))|0,0 = −µ>
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
We can get some derivative-free conditions for convex problems.
min f
(x)
h(x) = 0, g(x) ≤ 0
x∈Ω
So
f , gi
and
hi (x) = 0
Ω
should be convex.
is convex exactly if
hi
Richárd Molnár-Szipai
is linear.
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
First we consider equality constraints:
min f
(x),
h(x) = 0,
x∈Ω
Denition
An ane function
h
Ω if
{y : |y| < ε} for
is regular with respect to
C = {y : h(x) = y, x ∈ Ω}
contains
Richárd Molnár-Szipai
Global Optimization 5.
some
ε > 0.
Zero-order conditions and Lagrange multipliers
The primal function is dened by varying the right hand side of
the constraint:
Denition
The
primal function associated with the problem is
w (y) = inf {f (x) : h(x) = y, x ∈ Ω}
dened for all
y ∈ C.
Proposition
Suppose that f and Ω are convex, h is ane. Then the primal
function is convex.
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proof.
w (αy1 + (1 − α)y2 ) = inf {f (x) : h(x) = αy1 + (1 − α)y2 }
≤ inf {f (x) : x = αx1 + (1 − α)x2 , h(x1 ) = y1 , h(x2 ) = y2 }
≤ α inf {f (x1 ) : h(x1 ) = y1 } + (1 − α) inf {f (x2 ) : h(x2 ) = y2 }
= αw (y1 ) + (1 − α)w (y2 )
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proposition
Assume that Ω and f are convex, and h is an ane function.
Assume that h is regular with respect to Ω.
If x∗ is a solution, then there is λ such that x∗ solves the
Lagrangian problem
min f
(x) + λ> h(x),
Richárd Molnár-Szipai
x∈Ω
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proof
Let f ∗ = f (x∗ ). Dene the following sets
A = {(r , y) : r ≥ w (y), y ∈ C }
B = {(r , y) : r ≤ f ∗ , y = 0}
According to the separating hyperplane theorem, there is (s, λ) and
c such that
sr + λ> y ≥ c
(r , y) ∈ A
sr + λ y ≤ c
(r , y) ∈ B
>
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proof
From the inequality on B we get that s ≥ 0.
Suppose that s = 0. Then λ 6= 0.
From the inequality on B we get c = 0, as the hyperplane must
include (f ∗ , 0).
Thus the inequality on A says λ> y ≥ 0 for all y ∈ C . But this
contradicts the regularity of C .
Thus s > 0.
Without loss of generality, we might assume s = 1.
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proof
Then for any x ∈ Ω we have (f (x), h(x)) ∈ A and (f (x∗ ), 0) ∈ B .
From the separation inequalities we get
f (x) + λ> h(x) ≥ f (x∗ ) = f (x∗ ) + λ> h(x∗ )
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Example
Consider the problem of minimizing the length of the diagonal of a
rectangle subject to the perimeter being of length 4:
min
1
2
(x12 + x22 )
x1 + x2 = 2
x1 ≥ 0, x2 ≥ 0
The objective function is convex,
Ω = {x1 ≥ 0, x2 ≥ 0}
and we have one ane constraint.
Richárd Molnár-Szipai
Global Optimization 5.
is convex,
Zero-order conditions and Lagrange multipliers
Example
The solution using the rst order conditions is
x1 = x2 = 1
λ = −1.
The corresponding Lagrangian problem is
min
1
2
(x12 + x22 ) − (x1 + x2 − 2)
x1 ≥ 0, x2 ≥ 0
Richárd Molnár-Szipai
Global Optimization 5.
with
Zero-order conditions and Lagrange multipliers
Let us take a look at the case of inequality constraints:
min f
where
gi
(x),
g(x) ≤ 0,
x∈Ω
are convex functions.
Denition
The constraints satisfy the
Slater condition with respect to Ω, if for
D = {z ∈ Rp : g(x) ≤ z, x ∈ Ω}
there is a
z∈D
such that
z<0
Richárd Molnár-Szipai
(coordinatewise).
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Denition
The
primal function associated with the problem is
w (z) = inf {f (x) : g(x) ≤ z, x ∈ Ω}
Proposition
Suppose that Ω, f , and g are convex. Then so is the primal
function w .
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proposition
Assume that Ω, f , and g are convex. Also assume that there is a
point x1 ∈ Ω such that g(x1 ) < 0.
Then if x∗ is a solution, then there is µ ≥ 0 such that x∗ solves the
Lagrangian problem
min f
(x) + µ> g(x),
x∈Ω
Furthermore, µ> g(x) = 0.
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proposition
min f
(x),
h(x) = 0,
g(x) ≤ 0,
x∈Ω
Assume that Ω, f , and g are convex, h is ane. Also assume that
h is regular, and that there is a point x1 ∈ Ω such that h(x1 ) = 0,
and g(x1 ) < 0.
Then if x∗ is a solution, then there is λ and µ ≥ 0 such that x∗
solves the Lagrangian problem
min f
(x) + λ> h(x) + µ> g(x),
x∈Ω
Furthermore, µ> g(x) = 0.
Richárd Molnár-Szipai
Global Optimization 5.
Zero-order conditions and Lagrange multipliers
Proposition
Suppose f , h and g are functions on Ω. Suppose that there are
vectors x∗ ∈ Omega, λ, µ ≥ 0 such that
f (x∗ ) + λ> h(x∗ ) + µ> g(x∗ ) ≤ f (x) + λ> h(x) + µ> g(x)
for all x ∈ Ω.
Then x∗ solves
min f
(x),
h(x) = h(x∗ ),
g(x) ≤ g(x∗ ),
Proof
Trivial.
Richárd Molnár-Szipai
Global Optimization 5.
x∈Ω
Zero-order conditions and Lagrange multipliers
The result suggests, that we can guess Lagrange multipliers, and
solve the Lagrangian problem.
If
h(x) 6= 0,
or
g(x) 6≤ 0,
then we can modify the multipliers
accordingly, and resolve the Lagrangian.
The Lagrangian problem is unconstrained, we can use the methods
proposed in the previous sections.
Richárd Molnár-Szipai
Global Optimization 5.