Augmented Lagrangian Method
Svitlana Volkova
General Constrained Optimization Problem
I
Optimize nonlinear objective function
min f (x )
x ∈<n
I
Subject to nonlinear constraints:
c(x ) = 0
Sequential Quadratic Programming (SQP)
I
Step Generation: Apply Newton method to solve 1st order
optimality conditions:
1
min g(xk )T ∆x + ∆x T H(xk , λk )∆x
2
s.t. c(xk ) + J(xk )∆x = 0
Sequential Quadratic Programming (SQP)
I
Step Acceptance: Merit function balances the conflicting
goals of reducing the objective function f (x ) and the
constraint violation kc(x )k, e.g., L1 merit function:
ρ
Φ(x , ρ) = f (x ) + kc(x )k1
2
SQP Characteristics
I
Not a feasible method
I
Solving QP at each iteration is expensive in large scale
constrained optimization
I
Dependent on rapid algorithms for solving quadratic programs
I
Only guarantee to find a local solution
Penalty Methods
I
Main idea: solving a constrained optimization problem by
solving a sequence of unconstrained optimization problems
I
Another type of merit function - quadratic penalty
ρ
P(x , ρ) = f (x ) + kc(x )k22
2
x (ρ) = arg min P(x , ρ)
x ∈<n
I
The solution to original optimization problem:
lim x (ρ) = x ∗
ρ→∞
Limitations of Penalty Methods
I
lim x (ρ) = x ∗ is not computationally reasonable
ρ→∞
Limitations of Penalty Methods
I
lim x (ρ) = x ∗ is not computationally reasonable
ρ→∞
Limitations of Penalty Methods
I
lim x (ρ) = x ∗ is not computationally reasonable
ρ→∞
Limitations of Penalty Methods
I
lim x (ρ) = x ∗ is not computationally reasonable
ρ→∞
Introducing Augmented Lagrangian Method
I
I
lim x (ρ) = x ∗ is not computationally reasonable
ρ→∞
If we relax c(x = 0) by some value e.g., ρ1 λ then we can
achieve x ∗ with finite ρ
Problem Reformulation
I
We reformulate the original problem NLP
min f (x )
x ∈<n
s.t. c(x ) = 0
I
into a new problem PNLP:
min f (x )
x ∈<n
1
s.t. c(x ) − λ = 0
ρ
Introducing Augmented Lagrangian
I
Penalty Method:
ρ
x (ρ) = arg min f (x ) + kc(x )k22
2
x ∈<n
I
Augmented Lagrangian:
ρ
1
x (ρ, λ) = arg min f (x ) + kc(x ) − λk22 =
2
ρ
x ∈<n
ρ
2
= arg min f (x ) + kc(x )k − c(x )T λ
2
x ∈<n
ρ
.
x (ρ, λ) = arg min f (x ) − c(x )T λ + kc(x )k22
2
x ∈<n |
{z
}
LA (x ,ρ,λ)
I
optimal shift = optimal multiplier
1st Order Optimality Conditions for Original
Optimization Problem
I
Original problem
min f (x )
x ∈<n
s.t. c(x ) = 0
I
1st order optimality conditions for original problem
- c(x ) = 0
- ∇f (x ) − ∇c(x )T λ = 0 ⇒ g(x ) − J(x )T λ = 0
1st Order Optimality Conditions for Modified
Optimization Problem
I
Modified problem
min f (x )
x ∈<n
1
s.t. c(x ) − λ = 0
ρ
ρ
L(x , ρ, λ) = f (x ) − c(x )T λ + kc(x )k22
2
I
1st order optimality conditions for modified problem:
c(x ) −
1
λ=0
p
ρ
∇L(x , ρ, λ) = ∇f (x ) − ∇c(x )T λ + ∇kc(x )k22 =
2
T
T
= g(x ) − J(x ) λ + ρJ(x ) c(x )
g(x ) − J(x )T (λ − ρc(x )) = 0
Augmented Lagrangian vs. SQP
I
SQP generates a step and then pairs it with a merit function
I
AL similarly to penalty methods directly minimizes penalty
function
Extend Augmented Lagrangian Method to
Inequality Constraints
I
Inequality constraints:
min f (x )
x ,s∈<n
s.t. c(x ) ≥ 0
I
Use slack variables
min f (x )
x ,s∈<n
s.t. c(x ) − s = 0
s≥0
I
solving a sequence of bounded constrained problems