Conjugate Gradient
Optimization
Outline
•
•
•
•
CG Method
Non-linear CG
Solving Linear System of Equations
Preconditioned CG and Regularization
Quasi-Newton
Quasi-Newton Condition:
Condition: g’
g’–– gg ==Hdx’
Hdx’
2
2
Recall: d f(x)/dx ~ [df(x+dx)/dx-df(x)/dx]/dx
Step 1: Quadratic f(x+dx’) ~ f(x) + dx’ g + 1/2dx’ H dx’
D
Step 3: Define
D
Step 2: Gradient
f(x+dx’) ~ g + H dx’
f(x+dx’) = g’  g’ – g = Hdx’
Similar to FD approx. to 2nd dervative
dx _ g’
g
Kiss point x
dx’
dx
x*
g’
Conjugate Gradient
Quasi-Newton Condition:
g’ – g = Hdx’
(1)
Plane spanned by dx and dx’
dx’
x*
Bullseye ( f(x*),dx)=0
Bullseye ( f(x*),dx’)=0
D
D
dx _ g
g
Kiss point
dx
g’ at bullseye has no components in dx & dx’ plane
If dx’ points at bullseye, then dot product of dx with eq. 1 gives
-dxg = 0 = dxHdx’
(3)
Conjugacy Cond. -> search dx’ (conjugate to previous dx) hits bullseye
dx’ = bdx– g’
Above is conjugate to dx if b found
s.t. (dx’, Hdx)=0
Conjugate Gradient
Conjugacy Cond :
New Search Direc:
dx’ is conjugate to dx
for b s.t. (dx’,Hdx)=0
Solve for Conjugacy Step b
Step length a :
dx’Hdx = 0
dx’ = bdx – g’
( bdx – g’,Hdx)=0
(3)
(4)
dx’
x*
dx
(5)
b = g’Hdx/dxHdx (6)
a = (g,dx)/dxHdx
(7)
Hit Bullseye
No H!
Conjugate Gradient
For k =2:Niter
dx’
dx
No H!
end
Fletcher-Reeves
x*
Conjugate Gradient
Starting point x0 and take -g as initial direction
For k =2:Niter
dx’
x*
dx
Polack-Ribiere
end
Conjugate Gradient Properties
For i = 1:nit
%find b
p= dx +bg
%find a
dx’ = dx + ap
dx= dx’
x=x+ dx’
end
g’
dx’
x*
Kiss point
dx
dx’’
1. (p j ,Hpi )=0 for any i=j
2. Converges in N steps for NxN H with SPD property
3. Converges quickly if eigenvalues are clustered (i.e., round contours)
Solving Rectangular Linear Systems by
Regularized SD with Scaling
Given: H rectangular matrix s.t. Hx=g ill-conditioned
Let CH H x = Cg s.t. C approximates inverse H H
T
T
(k)
Soln: x = x – a [CH( H x - g) + l Gx
(k+1)
(k)
CG will converge in 2 steps!
Newton in one step!
400 1 x1
1
4 x2
=
5
2
Classroom Exercise:
1. Derive formula for b
2. Write CG code and
solve above equations
T
Solving Square Linear Systems by
HL
Regularized CG
Solving Lx = -g  e = xTg + 0.5 xT L x
Solving Rectangular Linear Systems by Regularized CG
Solving LTLx = -L Tg  e = xTg + 0.5 xT LTL x
Write Two Subroutines: 1). [d]=forward(m)
2). [g] = adjoint(r)
% predict data d=Lm from model
% predict model g=L’r from data residual r=(Lm-d)