Modified Newton Methods
Lecture 9
Alessandra Nardi
Thanks to Prof. Jacob White, Jaime Peraire, Michal Rewienski,
and Karen Veroy
Last lecture review
• Solving nonlinear systems of equations
– SPICE DC Analysis
• Newton’s Method
–
–
–
–
Derivation of Newton
Quadratic Convergence
Examples
Convergence Testing
• Multidimensonal Newton Method
– Basic Algorithm
– Quadratic convergence
– Application to circuits
Multidimensional Newton Method
Algorithm
Newton Algorithm for Solving F  x   0
x = Initial Guess, k  0
0
Repeat {
   
 x  x  x    F  x  for x
Compute F x k , J F x k
Solve J F
k
k 1
k
k
k 1
k  k 1
} Until
x k 1  x k ,

f x k 1

small enough
Multidimensional Newton Method
Convergence
Local Convergence Theorem
If
 
a) J F1 x k  
 Inverse is bounded 
b) J F  x   J F  y  
x y
 Derivative is Lipschitz Cont 
Then Newton’s method converges given a
sufficiently close initial guess (and
convergence is quadratic)
Last lecture review
• Applying NR to the system of equations we
find that at iteration k+1:
– all the coefficients of KCL, KVL and of BCE of
the linear elements remain unchanged with respect
to iteration k
– Nonlinear elements are represented by a
linearization of BCE around iteration k
 This system of equations can be interpreted as
the STA of a linear circuit (companion
network) whose elements are specified by the
linearized BCE.
Application of NR to Circuit Equations
Companion Network – MNA templates
Note: G0 and Id depend on the iteration count k
 G0=G0(k) and Id=Id(k)
Application of NR to Circuit Equations
Companion Network – MNA templates
Modeling a MOSFET
(MOS Level 1, linear regime)
d
Modeling a MOSFET
(MOS Level 1, linear regime)
Last lecture review
• Multidimensional Case: each step of iteration
implies solving a system of linear equations
• Linearizing the circuit leads to a matrix whose
structure does not change from iteration to
iteration: only the values of the companion
circuits (the nonlinear elements) are changing.
DC Analysis Flow Diagram
For each state variable in the system
Implications
• Device model equations must be continuous
with continuous derivatives (not all models do
this - - be sure models are decent - beware of
user-supplied models)
• Watch out for floating nodes (If a node becomes
disconnected, then J(x) is singular)
• Give good initial guess for x(0)
• Most model computations produce errors in
function values and derivatives. Want to have
convergence criteria || x(k+1) - x(k) || <  such that
 > than model errors.
Improving convergence
• Improve Models (80% of problems)
• Improve Algorithms (20% of problems)
Focus on new algorithms:
Limiting Schemes
Continuations Schemes
Outline
• Limiting Schemes
– Direction Corrupting
– Non corrupting (Damped Newton)
• Globally Convergent if Jacobian is Nonsingular
• Difficulty with Singular Jacobians
• Continuation Schemes
– Source stepping
– More General Continuation Scheme
– Improving Efficiency
• Better first guess for each continuation step
Multidimensional Newton Method
Convergence Problems – Local Minimum
Local
Minimum
f
At a local minimum,
0
x
Multidimensional Case: J F  x  is singular
Multidimensional Newton Method
Convergence Problems – Nearly singular
f(x)
x1
x0
Must Somehow Limit the changes in X
X
Multidimensional Newton Method
Convergence Problems - Overflow
f(x)
x
0
Must Somehow Limit the changes in X
X
1
x
Newton Method with Limiting
Newton Algorithm for Solving F  x   0
x = Initial Guess, k  0
0
Repeat {
   
 x  x  F  x  for x
 limited  x 
Compute F x k , J F x k
Solve J F
x k 1  x k
}
k
k 1
k
k 1
k 1
k  k 1
k 1
k 1

x
,
F
x

 small enough
Until
Newton Method with Limiting
Limiting Methods
• Direction Corrupting


limited x k 1 
i
xik 1 if xik 1  

limited x k 1
 sign  xik 1  otherwise

x k 1

• NonCorrupting


limited x k 1  x k 1


  min 1,
k 1

x





limited x
k 1

Heuristics, No Guarantee of Global Convergence

x k 1
Newton Method with Limiting
Damped Newton Scheme
General Damping Scheme
 
 
Solve J F x k x k 1   F x k for x k 1
x k 1  x k   k x k 1
Key Idea: Line Search

Pick  to minimize F x   x
k

F x   x
k
k
k 1

2
2
k
k

k 1

 F x   x
k
k
2
2
k 1
 
T
F x k   k x k 1
Method Performs a one-dimensional search in
Newton Direction

Newton Method with Limiting
Damped Newton – Convergence Theorem
If
 
a) J F1 x k  
 Inverse is bounded 
b) J F  x   J F  y  
x y
 Derivative is Lipschitz Cont 
Then
There exists a set of  k ' s   0,1 such that



F x k 1  F x k   k x k 1

 
  F xk
with  <1
Every Step reduces F-- Global Convergence!
Newton Method with Limiting
Damped Newton – Nested Iteration
x = Initial Guess, k  0
Repeat {
0
   
Solve J  x  x   F  x  for x
Find    0,1 such that F  x   x 
Compute F x k , J F x k
k
k 1
k 1
k
F
k
}
k
k
x k 1  x k   k x k 1
k  k 1
k 1
k 1

x
,
F
x
  small enough
Until
k 1
is minimized
Newton Method with Limiting
Damped Newton – Singular Jacobian Problem
x2
1
x
1
D
x
x
0
X
Damped Newton Methods “push” iterates to local minimums
Finds the points where Jacobian is Singular
Newton with Continuation schemes
Basic Concepts - General setting
Newton converges given a close initial guess
 Idea: Generate a sequence of problems, s.t. a problem
is a good initial guess for the following one
F (x)  0  Solve F  x    ,    0 where:
a) F  x  0  , 0   0 is easy to solve  Starts the continuation
b) F  x 1 ,1  F  x  Ends the continuation
c) x    is sufficiently smooth Hard to insure!
Newton with Continuation schemes
Basic Concepts – Template Algorithm
Solve F  x  0  , 0  , x   prev   x  0 
 =0.01,   
While   1 {
x0     x   prev 
Try to Solve F  x    ,    0 with Newton
}
If Newton Converged
x   prev   x    ,      ,   2
Else
1
   ,    prev  
2
Newton with Continuation schemes
Basic Concepts – Source Stepping Example
Newton with Continuation schemes
Basic Concepts – Source Stepping Example
R
Vs
+
-
v
Diode
1
f  v    ,    idiode  v    v  Vs   0
R
f  v,  
v

idiode  v 
v
1

 Not  dependent!
R
Source Stepping Does Not Alter Jacobian
Newton with Continuation schemes
Jacobian Altering Scheme
F  x    ,     F  x      1    x   
Observations
 =0 F  x  0  , 0   x  0   0
F  x  0  , 0 
x
I
 =1 F  x 1 ,1  F  x 1 
F  x  0  , 0 
x

F  x 1 
x
Problem is easy to solve and
Jacobian definitely
nonsingular.
Back to the original problem
and original Jacobian
Newton with Continuation schemes
Jacobian Altering Scheme – Basic Algorithm
Solve F  x  0  , 0  , x   prev   x  0 
 =0.01,   
While   1 {
x0     x   prev   ?
Try to Solve F  x    ,    0 with Newton
}
If Newton Converged
x   prev   x    ,      ,   2
Else
1
   ,    prev  
2
Newton with Continuation schemes
Jacobian Altering Scheme – Update Improvement
F  x      ,      F  x  0 ,   
F  x    ,  
x
 x       x   
F  x    ,  


F  x    ,  
x
Have From last
step’s Newton
 x       x   
0
Better Guess
for next
step’s
Newton


F  x    ,  



Newton with Continuation schemes
Jacobian Altering Scheme – Update Improvement
If
F  x    ,     F  x      1    x   
Then
F  x,  
 F  x  x  

Easily Computed
Newton with Continuation schemes
Jacobian Altering Scheme – Update Improvement
 F  x    ,  
x       x     

x

0




1
F  x    ,  


Graphically
x  
x0     
0
   
1

Summary
• Newton’s Method works fine:
– given a close enough initial guess
• In case Newton does not converge:
– Limiting Schemes
• Direction Corrupting
• Non corrupting (Damped Newton)
– Globally Convergent if Jacobian is Nonsingular
– Difficulty with Singular Jacobians
– Continuation Schemes
• Source stepping
• More General Continuation Scheme
• Improving Efficiency
– Better first guess for each continuation step