Relaxation Methods for Iterative Solution
to Linear Systems of Equations
Gerald Recktenwald
Portland State University
Mechanical Engineering Department
[email protected]
Primary Topics
• Basic Concepts
• Stationary Methods a.k.a. Relaxation methods
� Gauss Seidel
� Jacobi
� SOR
Relaxation Solution to Linear Systems
page 1
Direct and Iterative Methods
Solve
Ax = b
where A is n × n matrix, x and b are n element column vectors.
Gaussian elimination with backward substitution is a Direct Method
• A Direct Method obtains the solution in a finite number of steps
• Solution is equivalent to
−1
x=A b
Iterative Methods obtain a sequence of approximations to the solution
x
k
k = 0, 1, 2, . . .
such that (b − Axk ) → 0 as k → ∞.
Relaxation Solution to Linear Systems
page 2
Residual
The residual
r = b − Ax
is zero when x is the solution to Ax = b.
For an iterative method the residual at iteration k is
k
r = b − Ax
k
k = 0, 1, 2, . . .
If an iterative methods converges, then r k → 0 as k → ∞.
Relaxation Solution to Linear Systems
page 3
Relaxation Methods
(1)
Overview
•
•
•
•
Simple to program
Converges slowly for large systems of equations (large n)
Not a useful stand alone solution method
Key ingredient to multigrid methods
Examples
• Jacobi
• Gauss-Seidel
• SOR
Relaxation Solution to Linear Systems
page 4
Relaxation Methods
(2)
Basic Idea
• Update elements of the solution vector one element at a time
• “Solve” a nodal equation by assuming other nodal values are known
Apply the finite volume method to get
N
−aS φS − aW φW + aP φP − aE φE − aN φN = b
W
P
E
If k is the iteration counter, we “solve” for φk+1
P
k+1
φP
S
=
�
1 �
k
k
k
k
b + aS φ S + aW φ W + aE φ E + aN φ N
aP
Values of φW , φE , φN and φS are updated by applying the
iterative formula to the cells centered at those neighbor nodes.
Relaxation Solution to Linear Systems
page 5
Matrix Notation
(1)
J=ny+2
On a structured mesh the interior nodes can
be numbered sequentially with a mapping like
i=nxny
i = I − 1 + (J − 2)nx
i=nx+1
3
where nx is the number of cells in the x
direction and xi is the value of the ith
dependent field variable which is located at
the cells with indices I and J .
The compass point notation
2
i=1
i=2
2
3
i=nx
y
J=1
I=1
I=nx+2
x
−aS φS − aW φW + aP φP − aE φE − aN φN = b
becomes
−aS,ixi−nx − aW,ixi−1 + aP,ixi − aE,ixi+1 − aW,ixi+nx = bi
Relaxation Solution to Linear Systems
page 6
Matrix Notation
(2)
The natural ordering of nodes on a two-dimensional structured mesh yields a coefficient
matrix with five diagonals. Coefficients on the diagonals correspond to aP coefficient of
φP ↔ xi and the four neighbor coefficients aS , aW , aE , and aN .
A=
Relaxation Solution to Linear Systems
– aS
– aW aP – aE
– aN
page 7
Matrix Notation
(3)
In an unstructured mesh, the coefficient matrix is also sparse, but the non-zero elements
of the coefficient matrix do not lie along diagonals.
Unstructured Mesh
Non-zeros in coefficient matrix
0
10
20
30
40
50
60
70
80
90
0
10
20
30
40
50
nz = 361
60
70
80
90
To simplify the presentation, we’ll use a structured mesh.
Relaxation Solution to Linear Systems
page 8
Relaxation Methods
Use relaxation methods to solve
Ax = b
Relaxation methods do not work for all A, but they do work for standard finite-difference
and finite-volume discretizations of the Poisson equation.
• Jacobi
• Gauss-Seidel
• SOR – Successive Over-relaxation
Relaxation Solution to Linear Systems
page 9
Jacobi Iteration
(1)
A discrete model of the Poisson equation yields a system of equations that can be written
Ax = b. The equation for the ith nodal value is
n
�
aij xj = bi
j=1
where n is the total number of nodes in the domain.
Extract the diagonal term from the sum
aiixi +
n
�
aij xj = bi
j=1
j�=i
The preceding equation will not be satisfied until, in the limit as the number of iterations
increased to ∞.
Relaxation Solution to Linear Systems
page 10
Jacobi Iteration
(2)
“Solve” for the next guess xi.
�
�
n
�
1
xi =
b−
aij xj
aii
j=1
( �)
j�=i
In terms of the compass-point notation for the two-dimensional finite-volume method, the
Equation (�) is
(k+1)
φP
=
�
1 �
(k)
(k)
(k)
(k)
b + aS φ S + aW φ W + aE φ E + aN φ N
aP
Equation (�) is not really the solution for xi until all the other xj are known. In other
words, the value of xi from Equation (�) will be correct only when the system of
equations Ax = b is solved.
If matrix A has favorable properties, the value of xi from Equation (�) will be closer to
the solution than the previous guess at xi.
Relaxation Solution to Linear Systems
page 11
Jacobi Iteration
(3)
Let k be the iteration counter. The Jacobi iteration formula is
�
�
n
�
1
(k+1)
(k)
xi
=
bi −
aij xj
i = 1, . . . , n
aii
j=1
j�=i
This formula implies that two copies of the x vector are maintained in memory: one for
iteration k and one for iteration k + 1.
for k=1:maxit
% -- For nodes away from the boundaries
for i=(nx+1):n-nx
xnew(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ...
+ an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i);
end
x = xnew;
end
Relaxation Solution to Linear Systems
page 12
Jacobi Iteration
(4)
The Jacobi iterative formula can be written as a matrix equation.
Let D = diag(a11, . . . , ann) and B = D − A then
(k+1)
xi
=
�
�
n
�
1
(k)
b−
aij xj
aii
j=1
j�=i
is equivalent to
x
(k+1)
=D
−1
�
b + Bx
(k)
�
(1)
where x(k+1) and x(k) are the solution vectors at iteration k + 1 and k, respectively.
Relaxation Solution to Linear Systems
page 13
Jacobi Iteration
(5)
The matrix form of the iteration can be rewritten
x
(k+1)
= Hx
(k)
+d
(2)
where
H =D
−1
B
d=D
−1
b
The matrix formulation is primarily useful in the analysis of iterative methods and in toy
implementations in Matlab. The eigenvalues of H determine the convergence rate of
the iterations.
Relaxation Solution to Linear Systems
page 14
Stopping Criteria
(1)
Recall that the goal is to solve
Ax = b
where A is n × n matrix, x and b are n element column vectors.
Iterative Methods obtain a sequence of approximations to the solution
x
k
k = 0, 1, 2, . . .
such that b − Axk → 0 as k → ∞.
The residual
r = b − Ax
is zero when x is the solution to Ax = b.
Relaxation Solution to Linear Systems
page 15
Stopping Criteria
(2)
For an iterative method the residual at iteration k is
k
r = b − Ax
k
k = 0, 1, 2, . . .
If an iterative methods converges, then r k → 0 as k → ∞.
Since r is a vector, what does it mean that r k → 0?
Use a convenient vector norm to test whether �r� is small enough.
Relaxation Solution to Linear Systems
page 16
Jacobi Iteration
Solution to a toy problem
∂ 2T
∂ 2T
+
=0
∂x2
∂y 2
on 0 ≤ x ≤ L, 0 ≤ y ≤ W
T = 0 on three boundaries
uniform q on fourth boundary
(3)
Solution
4.5
4
3.5
3
2
1.5
1.5
1
1
0.5
0.5
0
0
Solution is obtained by Matlab code demoJacobi
Relaxation Solution to Linear Systems
page 17
Jacobi Iteration
(4)
Convergence of Jacobi iterations slow as mesh is refined.
0
10
8x8
16 x 16
32 x 32
−1
|| r ||1 / || r0 ||1
10
−2
10
−3
10
−4
10
0
200
400
600
800
Iteration
1000
1200
1400
Relaxation Solution to Linear Systems
page 18
Gauss-Seidel Iteration
(1)
In the Jacobi method x(k+1) is obtained from the frozen values of x(k).
(k+1)
The Gauss-Seidel method uses a similar formula for updating xi
(k+1)
xi
is used as soon as it is available
, but the new value of
values from
iteration k
y
values from
iteration k+1
x
Relaxation Solution to Linear Systems
page 19
Gauss-Seidel Iteration
(2)
Replace nodal values as sweep progresses through the domain
for k=1:maxit
% -- For nodes away from the boundaries
for i=(nx+1):n-nx
x(i) = (b(i) + as(i)*x(i-nx) + aw(i)*x(i-1) ...
+ an(i)*x(i+nx) + ae(i)*x(i+1) ) / ap(i);
end
end
Relaxation Solution to Linear Systems
page 20
Gauss-Seidel Iteration
(3)
Consider the sweep through the nodal values in order of increasing i:
�
�
n
�
1
(k)
=
b1 −
a1j xj
a11
j=2
(k+1)
x1
�
�
n
�
1
(k+1)
(k)
=
b2 − a21x1
−
a1j xj
a22
j=3
(k+1)
x2
(k+1)
x3
...
(k+1)
xi
�
�
n
�
1
(k+1)
(k+1)
(k)
=
b3 − a31x1
− a32x2
−
a1j xj
a33
j=4
�
�
i−1
n
�
�
1
(k+1)
(k)
=
bi −
aij xi
−
a1j xj
aii
j=1
j=i+1
Relaxation Solution to Linear Systems
(3)
page 21
Gauss-Seidel Iteration
(4)
Again, let
A
x
=
b
new values
D = diag(a11, . . . , ann)
and define
i
old values
=
Coefficients that
multiply new values
L = lower triangular part of A
U = upper triangular part of A
Then Gauss-Seidel iterations correspond
to the splitting of A
Coefficients that
multiply old values
A=D−L−U
Note that A �= LU , i.e.,, L and U are
not the usual factors of A.
Relaxation Solution to Linear Systems
page 22
Gauss-Seidel Iteration
(5)
Equation (3) can be rewritten as
(k+1)
aiixi
= bi −
or
(k+1)
aiixi
−
or
Dx
(k+1)
i−1
�
i−1
�
j=1
(k+1)
aij xi
j=1
− Lx
(k+1)
aij xi
(k+1)
−
= bi −
= b + Ux
=⇒ (D − L)x
(k+1)
n
�
a1j xj
n
�
a1j xj
(k)
j=i+1
(k)
j=i+1
(k)
= Ux
(k)
+b
or
x
Relaxation Solution to Linear Systems
(k+1)
= (D − L)
−1
(U x
(k)
+ b)
(4)
page 23
Gauss-Seidel Iteration
(6)
Note: The order in which the nodes are processed in the Gauss-Seidel iterations will
affect the path toward the solution. In other words, for two different orderings of the
nodes, the intermediate iterates of x wil be different.
The Jacobi iteration does not depend on the order in which the nodes are numbered.
Much of the classical theory of relaxation methods assumes that the nodes and numbered
in natural order.
Relaxation Solution to Linear Systems
page 24
SOR: Successive Over-Relaxation
(k+1)
Let x̂i
(1)
be the updated value of xi from the Gauss-Seidel formula, viz.
�
�
�
�
1
(k+1)
(k+1)
(k)
x̂i
=
bi −
aij xj
−
aij xj
aii
j<i
j>i
(k+1)
Instead of taking x̂i
(5)
as the value of xi at the k + 1 step, use
(k+1)
xi
(k+1)
= ω x̂i
(k)
+ (1 − ω)xi
(6)
where ω is a scalar weighting factor. The basic step in Equation (5) is called relaxation.
ω = 1 for Gauss-Seidel
ω < 1 causes the iterations to more slowly move toward the solution.
This is called under-relaxation.
ω > 1 causes the iterations to accelerate, i.e. to be more aggressive.
This is called over-relaxation.
Relaxation Solution to Linear Systems
page 25
Relaxation as an update
We can rewrite Equation (6) as
(k+1)
xi
or
(k)
= xi
(k+1)
xi
where
�
�
(k+1)
(k)
+ ω x̂i
− xi
(k)
= xi
(k+1)
∆xi
(k+1)
(k+1)
= x̂i
(k)
− xi
is the update to xi in iteration k.
(k+1)
is the direction in which the solution is changing.
∆xi
(k+1)
ω∆xi
(8)
+ ω∆xi
(k+1)
∆xi
(7)
is a magnified (ω > 1) or reduced (ω < 1) change to the solution.
Relaxation Solution to Linear Systems
page 26
SOR: Successive Over-Relaxation
(2)
Substitute Equation (6) into Equation (5)
�
�
�
�
ω
(k+1)
(k+1)
(k)
k
xi
= (1 − ω)xi +
bi −
aij xj
−
aij xj
aii
j<i
j>i
(9)
Multiply through by aii and rearrange to get
�
�
(k+1)
(k+1)
(k)
k
aiixi
+ω
aij xj
= (1 − ω)aiixi − ω
aij xj + ωbi
j<i
j>i
Use the splitting A = D − L − U and the preceding equation becomes
Dx
(k+1)
− ωLx
(k+1)
(D − ωL)x
(k+1)
x
(k+1)
(k)
(k)
= (1 − ω)D x + ωU x + ω b
�
� (k)
(k)
= (1 − ω)D x + ωU x + ω b
� (k)
−1 �
(k)
−1
= (D − ωL)
(1 − ω)D x + ωU x + ω(D − ωL) b
Relaxation Solution to Linear Systems
page 27
SOR: Successive Over-Relaxation
(3)
The update equation is
x
(k+1)
= (D − ωL)
Thus
−1 �
(1 − ω)D x
x
(k+1)
(k)
= Hx
� (k)
−1
+ ωU x + ω(D − ωL) b
(k)
+d
where
H = (D − ωL)
−1 �
d = ω(D − ωL)
(1 − ω)D x
−1
(k)
+ ωU
b
�
The speed of convergence depends on the eigenvalues of H .
Relaxation Solution to Linear Systems
page 28
Compare Relaxation Convergence
On a coarse 8 × 8 mesh
4
10
Jacobi
GaussïSeidel
SOR
3
||r||
1
10
2
10
1
10
0
10
0
Relaxation Solution to Linear Systems
20
40
60
Iteration
80
100
120
page 29
Compare Relaxation Convergence
On a coarse 16 × 16 mesh
5
10
Jacobi
GaussïSeidel
SOR
4
||r||
1
10
3
10
2
10
1
10
0
Relaxation Solution to Linear Systems
50
100
150
200
Iteration
250
300
350
400
page 30