Solving Systems of Linear
Equations: Iterative Methods
Dongshin Kim
Joongheon Kim
Jihoon Myung
Computer Networks Research Lab.
Dept. of Computer Science and Engineering
Korea University
Computer Networks Research Lab.
Korea University
Contents






Introduction
Basic Idea
Jacobi Method
Gauss-Seidel Method
Successive Over Relaxation (SOR)
Summary
2
Introduction (1/2)
 If systems of linear equations are very large, the
computational effort of direct methods is
prohibitively expensive
 Three common classical iterative techniques for
linear systems
– The Jacobi method
– Gauss-Seidel method
– Successive Over Relaxation (SOR) method
 Matlab’s built-in functions
3
Introduction (2/2)
 For systems that have coefficient matrices with the
appropriate structure – especially large, sparse
systems (many coefficients whose value is zero) –
iterative techniques may be preferable
 4 1 0 0 1 0 0 0 0 0
 1 4  1 0 0  1 0 0 0 0

 0 1 4 1 0 0 1 0 0 0

 0 0 1 4 0 0 0 1 0 0
 1 0 0 0 4  1 0 0  1 0

 0 1 0 0 1 4 1 0 0 1
 0 0 1 0 0 1 4 1 0 0

 0 0 0 1 0 0 1 4 0 0

 0 0 0 0 1 0 0 0 4 1
 0 0 0 0 0 1 0 0 1 4

 0 0 0 0 0 0 1 0 0 1
 0 0 0 0 0 0 0 1 0 0

 0 0 0 0 0 0 0 0 1 0
 0 0 0 0 0 0 0 0 0 1

0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0
0
0
0
0
0
0
1
0
0
1
4
1
0
0
1
0
0   u1  0.08 
 
0 0 0 0 0   u 2  0.16 
0 0 0 0 0   u3  0.36 

   
0 0 0 0 0   u 4  1.64 
0 0 0 0 0   u5  0.16 

   
0 0 0 0 0   u 6   0.0 
0 0 0 0 0   u 7   0.0 

   
 1 0 0 0 0   u8   1.0 

   
0  1 0 0 0   u9  0.36 
0 0  1 0 0  u10   0 

   
 1 0 0  1 0  u11   0 
4 0 0 0  1 u12   1.0 

   
0 4  1 0 0  u13  1.64 
0  1 4  1 0  u14   1.0 



0 0  1 4  1 u15   1.0 

   
 1 0 0  1 4  u16   2.0 
0
0
0
0
4
Basic Idea
 Convert the system Ax  B into the equivalent system x  Cx  d
a11x1  a12 x2  a13 x3  b1
a21x1  a22 x1  a23 x3  b2
a31x1  a32 x1  a33 x3  b3
x1 

a
a12
b
x2  13 x3  1
a11
a11
a11
a 21
x2  
x1
a 22
x3  
a31
a
x1  32 x2
a33
a33
a 23
b2

x1 
a 22
a 22

b3
a33
 Generate a sequence of approximation x (1) , x ( 2) ,... , where
x ( k )  Cx ( k 1)  d
5
Jacobi Method (1/5)
 Consider the two-by-two
system
1
x


y3
2x  y  6
2
x  2y  6
1
y   x3
2
 Start with x (1)  y (1)  1 / 2
 Simultaneous updating
– New values of the variables
are not used until a new
( 2)
y
iteration step is begun
x
( 2)
y ( 2)
1 (1)
1
11
  y 3   3 
2
4
4
(1)
1 (1)
1
11 y
  x 3   3 
2
4
4
x (1)
x ( 2)
6
Jacobi Method (2/5)
 Con’t
1 ( 2)
11
13
y 3   3 
2
8
8
1
11
13
  x ( 2)  3    3 
2
8
8
x (3)  
y (3)
7
Jacobi Method (3/5)
 Consider the three-by-three system
2 x1  x2  x3  1
x1  2 x2  x3  6
x1  x2  2 x3  3
x1 
 0 .5 x 2  0 .5 x 3  0 .5
x2  0.5 x1
 0 .5 x 3  3 .0
x3  0.5 x1  0.5 x2
 1 .5
 x1( k )   0.0 0.5  0.5  x1( k 1)   0.5
 (k )  
 ( k 1)  


x


0
.
5
0
.
0
0
.
5
x

3
.
0
 2  
 
  2

 x ( k )   0.5 0.5 0.0   x ( k 1)    1.5 
  3

 3  
 
 Start with x (0)  (0,0,0)
8
Jacobi Method (4/5)
 Matlab function for jacobi method
9
Jacobi Method (5/5)
y
 Discussion
x  2y  6
x  2 y  6
2x  y  6
y  2 x  6
y (2)
 A necessary and sufficient
condition for the convergence
of the Jacobi method
y (1)
x (1)
x ( 2) x
– The magnitude of the largest
eigenvalue of the iteration
matrix C be less than 1
10
Gauss-Seidel Method (1/5)
 Consider the two-by-two
system x   1 y  3
2
1
y   x3
2
 Start with x (1)  y (1)  1 / 2
 Sequential updating
– New values of the
variables are updated
immediately
x
( 2)
y
( 2)
y (2)
1 (1)
1
11
  y 3  3
(1)
2
4
4 y
1
11
13
  x ( 2)  3    3 
2
8
8
x ( 2)
11
Gauss-Seidel Method (2/5)
 Con’t
1 ( 2)
13
35
y 3  3
2
16
16
1
35
61
  x ( 3)  3    3 
2
32
32
x ( 3)  
y ( 3)
12
Gauss-Seidel Method (3/5)
 Consider the three-by-three system
2 x1  x2  x3  1
x1  2 x2  x3  6
x1  x2  2 x3  3
x1new 
x2 new  0.5 x1new
 0.5 x2 ( old )  0.5 x3 (old )  0.5
x3 new  0.5 x1new  0.5 x2 new
 0.5 x3 ( old )  3.0
 1 .5
( 0)
x
 (0,0,0)
 Start with
13
Gauss-Seidel Method (4/5)
 Matlab function for gauss-seidel method
14
Gauss-Seidel Method (5/5)
 Discussion
– The Gauss-Seidel method is sensitive to the form of the coefficient matrix A
– The Gauss-Seidel method typically converges more rapidly than
the Jacobi method
– The Gauss-Seidel method is more difficult to use for parallel
computation
15
Successive Over Relaxation (SOR)
(1/5)
 Introduce an additional parameter, ω, that may accelerate
the convergence of the iterations
a11x1  a12 x2  a13 x3  b1
a21x1  a22 x1  a23 x3  b2
a31x1  a32 x1  a33 x3  b3
a11x1  a12 x2  a13 x3  b1
a21x1  a22 x1  a23 x3  b2
a31x1  a32 x1  a33 x3  b3
x1new  (1   ) x1old  
x2 new  (1   ) x2 old  
x3 new  (1   ) x3 old  

a11
(b1  a12 x2 old   a13 x3 old  )

a 22

a33
(b2  a 21x1new  a 23 x3 old  )
(b3  a31 x1new  a32 x2 new )
16
Successive Over Relaxation (SOR)
(2/5)
 Consider the three-by-three system Ax  b
 4 2 0 
A   2 6  5
 0  5 11 
 8 
b   29 
 43 
1
x1new  (1   ) x1old    ( x2 old   2)
2
1
5
29
x2 new  (1   ) x2 old    ( x1new  x3 old   )
3
6
6
5
43
x3 new  (1   ) x3 old    ( x1new  )
11
11
17
Successive Over Relaxation (SOR)
(3/5)
 Required number of iterations for different values of the
relaxation parameter
( 0)
– Start with x  [0 0 ... 0]'
– Tolerance = 0.00001
ω
No. of iterations
0.8 0.9 1.0 1.2 1.25 1.3 1.4
44
36
29
18
15
13
16
18
Successive Over Relaxation (SOR)
(4/5)
 Matlab function for SOR
19
Successive Over Relaxation (SOR)
(5/5)
 Discussion
– The SOR method can be derived by multiplying the decomposed
system obtained from the Gauss-Seidel method by the
relaxation parameter w
– The iterative parameter w should always be chosen such that
0<w<2
20
Summary
 Gauss-seidel method
 Jacobi method
x1k   
a12 k 1 a13 k 1 b1
x2

x3

a11
a11
a11
x1k   
a12 k 1 a13 k 1 b1
x2

x3

a11
a11
a11
x 2 k   
a 21 k 1 a 23 k 1 b2
x1

x3

a 22
a 22
a 22
x 2 k   
a 21 k  a 23 k 1 b2
x1 
x3

a 22
a 22
a 22
x3  k   
a31 k 1 a32 k 1 b3
x1

x2

a33
a33
a33
x3  k   
a31 k  a32 k 1 b3
x1 
x2

a33
a33
a33
 SOR method
x1new  (1   ) x1old  
x2 new  (1   ) x2 old  
x3 new  (1   ) x3 old  

a11
(b1  a12 x2 old   a13 x3 old  )

a 22

a33
(b2  a 21x1new  a23 x3 old  )
(b3  a31 x1new  a32 x2 new )
21