An iterative least-square method suitable for solving large
sparse matrices
By I. M. Khabaza
The purpose of this paper is to report on the results of numerical experiments with an iterative
least-square method suitable for solving linear systems of equations with large sparse matrices.
The method of this paper gives satisfactory results for a
wide variety of matrices and is not restricted to real or
symmetric or definite matrices. We consider a system
of n equations:
— rfr
Here au =
frp
Let the eigensystem of the matrix A be:
a = A,, A 2 ,.. ., An = b
5i, $2, • . . , § „
where J-,- is the eigenvector corresponding to the eigenvalue A;. In general we can express the residual vector r
of equation (2) in the form
r = fll§, + a&2 + ... + a£n
(8)
Ax = b.
(1)
As a library program the method requires from the user
an auxiliary sequence which, given x, produces y = Ax.
Thus no knowledge of the elements of A is required. A
need not be a matrix; it could be any linear operator.
Given an approximate solution x we calculate the
residual vector r:
r = b - Ax.
where a, is the component of r along %,.
If we apply the iteration (3) 5 times the residual becomes
+ . . . + ang^n
(9)
(2)
We seek a better approximation x of the form
x' = x +f(A)r
where g, = g(A,).
Thus convergence occurs only if |g,| < 1 for all i.
The graph of g(A) in the range (a, b) will indicate the rate
of convergence. The iteration (3) will eliminate rapidly
the components corresponding to roots where g(X) w 0;
it will blow up components corresponding to roots where
|g(A)| > 1. We note that g(0) = 1.
(3)
where f(A) is a matrix polynomial of the form
f(A) = c , / + c2A + c3A2 + ...+
cmA"'-K
(4)
x
The true correction in (3) is A~ r, and from the CayleyHamilton theorem, A-1 may be represented as a polynomial in A of degree n— 1 in general. Thus f(A)r is an
approximation to A~xr, but/(^) is of order m—\ where
m is a small integer, about 3, whereas n, the dimension
of the vector x, may be large. The polynomial / is
determined by the vector c of dimension m:
c = (c,, c2, . . . , cm)T.
The program
We give here a description of the program developed
for testing the method of this paper. Starting with an
approximation .v we calculate the residual r; if no
approximation is known we can take x = 0. To obtain
the coefficients c,, c 2 ,..., cm of equation (4) we require
m multiplications by the matrix A and an extra multiplication to find the new residual r'\ the new approximation
x' is determined by using the same multiplications by A
used for determining c. For each further approximation,
obtained by iterating with a previously calculated set of
coefficients c, we require m — 1 multiplications by A to
calculate x' and a further multiplication to calculate r'.
Let
«=l|r||;
«'=||r'||.
(10)
(5)
The superfix T denotes transposition so that c is a column
vector.
We define the polynomial g(A) as follows:
g(A) = 1 - A/(A) = 1 - c,A - c2X2 - . . . - cjr.
(6)
The coefficients cuc2, . . ., cm are determined by the
method of least-squares so that ||g(/4)r|| is a minimum;
the double strokes here denote the usual norm, i.e. the
square root of the sum of the squares of the moduli of the
components of a vector. We take, as an example, m = 3.
We define /•, = Ar, r2 = Aru r3 = Ar2. Then we determine cuc2, c3 from the system of equations
anc2 + al3c3 = bt
a22c2 + a23c3 = b2
ai2c2 + a33c3 = b3.
After each iteration we compare v with v'. Iteration
with the same set c is continued as long as v' < Cv, where
C is a convergence control parameter; otherwise a new
set of coefficients c is calculated. The parameter C is
given a small value such as C=0-2, if we expect rapid
convergence, for slow convergence we take C = 0 • 8 or
0-9. If v' > v, as may happen if g(A) blows up some
(7)
202
Method for solving large sparse matrices
components of the residual, it may be thought that it
would be better to discard the latest approximation x'
and its residual r' and calculate a new set c from r. In
fact this turned out to be wrong. This phenomenon is
similar to over-relaxation. Indeed it often paid to iterate
once more although the residual is increasing. The
program contains further parameters, D and F (e.g.
D = 2, F = 10) which determine when to stop iterating
when the residual is increasing (e.g. when v' > Dv0
where v0 is the smallest previous residual), and when to
reject the latest approximation (e.g. when v' > Fv0).
The required solution is obtained when v' < E where E
is another parameter (e.g. £ = 0000001). Finally,
another parameter specifies the value of m which determines the dimension of c or the order of the polynomial
g(\) and the system of equations (7). Various small
values of m were tried; in most cases m — 3 seemed to be
the most suitable value.
\
\
Fig. 1
Table 1
c
3•91
-4 •76
1•81
Numerical examples
We give the numerical results forfiveexamples. In each
of thefirstfour examples the initial approximation x was
taken to be 0; a known approximation, however rough,
would of course be better. To measure the efficiency of the
method we count the number of multiplications by A
and this number is compared with the corresponding
number required to obtain the same accuracy by existing
methods. The existing methods tried were the method of
Gauss-Seidel with over-relaxation (see Martin and Tee,
1961, equations 2-5 and 2-12) and the method of
Conjugate Gradients (op. cit. equations 7-10 and 711).
In the first three examples the right-hand side b of
equation (1) was taken to be e = (1,1,. . J)7"; n = 20;
A = [1, W], the codiagonal matrix for which Au = 1,
AUi-x = Au+l = W, otherwise A,, = 0. We considered
the values W = — 0-25, — 0 5 , — 0-6, respectively. The
roots A, of A are given by the formula A, = 1+2W
V
0•035
0•003
c
3-08
-2- 96
0-90
V
0-000009
0-0000002
this required a total of 14 multiplications by A. To obtain
the same accuracy by the method of Conjugate Gradients
required 16 multiplications by A, and by the method of
over-relaxation required 14 multiplications by A,
counting each iteration as equivalent to one multiplication, and using the optimum over-relaxation parameter
w = 107. As in practical problems the optimum w is
not known; this suggests that the method of this paper
may be faster.
cosf—— )• This result is used to determine the range
\n + 1/
(a, b) of the roots of A and draw the graph of g(A) in
this range.
In each example we give a table. The c-column gives
the coefficients cu c2, c3. The v-column following it gives
the values of the residuals for approximations obtained
by repeated iteration with c.
The computation was carried out on the Ferranti
Mercury Computer of the University of London Computer Unit. This is a machine with a single-length,
floating-point accumulator consisting of 10 binary digits
for the exponent and 30 binary digits for the argument.
All calculations were carried out in single-length arithmetic.
Well-conditioned matrices
We consider the case when W = —0-25.
The roots of A lie between 0-5 and 1 • 5. The graph of
g(A) in Fig. 1 corresponds to the first set c of Table 1.
The final solution obtained has eight figure accuracy;
203
Ill-conditioned matrices
We consider the case W= — 0-5; this is the onedimensional Laplace operator.
The roots of A lie between 0 01 and 1-99 giving a
condition number of about 200. Fig. 2 gives the graph of
g(A) corresponding to the second and third sets c of
Table 2. The graph of the larger set is not drawn to scale,
it oscillates between 3 and —82 in the range.
The final solution obtained has eight figure accuracy;
this required a total of 48 multiplications by A. The same
equation was solved by the method of over-relaxation
using the optimum over-relaxation parameter w = 1 • 74',
to get the same accuracy required the equivalent of 67
multiplications by A (counting each iteration as equivalent to one multiplication). The method of conjugate
gradients broke down in this case; this is because, for
ill-conditioned matrices, some of the intermediate
calculation requires accumulation of double-length
products, which is not easy on a single-length accumulator.
We observe that the vectors c fall into two categories,
small and large, which occur alternately. We could make
use of this "alternate directions" phenomenon to
economize on the calculations of new sets c. Thus by
Method for solving large sparse matrices
qft)
Fig. 3
Fig. 2
Table 3
Table 2
c
12
-20
8
C
3-74
3-74
9-90
V
3-57 0137
-3-63 0132
108 0126
c
5-22
-516
1-41
v
2-57
2-40
2-26
8618
-209-93
117-49
vV
C
91-29 0 00028 4-89
-157-85 00087 -4-64
66-88
1-23
C
-3-47
901
-3-81
0-56
72-28
6-48
-7-71
2-36
v
0000005
v
1-58
5-54
-5-32
1 -37
02
08
017
007
003
002
004
-2-62
14-62
-606
C
-2-63 0-000007
13-26 0000015
-5-83
Ill-conditioned non-definite matrices
We consider the case W = —0-6.
The roots of A lie between — 0-19 and 2-19, and one
of the roots is equal to 0 • 0085, giving a condition number
of about 250. Fig. 3 gives the graph of g(\) corresponding
to two consecutive sets c.
Thefinalsolution has eightfigureaccuracy; this required
a total of 98 iterations. This large number is partly due
to the fact that g(S) cannot be numerically small throughout the range (a, b), since the origin is within the range
and g(0) = 1; but it is probably mostly due to the illcondition of the matrix. The method was also tried for the
case W = — 10; convergence was a little faster because
the matrix is better conditioned. In this case the roots
are about equally spread on both sides of the origin;
in such cases there is some advantage in taking m even,
e.g. m = 4.
c
9-78
25-27
-11-66
-
00096 —5-87
00073
9-85
00318 -3-68
V
C
- 1 1 0 4 00002
4-33
13-21 00007 -5-20
- 3-76
1-57
9-76 0001
-14-74 0003
4-85
applying the second and third sets alternately it was
possible to achieve the same accuracy, but it required a
total of 57 multiplications by A.
102
0-91
1-26
0-29
118
0 004
0 006
0012
V
0000023
0000022
0000038
7 09 0-000004
-8-62
2-77
The two-dimensional Laplace operator
In this case the components of the vector x correspond
to the values JC (/, j) of a function of two variables which
satisfies the Laplace equation in discrete form at the 81
internal points of a 10 by 10 square mesh. The internal
points correspond to i, j = 1, 2, . . ., 9. The boundary
points correspond to i = 0 or 10, or j = 0 or 10; at
these points x was specified by the formula
x(i,j) = i3 - 3y2.
(11)
We denote x{i, j), x{i, j-l), x(i—l, j), x(i, j+ 1),
x(i -f 1, j) by x0, JC|, x2, x3, x4, respectively. Then equation (1) takes the form
204
Method for solving large sparse matrices
Complex matrices
The method of this paper was also tried on an 11 by 11
complex matrix of the type which arises in Power
Systems analysis in Electrical Engineering (Laughton &
Humphrey Davies). Iterative methods in this field are
particularly suitable because a good approximation is
usually known; because the matrices are very sparse and
can be very large; because the solution is usually required
only to about four or five significant figures; and finally
because the solution itself is part of an iterative procedure
which may involve altering the coefficients of the matrix
after each iteration.
The matrix studied was typical: it is symmetric not
Hermitian; in any row the sum of the elements is 0, or
nearly 0, and the diagonal element is the largest in
modulus; the smallest diagonal element is 0—7-69/ and
the largest is 7-960—44-277/.
The coefficients a y and b, of equation (7) are defined
as follows:
a
= r"r
u =
where the superfix H denotes the complex conjugate
transpose.
+* 4 ) = 0 i,j = 1,2,.. .,9. (12)
The operation y = Ax is obtained by setting x = 0
at the 40 boundary points and applying the substitution
at the 81 internal points. The residual r = b — Ax is
obtained by first setting the boundary values according
to (11), and using the substitution
r
(I 4 )
*0
0 =
at the 81 internal points.
Equations (12) are equivalent to the equation Ax = A
where A has the form I—L—U (Martin & Tee, 1961,
equation 1-2). To speed up convergence (A, b) are replaced by 04',*') where A' = / - {I-L)~lU, b' =
{I—L)~ '*. This is done by replacing the substitutions (13)
and (14) by the following:
Mo=i(«i+"2+*3+*4) Jo=^o—"o i,j= 1,2,...,9 (13')
9. (14')
*4) ro=uo—xo i,y=l,2
In (13') as in (13) the values of u and x are set to 0 at
the boundary points. In (14') as in (14) the values of u
and x are set at the boundary points according to equation
(11). The values of u obtained in (14') form the solution
one would obtain after a single Gauss-Seidel iteration
(op. cit., equation 2-2), and r0 is the corresponding displacement.
The initial approximation in Table 4 was x = 0.
c
4-31
-5-89
2-58
10-64
-24-60
13-73
Table 4
c
v
704
49-52 29-9
100 —11-36
-1514
5-78
4-46
6-60
348
137
72-2
0
0
0
0
0037
0026
0033
0036
c
2-85
-2-79
0-94
Table 5
V
0-73
0-033+0-235/
0-46
00125-0004/
-0 • 00007-0 00013/ 0-34
C
V
0-964+1-652/ 0 0 5
0-396-0-596/ 2-46
-0010-0004/
0031
3-58+2-45/ 00009
0-025+0-121/
0028
0-63-2-12/
0 004-0 • 002/
-0-00002-0 00004/ 0 0278 -0-033-0-003/
0-574
0-254
0 097
0 030
0013
The last approximation has a maximum error of one
in the fifth significant figure; this accuracy required 55
multiplications by A. To obtain the same accuracy by
the method of Gauss-Seidel required much more than
100 multiplications.
0000041
0-000022
The final solution obtained has eight figure accuracy;
this required a total of 55 multiplications by A'. The same
equations solved by the method of over-relaxation, using
the over-relaxation parameter w = 1-518, required the
equivalent of 44 multiplications. In this case the method
of this paper requires more multiplications than the
method of over-relaxation. In practical problems, however, one has only a guess at the optimum w; thus taking
w =* 1 -4 requires more than 70 iterations to obtain the
same accuracy. It would appear that in such cases the
method of this paper still has an advantage over the
method of over-relaxation.
205
Concluding remarks
The method of this paper seems to be an efficient way
for solving large sparse matrices or linear operators.
It compares favourably with existing iterative methods.
It also copes with complex or non-definite or unsymmetric matrices, whereas existing methods usually
require the matrix to be symmetric definite. As for other
iterative methods, its advantage over direct methods is
that it requires few iterations if an approximation is
already known or if only fewfiguresaccuracy is required
or if the matrix is well conditioned; it requires little
storage if the matrix is sparse, and indeed A may be
stored not as a matrix but as a program to apply a linear
operator.
The method is being tried on Atlas for matrices of
order several hundreds which arise in Electrical and
Method for solving large sparse matrices
Structural Engineering; this was not possible before
because of the relatively small size of the Mercury fast
store; I hope to report on the results in the near
future.
I am indebted to the Director of the University of
London Computer Unit, Dr. R. A. Buckingham, for
encouragement, and to my colleague, Dr. M. J. M.
Bernal, for frequent discussions.
References
D. W., and TEE, G. J. (1961). "Iterative methods for linear equations with symmetric positive definite matrix", The
Computer Journal, Vol. 4, p. 242.
LAUGHTON, M. A., and HUMPHREY DA VIES, M. W. "Numerical methods for Power System Load Flow studies". To be published
in the Proceedings of the Institution of Electrical Engineers.
MARTIN,
Note on the numerical solution of linear differential equations
with constant coefficients
By R. E. Scraton and J. W. Searl
If, therefore, fp can be expanded in the form
The numerical solution of the differential equation
ay" + by'+ cy = f{x)
fp = Ao + AlP + A2p2 + A3p> + .
(1)
where a, b, c are constants and f(x) is a numerically
specified function, can be obtained by a variety of
methods. For automatic computation the RungeKutta method is normally used, but this may be unstable.
The procedure described below provides an alternative
method which has been found satisfactory where the
Runge-Kutta method has failed.
Suppose that f(x) is tabulated at interval h. In the
usual notation, let/,, denote f(x0 + ph) so that equation
(1) may be written
d
a
Jy + bhdy + ch2y
dp2
it can be shown that the solution of (2) satisfying the
required initial conditions is
= J'oD - chW2(p)] + ahyiUi(p)
+
h2[A0U2(P)+Al.l\U3(p)
+ A2.2 \U4(p) +
(7)
Thus
h\Aou2
(2)
=
dp
(6)
+AlAlu3+A2.2\u4...)
(8)
and
It is assumed that y0 and y0 are known, so that a
procedure for determining j , and y[ makes it possible
to tabulate both y and y' in a step-by-step manner.
Let the sequence Ar be defined by the equations
hyl — —
(9)
where
um = UJl) =
(3)
2
aXr+2 + bh\r+1 + ch \r = 0,
The coefficients Ar of equation (6) may be taken from
any polynomial interpolation formula. For automatic
computation, Lagrangian formulae are appropriate,
and a four-point formula will be used as an illustration,
viz:
r> 0
and let
+2
mKFJ
~
m\
' (m + l)\ ' ( w + 2 ) !
...
(m + 1)! ' (m+2)!
(4)
It is easily verified that
2p*-p + 2)/0
f
0
p"-
2
if m = 0, 1 I
(5)
(10)
Equations (8) and (9) may then be written
206