0097-4943/88 $3.00 + 0.00
Copyright© 1988 Pergamon Press pic
Comput. Math. Applic. Vol. 15, No. 3, pp. 185-202, 1988
Printed in Great Britain. All rights reserved
SOLVING LARGE AND SPARSE LINEAR LEAST-SQUARES
PROBLEMS BY CONJUGATE GRADIENT ALGORITHMS
2
Z. ZLATEV 1 and H. B. NIELSEN
1
Air Pollution Laboratory, Danish Agency of Environmental Protection, Rise- National Laboratory,
DK-4000 Roskilde, Denmark
2
Institute for Numerical Analysis, Technical University of Denmark, DK-2800 Lyngby, Denmark
(Received 26 August 1987)
Communicated by E. Y. Rodin
Abstract-Let A e 91"' x" (with m ~ n and rank( A) = n) and b e rJtm x 1 be given. Assume that an
approximation to x =A th =(AT A )- 1 ATb is to be calculated. This problem is transformed into an
equivalent problem with a symmetric and positive definite matrix C = D -I (RT)- 1A rAR-ID -I where D
is a diagonal matrix and R is an upper triangular matrix. The conjugate gradient (CG) algorithm is applied
in the solution of the system of linear algebraic equations Cy = d (y = DRx, d = D -I (RT) -I A Tb ). If
D = R = I, then the CG algorithm is in fact applied to the system of normal equations ATAx = ATb and
the speed of convergence could be very slow. If D and Rare obtained by some kind of orthogonalization
(DR = QT A with Q e 91"' xn satisfying QT Q =I; D is often equal to I and Q is never used in the CG
algorithm) and if the calculations are performed without rounding errors, then C = I and the CG
algorithm converges in one iteration only. Even if the orthogonalization process is carried out with
rounding errors, the matrix C is normally close to the identity matrix I and the CG algorithm is quickly
convergent. However, for large m and n the orthogonal decomposition is an expensive process (both in
regard to storage and in regard to computing time). Therefore it may be profitable to calculate an
incomplete orthogonal decomposition. This is achieved by introducing a special parameter T, a
drop-tolerance, such that all elements which in the course of the computations become smaller in absolute
value than Tare removed. Numerical examples are given to illustrate that the CG algorithm applied to
system Cy = d and used with incomplete factors D and R is very efficient for some classes of problems.
It should be emphasized that matrix C is never calculated explicitly; the whole work is carried out by the
use of A, D and R only.
1. INTRODUCTION
Let A eqtmxn, bE qtmx 1, mE .AI and n E .AI be given. Assume that m
Consider the problem of finding a vector x e qtn x 1 that satisfies
X=
~
n and rank(A)
Atb,
= n.
(l)
where At is the pseudoinverse of A [1].
Since
rank(A) = n =>At= (AT A)- 1 AT,
(2)
it is clear that (l) is equivalent to the system of normal equations
AT Ax= AT b.
(3)
B=ATA!I.c=ATb.
(4)
Denote
Then vector x can be found by solving the system of linear algebraic equations
Bx =c.
(5)
Matrix B is symmetric and positive definite. Therefore the well-known method of conjugate
gradients (2-6] could be written, for system (5), as follows:
Algorithm 1
Step 1-Calculate r0 = c- Bx0 = AT(b- Ax0 ) (x 0 being an arbitrary starting approximation).
185
186
Step 2-Set i = 0,
Z. ZLATEV
£0
= rJro, Po=
and H. B.
NIELSEN
'o·
Step 3-Perform the ith iteration:
~i = £;, S; =Bpi= A TAp;,
Yi = p S;, ct.;= ~;/y;,
X;+ 1 =X;+ r:t.;p;,
ri+ 1 = r;- r:t.;Bp; = r;- r:t.;S;,
J
£;+1 =rJ+Iri+l•
/3;=£;+1/~;,
Pi+ 1 = r;+ 1 + /3;P;·
Step 4-Carry out a termination check and if
(a) the process converges,
(b) the accuracy required is not achieved and
(c) the number of iterations, i, is smaller than a prescribed (by the user) upper bound,
·
then set i •= i + 1 and go back to Step 3.
Step 5-Perform the output operations required by the user and STOP.
Certain convergence and accuracy criteria are necessary in Step 4 of the above algorithm. These
will be discussed in Section 4. It should be noted that it is not necessary to compute explicitly matrix
B and vector c when Algorithm 1 is in use.
Let K(B) = IIBIIIIB- 1 1 denote the condition number of B. It is well-known [e.g. 4] that when
K(B) is large, then the speed of convergence of the CG algorithm may be very slow. Since
K(B) = [K(A )F, it is clear that one should attempt to accelerate the speed of convergence in cases
where K(A) is not small. In this paper it will be shown that this can be done by a kind of
preconditioning of matrix B based on the use of factors of the original matrix A of the problem
to be solved, that are obtained by an orthogonalization process. The method so obtained will be
applied for large and sparse matrices A. In this latter case an incomplete orthogonal factorization
of matrix A could efficiently be calculated by the use of positive values of a special parameter called
the drop-tolerance [7].
The contents of the following sections can be outlined as follows. In Section 2 a transformation
of Problem (I) will be described. A modification of Algorithm I for the transformed problem will
be given in Section 3. Some stopping criteria, based on convergence and accuracy tests, will be
discussed in Section 4. Numerical experiments will be given in Section 5. In the last section, Section
6, some concluding remarks will be presented.
2. TRANSFORMATION OF PROBLEM {I)
Assume that three matrices Q, D and R are available and are such that
A =QDR+E
is satisfied with some perturbation matrix E
hold:
E [)fm
x ".
(6)
Assume also that the following relationships
(7)
(/" being the identity matrix),
D
R
E
[)f" x" A
E [)f" x" A
D is diagonal
A
rank(D) = n,
R is upper triangular
A
rank(R) = n.
(8)
(9)
Let
C =D- 1(RT)- 1 ATAR- 1D- 1,
(10)
y =DRx,
(II)
d=
n-
1 (RT)- 1 ATb
(12)
and consider the square system of n linear algebraic equations
Cy =d.
(13)
Solving large and sparse linear least-squares problems
187
The following result can be proved by the use of (6)--(12):
Theorem 2.1
If(6)--(12) are satisfied and ifrank(A) = n, then (i) the two problems defined by (1) and (13) are
equivalent, (ii) matrix C defined by (10) is symmetric and positive definite, and (iii) matrix C reduces
to the identity matrix (in ffli" x ") when E = 0.
Proof (i) The problem defined by (13) can be obtained from the problem defined by (1) in the
following way. Since rank(A) = n, (1) can be rewritten as (3) (see Section 1). Since D -I and R -I
exist [see the last conditions in (8) and (9)], (3) could be rewritten as
D -I(RT)-I AT A (R-ID -I )(DR)x
= D -I(RT)-I ATb
(14)
and (13) can easily be obtained by the use of (10)--(12). In a similar way (1) can be obtained from
(13).
(ii) The second assertion of the theorem is nearly obvious because matrix C from (10) can be
represented as
(15)
(iii) It is clear that
(16)
and
AT= RT DQT + ET => D-I(RT)-I AT= QT + D-I(RT)-I ET.
(17)
Multiply the last equalities in (17) and (16). The result is
c =I.+ QT ED-I R-I + D-I(RT)-IETQ + D-I(RT)-I ET ED-I R-I
(18)
and it is immediately seen that C = I. when E = 0. 0
The statements formulated in the following remarks are corollaries of Theorem 2.1.
Remark 2.1
Since C is symmetric and positive definite, the method of conjugate gradients can be applied to
solve (13).
Remark 2.2
The result of the theorem is true for any matrices Q, D and R that satisfy (7)--(9). In the following
sections, however, it will be assumed that these matrices are obtained by some kind of
orthogonalization. The code discussed in [7] and [8] will be used in the numerical demonstrations
in Section 5.
Remark 2.3
If Q, D and Rare obtained by some kind of orthogonalization of matrix A, then it should be
expected that C is not very ill-conditioned [even when B from (4) is] and that the method of
conjugate gradients applied for the system of linear algebraic equations (13) will converge fast.
Remark 2.4
Matrix Q is not used in the transformed problem (13). This is an important fact when matrix
A is a large and sparse matrix, because Q is normally rather dense even when A is very sparse.
In the code, which is used in the orthogonal factorization of matrix A, this fact is exploited and
the elements of matrix Q are never stored [7, 8].
3. A CONJUGATE GRADIENT ALGORITHM FOR THE
TRANSFORMED PROBLEM
Assume that R and D are available. Then a conjugate gradient algorithm for the problem defined
by (13) can be described as follows.
188
Z. ZLATEV
and H. B.
NIELSEN
Algorithm 2
Step 1-Ca1culate r0 =d-Cy0 =D- 1(RT)- 1AT(b-Ax0 ) (x0 being an arbitrary starting
approximation).
Step 2-Set i = 0, £0 = rJ ro, Po= ro.
Step 3-Perform the ith iteration:
l>; = f;, q; = R- 1D- 1p;, S; = CP; = D- 1(RT)-I AT Aq;,
'Y; = p"f S;, IX;= J;/y;,
Yi+ I = Y; + IX;p; =X;+ I = X;+ IX;q;,
r;+ 1 = r; -IX;Cp; = r; -IX;S;, f;+ 1= rf+ 1ri+ 1,
P;=£;+1/l>;, Pi+l =ri+l +P;P;·
Step 4-Carry out a termination check and if
(a) the process converges,
(b) the accuracy required is not achieved and
(c) the number of iterations, i, is smaller than a prescribed (by the user) upper bound,
then set i := i + 1 and go back to Step 3.
Step 5-Perform the output operations required by the user and STOP.
The following remarks are useful in connection with the algorithm described above.
Remark 3.1
Algorithm 2 is formulated in connection with (13) in the sense that the factors A, AT, R, RT and
D of matrix C are used in the formulation. However, it is seen that one can calculate
approximations, X;, to the exact solution of (1) directly (see Step 3).
Remark 3.2
Matrix AT A is never calculated explicitly when Algorithm 2 is in use.
Remark 3.3
On some computers it is vital to compute some of the inner products in double precision.
This is because the ordinary precision is too crude for most practical problems. The numerical
experiments presented in Section 5 were performed on an IBM computer with machine accuracy
6
5
£M = 16- ~ I0- • In these experiments all inner products were calculated in double precision (with
13
~ 2 x I0- 16 ). This facilitates the comparison with an earlier code LLSS01 [7, 8], which
£M = 16is also used for calculation of the factors R and D in Algorithm 2.
Remark 3.4
The convergence and the accuracy criteria used to stop the process (see Step 4 in the algorithm
described above) will be presented in the next section.
4. STOPPING CRITERIA
The iterative processes defined by Algorithms 1 and 2 have to be stopped when it is expected
that one of the following conditions is satisfied:
(i) the iterative process does not converge,
(ii) the iterative process converges, but the speed of convergence is too slow, or
(iii) the accuracy required by the user is achieved.
The basic principles used in the stopping criteria applied in connection with Algorithms I and
2 are discussed below.
4.1. Convergence control
Assume that all calculations in Algorithms 1 and 2 are carried out without rounding errors. Then
00
X;-+ x
as
i -+ oo
=
x = x0 +
L IX;q;
i= I
(19)
Solving large and sparse linear least-squares problems
189
when Algorithm 2 is in use (for Algorithm 1, q; should be replaced by p; ). A necessary condition
for the convergence of the series in ( 19) is
/;-+0 as
i-+ oo ( /;
=def
)
(20)
lloc;q;l[ .
Therefore the iterative processes are sometimes considered as convergent if
def
RATE;
= ];/];_ 1 ::::;; RATE::::;;
1(i =I, 2, ... ),
(21)
where RATE is given by the user.
However, the iterative process could be convergent even if (21) is not satisfied for some values
of i. This is illustrated in Table 1.
It should be mentioned that the accuracy required (10- 10 ) is achieved in 98 iterations. This is
quite a good result and for this problem Algorithm 1 is competitive with the other methods
compared in Section 5.
The example discussed above (the results being given in Table 1) shows clearly that the
requirement imposed by (21) should be relaxed if the user is prepared to accept a large number
of iterations.
We have found that the following strategy works well: Define
RMEAN; = 0.25
3
L RATE;-j
def
(RATEk =
RATE
for
k ::::;; 0)
(22)
and introduce two integer variables NBAD and NBAD1 that count the number of "failures" as
follows: if RATE;> RATE then N BAD is increased by .1 according to:
RATE;> 5 4 3 2 1.5 I RATE
.1 = 7 6 5 4 3 2
(23)
At the beginning both NBAD and NBAD1 are zero, and NBAD is reset to 0 if
RMAX;
def
=
max(RMEAN;, RMEAN;_ 1 , RMEAN;_ 2 , RMEAN;_ 3 ) <RATE.
(24)
NBAD1 is reset to 0 whenever RATE;::::;; RATE; i.e. when NBAD is not increased. NBAD1 is
increased by I when RATE;> RATE.
The convergence criterion used in connection with Algorithms I and 2 is defined by the use of
RMEAN;, NBAD and NBAD1.
Table I. The variation of RATE, in the
solution of Problem 7 in the set of George
eta/. [14] with RATE= 0.9 and by the use
of Algorithm I
RATE,
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
C.A.M.W.A. 15;J-.C
0.74
0.88
1.00
0.77
0.83
0.79
1.04
0.82
0.89
0.97
0.73
0.85
1.01
0.73
0.90
1.00
0.83
1.02
190
Z. ZLATEV
and H. B.
NIELSEN
Convergence criterion
The iterative processes in Algorithms l and 2 are considered as convergent (fori= 1, 2, ... ) as
long as
(25)
(RMEAN; <RATE) v [(NBAD ~ 40) "(NBAD1 ~ 5)].
4.2. Control of the speed of convergence
In fact an attempt to control also the speed of convergence is made in the convergence criterion
given at the end of the previous paragraph (by introducing the user-supplied variable RATE in the
criterion). If RATE = 1.0, then a pure convergence criterion is achieved. One should be careful in
the choice of RATE. If the value of RATE is close to 1, then the number of iterations performed
could be large (but not larger than the value of parameter MAX IT supplied by the user). If RATE
is small, then the iterative process could be terminated too early. In the experiments, the results
of which are presented in the next section, RATE= 0.9 was used in connection with both
Algorithms 1 and 2.
4.3. Accuracy check
Assume that the convergence criterion (25) is satisfied at all stepsj E {1, 2, ... , i}. Assume also
that it is desirable to stop the iterative process if
\\x -x;\\ < EPSJ\x \\,
(26)
where EPS < 1.0 is prescribed by the user. The example in Table 1 indicates that it is not
appropriate to replace the check (26) by
/; < EPS 1\x;\\.
(27)
This latter check is often used when iterative refinement is the method applied in the solution
of (5) or (13) [e.g. 7-9]. It is clear that a more stringent accuracy check is needed when Algorithms
1 or 2 are in use. This is so because, among other reasons, the convergence control is relaxed (see
the criterion given in Section 4.1). The following accuracy check is used in the experiments with
both Algorithms 1 and 2. The approximation, found at the ith iteration by either Algorithms 1
or 2, is declared as acceptable when
/;< (1.0- RMAX;)EPS\\x;\\A RMAX;< RATE
(28)
and the iterative process is terminated. Note that (28) is rather conservative when the iterative
process converges slowly. If at the ith iteration RMAX;;;:;: RATE, then no attempt to evaluate the
accuracy achieved is made. If RATE is close to 1, then the factor 1.0- RMAX; could be rather small
when RMAX; <RATE is satisfied. Nevertheless, many numerical experiments indicate that this
accuracy check performs rather well in practice. This will be illustrated in the next section.
4.4. Other stopping criteria
Several other stopping criteria are to be used. The principles on which these criteria are based
are very simple. The iterative process in Algorithm 2 is terminated if any of the following conditions
is satisfied:
(i) Y; is non-positive,
(ii) OC; II P; II I \1 X; II is very small,
(iii) OC; II s; 11/11 r; 11 is very small,
(iv) 1>;/£;+ 1 is very small, or
(v) RATE;> 2 11 •
Similar stopping criteria are used in connection with Algorithm 1.
We define a number a to be "very small" compared with another number b, when their machine
representations A and B satisfy fl (B + O.lA) =B. A similar criterion (without the factor 0.1) is
used in UNPACK [10]. Our definition is roughly equivalent with \a\~ 10£M\b\.
It should be mentioned that the relation used in (v) has been found experimentally after many
trials with badly scaled matrices). This relation could probably be improved.
Solving large and sparse linear least-squares problems
191
4.5. Some remarks
Several remarks are needed in connection with the choice of stopping criteria.
Remark 4.1
All stopping criteria introduced in this section are based on heuristics, but a long sequence of
experiments indicates that the stopping criteria selected work rather well on a wide range of
test-examples (some of the results will be presented in the next section).
Remark 4.2
In the implementation of the two algorithms the last residual vector, r;+ 1 , is available on exit
and the user could evaluate II r; + 1 11. This norm could be used in the judgement of the accuracy
achieved (as a supplement to the error evaluation calculated by (28). The experiments indicate that
the accuracy of the solution evaluated by the device described in Section 4.3 tends to be pessimistic,
while the estimation of the accuracy based on the residual normal tends to be optimistic; thus, the
exact error norm is often between these two estimations.
Remark 4.3
Some stopping criteria are based on the assumption that the computational process is performed
without rounding errors. Assume that EPS is not very small and that matrix A is not very
ill-conditioned. Then one should expect the iterative process with the stopping criteria selected to
be safely carried out because the truncation errors are dominating over the rounding errors.
5. NUMERICAL RESULTS
Two codes based on the algorithms described in Sections 1 and 3 as well as on the stopping
criteria discussed in Section 4 have been developed and compared with the code for solving linear
least-squares problems presented in [7] and [8]. Numerical results obtained in the comparison will
be reported in this section. Some information about the implementation of Algorithms 1 and 2 is
needed before the presentation of the numerical results.
5.1. Implementation of the conjugate gradient algorithm in the codes
We have implemented the two algorithms, Algorithms 1 and 2, in codes LLSSOX and LLSS02
respectively. Both codes will be described in detail elsewhere. In this connection it suffices to tell
that we use well-known sparse matrix techniques for storing the matrices. All matrices involved
in the computations are stored in single precision, while all operations on vectors are implemented
in double precision.
In LLSS02 we use the code LLSSOl [7, 8] to compute an approximate decomposition as given
by (6). The perturbation matrix E has contributions from round-off errors and from dropping small
elements during the factorization: all elements that are smaller in absolute value than a certain
drop-tolerance T ;:;:: 0 are neglected. This serves to reduce the number of nonzero elements during
the factorization process and, eventually, in the upper triangular matrix R.
The orthogonal factorization in LLSS01 is performed by the Gentleman version [11, 12] of the
Givens transformations; see also Remark 6.4.
Let NZ denote the number of nonzero elements in A, and NN an upper limit for the number
of nonzero elements during the factorization (including some elbow room; see [9, Chapter 2]). The
total storage needed for the arrays in the two codes is listed in Table 2. It should be noted that
NN = 3NZ is normally a good choice. Furthermore, one of the integer arrays of length NN
and several one-dimensional arrays in double precision are never needed simultaneously; an
EQUIVALENCE statement can therefore be applied to reduce the storage needed in LLSS02.
Table 2. Storage needed for arrays in our implementation of Algorithms I and 2
Algorithm
Code
Integer
I
LLSSOX
LLSS02
2NZ
2NZ+ 2NN +4(m +n)
2
Real
NZ+n
NZ+NN+2n
Double precision
2m +5n
2m +6n
192
Z. ZLATEV
and H. 8.
NIELSEN
5.2. Organization of the experiments
Several common rules were used in all experiments presented in the next paragraphs. These rules
are shortly described below.
Column equilibration is performed if m > n. Both column equilibration and row equilibration
are carried out if m = n. This is done only in order to facilitate the choice of the drop-tolerance.
The equilibration time is included in the factorization time for all runs.
The starting value of the drop-tolerance is T = 2- 3 when m >nand may beT= 2- 5 when m = n.
If the factorization fails (which may happen when too many elements are removed in the course
of the orthogonalization), then T is multiplied by 2- 5 and a new attempt to perform the
orthogonalization is carried out. Such a situation may occur several times. If the iterative process
fails, then Tis multiplied by 2- 3 and a new orthogonalization is performed to calculate more
accurate factors D and R. This may also happen several times.
The storage actually needed is, roughly speaking, given by COUNT (in the sense that NN in Table
2 can be replaced by COUNT). If COUNT, which is calculated by the code, is close to NN, then
the elbow room is small, and some garbage collections have probably been carried out during the
factorization [cf. 7], thus increasing the computing time. In the table below we demonstrate that
if we solve the same problem with different values for the drop-tolerance T, then normally COUNT
is smallest for the largest T.
If several failures (caused by large values of the drop-tolerance) are registered and if the
drop-tolerance is reduced as explained above, then the factorization time given in the table under
consideration is the sum of all factorization times and all iteration times (except the last one) used
in the run. The value of COUNT obtained with the last (and smallest) drop-tolerance is given in
the tables.
The accuracy requirement is the same in all tests: an attempt to stop the iterative process when
I x- xP 1 2 / llx lb < I0- 10 is carried out. In order to facilitate the comparison of the accuracy actually
obtained with the accuracy evaluated by the code the test-examples were constructed as follows:
the systems are consistent (r = b - Ax = 0) and the right-hand side vectors b are chosen so that
all components of the solution vectors are equal to l in all experiments.
The numerical results are obtained by the use of an IBM 3081 computer at NEUCC (Northern
Europe University Computing Centre, Lyngby, Denmark). The FORTVS compiler with
OPTIMIZE = 3 is specified and the computing times are measured in seconds. The nonzero
elements of all matrices (A, R, D) and some of the vectors are stored in single precison (machine
accuracy £M = 16- 5 ~ I0- 6 ). All inner products are accumulated and stored in double precision;
see also Remark 3.3. This means that all vectors involved are stored in double precision, which
is indicated in Table 2 (machine precision £M = 16- 13 ~ 2.0 X I0- 16 when double precision is
specified on IBM 3081).
5.3. The Waterloo set of test examples
This set consists of ten test examples with rectangular matrices. Test examples from this set have
been used by George and Heath [13], George eta/. [14], Liu [15] and Manneback [16]. All ten test
examples were run both with T = 0 (Table 3) and with T > 0 (Table 4). In these runs the code
LLSSO 1 [7, 8] is compared with the new code LLSS02. The starting solution is improved by the
use of iterative refinement (IR) in LLSS01, while the preconditioned conjugate gradient (CG)
algorithm discussed in this paper is applied in LLSS02.
It is clearly seen from Table 3 that if T = 0, then the use of the IR option is better than that
of the preconditioned CG option. The opposite is true when T > 0 is used; see Table 4.
The storage used could be reduced when T > 0 is specified. Sometimes the reduction is very
considerable; compare, for example, the values of parameter COUNT for the seventh problem
in Tables 3 and 4. It is also seen that the value of COUNT could be even smaller than NZ when
T > 0 (compare the values of NZ and COUNT for the second problem in Table 4).
The iteration times are very small (compared with the factorization times) in Table 2. Therefore
the total computing times obtained by the IR option are close to the total computing times needed
to solve the problems with the direct orthogonalization method (i.e. by setting
x 0 = R- 1D- 2 (RT)- 1 AT b
(29)
176
320
320
712
712
225
400
402
784
269
313
1033
1033
1850
1850
784
1444
1512
1488
900
I
2
3
4
5
6
7
8
9
10
NZ1
1557
4732
4719
8755
8636
3136
5776
7152
7040
4208
NZ
1557
4732
4719
8755
8636
3136
5776
7152
7040
4208
Fact.
time
1.05
1.54
1.48
7.31
7.44
9.38
36.08
5.30
7.21
11.64
COUNT
3662
5282
5213
15,835
16,196
14,663
40,195
10,619
14,432
17,827
Evaluated error
4.5 X 10-"
2.4 X 10- 13
9.3 X 10- 13
3.4 X 10- 12
5.2 x w- 13
5.2xl0- 13
8.9 X 10-"
1.2xl0-"
1.5xl0- 12
2.2 x w-"
Exact error
5.o x w-~•
1.2 x w-~•
4.7 x w- 14
1.6xl0- 15
5.2 x I0- 14
4.7 X 10-!6
6.2 x w-~•
6.7 x w- 16
7.8 x w- 16
4.7xl0- 12
lters
3
3
5
3
4
3
3
3
3
4
Time
0.06
0.12
0.18
0.25
0.32
0.13
0.27
0.17
0.24
0.19
0.06
0.14
0.18
0.30
0.37
0.15
0.32
0.20
0.28
0.22
Time
2
2
4
3
4
3
3
3
3
4
lters
Exact error
2.1 x w- 1•
2.0 x w- 1•
5.9xl0- 15
1.8xl0-16
1.5xl0- 15
2.0 X 10- 16
1.9 x w- 1•
2.2 X 10- 16
2.7 X 10- 16
9.8xl0- 13
4.3 X
I.J X
2.9 X
2.3 X
8.3 X
2.2 X
2.3 X
1.4 x
2.1 x
4.1 x
w-"
10-"
10-14
10- 11
IO-I3
10- 14
10- 14
10· 13
w- 14
w- 14
Evaluated error
Preconditioned conjugate gradients
313
1033
1033
1850
1850
784
1444
1512
1488
I
2
3
4
5
6
7
8
9
10
900
m
Problem
176
320
320
712
712
225
400
402
784
269
n
1557
4732
4719
8755
8636
3136
5776
7152
7040
4208
NZ
1380
4064
4719
7871
8609
2651
4897
6062
6076
4201
NZ1
2021
4065
4727
8193
11,535
3210
6190
7564
8084
16,496
COUNT
0.41
0.65
1.43
2.61
4.56
0.86
1.85
1.85
1.76
14.11
Fact.
time
67
II
13
30
30
29
294
II
0.39
0.98
0.72
1.31
1.44
18.75
0.50
lters
1.05
Time
9.6xl0-"
does not converge
1.8xl0- 12
does not converge
9.2xto-"
1.2 x to-"
1.3 x to-"
2.2 x to-"
3.6 x w-"
5.2 x to-"
Exact error
Iterative refinement
X
10-ll
6.7 x to-"
1.0 x to-"
7.5 x w-"
4.8 X to·· II
9.8 x w-"
l.lxto·"
2.4
7.6xl0-"
Evaluated
error
0.32
1.61
0.25
2.11
0.83
0.39
0.70
0.80
1.69
0.39
Time
18
45
6
28
10
15
15
15
24
8
lters
4.2
2.5
2.2
9.8
1.3
1.0
6.6
1.0
9.7
4.1
x w-"
x w- 14
x w-l3
x to-"
x w- 13
X 10 II
x w-"
x w--"
x to-"
x w- 13
Exact error
3.4xlo·"
4.9xl0- 11
5.8 x w-"
9.( X to· II
8.5xl0- 12
8.( X 10 II
5.9 x to-"
9.( X 10- II
3.4 X 10· II
9.8 X 10- l3
Evaluated error
Preconditioned conjugate gradients
Table 4. Some results obtained in runs with the Waterloo test examples with T > 0 and EPS = 10 -lo. The parameters used are the same as those used in Table 3.
The use of the starting drop-tolerance T = 2- 3 was successful for all problems except problems 3, 5 and 10; the latter problems were solved successfully with T = 2- 13 ,
T = 2-• and T = 2- 18 respectively
n
m
Problem
Iterative refinement
Table 3. Some results obtained in runs with the Waterloo test examples with T = 0 and EPS = 10- 10• The parameters used are: m = number of rows, n number of
columns, NZ =number of nonzero elements in matrix A, NZ1 =number ofnonzeros after a scan with the drop-tolerance T (in this experiment NZ1 = NZ because T = 0),
COUNT= the maximal number of occupied locations in array AQDR, Fact. time= the factorization time, Time= the iteration time, lters =the number of iterations
made
'()
w
"'
~
a
a"
'0
"'
~
1l
~
~
.
~
§'
.g
5.
I»
~
OQ
~
::s
g>
194
Z. ZLATEV
and H. B.
NIELSEN
and accepting x 0 as a solution without any attempt to carry out an iterative process). These total
times are given in the second column of Table 5. The total computing times obtained by the
preconditioned CG algorithm are given in the fifth column of Table 5. The results obtained by the
pure CG algorithm (Algorithm 1 as implemented in LLSSOX) are given in the last three columns
of Table 5.
The "pure" orthogonalization method (i.e. the use of IR with T = 0) performs better than the
preconditioned CG algorithm for three examples; problems 2, 3 and 10. The reason is the fact that
the runs with the starting value T = 2- 3 failed and the drop-tolerance was reduced when problems
3 and 10 were solved (two and three times respectively). The attempt with the first drop-tolerance
(T = 2- 3 ) was successful for the second problem, but (i) the number of the removed elements is
not very large (compare the values of COUNT for this problem) and (ii) the number of iterations
for the preconditioned CG algorithm (45) is much larger than that for the "pure" orthogonalization
method (3). However, note that the increase of the computing time (for the examples where the
preconditioned CG algorithm does not perform better than the "pure" orthogonalization method)
is not large.
The preconditioned CG algorithm performs better than the "pure" orthogonalization method
for the other seven problems. For the fifth problem this is so in spite of the fact that the attempt
to calculate an orthogonal factorization with T = 2- 3 failed and the orthogonalization was
performed with T = 2- 8 • It is seen that the reduction in computing time can be very significant
when the preconditioned CG algorithm is used; for the seventh problem a reduction by a factor
of about 15 is registered. It must be emphasized here that the results obtained by the preconditioned
CG algorithm given in Table 5 are not the best possible. For example, the best computing time
obtained in the solution of the tenth problem is 2.93 s (found when no equilibration is performed
and T = 2 is used). However, it is not very easy to find the best value of the drop-tolerance.
Therefore, a common strategy (equilibration+ starting drop-tolerance T = 2- 3 ) is used in all
experiments. Of course, if a simulation process where many problems with the same matrix or with
similar matrices are to be solved is to be carried out, then it may be profitable to attempt to
determine an optimal (or nearly optimal) value of the drop-tolerance at the beginning of the
simulation process and then to solve the remaining problems using the value of the drop-tolerance
found.
The computing times obtained with the pure CG process (Algorithm I implemented in code
LLSSOX) are rather good when the process is convergent (or, more precisely, when the speed of
convergence is not very slow). It should be mentioned that the stringent accuracy requirement
(EPS = I0- 10 ) is favourable for the other two algorithms (first and foremost, for the "pure"
orthogonalization method, but also for the preconditioned CG algorithm). If EPS = 10- 3 is used,
then the results for the pure CG algorithm will become the best when this algorithm converges.
However, the pure CG algorithm is not convergent (or slowly convergent) in 50% of the problems
in the Waterloo set of test examples.
The storage needed for the pure CG algorithm is smaller than the storage needed for the other
two methods. In this case the factors D and R are not needed and, therefore, no orthogonalization
is carried out (this is expressed, in Table 5, by setting COUNT= 0). The storage needed when the
preconditioned CG algorithm is specified is normally smaller than the storage needed when the
Table 5. Comparison of the total times, the numbers of iterations and the values of COUNT obtained in the runs of the test examples
from the Waterloo set by three different algorithms
"Pure" orthogonalization method
Problem
I
2
3
4
5
6
7
8
9
10
Total time
1.11
1.66
1.66
7.56
7.76
9.51
36.35
5.47
7.45
11.83
Pure CG algorithm
Preconditioned CG algorithm
lters
COUNT
Total time
lters
COUNT
Total time
3
3
3662
5282
5213
15,835
16,196
14,663
40,195
10,619
14,432
17,827
0.73
2.26
1.78
4.72
5.39
1.25
2.55
2.65
3.45
14.50
18
45
6
28
10
15
15
15
24
8
2021
4065
4727
8193
11,535
3210
6190
7564
8084
16,496
1.34
5
3
4
3
3
3
3
4
does
does
does
does
1.41
2.94
3.64
9.27
does
lters
133
not converge
not converge
not converge
not converge
82
98
103
234
not converge
COUNT
0
0
0
0
0
Solving large and sparse linear least-squares problems
195
"pure" orthogonalization method is applied (or, in other words, when LLSSOI with T = 0) is in
use. The reduction could be very significant; see the values of COUNT obtained in the solution
of the seventh problem by these two options.
The numerical results indicate that no method is best for any problem and for any accuracy
requirement. It is also clear that the preconditioned CG algorithm leads to very significant
reductions both in computing time and in storage for some problems (compared with the "pure"
orthogonalization method).
5.4. The Harwell set of test examples
This set consists of 37 test examples, but only 5 of them are with rectangular matrices (the others
being with square matrices). Some information about these test examples is given in Ref. [18].
Matrices of this set have been used in many studies [e.g. 16, 18-20]. All tests from the Harwell set
were run in a similar manner as the tests of the Waterloo set. The results are presented in Tables
6 and 7.
It is seen from Table 6 that even for T = 0 the use of the preconditioned CG option is quite
competitive with the IR option. The same is true for the tests of the Waterloo set (see previous
paragraph). However, for some tests from the Harwell set the preconditioned CG algorithm is
slightly better than the IR option, while for all test examples of the Waterloo test the IR option
was slightly better than the preconditioned CG algorithm. For five tests from the Harwell set (SHL
0, SHL 200, SHL 400, STR 0, BP 0) the codes determined the exact solution; this means that the
residual vectors are exact zero vectors and the iterative processes (both the IR and the CG) are
practically not performed; the calculations being terminated after the determination of the first
residual vector).
The results for T > 0 are given in Table 7. In this case the starting value of the drop-tolerance
is T = 2- 3 if m > n or if
(m =n)
A
(max (jaiij)Jla;;l
I
~I
Vi)
A
(a;;=/; OVi)
:1G;.j~n
and T = 2-s otherwise. It is seen that for the strategy of varying the drop-tolerance chosen (see
also section 5.2) the preconditioned CG algorithm normally performs better than the IR algorithm;
for many tests the IR does not converge.
The total computing times, the iterations carried out and the values of COUNT obtained by the
IR option with T = 0 are given in the second, third and fourth columns of Table 8. The
corresponding figures obtained by the preconditioned CG algorithm with T > 0 are given in the
fifth, sixth and seventh columns of Table 8. The corresponding results obtained by the pure CG
algorithm are given in the last three columns of Table 7.
The "pure" orthogonalization method (the use of IR with T = 0) performs better than the
preconditioned CG algorithm for many test examples. In some cases this is so because the problems
solved are very small (IBM 32, WILL 57). In other cases (see the results for the problems of classes
STR and BP) this is so either because the attempt with the starting value of the drop-tolerance
is not successful (and the computations were repeated with a smaller value of the drop-tolerance)
or because the number of iterations is large (even when the run with the starting value of the
drop-tolerance was not successful and the drop-tolerance was reduced). It should be noted that
the increase of the computing time, for the examples where the preconditioned CG algorithm does
not perform better than the "pure" orthogonalization method, is normally not large.
The preconditioned CG algorithm performs better than the "pure" orthogonalization method
for all five test examples with rectangular matrices (i.e. when least-squares problems are to be
solved). The reduction both in computing time and in storage is sometimes very significant. For
problem ASH 608 the computing time is reduced by a factor of 6.57 and the value of COUNT
is reduced by a factor of 4.62 when the preconditioned CG algorithm is in use. A very significant
reduction is also achieved in the solution of the problems of class FS although the starting value
of Twas not successful for three of them; which means that the computing times for these three
examples are sums of two computing times: the computing time with the starting value of T, plus
the computing time with the reduced value of T. A rather significant reduction in computing
LUNDA
LUND B
ERIS 1176
GENT 113
IBM 32
CURTIS 54
WILL 57
WILL 199
ASH 292
ASH 85
ARC 130
SHL 0
SHL 200
SHL 400
STR 0
STR 200
STR 400
STR 600
BP 0
BP 200
BP 400
BP600
BP800
BPIOOO
BPI200
BP 1400
BP 1600
ASH 219
ASH 958
ASH 331
ASH608
ABB 313
FS 541-1
FS 541-2
FS 541-3
FS 541-4
JPWH 991
Name
of the
problem
n
147
147
1176
113
32
54
57
199
292
85
130
663
663
663
363
363
363
363
822
822
822
822
822
822
822
822
822
85
292
104
188
176
541
541
541
541
991
m
147
147
1176
113
32
54
57
199
292
85
130
663
663
663
363
363
363
363
822
822
822
822
822
822
822
822
822
219
958
331
608
313
541
541
541
541
991
4285
6027
~285
2249
2441
18,522
655
126
291
281
701
2208
523
1282
1687
1726
1712
2454
3068
3157
3279
3276
3802
4028
4172
4534
4661
4726
4790
4841
438
1916
662
1216
1557
4285
4285
NZ
NZ1
0.0 2249
0.0 2241
0.0 18,552
0.0
655
0.0
126
0.0
291
0.0
281
0.0
701
0.0 2208
0.0
523
0.0 1282
0.0 1687
0.0 1726
0.0 1712
0.0 2454
0.0 3068
0.0 3157
0.0 3279
0.0 3276
0.0 3802
0.0 4028
0.0 4172
0.0 4534
0.0 4661
0.0 4726
0.0 4790
0.0 4841
0.0
438
0.0 1916
0.0
662
0.0 1216
0.0
1557
0.0 4285
0.0 4285
0.0 4285
0.0 4285
0.0 6927
T
5662
5426
48,202
1428
347
593
468
2881
9571
1339
2836
1687
1726
1712
2454
6442
7050
8735
3276
8447
28,914
25,774
30,892
44,482
49,820
43,314
35,802
1176
10,071
2252
5702
3579
21,192
21,161
23,240
23,632
233,480
COUNT
1.76
1.65
18.46
0.15
0.04
0.07
0.05
0.42
2.36
0.22
0.36
0.30
0.23
0.26
0.15
0.81
0.91
1.31
0.30
0.83
3.95
3.71
4.95
9.04
10.77
8.49
6.41
0.25
4.35
0.56
2.11
0.87
7.29
7.19
8.58
9.68
334.55
0.10
0.10
0.79
0.14
0.01
O.oi
0.01
0.07
0.14
0,03
0.06
0.01
0.01
0.01
0.01
0.18
0.18
0.31
0.01
0.19
0.66
0.61
0.57
0.93
2.23
0.91
0.79
0.02
0.10
0.04
0.06
0.06
0.30
0.40
0.52
0.41
3.00
3
3
3
12
2
2
2
4
3
3
3
0
0
0
0
4
4
6
0
3
5
5
4
5
II
5
5
3
3
3
3
3
3
4
5
4
4
lters
5.6 X
1.3 x
5.o x
1.3x
3.6 x
2.7 x
4.3 x
2.3 x
17
16
10- 16
10"" 15
10-' 4
10-"
10- 14
10-"
10
w-
0.0
0.0
0.0
0.0
5.4 x 10-' 4
3.4 x 10-' 4
2.8 X 10· 13
0.0
4.4 x w-' 4
2.3xl0- 13
u x 10-"
7.6 x w·' 4
7.4 x 10-"
7.3 x w-"
5.8 x w-"
6.8 x w-' 4
4.2 x 10-"
3.4 x 10-' 4
6.9 x w-"
9.2 x w-"
3.2x10- 12
0.0
0.0
0.0
0.0
5.1 x 10-"
1.2 X 10· 13
3.2x10- 12
0.0
3.5 x w-"
1.2 X 10 l2
5.2 x 10·"
5.3 X 10· II
2.5 x w-"
4.7 X JO .. II
2.4 x w-"
4.5 x 10-"
7.9xl0- 15
8.7 x w-' 4
1.6 x w-' 4
4.9 x 10-' 4
8.4 x 10_"
2.3 x w-' 4
8.4 x 10-' 4
Exact error Evaluated error
2.7 x 10-' 4
4.0 X 10 II
2.0 x w-' 4
3.8 x w- 11
8.4 x 10-"
3.7 x w-"
6.5 x w-"
3.3xto·"
4.4 x 10-16
4.2 X 10. -II
4.4 x 10- 11
4.7 x w-' 6
1.2 x 10-"
2.1 x w- 11
I.Oxi0- 13
5.4 x 10-"
6.9 X 10 .J5
4.7 x 10-' 6
1.3xl0- 15
6.9 x w-"
1.9 x w-' 4
2.4 x w-"
Iterative refinement
Time
0.11
0.11
0.93
O.o7
O.oi
0.02
O.oi
0.09
0.13
0,03
0.05
0.01
0.01
0.01
0.01
0.20
0.17
0.24
0.01
0.24
0.61
0.57
0.66
0.86
1.15
0.84
0.74
0.02
0.09
0,03
0.05
0.05
0.26
0.36
0.48
0.37
2.65
= the drop-tolerance (T = 0
3
3
3
5
2
2
2
4
2
2
2
0
0
0
0
4
3
4
0
3
4
4
4
4
5
4
4
2
2
2
2
2
2
3
4
3
3
lters
3.0 x w-"
4.1 x w-"
6.8 x w- 16
6.1 x w-"
2.1 x 10-' 6
2.8 x w-' 6
3.o x w-' 6
1.6 x w- 14
3.8 x w-' 6
3.0 x w-' 6
1.4 x w-"
0.0
0.0
0.0
0.0
8.2 x 10-"
2.0 x w-"
3.2 x w-' 4
0.0
7.4 x 10-"
1.6x10-14
3.5 x 10-' 4
4.6 x w-"
9.1 x 10·' 4
3.1 x w-"
5.5 X 10 -l 4
1.9 X to· 14
1.3 x w-'6
1.9 x w-"
1.8 x w- 16
1.9 x w·"
4.9 X 10"" 16
4.8 x w-' 6
3.5 x w-"
2.4xl0- 14
4.9 x w- 14
1.6x10- 15
12
X 10
x w-"
x 10-' 4
X 10 II
x w-"
x w-"
x w-"
x 10- 14
X 10 II
X JO- II
x w- 11
0.0
0.0
0.0
0.0
9.4 x w- 16
8.9 X 10 12
3.5 X 10 l2
0.0
3.1 X (0 l3
2.5 x 10-"
1.4 x 10_"
1.0 X 10 l3
9.9 x 10-"
1.5 x 10_"
4.6 X 10 l2
1.3 X 10 l2
2.3 X 10 II
8.4 X 10 II
3.4 X 10 II
8.3 X 10 ·II
5.5 x 10-ll
9.9 X 10- II
6.3 x 10-"
1.4 X 10· II
5.4 X 10 II
9.4x 10"" 12
4.7
3.1
1.2
1.3
3.0
2.9
1.6
1.8
2.7
1.0
2.1
Exact error Evaluated error
Preconditioned conjugate gradients
EPS = 10 -w. The parameters used are as for Table 3, with T
in this experiment)
= 0.0 and
Fact. time Time
Table 6. Some results obtained in runs with the Harwell test examples with T
'1:)
"'z
!;;
z
;;;
~
::r::
P-
::s
I»
<!
>
;;j
N
r-
N
0\
m
147
147
1176
113
32
54
57
199
292
85
130
663
663
663
363
363
363
363
822
822
822
822
822
822
822
822
822
219
958
331
608
313
541
541
541
541
991
Name of
the problem
LUNDA
LUNDB
ERIS 1176
GENTI13
IBM 32
CURTIS 54
WILL 57
WILL 199
ASH 292
ASH 86
ARC 130
SHLO
SHL200
SHL400
STRO
STR200
STR400
STR 600
BPO
BP200
BP400
BP600
BP800
BP 1000
BP 1200
BP 1400
BP 1600
ASH 219
ASH 958
ASH 331
ASH 608
ABB 313
FS 541-1
FS 541-2
FS 541-3
FS 541-4
JPWH 991
NZ
2249
147
2241
147
1176 18,552
113
655
126
32
291
54
281
57
701
199
2208
292
523
85
1282
130
1687
663
1726
663
1712
663
2454
363
3068
363
3157
363
3279
363
3276
822
3802
822
4028
822
4172
822
4534
822
4661
822
822
4726
4790
822
4841
822
438
85
1916
292
662
104
1216
188
1557
176
541
4285
4285
541
4285
541
4285
541
6027
991
n
NZ1
z-w
z-3
z-3
z-3
z-3
z-3
z-5
z-w
z-w
z-3
1423
1749
13,252
655
98
222
213
665
1648
393
327
1437
1451
1426
1126
1406
1426
2833
2895
3308
3976
-1o
4124
2
4207
2 --lo
-1o
4318
2
-1o
4646
2
4758
-1o
4805
2
377
1720
592
1097
1395
759
2938
3120
2--10
3042
4886
z-5
z-5
z-5
z-"
z-5
z-5
z-5
z-5
z-5
z-5
z-5
z-5
z-5
z-5
2-5
z-5
z-5
2 -1o
z-5
z-5
z-w
T
5710
6220
24,824
1337
176
346
289
1875
4062
625
361
1437
1451
1428
1126
2819
3010
8033
2895
6912
24,574
23,343
21,692
26,545
41,404
36,006
34,716
379
1885
678
1234
1998
759
9405
11,838
10,159
43,965
COUNT
2.32
2.29
9.53
0.20
0.02
0.04
O.o3
0.20
0.83
0.08
0.05
0.14
0.14
0.16
0.09
0.33
0.31
2.13
0.20
0.67
4.60
4.00
5.57
8.12
10.06
3.15
9.14
0.05
0.32
0.10
0.20
0.38
0.11
1.34
1.86
1.40
32.18
Fact.
time
1.40
0.33
0.86
0.10
0.23
0.10
0.15
7.59
0.51
14
II
9
178
0.41
0.08
0.09
4.12
1.7x10- 12
1.1 X 10-12
6.7 X 10- 12
(.7
2.7
3.3
9.9
X
X
X
X
X
X
X
10-11
10-ll
10-ll
10-ll
10-l2
10-ll
10-ll
9.7 X 10-ll
1.8x10- 11
5.3 X 10-ll
1.1 X 10 -II
3.7 X 10-ll
7.7 X 10-ll
4.1 X 10-ll
3.8 X 10 -II
9.9 X 10-ll
8.6x10- 11
4.7 X 10· II
5.3 X 10-ll
6.4 X IQ" II
7.0 X 10· II
4.5 X 10· II
6.3 X 10 II
7.6 X 10· II
1.9x10-ll
4.2x10·-ll
3.9 X 10· II
7.6 X 10-l2
4.4 X 10-ll
2.1 X 10 II
4.2 X 10-ll
3.2 X 10 II
8.4 X 10 12
2.4 X 10 II
5.9 X 10 12
3.2 X 10-ll
3.8 X 10· II
4.3 X 10 II
5.5 X 10-ll
3.9 X 10 II
1.7 X 10 II
4.3 X 10 II
2.5 X 10- 11
9.9 X 10· II
7.9
8.8
1.8
Evaluated error
10 II
X 10··l2
12
X 10X 10- l2
X 10 J3
12
X 1013
X 105.4 X 10-IO
4.1x10- 12
4.7 X 10- 12
5.7 X 10- 13
9.9 X 10' 15
1.0 X 10· 14
3.2 X 10- 15
2.6 X 10- 12
2.8 X 10··l 2
2.1 X 10 12
3.2 X 10- 12
1.6 X 10 12
1.6 X 10'' 12
3.4 X 10··l 3
3.0 X 10- 12
1.2x10·11
1.8 X 10-IO
1.3 X 10- 8
7.3 X 10 12
2.5 X 10 l2
2.3 X 10· 13
1.1 X 10-12
2.3 X 10- l2
2.0 X 10- 12
9.5 X 10- 12
6.9xl0· 13
1.5 X 10 12
9.3 X 10 12
1.8 X 10- 12
5.0 X 10-ll
X
40
52
22
12
7
7
7
62
9
8
7
3
4
4
12
34
53
17
15
90
14
17
42
60
33
28
20
10
9
9
9
20
7
10
37
13
131
1.26
1.71
4.12
0.15
O.o3
0.04
0.04
1.01
0.30
O.o7
0.09
0.10
0.13
0.13
0.30
1.14
1.82
0.89
0.68
5.68
1.76
2.07
4.86
7.92
6.41
4.64
3.25
0.07
0.20
0.08
0.13
0.34
0.29
0.75
2.74
0.92
25.45
1.7
5.0
8.4
7.6
7.9
2.3
9.3
Exact error
lters
Evaluated error Time
does not converge
does not converge
does not converge
does not converge
4.2x 10- 13
4.3 X 10-l2
7.3 X 10- 13
does not converge
1.8 X 10- 12
Exact error
does not converge
does not converge
6.7 X 10-ll
1.1 X 10-ll
24
does not converge
does not converge
does not converge
9.9 X 10-ll
5.6 X 10- 12
203
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
4.8 X 10-ll
6.3 X 10- 12
16
7.2 X 10-ll
7.2 X J0- 12
12
9.5 x I0- 11
9.8 X 10- 12
12
7.8 X 10-ll
7.0 X 10- 12
12
does not converge
7.6 X 10-ll
3.1 X 10' 12
10
4.2 X 10 -II
7.4xl0·12
13
does not converge
3.7 X 10-ll
5.8 X 10· 12
22
does not converge
10
9
9
lters
0.03
0.05
0.04
Time
10
used are as in Table 3, with T =the drop-tolerance (by
Table 7. Some results obtained in runs with the Harwell test examples with T > 0 and EPS = 10- • The parameters
which the problem was successfully solved)
Preconditioned conjugate gradients
Iterative refinement
....
Pl
"'....
~
.....,
"'
g:
3"'
't:l
....
0
"'
til
Pl
"'
Pl
"
t=-
10
S'
~
~
10
::s
p.
"'
(JQ
....iil
(JQ
::s
s.
0
rl)
198
Z. ZLATEV
and H.
B. NIELSEN
Table 8. Comparison of the total times, the numbers of iterations and the values of COUNT obtained in the runs of the test examples from
the Harwell set by three different algorithms
Name
of the
problem
LUND A
LUND B
ERIS 1176
GENT 113
IBM 32
CURTIS 54
WILL 57
WILL 199
ASH 292
ASH 85
ARC 130
SHL 0
SHL 200
SHL 400
STR 0
STR 200
STR 400
STR 600
BP 0
BP 200
BP 400
BP 600
BP 800
BP 1000
BP 1200
BP 1400
BP 1600
ASH 219
ASH 958
ASH 331
ASH608
ABB 313
FS 541-1
FS 541-2
FS 541-3
FS 541-4
JPWH 991
"Pure" orthogonalization method
Total time
Iters
1.86
1.75
19.25
0.29
0.05
0.08
0.06
0.49
2.50
0.25
0.42
0.31
0.24
0.27
0.16
0.99
1.07
1.62
0.31
1.02
4.61
4.32
5.52
9.97
13.00
9.40
7.20
0.27
4.45
0.60
2.17
0.93
7.59
7.59
9.10
10.09
337.55
3
3
3
12
2
2
2
4
3
3
3
0
0
0
0
4
4
6
0
3
5
5
4
5
II
5
5
3
3
3
3
3
3
4
5
4
4
COUNT
5662
5426
48,202
1428
347
593
468
2881
9571
1339
2836
1687
1726
1712
2454
6442
7050
8735
3276
8447
28,914
25,774
30,892
44,482
49,820
43,314
35,802
1176
10,071
2252
5702
3579
21,192
21,161
23,240
23,632
233,480
Preconditioned CG algorithm
Total time
3.58
4.00
13.65
0.45
0.05
0.08
O.D7
1.21
1.13
0.15
0.14
0.24
0.27
0.29
0.39
1.47
2.13
3.02
0.88
6.35
6.36
6.07
10.43
16.04
16.47
12.79
12.39
0.12
0.52
0.18
0.33
0.72
0.40
2.09
4.60
2.32
57.63
Iters
40
52
22
12
7
7
7
62
9
8
7
3
4
4
12
34
53
17
15
90
14
17
42
60
33
28
20
10
9
9
9
20
7
10
37
13
131
COUNT
5710
6220
13,252
1337
176
346
289
1875
4062
625
361
1437
1451
1428
1126
2819
3010
8033
2895
6912
34,574
23,343
21,692
26,545
41,404
36,006
34,716
379
1885
678
1234
1998
759
9405
11,838
10,159
43,965
Pure CG algorithm
Total time
4.33
0.03
0.05
0.05
0.28
0.09
0.25
0.18
0.46
0.17
0.32
0.78
0.67
Iters
does not converge
does not converge
48
does not converge
II
14
14
does not converge
19
17
30
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
does not converge
39
34
29
34
78
24
does not converge
does not converge
does not converge
does not converge
COUNT
0
0
0
0
0
0
0
0
0
0
0
0
0
time is achieved in the solution of the difficult, for the "pure" orthogonalization method, problem
JPWH 991.
It should be emphasized here that the results obtained with the preconditioned CG algorithm
are not the best possible. The best results, obtained with an optimal value of the drop-tolerance
will nearly always be better than the results obtained with the "pure" orthogonalization method.
However, it may sometimes be difficult to find such an optimal value. Therefore, the common and
somewhat crude strategy for specifying and varying the drop-tolerance, which is outlined in Section
5.2, is used in connection with all examples in the previous paragraph and in this paragraph. It
should also be emphasized that the stringent accuracy requirement (T = IQ- 10 ) is clearly favourable
for the "pure" orthogonalization method. If the accuracy requirement is not so stringent, then the
performance of the preconditioned CG algorithm will be improved considerably for many of the
examples in which the number of iterations is large when EPS = IQ- 10 is in use, while the
performance of the "pure" orthogonalization method will remain practically the same.
Often the pure CG algorithm performs best when its convergence is sufficiently quick. However,
the convergence was prohibitively slow for many examples from the Harwell set (the expression
"does not converge" being used in Table 8 in all such situations).
The same conclusions as in the previous paragraph can be drawn with regard to the storage used:
the pure CG algorithm performs best (when it converges quickly, of course), while the preconditioned CG algorithm is normally better than the "pure" orthogonalization method (the first two
examples being exceptions within the Harwell set of test examples). The reduction of storage
obtained by the preconditioned CG algorithm may be very large; see the values of COUNT for
the last five test problems in Table 8.
Again, as in Section 5.4, the numerical results indicate that no method is best for any problem
and for any accuracy requirement.
199
Solving large and sparse linear least-squares problems
5.5. Tests with matrix generators
Several matrix generators were developed at RECKU (Regional Centre at the University of
Copenhagen) [21). Many tests with one of these matrix generators, MATRF2, were carried out.
MATRF2 is a subroutine which generates matrices depending on five parameters: m, n, c, r and
rx. The numbers of rows and columns in the desired matrix are specified by m and n respectively.
The positions of the nonzero elements could be varied by varying the value of c. The number of
nonzero elements is determined by the values of r and m; NZ = mr + 110. The magnitude of the
nonzero elements depends on the value of rx; more precisely,
It is clear that the number of matrices that can be produced with MA TRF2 by varying the
parameters m, n, c, r and rx is practically infinite and no huge files are needed to keep sets of these
matrices (each matrix can easily be produced by a simple call of the subroutine MATRF2).
Moreover, some properties of the software tested can efficiently be studied in a systematical way
by the use of matrices generated by MA TRF2. One of the experiments carried out in connection
with the software discussed in this paper is described below as an illustration of the above
statement.
It is easily seen that the condition numbers of the matrices generated by MA TRF2 will in general
be increased when larger values of parameter rx are selected (keeping the other four parameters
fixed). This is confirmed by many experiments in Zlatev et a/. [22]. This property of the matrices
produced by MA TRF2 has been used to compare in a systematical way the largest values of the
drop-tolerance T that can be specified for LLSSOl (where iterative refinement is applied in the
solution process) and for LLSS02 (where the conjugate gradients algorithm is applied). The first
four parameters were fixed in all runs (m = 500, n = 400, c = 150 and r = 12). This means that the
number of nonzero elements was the same, NZ = 6110, in all runs. Six values of rx were used (rx = 2\
k = 0, 3, 6, 9, 12, 15).
First the six matrices generated by MA TRF2 with these values of the parameters were
decomposed with T = 0.0 and the problems created as in the previous two paragraphs were solved
by the use of IR and CG. The results are given in Table 9. It is seen that the factorization times
and the storage requirements are the same in this part of the experiment (the orthogonal
decomposition is carried out with the same subroutines both when IR is used and when the CG
algorithm is chosen). The solution times are slightly different (being in general smaller when the
IR process is in use).
During the second phase of the experiment T = 2- 2 was used as a starting drop-tolerance in the
10
decomposition of each matrix. If the accuracy requirement, prescribed with EPS = 10- , was not
satisfied (either because the iterative process under consideration was not convergent or because
the speed of convergence was judged by the code to be very slow), then the drop-tolerance was
reduced by a factor of two (and sometimes several reductions were performed). In this way for
each value of rx and for each of the two iterative processes the largest value of the drop-tolerance
T, by which the accuracy requirement was satisfied, was found. The results are given in Table 10.
It is seen that in general larger values of the drop-tolerance could be used with the CG algorithm
and this leads to savings in both computing time and storage. Even if the two iterative processes
could be run with the same large value of T, which is the case for well-conditioned matrices, the
Table 9. Results obtained by matrices produced by the matrix generator MATRF2 with m10 = 500, n = 400, c = 150, r = 12 (NZ = 6110) and
six different values of parameter IX. The accuracy requirement is determined by EPS = 10- • The notation used is the same as in the previous
tables
Preconditioned conjugate gradients
Iterative refinement algorithm
IX
2"
2'
2.
2.
2"
2"
T
NZ1
COUNT
Fact. time
Sol. time
lters
0.0
0.0
0.0
0.0
0.0
0.0
6110
6110
6110
6110
6110
6110
62,459
60,979
61,998
62,284
62,886
61,991
46.46
43.40
45.39
45.45
47.05
45.50
0.61
0.61
0.61
0.81
0.84
1.20
3
3
3
4
4
6
T
NZ1
COUNT
Fact. time
Sol. time
Jters
0.0
0.0
0.0
0.0
0.0
0.0
6110
6110
6110
6110
6110
6110
62,459
60,979
61,998
62,284
62,886
61,991
46.46
43.40
45.39
45.45
47.05
45.50
0.72
0.72
0.72
0.72
0.94
1.12
3
3
3
3
4
5
200
Z. ZLATEV
and
H. B. NIELSEN
Table 10. Results obtained by matrices produced by the matrix generator MATRF2 with m = 500, n = 400, c = 150, r = 12 (NZ = 6110),
3, 6, 9, 12, 15) and with an attempt the largest values of T. The accuracy requirement is determined by EPS = 10- IO The
notation used is the same as in the previous tables
"' = 2' (k = 0,
Iterative refinement algorithm
"'
2"
23
2'
29
2"
2"
Preconditioned conjugate gradients
T
NZ1
COUNT
Fact. time
Sol. time
Iters
T
NZ1
COUNT
Fact. time
Sol. time
Iters
2-2
2-l
2-4
2-7
2-10
2-"
4231
3722
4381
4411
4540
4638
4231
3722
5307
6502
9885
12,963
0.22
0.20
0.67
1.34
2.30
3.38
1.24
20.75
1.03
0.89
3.36
29.97
28
469
20
16
51
396
2-'
2-2
2-'
2-2
2-4
2-8
4231
3722
3590
3623
4345
4415
4231
3722
3590
5017
5284
7097
0.22
0.20
0.19
0.53
0.75
1.45
0.92
0.98
I. I 2
2.15
2.07
1.77
19
20
23
40
38
29
CG algorithm leads to a reduction of the numbers of iterations and, thus, to savings in computing
time; see the first two lines in Table 10. Of course, the same effect could be traced by studying the
results presented in Sections 5.3 and 5.4, but here the relationship between the drop-tolerance used
and the condition number of the matrix decomposed is much more transparent.
If the condition number of the matrix is large, then the largest value of the drop-tolerance by
which the decomposition can successfully be carried out is normally small (being considerably
larger when the CG algorithm is in use). Of course, this is not a surprise. This fact has been
observed and justified in Refs [23] and [24].
Comparing the results given in Tables 9 and 10 it is seen that the use of a large drop-tolerance
may be profitable even if the speed of convergence is very slow; compare the results obtained by
the IR algorithm for the matrices generated by ex= 23 and ex= 215 in Tables 9 and 10.
It should be stressed that the results given in Tables 9 and 10 illustrate the conclusions made
in these paragraphs, but these conclusions were drawn by performing many other experiments
(using different values of parameters m, n, c, rand ex).
6. CONCLUDING REMARKS
Some remarks concerning the algorithms discussed in this paper and some possibilities for
further improvement of the algorithms are given below.
Remark 6.1
The method of conjugate gradients was originally proposed as a direct method by Hestenes and
Stiefel [4]. The interest to this method became much greater after the proposal made by Reid [25]
to use it as an iterative method. The method is described in many publications [e.g. 2, 3, 6]. It is
normally used in connection with symmetric and positive definite matrices, but some attempts to
apply the method also to some classes of unsymmetric matrices were carried out [e.g. 26, 27].
Remark 6.2
The matrix AT A appears in the formulation of the CG algorithm in connection with linear
least-squares problems. Therefore it seems that the convergence will depend on [K(A )f (at least
when the pure CG algorithm is in use). Our experiments indicate that this is not the case; in many
experiments the pure CG algorithm was convergent even when K (A) is rather close to the reciprocal
of the machine accuracy. Probably the reason for this is the fact that AT A is never calculated
explicitly and we only consider consistent problems. It should be noted here that Michlin [28] for
the case where m = n and where the CG algorithm is used as a direct method, has proved some
error estimates that depend on K(A) and not on [K(A)] 2, but these estimates are valid for K(A)
much smaller than the reciprocal of the machine precision.
Remark 6.3
The orthogonalization method used in this paper is not the only method that can be used in
connection with large and sparse matrices. Other methods are described in Refs [16, 18, 29-31].
The method proposed by George and Heath [13; see also 14, 19, 32] should be noted. This method
is very efficient in the case where the storage is an important factor. Surveys of the sparse matrix
Solving large and sparse linear least-squares problems
201
techniques used in connection with least-squares problems have recently be published by Heath
[31] and Ikramov [5].
Remark 6.4
The Givens-Gentleman orthogonalization [11, 12] is used during the decomposition. The idea
proposed by Gentleman [33] is used in the pivotal strategy. This means that the plane rotations
are carried out for the pair with minimal number of nonzero elements during each minor step [7,
8]. Thus, the pivotal row is not fixed. The use of a variable pivotal row has recently been used by
Liu [15] also.
Remark 6.5
The particular versions of the CG algorithm described by Algorithms 1 and 2 are not the only
possible. Some modifications have been performed by many authors [e.g. 26, 33-35]. We are
planning to try some other versions in the near future.
Remark 6.6
The strategy for dropping small elements should perhaps be changed. Instead of dropping aij
when Iaijl ~ T, it might give better results to drop it when Iaijl is sufficiently small relative to some
norm of the ith row or jth column-possibly in the "active" part of the matrix. We are continuing
the work towards an efficient implementation of a relative drop-tolerance. Already the use of an
absolute drop-tolerance has proven to be efficient for some classes of problems. It should be
mentioned that dropping small elements was first considered by Evans [36].
Remark 6.7
In the whole work the assumption rank( A)= n was imposed. In fact, a stronger assumption is
needed: matrix A should not be too close to a matrix A* with rank(A *) < n when a computer is
in use. For many practical problems it is desirable to determine the numerical rank of matrix A
and to use special algorithms when it is smaller than n. We are carrying out some work in this
direction, but we are still in the starting phase.
REFERENCES
I.
2.
3.
4.
5.
6.
7.
8.
9.
10.
II.
12.
13.
14.
15.
16.
17.
18.
R. Penrose, A generalized inverse for matrices. Proc. Cambridge Phil. Soc. 51, 406--413 (1951).
0. Axelsson and V. A. Barker, Finite Element Solution of Boundary Value Problems. Academic Press, Orlando (1984).
G. H. Golub and C. F. Van Loan, Matrix Computations. The Johns Hopkins University Press, Baltimore, MD (1983).
M. R. Hestenes and E. Stiefel, Methods of conjugate gradients for solving linear systems. J. Res. Natn. Bur. Stand.
49, 409-436 (1952).
H. D. Ikramov, Sparse linear least-squares problems. In Advances in Sciences and Technology: Mathematical Analysis,
Vol. 23 (Edited by R. V. Gamkrelidze), pp. 219-285. Academy of Sciences, Moscow (1985). [in Russian]
C. L. Lawson and R. J. Hanson, Solving Least Squares Problems. Prentice-Hall, Englewood Cliffs, NJ (1974).
z. Zlatev and H. B. Nielsen, Least squares solution of large linear problems. In Symposium I Anvendt Statistik 1980
(Edited by A. Hiiskuldsson, K. Conradsen, B. Sloth Jensen and K. Esbensen), pp. 17-52. NEUCC, Lyngby, Denmark
(1980).
Z. Zlatev, Comparison of two pivotal strategies in sparse plane rotations. Comput. Math. Applic 8, 119-135 (1982).
0. 0sterby and Z. Zlatev, Direct Methods for Sparse Matrices. Lecture Notes in Computer Science, Vol. 57, Springer,
Berlin ( 1983).
J. J. Dongarra, J. R. Bunch, C. B. Moler and D. G. Stewart, LINPACK-Users' Guide. SIAM, Philadelphia (1979).
W. M. Gentleman, Least squares computations by Givens transformations without square roots. J. Inst. Math. Applic.
12, 329-336 (1973).
S. Hammarling A note on modifications to the Givens plane rotations. J. Inst. Math. Applic. 13, 215-218 (1974).
J. A. George and M. T. Heath, Solution of sparse linear least squares problems using Givens rotations. Lin. A/g. Applic.
34, 69-83 (1980).
J. A. George, M. T. Heath and E. Ng, A comparison of some methods for solving sparse linear least-squares problems.
SIAM J. Sci. Statist. Comput. 4, 177-187 (1983).
J. W. H. Liu, On general row merging schemes for sparse givens rotations. SIAM J. Sci. Statist. Comput. 7, 1190--1211
(1986).
P. Manneback On some numerical methods for solving large sparse linear least squares problems. PhD Thesis.
Department of Mathematics, University of Namur, Namur, Belgium (1985).
I. S. Duff and J. K. Reid, Performance evaluation of codes for sparse matrices. Report No. CSS 66. A.E.R.E., Harwell,
England (1978).
I. S. Duff and J. K. Reid, A comparison of some methods for the solution of sparse overdetermined systems of linear
equations. J. Inst. Math. Applic 17, 267-280 (1976).
202
Z. ZLATEV and H. B. NIELSEN
19. J. A. George and E. Ng, An implementation of Gaussian elimination with partial pivoting for sparse systems. SIAM
J. Sci. Statist. Comput. 6, 390--409 (1985).
20. Z. Zlatev, J. Wasniewski and K. Schaumburg, Comparison of two algorithsms for solving large linear systems. SIAM
J. Sci. Statist. Comput. 3, 48fr501 (1982).
21. Z. Zlatev, J. Wasniewski and K. Schaumburg, A testing scheme for subroutines for solving large linear problems.
Comput. Chern. 5, 91-100 (1981).
22. Z. Zlatev, J. Wasniewski and K. Schaumburg, Condition number estimators in a sparse matrix software. SIAM J. Sci.
Statist. Comput. 7, 1175-1189 (1986).
23. Z. Zlatev, General scheme for solving linear algebraic problems by direct methods. Appl. numerical Math. 1, 177-186
(1985).
24. Z. Zlatev, Sparse matrix techniques for general matrices with real elements: pivotal strategies, decompositions and
applications in ODE software. In Sparsity and its Application (Edited by D. J. Evans), pp. 185-227. Cambridge
University Press, Cambridge (1985).
25. J. K. Reid, On the method of conjugate gradients for the solution of large sparse systems of linear equations. In Large
Sparse Sets of Linear Equations (Edited by J. K. Reid), pp. 231-254. Academic Press, New York (1971).
26. 0. Axelsson, Conjugate type methods for unsymmetric and inconsistent systems of linear equations. Lin. A/g. Applic.
29, 1-16 (1980).
27. 0. Axelsson and I. Gustavsson, A modified upwind scheme for convective transport equations and the use of a
conjugate gradient method for the solution of non-symmetric systems of equations. J. Inst. Math. Applic. 23, 321-337
(1979).
28. S. G. Michlin, On the rounding error in the conjugate gradients method. Vest. Leningrad St. Univ. 19, 15-23 (1984).
[in Russian].
29. A. Bjorck, Methods for sparse linear least squares problems. In Sparse Matrix Computations (Edited by J. R. Bunch
and D. J. Rose), pp. 177-199. Academic Press, New York (1976).
30. A. Bjorck and I. S. Duff, A direct method for the solution of sparse linear least squares problems. Lin. A/g. Applic.
34, 43-67 (1980).
31. M. T. Heath, Numerical methods for large sparse linear least squares problems. Siam J. Sci. Statist. Comput. 26,
491-513 (1984).
32. J. A. George, M. T. Heath and R. J. Plemmons, Solution of large scale sparse least squares problems using auxiliary
storage. SIAM J. Sci. Statist. Comput. 2, 4lfr429 (1981).
33. W. M. Gentleman, Row elimination for solving sparse linear systems and least squares problems. In Numerical Analysis
Dundee 1975 (Edited by G. A. Watson), pp. 122-133. Lecture Notes in Mathematics, Vol. 506, Springer, Berlin (1976).
34. P. Concus, G. H. Golub and G. Meurant, Block preconditioning for the conjugate gradient method. SIAM J. Sci.
Statist. Comput. 6, 22(}-252 (1985).
35. P. Concus, G. H. Golub and D. P. O'Leary. A generalized conjugate gradient methods for the numerical solution of
elliptic partial differential equations. In Sparse Matrix Computations (Edited by J. R. Bunch and D. J. Rose), pp.
309-332. Academic Press, New York (1976).
36. D. J. Evans, The analysis and application of sparse matrix algorithms in finite element method In The Mathematics
of Finite Elements and Applications (Edited by J. R. Whiteman), pp. 427-447. Academic Press, London (1973).