~I
-
C.,
14
01
Sw
.
.
0
4 )
)
V
'
C
C
0.
0)
00
cd0
0.-
ru
0
C13
0
H
-4
*
W4'c
C
I
On Direct Methods for Solving Symmetric
Systems of Linear Equations
James R. Bunch
---
3
irThi
SCI.Ite
C&Tov ,
Technical Report No. 33
-3
Computer Center
University of California
Berkeley
document
IThis
docuinent 'iri
been approved
for Public rele -s -Td saile; it
diztribution is untimJit,.
On Direct Methods for Solving SMetric Systems
of Linear Equations
by
James Raymond Bun:h
Abstract
There has been no stable direct method for solving symmetric
systems of linear equations which takes advantage of the symmetry.
If
the system is also positive definite, then fast, stable direct methods
(e.g., Cholesky and symmetric Gaussian elimination) exist which preserve
the symmetry.
tf
These methods are unstable for symmetric indefinite sys-
Such systems often occur in the calculation of eigenvectors.
s.
Gaussian eliminatiun with partial or complete pivoting is currently
recommended for solving symmetric indefinite systems, and here symmetry
is lost.
We present a generalization of symmetric Gaussian elimination,
called the diagonal pivoting method, in which pivots of order two as
F
well as one are allowed in the decomposition.
We show that the diagonal
pivoting method for symmetric indefinite matrices takes advantage of
symmetry Lo that only
and
2
6
n'n multiplications, at most
storage locations are required to solve
is a non-singular symmetric matrix of order
that the method
is
n .
I3
n3 additions,
A x - b , where
A
Furthermore, we--shwQ
nearly as stable as Gaussian elimination with complete
pivoting, while requiring only half the number of operations and half
the storage.
L.... .... .. .... . . .......
.............
We inzlude a listing of an Algol procedure for the diagonal
pivoting method, which is applicable both to symmetric definite and
indefinite systems.
We discuss the problem of symmetric band matrices and present
an algorithm only for the tridiagonal case.
Fur~ther, we discuss the
problem Of 'Oquillhrattng symmetric matrices while preserving symmetry
and we present a simple algorithim (and Algol procedure) for accomplishing
thiti.
Table of Contents
Page
Chapter 1
1.1
ti.2
1.3
1.4
Chapter 2
Introduction
Symmetric Systems of Linear Equations
Our Contribution
1
1
Summary
2
Origin of Symmetric Indefinite Systems
3
I'resentation of the Problem
2.1
2.2
2.3
Introduction
General Problem : Exact Arithmetic
General Problem: Failure of Previous Attempts
in Finite Precision Arithmetic
2.4 Condition of a Matrix
2.5 General Case : Stable Direct Methods
2.6 Symmetric Case : Direct Methods
2.7 Symmetric Positive Definite Case
2.8 Symmetric Case : Failure of Cholesky and
L D Lt Methods in Exact Ar.thmetic
2.9 Symmetric Case : Failure of L D Lt in
Finite Precision Arithmetic
2.10 Symmetric Case : Present Situation
2.11 Symmetric Case : Our Problem
Chapter 3 : Historical Survey
3.1
3.2
3.3
Introduction
Direct Methods
Indirect Methods
Chapter 4 : Diagonal Pivoting
4.1
4.2
4.3
4.4
4.5
4.6
4.7
Preserving Symmetry
The L D Lt Decomposition
Orthogonal Reduction to Diagonal Form
Lagrange's Method of Reduction
Kahan's Proposal
Pivotal Strategy
Parlett's Observation
Chapter 5 : The Decomposition for Diagonal Pivoting
5.1
5.2
.
I;
I
1
Definitions
Ti"
Thc
'i
lv'compo
I
[ ttion
ivot
5
5
5
6
7
7
10
10
11
12
12
13
14
J4
14
15
17
17
17
18
18
20
21
22
23
23
23
21
Page
Chapter 5
(Continued)
5.4
The
5.5
5.6
5.7
Bounding v
The Reduced Matrices
Criterion for Choosing a lxI or 2x2 Pivot
2x2
Pivot
Chapter 6 : The Complete and Partial Pivoting Strategies
24
25
27
27
30
6.1
6.2
Complete Pivoting
Bounding VC
30
31
6.3
6.4
6.5
Operation Count for Complete Pivoting
Partial Pivoting for Equilibrated Matrices
Bounding v
p
Criticism of the Partial Pivoting Strategy
31
33
33
6.6
Chapter 7 : Equilibration of Symmetric Matrices
7.1
7.2
7.3
7.4
7.5
34
36
36
36
37
38
7.6
Introduction
Equilibration of General Matrices
Difficulties with Symmetric Matrices
The Obvious Attempt
Equilibration of Lower Triangular and
Symmetric Matrices
The Algorithm for Null Rows in the Lower
7.7
7.8
Triangle
Summary of Equilibration of Symmetric Matrices
The Algorithm for Exponent Adjustment
40
41
42
Chapter 8 : Unequilibrated Diagonal Pivoting
8.1
8.2
8.3
8.4
8.5
Maximal Off-diagonal Element
Bounding vb
Comments on this Strategy
A Partial Strategy for Equilibrated Reduced
Matrices
k Partial Strategy for Unequilibrated Reduced
Matrices
Chapter 9 : Operation Count
9.1
9.2
9.3
9.4
9.5
9.6
9.7
T~~
Solution by Diagonal Pivoting
Summary of the Work Required
Forming (1) A - M D Mt
Solving (2), (3), (4)
Total Work Required
Upper Bound on Mults
Upper Bound on Adds
39
44
44
45
46
48
48
50
50
50
52
54
54
55
55
Page
56
Chapt?7 10 :Error Analysis for Diagonal Pivoting
10.1
10.2
10.3
10.4
10.5
10.6
10.7
10.8
Introduction
The Occurrences of Error
The Error Matrix E
Notation
Summary of the Error Analysis
(r)
The Decomposition for the Reduced Matrix
The Error Matrix F for the Decomposition
The Error Matrices M 1 , 6D, M 2
56
56
57
57
58
59
59
60
10.9
10.10
10.11
10.12
10.13
10.14
Floating Point Error Analysis
Floating Point Analysis for F
Summary of Floating Point Analysis for
Comments on the Bound for F
Floating Point Analysis for a D
Floating Point For M1 and M2
61
61
64
65
66
67
F
10.15 Floating Point Bound for iL
10.16 Comments on the Bound for E
Chapter 11 ; An A Posteriori Bound on Elemcnt Growth
for 0 < a < 1
11.1
11.2
11.3
11.4
11.5
11.6
11.7
11.S
11.9
Chapter 12
*
Introduction
Pivots
Hadamard's Inequality
Founding det A(k)
Fundamental Inequality
Bounding Pivot Growth
Bounding Element Growth
Comments on the Bound
Smaller Bound on Pivot Growth
68
70
71
71
71
72
73
73
74
75
78
79
An A Priori Bound on Element Growth for
= (i + /7)/8
82
12.1
12.2
12.3
12.4
Introduction
Lower Bound on c(as) h(n,a) for 0 < c I
Remarks 3n an Upper Bound for h(n,a)
An A Priori Bound for a - a 0 - (1 + 1)1-8
82
83
85
86
12.5
12.6
Bound on Element Growth
Conjecture for Gaussian Elimination
91
91
12.7
Conjecture for Diagonal Pivoting
92
12.8
The Optimal Choice of ci
92
80
I
I:
... . _
Page
Chapter 13 : Iterative Improvement
13.1
13.2
13.3
The Approximate Solution
The Iteration
Convergence of the Iteration
Chapter 14 : Symmetric Band Matrices
14.1
14.2
14.3
Gaussian Elimination for Band Matrices
Diagonal Pivoting for Symmetric Band Matrices
Symmetric Tridiagonal Matrices
Appendix A : Miscellaneous Results
A.1
A.2
A.3
Vtagcnai Pivoting for Positive Definite
Matrices
A Results foc Symmetric Hadamard Matrices
Gaussian Elimination for Tridiagonals
Appendix B : Algorithm for Symmetric Equilibration
B.1
B.2
Discussion
The Algol Procedure
Appendix C : Algorithm for Diagonal Pivoting
Bibliography
4J
94
94
95
96
98
98
99
99
A-I
A-1
A-2
A-3
B-I
B-I
B-i
C-I
Acknowledgtment
This dimsterraLion was prepared under the direction of
Profes~sor licresford N. Parlett, Chairman, Department of Computer
Science, Berkeley.
I am most grateful to Professor Parlett for
his being willing to discuss innumerable contentions and for the
go'nerous contribution of his time that he made.
I wish to thank ProfessoL William Kahan for several helpful
eonversations, aad I wish to thank the dissertation committee,
Professor Parlett, Professor C. Keith Miller, Department of Mathematices, and Professor H-ugh D. McNiven, Department of Civil Engineering,
toy reading the manuscript.
I d160 UWAk 'Mrs. Geri Stephen for great patience and exceptional
skill In typing the manuscript.
Research for this dissertation was supported in part by the
Office of Naval Research under the Computer Center Nonr Contract
3656(23).
Chapter 1 ; Introduction
1.1
Symmitric Syatem
of Linear Lquations
Let us consider direct methods for solving the system of linear
algebraic equations, A x - b, where
if
and
::
Tf
A
A
is symmetric.
is alo positive definite, then Cholesky's method (§2.7)
L D Lt
A
method (§2.6) are fast, stable, and preserve symmetry.
is symmetric but indef3nite (neither positi e definite nor
negative definite), Cholesky's method and the
L D Lt
method are unstable
.nd can produce very inaccurate results (§2.8).
At the present time, if
A
is symmetric indefinite, Gaussian
elimination with partial or complete pivoting is recommended for solving
the system (Fox, p. 80, 185), and thus the symmetry of
A
is ot no
advantage.
Is there an algorithm for the symmetric indefinite case wh'ch is
stable, is
faster than Gaussian elimination, and can take advantage
of the symmetry?
1.2
Our Contribution
We discuss the problem is Chapter 2 and review previous efforts in
Chapter 3.
In Chapters
4-6 we present a laethod, called diagonal pivoting,
which fulfills the above requirements whet restricted to equilibrated
matrices,
In Chapter 7 we present a metbod for equilibrating symmetric
matricua in a very simple manner.
A variation of the diaLonal pivoting
method is presented in Chapter 8i this method is applicable to unequili-
brated matrices and it fulfills. the above-mentioned requirements.
In
Chapter 9 we show that the diagonal pivoting method is almost as fast
as Cholesky or
analysis.
In Chapter 10 we perform a backwards error
L D Lt.
In Chapters 11-12 we show that the method is essentially as
stable as Gaussian elimination with complete pivoting (in the sense of
Wilkinson's analysis for Gauss'an elimination with completely pivoting,
Wilkinson (19b1)).
In Chapter 13 we show that iterative improvement is
as applicable here as it Is for Gaussian elimination.
In Chapter 14
we discuss the problem of symmetric band matrices.
All the results proved are applicable to complex systems where
A
is Hermitian.
1.3
Summary
Let
A
be an
n x n
We want to solve
matrix with
max IAij
ij ii
1
A x - b.
k-1
Let
-C
N
denote
C
N
+
I
ore constants independent of
Let
G.E.
Ni
C
i-O
where the
n
denote Gaussian elimin-aticn.
The situation is summarized in the following table:
C
0 < i < k
3
Restrictio-as
on A
Method
Number of
AuhLOAlications
G.E. with
complete
pivoting
det A
0
G.E, with
partial
pivoting
det A
0
Number of
Additions
Storage
2in
3 n
n
i
1
-n
/nf (n)
1
n3
symmetric
positive
definite
1
Diagonal
Pivoting
symmetric,
det A #0
1 n
T-
n
n
2
3
1 n2
4 <n 3c
- -1
3n
~1l/2 -
3
i
,
n
0 < a < I, f(n)
n2
2
Cholesky
Method
Here
-Bound on
Element
Growth
StaLlityl
k
-
h(n,a)
n f (n)
c"(ot) h (n. a)
is a function
k-2
dependent on the pivotal strategy, and
9§11.6-7.
for
a
In Chapter 12 we show that
c(a)
and
h(n,a)
are defined in
<.3.07(n-1) 0.446 < 3
c(c)h(n,)
v'nn
- (I + 17)18 -c
0
1.4
Origin of Symmetric Indefinite System's
The problem of indefinite systems cf linear equations is sometimes
dismissed as academic by the claim that physical pr-blems always generate
positive definite systems of linear equations.
However, the numerical
solution of natural problems often gives rise to situations which do not
have a physical origin.
We give two related examples.
In the Rayleigh Quotient Iteration for finding eigenvalues of a
positive definite matrix
£i
A
(Wilkinson (1965), p. 172, and particularly
4
(A
p. 629) we need to solve the systems
ri = xi
r
A x i/ x i
xi
Here
A - r I
-
ri
) Xi+
xi
where
cannot be definite because
lies between the extreme eigenvalues.
In the inverse iteration method for finding the eigenvector corres-
ponding to an approximation
positive definite matrix
ui+ 1 - vi+ / max (v i+ I )
A,
.A
\
to an intermediate eigenvalue
we need to solve
- X I
( A - X I)
can be indefinite even when
positive definite (Wilkinson (1965), pp. 618-635).
I!
v i+
of a
u,
I
A
is
5
Chapter 2
2.1
Presentation of the Problem
Introduction
The speed and storage capacities of current digital computers
allow us to solve large systems of linear equations by direct methods.
Here we shall consider direct methods for solving a system of linear
A x - b , whe-e
equations,
2.2
A
is symmetric and
det A 0 0
General Problem : Exact Arithmetic
First let us consider the solution, in exact arithmetic, of
A x
b , where
A
is general and
det A # 0 . We know that Gaussian
elimination will give the aolution provided that whenever a zero appears
in the leading diagonal position we interchange that row with a lower
row with non-zero leading element (such a row will exist since
det A 0 0), e.g. if
A ji
A
0
=
and if
0 , then we interchange rows
exists a permutation matrix
interchanges applied to
P A
P
j
I
is the least integer for which
and
j .
Or, equivalently, there
such that Gaussian elimination without
will give us the solution.
Since we could also do the same with columns instead of with rows,
there exists a permutation matrix
without interchanges applied to
Q
A Q
such that Gaussian elimination
,.,1l1
give us the solution.
In matrix notation the Gaussian elimination algorithm factors
into
and
A - L U , where
L
Thus "
and
U
L
is unit lower triangular, U
A
is upper triangular,
are unique when they exist (Wilkinson (1965), p. 204).
xact arithmetic there exist permutation matrices
P
and
Q
6
such that
P A
=
L
U
and
A Q = L2 U 2
,
provided only that
der AO0
2.3
General Problem : Failure of Previous Attempts in Finite
Precision Arithmetic
In finite precision arithmetic (Wilkinson, 1963) the above algo-
ricIhm can fail if we interchange only under the condition that the element in the leading diagonal position be zero.
Let
Then
A
A
[
L [J =
[ [
and
Li/:
b
[
However, if
n1 -or
1/1
c
and
n
small enough, then in finite precision arithmetic the operation
yields
U
-
i/c
C
and
x
c
be the matrix and vector of the values of
U
x, respectively, computed in finite precision.
Then
U
I
a -
b
-J.I
x-U'
If
and
=
Soxo
So
i
-I/c .
Let
and
are
n
a-
c- I
Kll)
cE, then
/
xi
of the computed solution is
2]
, but
xc
mU
co
and the error in the first component
Id
I
i
7
Condition of a Matrix
2.4
Wilkinson (1965, pp. 189-191) shows that the relative error in the
solution of a system of linear equations is bounded in proportion to
the condition number of
A , K(A) -
then
lel / Ix1 < p(A) K(A)
K(A)
is large.
Here
A
|A| |A I' > I
, ~C
i.e.,
if
e - x
xc
We should not expect a small error if
2I.]
[
A-'
has a very satisfactory condition number.
n
Using the one-norm,
1A1
ma
J
i-I
IAij1,
we have
K(A)
-
AI
1
1A- 1 1
=
[l + max(IclIn)]
11 - n,'l
The computer replaces
L
(A-l
Cn - I
Then
by
--I,
and the computed inverse
b bA-
is
Thus the trouble lies in
the Gaussian elimination algorithmn, not in the matrix
A
General Case : Stable Direct Methods
2.5
(a)
Direct Inversion
Direct inversion (the formation of
A
) of a system requires
each of multiplications and additions (Fox, pp. 177-179).
mination, however, requires only
additions.
3
n
-
n
Gaussian eli-
each of multiplications and
Thus we would prefer to use Gaussian elimination if we could
obtain a satisfactory solution.
8
(b)
Stability
Let us attempt to solve
# 0 . If xc
det A
may consider
x
A x
b
-
,
where
A
nxn
is
and
is the solution we obtain from the computer, we
to be the exart solution of the system
(A + E) y
We might say that the algorithm we use is stable if the elements of
are small in comparison to the corresponding elements of
the term is more often used when
1E
/ 1A1
is small.
b
-
E
A . Actually
I I
(Here
is any norm.)
(c)
Stability for Gaussian Elimination
Wilkinson (1960) showed that for Gaussian elimination we have:
.1 max(1,J)2
~-iJ2 1 Eiji < 2.01
a
rs
I
A(n-k+l)
of binary digits in the machine, k
reduced matrix of order
n-k+l
=
where
min(i,j), and
t
is the number
A(n- k+l)
is the
in the elimination process.
The important lesson from the above is that we must be interested
in keeping the elements in the reduced matrices small.
well-known strategies for choosing permutation matrices
There are two
P
and
such that Gaussian elimination without interchanges applied to
Q
P A Q
will provide sufficiently small element growth in the reduced matrices.
(d) Complete Pivoting
The first strategy, called complete pivoting, requires that we
bring the largest element in the reduced matrix into the leading diagonal
position.
This strategy is called complete since we search the entire
reduced matrix.
t
I
Wilkinson showed that this complete strategy gives
I
=--
-"
Imom
9
1
Ik+
max
i ,j
f(k) 2
where
A
< vf(k)max
rr
k
i ,j
r-2
In words, the elements in the reduced matrices can never become too
large; so this strategy is never bad.
bound is
max
A-n-k+J<) k
It is conjectured that the true
where
A
to real (A2.4).
Equivalently, the above says that there exist permutation matrices
P
to
and
Q
P A Q
such that Gaussian elimination without interchanges applied
gives
2.01 n3 / 2 f(n)
1E ij 1
max
i ,J
2
- t
i
max 1A
iJ
(e) Partial Pivoting
The second strategy, called partial pivoting, requires that we
bring the largest element in the first column of the reduced matrix
into the leading diagonal position.
This strategy is called partial
since we search only a part of the reduced matrix.
This is equivalent
to the application of Gaussian elimination without interchages to
4h-k+lV
k
where P is a permutation matrix. Here
max
;A< 2
-1I 1
0
This bound is sharp since
nxn , has
A(n)
nn
=
n
.
A
Thus
and this is very weak when
.
-
max
i
J
E
ji
0
0
0
i II
1
0
1
.
<-I 2.01 n 2- t
,
2n
where
max
IA
A
P A
is
I
j
n > t
Correspondingly, we could use a partial pivoting strategy in which
we bring the largest element in the first row of the reduced matrix into
I
10
the leading diagonal position.
Thus there exists a permutation matrix
applied to
such that Gaussian elimination without interchanges
- n
2
has an error matrix
E
with
< 2.01 n
max
tEijI max
A Q
l~i
iji,j
(f) The Error Matrix
We see that the error matrix
L,U , the matrix
E
is dependent on the decomposition
A , the right hand side
and
Q
E(L,
U, A, b, I',Q) , where
b , and the permutations
by which we can pre- and post-multiply
L U - P A Q
A , i.e.
P
E -
(For the partial pivoting
.
strategy on the first column of the reduced matrix, we take
Q - I
in
the above.)
Symmetric Case : Direct Methods
2.6
If
A
is symmetric, then we can only apply congruences to
we want to preserve symmetry.
A
if
In particular, whenever we interchange
two rows, we must also interchange the corresponding columns.
Thus
only a diagonal element can be brought into the leading diagonal position.
The cymmetric form of the Gaussian elimination decomposition
the decomposition
diagonal, and
congruences on
N
2.7
Lt
L D Lt
t
, where
L
is
is the transpose of
is a permutation matrix such that
unit lower triangular,
L
A , the error matrix is
L U
L D L
D
is
Since we may only perform
.
E
gives
-
E(L, D, A, b, N, N t )
where
NttA N
Symmetric Positive Definite Case
(a) Cholesky's Method
The Cholesky decomposition (Wilkinson (1965), pp. 229-232) is the
most well-known decomposition for a positive definite matrix
A
-i
I
t
(i.e.
x
Ax > 0
triangular.
If
x
0)
Here
A
L L
where
L
is lower
No interchanges are re4uired for stability.
L D Lt
(b)
for
11
_
Decomposition
L D Lt
is positive definite, then its
A
(the1
decomposition
symmetric form of Gaussian elimination) is stable in the absence of
unde.rtluw and overflow.
The elements of
L
can be arbitrarily large,
but if they do not overflow then in fact the error matrix
E
small as the error matrix for the Cholesky decomposition of
1/2
L - L D
(Note:
is
A
as
.
)
(c) Method of Congruent Transformations
This method (Westlake, p. 21; De Meersman and Schctsmans, (1964))
decomposition (b) with interchanges.
Here the largest diagonal element is
brought into the leading diagonal position at each step.
tive definite, then the elements of
uses
L
are bounded by
If
A
is posi-
and the
1
method is stable.
(d)
Summary
If A
If
A
is positive definite, then the above three methods are stable.
nxn , then each method requires
is
1 n3
6
n3
multiplications, and
n2
storage positions,
additions to solve
Ax - b
6
2.8
Symmetric Case : Failure of Cholesky and
Exact Arithmetic
If A
is symmetric but indefinite, then the
L D Lt
L D L
Methods in
decomposition,
the method of congruent transformations, and the Cholesky decomposition
fail in exact arithmetic for a matrix as simple as
A
- ==
[
1
that is, there exists no permutation matrix
=
N
such :hat
12
N A Nt
has an
L D L
Note that
or an
L Lt
L Lt
decomposition.
is always positive semi-definite.
Thus Cholesky
decomposition will fail in exact arithmetic whenever
A
dnd
such that
det A
'
0 ,i.e.
there exists no permutation
N
is indefinite
N A Nt -
tt
L Lt
where
'file
L
Is lower triangular.
L D Lt
decomposition and the method of congruent transfor-
mations will fail in exact arithmetic whenever all the diagonal elements
in a reduced matrix are zero, i.e. there exists no permutation
N A Nt
that
2.9
has an L D Lt
L D Lt
L D Lt
in Finite Precision Arithmetic
decomposition in finite precision can be unstable if
the diagonal elements are too small.
matrix
such
decomposition.
Symmetric Case : Failure of
The
N
A
The
L D Lt
decomposition on the
will produce the same incorrect solution on the
computer as we saw in §2.3 for Gaussian elimination without interchanges.
However, here there exists no permutation matrix
has a stable
L D Lt
decomposition.
Thus the
N
such that
L D Lt
N A Nt
decomposition
fails for symmetric indefinite matrices.
2.10
Symmetric Case : Present Situation
If we ignore the symmetry of
or partial pivoting to
A
and apply elitainption with complete
A , then in general
A
metric after the first step of the elimtintion.
will no longer be symWe then need
storage positions in the computer and we must perform
1
-
3
plications and
....
. ...
. . . .
...
. . . .....
3
n
additions.
. ....
. ..
- n2
3
n
multi-
But we also have stability for the
...
r
13
L U
decomposition.
This procedure is presently recommended for the
solution of symmetric indefinite systeas of linear equations (Fox;
p. 80, 185), and thus the symmetry of
2.11
Symnetric Case
ii
A
A
is of no aavautage.
Our Problem
is symmetric but indefinite, we would like. to find an
algorithm which gives a stable decomposition when applied to
where
N
N A Nt
is a suitable permutation matrix, but which also takes
advantage of the symmetry in order to require only
1 n
positions in the computer and to require only 6
n
n
storage
multiplications
1 3
and
-n
6
additions.
Our algorithm will fulfill all the above requirements wich the
1 3
3
exception that we will need between
n
and
n
additions.
t4
I
.. .. =
1
14
Chapter 3
3.1
Historical Survey
Introduction
Various methods have been proposed for symmetr;.c indefinite
f
Most of these methods have been unstable, while the stable
MaLrices.
methods have required operation counts of at least
n3/3, where an
operation Is defined to be a muItiplication follo-,wed by an addition.
Let us look at some of these methods.
3.2
Direct Methods
Usually direct muthods for symmetric indefinite systems are based
on the symmetric form of Gaussian eliminarion,
t
L D L
, which is unstdble
in the absence of pivoting.
The
L D L
method and its 'ariant, the method of congruent trans-
formations (in which we use the largest diagonal elment as the pivot
at each step (§2.7)) require
operations.
n
23 locations and
storage
But both methods are unstable (§2.7-2.9).
These methods
are of value only if It is known in advance that no element of
D
will
vanish or be small.
The Crout factorization (Hildebrand, pp. 429-435; Householder,
pp. 82-83) is also a modification of
mination, and thus requires
2
L D Lt , symmecric Gaussian
n-
storage locations and
2
2li-
6
operations, but it is also unstable.
These variants of
L D Lt
are unstz'ble.
direct methods that are not based on
I
II
I
I
I-I
I-I-I-II
Let us consider some
L D Lt
i--I
I
Is
15
The escalator method (Householder, pp. 78-79) uses the known
solution of a subsystem as a step in solving the complete system.
The problem lies In finding the solution of the subsystem.
Some headway in this problem was made by Parlett and Reid (1969).
They reduce a symmetric matrix to tridiagonal form by stabilized elementary congruences and solve the tridiagonal system by Gauss1~n elimination with partial
tior
and
3
n3
pivoting.
They require
n
storage loca-
operations, and the method is stable.
In October, 1965, W. Kahan (in correspondence with
R. De Meersmans
and L. Schotsmans) proposed a method for solving symmetric systems based
on Lagrange's theorem on the reduction of quadratic forms to diagonal
forms (§4.3 - 4.4).
Kahan proposed the generalization of the idea of
a pivot to include 2x2 submatrices (§4.5).
Then Schotsmans (1965) prepared an algorithm in which one searches
all the principal 2x2 submatrices for the one with largest determinant.
. n2
~ 1 n3
This algorithm requires
n
storage, but between
-n
and
n
3.3
operations (§5.4).
Indirect Methods
The Seidel iterative method (Householder, pp. 48-51, 81) and the
method of relaxation (Householder, pp. 48-51, 81) require
tions for each cycle.
:
|I
I
opera-
The number of cycles required depends on the
matrix, the starting values, and the needed accuracy,
ber of cycles exceeds
- n2
a
,
so at least
. =. . I.
n3
I "
Usually the nu-
operations are required.
II
I
16
The method of steepest descent (Householder, pp. 47-51, 82)
requires
n
- 2 n?
operations at each step.
steps are required.
Again, usually at least
So the number of operations is at least
2 n3
The congruent gradient method, also called the Stiefel-Hestenes
method, is a finite iterative method designed for positive definite
matrices, but it can be used for symmetric matrices (Fox, pp. 208-213;
Householder, pp. 73-78, 82).
of
n
steps.
Once again
It requires
- 2 n3
- 2 n2
operations are required.
(1967) for useful observations on this method.
I
operations at each
See Reid
r[
17
Chapter 4 ; Diagonal Pivoting
Preservig Symmetry
4.1
In order to L:ave a direct method for symmetric matrices which
will preserve symmetry, we can perform only congruences on the
A , i.e. if we premultiply
matrix
then we must also postmultiply by
The
4.2
L D Lt
by a non-singular matrix
Xt
t
L D L
method in greater detail.
to diagonal form by congruences.
A
X
Decomposition
Let us consider the
vert
A
We con-
Let u. consider the first
step of the decomposition:
Let
Lc
A-
1C
where
If a
B_
0
and
*
I1
A
L
,
B-CCtla
0
1
L
C/a
0 ,then
In-
is the identity matrix of order
n -i
n-1]
L D Lt
The variant of
called the method of congruent transforma-
tion. (Westlake,p. 21; De Meeramans and Schotsmans) uses the largest
diagonal element as pivot.
position of
N A Nt
,
where
This is equivalent to the
N
L D Lt
decom-
is a permutation matrix.
As we saw in §2.8 both methods are unstable for symmetric (indefinite) systems.
This instability results from our being unable to bring an offdiagonal element into the pivotal position.
Since we are using only
congruences, we can bring only a diagonal element into the pivotal
I
..
..
..-.
18
When some off-diagonal element Ai , j > i * is very large,
position.
we can bring Aji into the (2,1) position by congruences, but never
into the
(1,1)
Thus we cannot take advantage of this
position.
valuable information.
4.3
Orthogonal Reduction to Diagonal Form
t
Any real quadratic form
x
A x
of rank
r
can be reduced by an
orthogonal transformation to a diagonal form
x11
where
Al. ... Ar
2
+X
+
x2
r
r
are the non-zero eigenvalues of
A
(Mirsky, pp. 362-
363).
If
A
nxn
is an
symmetric matrix with
det
A
#
0
,
then the
above means that
Aff0A0
where
0
A = diag
1 ,...
,n
n
t
t
, the
is an orthogonal matrix whose
ponding to
Xi
i
are the etgenvalues of
A
and
column is an eigenvector corres-
A1
However, this
0 A 0
t
decomposition involves more work than
Gaussian elimination and requires the use of irrational operations.
For a finite-precision algorithm we would prefer a reduction
involving only rational operations.
4.4
Lagrange's Method of Reduction
In 1759 Lagrange devised a method for reducing a real quadratic
form of rank
r
by a real non-singular linear transformation to a
19
diagonal form
L 2 +
X
"
+
r
1
C
where
x2
r
"...,sr
are all non-zero (Mirsky, pp. 368-374), and the
number of positive (and negative) squares is invariant.
L D Lt
This method corresponds to the
metric matrix
A
A11U
Suppose
L D L
A
t
when the
...
L D Lt
for
r # s
decomposition exists.
-A nnl - O , while
decomposition for
# 0
A
since
decomposition of a sym-
det A # 0
does not exist.
det A
.
Then the
In this case, some
0
rs
A 11 -
Let us assume
(Xl,.. .,x)
x
- _xl,
x
A x
0 = A22
but
A12 0 0 , where
is a quadratic form in
xl,...,x
A
n
A
where
I
,xx
n]
In this case Lagrange proposed the following transformation:
(4.4.1)
x
-
This maps
+ Y
Y
2 A1
2
x
xI x2
-
into
2
x-
2 A1
2
3Y3
(
"Xn
. Yn
- ).
Thus
is
transformed into a quadratic form
P in Yl....'y
where the coeffi2
an
cients of
y
and y2 are non-zero. Then we can proceed with the
decomposition (Mirsky, p. 371-2; Gantmacher,
p. 19q).
Let us consider the above transformation in matrix form.
T y
Then
:z , where
-1
(4.4.2)
0
T =
and
2 0
In-2
is the identity matrix of order
1n-
n-2
K!
i
20
Thus
x
t
y
A x
t
Ct
(T t A T) y,
and
a symmetric matrix
A[I
T
(Tt A T)1 1 ,(Tt A T) 2 2 0 0
with
Hence we have avoided the problem of zeros on the diagonal of
by use of the
2x2
A
matrix
This procedure is also applicable to complex quadratic forms.
4.5
Kahan's Proposal
Ln 1965 W. Kahan (in correspondence with R. De Meersmans and
L. Schotemans) proposed that Lagrange's method could be made the basis
of a stable method which preserved symmetry.
Kahan adapted Lagrange's method to finite precision by observing
that the use of
in (4.4.2) corresponds to the use of a 2x2
submatrix as a pivot in a decomposition by linear transformations and
that a 2x2 could be chosen if the diagonal elements were zero or very
small (§§2.8-2.9).
Suppose we used a 2x2 submatrix
such a decomposition.
A
nxn ,
is
A a L1
0
P
Let
A
is 2x2 , C
B-CP 1 Ct1
L
P
-
,
is (n-2)x2
,
the identity matrix of order
as a pivot.
where
,
where
and
B
A
Let us look at
-
At . det
is (n-2)x(n-2).
L1 -
and
2x2
pivots be baL?
and
Ik
0
Then
is
k
How do we choose whether to use a lxl or a 2x2 pivot?
lxl
A
Can both
21
Pivotal Strategy
4.6
In the first, one
Kahan considered two pivotal strategies.
m
searches the entire matrix for
A i,
f
c
c
- wax
i,j,k
,
[A'
AjA
-
A2 1
All
then interchange rows and columns 1 and i and use
as a lxl pivot.
If m
IA1a A.k
columns 1 with j and 2 with
-
A2I , then interchange rows and
k , and use
[ Aj j Ajk]
ILA k Ak
as a 2x2 pivot.
Since we search the entire matrix, this is called a complete
pivoting strategy, In analogy with complete pivoting for Caussian
However, the searching here requires between
Elimination.
and
-
3
multiplications to find
n
the number of
requires
between
lxl
I n
6
I n3
and
2x2
for all steps (depending on
pivots used).
multiplications.
and
mc
The decomposition itself
Thus this strategy would require
multiplications to solve
n
3
6
(§5.4), which is more than for Gaussian elimination.
A x - b
Hence Kahan
rejected this strategy.
Kahan considered a second pivotal strategy in which we scan only
the first column and the main diagonal; this is called a partial pivoting
strategy, in analogy with partial pivoting for Gaussian elimination.
1 n2
1 n2
multin
and
.1 n
The searching here only requires between
2
4
plications.
We take
m
(A2
1
I
A
A.max
}
.
However, this partial
pivoting strategy is unstable.
I
22
A
At
Th en
mp
TThus JA I
O<c<<l.
, where
1'
2 A2
0
Th us
[
2
1
2
8
+ g - 1 C)
()21
AM
a 2x2 pivot, and the reduced matrix
If
c
operatio
I- 1
is
would be used as
is small enough, then in finite precision arithmetic the
1
3C
3 - 23 c
yields
-
.
+ 8 .
3
This can cause highly
inaccurate solutions, as in §2.3.
So this partial pivoting strategy is unstable.
For these reasons
Kahan rejected this method for use on symmetric systems.
4.7
Parlett's Observation
In 1967 B. Parlett observed that the examples for which the partial
pivoting strategy was unstable were also unequilibrated.
matrix
(§7.1).
A
is equilibrated if
max
j
A symmetric
IAijI - 1 for each row index
i
Parlett conjecture that the partial diagonal pivoting strategy
would be stable when applied to equilibrated matrices.
23
The Decompoition for Diagonal Pivoting
Chapter 5
Definitions
5.1
Let
A
M D Mt
to the "diagonal" form
A
reduce
symnetric non-singular matrix.
nxn
be an
We want to
by congruences, where
D
is a block diagonal matrix, each block being of order 1 or 2, &nd
is unit lower triangular with
if
,
0 0
D
and
1 - IA n
22
-A
The Decomposition
5.2
[
Let
and
-
A = C
L
is
P
if
j
0
"max
12 IAi
ij1
0 "xi ,J IA 1l ,
Let
H
H
s
B
,
where
j x j , where
p-l
exists, then
and
L
and
[
n-J, respectively.
C
j x (n-J),
is
B
is
(n-J)
x (n-J),
j - I or 2.
A - L1
'p.In-
p--
LL
IB-C
001
1
t
where
are the identity matrices of order
Any element of
C P-
will be called a
multiplier.
5.3
The
Suppose
lxl
P
between a matrix
call
Pivot
is of order 1.
P
(We shall not make a distinction
of order 1 and its element, which we shall also
P .)
i
24
Let us assume that we have already interchanged rowe and columns
so that
1PI
If
P1
Pi
I
l
Lemma I:
(1)
If
P
V
is of order 1
A ( n - l)
the,. let
t
C P- 1 C
- B -
Aj+~
J+l,l
we have the following:
(C P-
Ai~~+
i+l,j4-l
IPI
and
i
£
" U1 I
0
then
max I(c p-1 I < Po/W,
i
I
(ii)
is the maximum diagonal element.
0),
(n-1)
and Ai
ij
- max JAiji
P
Ai
A
1+1
and V
i
Since
P
(i.e.
exists
(C P-I~
Then
i.e.
(1 + Uo/j)o
max JA(nl)
i,j
Thus a
to
lxl
10 I i.e.
5.4
if
The
P
(n2 B
C P -1
exists
(i.e.
v
#
0).
is:
C p -I Ct
Then
A (n-2) -A
2
B- CAi
-
i+2,J+2
C
£2
2j
Since
JA111,
111
P-1
A11 A 1 - Ar 1 (AkZ A2 2 - Ak2 A 21, Ak2 A,1 - Ak. A2 1 ).
-
(C P)
is large relative
Pivot
-
A(2)
P'
is bounded away from zero.
is of order 2 and
Ak2)
IFP-
is useful iff
(k-l) at row of
Here the
let
P
WI/)JO
2x2
Suppose
(Akl,
pivot
1A22
_
221
v -
A11 A 2 - A21 1,
lAkil
and
<111 ' we have the following:
_
_--
IA.k21 <
0 , and
(C P -
~
C
il
C1 j
Lama 2:
(1)
(ii)
If
P
is of order 2 and
I
I (C P-1)
max
0
,
then
o + ii1 )Iv
11o
< (1 + 2o (jj
max
Idet PI " V
P)Iv)
'o
+
iJ
Thus a
2x2
pivot is useful iff we can bound
In particular from §5.3, we need to have
whenever
p1/p0
v
v
away from zero.
bounded away from zero
is near zero.
(Note that the use of the standard norm bound would give too crude
an analysis for Lemmas I and 2.)
5.5
Bounding
V
We can easily bound
A2
4A 2112~
11
Lea-3:
1
1A222
2
+ V2
IdetPI
from above, since
v
v - IA, A2 2
-
A:,2I.<
Thus we have:
+ M2.
2
V<_p2 + V
-0
1
This upper bound is sharp for
II
VO
-L0
0
0
-
0
l0
ow\n
o
But, as we saw in §5.4, we need a lowez bound on
V which bounds
I
i
26
v
away from zero when
V 1/M0
Clearly, such a lower bound
s small.
does not exist without interchanges.
Consider
A
0
- 0
with
1
P =10
001
We shall exhibit three different pivotal strategies in §6.1, §6.4,
and §8.1, which provide us with the necessary interchanges so that we
have a
2x2
pivot
Idet
P
with:
PI
(§6.2, §6.5, §8.2).
we have:
Assuming this lower bound,
Lenma 4:
If
Idet PI _ v>
I
then
>0
(i) max I(C P-1)ikl < P0/(PO - Ul )
for
k-l,2,
i,k
(ii)
(ii)ax I(n-2 )I.+MAX ij A'(
The lower bound on
2v100/(J
0
V > 0
1
0
0
is sharp for
l1
PO
L0
\
0
K_
A-
L 1- 0
0
Thus we have a good bound on element grovth in the reduced matrix,
since if
Vj/ O
1 0
1 . We
W shall see in
is small then
1
0
Chapter 10-12 that stability follows from this.
27
The Reduced Matrices
5.6
Av(n
On
Define
Let
A(k)
1 (k) . max
a
-A,
A
.(n)
)0
10
P
, (n)
1
1(k) . max IA(k) 1. and
(k),
1'
jj
(k )
o
Let
. JA(k) A2(k)
1
1
All considerations in §§5.2-5.5 hold for
5.7
k .
be the reduced matrix of order
0J
"y1
v
22
A (k)21
2
A(k)
Criterion for Choosing a ixl or 2x2 Pivot
We must find a proper criterion for deciding whether we shall use
a lxl or 2x2 pivot.
In Chapter 10 we shall show that the elements of the error matrix
are bounded in proportion to the elements in the reduced matrices.
For
stability we must ensure that the elements in the reduced matrices do
not become too large.
If we made our criterion to be the minimization of the number of
multiplications (additions), then we would want a lxl (2x2) pivot at
each step.
But this would be unstable.
Instead, let us aim to minimize the element growth that can take
place in the transformation from one reduced matrix to the next.
further temarks see §12.6.
Let
pivot for
F (k) be the growth factor permitted by choosing a
A(k) , where
J-1 or 2.
If the hypothesis of Lemma 4 holds (i.e. v(k) >()
for all
A~k) ),
jxj
then by Lmas 1 and 4:
2
k)2
For
-4
28
1 + jOIl
F
F=i
+ 21F-P
(k).
(k)
is not too small; while
has a good bound if p1
F
P
(k)
(k)
(k) (k)
has a good bound if p IPO
F2
is not too large. Thus we are
led to the following:
Definition:
0 < a < 1 , let
For
for each reduced matrix
(k) >a
P(k)
With
all
S
A~k)
,
S
be the following strategy:
choose a lxi pivot iff
(and a 2x2 pivot otherwise).
we have
F(k)
I
<
+ 1a
and
F 2k)
1+2/(l -
for
A(k)
But at any stage the choice of a 2x2 pivot carries us further
towards the complete reduction than does the choice of a lxl pivot.
Since the growth factors from reduced matrix to reduced matrix are
multiplicative,
tor
F(k )
factor
it is natural to compare the square of the growth fac-
permitted by choosing a lxl pivot for
F(k)
2
for a 2x2 pivot.
Thus the problem is to find
Theorem:
min
O<a<1
and is achieved by
Proof:
~.
max (( + 1/a) 2
a -a 0
The equation
quadratic with roots
L
A (k ) with the growth
max ((1 + I/a) 2 ,
min
,
I + 2/(1 - a))
-
1
+ 2/(1 -
c)}
(9 + 17)18
(1 + r17)/8
(1 + I/a) 2
-
1 + 2/(1 - a)
reduces to a
(1+±17)/8. Since the left side of the equation
29
is monotone decreasing, the right side is monotone increasing, and
a > 0 , the minimum is given by
a
0 =
(1 + ,r7)/8
q.e.d.
We immediately obtain the following bounds on multipliers and on
elements in the reduced matrices under strategy
S
Let
.
m
be any
multiplier.
K
Corollary 2:
V (k) >(k)
_U
2
Under strategy
<mi
a
a0
u
,
i for all
if
A
(k)
then:
1 1( k)2
Ia
For
S
I
for a lxl pivot
1/(1 - a) for a 2x2 pivot
(1 + /17)/8
(7i -
1)/2 < 1.562 for a lxl pivot
T + 7)/4 < 2.781 for a 2x2 pivot
Corollary 3:
Under strategy
(k)>(k)
)2
>III
O ) 2 _(k)
forall
P M ) < P0[(9 + /J)/
]
21
( n -t
)I2
S.
with
Akk
a - a0
A( ) ,thenfor
,
if
l<i<n
U0 ( 2 , 5 7 )n - i
In Chapters 11-12 we will give a much better bound on the elements
in the reduced matrices.
The strategy
S
allows us to proceed in the following order:
a
(1) calculate
(2)
if
(k)
Pi
V(k)
a0
and
(k)
ji0
pk
then we use a lxl rpivot;
(3) otherwise we find a 2x2 by some strategy.
Thus we
earch for a 2x2 for
A(k)
iff
(k1
1
a 0(k)
30
Chapter 6 : The Complete and Partial Pivoting Strategies
6.1
Complete Pivoting
As we saw in Chapter 4, the partial pivoting strategy can be
unstable when uNed on unequilibrated matric:s.
The trouble lies in
the fact that we do not have a lower bound on the 2x2 principal minors
which bounds them away from zero when the diagonal elements are small
(§5.5).
Let us therefore consider a complete pivoting strategy ("complete"
in the sense that we search over all the principal 2x2 minors, cf. §4.6).
By interchanging rows and the corresponding columns it is possible
to bring any diagonal element into the (1,1) position or any principal
2x2 submatrix into the leading 2x2 position.
Let
Vc
max JIAi
A2 1
-
i ,jii
The complete strategy involves:
(1) finding
il
max JAl
i
I
,
0
- max JAi I
i,j
(2) choosing a Ixl or 2x2 pivot according to
S
§5.7);
(3) for a lxl, interchanging so that IPI = l (§5.3);
(4)
for a 2x2, finding vc I.and interchanging so that Idet PI
(h
wor
2x2, b
firdachcrdc
(§5.4).
This would be repeated for each reduced matrix.
Vc
i
31
6.2
Bounding
However,
bound for
K
V
the result of all this work is that we do obtain a lower
in trnw of
vc
0
V
Theorem 1:
and
pI"
< 02 + 1"
Proof: The upper bound follows from Lemma 3 of §5.5. Let
IA, . Then v
max IAi
- AlI > IA A
-A 2 Iw
0 sC
rrso
rs
I,~
2
0
pI
2
A A > 2
rr as 20
since
= A2
and
rs
V
01
0
=max JA iii
q.e.d.
1
Thus Lemma 4 in §5.5 holds for
SC&,we
P
>
v
when
pl < 1i
choose a Ixl pivot for the reduced matrix
o
where
P
CL0
0
= (I
+
A( k )
According to
if
11i7)18.
In Chapters 10-12 we shall see that stability of this complete
pivoting strategy follows.
Operation Count for Complete Pivoting
6.3
Unfortunately, the operation count here is much larger than we
desire.
V(k )
The calculation of
additions.
Let
p
k
,
k(k-1)
multiplications and
be the number of lxl pivots used (so
pivots of order 2 are used).
indices
requires
1 < k < n
,
Let
for which
P)
A(k)
q - (n - p)/2
denote summation over those
uses a pivot of order
j
32
The searching for all the
(6.3.1)
1(') k(k - 1) + 1(2) k(k
equality iff p - n , while
6
-
requires:
multiplications and additions.
1)
n
1
I k(k - 1) n(n + 1)(n - 4)
r-1
n?2
Now (6.3.1) <
!
Vc(k)
n3 with equality iff
(6.3.1) >J
2J(2j
p - 0
q - n/2).
i-I
(i.e.
-
1)
.lna
n3 with
T
n(n + 2)(2n
From Chapter 9 we see that the rest of the work for solving
. n3
would require
in
multiplications, and between
6
I n3
4
-
A x -b
and
additions.
Thus the complete diagonal pivoting strategy requires between
n3
and
additions.
3
1-i
+.1
I
n3
multiplications, and between
l2
(To be exact,
p
n33 + 1 p3
_ n3
12
and
a
multiplications and
additions are required, where
p
is the number of
pivots used.)
Examples:
If
0
A
is
nxn
and positive definite, then
p - n
1
0
If Ais
nxn with
ni even, then
1)
p -0
lxi
ii
33
Partial Pivoting for Equilibrated Matrices
6.4
As we saw in §§6.2-6.3 the complete pivoting strategy is stable,
but it
involves more work that we are willing to perform, while in
Chapter 3 we saw that a partial pivoting strategy is unstable for
j
unequilibrated matrices.
We shall now show that a partial pivoting strategy is stable
when applied to equilibrated matrices (with equilibrated reduced
matrices).
Let
where
V
Let
be equilibrated, i.e. let
A
> 0
j
AiI
(usually we normalize by taking
- max IAll A
v
max
=
U0
0
for every
i
i ).
-A2
The partial strategy involves:
A
(1) equilibrating
p1
(2) finding
(thus we know
0
and choosing a lxl or 2x2 according to
(3) for a lxl, interchanging so that
(4) for a 2x2, finding
IPI
Ul
(§5.7);
05.3);
, and interchanging so that
v
S
Idet P1
Vp
(§5.4).
V
bounding
6.5
Now let us find a lower bound for
Theorem 2:
V0
0
1
1
<-
A
is equilibrated
2
2<%
2_
then
If
4
2
<i- + ]J"
0
1
p-
V
p
(max
lAijI
=
-
for every
i
34
The upper bound follows from Lemma 3 of §5.5
Proof.:
equilibration, either (1)
k > 2
IAklj - p0
with
p _o 0_i
- 0 .
p? -A:
,
or
,
If (i), then
.
(ii),
V
then
(ii)
p0
there exists integer
1-1 and, trivially,
> JAll Akk-
i1'
q.ced.
P2 _O-
Thus,
Sci
If
IAI1- U 0
By
Lemma 4 in §5.5 holds for
v
<
if
0 '
.
According to
we choose a lxl pivot for the equilibrated reduced matrix
of rdek
ff
of order k ifk
(k)
(k)
-1
,where
-
+
A(k)
/~/
Also, stability of the partial pivoting strategy for equilibrated
matrices follows from Chapters 10-12.
For
A (n )
A
only
,
required to calculate
k
requires between
v
2(n
(n)
)
1 2
2 n
6.6
1)
and
n(n - 1)
~
6
multiplications and additions are
(k)
v k
p
Thus the calculation of
P
nd
with between
-
for all
multiplications, compared
fcrallth
3
for all the
V(k)
in (6.3.1).
Citticism of the Partial Pivoting Strategy
The drawback to this method and the crite-.ion which we have found
for the pivoting strategy is that the matrix mus. be equilibrated at
the beginning and then each reduced matrix should be equilibrated.
But an algorithm for equilibrating symmetric matrices has never been
exhibited, i.e., for an arbitrary matrix
D 1 , D2
such that
21
we need
D
-
D2
1 . . . 2..
D
A D2
2
A , "Pe seek diagonal matriceb
is equilibrated, .nd if
in order to preserve symmetry.
. .. .
.
.
. .
.. . ..
. .
. .
. .
A
is symmetric,
35
We resolve the above predicament by two fundamentally different
approaches.
In Chapter 7 we exhibit an algorithm which can equili-
brate any symetric matrix in a very simple way.
In §§6.1-6.3 we
showed that complete diagonal pivoting avoids the problem of equilibration and is
stable, although the number of multiplications and
additions required is more than we desire.
But in Chapter 8 we shdll
exhibit a new version of diagonal pivoting which is
applicable to
unequilibrated matrices; we call this unequilibrated diagonal pivoting.
£his method will show that equilibration (in Wilkinson's sense) is
unnecessary for this strategy and this is the algorithm that we
recommend.
36
Chapter 7 : Equilibration of Symetric Matrices
7.1
Introduction
Wilkinson (1961) recommends that a matrix be equilibrated before
applying any algorithm for solving a system of linear equations.
matrix
A
A
is said to be equilibrated if all its rows and columns have
the same length in some norm.
Wilkinson's rounding error analysis for
Gaussian elimination (Wilkinson, 1961) gives the most effective resultN
when the matrix is equtI"brated, since a small perturbation of one row
(or column) Is then of the same magnitude as that of any other row
(or column).
7.2
Equilibration of General Matricee
In finite precision we modify the definition of equilibration.
(In this chapter we shall confine ourselves to the norm
10. - maxixil .)
i
A matrix
0 < max
-I
A
is row equilibrated if, for each row index i
< WO .
,AijIwhere
a is the number base of the floating
point system.
A matrix
index
PO<
j
A matrix
A
max
A
is column equilibrated If, for each columt,
JA1jI
<1O
is equilibrated if it is both row and column equili-
brated.
Usually we normalize by choosing
PO
=
1
in this chapter.
pO0
-
1 . We shall assume
8permits a matrix to be equilibrated by changes of
The use of
exponent only.
I
In order to row (column)-equilibrace
nb~ (D)
such that
is equilibrated.
DA
(A D )
A we seek a diagonal matrix
is row (column)-equilibrated.
1
To
2
However, there is no unique equibrated form of a
matrix for this norm (Forsythe and Moler, p. 45).
The various equili-
brated forms may differ greatly in their desirability for use in
Gaussian elimination, since the various equilibrations cause different
choices of the pivots (some are good choices, others are bad).
For the one-norm
(Ix0I
F em
ixi;) the equilibration is unique
but the convergence of the algorithm is slow (Sinkhorn (1964), (1967);
Siktkhorn and Knopp).
7,3
Difficulties Vith S
If
etric Matrices
is sbymmeic, then we allow
A
-2
1-
instead of
V0
as the lower bound it the definitions of row (and column)-equilibrated.
If
A
is syametric, then
equilibrated
1ff
A
A
is row equilibrated iff
A
is column
is equilibrated.
In order to equilibrate a symmetric matrix and still preserve the
symmetry we seek a diagonal matrix
Let
Le
dg
D
-
A
A
2
1
L
such that
- diag {1/2, D)
14/2]
diag {v' /4, /2
D
1
D
2
D A D
-
is equilibrat _d.
diag {i, V/2}
and
38
1)1 A D
Then
D3 A D3
1
/4
1
I_
1
D2 A
2
18
D2
,and
1
are all equilibrated.
-F
Criteria for choosing pivots are based on their size.
It is an
open problem whether all equilibrated forms of a symmetric matrix give
satisfactory pivots as judged by the usual criteria.
There is no known algorithm for the symmetric case.
The Obvious Attempt
7.4
Let us cc sider
an
nxn
D A D .
symmetric matrix.
that no row of
Let
Then
D - diag {dl,...,d n I
(DAD)ij - d i d
A
and
A
be
Let us assume
.
is all zero.
A
The method which seems the morc obvious is to equilibrated one
row at a time:
Let
D1
diag{dl,1,...,).
C Aij
Then
(D AD )ij
So choose
Now
DIAD1
d1
-
=
for
3,4 ' I
dI Aij
2
for
or
J'l, i#1
I All
for
ijml
max{
S2<_j 'IA
11
,
i 1, J0l
max<n IAij )
is symmetric andlmax
l
i
1
-(DIAD
.
However,
when we tr'/ -o equilibrate the second row we usually destroy the equilibration of the fitat row.
39
This method is related to the Sinkhorn algorithm which equilibran
tea in the one-norm,
1x
iii 1x
1 , rather than the -norm,
Ix1 I , which we are using (Sinkhorn (1964), (1967); Sinkhorn
|01.o -max
I
and Knopp; Marcus and Newman).
7.5
Equilibration of Lower Triangular and Symmetric Matrices
The problem with the method in §7.4 is that we try to do too much
at each step.
T + A + Tt ,where
A
is strictly lower triangular (i.e.
T
Let us consider
matrix and
T
Lemma 1:
If
At
A
T + A
Let us consider the ith row of
Proof:
Aij I
equilibrated,
max
ITijI < 1
Aji = Tij
each
i
is row-equilibrated, then
and
1 .
=
for
But
0
-
A
Hence
for
max
J > i ).
if
is equilibrated.
T + A
A . Since
IAjij < I
j > i.
isadiagonal
J
i
IAij
,
is rowsince
ik
q.e.d.
.
Now we shall show hcw to construct
row-equilibrated if
T + A
D
such that
has no all zero row.
D(T + A)D
is
In this case, by
Lemma 1, D A D is equil!.b:'-d.
In fact, it is eas-
uilibrate the lower triangular part
of a symmetrJc matrix and s..
Lemne 2:
for
di I
i
I
Let
B
be an
preserve symmetry.
lower triangular matrix (i.e. B
nxn
> i ) with no all-zero row.
max {
brated, where
1il
D
=
max
Id
l i< i-lI
diag
1 < i < n , let
For
Bi
d I ....d}
-j
n
-
}.
Then
D B D
is row equili-
0
40
by hypothesis.
Idj
dk BJki - 1
, and
1
0
B 11
" 1
have been chosen so that
dl,...,dl_ 1 > 0
Assume that
max
Id2 BI
So
d
Let
By induction.
Proof:
I < J < i-1
for
l<k~j
Let
d
i
max {
,
Idj BiI I
l<1max
By hypothesis,
1:3.
<~
di
> 0 .
j
Then
Hence the
D
-
diag {d
max
J
i
di
, . . ,d n
exist for
I . Then
But what do we do if
1 < i < n
D B D
Let
by induction,
is row-equilibrated.
q.e.d.
0
A1
or if, for some
i ,
Aij =0
1 < J < i ?
Let us form
di
.
The Algorithm for Null Rows in the Lower Triangle
7.6
for
ijI
Idi d
1
if
as In §7.5 with the exception that we set
D
for I < j < i .
Aij = 0
Thenfor all i,j
D A D
: l(D A D) iI
< 1
fails to be equilibrated only if for some
(a)
Aij = 0
(b)
max Idj AijI
j >i
1< j < i
for
,
i
or
< 1
th
row of A is not null, then the maximum in (b) is
If the i
i, define ei by e-l = max Id A I ; for all
positive. For such
other
i , let
ei =1.
Let
E =diag {el,...,e
n
41
Theorem:
Let
D
and
Let
E
A
be an
symmetric matrix with no null row.
be constructed as above.
a diagonal matrix, and
Proof:
nxn
If
6 A A
ei A I
Let
A - D E .
Then
A
is
is equilibrated.
then the maximal magnitude of row
i
in
D A D
E D A D E , while in all other rows the ith
is raised to I by forming
element is increased in magnitude but not in excess of 1
The theorem follows from §7.5.
7.7
Summary of Equilibration of Symmetric Matrices
If
A
is an
nxn
a diagonal matrix
that
q.e.d.
D A D
symmetric matrix with no null row, we can find
D
(in two sweeps, although only
is equilibrated (in the u-norm, |xi.
We can express
-
D
by the following algorithm:
_ {
max
Then, for
i
-
/1iV,
l<_ii
max
steps) such
For
for
if Aij
A
i16 jAij
Ai )
I < j
).
i = l(l)n
some
Aij - 0
if
lxii
max
i
=
6ii
1
=
n
j
0
for
,1<
J
<i.
l(l)n:
if
A
#0
for some
j,
< J < i
i
max
i+l <j<nI
Let
D
=
diag (dl...,d
n
I .
i Aji
Then
I
if
D A D
The work required to equilibrate an
all zero row is:
n
square roots
n(n+l)
multiplications
(n-I) 2
idul tions
Aij -0
fur
1<j<
i
is equilibrated.
nxn
symmetric matrix with no
42
In practice the algorithm can be expressed in a very simple
manner in Algol (see kppendix B) and can be performed in only
(We would actually set
then search for
7.8
fi "
0
max
16j AjI
i+l J<n
if
A
0
for
n
steps.
1 <j < 1
and
f - 0 )
±
iff
The Algorithm for Exponent Adjustment
6-2 <
In practice, we actually only require that
for every
i
max IAij I - 1
lj _n
instead of
for every
only adjust the exponents of the elements of
max
i
JAij
I
<
so that we need
A
Then our algorithm takes the following form:
Aij = Yij B( j
Let
AJ4 = 0
for
0
when we take
iiJ
Let
A
when we take
j
i
=
a
i
0
i
-l(ln
set
if
Aij - 0
for
1 < J
0
(a
ij
- 6
ma
i
set
i
6
'
p2X i1
i = l(1)n:
Al1
ii
ax{max
For
<1,
I
unless
<1 , unless
C4_
0
For
Iij
ij
where
yi
-1 <
, where
d
if
J
)} otherwise
61 #0
i
max
(aj - 6j)
li
<n
i
-.
if
6, = 0
43
This will give
S-2
<max
16d +
A i
< 1
for every
±
(This equilibration can be performed very rapidly in mach rne
language.)
44
Chapter 8
8.1
Unequilibrated Diagonal Pivoting
Maximal Off-diagonal Element
In order to obtain a lower bound of
W2_0
0p
for
v
in Theorem
1 in §6.3, we needed the fact that, due to equilibration, there existed
an element of maximal absolute value in the first column.
P1 < P0 I then there exist integers
jA11 i
i > j ,
with
i,j
WO * We need only bring the element
A
However, if
such that
up to the (2,1)
position and then we will have a 2x2 pivot with a maximal off-diagonal
element.
We shall call this variation unequilibrated diagonal pivoting.
Let
gers.
U 0 . max JAiji
i,j
Let
Let
pI = max
i
V =A
b
IArsI , where
r,s
are the least such inte-
lA il•
- A'
A
rr
=
ss
rs
This strategy involves:
(I) finding
u1
and the least integer
(2) finding
p0
and the least integers
k
IAkkI
[1
with IArl = PO
with
r,s
(3) choosing a lxl or 2x2 pivot according to
S
(§5.7);
(4) for a 1xl, interchanging rows and columns I with
IvI
= 1P
so that
r
and 2 with
(§5.3);
(5) for a 2x2, interchanging rows and columns I with
so that
k
Idet PI = Vb
and
IA211 - o0
(§5.4).
This procedure is repeated for each reduced matrix.
s
45
Note that calculating
of
n(n-l)
for
vC
and
Vb
requires only 2 multiplications instead
2(n-l)
for
vp
Clearly from the definitions of
vb, V,
and
and from Lemma
V
3 of §5.5, we have:
v
Lemma 1:
8.2
< V
-
Bounding
' V
p -
A
2 I
-
A21
Theorem 1:
If
IA11 A 2 2
b
Proof:
Since
+
2
0
2
1"
vb
Let us now bound
-
<
c -
vb
from. below.
and
-
From §8.1, we may assume
0
A22
00
IA2 1 1 ' O - then
< 120 +i W
<<
--
0 _P2
-1
P2
The upper bound follows from Lemma 1.
IA2
=
211 -0
0
'
b
IA
2
- A 11 -0
A
1
22
21
A
-A
0
1
>
2
2
22 -0
1
q°e. .
Here, as in §6.2 and §6.5, symmetry was used to get the lower bound
on
Vb
* By symmetry, if
If
IA12 1
A
- P0
PO=
IA
21 1
were not symmetric, then
(in fact we could have
JA12 1 ' 00
then
p0
=
1A
21 1
A12 ' 0).
does not imply that
Thus no such non-negative
lower bound on the determinant of 2x2 submatrices can exist for nonsymmetric matrices.
Thus Theorem I impltes that Lemma 4 in 55.5 holds for
vb
if
WI < PO * According to Sa (§5.7), we would choose a lxi pivot for the
reduced matrix A
of order k iff p k)>
W
where
t0 - (l +
T7)/8.
in Chapters 10-12, we shall see that stability of this strategy
(for unequilibrated matrices) follows from the above.
-
46
8.3
Comments on this Strategy
From §8.1, we see that we need do no searching over the 2x2 prin-
cipal minors, but we merely choose that principal minor with maximal
off--diagonal element.
O(1n3
for the
W k)
In Chapter 9 we shall see that this searching
and
requires between
-
.
I--
In
n3
additions
and no multiplications.
We used the terms "complete" and "partial" strategies in Chapter
4 to distinguish between searching over all the 2x2 principal minors
and over the 2x2 principal minors with off-diagonal element in the
first column.
In analogy with Gaussian elimination with complete pivoting, we
would like to call our strategy in §8.1 a complete pivoting strategy
since we search all of
(k)
A
for its maximal element, but we do not
wish to cause confusion with the use of the word "complete"
in Chapter
4 where it meant searching all the 2x2 principal minors.
Now, in analogy with Gaussian elimination with partial pivoting,
we ask if there could be a partial strategy where we search only the
first column
of
A
(k)
for its maximal element.
Such a partial s=trategy requires at most only
l
-
2
n (n-i)
addi-
tions to calculate the maximal element in the first column of all the
reduced matrices.
If such a partial strategy were
13
strategy would require only
additions
to solve
A x
b
a
a
= At
table, then this
multiplictions and
det A
0
• A #0 ,
de
6
3l
47
However, any such partial strategy is unstable for unequilibrated
matrices, as the following example shows:
A
where
0":aLx<1
8.4
1
-
and
1
O<cE<<I
A Partial Strategy for Equilibrated Reduced Matrices
A - A (n
Clearly, if
and each of its reduced matrices
A(k)
equilibrated (i.e.
max
ij
IJA I
(k)
= Pk
for each
A
(k) is
i ), then the partial
strategy, whereby we choose the 2x2 principal submatrix whose offdiagoral element is the maximal element in the first column of
A(k)
is stable since, by the equilibration, such a maximal element in the
first column of
A(k)
is also a maximal element of
But we would have to equilibrate
librate each reduced matrix
A
A(k)
at the start, and then equi-
A(k)
Let us now consider such a partial strategy when
A
is equilibrated,
but we do not equilibrate the reduced matrices.
8.5
A Partial Strategy for Uneguilibrated Reduced Matric,=s
Let
max
j
%j
A
be an
- p0
nxn
symmetric equilibrated non-singular matrix with
for each
i . We shall now consider the partial strategy
defined in §§8.3-8.4, but we shall not equilibrate the reduced matriceb
A(k)
for
k < n.
48
A( k )
Let
-) max
ij
Let
(We shall not actually calculate
(
(k)
A)
say.
k)I
A
interchanged
II rows and columns so that
a
JA (k),
.
A
0
(k),
.(k)
Let)I
X (k) .
A( n )
k ; let
be the reduced matrix of order
So
X(k)
We shall use a I1
Ij(k)
We shall assume we have
(k )
_
,wie
Then let
(n)
(in)
(k)
(k)
Pl
> a X
pivot iff
Let
.1
m be any
multiplier (§5.1).
I
(k)
> a XW
1
--
then we use
as a lxl pivot, and
A
1
Iml < x(k)/P(k) < i/a , while
max
(k l)' , 11(k) + X(k)/a <
IJA
(j +j i/-I(k
0
(k)
If
(k)
Then
I)
-
Il
Then
Im
then we Interchange so that
11(k) < a X(k)
(k) (p(k)
X
(P
22
(k)
()
)
-
2)
and
(P(k) + W(k) )V(k)
As in §5.7 we would choose
-k
.
IA (kI < (2.57) n
(k)
v(k) >k()
X
(k)
)/V
+
2)1 <(k) + X (k)'
max JA (k2
wax
So
.
JA2kI , -
a - a0
-
< (I + 2/(l1
(l + v'Tl
.
(k)
Then
A(k)
for each
i
But we cannot obtain a bound on
we cannot express bounds on
v (k)
A(k)
as in Chapters 11-12 since
in terms of
(k)
X( k)
'
49
We leave the significance of this method (§8.5) in relation to
the method in §§8.1-8.4 as an open question.
Chapter 9
Operation Count
Solution by Diagonal Pivoting
9.1
We now consider the amount of work required to solve
A X
-
B
by the (unequilibrated) diagonal pivoting method (Chapter 8), where
A
is an
nxn
non-singular symmetric matrix and
i.e. there are
k
B
is an
nxk
matrix,
right hand sides.
In matrix notation, we perform the following steps:
(i)
A= M D t
(2)
C =M
-1
(3) Y: D
t
B.
C
(4) X= M - t Y.
Here
M
-t
Let
-it
(M
means
p
)
.xl
be the number of
pivots of order 2 are used.
Let
pivots used.
A
So
q =
-p)
be the reduced matrix of order
Definition:
pivot [i] -
We shall use the term
plications (additions).
We shall use
Z (J
1 < i < n , such that
9.2
{l
if
A(i)
uses a
lxi
pivot
2
if
AW
uses a
2x2
pivot
L
ifA (i + l )
Mults (Adds)
uses a 2x2
to mean the number of multi-
We shall count a comparison as an addition.
)
to denote summation over those indices
pivot[i] = J1
Summary of the Work Required
Stcps (1),
(2),
pivot
(3),
(4)
In
99.1 req~tire;
i
i
51
1
3
+ (k +
Mults -
1 2
i3
)n
+ (- -2k)n
2
3
n 1 + (k +
-6
Adds
4
4
(53
)n+
4~
+ (k + 3/2)p + X(2)
2k)n+
l
n3 + (k + 5/8)nW + (2
4
n3
7)222
-(2k
3
2k)n +
U
/8)p +
4i
(1-1)
~-n4 +-i=p3
13
1
<1n3 + (k + -) n'
(k_- _)n
-3
For
k
1
l n3
(i.e. one right hand side), let us compare the work
required for the diagonal pivoting method (applicable to all non-singular
symmetric matrices) with that required for Cholesky's method (applicable
only to positive-definite matrices).
Cholesky
Diagonal Pivoting
Exact
Mults
Adds
Root
Reciprocals
3
1
n
3
+n
62
2
+
Lower Bound
3
7
-
6
2
3 + A3
1
n
3 2n
n
n
n
Upper Bound
3
2
-n
5
n
9+ n 2
3
6
1
n
3
0
2 n +
3
12
4
+
3
2
-
0
Since the time required for a computer to perform a multiplication
1i 3
is much longer than for an addition, the requirement of
plications and between
-
4
3
and
tory for symmetric indefinite matrices.
K
-
3
n3
-
6
n
multi-
additions is very satisfac-
5
n
52
If A
is positive definite, then Cholesky's method is preferable
to the diagonal pivoting method.
9.3
See also Appendix A (SA.l).
t
A-MD M
Forming (k)
First we decompose
A
into
A - M D M
Let us consider the reduced matrix
t
of order
AM
i
The following chart shows our course of action.
1()
find
find
Adds- i- I
Add
LL
P MH
1-- il
ulLs, Adds-1
no
yes
use lxi
use 2x2
inechange
I
~Mults-2
calculate det:t Ads -2I
interchange
I
Mut
-:calculate
multipliers
{
Mults, Ads 'omA(-)calculate
A)
Adds(
1
2
(11
multipliers
form
j
i-ii
A (1i-2)
:-
1-2
ults -6(-)
Adds - 2(1-2)
Mults, Adds
53
Thus for
A(1)
A.
2
Mults
i(i+l)
Mults - 12 - 3i
and
-
2
Recall that there are
For step (1):
pivots and
lxl
_13n +
Adds
-
(
(2) (ii)(3
i2 +
1) +
I
2 + 1 1 + _
y(2)
4
+
(
-7
1(2)
85n
8
Now let us bound
<
[(2)i
2
i)1 +
-
(1_1)2 +
(
-
/i)1
5
4)
2
(n-p) +
I(-2
8
5
+
n
3 [42) i
p+2J
+)
4
)-
1
5 2
1 3
n3 +
n2 -4
(
2
2)
1
(9.3.2)
(i-l)i + 31
7),
7
+Zp+3
2
6-
2x2
+ 31 - 7)
4i
(i+l) + -
19
(n-p)
-
pivotti] - 2
n + 11
S(i){43
n"
q
if
12
i2 +
"
(i-l)(j i+l)
1
i
++
1 n3
I2
n +
6
1
3
p
t +) ) (31(9.3.1)
if pivoc[i]
Adds -
+ 1(2)
(I i (
1
7 and
Ti ([+1) +
Tjl)
Mults -
Adds
=
.p +
1 t4l)
-
z
1 (1-2)
844
and
-2
1 (1-2)
) +
(n-p)
from above.
and
44
pivots.
54
(1)
(i-2) <
i
5
1 p2
< n
(1-2)
i
-
p{n 2 -(p+l)n
2
+ " p + !
P
i-n-p+l
So we have the following upper bounds for (1):
(9.3.4)
+
Mults <_
4
-6
1 n3 + L 2
1
n
Adds <1 3
9.4
Solving (2).
(93.3)
p
(I1 p +
+
p
2
4~
2
2
) n
i(p2+P+l )n +
4
I1 -Z+ 5
8
+T1
(4)
(3.
-1
M 1 B :
C
For (2)
n
3
Multe, Adds
-
(Note that
M
n+!-p) k
(10
a
[i] - 2.)
if pivot
-1
C :
D
Y
For (3)
3n
Mults
2p
-
, Adds - n
p
-
-t
I
-M Y
For (4) X
1n2
n2 -
Mults, Adds
1
n +
p)
k
Total Work Required
9.5
Thus steps
Mults
Mults <
6
)1
T1 + (k +1
3
n- + (k+
1 n
+
)
+
5
2
1) +
1
(4) together require:
(3),
(1), (2),
132+3
1
+
(L-
3
Adds =
5n
+ (k + g) n2 +
n
3
+ (k +
Adds( <
3
1
Adds >
1-
+ (k +
)
1+
5
2
)
3
p
p
2
1
n2
+ (k +
I
(.
p + 3
n + (k +
7
2k) n + (k-)
4
( -
Mults >6
)n
-2k)
4P
1(2)
2 +
2k)
n.
2k) n
k
k
-kn+
1P3
2k
(
3 7
-
2k) n.
i(I
3
)
p +
1
3l
1 P2+
i
(1-2)
p.
|i
Uper Bound on Mults
9.6
that is independent
We would like a minimal upper bound on Mults
f(x) - (k
Let
f(P) <
4
(-2k)
f
n
0 <p
p , where
of
(
.
By elementary calculus,
x-
-
Thus
" T2 (2k- n7)
2
n + -- ( k-7)
2
g n
9zn
+
25
n
+ (k +
i)
(wnen k
n + j-
-
n
ults <
2
Uer Bound on Adds
9.7
bound on Adds that is indeNow we would also like a minimal upper
p
peadent of
i
Ltgx
Let
Since
-- 4
~12
+( z_
x -
g(x
g'(x) > 0
(p)
10,n]
on
n
1 n3
+ (k +
32
+ - n22j
2 n
-
4 n + k-
n
(k -
k
8
)n
4 n
(_
we have
n
12
Adds < 3 n
x2 +
1 2 )n.Tu
- (k - -) n
(where
k
l)
Thus
12)
x.
56
Chapter 10 : Error Analysis for Diagonal Pivoting
10.1
Introduction
Let us attempt to solve
method.
If
consider
xc
x
A x - b
by the diagonal pivoting
is the solution we obtain from the computer, we may
as Lhe exact solution of the system
(A + E) z - b
As in §2.5, we are interested in showing that the elements of
small in comparison to the corresponding elements of
1960, 1961, 1963, 19b5).
Such an analysis of
E
A
E
are
(Wilkinson,
is called a back-
ward error analysis.
10.2
The Occurrences of Error
Suppose that we could perfocm the diagonal pivoting method in
exact arithmetic.
In order to solve
A x - b , we perform the fol-
lowing steps (see §9.1):
(1) A - M D M t
(the decomposition)
(2) c
b
(the new right hand side)
c
(solve the lxl and 2x2 systems)
-
M'
(3) y - D
(4) x
-
Mt y
(recover the solution)
However, in finite precision arithmetic, we have error at each
step.
Instead of decomposing
such that
M D Mt. A + F.
Mt y , we actually obtain
x -
(M + M2
)-
t y
A
into
H D Mt , we obtain
Instead of calculating
c - (M + M 1 )
for some perturbations
Thus we actually perform:
M -] b
,
b , y - (D + 6)
Ml ,
6 D , M2
M
and
D-
c ,
1
D
C ,
respectively.
57
(1) A + F
(2)
MDMt
cm (M + M)
(3) y
(D +
6
D)
b
1
c
(4) x= (M + M 2 )- t
10.3
The Error Matrix
Thus we have
E
b - (M+M) c
(M + M 1 )(D + 6 D)(M + M 2 )t x
-
+ H [6 D (M + M
(A +
E
-
+ D Mt
E) x - b .
1)
(M D Mt + M1 (D + 6 D)(M + M2)t
-
x
(M + M)(D + 6 D) y
M D Mt a A + F
but
.
,
while
Henc,
F + M 1 (D + 6 D)(M + M )t + M [6 D(M + M)t + D M
if we can bound the elements of
then we can bound the elements of
error lies in
, D , M
# M2
1'2'
and
6 D
E . We shall see that most of the
F , in other words, most of the error occurs when
obtaining the decomposition
10.4
F , M
H 1 MD
of
A
Notation
In the following sections, the symbol
ICI < 2 -
numbers with
in
<
(1.06) 2-
ber of binary digits in the computer.
E,n
,
will stand for any
where
t
iG the num-
Each occurrence of
C
or
n
in an equation should be indexed, but we shall suppress these indices
for the sake of clarity.
and
i < k < n
,
shall denote the summation over those indices
with pivotik]
= 1 and 2 respectively.
k
-tt
g(t) - 0 (2- ) if lim 2t g(t)
We shall write
is finite.
Summary of ihe Error Analysis
10.5
In the following sections we shall show:
(10.5.1)
t
IFjI 2
(10.5.2)
IEi jI
s
F - Ft
, and
I
for
IFij
+ £
where
,
2
2
max (1/a , 1/(-,1l
- t
£ < 2
(10.5.4)
1E
ma)
.
k
(k)
for
i
o
1
2 x
))
Also, for a - (a
0 , we shall show:
(k) + 31.6
IF 1 2'1 5.71
(10.5.3)
j
Ijj
(k)
( k) + [I + l1.02/(l-a)]
11 + 3.01/a]
'
(k)
L
0
0
J
< 2- t max p0(k) (23.54)n
k
n
where
II
is the one-norm:
1E1
=
1I
max
j i=1
In Chapters 11 and 12, we shall show for
a =
(10.5.5)
-t
rn f(n) 110 (3.07)
max IFijI < (15.8)n 2
iJ
(10.5.6)
IF
(10.5.7)
NEI < (23.54)n
< (15.8)n
2-
2
2
t
a0 :
nO
446
n f(n) 1O (3.07) n0.446
2- t
0
n f(n) V'
0 (3.07)n .446
For Gaussian elimination with complete pivoting (see §2.5):
(10.5.8)
@0< (2.01)n
2
2- t
,n f(n) 11O I where
For further remarks, see §10.12 and §10.16.
LU- A+ F .
59
The Decomposition for the Reduced Matrix
10.6
Let
a
A(r)
be the reduced matrix of order
pivotir]
*
Let
(see §9.1).
AB(r )
Thus
*
is
1 or 2
)
(r-s)xs
]
Air)
8
We shall use
.
where
P(r)
s
C(r)
a
M(r)
(P9
be the
(r-s)xs
sxs
(r)
= M(r)
S
in finite precision.
s
- )
A
But
S
r-s
Let
(r)t
-
M (r)
10.7
-
A r)+
G(r-s)
Fij
=
Let
be the
(M(r)
S
C
+ Ma
S
(r)
S(r)
The Error Matrix
j
(r-s)
+ (M Dt
-
S
F (-)-M~r
From §10.6, we have
>
(r-s)xs
Ar)
be the
S
Mr) p(r)
SS
FCr-s)
FG
be the
resulting from calculating
G
M(r) p(r) M(r)t
Hence
G
(r-s)×(r-s)
A (r)
-
M (r ) C Wt
error matrix
A r) + M r) c (r)t
r-s )
_f.S
pivot.
(
(
(
M(r)
Let
_ c(r) P(r)-i
S
reduced matrix of order
(r
axe
~(r(r)
)
i
as the
is
matrix resulting from calculating
in
nr)
finite precision.
error matrix :
C
P (r)
s
Let
a
C(r) t
c(r)
s
C (r)
Let
.
A(r)
p(r)
A (r
r * 1 < r < n
be the matrix resulting from deleting the first
rows and columns from
Lez
a
A(r)
PC
(
r)) p(r)]t
(r)t
S
Mrt + Ar)
,where
Wt
0 S(ra
F
for the Decomposition
A+FMDM
t
where
, where for
F
F
t
and for
i > J
I
I
'I
60
Gij = 2 ij
(k)
with
G(n-k)
G-k,--)
C(k
LI.(k)
if pivotfn-k+l] - 1
G(-kl
Ui-k-I
,j-k-i
(n-kl)
if pivot[n-k+l]
- 2
0
if pivot[n-k+11 - 0
C- (Q(n-k))
0
j > k :
and for
jk
if pivotin-k+l] = I
J-k,l
1
(0n-kl))j-k-1,
=
I
(0(
t2 n - k) )j-k-l,2
The Error Matrices
10.8
1
6 D,
if pivot~n-k+1] - 2
if pivot[n-k+l]
0
M2
From §10.6,
(M + MH) c = b
(Mi n -
J )
i
if pivot[n-j+l] -1
(2 (n-J -I))
Mij
( 2-l
(M2
for
while
i > j
Mj
Thus
if pivot[n-J+l] = 0
)i-J-l,2
0 if pivot[n-j+l] = 2
J+1 ,j
for every j , and Mij - 0 for j > i
,with
1
if pivotin.-j+l] - 2
(Ml)ij
H
= 0
for
J > i
and
I ij
(H)
= 0
if
1 J+1,j
pivot[n-j+1] = 2
(D + 6 D)y = c .
From §10.6,
blocks of order 1 and 2.
L
D
is a block diagonal matrix with
The blocks are the pivots
P(r)
S
in §10.6,
61
s - pivotir]
.
(M + H2 )
I2
10.9
Also,
x -y
6 D
has the same block structure as
D.
has the same structure asM
.M,
Floating Point Error Analysis
Since most computers now perform calculations In floating point
arithmetic rather than in fixed point (see Wilkinson, 1965, pp. 110188), we shall give an error analysis only in floating point.
z = fX(x * y) E
Then, from Wilkinson (1965), pp. 114-117, we have
jej < 2- t
(x * y) (1 + c) , where
used by the computer, and
*
,
t
is the number of binary digits
is any arithmetic operation (+, -,
x .
Further, we shall assume that the computer can accumulate innerproducts in floating point arithmetic.
Then
Ift (x1 Y1 +
+ X
...
Yn)
-
n
(XI yl +
+ xn Yn)I <
...
<3
-2t
<
(1.06) 2
x i yi
2 -2t
Case 1:
M
So
O r
o ij
r-)
Ar)
s,
.
JA
-
pivot[r]
I
and
are
l
n
is any number
and we shall suppress indices.
Floating Point Analysis for
Consider
, where
From
Inl < 2-' 0 ( 2 - t)
§10.4, we may write
10.10
iIy
As in §10.4, we shall assume
.
Inm < (1.06)
such that
)
i~I
CI
1
(r),
j
Let
r)
C
i > j
P
(r-l)xl
_ (11A
-
F
r)
I (elPj
PiJI
A
P
(1 + C)l (I + L)
I
=I
62
(r-l)
)(+
(C1
)1
(M1
c
(
Ei
So
(2 + c)
(r)
C(r)
-(r))
-1
2 -t
Cr-I)
0
-
r)
-t
A(r)l (I + E)
A (r)
j+1,1 11
(MI r)
()
(r
(r)/(r) <
(r )
IMr
(lOlO2)
1
-1
A Cr)-l
(r)
J
)
1
"ij
1
I
Il< (1 + 2-:/c
•
+ 2/c + 0( 2 -t)]
,
I <
IM r)p(r) 0 (r)t)
.2.1
(1
I(CC .1
)
t
I(M (r))
t
2-
IE-)I
(10.10.1)
(r)-1
£(C(r)
-
A
(r)
(1+
2
- t )
Thus
and
=
2-t/O
0
(r) 2- t [1/a + 0(2-t)]
0
Case 2:
s
-
pivot[r]
m
2
)
M2
and
So
dj
, 02
(
and
C2
are
(r-2)x2
)
(C)
r) 2 >
2
1
i (C(r))
+ (1+)
(r)>p
2
A (rj ) -(([( 2Cr))ij-(
( (r)
)
2
A121
P 2j
) ,( )2
dEi
'+
-
<j
-A
-
11
/a
and
<_ 2-/t
(r)
1(0
(Q r))
r)r).
t
+ 2-
I(M r))J P 1
So
)
(1 +
(r)-l.
(C r)
So
1J))
(C
- IAi+1,+lLuJli 0
1
( r)
<
I
(A(r))(r)
63
2
(M~r) ) i~~
(C
2
)
(A(r)
(r- )
2
1j
C
(Cr)
12
[2 + 2- t]
M2
)1
T(r)
A(r)
ii
)JI
+ [3 +
(,
I(M 2 r)
0 (2-t)]
+
2
(r) (r)
(r)
-A 1
A2
(r)'
A
A1 1
A(r)
A2r
21
0)
(I +
+
Cr)
E (2 +
IE)
21
C(r)(r
(2 + E) - A2
A2 2
22i2
2
Ar) A2 2
A
(1+
(r) (r)
22
A
(r)
M (r-)i )
A(r)
Now
Il (
1]
+ C)3
[
(1+ U)2
IG(r- 2 )I < 2-t
(10.10.3)
((r)
2
~J2
( 2
From §10.6, we have
) .
(r)
-
)
()
-2
) )(1 +
(I +
E)
(r)
- A22
+ A2(
21
(r)
Al(
11
22
C
From §10.5,
1(0 2
C)(r)
()
< to0
)i
W(r)
2-
t
(2 +
(r)
1r)L
(10.10.4)
r))
2-
2-)
(2 +
p(r)
(1+(
2
)
2-
1I
since
(3/(l - a) +
O<a< 1
0
(2-t)1
(O2( r) )12l
<
- t
- 2
{3/(l - a) + 0(2-t))
c
.
2
-
22
Similarly, we obtain
(10.10.5)
2 -t
O<{
+ a) < 1
Ct
r)2
t](r)
+ W1I2(r)
(r)2 2
Cr)
+
_
(1 + C 2 )/l
(O
2- t
t)
+
(r) _
0
But
2
+ pr))
+
Thus we have
i+
+--
64
Frm
106 106,wereal:
we recall:
I(C~r) (.(r)-l)
(10.10.6)
I2(r)2 p(r)-l + U2
0 (r)
M 2(r)
(r)+ (r))/ (r)< 1l
. (r)
Thus from (10.10.4) - (10.10.6), we have for
(10.10.7)
< (I + 2- t (3 +
I(. r)
But
2
-t]}/(l
j
a
1,2
-
a)
From (10.10.3) and (10.10.7), we conclude:
(r'2) l
-t oCr)t
+ 5/(1 - a) + 0(2-t)}
(10.10.8)
I1G
2<1
1r
o we n
Now we need to bound
recall:
M(r)
2
(r) (r) (r)t
P
02
j
(
(r) . C (r) +
2
0
From
§10.6, we
(r) p(r)
2
2
From (10.10.4), (10.10.5), and §10.6, we have
(10 .0 9)
since
i (r) p (r)
.. .. 2
p Cr) <
<<
2'iJ'
(r)
2-t
(1 +
[3(1
)/(!
+
-
c) +
C, r)
From (10.10.4), (10.10.5), and (10.10.9), we conclude:
(10.10.10)
10.11
I(M~r) p(r)
0
r)t)
I
<
+ 0 ( 2 "t)} 2 p(r)
Ot/(
Summary of Floating Point Analysis for
IG
I
2 t < [1 + 2/C + 0 (2-t)]
[1 + 5/(l - a) +
(10.11.2)
I(M D Ot)I
2
l/A
t <
16/11
-
a) +
0
0
2 t
i > J
p1 Wk) +
Wi
k)
(2 -t)]
0(2-t)
+
1
F
From §10.7, (10.10.8), and (10.10.10) we have for
(10.11.1)
2
i
(k)
+
1
"(
---------------------------
V
65
From (10.11.1) and (10.11.2), we conclude for
IFi1
(10.11.3)
I
2t < 11 + 3/a + O( 2 tfl
[I + 11/(1 - a) +
We shall now asp',me tha. the first
0(2 - t )
0
( 2 -t)
k
1
0(2- )
(k)
term in (10.11,3)
for all
a )
and
term in (10.11.3) is bounded by 0.02
when
t > 14
is true for
(k)O
,
t > 8
is bounded by 0.01 (which is true fot
that the second
,
i > j
(which
a - (0 a (I + /r7/8)).
Under these assumptions, we have:
(10.11.4)
IF I
1
while for
a -a
(10.11.5)
IF I
1
11(k) +
2 t < [1 + (3.01)/(J
1 + (11.02)/(1 - ) 1 1i]Jk
0
we have:
,
2'
+ 3.16 1" w
< 5.71
Comments on the Bound for
10.12
F
From (10.11.4), (10.11.5), and §10.3, we see that we have a good
bound on
F
(and hence on
the reduced matrices.(
E ) only if the maximal elements
< (2.57)n-k P
good bound on
have
t 4 50 .
F
Thus
if
n > t
OL- C
(10.11.5) does not automatically give a
.
At the present time, most computers
Hence (10.11.5) cannot guarantee a good bound on
for systems of order
in
do not grow too large.
From Corollary 3 of 55.7, we have for
(k )
(k)
F
> 50
i
66
According to Wilkinson (1965, p. 215), if
I < (2.01)
max IF
iJ
x(k) < 2n
max
k
n
k)
L U
-
A + F
,
then
for Gaussian elimination, where
for partial pivoting, while
MIx
(1) < rn f(n) )0
for complete pivoting.
In Chapter 11, we shall show that for the diagonal pivoting method
0 < a <1, we have:
with
V<n f(n) P00 c(a) h(n,u)
(
max
k
c(O
i
In Chapter 12, we shall show that for
0
n
>
2
.446
for
)
<
3.07
(n-1)
)
h(n,a
O
Then from (10.11.5) we have for
IFIa
F <
-t
rn f(n) P
a
a0
-
a0
:
[5. 7 1p + 31.6(n-p)/2]
(3.07)n0.446
i,j
where
max
i,j
p
IF I
is the number of
< 2
((n) t1
< 15.8n 2
-
pivots used. So
0 446
10.0 9 p)
[15.8n
(3.07)n
II
f(n) ii0
44 6
(3.07)n 0
which is within a factor 7.9(3.07)n 0
446
of the bound for
max iJ
IFi
I
for Gaussian elimination with complete pivoting.
10.13
Floating Point Analysis for
The blocks of
(a) Surpose
We want to solve
D
a D
are of order 1 or 2
[Dii]
D i Yi
is a block of order 1.
.ci
. but we obtain
due to the finite precision; pivotin-i+lJ
-
1
Then so is
[(6 D)il]
(Dii + (6 D)ii)y
and
IDiiI=
i
(n-i+l)
67
(6 D)iil
so
2-
(
D)1( I < 2
Thus
t
Then ptvot(n-1+l]
2 t
ni-
where
n -k l )
(nk+)
~
lu~
Now
and
=
0(2-t)
.
Since
1*
p(n-i+l) <
and
2
(n-+l)
~+ (+
(I +I n)
le
Since
2- t
(n-i+l)
- de
det r 2
[1 +
ci D +li+l (1+n)
-t
,In
Icl < 2
n) -
(I +
-
-t
0
, we have
0
(2 -t)]
cI Di+lI
Ci+l D i+
< 2
+)-V~,
(1+ri)] (1++
2- t
I<
-
yi+I . [Ci+l D1 1
yi=
is a block of order
- (n-+l)
2
D.+i+
ID ii Di
V
(n-k+l)
(n-i+l)
P(n-i+I)
j
DI+
|Di~
2.
C
.- i+l)
p0
Suppose
(b)
IDij-
. But
Ioii
-D
(6 D)
and
(ci/Di)(1 + c)
yi
Thus
I
+ a
(1
(1 +
and
) pjn-i+l)
I)J(l + 6)/f
+) ] (I
(2 -t)
2
(
D)ii+
(n-i+l) < 0
J
I
<21(6 D)
10.14
i
1(6 D)
Floating Point for
From §10.2, (M + M 1 )c
M1
b
and
and
1 + 0(2-t)]
(n-i+i)
0
(M + M 2 )
(n-i+1)
[+il
H2
x - y
( -+
)/(±v (nk+1))
after much manipulation, we have:
1(S )j
2
:)
- -- -_
--
--
--
--
-
--
--
--
-
68
From Wilkinson (1965, pp. 247-249), we have:
I(M ),,
for
i
< 2-
, it,.,)Il
.<q-2
t
11 + 0(2"t)]
I
[I + 0(2 -t)]
,
(Ml)l
= 0 = (Mt)
1
j
IMjl
(n
-
2 + I - J)
IM I
j
From §10.10, M li ' I
max (1/a,
i
So for
> j
l/(I--))
and
for
(1 +
0 ( 2 -t)]
(M)i1,
I(M1 )1
10.15
i > j
.
t,
(10.14.1)
and
IMii
< J,
(M
for
l(12)il
I
2-t
M2ijI <
max{lla,l/(l-a)}[l+O(2-t)13(n-2+i-j)/2
'_t [I +
2
, I(M2 )I
Floating Point Bound for
0
(2 -t)]
E
From §§10.3, 10.11, 10.13. and 10.14, we could express
terms of
Fij
and the elements of
M, D, M I, M 2, ard
6 D
Eij
.
in
But this
expression is very complicated.
n
Let us define
IAI-
max I
j i=&
usually called the one-norm.)
JAij
I
Clearly,
where
A
is
nxn.
IAijI < IAI
From §10.3, we now have
(10.15.1)
lE jI L IFijI +
c - UM I ID + 6DI I(M Mt)I+
12
E
,
where
NMI ,,6DI
,(+Mt)l+
2
ID
W1,I
2
(This is
69
3
IDI
. (1 + a) max
From §10.13,
(k)
k
16DI
2- t
i
(I + ct) max p (k) 11 + 0(2"t)]
0
k
IMI,
FLum 910.10,
0(2 - t ) (1 + max {1/a. 1(1-ci)} 3(n + 2)) - 0(2 ')
Let us assume
i.e.
n 2 2- t << 1
let us assume
IM 1, IMtl < 2-
2-
1'
c .
Let
c< 2- t (I + a) max
(10.15.2)
k
(10.15.3)
c < 2
11 +
0
( 2 -t)]
(10.15.1), and (10.15.2) we conclude:
,(k)
t
IJEI 2 < [1 + 3.0/a] Yj
+ (!+ a) max p
k
From (IG.II.5.
a
1 + (n-1) max {1/a, 1/(-a))
0(k) B(2 + a) [1 + 0(2-t)]
maxaxk1 (k)) (7.734)n2
k
From (10.11.4),
(10.)5.4)
8
8 < 2.781n , so
,Owe have
a -
Then
1t- + 0(2-')].
Now we can bound
For
IMtI < 1 + (n-1) max {I/a, 1/(1.-a)} [1 + 0(2-)]
+ 13 + 11.02/(1-a)]
PO
(k)
t o(
B(2 + 1)[1 + 0(2
(10.15,1), and (10.15.3)
we conclude for
0
(10.15.5)
Since
(10.15.6)
1E1
I
< 2
max p
k
.E| ,i Irl + c
1E1 < (23.54)n2
,
15.80n + 7.74 nrt
from (10.11.5) and (0.15.3):
ar
k
(k)
ICi
I
|
It
70
Comments on the Bound for
10.16
E
E
have a good bound or
max p(k) < r
0
k
In Chapter 11 we shall show that
for the diagonal pivoting method for
I
, (23.54)n
<i:
0 < a < 1
c(gO ) h(n'a O ) < 3.07(n-1) 0 .446
Shall silow that
-t
2
2
in order to
P(k)
From §10.12 and §10.15, we need to bound tile
n
,
f(n)
0
0
h(nCL)
c
C(a)
In Chapter 12 we
Then we would have
0 (3.C7)(n-1) 0.446
For Gaussi3n elimination, in order to solve
A x
-
b
,
in finite
precision we actually perform:
(1)
LU - A + F
(2) c-
(L + SL) - 1 b
(3) x= (U+6u)
c.
From Wilkinson (1965, p. 215, pp. 248-252), we have (cf. §2.5):
max
1E1
< 2.01n
2
and
max I1
2On
FI 2.01n 2
-t
max
-
2 t max
k
k)
P(k) [1 + 0(2"t)1
0
For complete pivoting, max
k
Thus our bound on
36(n-i) 0 .446
1El
n f(n) p
0
0
(Wilkinson, 1961).
for diagonal pivoting differs by a factor
from the bound on
complete pivoting.
<
O
1 (k)
IE1
for Gaussian elimination with
I
71
Chapter U1
An A Posteriori Bound on Element
0 < a
Growth fuo
11.1
1
Introduct ion
We saw in Chapter 10 that the bouna on the elements of the error
matrix depends on the maximum of the elementb of the reduced matvices.
Let
det A
A(n )
A
# 0 .
Let
We saw in
be the original symnetric matrix of order
(k)
A'
be the reduced matrix of order
§8.4 that
max
A( k
<(
2 5 7 )n-k
max
n
wi1h
k
Aij
But we
ije
: ,
"i,)
shall show in §§11.2-11.7 that we cau get a much better bound by the
use of Wilkinson's techniques for Gaussian elimination with complete
pivoting (Wilkinson, 1961; pp. 281-285).
His proof depended or, the fat that the pivots in Gaussian elimination with complete pivoting were maximal elements in the reduced
Our pivoLs are rot necessarily maximal element;,
matrices.
but they
are closely related to the maximal elements in the reduced matrices.
We shall assume that we use strategy
v (k)
0<cat-,andthat
Lemma 4, §5.5).
Sies in §6.1,
11.2
(k)
-
(A)
I
Sa
-
for all
A
cc
(see
In particular, this lower hound holds for the strate-
§6.4,
A( k )
-
for any
(§5,7)
.. d §8.1.
The Pivots
Let
-
>
2
be the reduced matrix of order
-
-
k
72
Let
pk)
mx
(A
i,j
(k)
and
ij
)A
ai i
•
i
Let us assume that, whenever we shall use a 2x2, interchanges have
v(k) .A(k) A(k) - A(k) 2 1 >(k)2
(k) 2
.nsr11
22
1
21
ensured that
(In parcicular, thiF assumption holds for the strategies in
§6.1-6.2,
and §§8.1-.8.2.)
§§6.4-6.5,
From strategy
I
pivot(]
S.
(§5.7) and §9.1,
Wk
if
i.e.
(-K
<if
)
W
2O
we recall:
,
1 (k+l)
(-0
if
Vj l
,
(k)
i.e.
A (k)
A
i
if
A
uses a lXl pivot
k)
uses a 2x2 pivot
+l )
ie.
if
A(k
i.e.
if
A
uses a 2x2 pivot.
Let us now define
p.
"
Pi
"
The
pk
(0c)
if
pivotik]
/V( k )
if
pivot~k] - 2
i
if
pivottkl-
-1
-+7 if pivotLk] - 0
will be called pivots.
From the decomposition
A - H D Mt
Idet A(k) I -p.. .pN ,
ince
and
2
we see that
Idet Al - p1 .. p
det M - 1
hadamard's Inegality
11.3
By idamard's Inequality (Gantmachcr, pp. 253-254),
(11.3.1) Idet-Al!(
< { 11
nI
n
~
2
Ai~
1/2
iml j-I
since
p
a
(11.3.2) ldet Akk)I
Also
x 'A€
140
~
(k))k
nZn/2
<(nv
(n)2
(n).n
- (n jj 0
o
I
73
det A(k)
Bounding
11.4
(11.4.1)
p1
"
(2) If pivotik] - 2
2
, (k) > P(k)
Thus
- 1i(k)
v:(k)
p (k )2 < V(k)(l
then
c 2)
-
1(
p0
(Pk iv/a)k
If we assume
(k)
A (k)
for all
oW
So
< (/P(k)k
det A(k),
k
(k)
p (k) >
(1) If pivot[k] = 1
(k)
(k)
_2
2
a
c)
k*V
Idet A(k)
(11.4.2) p1 .' Pk
.
(k
.kI(1
)k
-c)
Fundamental Inelit
11.5
From §11.4 (and later in §11.5), we are led to the following;
Definition:
(11.5.1)
k
SI/C, 2
if pivot[k] - 1
1/(l-ca 2 )
if pivot[k] - 2
I
k+l
1+1/k
if pivot[k] - 0
I
From (11.4.1) and (11.4.2) we have:
(11.5.2)
Pl "'
Pk )
(
Pk
k
if
pivot[k]
# 0, 1 < k < n
We would like to have a similar equation for
pivot[k) = 0
for
ouv"analysis in §11.6.
If
P1
P
pivot[k] - 0 , then
Pk Pk+1
P
Since
+l
pivot[k+l] ,k+1.
6k+1
k
1 k+l
l-
k+1
1+1/k
2
.
, where
By
(11.5.2),
8k+l 1
(I
-
ci
and
by (11.5.1),
I
74
8
P1 "'" Pk < [(k+l)
]
k+l
(k+l)/2
)k
(11.5.3)
P
11.6
Bounding Pivot Growth
Define
Pkk
Idet A (n)
Since
) k
P
for all k
,
"
Thus
1< k < n
qk = log Pk . From (11.5.3),
k-- 1
1 qi < (k-)
Ji
(11.6.1)
( -
k
= p1
qk + 2k log (k
pn
...
,
k)
we have:
(n , n
(11.6.2)
log Idet A
-
Dividing (1l.b.l" by
I q
i-i
k(k-1)
for
2 < k < n-i
and (11.6.2) by
n-i
and adding we have:
q
+ q 2 /2 + q 3 /3 +
nI log
qn- /(n-2)
2
...
)/(k-)
k=2
+ q 2/2 + q 3/3 +
q
+ q n 1i(n
...
n1
- r(r-l)
--I
rink
after observing that
(11.6.3)
log (k
+
Idet
+ qn-1/(n-1) + qn/(n-1)
+ qn/(n-l)
<
lo1
-
i)
- =k-i
n-l
n i (k
n-i
1k~/(k-1)
+- I
k=2
From (11.5.3),
Idet A(n)I < (Y/r-T- P n
r
(11.6.4)
f(r)
(11.6.5)
h(r,a){
R(
I ki/(k- )l 1/2
k-2
r
1/ 2
11 ak 1/(k-1)}
k-2
We define
log Idet A (0)
75
From (11.6.3) we now have
q, + qn/(n-l) < log f(n-1) + log h(n-l,a) +
n2(n-1)
log (n n) + n q /(n-1)
n
n
n
With simple manipulation we have
-
q
2n-I
< log f(n-l) +
log n + log h(n-l,a)
1
+ 2(n--l)
log 8 n + 1j log (n
8)
1
log f(n) + log h(n,cL) +1
log (n
n)
n
Thus we conclude
(11.6.6)
p/P
<n
An
f(n) IX
Similarly, we have for
h(n,t)
1 < k < n
< AkPn
-k + 1 f(n - k + 1) *'Pk/pnn-k+l
(11.6.7)
(
<_
n-k+l
(11.6.6) and (11.6.7) hold for all
(§5.7) provided that v ( k) > P(k) 2
a
_
-
OL , 0 < a < 1
k) 2
S
)
We now have a bound on pivot growth.
bounding element growth (§11.1).
are interested in bounding
(1)
A
under strategy
A( k )
for all
(§55)
But we are interested in
finishes with a 2x2 pivot we
while if A finishes with a lxl pivot,
1
Boui.dxn- 7lement Growth
We shall now express
. ..
p2)
0(2)
If
,
( )
we must bound
11.7
k + l,a)
l h(n-
(n )
(2)
P0
and
p
..
P'
VPI'Pn'
"n
and
a
Pn
in terms of
76
Let
(n)
,U
A-A
U
0
0
n
III
1
But
cL2 )
-
<
1
if
If pivot[n]
2 p0"
< /I -t
+
0_5.5)
F0
Pn
if pivot[n]
if pivot[n]
-
and
2
-
2, then
-
V1
<
L0
,
and
Thus
I
0 if pivot[n] - 2
A'
1/a 2
NOw
,
pivot[n] - 2
always.
0
-< _
(by
I
if pivot[n] - 1
i/a2
L1/
(n)
pivotin] -
if
r
8n
V
i
pivot(n] , 0 , pn
Since
u(n) V
,W
U
(1
Bn-) -
if pivot[n-i] - 1
if
-
n--(n On)1/(-
pivot[n-i] - 2
if pivotin-I]
-
0
Let us define:
(11.7.1)
rea
1__1x
-
}
_ (_a
V(1I__)I
Thus we have:
(11.7.2)
<m (a) V0
n
,
and
if pivot[n] a 1
M(C) 1o
(11.7.3)
VW9T
Pn
<
n-_n,11 (n
From (11.6.4) and (11.6.5),
(11.7.4)
(/;-)i/(nl)
n
-i)r7-")
,+c(I-at
we have
f(n-1) h(n-l,t) - f(n) h(n-l,x)
if pivot n]
-
2
77
We have two cases (§11.6) as to whether:
(a) the last pivot is lxi,
i.e. pivot[l] - 1
,
(b) the last pivot is 2x2, i.e. pivot[2] - 2
"
p
pivot[i] - 1 . Then
Case (a): Let
() . (1)
From (11.6.6) and (11.7.2) we have
(11.7.5)
<
1.1(1) "
V (2 )
>
(1
(2)
2
Cta
)
f(n) p
P(2)'2
0( 2 )
From (11.6.7) with
k
P(2) <
and
(2)
(2)'
>
v
Hence
I
(11.7.7)
P2 "
(2)
Since we have assued
P21V-r_
(11.7.6)
m(a) h(n,a)
pivot[2] a 2 . Then
Case (b) : Let:
(2)
41
e 1
r
-
2
and from (11.7.6) we have
n- f(n-1) h(n-l,ct) Y
p
Now we have two cases as to whether the first pivot is (i) lxl
or (ii) 2x2.
(i) Let pivot[n] - 1 . From (11.7.3) and (11.7.),
<_ rnl
But
f(n-1) < f(n)
(11.7.8)
and
h(n-l,a) < h(n,a)
P(2)
0
< vn f(n)
(ii) Let pivot[n]
0 (2) <
UO m(a) h(n,t)/
-
_
'-a
a
2)
0
0I
__
Thus
.
2 . From (11.7.3), (11.7.4), and (11.7.7),
f (n) P h (n, a) A772(
~n
F
2
a"
h(n-l,ot) m(at) p0/
f (n-1)
Pa(2
__
I
78
But
2+'/(l -a
(11.7.9)
P(2) <
2
) < m(a)/T-_
from (11.7,1).
Hence
/rn f(n) pO me(a) h(n,a)/,r-
Let us now define:
(11.7.10)
c(a)
-
i
m(a)
if
where
m(a)
- max il/a,
o1
pivot[2] - 2
(la)/(1-a') }
Then we conclude for
f pivot[2]
0 < a < 1
2 : u
2~
0
)
Similarly, we have for each reduced matrix
(11.7.12)
w (k) < VW00
k+j f(n - k+j) )0
A(k)
c(k,a) h(n - k+j, a)
j
1if
where
j - pivot[k] ,
c(k,a) -
(a)
x
i/I
and
2
if
j
=2
0 < a < I
Hence, for all
(11.7.13)
11.8
I
A
,
(k) < vr f(n)
we have for
0 < a < 1
c(a) h(n,a)
Comments on the Bound
The bound in (11.7.13) holds under strategy
under Lite assumption
V
(k)
-
S
for all
, 0 < a < 1
A
,
and
In parti-
cular, (11.7.13) holds for the unequilibrated diagonal pivoting strategy (9§8.1-8.2), the complete strategy (9§6.1-6.3), and the partial
I
79
equilibrated strategy (9§6.4-6.5).
(For the partial equilibrated stra-
tegy, we assume that, given a reduced matrix
we equilibrated
eacit row is
A(k)
Vk) - max JA(k)a
I
'Jj
A(k) so that the maximal element in absolute value in
pok)
)
/n t(n) vO
Wilkinson (1961) obtains
In the reduced matrices for solving
as the bound on the elements
A x - b , J0 . max
i,j
Gaussian elimination with couplete pivoting.
c(a) h(n,)
with
since our
IAij I , by
We here the extra factor
pivots are not necessarily maximal elements of
the reduced matrices.
We call the bounds in (11.7.11) - (11.7.13) a posteriori, since
8
we cannot calculate the
k
terms of
tion of the blocks of order 1 and 2 in
A - M D Mt
D
until we know the pnsi-
for the decomposition
In Chapter 12, we shall give a bound on
independent of the structure of
(1 + Y17)/8
11.9
h(n,a)
D , for the value
c(a) h(n,a)
a -
0 =
(§5.7).
Smaller Bound on Pivot Growth
If
H
is an
is the minimum
nxn
positive-definite Hermitian matrix and
eigenvalue of
n
detR<-
(11.8.1)
i- 1HHii
i-1
If
A
is an
nxn
-
A
,
then (Shisha, p. 173):
n-2
mai (nX
n
min
i2)
11
•
n~j>>I> 1
non-singular real symmetric matrix and
the minimum of the absolute values of the eigenvalues of
setting
H - At A
Xmi n
in the above, we have
A
,
X
then,
is
80
1 Ai)
( nk
2.
H
n
Idet Al'
(11.8.2)
-~
1=1
j
is the
A
where
th
n j>~l1
j
A
column of
)
Idet Al2 < (n tj2
Then
A t AI 2
X2 0-2)
-
C(A)
,
where
7 /(n P
lAt A I
2(n-2)
(11.8.3) c(A) - 1 - X
2
2n
Using an analysis similar to that in §§11.2-11.7, we obtain:
1
if pivotil] - I
)
(11.8.4)
/n
<
{If pivot[Zi
-
2 : P(2)
1
1
where
2
(r))r(r£(AI
n
(A)n -
-
T(A)
f(n) p 0 c(a) h(n,a) T(A)
and--
r-2
C(A(r) )
-1 r
2(r-2)
-
L
ECA
r t j
'i
A r)t A(r)12J
2
is the reduced matrix of order
r
(A
=
A (n ) ) ,
the absolute values of the eigenvalues of
th
j
column of
If
A
is the minimu- of
X
(r)
where
, and
A
(r)
is the
A
det A # 0,
A(r)
A
,
A(r)
then
IrI > 0
not Ladamard (see §12.6), then
then
A(r)
"(r )2 r
/ [r V
for each
c(A(r))
will use a 1×i pivot and
<
1
A (r - 1 )
.
A(r)
If
If
A(r)
A(r)
is Hadamard,
will not be Hadamard
(see Appendix A).
Thus
'(A) < 1
if det A # 0 . Hence (11.8.4) gives a lower bound
than does (11.7.11), but (11.8.4) is so complicated we are unable to use
it to advantage, and we present it merely for its academic interest.
81
Similarly, for Gaussian elimination with complete pivoting, we
can obtain r- f (n) .0 T(A)
as the bound on the elements in all the
A x - b
reduced matrices when solving
If we replace
of
A r)o
te
Xr
in
T(A)
minimum modulus.
of_
by
,
l 0 . max
ii
10 r I
lAijI
, det A 0 0
the singular value
82
Chapter 12
An A Priori Round on Element Growth
for
12.1
a - a 0 = (1 + Vi7)8
Introduction
In (11.7.13) we obtained
rn f(n) PO c(a) h(n,a)
the element growth in the reduced matrices, for any
as a bound on
a
with
0 < a < 1
From (11.5.1), (11.6.4), (11.6.5), (11.7.1), and (11.7.10)
we
recall;
if pivot!k] - 1
ak
1/(
=
f
-
E
( k
2
2
)
if pivot~k] - 2
/
if pivot(kJ - 0
i H~nkl/(k-i)
k
h (n, )2
H11
n
k-2
k-2
1
Me(a)
= max
,
I
if pivot[l] = 1
i1
if
1
---
p
1/(k-i)
k
1-ap
}
,
The term c(a) h(n,a)
and
c(a)
-
mn(a)
X
2
pivot 12)
arose from the fact that our pivots are not
necessarily the maximal elements of the reduced matrices, but can be
expressed as multiples (involving
a) of such maximal elements.
The
bound in (11.7.13) is a posteriori since we cannot calculate
c(a) h(n,a)
until we know the pivot selection, i.e. until we know the
position of the blocks of order I and 2 '1 the block diagonal matrix
for our decomposition
0a
< I.
A "M
D Mt
for a given value of
a
D
83
We would like an a priori bound on the element growth, i.e. a
bound independent of the selection of a lxl or a 2K2 pivot at each
stage.
We would also like the a priori bound to hold for all
0 < a < 1.
Let
a
But no such bound exists, as we shall now show.
p be the number of lxl pivotu for the deccmpc'sition (§9.1),
i.e. there are
blocks of order I in
p
D
above.
If
p - n
(e.g.
for a positive definite matrix, see Apptndlx A), then
h~n)
h(n,T)
l/ (k-1)
-n
=
(1/a)
I
w
-
a-1 0 .
as
fp
-
0 (see §6.3), then
k-2
h(n,a) >
n
H (i/ilk-2
2)I(kl)
w
as
a - I
Thus we shall give art a priori bound for element growth only for
the value
c
-
a0
" (1 + vr17")/
Lower Bound on
12.2
(§5.7).
c(a) h(na)
We shall now find a lower bound on
for
0 < at < 1
c(a) h(n,)
0 < a < 1 , in order to show that the upper bound for
for all
,
a - a0
that
we obtain in §12.4 is a reasonable bound, i.e. that some other choice
of
a
would not provide us with a much better upper bound on
c(a) h(n,a)
Since
min c(a) h(n,a) > min
O<CL<1
0<,<l
(a) x
sin
O<a< 1
only find lower bounds for the latter two minima.
Lemma 1:
II
)4
min
O<OL<l1
c(a) > 2.029
.
h(n,a)
,
we need
84
Proof:
c(1r
c(a)
m n
l)
- /2 + 3/17> 2.029 .
q.e.d.
We shall later need the following:
n+l
Lemma 2:
< k n
1 ( --) n*1
L
(r=
k-m k
l~
<
k-m+l(
1
1 <k+l
-<dx< .
k+- < k
xn~
kx,
1~
for n,m> 1.
By elementary calculus,
<~~~~~
o
u-L
<f~ +1-1
<x
<
- lo
Proof:
I
min
Lemma 3:
h(n,C)
Thus
q.e.d.
h(n,a).
min
. lower bound for
Pow we cau find
n > m >1.
for
> n0.3465
> nlog
n
Proof-
Let
Y'1/(k-1)
o(a) -
.
If
then
n ,
p
k-2
h(n,)
-
(W/a)w( n)
where~~~ 2
where
L-N)
11
Thus
h(nCA)
If
0
p
h(n,a)
then
-2)
(.//
-
w (
n)
t(n)
~ 2?(J_ ~ ) 1,1 (2j1]l(j2
2j-1
> b(,)w(n)
, where
min b(a) - V2 , and i
b '%) - max fl/a, 11lyi-}
attained by
a - liV2 .
BuL
.
From Lemma 2,
O<wa 3.
w(n) > log(n)
.
hence
min
h(n.cz) >
(,)log n
n
n
q.e.d.
r01< <
From Lemnas 1 and 3 we obtain a lower bound for
for
0 < a < 1
Theoram 1:
ain
c(a) h(n,a) > (2.029)n
0<a<1L
G.3465
log b,1
> n
0
.
346
c(a) h(n,a)
85
c(1 0 ) h(n,a 0) < 3.07(n-1)0.446
In §12.4 we chall show that
on
which does not differ much from the minimum of
h(n,co)
C()
0 < a < 1
for
c(,t) h(n,a)
(95.7) will provide us with an upper bound
Ct O0
Thus our choice of
h(na)
Remarks on sn Upper Bound for
12.3
We cannot evaluate the
0k
until we knaw the pivotal selection,
independent of the pivotcl selection,
Bk
but if ue could bound all the
then we can obtain an upper bound.
is independent of
h(n,
8
If
Theorem 2:
n, then for
)ih(r--1,. ) < ¥(n-].
g
y{-(-2
where
~
lg
n-I
1/(k-l) < log ( -
1
From Lema 2 of §12.2,
k-r>3
But
{ i c1 / ( C -I )
h(n,c)/h(r-l,Ci) <
xlogy
k
>_ '
2 <~k
<
r-l
,
8
k
independent of the pivotal
atud if we can consider the worst possible cases
k -
of pivot selection for
have a bound for
q.e.d.
.
we can bound
This shows that if
strategy for
x,y > 0
n-1
/2 < c
k-r
ior
y.og x
=
r > 3
"
1
Thus
and
0 < a < I
n
Proof:
c > 0
k > r ,where
for all
k<c
h(n,ci)
.
we must have
2,...,r-1
and bound
h(r-l,a) , then we
(In order to consider the cases for
r
reasonably small.)
In §12.4 we shall do this for
a - a0
,
and we shall have
r - 5
I
.
--
-. ----------
-
--
~-
----.-------..--
86
For
a
aO
-
a
a
An A Priori Bound for
12.4
= (
(1 + Y7)/8
ras,
+ a;) 7(1
* max (
!/a
ri
(12.2.1)
d(a)/a 0
c(a)
where
if
pivotil) - 1
d(a)
LI I/
Now we shall show that for
lxI pivots for
k > 5
.
Thus, for
a
a0
a2
k
Recall §9.1 or §11.2
if plvoti2]
-
2
is maximal for
for the definition of
pivotik]
Lemma 4:
B1/(k-l)
k
For
l/k
<
6k+1
-
Proof:
a0
and
)/(k-1)
k > 5
+ 1/k
1/k
Ikk+1
and
- (/a2)i/(k-l)
if pivot[k] - 0 , then
,
From (11.5.1), if pivot[k]
/k+l
1/ - a 2 )
8 1/(k-l)
a -
-
0
,
1+1/k
+ I/k
f
then for
I
Hence
(k+l)k+l/kk < a 2 (i/a2
iff
(k+l) log (k+l) - k log k < log a2 + 2 k log (1/a
iff
g(k,a) > 0
for
k > k0
for some integer
g(x,a) - 2 x log (1/C& 2 -1) + log
where
g(x,a) - 2 log (1/a2 -1)
ax
-
1 )2k
x > l/[(1/C
2
-
=
g(x, C)
0_
-
1)
k0 > 1
a 2 _ (x+l) log (x+l) + x log x
log (I +
/x) > 0
- 1)2 - i]
In order to evaluate this inequality, we now fix
G(x)
at<
k+1
kc
iff
0 <
at
a0 .
Let
r
87
g(x'0
G'(x) -x
Thus
g(k, 0 ) > 0
Thus
x > 0.94
x > 0.94.
monotone increasing function for
G(5) - 3.698.
iff
> 0
Now
So
.
G(x)
is a
q.e.d.
k > 5
for
and
G(4) - -0.476
Now we can apply Theorem 2 of §12.3 and Lemma 4 to obtain the
[
following two necessary lemas.
-log a
log a
If pivot[5] 0 2
Lemma 5:
<
Proof:
log
L
<
a.
0
3
.
0k
By Theorem 2 of §12.2 and Lemma 4,
0.446
-log a 0
(n-i)
< 0.613 (n-)
.
If pivoti5l
Lemma 6:
-
h(n,a0 )/h(4,±)
q.e.d.
h(n,ao)/h(3,
2 , then
O)
0 446
< 0.717 (n-i) .
If pivot(5] = 2 , then pivot[6] 0 2 , so by Theorem 2 of
-log a 0
log (0
Proof:
§12.2 and Lemma 4,
pivot[5] -
0
0 4 46
0.613 (n-i)
(n-1)
h(n,ao)/h(4,a O ) < 3
then
,
2,
/
1/3
1
/4 1/2
00
(4 85
1
oga
3
4g
0 ( 4
Since
(n-l)
h(n,ci)/h(5,a) < 4
1
q.e.d.
1/2 < 0.717
a4
1
(2
Now we need only to bound
1
2
{82 83
1/2
for pivot[51,
2
and
1
3
84 }
1/2
for pivot[5] # 2
(iL0)3/2 if pivotf2=I
Lemma 7:
If pivot[3] -
1
,
then
h(3,a O ) -
L
0(i.c)
< 1.96.
kI
[
if pivot[2l
2
88
Since pivot[3]
Proof:
1
-
and pivot 12] 0 0
0 Tr
,
0
if pivot[2] -
-
So
1
a2
,
if
02
1-'
Lemma 8:
h(3,a O) < 1.96.
and
pivot[2] -2q.e.d.
2 , then
If pivo13]
h(3,a)
< 2.74.
3/2
Proof:
Since pivot[3] - 2
,
S3
and
-
2
2*
i
q.e.d.
Lemma 9:
h(4,aO
t0
If pivot[3]
1/3
x/ r
-
0 , then
if pivot[2] = 1
0
if pivot[2] - 2
1
4/3
Proof:
Since pivot[3]
0,
a4 '
1
and pivot[2] 0 0 .
So
'14
9
a
=
)
0
0
B2
.
1
if pivot[2]
= 2
Now we put Lemmas 5-9 together to get a bound on
which is independent of the pivotal selection.
q.e.d.
c(a0 ) h(n,c
0)
89
c(
From (12.2.1),
Proof:
1
d(
0
<
h(na O)
c(ca)
Theorem 3:
3.07 (n-1)
4
)- d(%o)/
if pivo:[il
-
0.446
for
3 rn
'
n > 2
0
1
)"
l//'
;-
if pivoci2] 2
Now we must consider various situations.
0
pivot[51 - 2 : Then pivot[3]
Case I:
(n-i)
h(noa
0)/h(3,aO) < 0.717
(a) If pivot[3]
By Lemma 6,
.
0.446
1:
-
h(3,
Then, by Lemma 7,
O)
< 1.96
Since
d(
O) <
A-0
,
(n-1)0.446
c(Q0 ) h(n,co) < 2.39
(b) if pivot[3]
Then pivot[l] f1
So
c(O0 ) h(n.o 0 )
Case II:
<
.
2:
So
3.07 (n-i)
d(c 0 ) - 1
.
By Lemma 8,
h(3,.)
0.446
pivot[5] 0 2 : Then pivot[4]
# 0
.
By Lemma 5,
h(n,CO)/h(4,ot 0) < 0.613 (n-1)
(a) If pivot[ 4 )
By Lemma 9,
-
2;
d(a0 ) h(4,ct 0 )
2.58
2.48 (n-1)0.446
(b)
Then
4
If pivot[ ] - 1
4-
1/a'
<
and pivot[3]
0
.
Thus
c(a O) h(n,c 0 ) <
2.74
-
-1
90
(i)
If pivoti3J = 2
Then pivotil'
I
.
c(%o)h(n,a)
Thus
h(3,caO ) < 2.74.
(1/(X0 )(2.74)
d(aO ) - 1
so
,
By Lemma 8,
<
0446
0446
16
< 3.04 (n-1)
/4 (0.613)(n-1)
(ii) If pivot[3]
1
(1/Co) 3 /2
I
if pivot[2]
By Lemma 7, h(3,c O ) -
01
1
if pivot[2] - 2
We must use this form of Lemma 7 in order to prove the theorem.
Once again we have two cases.
(A) If pivot[2]
-
1
Then pivotl] - 1 , and
c(c
O
h(n,a
2.17 (n-1)
(B)
Then
(1/0o1
)
< (1/a017/6
So
(0.613)(n-1)0.446
0.446
If pivot[2] - 2:
d(a 0 )
1//
00
l/ [I/(
11/6
-
-
.
So
c(
O
c0%
h(n,C O
hnc
<
0
(n-1) O ' 4
- a')] (0 .613) (n-1)0"4
0.446 <
< 2.36
.6(-)0.446
Tius, in all cases, we have
for
1 .
d(a)
0
c(aO ) h(n,a 0 ) < 3.07 (n-1) 0.446
n > 2
Now
3.07 n0 .446 < 3Vn/
for
n > 2
q.e.d.
A
-1
91
Bound on Element Growth
12.5
From §11.7 end §12.5, we see that the elements in all the reduced
*
matrices are bounded by
-max
lA Ii
v'i-f(n) p10 (3.07)(n-)-)0.446,
, under strategy
S
with
a -ao
where
. (1 + r/17)18
the pivotal strategies described in §6.1, §6.4, and particularly
for
M8..
For Gaussian elimination with complete pivoting (Wilkinson, 1961;
rn f(n) p.1
pp. 281-285). the bound on element growth is
a - a 0 I our bound for diagonal pivoting is within a
Thus, for
0.446
of Wilkinson's bound for Gaussian elimination
factor 3.07(n-1)
with complete pivoting.
Conjecture for Gaussian Elimination
12.6
It is conjectured that the best possible bound ;.s
n
for real
matrices under Gaussian elimination with complete pivoting (fryer, p.
343).
The conjecture is false for complex matrices (Tornheicr, 1965).
For real matrices, the best possible bound is
is
2-1/4 for
A matrix
H
n
for
n =1,2,4
and
n-3.
R
are orthogonal.
(Davis, p. 327).
is a Hadamard matrix if
Then the order of
H
IJ
Thus, a Hada-Ard matrix of order
at least
n
- 1
and tha rows of
is 2 or is divisible by 4
Under Gaussian elimination,
p, 343).
I
n
111(1)1
> n
(Cryer,
has element growth of
r
--
92
L 1J(Davis,
Fl
is generated by tensor products of
If H
and if the order of
n -
is
H
2k
hn
,
p. 326),
1111-n
-
2k
Conlecture for.Diagonal- Pivoin
12.7
If
Ai Is as in the previous paragraph,
then
It
symmetric and
is
the diagonal pivoting method uses a lxi pivot at each step under stra-
0 < a <1
SOLfor
tegy
1H11l
and
,
-mn=m2
We conjecture that the optimal bound for diagonal pivotin~g is of
the form
F
iI
has
/
n -2
for
>.
1
+ l/aL)/2
-(a
,q(cL)
,
and
_
L1
since
0 < a <
as its bound on element growth, where
a + 1/a
Thus,
We need a function q((%)
n q(00) .
<
1.10
q~CL0
Or, we could conjecture a best possible bound of the form
n q(n,aL)
The optimal choice of
mizes
q(aL)
such that
q(n,a)
in
aL
n
for
§12.7 for all
minimizes
q(2,a0 )
1.10
for
0 < a < 1
n
Or, we could seek a sequence of
q(n,a)
.
in §12.7.
is
the value which mini-
But we do not know
a
q(aL)
n >2
We could choose
Ai
a
and
a
The Optimal Choice of
12.8
or
+ 1/a)/2
q(2,ux) -(a
.Then
f(n) Pc(at) h(n,ax)
a
to minimize
c(a) h(n,a)
in (11.7.11).
But
is merely a bound on the element growth and is,
by no means, the best possible bound.
93
Since
min
) h(n O
c( 0
c(a) h(n,)
)
< 3.07 (n-1)0.446
0.3465
0;
> (2.029) n
(§12.4), while
0
(§12.2), our choice of
a
-
0
gives us a bound (ind -pendent of the pivotal selcctioa) for
c(ctO ) h(n,(%O )
which does not differ much from a lower bound for
min c(a) h(n,a)
0<< 1
than
min
q(cx)
We assert, further, that
q(ao)
is not much greater
.
O<OL<I
We note that
-
1
,
inf q(2,)
-
1 - q(2,1)
But this implies that
so we would use a lXl iff one of the diagonal elements were
maximal, i.e. if
WI
<
0
we must use a 2x2 and thus for
n - 2
no
decomposition would be performed, which is to be expected since the
minimal element growth occurs when we do no decomposition at all.
inf
q(2,ci) - 1
=
q(2,1)
does not provide us with any information for
th
the general n
Thus
order case if we require
an
to minimize
q(n,)
94
Chapter 13
13.1
Iterative Improvement
The Approximate Solution
Let us assume now that we have obtained an approximate solution
z
to the system
Ax-b
diagonal pivoting.
A
,
,
det A
The approximate solution
0
z
,
by the method of
to
A x
-
b
can be
considered the exact solution to the system (A + E) z - b
In exact aritunetic, the method of diagonal pivoting would perform the following steps (see §10.2):
(1) A =M D Mt
(2) c= M
1
b
(3) y f
1
c
(4) x =M
-
t y
However, in finite precision we have error at each step.
error matrices
F , M1
,
D
,
and
M2
,
For some
we actually perform (see §10.2):
(1) A + F - MD Mt
(2) c-
(M + M1) -
(3) y-
(D + 6 D) -I c
(4)
(M + M2 ) - t y
x
'
I
b
From §10.3, we see that
z
is the exact solution to
(A + E)z
where
t
+ M J6 D (M + M2)t + D M ]
(D + 6 D)(M + M
E - F + M
2 )
1
From (10.15.6),
we have for
=
0 0
(0
is the one-norm):
-
b
a
95
(k)
0
2-t
iEi < (23.54) n
k
In Chapters 11 and 12 we have shown for
max p(k) <
00
k
0
(3.07)0(n-1) .46
n f(n) o
110 . max IA jI . Hence, for
ij
< (72.3) n 2.9446 f(n)
-t
where
|E
13.2
a "
0
The Iteration
Let
x
a = a0 :
to
x,
z
For
.
A x - b
m
-
1,2,...
we obtain an improved solution
by the following
(1) rm - b -A
xm
(2)
-
(A + E) d
r
mm
(3) xm+1
x
+d
We shall assume that (1) and (3) are done exactly.
This is a
reasonable assumption if accumulated inner products are used (see
Wilkinson (1965), pp. 116-117).
Thus we shall assume the only error occurred when we tried to solve
A d - r
m m
but performed (2)
m
(A + E) d
-
r
m
instead.
The iteration is meaningful only if the matrix
A
sented exactly in the computer, i.e. if the elements of
can be repreA
are known
exactly and no round-off occurs when the numbers are read into the
computer.
i
i --
96
13.3
Convergence of the Iteration
xm defined in §13.2 converge to the
We must now show that the
solution
xA
b
s
Theorem: Let x
A- 1 b
b - A xj , (A + ) d=r
r
lira Ix
xl
-
(A + E) d
Ax-Ax
.
(I + A
I
- 1
IA
a
r
- A
1
El
a < 1/2
Let
d
Then
x
b-Ax
1
.
- x) =E(x
So
-x)
(A + E)(x
-x
Since
det A
.
0,
E(x - x)
r
El < 1/2 .
-- |
-1 -xl
-xl
<Ax
<
Assume
IA , and x
(A + E)(x
Thus
E)(x -x)
rn
Let
L
0
Proof:
Ix
b
Let
T =
a/(1
-
.
Since
T < 1 , lim Ix
-
1
T <
So
T IX _
IA - 1 EII(1--rn-TIx
IA- 1 El)
1m - 1 Ix, - xl
a)
xl
x
-0
rq.e.d.
Thus the iterative vectors converge to the solution
provided that
norm, i.e.
IA- 1 El < 1/2
Ix.+
1
< Ix
-xl
and the convergence in monotone in the
-
xl.
IA -1 El < 1/2
n
1
i-i
K(A)
If
Ixii
(72.3) n
A1 IA-1 1 ,then
.
n
and
1A1-
max
I
2,9446
Corollary:
b
.
Now we must give conditions under which
use the one-norm: 10
x - A
!
i-I
IA 1jl
_t<1/,whr
f(n) K(A) 2
lim Ixm - xl
0
where
< 1/2 , where
x - A- b
We shall
97
From §13.1,
Proof:
IA|1 1 p0 < K(A)
po< IAI , so
f(n) K(A) 2
t
f(n)
|El < (72.3) n
.
Hence
|A- 1 El
But
0
94
< (72.3) n2.
46
ard the corollary follows from the previous theorem.
q.e.d.
We see that the convergence of the iterates depends on the condition number of
A , and so we cannot expect iterative improvement to
be of value for ill-conditioned matrices.
For further remarks on
condition numbers, see Wilkinson (1965); and on iterative improvement,
noted in
see Fox (pp. 49-53, 109-113) and Moler (pp. 316-321). As we
while
§13.2 the iteration is meaningful only if no round-off occurred
reading
A
into the computer and if the elements of
A
are exact.
I
98
Chapter 14
Symmetric Band Hatrices
Gaussian Elimination for Band Matrices
14.1
Let
A
be an
2m+l << n , i.e.
nxn
non-singular band matrix with band widrh
Aij - 0
We could store
tions by ignoring
A
Aij
li-Jl
f
in
(2m+l)n
for
li-Ji
band structure while solving
> m
> m .
A x
-
n 2 loca-
locations rather than
If we could preserve the
b , then we would save storage
and thus be able to solve band matrices of very large order in relatively few storage locations.
If we use Gaussian elimination with complete pivoting, then we
must interchange to bring the maximal element to the leading diagonal
position.
n
This could destroy the band structure and we would need
locations to store
L
and
U
in the decomposition.
If we use Gaussian elimination with partial pivoting, then
is unit lower triangular with
L
= 0
upper triangular with
if
li-jl
stored in
mn
and part of
Ul
0
locations and
U
U
in
can be written over
if
ji-Ji > m+l
> 2m+l .
(2m+l)n
Thus
locations.
and
L
U
L
is
can be
Since
A , we would nced only
L
mn
additional storage locations.
Thus if
if
b
A
is an
nxn
band matrix with band width
is a vector of length
partial pivoting requires only
and
n , then Gaussian elimination with
(3m+2)n
storage locations to solve
A x - b . Furthermore, this method requires only
multiplications and additions.
2ar+l
- (2m2 + 4m + l)n
99
Diagonal Pivoting for Symmetric Band Matrices
14.2
Let
width
A
If we use diagonal pivoting (see Chapters 5, 6 and 8)
2m+l .
to solve
symmetric non-singular band matrix with band
nxn
be an
A x - b
then interchanges can destroy the band structure
,
I2
and we would need
storage locations.
2
We have investigated many variations of the diagonal pivoting
method for the symmetric band case, but these algorithms have either
been unstable or have required more storage and operations than
Gaussian elimination with partial pivoting.
At the present time we recommend Gaussian elimination with partial
pivoting rather than the diagonal pivoting method (see Chapters 5 and
8) for symmetric band matrices, and thus we are unable to take advan(For the special case of symmetric tridiagonal
tage of the symmetry.
matrices, see §14.3.)
Symmetric TridjgAonal Matrices
14.3
We are able to present a stable algorithm for the symmetric tri-
diagonal case which requires less storage than does Gaussian elimination with partial pivoting.
Let
i.e.
A
be an
Aij - 0
max
i l
li-Jil
Ai+li
A ii+i -bi
2<i<n
if
nxn
'
ax
l<i<n-i
synmetric non-singular trLdiagonal matrix,
> 1 .
for
Ibmil < a
Let
A
1 < i < n-1.
-
ai
for
Asume
1 < i < n
Ial
and
< c ,while
100
Let us consider only the first step, which is typical (cf.
Appendix A, §A.3).
b
If
0
we have nothing to do, and
,
A
A(n-1) -
i+1,j+l
ij
bI
Suppose
2
r >
# 0
Let
.
.
If Jail _ 2 r b
Ixl pivot. Then M 2 1
A (n-1) = A
i+1,j+1
ii
la1 I <
If
2 -r
2
-
since
2
Now
r 8 < 1
as a 2x2
II
-
b
32
b 2 /(a
<
2
)
-r
81(i
'
2 -r
2
- b2)
A(n-2)
ij
lb 2 1/[1b1 l
1M31 1
-
a
and
2
b1
,
M3
2
-a
1
a2
8)
8)
> 0
a 2 - b2)
1
Thus
A n 2)
i+2,j+2
(1 -
2
r
8)]
2-r a
])i <All
8I(1 -
2
8)
,
while
<__
8 otherwise.
We see that the bounds on the elements of
A( n -
b 2 /(a
therwise.
. A
(n-2)1
-r-
and
lb I
_ 2 r/
8 otherwiee.
<
b2 , then we shall use
b
is tridiagonal,
2
A(n-l)
as a
by assumption.
M31
A (n-2) _ a
11
3
1M3 2 1
and we shall use a,
a 2 - M1
, and
- 2 -K 2 1 b
is tridiagonal, IM2 11
1
A (n-)
while
,
a, # 0
A
A11
11
then
b/a
(Note that we make no interchanges.) Then
2
> b2 (1 - 2- r
1 >b - , a2 > b2 (1 - 2 r a
1
21)
b1
1
1
1
pivot.
Ia
I
,
Thus
+ 2r
S(n-)
IA(
be a non-negative integer such that
r
2)
for
2x2
A (n -
are independent of the bound
)
for
a
on
a lxl
Jal
F
101
Thus the pattern continues throughout all the reduced matrices.
we conclude ; each reduced matrix
IA k
< max {6 + 2r , 6/(
A(k)
Hence
is tridiagonal,
IAi'(I
- 2r 8)} ,while
<B
otherwise.
1
I
Usually we normalize by choosing
Ai
< max (1 + 2r
1/(1
for each reduced matrix
A
1 < I + 2r
k)
Max max JA(k)I <
k
A
-
2r)}
(k)
.
Then,
IA(kI
while
Since
2
r
>B-
2
r
(- 3
r-
for
otherwise,
r > 1 . Hence
Thus given any positive integer
1
1
r
we have
,
1)
iJi
A backward error analysis of this algorithm shows tht it is very
stable (since the elements of all the reduced matrices are bounded by
2r
I +
, which takes on its minimal value
Thus we can decompose
A - M D Mt , where
with blocks of order 1 and 2, and
M
- 0
il
if
D
i+l'i
#0
3 for
M
and with
r - 1 ).
D
is block diagonal
is unit lower triangular with
IM
'ij'I
-O
if
1 >
i
>+
+2
We shall need an n-vector array to record the pivotal selection.
We set pivot[k] - 1 (2) if we use a lxl (2x2) pivot for
pivotik] - 2 , we set pivot[k-1] - Mnk+3,n-k+l .
storage locations to store the rest of
2n
over
A ).
Thus we need only
3n
M
and
A~k)
Then we need only
D
(these we write
storage locations for this algorithm.
From §14.2, we see that Gaussian elimination requires
and
A,
7n
§A.3).
If
5n
storage
operations, and is very stable for tridiagonals (see Appendix
102
We would also like the number of operations required for our algorithm to be less than
7n . However, if we use the algorithm in the
1
manner in which we have expressed it,
and
5n
8;-- n - 2- p
2
additions are required, where
pivots used.
Thus between
6n
(We ignore multiplication by
and
2
1
2
p
1
82
multiplications
is the number of
lxl
multiplications are required.
in the count.)
However, we can reduce the number of multiplications from
1
1
&2 n - 2- p
7
to
1
n
-
3
p
if we implement the algorithm in the
following manner:
(We present the first step of the algorithm in Algol form):
if b[l]
else
- 0 then M[2,1]
:- 0
begin temp :- al1]/b[l];
if
abs (temp) > abs (b[lJ)x2+(-r) then
begin M[2,1]
else
:. 1/temp; a12]
begin calc :M[3,1
:--
:- a[2]
-
M[2,l]xb[l] end
tempxa[2] - b[l];
: -
b[2]/calc; M1[3,2]
a[3] :- a13
tempxM[3,1];
- M[3,2]xb[2] end
end;
A backward error analysis shows that this implementation of the
algorithm is also
very stable (since all the reduced matrices are
bounded by
).
1 +
2r
Now min (1 + 2r) -3
r>l
as
r
-
for
(since the larger
rl
,while
1
3
( 71-n--Ip) -#6n
2
2
r is the more likely the choice of a
103
lxi pivot becomes).
But
(1 +
2r)
as
r
*
,
not have a good bound on the error matrix for large
(i.e. as close to
6n
2
r
.
Thus in
r > 1 so that
is reasonably small
practice, we must make some reasonable choice of
r
1+2
is not too large but so that 71
3
50
and thus we would
2jpv sraoal
ml
as possible, and hopefully not more than
7n ).
We have considered many versions of diagonal pivoting for the
tridiagonal case.
The minimal storage possible is
3n . The above-
mentioned algorithm had the least operation count of all the versions
studied.
Since this algorithm requires between
tions in comparison to
7n
6n
and
7i n
multiplica-
for Gaussian elimination with partial
pivoting,we can recommend this algorithm for general use only if
storage of
3n
rather than
5n
is crucial to the user.
A-1
Appendix A
Diagonal Pivoting for Positive Definite Matrices
A.1
If
A
nxn
is an
maximal element of
to
Miscellaneous Results
S
A
symmetric positive definite matrix, then the
is on its diagonal.
So
p
. 0
we use that r-qximl diagonal element as pivot.
,
is also positive de ir.'te.
p - n
Thus
(where
p
is
and, according
But
(n-1
-
A0
1)
the number of
lxl pivots used in the decomposition.)
Since
cessary.
(k)
the
V(k) =
(k) for each
0
A(k) , calculating
1
(This calculation would require
Thus if we know that
.)
omit the calculation of the
p
(k)
k
A
,
n
L D Lt
is unne-
additions for all
and our method is identical to the
If we also omit the cal-
and use the first diagonal element as a lx' pivot
p(k)
(non-zero since
03
is positive definite, we may
method of congruent transformations (§2.7).
culation of
p 0(k )
A
is positive definite), our method is identical to
(92.7).
From the above we see that the
congruent transformations (i.e.
L D L t method and the method of
L D Lt
with pivoting on the diagonal)
are special cases of the c.iagonal pivoting method, and either of these
may be used if
on this topic.)
A
is definite.
(See §§2.6
-
2.11 for further remarks
I
A-2
In our algorithm for the diagonal pivoting method in Appendix
C,
0J
we allow the following options:
(I)
If
(2) If
(a)
A
is indefinite, then we must use diagonal pivoting.
A
is definite, then we may use:
L D
with pivoting on the diagonal (by omitting the
calculation of the
A2 (b)
I'An
,
|
A*2If
ij
LBD Lt
p (k) ),or
i
i
(by omitting the calculation of the
uA Result
loih
thefordiagonal
oSymmetric
pvoting
o method
symtiGa
Hadamard
Hatricee
nxn
real matrix
H
is liadamard if
IIIl1
P.k
and the
sin pemnation
110
for all
<~~is
f following
A
sidfnie 0 re
s Usually
dit au we a normalize
eawthe
nitin
atio
iI
where
po>
, andtipoperies
H hnw
Ht -n pI 1. yum
by
th
choosing
Ilus,
ti
0 .1
.
Thus all the elements of
and the columns of
t
te
th
columnnof
if
H,
H
are of the same modu-
H
re mutually orthogonal, i.e. if
t
then H H~ -n6
whete
H
is
i -j
is the Kronecker delta.
6
ij
first step under any strategy.
Theorem:
columns ofs HH(n-1)
Then the reduced matrix
H (-)has
is not Hadamard, and the angle between any two
i
e/3
A- 3
B-
H
1-
Let us assume
Proof:
We use
1
as pivot.
H
Let
(n-1).
Bij - Hi+lJ+ I
Then
Y
Bks
ki(r+lk+l
s+ll Hk+I 1 /H1 1 )
=
H
ijI
H
(Ii
/H)
Ht H
(H
/H )Ht
1 r+l s+1,1 11
r+l,l 11
1
+1
/H2)Pt
H
(H
r+l, 1 s+1,1 11
1
r 0 s
If
B
Further
B
IB 12 = Bt
r
r
r
cos O(r,s) -
r
B
and
r+1
i,j
Bt B 0 0
r s
Thus
r
H
Ht
, then
for all
I
t
H
. t
~
r+l s+1
tH
n-I.
Hr+ll Hk+,/HIl)
(H-
-
(if
6+i,k+]. -
I
1 < ij
n-I
n-I
X B
k-l
BBtB
r s
l,/HI1for
Hi+l,1 HJ+
-
0
,40
t
and
for
/H2
H
+ Ht H 0
H
r+l-1 s+1,1 11
1 1
s+1
r+1
S+1
H
Bt B - - n
r s
so
/H2
r+1, 11I
- 1/2 , so
B
O(r,s)
is not Hadamard.
H (n1)
2
= n + n (H
cos O(r,s)
since
n.
H1
s,
r
+
2n . Define
r
B /(IB C IB 1).
s
s
r
±
1-3
for
Then
r 0 s
q.e.d.
A similar result holds for Hermitian Hadamard matrices.
Gaussian Elimination for Tridiagonals
A.3
Let
li-il
T
>1x).
be an
suppose
nxn
tridiagonal matrix
ITn[! < ci
,
and
I~j
(i.e.
T j - 0
is
< S otherwise.
for
A-4
Theorem:
Let
T
be as above.
T~ k
Then for any reduced matrix
((k)
under Gaussian elimination with partial pivoting
<
2
otherwise.
k - n-1 is typical.
The situation for
IT11 1 > IT2 1 1 , then we use
If
(n-1)
Ti
<
I
, and
Proof:
-
is tridiagonal,
,(k)I
2(k)I
IT
T
(k )
/T
while
'2
- 21
22 12 1i+l,j+l
r Is idiagonal,
ITJ11>
T 1 1 as the pivot.
Totherwise.
T(
< 2
So
and!IT
Thus
'>;
row
ot he rwise.
ITIIl
If
an
T(1
ij
IT.(n-1)I
IT 211
then we interchange
seTastepio.
21
while
<
oT(n-1) ..
s
- T
te first and second rows
T/
l2
i+1,j+1
28 , and
IT(<
otherwdie.
o
-
Thus
T (n - 1 )
is tridiagonal,
< 18 otherwise.
From the theorem and §2.5,
max max IT
k ij
q.e.d.
we conclude that Gaussian elimination
with partial pivoting is very stable for tridiagonal matrices
since
(n-1).-T
22/21 ' 12
< 2 m-x ITI
i
i
T,
T/
-T2 3 T 1 1 T 2 1 '
B-I
Appendix B
B.1
Algorithm for Symmetric Equilibration
Discussion
The following Algol procedure will equilibrated any
metric matrix
diagonal.
A
elements of
B.2
A
so that
D A D
is replaced by
D
is equilibrated, where
nxn
D
symis
D A D , and the inverses of the diagonal
are stored in the vector
d
(See ChapL! r 7.)
The Algol Procedure
procedure symequil (A, n, d);
value n; array A,d; integer n;
comment the symmetric matrix
A
of order
n
brated and the symmetric equilibrated matrix
stored in A , where
ct.1
:=
begin
for
1 until
n do
iij :- sqrt (abs (A[i,i]));
3
:-
1 step I until
begin t
i-i
do
:- abs (A[iJ]);
if t > d[i] then d[(I
:-
t
end;
if d[i] 0 0 then
begin for j
:= I step 1 unti: i do
A[i,j]
for j :-
i
- Alijl/d[Li];
step 1 until n do
AIJ,i] :
_i
D A D
-*1
U
= diag (dill,..., din]);
begin integer i.j; real t;
for i
is equili-
AiJ,i]/dti];
ic
B-2
end;
end;
for i
:-
if d[i]
begin
1 stEp I
-
until n do
0 then
for j :-
i+l step 1 until n do
begin t ;- abs (AIJ,i]);
if
t > d[i]
then di|
:- t
end;
if d(i] - 0 then
for J
:-
oto alarm;
i+l step 1 until n do
A[J,i] := A[J,i]/d[iJ;
goto out;
end;
alarm: print ('this matrix has a null rows)
out
end;
c-i_
C-1
Algorithm for Diaoal
Appendix C
Pivoting
The following listing of an Algol procedure will solve
A
by the diagonal pivoting method, where
symmetric matrix and
B
is an
is a vector of length
A X
-
B
non-singular
nxn
n .
The L D Lt method (symmetric Gaussian eliminatio' and the
method of congruent transformations (L D Lt with pivoting on the
diagonal) are special cases of the diagonal pivoting algorithm.
2, and
M[i+l,iJ
=
0
when
is declared
X
the solution
AX
=
B
D
are written
B
and
is
[1 : n
B , i.e.,
is stored in
.
X[i]
Upon exit,
is
BKil
stored in
If
to
and
A
n, 1 : n]
[1
0 . H
D[i+l,i]
over the lower triangular part of
A
is unit lower
is symmetric block diagonal with blocks of order 1 or
D
triangular,
H
A = H D Mt , where
is decomposed into
A
part.
is assumed to be stored only in its lower triangular
A
The matrix
is indefinite, then set
A
DEF
=
0
and the general
diagonal pivoting method is used.
If
is (positive or negative) definite, then we may omit the
A
calculation of the maximum off-diagonal element in the reduced matrices.
If
DEF
L D Lt
=
2
then this is omitted and the algorithm is identical to
with pivoting on the diago"l.
The pivoting on the diagonal
C-2
may also be omitted if desired by setting
algurithm is identical to
DEF
1
,
and then the
L D Lt
The algorithm,as presented below, is by no means, in its most
efficient form.
In particular, as written, no advantage of symmetry
Instead of using only the lower triangular
is taken to reduce storage.
part of
Ajl:n, 1:n], the algorithm should be coded so that the lower
triangular part of
A
n (n+l) . Further,
is stored in a one-dimensional array of length
B[1 : n]
for solving a system with
k
could be replaced by
right hand sides.
B[l:n, 1:k]
C-3
*pw('CF IIPF
[V1
tm)Ays
lkP ,p
(A I.p orUEF)
Ie
tINTFrEP* NiOEF
to
';tLvFS; A X 0 R BY THF L)TAGONAL PIVOTING MFTHOD
.8Tr
AfP
A SYM AFTRIC MATPIX oF ORDERP N AND
P IS~ A VWrT)R OF LENGTH N t
*C~t-rN.T* A~ L5 A:Ur'ED To HE rTOPEn ONLY IN ITS LOWER
t ('1,IAP f;T i
W
lkIAl tLLAP PART, M AND D APF WRITTEN OVER A WHERIE
A = 1 C m TRANSPnSE. M IS IuNIT LOWER TPIANGULAR,
Al' : I' IS RLOCK DIAGONAL WITH 13LOCKS OF ORDER I OR 2.
AN.
"/11./
a (0WHE~t
#(KI.,*OT
EQUALS 0 ..
3fCfOJMFvT* IF A 14; It..EFImITE9 SET DEF *0ANO
THE DIAGONAL
PIVOTIIc, tETHOC IS USEr.,
*CnVMFNt\T
IF jM IS (POSITIVE C10 NEGATIVE) DEFINITE, THEN
SIT IF =1 AKC L C L TRANSPOSE WITHoUT PIVOTING
w1Itl HE -jrEfl CR SET 0FF a 2 AND L 0 L TRANSPOSE
W
I
c'ir
CN THE DIAGONAL WILL OF USFO .,
w-nt (*I.
$IN-TE.J-*
$Pl A1.1
OAPP'AYt
()FT.
AVF* TFHP,
ALPHAto
*
sAPpAV* CIAt.oE (/,.N/)
t
Phw~T
.91*en)*
(AeKvf~..JqM1)
*pLl()cFI-0pE;*~ HA~.IIA,
ifVAL'JF$
tI\
99
~rH~~*PAL$ Ml
taRPAY*
*o
*IftTFGER*
Kt.'qj
to
THF
t- AXIMUM. MF THE
FCR K~ OLEO$
141
DIAGONAL
OF A ,
ml1
I *LEO* N, AND J IS THE
=Afi4(A(/K*K/))
*9 J on K so
qTEP* I *UNTIL* N *00*
*I*AS(A(111T/)) tGREATFR% "I THFI*
*PH.-IN* pi
au AlBqcA(/I,I/fl
vgJ
i
IFr~l*
at
.9I
trotPFI T~tCAICULATFS
umAX A(A/,)
*PFCGIN**IN1Fnkk*
*FOP* I
K1
*
a
.9
*ENO* **
MAYIP~.
*PIWOCF[UUP.* MAX ^(AvKvr9R9SH0*L.ml) s
*VALUE'K Nt mILaM1
*ARPAY*
A to *TNTEGF.R* KNspoc,,L .9 *PFAL* ?PO.m1 ..
*roCtffj~iNt
CALCULATE-, 100 a MAX ARS(A(/Ioj/) ) FOR
I- *LF'~W It. OLEO* N- AtM THF INTEGERS R ANO S SUCH
TpaT AES(a(/PqS/l1 z M6, IT IS ASSUMED THAT MI
PAiX (AES fA(/IT/) ) ) AND, vIa ABS( A(/LvL/) )
iFCT~:**I~T(;~*It'J .9 140 9= ml
P *a S .m L so
*sF001 j
*STFP* I *UNTIL* "-I *n0*
.
I
C-4
1 OUNT1L;9 N f$oO*
1 OuNTIL* N $00$
*STEP*
&Hq ( A(/y.J/1 . *GPFATERO MO *THNJ*
j ,:KSTFP*
*il
I
$*L0P*
irrnpV
*TF*
*I4Er.TN*
.
or AS~S ( At/TJ/)
q. S or .j 0END*
an I
)
PCO
.
*,
tpRCunuUP * It.TFPCNANC-E CAtKel) .
*VAI.k-F* KoI so *ARRAY0 A .0 *INTEGER* Kol
..
IlNTFQC ANG-ES ROW ANDl CnLUJMN K WIT14 POW
CO)LUM'N) I wl-EpF K *CEn$ I AND A 15 THE REDUCED
~t0L
S9
.'I
m!ATPIX O~F CPrFP
* Fr:IN* 0LdALj! !FPP v. *INTFGFR* J .
N *00**
, .= K+1 *SlEpt I *uNTIt.
*rIoP*
$I
*PC INI TFvP ow At/JoK/) so tA/JqK/) qm A(/Jl1/1
~j*EN*
TVMP
o3
A(/,jei/)
*FnPs i tc Tol *STEP* I1 *UNTIL* K-1 000*
*rv-~T
L
.= TFMF *Etn*
A(/Kt../)
0f*i
ALPHA
A / 0 /
( T l )
.
TF P
soflA
=A(/KoK/)
(HIpoCF
It
Ti
~PTH))/8
*
v,
.
STAPI..
cHF,lN* rwANLF
(/7/)
or I to
(A*Ktl)
*F~1N% !;TEc2CH4ANcE
*(<TC* P~vCTCNF *FN1t*
*GO1O$ PIVOTONE *END*
'v
CHANGE
fMl *NOT i-FSS* N~O * ALPHA *TH.FN*
* AFrINX lK1EPrHANG-F (AKo1) at CHANGE
*g
sr.C;TC' IPTvCTCNE *F~)*
*IF*
(/1/)
(/I/)
.
or K .9
90 x
TNTEPCMANGF (AqSvI) #9
I *TwFNO
*
R
on
CH.4ANCE C/I/)
P *CRtATFct$ 1.1 *1i.4NO IKNTEPCHANGE (AtP9I.1)
*g
gli-s 5 $GPIEATFQ$
*IF$
2
*,-OTO*
PIVOTTwn
.9
ts
P~vOTOfI'F
I*enM~rT*
*IF#
THEN* *GOTn* ALAPM t
kvF U E A IEI PTVOT
011111)~2 6
*FOP* J
z: 141 $STEP'*
*Cfl.,~E t
gF0o,* j .= 1+]
z' *: 1
Ito#r
1 *UNTIL$
N *00*
HvSr BEEK, SET FOUAL
/jvl/)
fITEP* I *UNTIL* N *rO0*
J t'0
I *UNTIv_*
*rTEI*
A1*l
I(J K).=
TO TIE MULTIPLIER
so
c-5
srnpMFr'Tt 1'+.. A(,e/
HAVE tiFEN~ SET To THEIR NEW VALUES
*CflMMFe'T* PTvOH,/1/ a I Ir WE UqE A IXI AT ROW I as
*IF* I *PFT U;E.TEP*
1C~#~F~'*
FSE
AGl0
t4~~
IFS
TSFP
z
A(/Jql/)
rZOR 0 STAR
*THEN A
FIN,( Cy
TUTL *
.
J-1, *DOALAR
T*2
/Kl/*
A(/J.oc/) - A(/Kqt/)*A(/Jo1/)A(/J.14.1/) a
*r0N'Ft4T* T.E *A(/,gK/ HAVE REE~' FET Tn TH~EIR NEW VALUES as
SAVJF .3 A.(/JsT/) as TEMP .uAC/J9I1/1) go
A(/JgI/) .=(/.,.1)SV
A(/I*39I/)TFMP)/lET so
/d/sT'/).= (A(/1I/)*TFMP - A(/I#)9I/)*SAVE)/OET a
~C~iM~~T
A(~g~/)ANn A(/Jo1.1/)
HAVE BEEN SET EQUAL
IC THE APPInPPIATF MULTIPLIFR "
At/jtj/)
*r' Ac/,.,j/)
AC/JI/)ogSAVE - A(/Jtl.1/)TEMP .,
.8
*Crn lF"ITt PTv0T(/I/) a 2 IF WE kir.,A 2X2 AT N~OW I AND THEN4 DET IS
PIVOT (/I/) -= ? -9 PIVOT 1/1.1/) sc !)ET .w 1 on 1+2
*IF* I t?'CT (PFATEP* K~ :THEN* *rI0Tfl9 START
*FI 5F* *GO~lr*
FI~r) C .
*Cti11-4ft'Ts Inv.
F')Fmr =
m~ lIKvESF
TTM.FO, 14 A0f) STORE
..
IT IN 8
.9
T
P~FPiAT.
~vF*~H(/J/l .' f(/I/) got R(/CHANGE!/I/)
CtH~h'iF/I/) I)
.q AVE so
*IF;, P~vrT(/I/l = I THEN*
*k.FG1N*
*rH
.=: 71 * TEr1* I *INTIL* N *Cn*
[,(/j/)
.. = kU/.j/) - A(/joT/I 4b P./I/) e
I)
.
RC/
1.
lo.
S
SAE
*Fkiv{0
.
*FLSE49
*pCIN*
~/Ii,).' (/I*1/)
CHArFI4I/)
eon SA'
*FOP$ j *z 1.2 iiSIEPs 1 *IUNTIL*
P(/
I .=
*IF*
I
I+? *Etn#
st\Cl
. P(/
at
CHANflE(/I*1/)
N tflO*
.,
GliFATER*$
*THEN,$
$C6oTO* REPEAT
at
as~
srrFTg
iC-4 SflL\F
r.
Y
c
Ar'fl STnPE Y IN THE VECTOR 8 .
as
c-6
SOL VF..
PIV~i1
11JFS
i *T
(/I/)
I $(EfTvFW* h. *Tl4EN'
OFICF* *r'ryTfl SO, VE
joffs
4V~Nrs
LTM
.
R(/HI/)
*COTO*
au BUT41/I
o, SAVE
FiNfl x
DET
.9
ou PIVOT(/!4j/)
.
1FAP(/I41.1.1/) - SAVFOA(/141.I/) )/OET go
M(/1)
*C
)/OET *
SAVE*A(/ItT/) -TFMPOAC/I.1,I/)
I~'/.W)*:
*1F*I#i-'iTEPgf
OGOTO0 FIN~DX
N OTE,
*r7OTO*
*FLSE*
SOLVE
X am INVFRSE TRANSPOSE TImES Y wH'EPE Y IS
I~CAw c-OtVE
Ts
SYI'RFr IN TWF VFCTOP 8 ANfl STrnRE X IN 8 t
*()mmUF
F t:IX
CALC so
PlvCT(/I/
W$I
*FOP*
T.: *.STEP*
J
SAVF .= tb(/T/)
83(/
WrR* J *a
a
1 *Ti.EPm*
1 *UNTIL* N
az R/
SAVF .*
)* .=M('(I)
*,B(/!/)
*FCR* K on
*FLc E* 1'9LQ.INl
T.1
*STEP*
I-le!
1 *tINTIL*
son*~
CH4ANtE(/I/)I).
I *z
I-1
*END*
*00n
N *PO
*EGIN$
a~ 1-1, 1 000
OFCP$ K
..
R(/S(/) on RU/ CHANGE(/K/)I).
SAVF ex P(/K/) .,
*END$,#
.
oft
AVF go
BU/C"1ANCr(/K/)
=~ 1-2 tENI:
I
7 $f,~T LFsci 1 *THFNI* *Gr.Tfl* CALC .
*FNfl
$IF*
*G~OTr'
OtiT
ALAPPM
nULTPLJT
OUT
.
..
(61
*FN[O$
#(*
*(s
PIVClT ,C
SIrtrULAW
MATPTX
)
~
2.
Bibliography
Cryer, C. W.,
"Pivot size in Gaussian elimination", Numerische
Mathematik, 12 (1968), pp. 335-345.
Davis, P. J.,
The Mathematics of Matrices, Blaisdell, New York (1965).
Forsythe, G. and C. Moler, Computer Solution of Linear Algebraic
Systems Prentice-Hall, New Jersey (1967).
Fox, L.,
An Introduction to Numerical Linear Algebra, Oxford Uni-
versity Press, London (1964).
Gantmacher, F. R..
The Theory of Matrices, v. 1, Chelsea, New York
(1959).
Hildebrand, F. B.,
Introduction to Numerical Analysis, McGraw-Hill,
New York (1956).
Householder, A. S., The Theory of Matrices in Numerical Analysis,
Blaisdell, New York (1964).
Marcus, M. and M. Newman, "The permanent of a symmetric matrix",
Notices Amer. Math. Soc., v. 8
(1961), p. 595.
Meersman, R. de and L. Schotsmans, "Note on the inversion of symmetric
matrices by the Gauss-Jordan method", I.C.C. Bulletin, 3 (1964),
pp. 152-5.
Mirsky, L., An Introduction to Linear Algebr,
Clarendon Press,
Oxford (1955).
Moler, C.,
"Iterative
refinement in floating point", Journal A.C.M.,
14:2 (1967), pp. 316-321.
i
I
I
Parlett, B. N., and J. K. Reid, "On the solution of a system of
linear equations whose matrix is symmetric but not definite",
M.I.T. (to appear).
Reid, J. K.,
"A note on the least square eolution of a band system
of linear equations by Householder reductions", Computer Journal
10:2 (1967), pp. 188-189.
Schotamans, L., "Gauss-Jordan inversion of symmetric matrices, using
a 2x2 pivot submatrix", C.E.N, Report No. 621-42/65-153, Mol,
Belgium (1965).
Shisha, 0., editor, Inequalities, Academic Press, New York (1967).
Sinkhorn, R., and P. Knopp, "Concerning non-negative matrices and
doubly stochastic matrices", Pacific Journal of Math., 21:2
(1967).
Sinkhorn, R., "Diagonal equivalence to matrices with prescribed row
and column sums", Amer. Math. Monthly, 74:4 (1967).
Sinkhorn, R., "A relationship between arbitrary positive matrices and
doubly stochastic matrices", Ann. Math. Statistics, 35 (1964),
pp. 876-879.
Tornheim, L., "Pivot size in Gauss reduction", Tech. Report, Calif.
Res. Corp., Richmond, Calif. (1964).
Tornheim, L.,
"Maximum third pivot for Gaussian reduction", Tech.
Report, Calif. Res. Corp., Richmond, Calif.
(1965).
Weatlake, J., Handbook of Numerical Matrix Inversion and Solution of
Linear Equations, Wiley, New York (1968).
Wilkin-on, J. H.,
"Rounding errors in algebraic processes", Infor-
mation Processing,
"Proceedings of the International Conference
on Information Proccssing, UNESCO, Paris (1959).
1 II II I III
II
I
I
Wilkinson, J. H., "Error analysis of direct methods of matrix inversion", Journal A.C.M., 8:3 (1961), pp. 281-330.
Wilkinson, J. H., Rounding errors in algebraic processes, Notes on
Applied Science No. 32, Her Majesty's Stationery Office, London;
Frentice-Hall, New Jersey (1963).
Wilkinson, J. H., The Algebraic Eigenvalue Problem, Clarendon Press,
Oxford (1965).
LAI
E,,'VI
'wI