Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
629
A DIRECT ALGORITHM FOR SOLVING ILL-CONDITIONED
LINEAR ALGEBRAIC SYSTEMS
X. J. Xue’, K. J. Kozaczek2, S. K. Kurtzl,
and D. S. Kurtz2
1The Pennsylvania State University, University Park, PA 16802
‘HyperNex, Inc., State College, PA 16801
ABSTRACT
In this paper, a new algorithm is presented to directly solve the linear algebraic system Ax=b,
where A is an y1x it coefficient matrix which may be singular or ill-conditioned. By writing the
system as an expanded matrix A’= [ ~ibi E], where E is an n x y1 unitary matrix, one can
transform A into a unitary matrix through the row-transformations with complete pivoting and
proper zeroing, It is shown that the algorithm can provide a solution in the non-null subspace
of the solution space, if matrix A is singular. The criteria for curing ill-conditions are related to
the numerical precision of computers. Numerical examples demonstrate the power of the new
algorithm.
INTRODUCTION
As linear algebraic systems are very often ill-conditioned in engineering applications, methods for
solving such ill-conditioned systems have been studied for a long time [Golub71, Rice81, Press92,
Tamburini93, Kim96]. Rice even stated in 1981 that “if the problem is ill-conditioned, then no
amount of effort, trickery, or talent used in the computation can produce accurate answers except
by chance.” However, the singular value decomposition (SVD) methods developed later in the
1980’s [Golub71, Press921 have been proven to be a cure for such ill-conditioned LAS, although
the methods generally converge very slowly. For an LAS in the form of
Ax=b
(1)
one seeks a solution x for a given vector b and a given y1x n coefficient matrix A. The most
important direct method for solving linear algebraic systems is the Gauss-Jordan elimination
method, but this method itself is not applicable to the ill-conditioned cases. Recently, Olschowka
and Neumaier showed that if the matrix A is positive definite but not diagonally dominant, a best
pivot series can be found through an assignment-like algorithm [Olschowka96]. This work
stimulated us to seek for an algorithm that directly solves ill-conditioned linear algebraic systems.
In this paper, we will show that a direct algorithm can provide a solution in the non-null subspace
of the solution space when the matrix A is singular. Combining complete pivoting and zeroing
techniques with the Gauss-Jordan elimination, the new algorithm can cure the singularity of the
linear algebraic systems. As a side benefit it also eliminates round-off errors.
The basic idea of our algorithm can be described as follows. Let us write the system as an
expanded matrix
A’= [AibiE]
(2)
where E is an y2x YZunitary matrix. The approach can now be elaborated as follows. One
implements row-transformations in A ’ to transform A into a unitary matrix. Denoting a submatrix
of A composed of elements of ajk, j E [i ,..., n],k E [i,. . .,n] as Ai, i E [l,. . .,n] , a pivot is the
629
This document was presented at the Denver X-ray
Conference (DXC) on Applications of X-ray Analysis.
Sponsored by the International Centre for Diffraction Data (ICDD).
This document is provided by ICDD in cooperation with
the authors and presenters of the DXC for the express
purpose of educating the scientific community.
All copyrights for the document are retained by ICDD.
Usage is restricted for the purposes of education and
scientific research.
DXC Website
– www.dxcicdd.com
ICDD Website
- www.icdd.com
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
element with maximum magnitude in the submatrix Ai. This selected element will exchange
position with element aii. If one finds that aii=O, i <n after pivoting, then the null subspace in the
vector space defined by b is determined, and hence one has aj=l, ajk=O, b,-0, ej=l, and ejk=O,
forjE[i ,..., n],kE[i ,..., n], j f k , which is called the zeroing operation. In dealing with illconditioned systems, one can initially choose a sufficiently small positive E. If la,/n,l I E for the
pivot aii, where nCis the maximum magnitude of element in matrix A, then a zeroing operation is
performed.
A more detailed description of the algorithm is given in Section 2. The numerical examples
are used to demonstrate the new algorithm in Section 3. The concluding remarks are provided in
the last section.
METHOD
For the system of linear equations described by Eq. 1 with a square nonsingular coefficient matrix
A, the most important solution algorithm is the systematic elimination method of Gauss-Jordan
when the inverse of A is needed. For instance, the standard deviations of variables modeled by
(Xi-square method are determined by A-‘. The numerical stability of this algorithm is guaranteed
only if the matrix A is symmetric positive definite, and diagonally dominant, or an H-matrix for
short. Unfortunately, in general linear systems the coefficient matrices are often singular or more
frequently ill-conditioned.
Def. 1: An y1x y1coefficient matrix is singular ifin the matrix one or more rows (columns) are the
linear combination of other rows (columns). In this case,
r(A)+
Cl
that is, the rank of the matrix A is less than n.
Def. 2: The coefficient matrix is said ill-conditioned ifin the matrix one or more rows (columns)
are very close to the linear combination of other rows (columns).
0
The following three Facts are the basis of Gauss-Jordan elimination. The proof of these
observations is straightforward.
q
Fact 1: Interchanging any two rows of A ’ does not change the solution x.
Fact 2: Interchanging any two columns of A gives the same solution x if the corresponding rows
0
of x and b are interchanged simultaneously.
Fact 3: The solution x is unchanged if any row in A ’ is replaced by a linear combination of itself
0
and any other row of A ‘.
Actually, each of these Facts corresponds to an operation in Eq.2, thus three types of
operations are now available. The original Gauss-Jordan elimination solely uses Fact 3 as
operation to reduce the matrix A to a unitary matrix. When this is accomplished, the vector b in A ’
becomes the solution x.
Def. 3: The portion of matrix A composed by elements of ajk’s j E [i,. . .,n],k E [i,. . .,n] is called
q
a submatrix ofA and denoted as Ai, i E [l,. . ., n] .
Def. 4: A completepivot in submatrix Ai, i E [l, . . . , n] is the element such that
.,..., n],k~[i ,..., n].
aP’ =maxajk,j~[i
I I
Def. 5: The completepivoting of submatrix Ai includes two steps:
(1) Find the complete pivot aPiin the submatrix of Ai;
630
630
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
631
0
(2) Interchange rows and columns to exchange the position of aPi and aji.
It is well known that an algorithm of the Gauss-Jordan elimination form with complete
pivoting can solve the problem of linear algebraic system as long as the matrix A is an H-matrix. If
the matrix A is singular, then the concepts of null subspace and non-null subspace (range) should
be first introduced.
Def. 5: (Null subspace) If the coefficient matrix A in Eq. 1 is singular, then there is some subspace
of x called the null subspace, that is mapped to zero, i.e.,
&,GQI=O.
(4)
0
The nullity of a null subspace equals to the number of rows of A,,rl.
Def. 6: (Non-null subspace, or range) If the coefficient matrix A in Eq.1 is singular, and the nullity
of A is no more than (n-l), then there is some subspace of x called the range defining a non-null
mapping, i.e.,
AmngeXmnge=b
’
(5)
q
where the dimension of b’ is n-the nullity of A.
By partitioning Eq.1 in the light of nullity of matrix A, this linear algebraic system can be
written as
[AT
;_1[x;::l=jb’]
As
r(A,,,s,)=n-(nullity(A))
(6)
the following Lemma holds.
0
Lemma 1: For a singular n x n matrix, its rank plus its nullity equals n.
By applying an algorithm of Gauss-Jordan elimination with a complete pivoting in Eq.2, the
elements of matrix A corresponding to the non-null and null subspaces are moved to the right-up
and left-down corners, respectively. From Lemma 1, the rank of the non-null submatrix Arange
is n
- (nullity(A)). The solution x,,, of the linear algebraic system thus spans on b ’ and xnc,,ll=O,
leading to the following Theorem.
Theorem 1: If the coefficient matrix A is singular and the number of singularities is no more than
(n-l), then the algorithm of Gauss-Jordan elimination with complete pivoting can find a solution
vector x which spans on the non-null subspace of solution space.0
Numerically, in the algorithm another type of operation than that used in the Gauss-Jordan
elimination should be defined to clean up the null subspace. This is the zeroing operation.
Def. 7: After implementing the Gauss-Jordan elimination with complete pivoting in Eq.2, if a ‘PO,
one can set
a&=1,ajk=O,b]=O,e&=1, and ejk=O,for j E [i, . . . , n], k E [i, . . . , n], j f k (7) III
If the coefficient matrix A in Eq.2 is ill-conditioned, then some pivots will be very small
but not exactly equal to zero after implementing Operation 3. This is a very complicated situation,
because the solution x will be ruined by the round-off errors if these very small pivots are not
excluded from A. The cure for such a situation is the following. One can choose a sufficiently
small positive E, called the ill-condition control parameter, and denote
n,=rnaxaik,j~[l
,..., n],ke[l,..., n].
(8)
I I
If
laii/n,I,i EL..,nl
(9)
631
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
632
for the pivot aii, the zeroing operation is then called. It is suggested that the constant E be chosen
as 1O-24for the double precision (64 bits) floating point operations. Once A becomes a unitary
matrix in A ‘> the vector x is equal to the vector b in A ‘. The variances of x, noted as 02(x), are
specified by the diagonal elements of E in A, i.e.,
02(x)=diag(E)
(10)
sinceE=A-’ in A ‘.
The quality of solution x may be verified by measuring the residual due to reducing the
number of variables in this linear algebraic system from n to n - (nullity(A)).
r range = &
range x,.ge
-
(11)
b9
The applications of this algorithm will be discussed in the next section. One will find that the
parameter E is vital for the successful application of this algorithm.
EXAMPLES
As the singular case is relatively simpler than the ill-condition case, we will concentrate on the
later case. A famous example from [Rice811 is used to verity the stability of our algorithm. Let us
consider a linear algebraic system
Without pivoting and zeroing, one can obtain a solution x and corresponding residual r by the
Gauss-Jordan elimination as the following
x=[O 1O-2o-10”’ 10~“]‘, ~1.6384~10~.
Applying our new algorithm described in section 2, one finds that n,=105’. By choosing different
$s, the corresponding solutions and residuals are summarized in Table 1.
Table 1. Calculation
Parameter E
1o-l2
1o-24
1o-36
1o-48
results with parameter E ranging from lo-l2 to lOA*.
Solution vector x=[ xl x2 x3 x4]
[O 0 1o-5o10-50]t
[O 0 1o-5o1o-5o]t
[-lo-20 lo-2qo-30 1(y40]t
[-lo-20 l(pO_l@30 l(y40] t
t
Residual r
0.0
0.0
1.732050
1.732050
Null subspace
Xl,
x2
x2,
x2
None
None
These results suggest that the Gauss-Jordan eliiation
is almost useless without pivoting
as the residual in this example has reached the magnitude of 104. With only pivoting and without
zeroing, the Gauss-Jordan elimination is still unstable because the solutions in the non-null
subspace are overwhelmed by the round-off errors as shown in the case of E =10-36, even though
in this case the residual is more acceptable. Overall, the results listed in Tab.1 suggested that for
32-bit double precision machines, taking the ill-condition control parameter E =10-24 is a good
initial guess in the sense of separating the null subspace from the solution space.
632
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
Copyright(C)JCPDS-International Centre for Diffraction Data 2000, Advances in X-ray Analysis, Vol.42
REMARKS
In this paper, a new algorithm is presented to directly solve a linear algebraic system Ax=b, where
A is an y1x y1 coefficient matrix with at most (n-1) singularities or ill-conditions. By writing the
system as an expanded matrix A’ = [A i b i E], one can transform A into a unitary matrix through
the row-transformations with complete pivoting and proper zeroing. We showed that the
algorithm can provide a solution in the non-null subspace of the solution space, whatever the
matrix A is singular or ill-conditioned.
A numerical example from [Rice811 is employed to show the power of the new algorithm,
and suggested that the ill-condition controlling parameter E =10-24 is a good initial guess in the
sense of separating the null subspace from the solution space for 32-bit double precision
machines. This criteria has been successfully used in our intensive X-ray difhaction data
parameterization for residual stress calculations which are reported elsewhere in this conference
[Xue98].
REFERENCES
[Golub71] G.H. Golub & C. Reinsch, in “Linear Algebra, Handbook for Automatic
Computation”, Chapter 10, eds.: J.H. Wilkinson and C. Reins&, Springer-Verlag, New
York, 1971
[Kim961 H.J. Kim, et al., “A new algorithm for solving ill-conditioned linear systems”, IEEE
Trans. Magnetics, 33: 1373-1376, 1996
[Olschowka96] M. Olschowka & A, Neumaier, “A new pivoting strategy for Gaussian
elimination”, Linear Algebra & Its Applications, 240: 13 1- 15 1, 1996
[Press921 W.H. Press et al., “Numerical Recipes: The Art of Scientific Computing”, Chapter 2,
Cambridge, 1992
[Rice811 J.R. Rice, “Matrix Computations and Mathematical Software”, McGraw-Hill, New
York, 1981
[Tamburini93] U. Anselmi-Tamburini & G. Spinolo, “On the least-squares determinations of
lattice dimensions: A modified singular value decomposition approach to ill-conditioned
cases”, J. Appl. Cryst., 26: 5-8, 1993
[Xue98] X.J. Xue, K.J. Kozaczek, D.S. Kurtz & S.K. Kurtz, “Estimation of residual stresses in
polycrystalline aluminum interconnects with &i-square fitting”, The 47’h Annual Denver
X-ray Conference, Colorado Springs, CO, August 3-7, 1998
633
633