2.3 ERROR ANALYSIS AND NORMS
37
like this at each step we can eliminate all variables both below and above the diagonal.
The end result is an equivalent diagonal system, thus eliminating the need to backsolve
to obtain the x;'s. This is called the Gauss-Jordan elimination process. Count the number
of multiplications necessary to solve the linear system by this process and compare it with
the operations count of Gauss elimination.
2.3 ERROR ANALYSIS AND NORMS
There are many different types of linear systems or equations each having their
own special characteristics. Thus we can hardly expect any particular direct
method, like Gauss elimination, to be the best possible method to use in all cir­
cumstances. Moreover, ir we do use a direct method, our computed solution will
almost certainly be incorrect due to round-off error. Therefore we need some way
to determine the size or the error or any computed solution and also some way to
improve this computed solution. In this section, we will briefly develop a little of
the theoretical background that is needed both for the analysis of errcrs and for
the analysis of the "iterative" algorithms of Section 2.4 which generate a seq uence
of approximate solutions to Ax = b.
The subject of error analysis must be approached somewhat carefully, since
a particular computed solution (say xJ to Ax = b may be considered badly in
error or quite acceptable depending on how we intend to use the vector XC" For
example, let x, denote the "true" solution of Ax = b and let r = AXe - b be the
residual vector. Then, r = AXe - b = AXe - Ax, or
Xc -
x, = A -
1r.
(2.34)
If A -1 has some very large entries, then Eq. (2.34) demonstrates the fact that the
residual vector, r, might be small, and yet X e might be quite far from x,, Depending
on the context of the problem that gave rise to the equation Ax = b, we might be
happy with having AXe - b small or we might need X e to be near x,,
2.3.1 Vector Norms
In the discussion above, we were forced in a natural way to use words like "large"
and "small" to describe the size of a vector and words like "near" and "far" to
describe the proximity of two vectors. Also, in subsequent material we will be
developing methods which generate sequences ofvectors, {x(klH"= 1> which hopefully
converge to some vector x. In these methods we will need to have some idea of
how "close" each X(k) is to X in order that we may know how large k must be so
that X(k) is an acceptable approximation to x. This will tell us when to terminate
the iteration that generates the sequence. Thus we must have some meaningful way
to measure the size of a vector or the distance between two vectors. To do this,
we extend the concept of absolute value or magnitude from the real numbers to
38
SOLUTION OF LINEAR SYSTEMS OF EQUATIONS
vectors. For a vector
x
=
Xz
X1J
(2.35)
r
x:"
we are already familiar with the Euclidean length Ixl = J(x 1)2 + (X2)2 + ... + (x,Y,
as a measure of size. It turns out, as we shall see, that there are other measures of
size for a vector that are also practical to use in many computational problems.
It is this realization that leads us to the definition of a norm.
To make our setting precise, let R" denote the set of all n-dimensional vectors
with real components
x,. x, ....• x"
ce,}
(2.36)
A norm on R" is a real-valued function 11'11 defined on R", satisfying the three
conditions of (2.37) below (where, as before, e denotes the zero vector in R"):
Ilxll ~ 0, and IIxll = 0 if and only jf x = 0:
(2.37)
Ilo:xll = lalllxll, for all scalars!Y. and vectors x;
Ilx + yll ~ Ilxll + Ilyll, for all vectors xand y.
the quantity Ilxll is thought of as. being a measure of the size of
a)
b)
c)
As noted above,
the vector x, and double bars are used to emphasize the distinction between the
norm of a vector and the absolute value of a scalar. Three useful examples of norms
are the so-called {p norms, II· lip, for R"; p = 1, 2, co; given by
Ilxll = Ixll + Ix 1+ .. , + Ix,,1
II x l12 = J{XJ2 + (X2)2 + ... +
Ilxll = max{lx11, Ixzl,"', Ix"l}
2
l
(2.38)
(X")2
ce
Example 2.11.
By way of illustration for R J ,
Ilxlll
= 5,
Ilxlh
=
3,
Ilxll",
= 2.
(2.39)
To further emphasize the properties of norms, we now show that the definition
of the function IIxlll in (2.38) satisfies the three conditions of (2.37). Clearly, for
2.3 ERROR ANALYSIS AND NORMS
each
xE R", Ilxlll ;:::
39
0. The rest of (a) is also trivial, for if x = 9, then
Ilxlll = 0.
and
Conversely, if
IIxlll
and
then Ixd + IX21
For part (b),
+ ... +
IxlIl
=
= 0,
°and hence XI = X2 = ... =
XII
=
0, or
x=
9.
IXX'
ax
=
Cf. X 2
.
f J
IXXII
.
y
=
l~:l
.,
then
x+y=
YII
XI + Yll
+. Y2
Xl
f
XII
+
YII
and therefore
Ilx + ylll = IX I + YII + IX2 + JIll + ... + IX + YIII
~ (Ixd + IX 21 + ... + Ix + (IY,I + Ihl + ... + IYII/)
= Ilxll l + lIylk
II
lI /)
Similarly (Problem 7) it is easy to show that 11'11<10 is a norm for R". The re­
maining t p norm, 11·112, is not handled quite so easily. In this case, the triangle
inequality in condition (c) does not follow immediately from the triangle inequality
for absolute values, and (c) is usually demonstrated with an application of the
Cauchy-Schwarz inequality (see Section 2.6).
Having the concept of a norm, we can now make precise quantitative state­
ments about size and distance, saying AXe - bis small if IIAx e - bll is small and
saying Xc is near x, if Ilxe - x,II is small. The definition of a norm also has some
Rexibility. For example, we might have reason to insist that the first coordinate
of the residual vector r = AXe - b is quite critical and must be small, even at
the expense of growth in the other coordinates. In this case we might select a norm
be
40
SOLUTION OF LINEAR SYSTEMS OF EQUATIONS
like the one below, where x is as in (2.35)
IIXII =
I I
max {lOIX 11, X2/' X31, ... ,
Ixnl}
Ilxll ::;:
A norm such as this emphasizes the first coordinate. For example, saying that
10- 5 implies that Ix;/ ::;: 10- 5 for i = 2,3, ... ,11 and jxd ::;: 10- 6 . Weightings of
this sort are quite common in problems that involve fitting curves to data and
we shall see some examples when we discuss topics such as least-squares fits.
An idea related to norms which weight different coordinates differently is
the concept of relative error. This is the simple and practical notion that when
the size of the error Xc - X, is measured, we should take into account the size of
the components of XI' That is, if Xl has components of the order of 104 , then an
error of 0.01 is probably acceptable, while if Xl has components which are generally
of the order of 10 - 4, then an error like 0.01 is disastrous. Thus if 11'/1 is a norm for
W, we define c to be the absolute error and define the quantity c to be the relative error. We will usually be more interested in the relative error
than the absol ute error.
JJx
Example 2.12.
x,11
xlll/llxlii
Consider the two vectors
x,
Then the vector
Ilx
Xc -
0.000504]
0.000397]
0.000214
=
[
and
Xc
= 0.000186 .
[
0.000309
0.000342
x, is given by
Xc -
x,
=
-
[
0.000107:
0.000028 .
0000033
One measure of the absolute error is Ilxc - x,III = 0.000168, so Xc seems a reasonably good
approximation to Xl' However, checking the relative error, we see that
-".-llx---,c,.--..,.,-x-"Ic.....11
Ilx,lll
=
0.000168 "" 0.183
0.000920
which more nearly reflects the true state of afrairs, i.e., that our approximation Xc is in error
by nearly 20 percent. Relative errors become particularly important in computer programs
where a criterion is needed for terminating an iteration.
2.3.2 Matrix Norms
In Eq. (2.34) we see that in order to measure the size of Xc - XI' we must also be
able to measure the size of the vector (A -Ir), since Xl is not known. If we knew
A-I, then we could simply multiply A - I times r and measure the size of (A -Ir)
by one of the vector norms of the previous section. However, since A-I is not
known, we must use a different approach, since it would be quite inefficient to
2.3 ERROR ANALYSIS AND NORMS
41
accurately compute A - I just to check the accuracy of Xc- We are thus led to
consider a way of measuring the "size" or "norms" of matrices as weJl as vectors.
We shall do this in a way such that we are able to estimate the "size" of A - I by
knowing the "size" of A and then to estimate the norm of (A - lr). The concept of
matrix norms is not limited to this particular problem, i.e., estimating c but is practically indispensable in deriving computational procedures and error
estimates for many other problems, as we shall see presently.
By way of notation, let M n denote the set of all (n x n) matrices and let (!)
denote the (n x n) zero matrix. Then, a matrix norm for M n is a real-valued function
11'11 defined on M n satisfying for all (n x It) matrices and A and B:
Ilx
a)
b)
c)
d)
xIII,
IIAII ~ 0 and IIAII = 0 if and only if A = (!)
IIIXAII = !Cl.IIIAIl for any scalar CI.
IIA + BII ~ II A II + IIBII
IIABII ~ IIAIIIIBII.
(2.40)
The addition of condition d) should be noted. Thus matrix norms have a triangle
inequality for both the operation of addition and multiplication.
Just as there are numerous ways of defining specific vector norms, there are
also numerous ways of defining specific matrix norms. We will concentrate on
three that are easily computable and are intrinsically related to the three basic
vector norms discussed in the previous section. Specifically, if A = (a i ) E M n ,
we define
IIAII
I
= Max
1:S;):S;n
IIAII. '"
IIAIIE =
[t laijl]
-(Maximum absolute column sum)
i= I
l'!~<'. [~ la'll] -(M,,;mum absolute
n
'Ow
sum)
(2.41)
n
LL
(ay.
i= I j= I
Example 2.13.
Let
A
then
IIAII,
=
Max{l, 2, 25, 3) = 25,
~ l~ ~ l~
IIAII""
=
]
Max{lO, 8, 7, 6}
lO, and
IIAIIE
JiSt
= .
~
13.454.
These three norms are stressed here because they are compatible with the
t'1' t' ,,,,, and t 2 vector norms, respectively. This means that given any matrix
42
SOLUTION OF LINEAR SYSTEMS OF EOUATIONS
A E M n , then it is true for all vectors x E R", that
(The reader should be careful to distinguish between vector norms and matrix
norms since they bear the same subscript notation in two of the three cases. This
distinction is clear from the usage. For example, Ax is a vector so IIAxll1 denotes
the use of the 11'11 I vector norm, whereas IIAIII denotes use of the matrix norm.
The reader should also note that the pairs of matrix and vector norms are com­
patible only in the orders given by (2.42) and cannot be mixed. For example,
let A be given as in the above example and let x = (0,0,1, O)T. Then Ilxlll = 1,
but IIAxll1 = 25 and IIAII001lxl1 1 = 10. That is, we cannot expect to have the
inequality IIAx!/l ~ IIAllool/xll oo or IIAxll, ~ IIAIIoollxlid
Compatibility is a property that connects vector norms and matrix norms
together. For example, in (2.34) we had (xc - x.) = A -If. Using the idea of
compatible vector and matrix norms, we can estimate Ilx c - x,iii in terms of
IIA - 1111 and IIrll l :
As we shall see in Section 2.4, compatibility is also crucial to a clear understanding
of iterative methods.
Thus far we have not shown that the three matrix norms satisfy the com­
patibility properties, (2.42), or even that their definitions given by (2.41) satisfy
the necessary norm properties given by (2.40). For the sake of brevity we shall
supply only the necessary proofs for the 11'111 norm, leaving the iI'lIoo and II·IIE
norms to the reader.
Let A = (ai) E iIIl n and write A in terms of its column vectors, A = [A I,
A2, ... , Anl Let x = (x" X2, ... ,Xn)T be any vector in Rn, and recall ,from
Section 2.1 that Ax may be written as Ax = x,A, + X2AZ + ... + xnA n. Since
Ax is an (n x 1) vector, we use (2.37) to get
+ X2A2 + .,. + xnAnll 1
~ IIx1AIII1 + II x 2A2111 + .,. + IlxnAnll l
= IxllllAIIL + IX2111A2111 + ... + IxnlilAnl1 1
IIAxll1 = II x ,A,
~ (ixt\ + IX21 + ,., + IXnj)C~ja5~,IIAjlll) ==
(2.43)
IIAlldlxl/I'
Thus we have shown compatibility for the 11'111 norm.
It is trivial to see that parts (a) and (b) of (2.40) hold for 1I·lk Since the ith
column of A + B is precisely the ith column of A plus the ith column of B, part
(c) of (2.40) is also easily seen to be true. For part (d) of (2.40), we recall from Section
2.1 that the ith column of AB equals AB i , where Bi is the ith column of B. By
compatibility, IIABdl1 ~ IIAIIIIIBdll' Now choose i such that IIBdil ~ IIBjll, for
2.3 ERROR ANALYSIS AND NORMS
43
~ j ~ n. Then IIBII I = IIBdl to and
IIABII
I
= max IIABjll 1 ~ max I/AIlII/Bjl/ 1 = IIAIIII/B,III = IIAIIIIIBIII>
isjS"
lS]511
and thus part (d) of (2.40) is satisfied. We have therefore shown that IIAII I =
maxlsj:SIID~1 laul is a matrix norm and that it is compatible with the 11·111
vector norm.
We conclude this material with an observation on matrix norms, considering
the three numbers K p ; P = 1,2, 00; where if A E Mil is given, then
Kp
= inf{K E R I : IIAxllp
~ Kllxllp, for all x
E
R"}
(2.44)
(where "inf" denotes infimum or greatest lower bound). It can be shown that K I =
IIAII I and K"" = IIAlloo, although it goes beyond our purposes to do so. We intro­
duce these numbers, however, since K z i= IIAlb and this explains why we use
the subscript "E" instead of "2." Thus, although we have IIAxl/ z ~ IIAIIEllxl12
(compatibility), there is a matrix norm K 2 == I/AIIz, smaller for most matrices
than IIAIIE' such that IIAxl12 ~ IIAI/zllxllz, for all x E R". I/Aliz is rather unwieldly
in computations and involves some deeper theory to derive, and so we are satisfied
10 use the easily computable and compatible IIAII£ in place of IIAI12 (see Theorem
3.11 ).
2.3.3 Condition Numbers and Error Estimates
In this section, we use the ideas of matrix and vector norms to provide some more
information that is useful in helping to determine how good a computed (approx­
imate) solution to the system Ax = b is. The norms used below can be any pair
of compatible matrix and vector norms; for convenience we have omitted the
sUbscripts. The reader can tell from the context whether a particular norm is a
matrix norm or a vector norm. The following theorem provides valuable infor­
mation with respect to the relative error.
Theorem 2.1. Suppose A E Mil is non-singular and Xe is an approximation to XI>
the exact solution of Ax = b, where b i= 9. Then for any compatible matrix and
vector norms
Proof:
Again by (2.34),
Xc -
XI
=
A -I r, where r
= AXe -
patibility conditions, (2.42), IIxe - XIII ~ IIA-Illlirl/
=
b. Thus by the com­
I/A-11I1IAx e - bll. Now,
AX I = b, so IIAllllx,l1 ~ Ilbll, and IIAll/llbl1 ~ l/llx,lI. Thus,
!Ix, - XIII < IIA -IIIIIAx - bll ~
Ilxlll
'1Ibll'
44
SOLUTION OF LINEAR SYSTEMS OF EQUATIONS
establishing the right-hand side of (2.45). Now,
IIAx, - bll
and, since
IIAII
IIrll
=
IIAx, - Ax,II ::;; IIAllllx, - xIII,
=
> 0,
Ilx, - x,11 2 IIAxe - bll/IIAII·
!lxlll : ; IIA -lllllbll, or 1/llx,11 2 l/IIA -I!I!lbll
Also, x, = A -lb, so
Combining
these last two inequalities establishes the left-hand side of (2.45). •
Note the appearance of the term IIAII!lA - III in both the upper and lower
bounds for the relative error. This term is usually called the condition number
and its size is a measure of how good a solution we can expect direct methods to
generate. It is easy to show (Problem 8) that the condition number satisfies the
inequality: IIAIIIIA -III 2 1. The number IIA -III is valuable not only in terms of
the condition number, but also in the "primitive" error estimate of (2.34), which
we rephrase using compatible norms as
(2.46)
Since we can compute IIrll, any information about IIA - 111 can be utilized, but as
mentioned above, A-I is not normally available. A way to get a lower estimate
for IIA- 111 is provided by noting that for any X in R",
!lx!l
=
IIA -I Axil::;; IIA -IIIIIAxll
(2.47)
and thus
~::;;IIA-III.
II Axil
Example 2.14.
(2.48)
Consider the linear system
6x 1
lOx I
6x I
+
+
+
6x l
8Xl
4x l
+
+
+
3.00001xJ = 30.00002
4.00003xJ = 42.00006
2.00002xJ = 22.00004
which has the unique solution XI = 1, Xl = 3, XJ = 2. This example serves to illustrate the
notion of an "ill-conditioned" system as well as serving as an example of how we can use matrix
norms to analyze the results of a computational solution. (Recall: A system of linear equations
is called ill-conditioned if "small" changes in the coefficients produce "large" changes in the
solution.) Before solving this system, note that the perturbed system below
6x 1
10x i
6xj
+
+
+
6Xl
8Xl
4x l
+
+
+
3.00001 XJ = 30
4.00003xJ = 42
2.00002xJ = 22
has a unique solution Xl = I, Xl = 4, xJ = 0, which is substantially different from the solution
of the first system, even though the coefficient matrices are the same and the constants on the
right-hand side are the same through six significant figures. Therefore we call these two systems
ill-conditioned.
2.3 ERROR ANALYSIS AND NORMS
45
The first system also demonstrates that we should be cautious about how we determine
whether an estimate to a solution is a good estimate or not. If we try Xl = 1, x 2 = 4, and
XJ = 0 in the first sys.tem, we obtain the residual vector
-2. x 10r =
5
]
-6. x 10- 5
[
-4. X 10- 5
whereas the act ual error in lIsing this est imate is on the oreler of 105 times as large as the residual
vector would indicate. By (2.34) it follows that the inverse of this coefficient matrix has large
entries.
To see what happens when Gauss elimination is used to solve the first system of equations,
a single precision Gauss elimination routine was employed, and found
XI]
X2
[x
=
J
[0.9999898]
1.907699 ,
4.184615
0.000019]
0.000076 ,
r =
[
0.000010]
1.092301
x, - x, =
[
0.000049
- 2.18461 5
These results are typical if this system is solved on any digital device which has six to eight
place accuracy. A double precision computation on a computer would give almost exactly
the correct answer, but going to double precision is obviOUSly not a cure-all, for there are
simple examples like the one above for which double-precision arithmetic is not sufficient.
While this example is somewhat contrived (so that the essence of the problem
is not hidden in a mass of cumbersome calculations), there are many real-life
problems which are ill-conditioned, particularly in areas such as statistical analysis
and least-squares fits. Thus it is appropriate that a person who must deal with
numerical solutions of linear equations have as many tools at hand as possible
in order to test computed results for accuracy. Often a very reliable test is simply
a feeling for what the computed results are supposed 10 represent physically, i.e.,
do the answers fit the physical problem? In the absence of a physical intuition
for what the answer should be, or in terms of a tool for a mathematical analysis
of the significance of the computed results, the estimates of Theorem 2.1, (2.46),
and (2.48) provide a beginning.
We continue now with Example 2.14 to illustrate how the 11·111 and 11·1j" norms can be
readily used to analyze errors in computed solutions. Thus Ax = b has x, as its solution
vector, where
A
=
[
6
LO
6
8
3.00001]
4.00003 ,
6
4
2.00002
30.00002]
4200006 ,
b =
[
(2.49)
22.00004
In order not to obscure the point of this example, let us suppose rhat a computed estimate
to the solution, x" and hence the residual r = Ax, - b, are given by
and
r
-0.00002]
-0.00006
=
[
-0.00004
(2.50)