AMS526: Numerical Analysis I
(Numerical Linear Algebra)
Lecture 7: Sensitivity of Linear Systems
Xiangmin Jiao
Stony Brook University
Xiangmin Jiao
Numerical Analysis I
1 / 18
Outline
1
Condition Number of a Matrix
2
Perturbing Right-hand Side
3
Perturbing Coefficient Matrix
4
Putting All Together
Xiangmin Jiao
Numerical Analysis I
2 / 18
Condition Number of Matrix
Consider f (x) = Ax, with A ∈ Rm×n
κ=
kJk
kAkkxk
=
kf (x)k/kxk
kAxk
If A is square and nonsingular, since kxk/kAxk ≤ kA−1 k
κ ≤ kAkkA−1 k
We define condition number of matrix A as
κ(A) = kAkkA−1 k
It is the upper bound of the condition number of f (x) = Ax for any x
For any induced matrix norm, κ(I ) = 1 and κ(A) ≥ 1
Note about the distinction between the condition number of a problem
(the map f (x)) and the condition number of a problem instance (the
evaluation of f (x) for specific x)
Xiangmin Jiao
Numerical Analysis I
3 / 18
Geometric Interpretation of Condition Number
Another way to interpret at κ(A) is
κ(A) = sup
δx,x
kδf k/kδxk
supδx kAδxk/kδxk
=
kf (x)k/kxk
inf x kAxk/kxk
Question: For what x and δx is the equality achieved?
Xiangmin Jiao
Numerical Analysis I
4 / 18
Geometric Interpretation of Condition Number
Another way to interpret at κ(A) is
κ(A) = sup
δx,x
kδf k/kδxk
supδx kAδxk/kδxk
=
kf (x)k/kxk
inf x kAxk/kxk
Question: For what x and δx is the equality achieved?
Answer: When x is in direction of minimum magnification, and δx is
in direction of maximum magnification
Define maximum magnification of A as
maxmag(A) = max kAxk
kxk=1
and minimum magnification of A as
minmag(A) = min kAxk
kxk=1
Then condition number of matrix is κ(A) = maxmag(A)/minmag(A)
For 2-norm, κ(A) = σ1 /σn , the ratio of largest and smallest singular
values (in later sections)
Xiangmin Jiao
Numerical Analysis I
4 / 18
Example of Ill-Conditioned Matrix
Example
1000 999
Let A =
. It is easy to verify that
999 998
−998
999
−1
A =
. So
999 −1000
κ∞ (A) = κ1 (A) = 19992 = 3.996 × 106 .
Xiangmin Jiao
Numerical Analysis I
5 / 18
Example of Ill-Conditioned Matrix
Example
A famous example is Hilbert matrix, defined by hij = 1/(i + j − 1),
1 ≤ i, j ≤ n. The matrix is ill-conditioned for even quite small n.
For n ≤ 4, we have


1 1/2 1/3 1/4
 1/2 1/3 1/4 1/5 

H4 = 
 1/3 1/4 1/5 1/6  ,
1/4 1/5 1/6 1/7
with condition number κ2 (H4 ) ≈ 1.6 × 104 , and κ2 (H8 ) ≈ 1.5 × 1010 .
Xiangmin Jiao
Numerical Analysis I
6 / 18
Outline
1
Condition Number of a Matrix
2
Perturbing Right-hand Side
3
Perturbing Coefficient Matrix
4
Putting All Together
Xiangmin Jiao
Numerical Analysis I
7 / 18
Condition Number of Linear System
What is the condition number for f (b) = A−1 b?
Xiangmin Jiao
Numerical Analysis I
8 / 18
Condition Number of Linear System
What is the condition number for f (b) = A−1 b?
Answer: κ ≤ κ(A) ≡ kAkkA−1 k, as in matrix-vector multiplication
Theorem
Let A be nonsingular, and let x and x̂ = x + δx be the solutions of Ax = b
and Ax̂ = b + δb, respectively. Then
kδbk
kδxk
≤ κ(A)
,
kxk
kbk
and there exists kbk and kδbk for which the equality holds.
Question: For what b and δb is the equality achieved?
Xiangmin Jiao
Numerical Analysis I
8 / 18
Condition Number of Linear System
What is the condition number for f (b) = A−1 b?
Answer: κ ≤ κ(A) ≡ kAkkA−1 k, as in matrix-vector multiplication
Theorem
Let A be nonsingular, and let x and x̂ = x + δx be the solutions of Ax = b
and Ax̂ = b + δb, respectively. Then
kδbk
kδxk
≤ κ(A)
,
kxk
kbk
and there exists kbk and kδbk for which the equality holds.
Question: For what b and δb is the equality achieved?
Answer: When b is in direction of minimum magnification of A−1 , and
δb is in direction of maximum magnification of A−1 .
In 2-norm, when b is in direction of maximum magnification of AT ,
and δb is in direction of minimum magnification of AT .
Xiangmin Jiao
Numerical Analysis I
8 / 18
Singular and Nearly Singular Linear System
Question: What is condition number of Ax if A is singular?
Xiangmin Jiao
Numerical Analysis I
9 / 18
Singular and Nearly Singular Linear System
Question: What is condition number of Ax if A is singular?
Answer: ∞.
We say a matrix is nearly singular if its condition number is very large
In other words, columns of A are nearly linearly dependent
If A is nearly singular, for matrix-vector multiplication, Ax, error is
large if x is nearly in null space of A
If A is nearly singular, for linear system Ax = b, error is large if b is
NOT nearly in null space of AT
Therefore, ill-conditioning (near singularity) has a much bigger impact
on solving linear system than matrix-vector multiplication!
Xiangmin Jiao
Numerical Analysis I
9 / 18
Ill Conditioning Caused by Poor Scaling
Some matrices are ill conditioned simply because they are out of scale.
Theorem
Let A ∈ Rn×n be any nonsingular matrix, and let ak , 1 ≤ k ≤ n denote the
kth column of A. Then for any i and j with 1 ≤ i, j, ≤ n,
κp (A) ≥ kai kp /kaj kp .
This theorem indicates that poor scaling inevitably leads to ill
conditioning
A necessary condition for a matrix to be well conditioned is that all of
its rows and columns are of roughly the same magnitude.
Xiangmin Jiao
Numerical Analysis I
10 / 18
Estimating Condition Number
We would like to estimate κ1 (A) = kAk1 kA−1 k1 without computing
A−1 , but allow LU factorization of A
For any vector w ∈ Rn and kw k1 = 1, we have lower bound
κ1 (A) ≥ kAk1 kA−1 w k1
If w has a significant component in direction near maximum
magnification by A−1 , then
κ1 (A) ≈ kAk1 kA−1 w k1
Note statement on p. 132 of textbook “Actually any w chosen at
random is likely to have a significant component in the direction of
maximum magnification by A−1 ” is unjustified for large n in 1-norm
Good estimators conduct systematic searches for w that approximately
maximizes kA−1 w k1
Xiangmin Jiao
Numerical Analysis I
11 / 18
Outline
1
Condition Number of a Matrix
2
Perturbing Right-hand Side
3
Perturbing Coefficient Matrix
4
Putting All Together
Xiangmin Jiao
Numerical Analysis I
12 / 18
Non-singularity of Perturbed Matrix
Theorem
If A is nonsingular and
kδAk/kAk < 1/κ(A),
then A + δA is nonsingular.
Xiangmin Jiao
Numerical Analysis I
13 / 18
Non-singularity of Perturbed Matrix
Theorem
If A is nonsingular and
kδAk/kAk < 1/κ(A),
then A + δA is nonsingular.
Proof.
kδAk/kAk < 1/κ(A) is equivalent to kδAkkA−1 k < 1. Suppose A + δA is
singular, then ∃y 6= 0 such that (A + δA)y = 0, and y = −A−1 δAy .
Therefore, ky k ≤ kA−1 kkδAkky k, or kA−1 kkδAk ≥ 1.
If A + δA is the singular matrix closest to A, in the sense that kδAk2 is
as small as possible, then kδAk2 /kAk2 = 1/κ2 (A)
Xiangmin Jiao
Numerical Analysis I
13 / 18
Linear System with Perturbed Matrix
Suppose Ax = b and Âx̂ = b where  = A + δA. Let δx = x̂ − x and
x̂ = x + δx.
We would like to bound kδxk/kxk, but first we bound kδxk/kx̂k
Theorem
If A is nonsingular, and let b 6= 0. Then
kδAk
kδxk
≤ κ(A)
.
kx̂k
kAk
Proof.
Rewrite (A + δA)x̂ = b as Ax + Aδx + δAx̂ = b, where Ax = b. Therefore,
kδxk ≤ kA−1 kkδAkkx̂k.
Therefore,
kδxk
kδAk
≤ kA−1 kkδAk = κ(A)
.
kx̂k
kAk
Xiangmin Jiao
Numerical Analysis I
14 / 18
Linear System with Perturbed Matrix Continued
Ax = b and Âx̂ = b where  = A + δA. Let δx = x̂ − x and
x̂ = x + δx.
Theorem
If A is nonsingular and kδAk/kAk < 1/κ(A), and let b 6= 0. Then
κ(A)kδAk/kAk
kδxk
≤
.
kxk
1 − κ(A)kδAk/kAk
Proof.
kδxk ≤ kA−1 kkδAkkx̂k ≤ kA−1 kkδAk(kxk + kδxk). Therefore,
1 − kA−1 kkδAk δx ≤ kA−1 kkδAkkxk,
where kA−1 kkδAk = κ(A)kδAk/kAk.
We typically expect κ(A)kδAk kAk, so the denominator is close to 1.
Xiangmin Jiao
Numerical Analysis I
15 / 18
Outline
1
Condition Number of a Matrix
2
Perturbing Right-hand Side
3
Perturbing Coefficient Matrix
4
Putting All Together
Xiangmin Jiao
Numerical Analysis I
16 / 18
Perturbed RHS and Matrix
Suppose Ax = b and (A + δA)(x + δx) = (b + δb), where
 = A + δA, b = b + δb and x̂ = x + δx.
Theorem
Let A be nonsingular, and suppose x̂ 6= 0 and b̂ 6= 0. Then
!
!
kδxk
kδAk kδbk kδAk kδbk
kδAk kδbk
≤ κ(A)
+
+
≈ κ(A)
+
.
kx̂k
kAk
kAk kb̂k
kAk
kb̂k
kb̂k
Theorem
If A is nonsingular and kδAk/kAk < 1/κ(A), and let b 6= 0, then
kδxk
κ(A)(kδAk/kAk + kδbk/kbk)
.
.
kxk
1 − κ(A)kδAk/kAk
Roughly speaking, κ(A) determines loss of digits of accuracy in x in
addition to loss of digits of accuracy in perturbations in A and b
Xiangmin Jiao
Numerical Analysis I
17 / 18
A Posteriori Error Analysis Using Residual
Suppose x̂ is a computed solution of Ax = b, and residual r̂ = b − Ax̂.
How to bound error in x − x̂?
Theorem
Let A be nonsingular, let b 6= 0. Then
kδxk
kr̂ k
≤ κ(A)
.
kxk
kbk
If the residual is tiny and A is well conditioned, then x̂ is an accurate
approximation to x.
For a posteriori error bound, one needs to estimate kr̂ k and κ(A)
Xiangmin Jiao
Numerical Analysis I
18 / 18