Chapter 4: Rank Deciency and Ill-Conditioning
Math 4/896: Seminar in Mathematics
Topic: Inverse Theory
Instructor: Thomas Shores
Department of Mathematics
AvH 10
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Chapter 4: Rank Deciency and Ill-Conditioning
Outline
1
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Chapter 4: Rank Deciency and Ill-Conditioning
Basic Theory of SVD
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Theorem
(Singular Value Decomposition) Let G be an m
Then there exist m
×m
orthogonal matrix U , n
× n real matrix.
× n orthogonal
× n diagonal matrix S with diagonal entries
σ1 ≥ σ2 ≥ . . . ≥ σq , with q = min{m, n}, such that U T GV = S .
Moreover, numbers σ1 , σ2 , . . . , σq are uniquely determined by G .
matrix V and m
Denition
p
p
With notation as in the SVD Theorem, and U , V
the matrices
consisting of the rst p columns of U , V , respectively, and S
rst p rows and columns of S , where
σp
p
the
is the last nonzero singular
value, then the Moore-Penrose pseudoinverse of G is
G
†
= Vp Sp−1 UpT ≡
p
X
j =1
1
σj
T
Vj Uj .
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Matlab Knows It
Carry out these calculations in Matlab:
> n = 6
> G = hilb(n);
> svd(G)
>[U,S,V] = svd(G);
>U'*G*V - S
>[U,S,V] = svd(G,'econ');
> % try again with n=16 and then G=G(1:8)
> % what are the nonzero singular values of G?
Chapter 4: Rank Deciency and Ill-Conditioning
Applications of the SVD
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Use notation above and recall that the null space and column space
(range) of matrix G are N (G )
R (G )
= {y ∈ R m | y = G x ,
Theorem
(1) rank (G )
=p
and A
=
p
X
j =1
= {x ∈ R n | G x = 0 }
x
and
∈ Rn } = span {G1 , G2 , . . . , Gn }
σj Uj VjT
= span {Vp+1 , Vp+2 , . . . , Vn },R (G ) =
span {V1 , V2 , . . . , Vp }
(2)N (G )
(3)N
G
T = span {U , U , . . . , U },R (G ) =
p +1 p +2
m
p}
span {U1 , U2 , . . . , U
(4) m†
= G †d
is the least squares solution to G m
=d
of minimum
2-norm.
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Heart of the Diculties with Least Squares Solutions
Use the previous notation, so that G is m
×n
with rank p and
SVD, etc as above. By data space we mean the vector space
and by model space we mean
Rn .
Rm
No Rank Deciency:
This means that p
= m = n.
Comments:
This means that null space of both G and G
T
are trivial (both
{0}).
Then there is a perfect correspondence between vectors in
data space and model space:
Gm
= d,
m
= G − 1 d = G † d.
This is the ideal. But are we out of the woods?
No, we still have to deal with data error and ill-conditioning of
the coecient matrix (remember Hilbert?).
Heart of the Diculties with Least Squares Solutions
Use the previous notation, so that G is m
×n
with rank p and
SVD, etc as above. By data space we mean the vector space
and by model space we mean
Rn .
Rm
No Rank Deciency:
This means that p
= m = n.
Comments:
This means that null space of both G and G
T
are trivial (both
{0}).
Then there is a perfect correspondence between vectors in
data space and model space:
Gm
= d,
m
= G − 1 d = G † d.
This is the ideal. But are we out of the woods?
No, we still have to deal with data error and ill-conditioning of
the coecient matrix (remember Hilbert?).
Heart of the Diculties with Least Squares Solutions
Use the previous notation, so that G is m
×n
with rank p and
SVD, etc as above. By data space we mean the vector space
and by model space we mean
Rn .
Rm
No Rank Deciency:
This means that p
= m = n.
Comments:
This means that null space of both G and G
T
are trivial (both
{0}).
Then there is a perfect correspondence between vectors in
data space and model space:
Gm
= d,
m
= G − 1 d = G † d.
This is the ideal. But are we out of the woods?
No, we still have to deal with data error and ill-conditioning of
the coecient matrix (remember Hilbert?).
Heart of the Diculties with Least Squares Solutions
Use the previous notation, so that G is m
×n
with rank p and
SVD, etc as above. By data space we mean the vector space
and by model space we mean
Rn .
Rm
No Rank Deciency:
This means that p
= m = n.
Comments:
This means that null space of both G and G
T
are trivial (both
{0}).
Then there is a perfect correspondence between vectors in
data space and model space:
Gm
= d,
m
= G − 1 d = G † d.
This is the ideal. But are we out of the woods?
No, we still have to deal with data error and ill-conditioning of
the coecient matrix (remember Hilbert?).
Heart of the Diculties with Least Squares Solutions
Use the previous notation, so that G is m
×n
with rank p and
SVD, etc as above. By data space we mean the vector space
and by model space we mean
Rn .
Rm
No Rank Deciency:
This means that p
= m = n.
Comments:
This means that null space of both G and G
T
are trivial (both
{0}).
Then there is a perfect correspondence between vectors in
data space and model space:
Gm
= d,
m
= G − 1 d = G † d.
This is the ideal. But are we out of the woods?
No, we still have to deal with data error and ill-conditioning of
the coecient matrix (remember Hilbert?).
Heart of the Diculties with Least Squares Solutions
Use the notation m†
= G † d.
Row Rank Deciency:
This means that d
= n < m.
Comments:
This means that null space of G is trivial, but that of G
T
is
not.
Here m† is the unique least squares solution.
And m† is the exact solution to G m
=d
exactly if d is in the
range of G .
But m is insensitive to any translation d
d0
∈N
G†
+ d0
with
Heart of the Diculties with Least Squares Solutions
Use the notation m†
= G † d.
Row Rank Deciency:
This means that d
= n < m.
Comments:
This means that null space of G is trivial, but that of G
T
is
not.
Here m† is the unique least squares solution.
And m† is the exact solution to G m
=d
exactly if d is in the
range of G .
But m is insensitive to any translation d
d0
∈N
G†
+ d0
with
Heart of the Diculties with Least Squares Solutions
Use the notation m†
= G † d.
Row Rank Deciency:
This means that d
= n < m.
Comments:
This means that null space of G is trivial, but that of G
T
is
not.
Here m† is the unique least squares solution.
And m† is the exact solution to G m
=d
exactly if d is in the
range of G .
But m is insensitive to any translation d
d0
∈N
G†
+ d0
with
Heart of the Diculties with Least Squares Solutions
Use the notation m†
= G † d.
Row Rank Deciency:
This means that d
= n < m.
Comments:
This means that null space of G is trivial, but that of G
T
is
not.
Here m† is the unique least squares solution.
And m† is the exact solution to G m
=d
exactly if d is in the
range of G .
But m is insensitive to any translation d
d0
∈N
G†
+ d0
with
Heart of the Diculties with Least Squares Solutions
Use the notation m†
= G † d.
Row Rank Deciency:
This means that d
= n < m.
Comments:
This means that null space of G is trivial, but that of G
T
is
not.
Here m† is the unique least squares solution.
And m† is the exact solution to G m
=d
exactly if d is in the
range of G .
But m is insensitive to any translation d
d0
∈N
G†
+ d0
with
Heart of the Diculties with Least Squares Solutions
Column Rank Deciency:
This means p
= m < n.
Comments:
This means that null space of G
T
is trivial, but that of G is
not.
Here m† is a solution of minimum 2-norm.
And m†
m0
+ m0
∈ N (G ).
is also a solution to G m
=d
So d is insensitive to any translation m†
m0
∈ N (G ).
for any
+ m0
with
Heart of the Diculties with Least Squares Solutions
Column Rank Deciency:
This means p
= m < n.
Comments:
This means that null space of G
T
is trivial, but that of G is
not.
Here m† is a solution of minimum 2-norm.
And m†
m0
+ m0
∈ N (G ).
is also a solution to G m
=d
So d is insensitive to any translation m†
m0
∈ N (G ).
for any
+ m0
with
Heart of the Diculties with Least Squares Solutions
Column Rank Deciency:
This means p
= m < n.
Comments:
This means that null space of G
T
is trivial, but that of G is
not.
Here m† is a solution of minimum 2-norm.
And m†
m0
+ m0
∈ N (G ).
is also a solution to G m
=d
So d is insensitive to any translation m†
m0
∈ N (G ).
for any
+ m0
with
Heart of the Diculties with Least Squares Solutions
Column Rank Deciency:
This means p
= m < n.
Comments:
This means that null space of G
T
is trivial, but that of G is
not.
Here m† is a solution of minimum 2-norm.
And m†
m0
+ m0
∈ N (G ).
is also a solution to G m
=d
So d is insensitive to any translation m†
m0
∈ N (G ).
for any
+ m0
with
Heart of the Diculties with Least Squares Solutions
Column Rank Deciency:
This means p
= m < n.
Comments:
This means that null space of G
T
is trivial, but that of G is
not.
Here m† is a solution of minimum 2-norm.
And m†
m0
+ m0
∈ N (G ).
is also a solution to G m
=d
So d is insensitive to any translation m†
m0
∈ N (G ).
for any
+ m0
with
Heart of the Diculties with Least Squares Solutions
Row and Column Rank Deciency:
This means p
< min {m, n}.
Comments:
This means that null space of both G and G
Here m† is a least squares solution.
We have trouble in both directions.
T
are nontrivial.
Heart of the Diculties with Least Squares Solutions
Row and Column Rank Deciency:
This means p
< min {m, n}.
Comments:
This means that null space of both G and G
Here m† is a least squares solution.
We have trouble in both directions.
T
are nontrivial.
Heart of the Diculties with Least Squares Solutions
Row and Column Rank Deciency:
This means p
< min {m, n}.
Comments:
This means that null space of both G and G
Here m† is a least squares solution.
We have trouble in both directions.
T
are nontrivial.
Heart of the Diculties with Least Squares Solutions
Row and Column Rank Deciency:
This means p
< min {m, n}.
Comments:
This means that null space of both G and G
Here m† is a least squares solution.
We have trouble in both directions.
T
are nontrivial.
Chapter 4: Rank Deciency and Ill-Conditioning
Outline
1
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Covariance and Resolution
Denition
The model resolution matrix for the problem G m
Rm
=
=d
is
G †G
Consequences:
Rm
= Vp VpT ,
If G m
which is just I
n
if G has full column rank.
true = d, then E [m† ] = Rm mtrue
Thus, the bias in the gereralized inverse solution is
E [m† ] − mtrue
V
= [Vp V0 ].
= (Rm − I ) mtrue = −V0 V0T mtrue
with
Similarly, in the case of identically distributed data with
σ 2 , the covariance matrix is
Vi ViT
2 † G † T = σ 2 Pp
Cov (m† ) = σ G
i =1 s 2 .
variance
i
From expected values we obtain a resolution test: if a
diagonal entry are close to 1, we claim good resolution of that
coordinate, otherwise not.
Chapter 4: Rank Deciency and Ill-Conditioning
Outline
1
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Instability of Generalized Inverse Solution
The key results:
× n square matrix
G
−1 cond2 (G ) = kG k2 G
= σ1 /σn .
2
For n
This inspires the denition: the condition number of an m
matrix G is
Note: if
σ1 /σq
σq = 0,
where q
×n
= min {m, n}.
the condition number is innity. Is this notion
useful?
0
If data d vector is perturbed to d , resulting in a perturbation
0
of the generalized inverse solution m† to m† , then
km0† −m† k2
kd0 −dk2
≤
cond (G )
k dk 2 .
km† k2
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Instability of Generalized Inverse Solution
The key results:
× n square matrix
G
−1 cond2 (G ) = kG k2 G
= σ1 /σn .
2
For n
This inspires the denition: the condition number of an m
matrix G is
Note: if
σ1 /σq
σq = 0,
where q
×n
= min {m, n}.
the condition number is innity. Is this notion
useful?
0
If data d vector is perturbed to d , resulting in a perturbation
0
of the generalized inverse solution m† to m† , then
km0† −m† k2
kd0 −dk2
≤
cond (G )
k dk 2 .
km† k2
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Instability of Generalized Inverse Solution
The key results:
× n square matrix
G
−1 cond2 (G ) = kG k2 G
= σ1 /σn .
2
For n
This inspires the denition: the condition number of an m
matrix G is
Note: if
σ1 /σq
σq = 0,
where q
×n
= min {m, n}.
the condition number is innity. Is this notion
useful?
0
If data d vector is perturbed to d , resulting in a perturbation
0
of the generalized inverse solution m† to m† , then
km0† −m† k2
kd0 −dk2
≤
cond (G )
k dk 2 .
km† k2
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Instability of Generalized Inverse Solution
The key results:
× n square matrix
G
−1 cond2 (G ) = kG k2 G
= σ1 /σn .
2
For n
This inspires the denition: the condition number of an m
matrix G is
Note: if
σ1 /σq
σq = 0,
where q
×n
= min {m, n}.
the condition number is innity. Is this notion
useful?
0
If data d vector is perturbed to d , resulting in a perturbation
0
of the generalized inverse solution m† to m† , then
km0† −m† k2
kd0 −dk2
≤
cond (G )
k dk 2 .
km† k2
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Properties of the SVD
Covariance and Resolution of the Generalized Inverse Solution
Instability of Generalized Inverse Solutions
Chapter 4: Rank Deciency and Ill-Conditioning
Instability of Generalized Inverse Solution
The key results:
× n square matrix
G
−1 cond2 (G ) = kG k2 G
= σ1 /σn .
2
For n
This inspires the denition: the condition number of an m
matrix G is
Note: if
σ1 /σq
σq = 0,
where q
×n
= min {m, n}.
the condition number is innity. Is this notion
useful?
0
If data d vector is perturbed to d , resulting in a perturbation
0
of the generalized inverse solution m† to m† , then
km0† −m† k2
kd0 −dk2
≤
cond (G )
k dk 2 .
km† k2
Instructor: Thomas Shores Department of Mathematics
Math 4/896: Seminar in Mathematics Topic: Inverse Theo
Stability Issues
How these facts aect stability:
If cond (G ) is not too large, then the solution is stable to
perturbations in data.
If
σ1 σp ,
there is a potential for instability. It is diminished
if the data itself has small components in the direction of
singular vectors corresponding to small singular values.
If
σ1 σp ,
and there is a clear delineation between small
singular values and the rest, we simple discard the small
singular values and treat the problem as one of smaller rank
with good singular values.
If
σ1 σp ,
and there is no clear delineation between small
singular values and the rest, we have to discard some of them,
but which ones? This leads to regularization issues. In any
case, any method that discards small singular values produces
a truncated SVD (TSVD) solution.
Stability Issues
How these facts aect stability:
If cond (G ) is not too large, then the solution is stable to
perturbations in data.
If
σ1 σp ,
there is a potential for instability. It is diminished
if the data itself has small components in the direction of
singular vectors corresponding to small singular values.
If
σ1 σp ,
and there is a clear delineation between small
singular values and the rest, we simple discard the small
singular values and treat the problem as one of smaller rank
with good singular values.
If
σ1 σp ,
and there is no clear delineation between small
singular values and the rest, we have to discard some of them,
but which ones? This leads to regularization issues. In any
case, any method that discards small singular values produces
a truncated SVD (TSVD) solution.
Stability Issues
How these facts aect stability:
If cond (G ) is not too large, then the solution is stable to
perturbations in data.
If
σ1 σp ,
there is a potential for instability. It is diminished
if the data itself has small components in the direction of
singular vectors corresponding to small singular values.
If
σ1 σp ,
and there is a clear delineation between small
singular values and the rest, we simple discard the small
singular values and treat the problem as one of smaller rank
with good singular values.
If
σ1 σp ,
and there is no clear delineation between small
singular values and the rest, we have to discard some of them,
but which ones? This leads to regularization issues. In any
case, any method that discards small singular values produces
a truncated SVD (TSVD) solution.
Stability Issues
How these facts aect stability:
If cond (G ) is not too large, then the solution is stable to
perturbations in data.
If
σ1 σp ,
there is a potential for instability. It is diminished
if the data itself has small components in the direction of
singular vectors corresponding to small singular values.
If
σ1 σp ,
and there is a clear delineation between small
singular values and the rest, we simple discard the small
singular values and treat the problem as one of smaller rank
with good singular values.
If
σ1 σp ,
and there is no clear delineation between small
singular values and the rest, we have to discard some of them,
but which ones? This leads to regularization issues. In any
case, any method that discards small singular values produces
a truncated SVD (TSVD) solution.
Stability Issues
How these facts aect stability:
If cond (G ) is not too large, then the solution is stable to
perturbations in data.
If
σ1 σp ,
there is a potential for instability. It is diminished
if the data itself has small components in the direction of
singular vectors corresponding to small singular values.
If
σ1 σp ,
and there is a clear delineation between small
singular values and the rest, we simple discard the small
singular values and treat the problem as one of smaller rank
with good singular values.
If
σ1 σp ,
and there is no clear delineation between small
singular values and the rest, we have to discard some of them,
but which ones? This leads to regularization issues. In any
case, any method that discards small singular values produces
a truncated SVD (TSVD) solution.