Rank-Deficient Least Squares
Problem
Βαγγέλης Δούρος
EY0619
1
Mini Revision
2
First Remarks (1)
z
z
z
z
3
Problem Statement: Given A ∈ R ,m>=n,
m
r=rank(A)<n and b ∈ R , find x such that min x || Ax − b ||2
Theorem : There are n-r dimensional set of vectors x
that minimize || Ax − b ||2
Proof: Let S=Au=0. Dimension of S is n-r. If x
minimizes || Ax − b ||2 , so does x+u.
Conclusion: The least squares solution is not unique!
mxn
First Remarks (2)
z
Theorem: Let σ min = σ min ( A) > 0 ,the smallest singular
T
A
=
U
Σ
V
value of
.
| unT b |
i) If x minimizes || Ax − b ||2 then || x ||2 ≥
,where un
σ min
is the last column of U
ii) changing b to b + δb can change x to x + δx, where
|| δ x ||2 ≤
|| δ b ||2
σ min
• Conclusion: When A is nearly rank deficient, solution
x is ill-conditioned (and possibly very large)
• What can we do?
4
Regularization
z
z
5
That is to impose extra conditions to the
solution so as to convert an ill-conditioned
problem to a well-one.
Let’s try to approach it to an exactly rankdeficient least squares problem, using the
SVD method.
SVD: Theorem
z
Σ1 rxr, U1,V1 with r
Columns
6
SVD: Proof (1)
7
SVD: Proof (2)
z
8
Key point: The minimum norm solution x is unique
and may be well-conditioned provided σmin is not
too small
Problems…
z
z
z
z
9
Let A = ⎛
⎜
1 0 ⎞,
⎟
0
0
⎝
⎠
b = [1,1]T . Applying the
previous theorem we take the unique solution x = [1, 0]
and the condition number is 1.
1 0 ⎞ b = [1,1]T
Let A = ⎛⎜
. Applying the previous
⎟,
⎝0 ε ⎠
T
x
=
[1,1/
ε
]
theorem we take the unique solution
and the
condition number is 1/ε >> 1.
So, for a slight different example things are much more
difficult…
It needs decisiveness!!!
T
Truncated SVD
z
z
We are going to define a measure tol, setting each
singular value smaller than tol to zero!!
Let U Σ V T the truncated SVD. As
|| U Σ V Τ − U ΣV Τ ||2 =|| U (Σ − Σ)V Τ ||2 ≤|| U ||2 || (Σ − Σ) ||2 || V ||2 =|| (Σ − Σ) ||2 ≤ tol
•
10
we can use the truncated method instead of the
original.
The advantage: Not only do we compute the
minimum solution, but also we minimize the
condition number!
QR Method
⎛ R11
A = QR = Q ⎜
⎝ 0
R12 ⎞
⎟ and
R22 ⎠
R11 is of order r
z
Let
z
Provided that || R22 ||2 0 and [Q, Q ] square and orthogonal
T
T
⎡
⎤
⎡
Q
Rx
−
Q
b⎤ 2
2
2
|| Ax − b ||2 =|| ⎢ T ⎥ ( Ax − b) ||2 =|| ⎢ T
⎥ ||2 =
⎣⎢Q ⎦⎥
⎣⎢ −Q b ⎦⎥
=|| Rx − QT b ||2 + || Q T b ||2
2
2
Q = [Q1 , Q2 ], x = [ x1 , x2 ]T
T
2
|| Ax − b || =|| R11 x1 + R12 x2 − Q1 b || + || Q b || + || Q b ||2
2
2
z
11
Is minimized choosing
T
2
2
T
2
2
2
⎡ R11−1 (Q1T b − R12 x2 ) ⎤
x=⎢
⎥
⎣ x2
⎦
QR Method: Bad Scenario
z
Let
z
Though condition number of the matrix increases
rapidly, || R22 ||2 is not small!! (try it with matlab)
So original QR failures to recognize nearly rank
deficient R!
•
12
⎛1/ 2 1 0 ⎞
⎜
⎟
A=⎜
% 1 ⎟
⎜
⎟
1/
2
0
⎝
⎠
A=QR, Q=I,R=A .
QR with Pivoting
z
z
z
z
13
We factorize AP=QR, P: permutation matrix
At step i, we select from the unfinished part of A
(columns i Æn, rows i Æm) the column of largest
norm and we exchange it with the i-th column.
So, we try to keep R11 as well-conditioned as
possible and R22 as small as possible
However, this greedy strategy fails to some types of
matrices (Kahan matrices)
RRQR
z
Let A mxn and singular values σ 1 ≥ σ 2 ≥ ... ≥ σ n ≥ 0
R ⎞
⎛R
AP = QR = Q ⎜
⎟ , R
11 is of order r
⎝ 0 R ⎠
RRQR=rank-revealing QR factorization is the
σ
cond
(
R
)
=
,|| R || = σ ( R ) = σ
one where
σ
Find a permutation that maximizes σ m in ( R1 1 )
Find a permutation that minimizes σ max ( R22 )
11
•
12
22
1
11
•
•
14
22
r
2
max
22
r +1
Which Method Should I Choose?
z
z
z
z
z
15
Tradeoff between reliability and speed
QR without pivoting is fastest but less
reliable…
… SVD is slowest but most reliable…
QR with pivoting/RRQR are in the middle.
Generally speaking, if m>>n, all algorithms
cost about the same
References
z
z
z
z
16
James W. Demmel, “Applied Numerical
Linear Algebra”
http://www.ms.uky.edu/~bai/Class/lecture19.
ps
http://www.cse.uiuc.edu/heath/scicomp/notes
/chap03.pdf
http://www.mcs.anl.gov/prism/lib/techsrc/wn3
0.ps
Ευχαριστώ!
17