Rank Revealing QR factorization
F. Guyomarc’h, D. Mezher and B. Philippe
1
Outline
• Introduction
• Classical Algorithms
⋆ Full matrices
⋆ Sparse matrices
• Rank-Revealing QR
• Conclusion
CSDA 2005, Cyprus
2
Situation of the problem
See Bjorck SIAM96 : Numerical Methods for Least Squares Problems.
Data : A ∈ Rm×n , b ∈ Rm.
Problem P :
Find x ∈ S such that kxk minimum
where S = {x ∈ Rn | kb − Axk minimum }.
(k.k ≡ k.k2)
The solution is unique : x̂ = A+b.
Property :
x∈S
iff
AT (b − Ax) = 0,
iff
AT Ax = AT b.
The last equation is consistent since R(AT A) = R(AT ).
CSDA 2005, Cyprus
3
Rank-Revealing QR factorization
Theorem : A ∈ Rm×n with rank(A) = r (r ≤ min(m, n)).
There exist Q, E, R11 and R12 such that
• Q ∈ Rm×m is orthogonal,
• E ∈ Rn×n is a permutation,
• R11 ∈ Rr×r is upper-triangular with positive diagonal entries,
• R12 ∈ Rr×(n−r),
and AE = Q
R11 R12
0
0
!
.
RRQR
The factorization is not unique. Let RRQR be any of them.
CSDA 2005, Cyprus
4
Actually, if we consider a complete orthogonal decomposition
A=Q
T 0
0 0
!
VT
where Q and V are orthogonal and!T triangular with positive diagonal
T −1 0
+
QT .
entries, we have A = V
0 0
Such a factorization can be obtained! from the previous one by perT
R11
forming a QR factorization of
T
R12
CSDA 2005, Cyprus
5
Column pivoting strategy
Householder QR factorization using Householder reflections :
H1 A =
A=
v = A1..12,1 + ||A1..12,1||e1
t
H1 = I12 − 2 vvvt v
CSDA 2005, Cyprus
v = A2..12,2 + ||A2..12,2||e1
t
H2 = I11 − 2 vvvt v
6
Sparse QR factorization
Factorizing a sparse matrix implies fill-in in the factors.
The situation is worse with QR than with LU since when updating the
tailing matrix :
• LU : the elementary transformation x −→ y = (I − αveT
k )x keeps x
invariant when xk = 0.
• QR : the elementary transformation x −→ y = (I − αvv T )x keeps x
invariant when x⊥v.
Since AT A = RT R, the QR factorization A is related to the Cholesky
factorization of AT A. It is known that a symmetric permutation on a
sparse s.p.d. matrix changes the level of fill-in.
CSDA 2005, Cyprus
7
Therefore, a permutation of the columns of A changes the fill-in of R.
⇒ there is conflict between pivoting to minimize fill-ins and pivoting
associated with numerical properties.
We choose to decouple the sparse factorization phase and the rankrevealing phase
For a standard QR factorization of a sparse matrix :
Multi-frontal QR method [Amestoy-Duff-Puglisi ’94]
Routine MA49AD in Library HSL.
CSDA 2005, Cyprus
8
Column pivoting strategy
Householder QR factorization using Householder reflections :
H1 A =
A=
v = A1..12,1 + ||A1..12,1||e1
t
H1 = I12 − 2 vvvt v
v = A2..12,2 + ||A2..12,2||e1
t
H2 = I11 − 2 vvvt v
Householder QR with Column Pivoting (Businger and Golub) :
At step k, the column jk ≥ k defining the Householder transformation
is chosen so that kA(k : m, jk )k ≥ kA(k : m, j)k for j ≥ k.
CSDA 2005, Cyprus
9
Some properties on R11
At step k :


(k)
(k)
R
R12 
A(k) ≡ Hk · · · H1AE(k) =  11
.
(k)
0
A22


(k)
R
(k, :) 
(k)
satisfies R11 (k, k) ≥ kak.
Any column a of  12 (k)
A22
(k)
(k)
Moreover kAk ≥ kR11 k ≥ R11 (1, 1)
(k)
(k)
R11 (k, k) ≥ σmin(R11 ) ≥ σmin(A),
(k)
R (1,1)
(k)
.
This implies that : cond(A) ≥ cond(R11 ) ≥ 11
(k)
R11 (k,k)
CSDA 2005, Cyprus
10
Bad new
(k)
The quantity
R11 (1,1)
(k)
R11 (k,k)
(k)
cannot be considered as an estimator of cond(R11 )
since it can be of different order of magnitude :
Example (Kahan’s matrix of order n) :
for θ ∈ (0, π
2 ), c = cos θ and s = arcsin θ:
MATLAB : d=c .ˆ [0:n-1] ; M=diag(d)*(eye(n)-s*triu(ones(n),1));
n=100 ; θ = arcsin(0.2) :
σmin(R11) = 3.7e − 9 ≪ R11(100, 100) = 1.3e − 1
(k)
Therefore a better estimate of cond(R11 ) is needed.
CSDA 2005, Cyprus
11
Incremental Condition Estimator (ICE) [Bischof 90]
which is implemented in LAPACK :
(k)
(k−1)
it estimates cond(R11 ) from an estimation of cond(R11 ).
However, the strategy which consists in stopping the factorization when
1
(k+1)
)
≤ cond(R11
ǫ
may fail in very special situations :
Counterexample :
M = Kahan’s matrix of order 30 with θ = arccos(0.2)
(17)
(16)
cond(R11 ) < 1ǫ < cond(R11 )
(k)
cond(R11 ) <
indicates a numerical rank equal to 16 although the numerical rank of
M computed by SVD is 22.
CSDA 2005, Cyprus
12
Pivoting strategies
Convert R to reveal the rank by pushing the singularities towards the
right end
• Apply an Incremental Condition Estimator to evaluate σmax and
σmin of R1→i,1→i,
• if σmax /σmin > τ , move column i towards the right end and reorthogonalise R using Givens rotations
CSDA 2005, Cyprus
13
Pivoting strategies
• RRQR uses a set of thresholds to overcome numerical singularities,
• A predifined set of thresholds might fail


2
0
0
0


A =  0 10−6
1
1

0
0
10−3 −10−3
rank=2 if τ = {107}, rank=3 if τ = {104, 107}
• RRQR adapts the thresholds to avoid failure
⋆ Upon completion, check if R22 < σmax
τ ≃ σmin
⋆ On failure, insert new thresholds and restart algorithm
CSDA 2005, Cyprus
14
Numerical results
50
10
smax/smin
45
cond([q,r,p]=qr(A)
10
RRQR, thres=1e3,1e7,1e12,1e25
RRQR, thres=1e3,1e9,1e25
40
10
35
condition number
10
30
10
25
10
20
10
15
10
10
10
5
10
0
10
0
CSDA 2005, Cyprus
5
10
15
20
25
30
35
40
kahan(50,acos(0.2)), cond=2.4721e+49, rank=20, size=50x50, req=0
45
50
15
Numerical results
50
10
smax/smin
45
10
cond([q,r,p]=qr(A)
RRQR, thres=1e30
40
10
RRQR, thres=1e20
RRQR, thres=1e25, req=2
35
condition number
10
30
10
25
10
20
10
15
10
10
10
5
10
0
10
0
CSDA 2005, Cyprus
5
10
15
20
25
30
35
40
kahan(50,acos(0.2)), cond=2.4721e+49, rank=20, size=50x50
45
50
16
Second strategy
The k + 1 column is not the worst usually for the condition number.
Can we select the worst one for the conditionement of R11 ?
Then we reject this worst column to the end.
k+1
j
k+1
We need to recompute the singular values and vectors estimates.
Reverse ICE
CSDA 2005, Cyprus
17
Second strategy
Random matrix, n=500
20
10
Exact
Estimated tol=1e−12
Estimated tol=1e−7
15
10
10
10
5
10
0
10
0
CSDA 2005, Cyprus
100
200
300
400
500
18
Conclusion
Work is still on progress :
Case where R is sparse
• ICE might fail, so does the Reverse ICE.
• Using the elimination tree, we can try to keep R as sparse as
possible.
CSDA 2005, Cyprus
19