RELIABLE CALCULATION OF
NUMERICAL RANK, NULL SPACE BASES,
BASIC SOLUTIONS AND
PSEUDOINVERSE SOLUTIONS USING
SUITESPARSEQR
Leslie Foster
Mathematics Department, San Jose State University
and
Timothy Davis
CISE Department, University of Florida
June 15, 2011, Householder XVIII
L ESLIE F OSTER M ATHEMATICS D EPARTMENT, S AN J OSE S TATE U NIVERSITY
RELIABLE
AND CALCULATION
T IMOTHY DAVIS OF
CISE
NUMERICAL
D EPARTMENTRANK,
, U NIVERSITY
NULL OF
SP
SPQR_RANK PACKAGE
For sparse, numerically rank deficient matrices
SPQR_BASIC: basic solutions to least
squares problems
SPQR_NULL: sparse, orthonormal null
space bases
SPQR_PINV: pseudoinverse solutions to
least squares problems.
SPQR_COD: pseudoinverse solutions to
least squares problems using a complete
orthogonal decomposition
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
SPQR_RANK PACKAGE, CONTINUED
Goal: reliable determination of the numerical
rank (the number of singular values > input
tolerance τ ) in the sense that a warning flag
warns the user if the the calculated numerical
rank may be incorrect
The routines achieve the goal for matrices with
a well defined numerical rank and almost always
achieve goal for any matrix
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
T OOL : SPQR
SPQR from SuiteSparseQR (Tim Davis, 2011):
High Performance Multifrontal Sparse QR
Factorization
Estimates numerical rank; often an accurate
estimate, but not always
Can return Q as sparse Householder
transforms
sparse QR in MATLAB 7.9 and higher
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
AN EXAMPLE
A: HB\will199 from UF Sparse Matrix Set
199 × 199 structural problem, Harwell-Boeing
A = [ A , zeros(100,1)
√ ];
b = ones(199,1) / 199; ⇒ ||b|| = 1
SPQR:
x=A\b;
norm(x), norm( b - A*x)
1.6 × 1014 .094
SPQR_RANK:
x = spqr_basic(A,b) ; norm(x), norm( b - A*x )
1.8
.086
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
S ECOND T OOL : I NVERSE SUBSPACE ITERATION
Algorithm SPQR_SSI: apply to k × k triangular
matrix R, returns approximate singular values
<= τ , and one singular value > τ plus singular
vectors
block power method applied to R −1 R −T
increase block size until estimate of σr of
R>τ
References: Chan, Vogel, Gotsman, Toledo
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHM SPQR_NULL
A = [ gallery(’kahan’,100) , zeros(100,2) ] ;
SPQR to AT :


R
b = Q  0  or
AT P = Q R
0
A = P RT 0 0 QT
R is 100 × 100, numerically singular with no
small diagonal entries. SPQR concludes that R
is full rank.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHM SPQR_NULL CONT ’ D
SPQR_SSI to R:
s1 0 T
v1 v2
R v u1 u2
0 s2
s1 , s2 estimates of smallest singular values of R.
R T u2 = v2 s2 , s2 = 9 × 10−17 , ||u2 || = ||v2 || = 1.


u2 0 0
⊥ null space basis = N = Q  0 1 0  = Q X
0 0 1
Recall: A = P R T 0 0 Q T
||A N|| = 9 × 10−17 ; numerical rank correct.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHM SPQR_NULL, WARNING FLAG
A = [ gallery(’kahan’,100) , zeros(100,2) ] ;
[N, stats ] = spqr_null(A); stats
flag: 0
rank: 99
tol: 1.8e-13
est_sval_upper_bounds: [1.18e-03 1.4e-16]
est_sval_lower_bounds: [1.12e-03 0]
sval_numbers_for_bounds: [99 100]
1.12e-03 > 1.8e-13 ≥ 1.4e-16
So numerical rank appears correct, flag = 0.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHM SPQR_NULL, EST. SING . VALUE BOUNDS
Tools (Golub, Bjorck, Parlett, . . . ) for estimating
bounds on singular values:
singular value interlace theorem
singular value minimax characterization
perturbation bounds on singular values
residual error bounds for eigenvalues
in SPQR, AP = QR + E, ||E||F known
“Bounds” are estimations which work well in
practice in our tests.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHM SPQR_NULL, STORAGE
Store N implicitly in terms of Q and X.
Store Q using sparse Householder factors.
X usually has few (or no) dense columns.
Example: 321,671 × 321,671 sparse matrix
(Sandia\ASIC_320ks) with numerical nullity
of 222,481.
N in implicit form:
2 × 109 bytes.
N as explicit matrix: 573 × 109 bytes.
SPQR_RANK provides tools to use implicit
N.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A LGORITHMS (SPQR_BASIC, SPQR_PINV,
SPQR_COD)
Algorithms SPQR_BASIC, SPQR_PINV,
SPQR_COD, as well as SPQR_NULL, use:
SPQR
subspace iteration to check and correct, if
needed, the numerical rank
estimation of singular value bounds to
determine a warning flag
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
T ESTING : SJSU S INGULAR M ATRIX DATABASE
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
M ATRIX T EST S ET
SJSU Singular Matrix Database at
http://www.math.sjsu.edu/
singular/matrices/
767 numerically singular matrices (with
tolerance max(m, n) eps(normest(A)) )
Matrices from UF Sparse Matrix Collection
(Davis) & Regularization Tools (Hansen)
Matrices come from real world applications
or have characteristic features of real world
problems
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
P ROPERTIES OF T EST S ET M ATRICES :
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
P ROPERTIES OF T EST S ET M ATRICES ( CONT ’ D ):
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
PARAMETERS IN EXPERIMENTS
1
2
3
tolerance, τ , defining numerical rank,
= max(m, n) ∗ eps(estimate of ||A||)
= O((machine eps)||A||)
where A is m × n, || || is 2 norm
Essentially the same choice as the
MATLAB’s RANK and NULL commands
other parameters set to default values
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
ACCURACY OF R ANK C ALCULATION (4 ROUTINES ):
hard to see, so focus on SPQR_BASIC:
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
ACCURACY OF R ANK C ALCULATION (SPQR_BASIC):
warning flag had no false positives
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
ACCURACY OF N ULL S PACE BASES (4 ROUTINES ):
hard to see, so focus on SPQR_NULL:
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
ACCURACY OF N ULL S PACE BASES (SPQR_NULL):
SPQR_NULL as good or nearly as good as
MATLAB’s SVD
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
Q UALITY OF BASIC S OLUTIONS
SPQR_BASIC quality similar to dense QR basic
solutions, in most cases
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
ACCURACY OF P SEUDOINVERSE S OLUTIONS
SPQR_COD solutions are close to solutions
calculated by MATLAB’s PINV
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
RUN T IMES : SPQR_COD VERSUS MATLAB’ S SVD:
SPQR_COD 10,000+ times faster than
MATLAB’s PINV
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
RUN T IMES : SPQR_BASIC, SPQR_COD & SPQR
SPQR_BASIC 22% slower than SPQR
SPQR_COD is 350% slower, on average
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
M EMORY U SE : SPQR_BASIC, SPQR_COD & SPQR
SPQR_BASIC 1% more memory than SPQR
SPQR_COD 235% more memory, on average
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
A RE THE SPQR_RANK ROUTINES RELIABLE ?
Is the user warned when the numerical rank
may be incorrect?
For entire data set no false positives for
SPQR_BASIC, SPQR_PINV, SPQR_COD.
For one matrix SPQR_NULL had a false
positive (rarely)
Singular values decay gradually for that
matrix ( Sandia\oscil_dcop_33)
So in our testing the routines were always
reliable for matrices with a well defined
numerical rank and almost always for any
matrix.
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA
C ONCLUSIONS
SPQR_RANK provides tools for sparse, rank
deficient matrices
Similar tools to those provided by dense
matrix MATLAB code
Basic and pseudoinverse solutions,
orthonormal null space bases
reliable in the sense that warning flag warns
user if rank may be incorrect, almost always
References
Paper submitted to ACM TOMS, 2011
Paper and code: Google “Les Foster”
L ESLIE F OSTER AND T IMOTHY DAVIS
H OUSEHOLDER XVIII, J UNE 12-17, 2011, TAHOE C ITY, CA