21/11/2011
CS475 / CM375
Lecture 20: Nov 17, 2011
Singular value decomposition
Reading: [TB] Chapter 31
CS475/CM375 (c) 2011 P. Poupart & J. Wan
1
Singular Value Decomposition
• Geometric view:
• Let be the unit sphere in .
is an ellipse in . The image CS475/CM375 (c) 2011 P. Poupart & J. Wan
2
1
21/11/2011
Interpretation
• The singular values of are the lengths of the principal semi‐axes of : , , … ,
⋯
– Convention: 0
• The left singular vectors of are the unit vectors ,…,
in the direction of the principal semi‐
axes.
• The right singular vectors of are the unit vectors ,…,
∈ such that CS475/CM375 (c) 2011 P. Poupart & J. Wan
3
Reduced SVD
Σ
• Decomposition: – Picture:
– Here Σ
,
and have orthonormal columns.
Σ
• Equivalently: – Hence ∀
1,2, … ,
– Picture:
CS475/CM375 (c) 2011 P. Poupart & J. Wan
4
2
21/11/2011
Full SVD
• Extend →
orthogonal
• Accordingly, Σ → Σ
• Then Σ
Σ
0
where , ,
– Picture:
CS475/CM375 (c) 2011 P. Poupart & J. Wan
5
SVD vs Eigendecomposition
• They both diagonalize a matrix . SVD uses 2 bases (left and right singular vectors). Eigendecomposition
uses 1 basis (eigenvectors)
• SVD uses orthonormal vectors where as eigenvectors are not orthonormal in general
• Not all matrices have an eigendecomposition. But all matrices have a singular value decomposition
CS475/CM375 (c) 2011 P. Poupart & J. Wan
6
3
21/11/2011
Matrix properties of SVD
• Let ∈
, min , , # of nonzero singular values of .
• Theorem:
• Proof: The rank of a diagonal matrix = # of nonzero diagonal entries. Since Σ , then
, orth ⟹
Σ
CS475/CM375 (c) 2011 P. Poupart & J. Wan
7
Matrix properties of SVD
,…,
• Theorem:
and • Theorem:
Note: ,…,
⋯
and , CS475/CM375 (c) 2011 P. Poupart & J. Wan
∑
8
4
21/11/2011
Matrix properties of SVD
• Proof: A
CS475/CM375 (c) 2011 P. Poupart & J. Wan
9
Matrix properties of SVD
• Theorem: The nonzero singular values of are the square roots of the nonzero eigenval. of or .
and are similar to Σ
• Proof:
• Theorem: If , then particular, if is SPD, then CS475/CM375 (c) 2011 P. Poupart & J. Wan
: ∈Λ
Λ .
. In 10
5
21/11/2011
Matrix properties of SVD
• Theorem: the condition number of ∈
is
• Proof:
CS475/CM375 (c) 2011 P. Poupart & J. Wan
11
Computing the SVD
• Recall:
Σ
Σ
Σ Σ
Σ
∴ eigenvalues of are σ
CS475/CM375 (c) 2011 P. Poupart & J. Wan
12
6
21/11/2011
An SVD algorithm
(1) Form (2) Compute the eigendecomposition
(3) Compute Σ
⋱
, , Λ
Λ
⋱
(4) Solve the equation Σ
for orthogonal (by QR factorization)
CS475/CM375 (c) 2011 P. Poupart & J. Wan
13
An SVD algorithm
• Unstable algorithm
– Suppose is computed stably, i.e.,
– Take square root to get – If ≪|
|, (e.g., :
), then
⟹ loss of accuracy CS475/CM375 (c) 2011 P. Poupart & J. Wan
14
7
21/11/2011
Example
• Find the SVD of 0
3
0
1/2
0
0
• Method 1:
CS475/CM375 (c) 2011 P. Poupart & J. Wan
15
Example (continued)
• Method 2:
• Method 3: CS475/CM375 (c) 2011 P. Poupart & J. Wan
16
8
21/11/2011
Example (continued)
• Method 3 (continued):
CS475/CM375 (c) 2011 P. Poupart & J. Wan
17
9