Extremum Properties of
Orthogonal Quotients Matrices
By
Achiya Dax
Hydrological Service, Jerusalem , Israel
e-mail: [email protected]
The Eckart – Young Theorem (1936)
says that the “truncated SVD” matrix
Ak = Uk Dk Vk
T
solves the least norm problem
minimize
subject to
F ( B ) = || A - B|| F 2
rank(B)  k
( Also called the Schmidt-Mirsky Theorem. )
Ky Fan’s Maximum Principle (1949)
S a symmetric positive semi-definite n x n matrix
Yk the set of orthogonal n x k matrices
l1 + … + lk = max { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [v1 , v2 , … , vk] .
giving the sum of the largest k eigenvalues.
Outline
* The Eckart – Young Theorem
and Orthogonal Quotients matrices.
* The Orthogonal Quotients Equality.
* The Symmetric Quotients Equality
and Ky Fan’s extremum principles.
* An Extended Extremum Principle.
The Singular Value Decomposition
A = U S VT
S = diag {s1 , s2 , … , sp } ,
p = min { m, n}
U = [u1 , u2 , … , up] ,
UT U = I
V = [v1 , v2 , … , vp] ,
VT V = I
AV = US
AT U = V S
A vj = sj uj , AT uj = sj vj
j = 1, … , p .
Low - Rank Approximations
Ak =
T
Uk Dk Vk
Dk = diag {s1 , s2 , … , sk } ,
Uk = [u1 , u2 , … , uk] ,
UkT Uk = I
Vk = [v1 , v2 , …
VkT Vk = I
, v k] ,
The Eckart – Young Theorem (1936)
says that the “truncated SVD” matrix
Ak = Uk Dk Vk T
solves the least norm problem
minimize
subject to
F ( B ) = || A - B|| F 2
rank(B)  k
( Also called the Schmidt-Mirsky Theorem. )
Rank - k Matrices
B = Xk Rk YkT
where
Rk is a k x k matrix
Xk = [x1 , x2 ,
… , xk ] ,
Yk = [y1 , y2 , …
, yk ] ,
XkT Xk = I
YkT Yk = I
The Eckart – Young Problem
can be rewritten as
minimize F ( B ) = || A - Xk Rk YkT || F 2
subject to
XkT Xk = I and YkT Yk = I .
Theorem 1: Given a pair of orthogonal
matrices, Xk and Yk , the related
“Orthogonal Quotients Matrix”
XkT A Yk =
( xiTAyj )
solves the problem
minimize
F ( Rk ) = || A - Xk Rk YkT || F 2
Notation:
Xk - denotes the set of all real m x k
orthogonal matrices Xk ,
Xk = [x1 , x2 , … , xk] ,
XkT Xk = I
Yk - denotes the set of all real n x k
orthogonal matrices Yk ,
Yk = [y1 , y2 , … , yk] ,
YkT Yk = I
Corollary 1 :
The Eckart – Young Problem
can be rewritten as
minimize F ( Xk,Yk ) = || A - Xk Rk YkT || F 2
subject to
where
XkXk and YkYk ,
Rk is the “Orthogonal Quotients Matrix”
Rk = XkT A Yk .
The Orthogonal Quotients Equality
For any pair of orthogonal matrices,
XkXk and YkYk ,
||A - Xk Rk YkT || F 2 = ||A|| F 2 - || Rk || F 2
where
Rk is the orthogonal quotients matrix
Rk = XkT A Yk .
Corollary : The Eckart–Young Problem
minimize F ( Xk,Yk ) = || A - Xk Rk YkT || F 2
subject to
XkXk and YkYk .
is equivalent to
maximize ||XkT A Yk || F 2
subject to
XkXk and YkYk ,
and the SVD matrices
Uk
giving the optimal value of
and
Vk
solves both problems,
s12 + s22 + … + sk2 .
Question: Is the related minimum problem
minimize || (Xk)T A Yk || F 2
subject to XkXk and YkYk ,
solvable ?
Using the Orthogonal Quotients Equality
the last problem takes the form
maximize F ( Xk,Yk ) = || A - Xk Rk YkT || F 2
subject to
XkXk and YkYk .
Note that the more general problem
maximize
F ( B ) = || A - B|| F 2
subject to
rank(B)  k
is not solvable .
Returning to symmetric matrices
How we extend the
Orthogonal Quotients Equality
to symmetric matrices ?
The Spectral Decomposition
S = ( Sij )
a symmetric positive semi-definite n x n matrix
With eigenvalues
l1 l2 ln 
and eigenvectors
v1 , v2 , … , vn
S vj =
lj vj , j = 1, … , n
.
SV=VD
V = [v1 , v2 , … , vn] , VT V = V VT = I
D = diag { l1 , l2 , … , ln }
S = V D VT = S lj vj vjT
Recall that
Rayleigh Quotient Matrices
Sk = YkT S Yk
play important role in
Ky Fan’s Extremum Principles .
Ky Fan’s Maximum Principle
S a symmetric positive semi-definite n x n matrix
Yk the set of orthogonal n x k matrices
l1 + … + lk = max { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [v1 , v2 , … , vk] .
which is related to the largest k eigenvalues.
Ky Fan’s Minimum Principle
S
a symmetric n x n matrix .
Yk the set of orthogonal n x k matrices .
ln-k+1 + … + ln =min { trace ( YkT S Yk ) | YkYk }
Solution is obtained for the Spectral matrix
Vk = [vn-k+1 , … , vn] ,
which is related to the smallest k eigenvalues.
Question : Can we formulate the
Symmetric Quotients Equality
in terms of
trace ( Sk ) = trace( YkT S Yk ) = S yjT S yj
The Symmetric Quotients Equality
S
a symmetric n x n matrix
YkYk an orthogonal n x k matrix
Sk =YkT S Yk
the related “Rayleigh quotient matrix”
trace (S - Yk Sk YkT ) = trace( S ) - trace( Sk )
Corollary 1 : Ky Fan’s maximum problem
maximize trace (Yk S YkT )
subject to
YkYk ,
is equivalent to
minimize trace ( S - Yk Sk YkT )
subject to
YkYk .
The Spectral matrix Vk = [v1 ,v2 , … ,vk]
solves both problems,
giving the optimal value of l1 + … + lk .
Recall that:
The Eckart–Young Problem
minimize F ( Xk,Yk ) = || A - Xk Rk YkT || F 2
subject to
XkXk and YkYk .
is equivalent to
maximize ||XkT A Yk || F 2
subject to
XkXk and YkYk ,
and the SVD matrices
Uk
giving the optimal value of
and
Vk
solves both problems,
s12 + s22 + … + sk2 .
Corollary 2 : Ky Fan’s minimum problem
minimize trace (Yk S YkT )
subject to
YkYk ,
is equivalent to
maximize trace ( S - YkT Sk Yk )
subject to
YkYk .
The matrix
Vk = [vn-k+1 , … , vn]
solves both problems,
giving the optimal value of ln-k+1 + … + ln .
Extended Exremum Principles
Can we extend these extremums
from eigenvalues of symmetric matrices
to singular values of rectangular matrices ?
Notation: 1 m*  m , 1  n*  n ,
Xm* - denotes the set of all real m x m*
orthogonal matrices Xm* ,
Xm* = [x1 , x2 , … , xm*] ,
Xm*T Xm* = I
Yn* - denotes the set of all real n x n*
orthogonal matrices Yn* ,
Yn* = [y1 , y2 , … , yn*] , Yn*T Y* = I
Notations :
Given Xm* Xm* and Yn* Ym* ,
the m*x n* matrix
( Xm*)TA Yn* = ( xiTAyj )
is called “Orthogonal Quotients Matrix” .
Notations :
The singular values of the
Orthogonal Quotients Matrix
( Xm*)TA Yn* = ( xiTAyj )
are denoted as
h1  h2  …  hk  0 ,
where
k = min { m*, n* } .
Questions :
Which choice of orthogonal matrices
Xm* Xm* and Yn* Ym* ,
maximizes (or minimizes ) the sum
(h1)p + (h2)p + … + (hk)p
where p > 0 is a given positive constant .
An Extended Maximum Principle:
The SVD matrices
Um* = [u1 , u2 ,
… , um*] and
Vn* = [v1 , v2 , …
, vn*]
solve the problem
maximize F ( Xm* , Yn* ) = (h1)p + (h2)p + … + (hk)p
subject to Xm* Xm*
and Yn* Yn* ,
for any positive power p > 0
,
giving the optimal value of
(s1)p + (s2)p + … + (sk)p .
An Extended Maximum Principle
That is, for any positive power p > 0 ,
(s1)p + (s2)p + … + (sk)p =
max{ (h1)p + (h2)p +…+ (hk)p | Xm* Xm* and Yn* Yn*},
and the maximal value is attained for the matrices
Um* =[u1 , u2 , … ,um*] and Vn* =[v1 ,v2 , … ,vn*] .
The proof
is based on “rectangular” versions of
Cauchy Interlace Theorem
and
Poincare Separation Theorem .
Corollary 1 : When p= 1 the SVD matrices
Um* = [u1 , u2 ,
… , um*] and
Vn* = [v1 , v2 , …
solve the “Rectangular Ky Fan problem”
maximize F ( Xm* , Yn* ) = h1 + h2 + … + hk
subject to Xm* Xm*
and Yn* Yn* ,
giving the optimal value of
s1 + s2 + … + sk .
, vn*]
Corollary 2 : When p= 1 and m* = n* = k
the SVD matrices
Uk = [u1 , u2 ,
… , uk] and
Vk = [v1 , v2 , …
, vk]
solve the maximum trace problem
maximize F ( Xk , Yk ) = trace ( (Xk)T A Yk )
subject to
Xk Xk and Yk Yk ,
giving the optimal value of
s1 + s2 + … + sk .
* See also Horn & Johnson, “Topics in Matrix Analysis”, p. 195.
Corollary 3 : When p= 2 the SVD matrices
Um* = [u1 , u2 ,
… , um*] and
Vn* = [v1 , v2 , …
, vn*]
solve the “rectangular Eckart–Young problem”
maximize F ( Xm* , Yn* ) = || (Xm*)T A Yn* || F 2
subject to Xm* Xm*
and Yn* Yn* ,
giving the optimal value of
(s1)2 + (s2)2 + … + (sk)2 .
An Extended Minimum Principle
Question: Can we prove a similar
minimum principle ?
Answer: Yes, but the solution matrices
are more complicated.
An Extended Minimum Principle:
Here we consider the problem
minimize F(Xm* ,Yn* ) = (h1)p + (h2)p + …
subject to
+ (hk)p
Xm* Xm* and Yn* Yn* ,
for any positive power p > 0 .
The solution matrices are obtained by deleting some
columns from the SVD matrices
U = [u1 , u2 , … , um] and V = [v1 , v2 , … , vn] .
Summary
* The Eckart – Young Theorem
and Orthogonal Quotients matrices.
* The Orthogonal Quotients Equality.
* The Symmetric Quotients Equality
and Ky Fan’s extremum principles.
* An Extended Extremum Principle.
The END
Thank You