Properties of the Horst
Algorithm for the
Multivariable Eigenvalue
Problem
Michael Skalak
Northwestern University
Outline of Problem
Given n1 ,..., nm  such that
definite matrix

m
n  n and a symmetric, positive
i 1 i
 A11 A12  A1m 
A

A

A
21
22
2m 

A
 

 


 Am1 Am 2  Amm 
with Aii  R n n
i
i
Multivariable Eigenvalue Problem
• Find real scalars 1 ,..., m and a real column vector
such that x  R n
Ax  x
xi  1, i  1,..., m
  diag{1 I [ n1 ] ,..., m I [ nm ]}
where I [ n ] is the identity matrix of size ni
n
x

R
and
is partitioned into blocks
T T
x  [ x1T ,..., xm
]
i
with xi  R n
i
Example
Given the symmetric and positive definite matrix n1  2 , m  2 , n2  3




A





 1.950  0.349  3.645 
 2.052  2.162  1.473  



8
.
990
0
.
297
0
.
581

 
 0.297 10.367 4.638  

 
 0.581 4.638 12.442 
 6.740  3.821
 3.821 9.919 


2.052 
 1.95
 0.349  2.162


  3.645  1.473 
the vector x   0.177

0.994
is a solution, as Ax  12.405x1
0.131
17.841x2 
 0.684
 0.717
Statistical Application
Find the maximum correlation coefficient of m random variables, each of
size ni i  1,..., m

T
Maximize x Ax
subject to
xi  1 i  1,..., m
Hence the solution is the global maximum of x T Ax for vectors in
Bn1  ...  Bnm, where Bn is a ball of radius 1 centered at the origin in
dimensions.
n
Power Method

The power method finds the eigenvector with the largest
eigenvalue for the usual single-variate eigenvalue problem.
for k  1,2,...
y ( k )  Ax ( k )
( k )  y ( k )
x ( k 1) 
end
y (k )
( k )
Horst Algorithm
Finds the
x which maximizes x T Ax
for k  1,2,...
for i  1,..., m
y
(k )
i
:
m
A
j 1
ij
x (j k )
(i k ) : yi( k )
( k 1)
i
x
:
yi( k )
(i k )
end
end
Proven to converge monotonically by Chu and Watterson
[SIAM J. Sci. Comput. (14), No. 5, pp. 1089-1106]
Example
For that same matrix, consider the Horst algorithm with the
starting point x(0)  0.707 0.707 0.577 0.577 0.577
First iteration:
2
(1)
1
y
  A1 j x1( 0 )  .884 3.398
j 1
1(1)  y1(1)  3.511
( 2)
1
x

y1(1)
1(1)
 0.252 0.968
2
y
(1)
2
  A2 j x2( 0 )  8.527
7.059
6.578
j 1
(21)  y2(1)  12.807
x
( 2)
2

y2(1)
(21)
 0.662
0.548 0.511
Dependence on Initial Conditions
Convergence point can depend on initial conditions:
T
x(0)  0.447 0.894  0.802 0.535  0.267   0.108 0.994 0.131  0.684  0.717 x Ax  30.284
x(0)  0.707 0.707 0.577 0.577 0.577   0.900 0.436  0.023 0.464 0.885 xT Ax  31.414
Like many other maximization algorithms, the Horst algorithm can converge to a
local instead of global max.
Results



For any ni , ni  1 can have at least as many
convergent points
 m 1 


 3 
For any m, there can be at least 3
convergence
points, and as few as one.
In at least a nontrivial special case (two convergence
points, m  2, n  3 ) the portion of the region which
converges to the global max can not be arbitrarily
small
Number of Convergence Points
There exist 3 matrices, m  1, 2, 3 ni  1 for all i such that there exist m
convergence points.
The block matrix, with Am meaning a matrix of size m with m convergence
points
 Am1
 0

0 

Am2 
is symmetric, positive definite, and has m1m2 convergence points
With a little manipulation, this proves that for any size m there exist
 m 1 


 3 
matrices with at least 3
convergence points.
Convergence to Global Max
Suppose there is some transformation on the matrix that can
arbitrarily move eigenvectors arbitrarily. After the
transformation, the matrix is rescaled so that the largest
element remains constant.
 a  b c  
 b d e  
  


 c   e f  


d xT Ax
 a  b  c dx1  b  d  e dx2  c  e  f dx3
dx
Case 1
The difference of the values between the local mins
and the global max is bounded    0 . Then the
derivative must increase without bound. However,
since all elements of the matrix are less than a
constant, this cannot happen.
Case 2
The values of the local mins approach the global max as
the vectors approach. Since one of the local mins is
the global min, the function become closer and closer
to constant, which cannot happen since the derivative
is bounded below in at least one direction.