Global Optimality of the Successive MaxBet Algorithm
Mohamed HANAFI
USC
ENITIAA de NANTES
France
and
Jos M.F. TEN BERGE
Department of psychology
University of Groningen
The Netherlands
Global Optimality of the Successive MaxBet Algorithm
Summary.
1. The Successive MaxBet Problem (SMP).
2. The MaxBet Algorithm.
3. Global Optimality : Motivation/Problems.
4. Conclusions and Open questions.
1. The Successive MaxBet Problem (S.M.P)
 
A  A kj
s.p.s.d
k , j  1,2, , K
K, K  Blocks Matrix
 A11
A
21

A
 

A K1
A12
A 22

AK2
 A1 K 

 A2K 
A kj

   pk , p j 

 A KK 
1. The Successive MaxBet Problem (S.M.P)
order 1
Maximize
u   u Au 
'
Subject to
K
u A
k , j 1
K
'
k
kj
uj
'
u
u

1
u u k  u k ukk  1,2,  , K
''
k
k 1
1. The Successive MaxBet Problem (S.M.P)
order s

1
2 
 s 1

k  1,2,  , K
Uk  uk
uk
Maximize
u1 , u 2 ,..., u K   u Au
'
Subject to
'
ukuk  1
{U u
'
k
 uk
k
0
k  1,2,  , K
2. The Successive MaxBet Algorithm
Ten Berge (1986,1988)
Order 1
1. Take arbitrary initial unit length vectors
2. Compute :
vk 
uk
k  1,2,  , K
K
A
j 1
kj
uj
3. rescale vk to unit length, and set uk= vk
4. Repeat steps 2 and 3 till convergence
k  1,2,  , K
k  1,2,  , K
2. The Successive MaxBet Algorithm
Order s

Ten Berge (1986,1988)
 
A kj  I pk  U k U A kj I p j  U j U j
'
k
1. Take arbitrary initial unit length vectors
2. Compute :
vk 
'
uk
k  1,2,  , K
K
A
j 1
kj
uj
3. rescale vk to unit length, and set uk= vk
4. Repeat steps 2 and 3 till convergence

k  1,2,  , K
k  1,2,  , K
Property 1 : Convergence of the MaxBet Algorithm
u
u
Property 2 : Necessary Condition of Convergence
 A11 A12
A
A
21
22

 


 A m1 A m 2
 A1m   u1   1u1 
'





 A 2 m   u 2   2u 2  u k u k  1


        k  1,2,  , K
  

 A mm  u K  K u K 
u  1  2  ....  K
3. Motivation and results
1. MaxBet Algorithm depends on the starting
vector
2. MaxBet algorithm does not guarantee the
computation of the global solution of SMP
3. Motivation and results : an example
K 2
A11
A
A 21
p1  2
p2  3
43
23
-13
0
-7
23
31
10
1
0
-13
10
64
-19
-2
0
1
-19
24
18
-7
0
-2
18
58
A12
A 22
Starting Vector
Solution Vector
Function value
{
0.67
0.36
0.20
0.53
0.30
u 
*
0.94
0.31
-0.92
0.35
0.11
{
u
{
f(u)= 10621
42185
64023
v 
*
v
0.64
0.31
0.64
0.24
0.10
0.69
0.72
0.58
-0.43
-0.68
f(v)= 9825.1
3846.7
5978.4
3. Motivation and results: Two Questions
Q1. How can we know that the solution computed
by the Maxbet algorithm is global or not ?
Q2. When the solution is not global, how can
we reach using this solution the global solution ?
3. Motivation and Results : Proceeding
Global solution of SMP
Spectral properties (eigenvalues and
eigenvectors) of A 1 ,2 ,..., K 
A12
 A11  1I p1
 A
A


I
21
22
2
p2

A 1 ,2 ,..., K  




AK2
 A K 1



A2K





 A KK  K I pK 

A1 K
RESULT 1
Result 1
When, for a solution, {u, 1, 2, …,K} satisfies
 A11
A
 21
 

A K1
A12
A 22

AK 2
 A1K   u1   1u1 





 A 2 K   u 2   2 u 2 


      
  

 A KK  u K  K u K 
A 1 ,2 ,..., K is negative semidefinite,
then u is the global solution of SMP.
ELEMENTS OF PROOF (Result 1)
we have :
vAv= vAv1v1v12v2v2 …KvKvK
= f(v)12…K,
the matrix A is negative semidefinite,
hence
f(v)1+2+…+K= f(u)
3. Motivation and Results
To what extent the previous sufficient condition (Result 1):
(matrix A  , ,...,   is negative semi definite)
1 2
K
is necessary ?
RESULT 2
Result 2
A is 2,2 blocks matrix
When u is the global maximum of S.M.P it verifies :
 A11 A12   u1   1u1 

A




 21 A 22  u 2  2u 2 
A12 
A11  1I p1
A1 ,2   

A
A


I
21
22
2
p2 

then matrix A  ,  is negative semi definite
1 2
ELEMENTS OF PROOF (Result 2)
Suppose A 1 ,2 
has a positive eigenvalue
A1 ,2 w  w
 0
1. w is block-normed vector
2. w is not block-normed vector
2.1. w is not block orthogonal to u
2.2. w is block orthogonal to u
1. w is block-normed vector
w A1 ,2 w  0
'
w Aw  1  2  u
'
w is better solution than u
2. w is not block-normed vector
2.1. w is not block orthogonal to u
v  u  w
( w1 ' w 1  w 2 ' w 2 )
 
(w1 ' w1  w 2 ' w 2 ) 2  8(u 2 ' w 2 ) 2
2
2
2
16(u 2 ' w 2 )
 
2
2
(w1 ' w1  w 2 ' w 2 )  8(u 2 ' w 2 )
2
v is better solution than u
w is not block-normed vector
2.2. w is block orthogonal to u
 d1 ' u1  0

d 2 ' u 2  0
 (d 2 ' u 2 )d1 
q

 (d1 ' u1 )d 2 
w  w  tq
*
v  u  w
*
v is better solution than u
RESULT 2
Result 3
A is K , K  blocks matrix
with all elements are positive K
p   pk
A  agh g , h  1,2...., p
 
k 1
When u is the global maximum of S.M.P
then matrix A 1 ,2 ,..., K  is negative semi definite
ELEMENTS OF PROOF (Result 3)
A is K , K  blocks matrix
with all elements are positive
Suppose A 1 ,2 ,..., K  has a positive eigenvalue
A1 ,2 ,..., K w  w
 0
ELEMENTS OF PROOF (Result 3)
w has all elements of the same sign
K


w A1 ,2 ,..., K w  w Aw   w w k k
'
'
k 1
'
k
u has all elements of the same sign
u  
p
u u a
g h
g ,h
p
gh
  | u g uh | agh
g ,h
Result 4
K 2
A is K , K  blocks matrix
with not all elements of the same sign
The sufficient condition (Result 1) :
(matrix A  , ,...,   is negative semi definite)
1 2
K
is not necessary
ELEMENTS OF PROOF (Result 4)
A
45
-20
5
6
16
3
-20
77
-20
-25
-8
-21
K 3
5
-20
74
47
18
-32
p1  2
6
-25
47
54
7
-11
16
-8
18
7
21
-7
p2  2
3
-21
-32
-11
-7
70
p3  2
ELEMENTS OF PROOF (Result 4)
Random research with 10.000.000 starting vectors
u=
0.49
-0.87
0.80
0.59
0.56
-0.82
(u) =378.96
 =0.48
4. General Conclusions
- Possible Application
in statistics :
Multivariate Methods (Analysis of K sets of data )
1. Generalized canonical correlationAnalysis:
Horst (1961)
2. Rotation methods :
MaxDiff, MaxBet, generalized Procrustes Analysis
Gower(1975); Van de Geer(1984);Ten Berge (1986,1988)
3. Soft Modeling Approach :
Estimation of latent variables under mode B
Wold (1984); Hanafi (2001)
4. Perspective and Little Open Question
- Necessary condition for the case K=3 when matrix A has not all
elements of the same sign?
-
X1 n, n
X 2 n, n
a X1a  0
 '
a X2a  0
'
a  0 ??
Motivation: Illustration 1
MaxBet Algorithm depends on the starting vector
f u 
 u1 
u 
2

u
  
 
u K 
u'k u k  1
k  1,2,..., K
The Successive MaxBet Problem (S.M.P)
and
Multivariate Methods


K


X  X1 X 2  X K  n, p 

k
k 1


Xk n, pk 
k  1,2,, K
Some multivarite methods
Generalized canonical correlation methods
Rotation methods(Agreement methods)
SOFT MODELING APPRAOCH(Approch)
Rotation methods
'
XX
A
n
 
A  A kj
 X XX
XXX
k
,
j

1
,
2
,

,
K
A kj 2
k

j
A kj   n nn k  j
k , kj, j 1,21, 
, K, K
A kj  
2
,

'

X
X


0
k

j
k
j

SS
MPP==MaxBet
method
method
KM
kMaxBet
 j

Vande
deGeer
Geer(1984)
(1984)Ten
TenBerge
Berge(1986,1988)
(1986,1988)
nVan
S
M
P
=
MaxDiff
method
S M P = Generalized Procrustes Analysis
'
k
' '
kk j j
j
Van de Geer (1984) Ten Berge (1986,1988)
Gower(1975), Ten Berge (1986,1988)
Generalized
Mode canonical
B soft modeling
correlation
approach
methods
Xk
SVD
  
A AAkj A kj
X k  Pk  k W
'
k
'
k
P Pj P' P
A kj   kj A  k j
kjn
n
 kjmethod(1961)
  jk  0,1,1
SMP = Horst
S M P = Soft Modeling Appraoch (Hanafi 2001)
'
Maximize u Au
Maximize u A  mIu
'
Subject to u'k u k  1
k  1,2,..., K
'
Subject to u k u k  1
k  1,2,..., K
'

u
Au  mK
u A  mIu  u Au  mu u
'
'
'
'
uk uk
 1 k  1,2,..., K
Maximize
u1 , u 2 ,..., u K   u Au
Subject to
'
u k u k  1 k  1,2,  , K
'
 A11
A
 21
 

 A m1
A12
A 22

A m2
A1m   u1   1u1 
 A 2 m   u 2   2u 2 


      
  

 A mm  u K  K u K 

u0
Multivariate Eigenvalue Problem
Watterson and Chu(1993)
 A11 A12
A
A
21
22

 


 A m1 A m 2
 A1m   u1   1u1 
 A 2 m   u 2   2u 2  u ' u  1
k k


        k  1,2,  , K
  

 A mm  u K  K u K 
number of solutions  2 p1  p2  ....  pk 