Symmetry Breaking Bifurcation
of the Distortion Problem
Albert E. Parker
Complex Biological Systems
Department of Mathematical Sciences
Center for Computational Biology
Montana State University
Collaborators:
Tomas Gedeon
Alexander Dimitrov
John P. Miller
Zane Aldworth
Bryan Roosien
A Class of Problems
We use
 Numerical continuation
 Bifurcation theory with symmetries
to analyze a class of optimization problems of the form
max F(q,)=max (G(q)+D(q)).
q
q

The goal is to solve  for  = B (0,), where:


y ) |  q( z | y )  1, y  Y   n
zZ

•
. : q( z |

•
•
•
•
•
G and D are infinitely differentiable in .
G is strictly concave.
D is convex.
G and D must be invariant under relabeling of the classes.
The hessian of F is block diagonal with N blocks {B} and B=B if q(z|y)=
q(z|y) for every yY.
B 0 0 

 1

q F   0  0 
0 0 B 
N 

Problems in this class
• Deterministic Annealing (Rose 1998)
max H(Z|Y) +  D(Y,Z)
q
Clustering Algorithm
• Rate Distortion Theory (Shannon ~1950)
max –I(Y,Z) +  D(Y,Z)
q
Optimal Source Coding
• Information Distortion (Dimitrov and Miller 2001)
max H(Z|Y) +  I(X,Z)
q
Used in neural coding.
• Information Bottleneck Method (Tishby, Pereira, Bialek 2000)
max –I(Y,Z) +  I(X,Z)
q
Used for document classification, gene expression,
neural coding and spectral analysis
Solution q* of max F(q) when p(X,Y):= 4 gaussian blobs
q
X
Q(Y |X)
Y
q(z|y)
Z
Q*(Z|X )
The optimal quantizer q*(Z|Y)
p(X,Y)
I(X,Z) vs. N
Z
Z
Some nice properties of the
problem
 The feasible region , a product of simplices, is nice.
Lemma
 is the convex hull of vertices ().
y1
 The optimal quantizer q* is DETERMINISTIC.
y2
y1
y3
y2
Theorem The extrema of  lie generically on the vertices of ..
Corollary The optimal quantizer is invariant to small perturbations
in the model.
y3
Annealing G(q)+D(q)
At    *  1, we observe bifurcatio n!
max G(q)   * D(q)
q
At   0
max G ( q )
q
Annealing G(q)+D(q)
At  = B (0,)
max G(q)  D(q)
q
At   0
max G ( q )
q
Conceptual Bifurcation Structure
q*
q* 
1
N

Conceptual Bifurcation Structure
q* (YN|Y)
q* 
1
N
Bifurcations of q*()

Observed Bifurcations for the 4 Blob Problem
The Dynamical System
Goal: To efficiently solve  maxq  (G(q) +  D(q)) for each , incremented
in sufficiently small steps, as   B.
Method: Study the equilibria of the of the flow

 q 

 
    q , L (q,  ,  ) :  q ,  G(q)   D(q)    y   q( z | y)  1 

yY
 z

 

•
The Jacobian wrt q of the K constraints {zq(z|y)-1} is J = (IK IK … IK).
•
The first equilibrium is q*(0 = 0)  1/N.
•
 q F
 q., L (q,  ,  )   T
 J
J

0
determines stability and location of
bifurcation.
Some theoretical results
Assumptions:
• Let q* be a local solution to  and fixed by SM .
• Call the M identical blocks of q F (q*,): B. Call the other N-M blocks of q F
(q*,): {R}.
• At a singularity (q*,*,*), B has a single nullvector v and R is nonsingular for
every .
• If M<N, then BR-1 + MIK is nonsingular.
Theorem: (q*,*,*) is a bifurcation of equilibria of  if and only if
q, L(q*,*, *) is singular.
Theorem: If q, L(q*,*,*) is singular then q F (q*,*) is singular.
Theorem: If (q*,*,*) is a bifurcation of equilibria of , then  *  1.
Theorem: dim (ker q F (q*,* )) = M with basis vectors w1,w2, … , wM
v if  is the i th unresolved class
[ wi ]  
0 otherwise
Theorem: dim (ker q, L (q*,*,*)) = M-1 with basis vectors
M 1
 wi   wM 

   
0
0
i 1
  
Bifurcations with symmetry
To better understand the bifurcation structure, we capitalize on
the symmetries of the optimization function F(q,).
The “obvious” symmetry is that F(q,) is invariant to relabeling
of the N classes of Z
The symmetry group of all permutations on N symbols is SN.
q 
The action of SN on   and q, L (q, , ) is represented by the
 
finite


Lie Group
0 
 

 : 
 0

 K n
 |   P
I 

K K 

n K
where P is a “block permutation” matrix.
What do the bifurcations look like?
The Equivariant Branching Lemma gives the existence of
bifurcating solutions for every subgroup of  which fixes
a one dimensional subspace of ker q,L (q*,,).
Theorem:
Let (q*,*,*) be a singular point of the flow
 q 
    q , L (q,  ,  )

 
such that q* is fixed by SM. Then there exists M
bifurcating solutions, (q*,*,*) + (tuk,0,(t)), each fixed by
group SM-1, where
( M  1)v

[u k ]   v
0

if  is the k th unresolved class
if   k is any other unresolved class
otherwise
For the 4 Blob problem:
The subgroups and bifurcating directions of the
observed bifurcating branches
subgroups:
bif direction:
S4
S3
S2
(-v,-v,3v,-v,0)T (-v,2v,0,-v,0)T
1
(-v,0,0,v,0)T
… No more bifs!
Partial lattice of the isotropy subgroups of S4
(and associated bifurcating directions)
 3v 
 
 v
 v
 v
 
0 
S4
 v
 
 3v 
 v
 v
 
0 
S3
S2 S2 S2
 0 
 
 2v 
 v
 v
 
0 
 0 
 
 v
 2v 
 v
 
0 
 0 
 
 v
 v
 2v 
 
0 
S3
S3
S2 S2 S2
 2v 
 
 0 
 v
 v
 
0 
 v
 
 0 
 2v 
 v
 
0 
 v
 
 0 
 v
 2v 
 
0 
1
S2 S2 S2
 2v 
 
 v
 0 
 v
 
0 
 v
 
 2v 
 0 
 v
 
0 
 v
 
 v
 0 
 2v 
 
0 
 v
 
 v
 3v 
 v
 
0 
S3
 v
 
 v
 v
 3v 
 
0 
S2 S2 S2
 2v 
 
 v
 v
 0 
 
 0 
 v
 
 2v 
 v
 0 
 
 0 
 v
 
 v
 2v 
 0 
 
 0 
Bifurcation Structure
 3 F (q* ,  * )
[v]k [v]m [v]l .
Let T(q*,*) = 
k , m ,l qk qm ql
Transcritical or Degenerate?
Theorem: If T(q*,*)  0 and M>2, then the bifurcation at (q*,*) is
transcritical. If T(q*,*) = 0, it is degenerate.
Branch Orientation?
Theorem: If T(q*,*) > 0 or if T(q*,*) < 0, then the branch is supercritical
or subcritical respectively. If T(q*,*) = 0 , then 4qqqq F(q,) dictates
orientation.
Branch Stability?
Theorem: If T(q*,*)  0, then all branches fixed by SM-1 are unstable.
Other Branches
The Smoller-Wasserman Theorem ascertains the existence
of bifurcating branches for every maximal isotropy
subgroup.
Theorem: If M is a composite number, then there exists
bifurcating solutions with isotropy group <p> for every
element  of order M in  and every prime p|M. The
bifurcating direction is in the p-1 dimensional subspace of
ker q,L (q*,,) which is fixed by <p>.
We have never numerically observed solutions fixed by
<p> and so perhaps they are unstable.
Lattice of the maximal isotropy subgroups <p> in S4
S4
 (1423) 
A4
 (1324) 
 1234  
2
 1324  
2
 v 
 
 v 
 v
 
 v
 
 1243 
2
 v 
 
 v
 v
 
 v 
 
 v 
 
 v
 v 
 
 v
 
The efficient algorithm
Let q0 be the maximizer of maxq G(q), 0 =1 and s > 0. For k  0, let (qk , k
) be a solution to maxq G(q) +  D(q ). Iterate the following steps until
K =  max for some K.
   qk 

1. Perform  -step: solve  q , L (qk , k ,  k )
        q , L (qk , k ,  k )
  k
   qk 
for     and select  k+1 = k + dk where


k

dk = s /(||qk ||2 + ||k ||2 +1)1/2.
2. The initial guess for qk+1 at  k+1 is qk+1(0) = qk + dk  qk .
3. Optimization: solve maxq G(q) +  k+1 D(q) to get the maximizer qk+1 ,
using initial guess qk+1(0) .
4. Check for bifurcation: compare the sign of the determinant of an identical
block of each of q [G(qk) +  k D(qk)] and q [G(qk+1) +  k+1 D(qk+1)]. If a
bifurcation is detected, then set qk+1(0) = qk + d_k u where u is in Fix(H)
and repeat step 3.