Statistical Mechanical Approach to
Probabilistic Information Processing:
Cluster Variation Method and Belief Propagation
Kazuyuki Tanaka
GSIS, Tohoku University, Sendai, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/
Collaborators
Muneki Yasuda (GSIS, Tohoku University, Japan)
Sun Kataoka (GSIS, Tohoku University, Japan)
D. M. Titterington (Department of Statistics, University of Glasgow, UK)
13 March, 2013
WPI, Tohoku University, Sendai
1
Outline
1. Supervised Learning of Pairwise Markov
Random Fields by Loopy Belief Propagation
2. Bayesian Image Modeling by Generalized
Sparse Prior
3. Noise Reductions by Generalized Sparse Prior
4. Concluding Remarks
13 March, 2013
WPI, Tohoku University, Sendai
2
Probabilistic Model and
Belief Propagation
Bayesian Networks
Bayes Formulas
Probabilistic
Information Processing
M i  j  fi   Constant

 U ij fi , f j
fj
Probabilistic Models
Belief Propagation
=Bethe Approximation
  M j k  f j 
kj
j  k { j , k}  E |
Markov Random Fields
P f  
U ij  fi , f j 
{i , j}E
i
j
Message
=Effective Field
V: Set of all the nodes (vertices) in graph G G  (V , E )
E: Set of all the links (edges) in graph G
13 March, 2013
WPI, Tohoku University, Sendai
3
Mathematical Formulation of
Belief Propagation
Similarity of Mathematical Structures between Mean Field
Theory and Belief Propagation
Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding
corrupted messages, Europhys. Lett. 44 (1998).
M. Opper and D. Saad (eds), Advanced Mean Field Methods
---Theory and Practice (MIT Press, 2001).
Generalization of Belief Propagation
S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy
approximations and generalized belief propagation algorithms,
IEEE Transactions on Information Theory, 51 (2005).

Interpretations of Belief Propagation based on Information
Geometry
S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy,
and information geometry, Neural Computation, 16 (2004).
13 March, 2013
WPI, Tohoku University, Sendai
4
Generalized Extensions of Belief Propagation based
on Cluster Variation Method
Generalized Belief Propagation
J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy
approximations and generalized belief propagation algorithms, IEEE
Transactions on Information Theory, 51 (2005).
Cluster Variation Method
R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 81 (1951).
T. Morita: Cluster variation method of cooperative phenomena and its
generalization I, J. Phys. Soc. Jpn, 12 (1957).
13 March, 2013
WPI, Tohoku University, Sendai
5
Applications of Belief Propagations
Image Processing
K. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J.
Phys. A, 35 (2002).
A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing,
Proceedings of IEEE, 90 (2002).
Low Density Parity Check Codes
Y. Kabashima and D. Saad: Statistical mechanics of low-density parity-check codes
(Topical Review), J. Phys. A, 37 (2004).
S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density
parity-check codes, IEEE Transactions on Information Theory, 50 (2004).
CDMA Multiuser Detection Algorithm
Y. Kabashima: A CDMA multiuser detection algorithm on the basis of belief
propagation, J. Phys. A, 36 (2003).
T. Tanaka and M. Okada: Approximate belief propagation, density evolution, and
statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on
Information Theory, 51 (2005).
Satisfability Problem
O. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase
transitions in optimization problems, Theoretical Computer Science, 265 (2001).
M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random
satisfability problems, Science, 297 (2002).
13 March, 2013
WPI, Tohoku University, Sendai
6
Interpretation of Belief Propagation
based on Information Theory
 Q f  
  F [Q]  ln Z
DQ P    Q( f ) ln
 P f  
f

1
P f  
U ij fi , f j

Z {i, j}E


Qij ( fi , f j ) 
Q f     Qi ( fi )  



Q
(
f
)
Q
(
f
)
Bethe Free Energy
i
i
j
j
 iV
 {i, j}E

Trial 
KL Divergence


F Q    Q( f )   ln U fi , f j  ln Q f 


Entropy
f
 {i, j}E


Qi ( fi ) 
 Q( f )

Qij ( fi , f j ) 
f \{ f }
i
One-Body Distribution
 Q( f )
f \{ f i , f j }
Two-Body Distribution
Marginal Probability
13 March, 2013
WPI, Tohoku University, Sendai
7
Supervised Learning of Pairwise Markov
Random Fields by Loopy Belief Propagation
Prior Probability of natural images is assumed to be the following
pairwise Markov random fields:
Pr{F  f }  P f  
U  fi  f j 
{i , j}E
f i  0,1,, q  1
13 March, 2013
WPI, Tohoku University, Sendai
8
Supervised Learning of Pairwise Markov
Random Fields by Loopy Belief Propagation
Supervised Learning Scheme by Loopy Belief Propagation in
Pairwise Markov random fields:




q 1 q 1
 
1
P f   U fi  f j , ln U fi  f j    K m, n m  fi  n f j
2 m 0 n 0
{i , j}E
q 1 q 1

  m  fi  n  fi    m,n ,  0  fi  
fi 0 fi 0
q 1
K m,0  K 0, m  1  i    m  fi ln Qi  fi 
fi 0

1
q
Histogram from
Supervised Data
q 1 q 1
    m  fi  0  f j Qij  fi , f j 
{i , j}i f i  0 f j  0
K m, n  K n, m 
13 March, 2013
q 1 q 1
   m  fi  0  f j Qij  fi , f j 
fi 0 f j 0
WPI, Tohoku University, Sendai
M. Yasuda, S. Kataoka and
K.Tanaka, J. Phys. Soc. Jpn,
Vol.81, No.4, Article
No.044801, 2012.
9
Supervised Learning of Pairwise Markov
Random Fields by Loopy Belief Propagation
Supervised Learning Scheme by Loopy Belief Propagation in
Pairwise Markov random fields:


 1

0
.
45

P f   exp  ln U fi  f j  ~ exp    fi  f j


 2

{i , j}E
 {i, j}E




13 March, 2013

WPI, Tohoku University, Sendai
10
Bayesian Image Modeling by
Generalized Sparse Prior
Assumption: Prior Probability is given
as the following Gibbs distribution with
the interaction  between every nearest
neighbour pair of pixels:
 1
p
1
Pr{F  f | p,  }  P f p,   
exp     f i  f j 
Z  p,  
 2 {i , j}E

f i  0,1,, q  1
p=0: q-state
Potts model
p=2: Discrete Gaussian
Graphical Model
13 March, 2013
WPI, Tohoku University, Sendai
11
Bayesian Image Modeling by
Generalized Sparse Prior
 1
p
1
Pr{F  f | p,  }  P f p,   
exp     f i  f j 
Z  p,  
 2 {i , j}E

q=16
p=0.2
1
E
 
{i , j}E f
f i  f j P  f p,  
p
Loopy Belief Propagation
13 March, 2013

WPI, Tohoku University, Sendai

1
ln Z  p,  
V

12
Bayesian Image Modeling by Generalized
Sparse Prior: Conditional Maximization
of Entropy
Pr{F  f | p, u}


 arg max   P( z ) ln P( z )
P( f ) 
 z
 z

z {i , j }E
p
i
 z j P( z )  u E
 
Pr{F  f | p, u}  P  f p,   p, u 
Lagrange Multiplier
 1

exp    p, u   f i  f j
 2
Z  p,   p, u 
{i , j}E

1
K. Tanaka, M. Yasuda and D.
M. Titterington: J. Phys. Soc.
Jpn, 81, 114802, 2012.
13 March, 2013





p



Loopy Belief Propagation
WPI, Tohoku University, Sendai
13
Bayesian Image Modeling by
Generalized Sparse Prior:
Conditional Maximization of Entropy
Critical Point by
Linear Response in
LBP
q=16
p=0.2
Mandrill
Lena
K. Tanaka, M. Yasuda and D. M.
Titterington: J. Phys. Soc. Jpn, 81,
114802, 2012.
13 March, 2013
WPI, Tohoku University, Sendai
14
Prior Analysis by LBP and Conditional Maximization
of Entropy in Generalized Sparse Prior
Prior Probability
 1
Pr{F  f | p, u}  P f p,   p, u  
exp     p, u   f i  f j
Z  p,   p, u 
{i , j }E
 2
1
q=16 p=0.2
  p, u , u 
*
13 March, 2013
*
q=16 p=0.5
K. Tanaka, M. Yasuda and
D. M. Titterington:
J. Phys. Soc. Jpn, 81, 114802, 2012.
1
u* 
f i*  f j*

E {i , j}E
WPI, Tohoku University, Sendai
p
15
p




Prior Analysis by LBP and Conditional
Maximization of Entropy in Generalized
Sparse Prior
ln Pr{F  f * | p, u *}
Free Energy of Prior
Log-Likelihood for
1
*
*
* p
p when the original     p, u   f i  f j  ln Z  p,   p, u * 
2
{i , j }E
image f* is given
LBP
q=16
p*  arg max ln Pr{F  f * | p, u *}
p
1
u 
E
*

{i , j }E
fi  f
*
* p
j
K. Tanaka, M. Yasuda and D.
M. Titterington: J. Phys. Soc.
Jpn, 81, 114802, 2012.
13 March, 2013
WPI, Tohoku University, Sendai
16
Noise Reductions by Generalized
Sparse Prior
Posterior




Pr{Original Image | Degraded Image }
Assumption: Degraded image is
generated from the original image by
Additive White Gaussian Noise.
Degradation Process




 Pr{Degraded Image | Original Image }
Prior


 Pr{Original Image }
Pr{G  g | F  f ,  }
1

2
  exp  
(
g

f
)

i
i
2
 2

iV
 0
13 March, 2013
WPI, Tohoku University, Sendai
17
Noise Reductions by Generalized Sparse Prior
Posterior




Pr{Original Image | Degraded Image }
Posterior Probability
Pr{F  f | G  g , p,  , u}
Degradation Process



 Prior


 Pr{Degraded Image | Original Image } Pr{Original Image }



 arg max   P( z ) ln P( z )
P( f )
 z



z

z
P
(
z
)

u
E
z {i
i
j


, j }E

2
2
( zi  g i ) P ( z )   V 


z iV

p
Lagrange Multipliers
Pr{F  f | G  g, p,  , u}  P f g, p, B, C 
 1
1
1
2
 fi  gi   C  fi  f j

exp  

Z  g , p , B, C 
2 {i , j}E
 2 B iV
K. Tanaka, M. Yasuda and D.
M. Titterington: J. Phys. Soc.
Jpn, 81, 114802, 2012.
13 March, 2013
WPI, Tohoku University, Sendai
p
18




Noise Reductions by
Generalized Sparse Priors
and Loopy Belief Propagation
Original Image
Degraded Image
Restored
Image
K. Tanaka, M. Yasuda
and
D. M. Titterington:
J. Phys. Soc. Jpn, 81,
114802, 2012.
p=0.2
13 March, 2013
p=0.5
WPI, Tohoku University, Sendai
p=1
19
Noise Reductions by
Generalized Sparse Priors
and Loopy Belief Propagation
Original Image
Degraded Image
Restored
Image
K. Tanaka, M. Yasuda
and
D. M. Titterington:
J. Phys. Soc. Jpn, 81,
114802, 2012.
p=0.2
13 March, 2013
p=0.5
WPI, Tohoku University, Sendai
p=1
20
Summary
Formulation of Bayesian image modeling for
image processing by means of generalized sparse
priors and loopy belief propagation are proposed.
Our formulation is based on the conditional
maximization of entropy with some constraints.
In our sparse priors, although the first order
phase transitions often appear, our algorithm
works well also in such cases.
13 March, 2013
WPI, Tohoku University, Sendai
21
References
1.
2.
3.
4.
5.
6.
7.
S. Kataoka, M. Yasuda, K. Tanaka and D. M. Titterington: Statistical Analysis of the
Expectation-Maximization Algorithm with Loopy Belief Propagation in Bayesian Image
Modeling, Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3,
pp.50-63,2012.
M. Yasuda and K. Tanaka: TAP Equation for Nonnegative Boltzmann Machine:
Philosophical Magazine: The Study of Condensed Matter, Vol.92, Nos.1-3, pp.192-209,
2012.
S. Kataoka, M. Yasuda and K. Tanaka: Statistical Analysis of Gaussian Image Inpainting
Problems, Journal of the Physical Society of Japan, Vol.81, No.2, Article No.025001, 2012.
M. Yasuda, S. Kataoka and K.Tanaka: Inverse Problem in Pairwise Markov Random
Fields using Loopy Belief Propagation, Journal of the Physical Society of Japan, Vol.81,
No.4, Article No.044801, pp.1-8, 2012.
M. Yasuda, Y. Kabashima and K. Tanaka: Replica Plefka Expansion of Ising systems,
Journal of Statistical Mechanics: Theory and Experiment, Vol.2012, No.4, Article
No.P04002, pp.1-16, 2012.
K. Tanaka, M. Yasuda and D. M. Titterington: Bayesian image modeling by means of
generalized sparse prior and loopy belief propagation, Journal of the Physical Society of
Japan, Vol.81, No.11, Article No.114802, 2012.
M. Yasuda and K. Tanaka: Susceptibility Propagation by Using Diagonal Consistency,
Physical Review E, Vol.87, No.1, Article No.012134, 2013.
13 March, 2013
WPI, Tohoku University, Sendai
22