Physical Fluctuomatics
7th~10th Belief propagation
Appendix
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
[email protected]
http://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics (Tohoku
University)
1
Textbooks
Kazuyuki Tanaka: Introduction of Image
Processing by Probabilistic Models,
Morikita Publishing Co., Ltd., 2006 (in
Japanese) , Chapter 5.
References
H. Nishimori: Statistical Physics of Spin Glasses and
Information Processing, ---An Introduction, Oxford
University Press, 2001.
H. Nishimori, G. Ortiz: Elements of Phase Transitions and
Critical Phenomena, Oxford University Press, 2011.
M. Mezard, A. Montanari: Information, Physics, and
Computation, Oxford University Press, 2010.
Physics Fluctuomatics (Tohoku
University)
2
Probabilistic Model for
Ferromagnetic Materials
P(1,1)  P(1.  1)  p
P(1,1)  P(1.  1)
1
 p
2
p


p
p

a1  1 a2  1

p

1
1

1
1
1
P(1.  1)  P(1.  1)  P(1,1)  P(1,1)
Physics Fluctuomatics (Tohoku
University)
3
Probabilistic Model for
Ferromagnetic Materials
p
p

a1  1 a2  1
1
1


1
1
1

# of Blue Lines 1
P(a )  p
(  p) # of Red Lines
2
>
=
>
Prior probability prefers to the configuration with
the least number of red lines.
Physics Fluctuomatics (Tohoku
University)
4
More is different in Probabilistic Model
for Ferromagnetic Materials
p


Sampling by Markov
Chain Monte Carlo
method
p
p


p

1
p
2

1
p
2
Small p
Large p
Disordered
State
Ordered
State
More is different.
Critical Point
(Large fluctuation)
Physics Fluctuomatics (Tohoku
University)
5
Fundamental Probabilistic Models
for Magnetic Materials
exp( ha)
P(a) 
 exp( ha)
a  1
e
h
h0
e

+1
h
1
a  1
Since h is positive, the probablity of up spin is larger
than the one of down spin．
Average
m
 aP(a)  tanh( h)
h ：External Field
a  1
Variance
V a  
2
2
(
a

m
)
P
(
a
)

1

tanh
( h)

a  1
Physics Fluctuomatics (Tohoku
University)
6
Fundamental Probabilistic Models for
Magnetic Materials
P(a1 , a 2 ) 

exp( Ja1a 2 )
 exp( Ja1a2 )
a1  1 a 2  1
a1  1
a2  1
J ：Interaction
J 0
eJ

eJ
1
Since J is positive, (a1,a2)=(+1,+1) and
(1,1) have the largest probability．
Average
m1 
Variance
  a1P(a1, a2 )  0
a1  1 a 2  1
V a1  

1
+1
+1

eJ

+1
1
eJ
1
+1



2
(
a

m
)
 1 1 P(a1, a2 )  1
a1  1 a 2  1
Physics Fluctuomatics (Tohoku
University)
7
Fundamental Probabilistic Models
for Magnetic Materials

a  (a1 , a2 ,, a N )


1
P(a )  exp  E (a ) 
Z

Z   exp(  E (a ))

a

E (a )  h ai  J
iV
E：Set of All the neighbouring Pairs of Nodes
 ai a j
{i , j }E
h
J
(V , E )
h
J




E (a )  E (a ' )  P(a )  P(a ' )

1 N
a
P
(
a
)
Problem: Compute m 

i
N i 1 a
Physics Fluctuomatics (Tohoku
University)
Translational Symmetry
8
Fundamental Probabilistic Models
for Magnetic Materials

a  (a1 , a2 ,, a N )
h
J
ai  1


1
P(a )  exp  E (a ) 
Z

E (a )   h ai  J
iV
h
a a
{i , j }E
i
J
Translational Symmetry
j
Problem: Compute

1 N
m  lim lim
ai P ( a )

h  0 N   N i 1 a
Spontaneous Magnetization
Physics Fluctuomatics (Tohoku
University)
9
Mean Field Approximation
for Ising Model

E a    h ai  J
iV
a a
{i , j }E
i
j
We assume that the probability for configurations
satisfying (ai  m)( a j  m)  0 ({i, j}  E )
are large.
h
ai a j  ma j  mai  m 2

E (a )   (h  4 Jm)ai
Jm
Jm
i
Jm
Jm
iV
Physics Fluctuomatics (Tohoku
University)
10
Mean Field Approximation
for Ising Model


1
P(a )  exp(  E (a ))   Pi (ai )
Z
iV

E (a )   (h  4 Jm)ai
iV
We assume that all random variables ai are independent of
each other, approximately.
N

1
m   ai P(a )  tanh( h  4 Jm)
N i 1 a
m  (m)
Fixed Point Equation of m
Physics Fluctuomatics (Tohoku
University)
11
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
 
*  *
M  M
Physics Fluctuomatics (Tohoku
University)
12
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
y   (x)
0
M*
Physics Fluctuomatics (Tohoku
University)
x
13
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
y   (x)
0
M*
Physics Fluctuomatics (Tohoku
University)
M0
x
14
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
M 1  M 0 
M1
0
y   (x)
M*
Physics Fluctuomatics (Tohoku
University)
M0
x
15
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
M 1  M 0 
M 2  M 1 
M1
0
y   (x)
M * M1
Physics Fluctuomatics (Tohoku
University)
M0
x
16
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
M 1  M 0 
M 2  M 1 
M1
M2
0
y   (x)
M * M1
Physics Fluctuomatics (Tohoku
University)
M0
x
17
Fixed Point Equation and
Iterative Method
•Fixed Point Equation
*
 
y
yx
M M
*
•Iterative Method
M 1   M 0 
M 2   M 1 
M 3   M 2 

M1
M2
0
y   (x)
M * M1
Physics Fluctuomatics (Tohoku
University)
M0
x
18
Marginal Probability Distribution in
Mean Field Approximation
Pi ( ai ) 



P
(
a
  )
 

a1 a 2
a i 1 a i 1
aN
1
exp(( h  4 Jm) ai )
Zi
m
 ai Pi (ai )
h
Jm
Jm
i
Jm
Jm
a i  1
m  tanh(( h  4 J )m)
Physics Fluctuomatics (Tohoku
University)
Jm：Mean Field
19
Advanced Mean Field Method
h
：Effective Field
Bethe Approximation

1
Pi ( ai ) 
exp(( h  4 ) ai )
Zi
1
Pij ( ai , a j ) 
exp(( h  3 )( ai  a j )  Jai a j )
Zi
Pi (ai ) 
 Pij (ai , a j )
a j  1
h
  arctanh (tanh( J ) tanh( h  3 ))
Fixed Point Equation for 
J 

h
Kikuchi Method (Cluster Variation Meth)
Physics Fluctuomatics (Tohoku
University)
20
Average of Ising Model on
Square Grid Graph
h


1
Pa  
exp 
h a i  J  a i a j

Z
{i , j }E
 iV
lim




J
h
J

lim  ai P(a )
h  0 N   a
(a)
(b)
(c)
(d)
Mean Field Approximation
Bethe Approximation
Kikuchi Method (Cluster Variation Method)
Exact Solution （L. Onsager，C.N.Yang）
1/ J
Physics Fluctuomatics (Tohoku
University)
21
Model Representation in Statistical
Physics
Pr{ A1  a1 , A2  a2 ,, AN  a N }  P(a1 , a2 ,, a N )
 

Pr{ A  a}  P(a )
Gibbs
Distribution
Energy Function


1
P (a )  exp(  E (a ))
Z
Free Energy

A  ( A1, A2 ,, AN )
Partition Function

Z   exp(  E (a ))

a

F   ln Z   ln(  exp(  E (a )))
Physics Fluctuomatics (Tohoku
University)

a
22
Gibbs Distribution and Free Energy


1
Gibbs Distribution P(a )  exp(  E (a ))
Z
Free Energy
 

 ln Z   ln   exp(  E (a )) 
 a

Variational Principle of Free Energy Functional F[Q]
under Normalization Condition for Q(a)

min {F [Q] |  Q(a )  1}  F [ P]   ln Z

a
Q



F [Q]   E (a )Q(a )   Q(a ) ln Q(a )

a

a
Free Energy Functional of Trial Probability Distribution Q(a)
Physics Fluctuomatics (Tohoku
University)
23
Explicit Derivation of Variantional
Principle for Minimization of Free Energy
Functional

min {F [Q] |  Q(a )  1}  F [ P]   ln Z
Q

a











LQ  F Q    Q(a)  1   ( E (a )  ln Q(a ))Q(a )    Q(a )  1
 
 
 

a
a
a






LQ
  E (a )  ln Q(a )  1    0
Q(a )




exp

E
(
a
)


ˆ
Q
(
a
)


P
(
a
)

Qˆ (a )  exp  E (a )    1
 exp  E (a )

a
Normalization Condition
Physics Fluctuomatics (Tohoku
University)
24
Kullback-Leibler Divergence and
Free Energy

  Q( a ) 
DQ P   Q(a ) ln     0

 P( a ) 
a




 Q(a )  0,  Q(a )  1



a




Q(a)  P(a)  DQ P  0


1
P (a )  exp(  E (a ))
Z
 


D[Q | P]   Q(a )E (a )   Q(a ) ln Q(a )  ln Z


a
a

F [Q ]
 F [Q]  ln Z


arg min {F[Q] |  Q(a )  1}  arg min {D[Q | P] |  Q(a )  1}
Q

a
Q
Physics Fluctuomatics (Tohoku
University)

a
25
Interpretation of Mean Field
Approximation as Information Theory
Minimization of Kullback-Leibler Divergence between

Q(a )   Qi (ai )


1
Pa   exp(  E (a ))
Z
and
iV

  Q( a ) 
DQ P   Q(a ) ln   

 P( a ) 
a
Marginal Probability Distributions Qi(ai)
are determined so as to minimize D[Q|P]


Qi (ai )   Q(a )   Q(a )

a \ ai
a1
a2
Physics Fluctuomatics (Tohoku
University)
ai 1 ai 1 ai  2
aN
26
Interpretation of Mean Field
Approximation as Information Theory

a  (a1 , a 2 , , a|V | )


1
P(a )  exp  E (a ) 
Z

E ( a )   h ai  J
iV
h
J
ai  1
a a
{i , j }E
i
a1
a2
ai 1 ai 1 ai  2
J
j


Pi (ai )   P(a )   P(a )

a \ ai
h
Translational Symmetry
aN
Problem: Compute


1
1
m
a
P
(
a
)

a
P
(
a
)



i
i i
| V | iV a
| V | iV a 1
i
Magnetization
Physics Fluctuomatics (Tohoku
University)
27
Kullback-Leibler Divergence in
Mean Field Approximation for


1
Ising Model
P(a )  exp  E (a ) 
Z


  Q( a ) 
E
(
a
)   h ai  J  ai a j
DQ P   Q(a ) ln   
iV
{i , j }E

P
(
a
)


a

Qi (ai )   Q(a )

Q(a )   Qi (ai )

a \ ai

      Q(a )
iV
a1
a2
DQ P  FMF Qi | i V   ln Z 
ai 1 ai 1 ai  2
aN
FMF [{Qi | i  V }]  h  Qi ( )
iV   1
J
 ( Q ( ))( Q ( ))    Q  ln Q  
{i , j }E
 1
i
 1
j
Physics Fluctuomatics (Tohoku
University)
iV
 1
i
i
28
Minimization of Kullback-Leibler
Divergence and Mean Field Equation
{Qˆ i ( )}  arg min {D[Q | P] |  Qi ( )  1, i V }
{Qi }
Set of all the neighbouring
nodes of the node i
i  j {i, j}  E
i

Variation


1
ˆ
ˆ
Qi    exp   (h  J   Q j ( ))  (i  V )
Zi
ji   1




ˆ
Z i   exp   (h  J   Q j ( )) 
  1
ji   1


Fixed Point Equations for {Qi|iV}
Physics Fluctuomatics (Tohoku
University)
29
Orthogonal Functional Representation
of Marginal Probability Distribution of
Ising Model

a  (a1 , a2 ,, a N )
ai  1


Qi (ai )   Q(a )      Q(a )

a \ ai
a1 a 2
a i 1 a i 1

mi   ai Qa  

a
1 1
Qi (ai )   mi ai
2 2
 Qi (ai )  c  dai
aN
 ai Qi (ai )
a i  1
(ai 2  1)
1
1
 Qi (ai )   (c  dai ) 2c  c  2  Qi (ai )  2
a i  1
a i  1
a i  1
1
1
 ai Qi (ai )   ai (c  dai ) 2d  d  2  ai Qi (ai )  2 mi
a i  1
a i  1
Physics Fluctuomatics (Tohoku
University)
a i  1
30
Conventional Mean Field Equation in
Ising Model

E ( a )   h ai  J
iV
 ai a j
h
J
h
{i , j }E
J
V
m1  m2    m N  m
Translational Symmetry

1 1
1 1
ˆ
Pi (ai )   P(a )  Qi (ai )   mi ai   mai

2 2
2 2
a \ ai

 1
1
Qˆ i (ai )  exp  (h  J   Qˆ j ( ))ai   exp(( h  4 Jm)ai )
Zi
ji   1

 Zi
m  tanh( h  4 Jm)
Fixed Point Equation
N
| i | 4 (i  V )

1
ai P ( a )  m

N i 1 a
Physics Fluctuomatics (Tohoku
University)
31
Interpretation of Bethe Approximation (1)

a  (a1 , a 2 , , a|V | )


1
P(a )  exp  E (a ) 
Z
Translational Symmetry
h
h
J
ai  1

E ( a )   h ai  J
iV
a a
{i , j }E
i
J
j

1
 ij (ai , a j )
P( a ) 
 ij (ai , a j ) Z  



a {i , j }E
Z {i , j}E
 1

1
 ij (ai , a j )  exp  
hai 
ha j  Jai a j 
| j |
 | i |

Compute Pi (ai )   P(a)   P(a)

a \ ai
and
a1
a2
ai 1 ai 1 ai  2
aN


Pij (ai , a j )   P(a )      P(a )

a \{ ai , a j }
a1
a2
ai 1 ai 1 ai  2
Physics Fluctuomatics (Tohoku
University)
a j 1 a j 1 a j  2
aN
32
Interpretation of Bethe Approximation (2)

  Qa  
DQ P   Q(a ) ln     0

a
 Pa  
1
P x  
 ij ai , a j 

Z {i , j}E
Free Energy
KL Divergence
DQ P  F Q  ln Z 

F Q    Q(a )

a
Qij (ai , a j )

  Q( a )


a \ ai , a j



 ln  ij ai , a j    Q(a ) ln Qa 
{i , j }E

a







    Q(a ) ln  ij ai , a j    Q(a ) ln Qa 



{i , j }E ai a j  a \ ai , a j 
a



   Qij ai , a j ln  ij ai , a j    Q(a ) ln Qa 
{i , j }E ai
aj
Physics Fluctuomatics (Tohoku
University)

a
33
Interpretation of Bethe Approximation (3)
KL Divergence
F Q  
Free Energy
DQ P  F Q  ln Z 
Q  ,   ln   ,  
 
 
ij
{i , j }E
ij


  Q(a ) ln Qa 

a



Qi (ai )   Q(a )

a \ai
 Qij  ,  ln  ij  ,  
{i , j }E 

1
Pa  
 ij ai , a j 

Z {i , j}E

Qij (ai , a j )   Q(a )


a \ ai , a j


  Qi   ln Qi  
Bethe Free
iV 
Energy


    Qij  ,   ln Qij  ,     Qi   ln Qi     Q j   ln Q j  
{i , j }E   



Physics Fluctuomatics (Tohoku
University)
34
Interpretation of Bethe Approximation (4)


DQ P  FBethe Qi , Qij   ln Z

   Q  ,  ln   ,     Q  ln Q  
FBethe Qi , Qij  
{i , j }E 

ij
ij
iV
i

i
arg min DQ P   arg min F


    Qij  ,   ln Qij  ,     Qi   ln Qi     Q j   ln Q j  
{i , j }E  




Q
Q


arg min DQ P   arg min FBethe Qi , Qij 
Qi ,Qij 
Q
Q  ,    1
 Q    
 
i
ij
Qi     Qij  ,  
Physics Fluctuomatics (Tohoku
University)

35
Interpretation of Bethe Approximation (5)


arg min  FBethe Qi , Qij  Qi     Qij  ,  ,  Qi     Qij  ,    1
Qi ,Qij  



 


Lagrange Multipliers to ensure the constraints




LBethe Qi , Qij   FBethe Qi , Qij 


   i ,{i , j}   Qi     Qij  ,  
iV ji 







  i   Qi    1    ij   Qij  ,    1
iV
 
 {i , j}E   

Physics Fluctuomatics (Tohoku
University)
36
Interpretation of Bethe Approximation (6)


LBethe Qi , Qij   FBethe Qi , Qij     i ,{i , j}   Qi     Qij  ,  
iV ji 











  i   Qi    1    ij   Qij  ,    1
iV
 
 {i , j}E   

   Qij  ,   ln  ij  ,     Qi   ln Qi  
{i , j }E 


iV

















Q

,

ln
Q

,


Q

ln
Q


Q

ln
Q




ij
i
i
j
j
  ij

{i , j }E  










   i ,{ j , j}   Qi     Qij  ,     i   Qi    1    ij   Qij  ,    1
iV ji 


 iV  
 {i , j}E   

•
Extremum Condition



LBethe Qi , Qij   0
Qi xi 



LBethe Qi , Qij   0
Qij xi , x j 
Physics Fluctuomatics (Tohoku
University)
37
Interpretation of Bethe Approximation (7)



LBethe Qi , Qij   0
Qi xi 



LBethe Qi , Qij   0
Qij xi , x j 
Extremum
Condition
 1
 Q12 a1 , a2   12 a1 , a2  exp 1,{1, 2} (a1 )  2,{1, 2} (a2 ) 
Qi ai   exp 

(
a
)

i ,{i , k }
i 

 | i | 1 ki

exp i ,{i , j} (ai )  
M
k i
(ai )
ki \ j
Q1 a1  
1
1
M 21 a1 M 31 a1  Q12 a1 , a2  
M 31 a1 M 41 a1 M 51 a1 
Z1
Z12
 M 41 a1 M 51 a1 
 12 a1 , a2 M 62 a2 M 72 a2 M 82 a2 
Physics Fluctuomatics (Tohoku
University)
38
Interpretation of Bethe Approximation (8)



LBethe Qi , Qij   0
Qi xi 
M 31
4
M 41

3
1
M 31
M 21
2
4
M 41
3
1
12
M 51
M 51
5


LBethe Qi , Qij   0
Qij xi , x j 
8M
2
Extremum
Condition
82
M 72
7
M 62
5
6
Q1 a1  
1
1
M 21 a1 M 31 a1  Q12 a1 , a2  
M 31 a1 M 41 a1 M 51 a1 
Z1
Z12
 M 41 a1 M 51 a1 
 12 a1 , a2 M 62 a2 M 72 a2 M 82 a2 
Physics Fluctuomatics (Tohoku
University)
39
Interpretation of Bethe Approximation (9)
Q1     Q12  ,  

M 12  
  12  ,  M 31  

 M 41  M 51  
Message Update Rule
M 31
4
M 41
3
M 31
M 21
1
2
4
M 41
3
1
8M
W12
M 51
M 51
5
2
82
M 72
7
M 62
5
6
Q1 a1  
1
1
M 21 a1 M 31 a1  Q12 a1 , a2  
M 31 a1 M 41 a1 M 51 a1 
Z1
Z12
 M 41 a1 M 51 a1 
 12 a1 , a2 M 62 a2 M 72 a2 M 82 a2 
Physics Fluctuomatics (Tohoku
University)
40
Interpretation of Bethe Approximation (10)
12  ,  M 31  M 41  M 51  

M 12    
 12  ,  M 31  M 41  M 51  


Message Passing Rule of
Belief Propagation
M 31
4
M 41
4
3
1
M 12
2
3
8
1
2
5
6

a2
7
3
M 51
5
=
4
It corresponds to Bethe approximation
in the statistical mechanics.
Physics Fluctuomatics (Tohoku
University)
1
2
5
41
Interpretation of Bethe Approximation (11)
 ij  ,    M k i  

  1
ki \ j
M i  j   
  ij  ,    M k i  
  1  1
ki \ j
M i  j    exp i  j 
i  j  arctanh (tanh( J ) tanh( h 
i  j  

ki \ j
k i
))
Translational Symmetry
  arctanh (tanh( J ) tanh( h  3 ))
Physics Fluctuomatics (Tohoku
University)
42
Summary
Statistical Physics and Information Theory
Probabilistic Model of Ferromagnetism
Mean Field Theory
Gibbs Distribution and Free Energy
Free Energy and Kullback-Leibler Divergence
Interpretation of Mean Field Approximation
as Information Theory
Interpretation of Bethe Approximation as
Information Theory
Physics Fluctuomatics (Tohoku
University)
43