Correctness of Loopy Belief Propagation
for Hyperparameter Estimation in
Probabilistic Image Processing
Kazuyuki Tanaka
Graduate School of Information, Tohoku University, Japan
http://www.statp.is.tohoku.ac.jp/~kazu/
Collaborators
Masato Okada (RIKEN, Japan)
Hayaru Shouno (Yamaguchi University, Japan).
December 18, 2003
Glasgow University
1
Image Processing
and Magnetic Material/Statistical Mechanics
Regular lattice consisting of a lot of nodes.
Interactions among neighboring nodes
Output images are
determined from a
priori information and
given data.
Similarity
Ordered states are
determined from
interactions and
external fields.
1 1 1

1
1 1 1
9

1 1 1
Para
It is difficult for conventional
filters to treat fluctuation in data.
December 18, 2003
Glasgow University
Critical Ferro
Fluctuation is enhanced
near critical temperature.
2
Belief Propagation
and Statistical Mechanics
Probabilistic model with no loop
Belief Propagation
(Lauritzen, Pearl)
Probabilistic model with some loops
Probabilistic model with no loop
Transfer Matrix =Belief Propagation
Bethe Approximation
= Loopy Belief Propagation
Probabilistic model
with some loops
December 18, 2003
Transfer Matrix
Recursion Formula
for Beliefs and Messages
Bethe/Kikuchi Method
Cluster Variation Method
Glasgow University
3
Belief Propagation
in Graphical Model
Y. Weiss and W. T. Freeman, Correctness of belief
propagation in Gaussian graphical models of arbitrary
topology, Neural Computation, 13, 2173 (2001).
Y. Weiss, Comparing the mean field method and belief
propagation for approximate inference in MRFs,
Advanced Mean Field Methods, MIT Press (2001).
Average is good but variance is not enough.
December 18, 2003
Glasgow University
4
Gaussian Graphical Model






2 
2 
f x, y  f x 1, y  
f x, y  g x, y 


1
2

  f   exp     2





Z
2
f x, y  f x, y 1


 ( x, y ) 
2





Free energy  ln Z
y

and average

 f x, y   f df
can be calculated
x
December 18, 2003
by using the multi - dimensiona l
Gauss integral formula.
Glasgow University
5
Main Purpose
Correctness of the hyperparameter estimation
in the Bethe Approximation (Loopy Belief
Propagation) in Gray-Level Image Restoration
Comparison in the maximum marginal
likelihood estimation for Gaussian graphical
model.
Bethe Approximation (Loopy Belief
Propagation)
Mean-Field Method and Exact Result.
December 18, 2003
Glasgow University
6
Bayesian Image Restoration
Noise
transmission
Degraded Image
Original Image
P(Original Image Degraded Image )
P(Degraded Image Original Image ) P(Original Image )

P(Degraded Image )
December 18, 2003
Glasgow University
7
Bayes Formula and
Probabilistic Image Processing
y
f
Prior Probability
Original Image

Pixel
Pg f ,  
P f  

Degradation Process

f  f x, y x, y   
xPosterior
Probability
g
g
P f g,  ,   
Degraded Image


g  g x, y x, y   
P g f ,  P  f  
P g  ,  
hx, y  g,  ,     z x, y Pz g ,  ,  dz
December 18, 2003
Glasgow University
8
Hyperparameter Estimation by
Maximization of Marginal Likelihood
ˆ , ˆ   arg max Pg  ,  
 , 
Pg  ,     Pg z ,  P z  dz
fˆx, y  hx, y  g , ˆ , ˆ    z x, y Pz g , ˆ , ˆ dz
y

P f  
x
Original Image
f  f x, y x, y  
December 18, 2003
Pg f ,  
f
g
g
Marginalize
Pg  ,  
Marginal Likelihood
Glasgow University
g
Degraded Image
g  g x, y x, y  
9
Degradation Process
and Prior Probability
Degradation Process
f x , y , g x , y   , 



1
1
2
P g f ,    
exp  
f x, y  g x, y 
2
2


2



( x, y )
Prior Probability
 
2
P f   
exp  
f x, y  f x 1, y 

ZPR   ( x, y )
 2


1
 
  exp  
f x, y  f x, y 1
 2
( x, y )

December 18, 2003
Glasgow University


2


10
Bayes Formula
and Posterior Probability
P f g,  ,   
P g f ,  P  f  
P g  ,  




1

 x, y f x, y  f x, y , f x 1, y  f x, y , f x. y 1

Z POS  g ,  ,   ( x, y )








1
2
 x, y f x, y  exp   2 f x, y  g x, y 
 2



 1
 f x, y , f x ', y '  exp    f x, y  f x ', y '
 2
Pg  ,     Pg z,  Pz  dz 
December 18, 2003
Glasgow University

2


ZPOS  g,  ,  
2 

||
ZPR  
11
Marginal Probability
Px , y ( f x , y )     f x , y  z x , y Pz g ,  ,  dz
hx, y  g ,  ,     z x, y P z g ,  ,  dz



December 18, 2003
Px, y  d
Glasgow University
12
Exact Results of Gaussian
Graphical Model









f x, y  g x, y 2 
f x, y  f x 1, y 2  

1
2

  f   exp     2




Z
2
(
x
,
y
)



f

f



x, y
x, y 1
2





Multi-dimensional Gauss integral formula
2   exp   1 g TC I  C 1 g  y


det I  C 
 2

Z


f x, y   f df   I  C 1 g

x
x, y C x' , y '  4 x, x ' y , y '   x, x '1 y , y '   x, x '1 y , y '
  x, x ' y , y '1   x, x ' y , y '1
December 18, 2003
Glasgow University
13
Mean-Field Approximation
for Gaussian Graphical Model
mx, y   f x, y   f df
Substitute
 f x,y  mx,y  f x1,y  mx1,y   0
 f x,y  f x 1,y 2  f x,y 2  2 f x,y f x 1,y  f x 1,y 2
 2  2 f x , y f x 1, y
 2  2mx , y f x 1, y  2mx 1, y f x , y  mx , y mx 1, y
Mean-Field Equation
mx , y 
December 18, 2003
g x, y  mx 1, y  mx 1, y  mx, y 1  mx, y 1
  4
Glasgow University
14
Mean Field Approximation In Variational
Principle of Mean Field Free Energy in
Gaussian Graphical Model
1 
 f  
Wx , y f x , y


Z ( x, y )


x 1, y


W
(
f
,
f
)
x
,
y
x
,
y
x

1
,
y




Wx, y ( f x, y )Wx 1, y ( f x 1, y ) 
(
x
,
y
)





x , y 1

W
( f x, y , f x, y 1 ) 
x, y


  Wx, y ( f x, y )Wx, y 1( f x, y 1 ) 
 ( x, y )



 

exp  
f x, y  g x, y 2 
 2

Wx , y f x , y 

 
2
exp



g

d
x
,
y
  2













 

exp  
f x, y  g x, y 2 
f x', y '  g x', y ' 2 
f x, y  f x', y ' 2 
2
2
 2

Wxx,'y, y ' f x, y , f x', y ' 
 
 
2 
2 
2


exp



g


'

g




'

dd '
x
,
y
x
'
,
y
'
   2
2
2


December 18, 2003



Glasgow University


15
Factorizable Form in Mean Field Approximation
 Q z  
dz
DQ     Q( z ) ln 
  z  
Kullback-Leibler Divergence


Qx, y ( f x, y )    f x, y  z x, y Qz dz
Q( f ) 
 Qx, y  f x, y 
( x, y )
December 18, 2003
Glasgow University
16
Mean-Field Approximation
for Gaussian Graphical Model
Q( f ) 
 Qx, y  f x, y 
( x, y )


DQ    FMF Qx, y  ln Z   0
FMF [{Qx, y }] 
 Qz dz  1
 D[Qx, y | Wx, y ]
( x, y )

 D[Qx, yQx 1, y | Wxx,y1, y ]  D[Qx, y | Wx, y ]  D[Qx 1, y | Wx 1, y ]
( x, y )

 D[Qx, yQx, y 1 | Wxx,,yy 1]  D[Qx, y | Wx, y ]  D[Qx, y 1 | Wx, y 1]
( x, y )
 : Upper Bound of Free Energy
F[  ]   ln Z  FMF Qx, y


Qx, y ( f x, y )    f x, y  z x, y Qz dz
December 18, 2003
Glasgow University
17
Mean-Field Approximation and Variational
Principle of Mean-Field Free Energy for
Gaussian Graphical Model
 x, y  arg Qmin FMF Qx, y 
x, y
FMF [{Qx, y }] 


Qx, y ( f x, y )    f x, y  z x, y Qz dz
 D[Qx, y | Wx, y ]
( x, y )
Mean-Field
x 1, y


D
[
Q
Q
|
W
]

D
[
Q
|
W
]

D
[
Q
|
W
]
 x, y x 1, y x, y
x, y
x, y
x 1, y
x 1, y
Free Energy
( x, y )
  D[Qx, y Qx, y 1 | Wxx,,yy 1 ]  D[Qx, y | Wx, y ]  D[Qx, y 1 | Wx, y 1 ]

( x, y )
MF Eq.
 x, y  f x, y      f x, y  z x, y  z dz 
mx , y 

 1

exp     f x, y  mx, y 2 
2
 2

g x, y  mx 1, y  mx 1, y  mx, y 1  mx, y 1
December 18, 2003
  4
Glasgow University
Fixed Point
Equation
18
Approximate Marginal Probability in Bethe
Approximation (=Loopy Belief Propagation)
 x, y ( f x, y )     f x, y  z x, y  z dz

1
Z x, y


 f x, y g x, y M xx,y1, y  f x, y M xx,y1, y  f x, y 



 M xx,, yy 1 f x, y M xx,, yy 1 f x, y
 xx,',yy ' ( f x, y , f x ', y ' )




   f x, y  z x, y  f x ', y '  z x ', y '  z dz

1
Z xx,y1, y



 f x, y g x, y   f x, y , f x 1, y  f x 1, y g x 1, y

 
 
 
 M xx12,,yy  f x 1, y M xx11,, yy 1  f x 1, y M xx11,, yy 1  f x 1, y 
 M xx,y1, y f x, y M xx,, yy 1 f x, y M xx,, yy 1 f x, y
December 18, 2003
Glasgow University
19
Approximate Marginal Probability in Bethe
Approximation (=Loopy Belief Propagation)
 x, y  f x, y   
 x 1, y
 x, y

 f x, y , z x 1, y dz x 1, y
( x, y  1)
( x, y  1)
M xx,,yy 1 ( f x, y )
M xx,,yy 1 ( f x, y )
M xx,y1, y ( f x, y )
( x  1, y )
( x  1, y )
( x, y )
M
x , y 1
x, y
M xx,y1, y ( f x, y )
( f x, y )
1
Z x, y
M xx,,yy 1 ( f x, y )


 M xx,, yy 1 f x, y M xx,, yy 1 f x, y
December 18, 2003

1
Z xx,y1, y
M xx12,,yy ( f x1, y )
( x  2, y )
M xx11,,yy 1 ( f x1, y )
( x, y  1)


( x  1, y )
( x, y )
( x  1, y )
x 1, y
 f x, y g x, y M xx,y1, y  f x, y M xx,y1, y  f x, y   x, y  f x, y , f x 1, y  

M xx11,,yy 1 ( f x1, y )
M xx,y1, y ( f x, y )
( x, y  1)
 x, y  f x, y  
( x  1, y  1)

( x  1, y  1)


 f x, y g x, y  f x, y , f x 1, y  f x 1, y g x 1, y

 
 
 
 M xx12,,yy  f x 1, y M xx11,, yy 1  f x 1, y M xx11,, yy 1  f x 1, y 
 M xx,y1, y f x, y M xx,, yy 1 f x, y M xx,, yy 1 f x, y
Glasgow University
20
Message Update Rule in Bethe
Approximation (Loopy Belief Propagation)

 x, y    ,  M xx,y1, y  M xx,,yy 1 M xx,,yy 1 d

x 1, y
x, y 1
x, y 1












,

'
M

M

M
x, y
x, y
x, y  d d '
  x, y

M xx, y1, y   
 
( x, y  1)
M xx,,yy 1( f x, y )
M xx,y1, y ( f x, y )
( x  1, y )
M xx, 1y, y ( f x1, y )
( x, y )
M
x , y 1
x, y

Fixed-Point Equations
( x  1, y )
( f x, y )
 
  
M  M
Natural Iteration
( x, y  1)
December 18, 2003

x ', y '

 1 x ', y '
x, y
x ', y '
x ', y ' 2 
M x, y ( ) 
exp   x, y    x, y 
2
 2

Glasgow University
21
Message Update Rule of Bethe Approximation
(Loopy Belief Propagation)
xx,y1, y
 xx, y1, y 


1, y
   xx,, yy 1  xx,, yy 1  xx
,y

1, y
    xx,, yy 1  xx,, yy 1  xx
,y
g x, y   xx,, yy 1xx,, yy 1   xx,, yy 1xx,, yy 1   xx,y1, y xx,y1, y
  xx,, yy 1  xx,, yy 1  xx,y1, y
Fixed-Point Equations
( x, y  1)
 xx,, yy 1, xx,, yy 1
 xx,y1, y , xx,y1, y
( x  1, y )
( x  1, y )
( x, y )
 xx,, yy 1, xx,, yy 1
 xx, y1, y , xx,y1, y
Natural Iteration
( x, y  1)
December 18, 2003


  
      
 
 
Glasgow University
22
Approximate Normalization in Bethe
Approximation (Loopy Belief Propagation)
ln Z 
 ln Z x, y
( x, y )

( x, y )
  ln Z xx,, yy 1  ln Z x, y 1  ln Z x, y 

 ln
x 1, y
Z x, y
 ln Z x 1, y  ln Z x, y
( x, y )
December 18, 2003
Glasgow University
23
Bethe Approximation and Variational
Principle of Bethe Free Energy for
Gaussian Graphical Model
1 
 f  
Wx , y f x , y


Z ( x, y )


x 1, y


W
(
f
,
f
)
x
,
y
x
,
y
x

1
,
y




Wx, y ( f x, y )Wx 1, y ( f x 1, y ) 
(
x
,
y
)





x , y 1

W
( f x, y , f x, y 1 ) 
x, y


  Wx, y ( f x, y )Wx, y 1( f x, y 1 ) 
 ( x, y )



 

exp  
f x, y  g x, y 2 
 2

Wx , y f x , y 

 
2
exp



g

d
x
,
y
  2













 

exp  
f x, y  g x, y 2 
f x', y '  g x', y ' 2 
f x, y  f x', y ' 2 
2
2
 2

Wxx,'y, y ' f x, y , f x', y ' 
 
 
2 
2 
2


exp



g


'

g




'

dd '
x
,
y
x
'
,
y
'
   2
2
2


December 18, 2003



Glasgow University


24
Factorizable Form in Bethe Approximation
 Q z  
dz
DQ     Q( z ) ln 
  z  


Qx, y ( f x, y )    f x, y  z x, y Qz dz



Qxx,',yy ' ( f x, y , f x', y ' )    f x, y  z x, y  f x', y '  z x', y ' Qz dz


Qxx,y1, y ( f x , y , f x 1, y ) 
Qxx,,yy 1 ( f x , y , f x , y 1 ) 
 

Q( f )    Qx, y  f x , y  



 ( x , y )
 ( x , y ) Qx, y ( f x , y )Qx 1, y ( f x 1, y )  ( x , y ) Qx , y ( f x , y )Qx , y 1 ( f x , y 1 ) 
December 18, 2003
Glasgow University
25
Basic Framework of Bethe Approximation
 
DQ    FBethe Q  ln Z 
FBethe [{Q }] 

( x , y )
D[Qx , y | Wx , y ] 

 D[Q
( x , y )
 D[Q
( x , y )
x , y 1
x, y
x 1, y
x, y

| Wxx, y1, y ]  D[Qx , y | Wx , y ]  D[Qx 1, y | Wx 1, y ]
arg min DQ P   arg min F

| Wxx, ,yy 1 ]  D[Qx , y | Wx , y ]  D[Qx , y 1 | Wx , y 1 ]
Q
Q
  arg min FBethe Q 
Q
Constraint Conditions









  Qxx,, yy 1  f x, y , z x, y 1 dz x, y 1   Qxx,, yy1 z x, y 1, f x, y dz x, y 1



Qx, y f x, y   Qxx,y1, y f x, y , z x 1, y dz x 1, y   Qxx, y1, y z x 1, y , f x, y dz x 1, y


December 18, 2003
Glasgow University
26
Propagation Rule of Bethe Approximation
Update Rule is reduced to Loopy Belief Propagation
 x, y  f x, y  
1
Z x, y










Wx, y f x, y M xx,y1, y f x, y M xx,y1, y f x, y M xx,,yy 1 f x, y M xx,,yy 1 f x, y

 xx,y1, y f x, y , f x 1, y 
1


 
 

 
Wxx, y1, y f x, y , f x 1, y M xx,y1, y f x, y M xx,, yy 1 f x, y M xx,,yy 1 f x, y
x 1, y
Z x, y





 M xx12,,yy f x 1, y M xx11,,yy 1 f x 1, y M xx11,,yy 1 f x 1, y


 x, y    ,  M xx,y1, y  M xx,,yy 1 M xx,,yy 1 d

x 1, y
x, y 1
x, y 1












,

'
M

M

M
x, y
x, y
x, y  d d '
  x, y

M xx, y1, y   
 
December 18, 2003
Glasgow University
27
Image Restoration
The original image is generated from the prior probability.
(Hyperparameters: Maximization of Marginal Likelihood)
Degraded Image (  40)
Original Image (  0.001)
Mean-Field Method
Loopy Belief Propagation Exact Result
ˆ  0.000298
ˆ  0.000784
ˆ  0.001090
ˆ  29.1, MSE  538.5
ˆ  37.8, MSE  302.8
ˆ  39.4, MSE  297.6
December 18, 2003
Glasgow University
28
Numerical Experiments
of Logarithm of Marginal Likelihood
The original image is generated from the prior probability.
(Hyperparameters: Maximization of Marginal Likelihood)
Original Image (  0.001)
Degraded Image (  40)
MFA
-5.0
-5.0
MFA
1
ln Pg ˆ ,  
||
LPB
1
ln Pg  , ˆ 
||
-5.5
Exact
LPB
Exact
-6.0
10
20
30 40
Mean-Field Method
ˆ  0.000298, ˆ  29.1
December 18, 2003
 50
60
-5.5
0
Loopy Belief Propagation
ˆ  0.000784, ˆ  37.8
Glasgow University
0.0010  0.0020
Exact Result
ˆ  0.00109, ˆ  39.4
29
Image Restoration
(MSE: Mean Square Error)
Degraded Image
Mean-Field
Method
Loopy Belief
Propagation
MSE: 1409
MSE: 593
MSE: 324
Exact Result
Lowpass
Filter
Wiener Filter
Median Filter
MSE:306
MSE: 268
MSE: 369
Original Image
December 18, 2003
Glasgow University
MSE: 259
30
Image Restoration
Original Image
Exact Result
MSE:315
December 18, 2003
(MSE: Mean Square Error)
Degraded Image
Mean-Field
Method
Loopy Belief
Propagation
MSE: 1512
MSE: 591
MSE: 325
Lowpass
Filter
Wiener Filter
Median Filter
MSE: 411
MSE: 545
Glasgow University
MSE: 447
31
Summary
Probabilistic Image Processing by Bayes Formula
and Bethe Approximation (Loopy Belief
Propagation in Solvable Model
The results obtained by the loopy belief
propagation represent systematic improvements
over those obtained by the mean-field
approximation.
Future Problems
Segmentation
Image Compression
Motion Detection
Color Image
December 18, 2003
EM algorithm
Statistical Performance
Line Fields
Glasgow University
32
Appendix A: Gaussian Graphical Model and
Statistical Performance
2




f

h
g
,

,

Pg f ,  P f  dfdg
  x, y x, y
  ln Pg  ,   Pg f ,  P f  dfdg
hx, y  g,  ,     z x, y Pz g ,  ,  dz
g
Pg f ,  
P f  
f

2
y
g
x
Original Image
f  f x, y x, y  
December 18, 2003
Marginalize
Pg  ,  
Marginal Likelihood
Glasgow University
g
Degraded Image
g  g x, y x, y  
33
Appendix A: Gaussian Graphical Model and
Statistical Performance


1
2


f

h
g
,

,

P g f ,  P  f  dfdg

x, y
x, y


|  | ( x, y )
1
2


|  | ( p , q ) 1   2  p, q 
  ln Pg  ,  Pg
f ,  P f  dfdg

||
1
  p, q  

1  ln 2  

ln

2
2 ( p, q )  1   2 ( p, q ) 
y


x
 2q 
 2p 

  2 cos
  p, q   4  2 cos
 Ly 
 Lx 


  ( x, y) x  1,2,, Lx , y  1,2,, Ly
December 18, 2003

Glasgow University
34
Appendix A: Gaussian Graphical Model and
Statistical Performance


1
2


f

h
g
,

,

Pg f ,  P f  dfdg

x, y
x, y


|  | ( x, y )
  30
  40
  50
244.5 296.3 336.0
1
ln Pg  ,  Pg f ,  P f  dfdg


||
December 18, 2003
  30
  40
  50
-5.2964
-5.5190
-5.7067
Glasgow University
35