Generalized Belief Propagation
for Gaussian Graphical Model
in probabilistic image processing
Kazuyuki Tanaka
Graduate School of Information Sciences,
Tohoku University, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/
Reference
K. Tanaka, H. Shouno, M. Okada and D. M. Titterington:
Accuracy of the Bethe Approximation for Hyperparameter Estimation in
Probabilistic Image Processing, J. Phys. A: Math & Gen., 37, 8675 (2004).
6 September, 2005
SPDSA2005 (Roma)
1
Contents
1.
2.
3.
4.
5.
Introduction
Loopy Belief Propagation
Generalized Belief Propagation
Probabilistic Image Processing
Concluding Remarks
6 September, 2005
SPDSA2005 (Roma)
2
Bayesian Image Analysis
Noise
Transmission
Original Image
Degraded Image
Degradation Process
PrioriProbability

 A

PrDegraded Image Original ImagePrOriginal Image
PrOriginal Image Degraded Image

PrDegraded Image



A PosterioriProbability
MarginalLikelihood
Graphical Model with Loops
=Spin System on Square Lattice
Bayesian Image Analysis + Belief Propagation
→ Probabilistic Image Processing
6 September, 2005
SPDSA2005 (Roma)
3
Belief Propagation
Belief Propagation
(Lauritzen, Pearl)
Probabilistic model with no loop
= Transfer Matrix
Probabilistic model with some loops
Approximation→Loopy Belief Propagation
Generalized Belief Propagation
(Yedidia, Freeman, Weiss)
Loopy Belief Propagation (LBP) = Bethe Approximation
Generalized Belief Propagation (GBP)
= Cluster Variation Method
How is the accuracy of LBP and GBP?
6 September, 2005
SPDSA2005 (Roma)
4
Gaussian
Graphical Model f  fi i   fi   ,
 1
1
1
2
2
 f   exp    i  f i  gi    ij  f i  f j  
Z
2 ijN
 2 i

Free energy  ln Z
 f f df  H
and average  f f df
g
H :    matrix
can be calculated
by using the multi - dimensiona l
Gauss integral formula.
6 September, 2005
1
H ij
SPDSA2005 (Roma)
1    ij /  i i  j 
  j ijN 

ij  N 
  ij / i
 0

otherwise 

5
Probabilistic Image Processing by
Gaussian Graphical Model and
Generalized Belief Propagation
How can we construct a probabilistic image
processing algorithm by using Loopy Belief
Propagation and Generalized Belief
Propagation?
How is the accuracy of Loopy Belief
Propagation and Generalized Belief
Propagation?
In order to clarify both questions, we assume
the Gaussian graphical model as a posterior
probabilistic model
6 September, 2005
SPDSA2005 (Roma)
6
Contents
1.
2.
3.
4.
5.
Introduction
Loopy Belief Propagation
Generalized Belief Propagation
Probabilistic Image Processing
Concluding Remarks
6 September, 2005
SPDSA2005 (Roma)
7
Kullback-Leibler Divergence of
Gaussian Graphical Model
 Qz  
dz  F    ln Z 
DQ     Qz  ln 

  z  



1
2
F      i Vi  mi  g i 
i 2
1
2


   ij Vi  2Vij  V j  mi  m j
ijN 2
Entropy Term


  Qz  ln Qz dz
Vi    zi  mi  Qz dz
2
mi   zi Qz dz
6 September, 2005
Vij   zi  mi z j  m j Qz dz
SPDSA2005 (Roma)
8
Loopy Belief Propagation
Trial
Function
Tractable
Form
6 September, 2005
Qij  f i , f j  



Q f     Qi  f i  
 i
 ijN Qi  f i Q f j  
Qi  f i      f i  zi Qz dz
Qij  f i , f j      f i  zi   f j  z j Qz dz
SPDSA2005 (Roma)
9
Loopy Belief Propagation
Qij  f i , f j  



Q f     Qi  f i  
 i
 ijN Qi  f i Q f j  
Trial Function
Marginal Distribution of GGM is also GGM
Q f     f  z Qz dz

 1

T
1
exp   f  m  A f  m 
 2

det A 
1
2 

m   m A 
 i
6 September, 2005
i
 ii
 Vi
SPDSA2005 (Roma)
A 
 ij
 Vij
10
Loopy Belief Propagation
Qij  f i , f j  



Q f     Qi  f i  
 i
 ijN Qi  f i Q f j  
Q f  

 1

T
1
exp   f  m  A f  m 
 2

det A 
1
2 

F    F mi ,Vi , Vij 

Bethe Free Energy in GGM


1
1
2
2
   i Vi  mi  g i     ij Vi  2Vij  V j  mi  m j 
i 2
ijN 2


 1

 1

2
   i 1  ln 2Vi    1  ln 2  det Aij 
 2
 ijN  2

i
6 September, 2005
SPDSA2005 (Roma)
11

Loopy Belief Propagation

  0   m  g  
F mi ,Vi ,Vij 

mi
i
i
0 
i
F mi , Vi , Vij 
Vi

i

Vij
 m  m   0



j ijN




j ijN
j
  i Ai 
1
ij
ij

m is exact
A 



1
j ijN


ii 


ij
ij
ij
ii
0
0
1
1 
2


1

1

4

V
V

ij
i j

2 ij 
6 September, 2005
i
1
1 
1
Vi 
  i    ij   Aij
 i  
 j ijN 
 j ijN 
Vij 
ij
  0    A 
F mi , Vi , Vij 
m  H 1g
SPDSA2005 (Roma)
 Vi Vij 

Aij  

V
V
j 
 ij
12
Iteration Procedure
 
V Ψ V
*
Fixed Point Equation
*
Iteration
V
(1)
V
( 2)
V
( 3)
 
 Ψ V 
 Ψ V 
Ψ V
(0)
y
(1)
V (1)
( 2)
0
yx
y   (x)
V * V (1)
M (0)
x

6 September, 2005
SPDSA2005 (Roma)
13
Loopy Belief Propagation and
TAP Free Energy
1 
2

Loopy Belief Propagation
Vij 

1

1

4

V
V

ij
i j


2

ij
1
  ij  Aij ij  0
3 2
2
5
  ijViV j   ij Vi V j  O  ij
A ii  Vi A ij  Vij
  0
 
 
ij
TAP Free
Energy


F mi , Vi , Vij 


Mean Field Free Energy

1
1
2
2
   i Vi  mi  g i     ij Vi  V j  mi  m j 
i 2
ijN 2
 
 1
 1
2
4
  1  ln 2Vi     ij ViV j  O  ij
2
 2 ijN
i 
6 September, 2005
SPDSA2005 (Roma)
14

Contents
1.
2.
3.
4.
5.
Introduction
Loopy Belief Propagation
Generalized Belief Propagation
Probabilistic Image Processing
Concluding Remarks
6 September, 2005
SPDSA2005 (Roma)
15
Generalized Belief Propagation
Cluster: Set of nodes
Subcluster      '     and  '  
Basic Cluster Set B :   B and  ' B     ' B
C  B        ' for   B and  ' B
    1 
Every subcluster of the
element of B does not
belong to B.
1
2
3
4

6 September, 2005
2
1
3
3
4
B
 '  ' , 'C
  '
C  B 
Example: System consisting of 4 nodes
1

2
12  1, 12  2, 23  2, 23  3,
4
34  3, 34  4, 14  1, 14  4
 12   23   34   14  1
 1   2   3   4  1
SPDSA2005 (Roma)
16
Selection of B in LBP and GBP
LBP
(Bethe Approx.）
1
1
2
4
B
2
2
3
6
5
4
5
5
6
8 8 8
9
1
2
3
4
5
6
7 7
7
8
9
1
2 2
3
4
5 5
6
4
5 5
6
7
8 8
9

6 September, 2005
B
GBP
(Square Approx.
in CVM)
4
5
SPDSA2005 (Roma)
3
6
9
17
Selection of B and C in Loopy
Belief Propagation
LBP
(Bethe Approx.）

B
The set of Basic Clusters
6 September, 2005
SPDSA2005 (Roma)
C
The Set of Basic Clusters
and Their Subclusters
18
Selection of B and C in
Generalized Belief Propagation
GBP
(Square Approximation in CVM）

B
The set of Basic Clusters
6 September, 2005
SPDSA2005 (Roma)
C
The Set of Basic Clusters
and Their Subclusters
19
Generalized Belief
Propagation
 Q f 
   
Qf  
Trial Function
  C 
Marginal Distribution of GGM is also GGM
Q f     f  z Qz dz 

A   V
A   V
 ii
i
ij
ij
 1

T
1
exp   f  m  A f  m 
 2

det A 
1
2 

F    F mi , Vi , Vij 



1
1
2
2
   i Vi  mi  g i     ij Vi  2Vij  V j  mi  m j 
i 2
ijN 2



 1


    1  ln 2  det A 
 2

 C
6 September, 2005
SPDSA2005 (Roma)
20
Generalized Belief Propagation


F mi ,Vi ,Vij
 0  i mi  gi  
ij mi  m j  0

mi
j j  ,i  , C


0 
F mi , Vi , Vij 
Vi

i 
j
  0  
F mi ,Vi ,Vij 
Vij
V  Ψ V 
6 September, 2005
ij
  ij 
j  ,i  , B 

 i 


 

m  H 1g
m is exact
 i  , C 
, j  , C 
 
1




A
0


ii
 
   A
1
ij
0
V  Vi ,Vij i  , ij  N 
SPDSA2005 (Roma)
21
Contents
1.
2.
3.
4.
5.
Introduction
Loopy Belief Propagation
Generalized Belief Propagation
Probabilistic Image Processing
Concluding Remarks
6 September, 2005
SPDSA2005 (Roma)
22
Bayesian Image Analysis
Noise
Transmission
Original Image
Degraded Image
Degradation Process
PrioriProbability

 A

PrDegraded Image Original ImagePrOriginal Image
PrOriginal Image Degraded Image

PrDegraded Image



A PosterioriProbability
MarginalLikelihood
6 September, 2005
SPDSA2005 (Roma)
23
Bayesian Image Analysis
f i , gi   ,
Degradation Process
g i  f i  ni

ni ~ N 0,  2

P g f ,    
i

1
 1
2
exp   2  f i  gi  
2 
 2

Additive White
Gaussian Noise
Transmission
Original Image
6 September, 2005
Degraded Image
SPDSA2005 (Roma)
24
Bayesian Image Analysis
f i , g j   ,
A Priori Probability

 1
P f   
exp    f i  f j

Z P R   ijB
 2
1

2



Generate
Similar?
Standard Images
6 September, 2005
SPDSA2005 (Roma)
25
Bayesian Image Analysis
P f g ,  ,  
Original Image f
Degraded Image g
A Posteriori Probability
P f g ,  ,   

Pg f ,  P f  
P g  ,  

1

Wij f i , f j

Z POS g,  ,   ijB






1
1
 1

2
2
Wij f i , f j  exp   2  f i  g i   2 f j  g j   f i  f j 2 
2
8
 8

Gaussian Graphical Model
6 September, 2005
SPDSA2005 (Roma)
f i , g j   ,
26
Bayesian Image Analysis
P f  
A Priori Probability
y

Pixels
Pg f ,  
f
Original Image
Degraded Image
f  fi i  
x
A Posteriori
Probability
g
g
P f g,  ,   
Degraded Image
g  gi i  
P g f ,  P  f  
P g  ,  
ˆf  f P f g,  ,  df   f P f g,  ,  df
i
i
 i
 i i

6 September, 2005
SPDSA2005 (Roma)
27
Hyperparameter Determination by
Maximization of Marginal Likelihood
ˆ , ˆ   arg max Pg  ,  
f i P  f g , ˆ , ˆ df

  
Pg  ,     Pg f ,  P f  df
,
fˆi 
P f , g  ,    Pg f ,  P f  
y

P f  
Pg f ,  
f
g
g
x
Marginalization
Original Image
f  fi i  
6 September, 2005
Pg  ,  
Marginal Likelihood
SPDSA2005 (Roma)
g
Degraded Image
g  gi i  
28
Maximization of Marginal Likelihood by
EM (Expectation Maximization) Algorithm

  Pg f ,  P f  df
Marginal P g  ,  
Likelihood
y

x
Q-Function
Q ,   ' ,  ' , g    P f g ,  ' ,  'ln P f , g  ,  df


 

P g  ,    0

 

Incomplete Data
g  gi i  
Equivalent
   

0
    Q ,   ' ,  ' , g 

  ',  '

6 September, 2005
SPDSA2005 (Roma)
29
Maximization of Marginal Likelihood by
EM (Expectation Maximization) Algorithm

  Pg f ,  P f  df
Marginal
P g  , 
Likelihood
y

x
Q-Function
Q ,   ' ,  ' , g    P f g ,  ' ,  'ln P f , g  ,  df
EM Algorithm
Iterate the following EM-steps until convergence:
E - Step : Q ,   t ,  t    P f g,  t ,  t ln P f , g  ,  df .
M - Step :  t  1,  t  1  arg max Q ,   t ,  t .
 ,  
A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data
via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).
6 September, 2005
SPDSA2005 (Roma)
30
Image Restoration
The original image is generated from the prior probability.
(Hyperparameters: Maximization of Marginal Likelihood)
Degraded Image (  40)
Original Image (  0.001)
Mean-Field Method
Loopy Belief Propagation Exact Result
ˆ  0.000298
ˆ  0.000784
ˆ  0.001090
ˆ  29.1, MSE  538.5
ˆ  37.8, MSE  302.8
ˆ  39.4, MSE  297.6
6 September, 2005
SPDSA2005 (Roma)
31
Numerical Experiments
of Logarithm of Marginal Likelihood
The original image is generated from the prior probability.
(Hyperparameters: Maximization of Marginal Likelihood)
Original Image (  0.001)
Degraded Image (  40)
MFA
-5.0
-5.0
MFA
1
ln Pg ˆ ,  
||
LPB
1
ln Pg  , ˆ 
||
-5.5
Exact
LPB
Exact
-6.0
10
20
30 40
 50
60
-5.5
0
0.0010  0.0020
Mean-Field Method
Loopy Belief Propagation
Exact Result
ˆ  0.000298, ˆ  29.1
ˆ  0.000784, ˆ  37.8
ˆ  0.00109, ˆ  39.4
6 September, 2005
SPDSA2005 (Roma)
32
Numerical Experiments
of Logarithm of Marginal Likelihood
Original Image (  0.001)
EM Algorithm with
Belief Propagation
0.002
Degraded Image (  40)
Exact ˆ  0.00109, ˆ  39.4
LBP ˆ  0.000784, ˆ  37.8
MF ˆ  0.000298, ˆ  29.1
0.001
LPB
MFA
0
0
6 September, 2005
Exact
SPDSA2005 (Roma)
50
 t 
100
33
Image Restoration by Gaussian Graphical Model
Original Image
Degraded Image
EM Algorithm with
Belief Propagation
MSE: 1512
MSE: 1529
6 September, 2005
SPDSA2005 (Roma)
34
Image Restoration by
Gaussian Graphical Model
Original Image
Degraded Image
Mean Field Method
MSE:611
MSE: 1512
LBP
MSE:327
6 September, 2005
TAP
GBP
MSE:320
MSE: 315
SPDSA2005 (Roma)
MSE 
Exact Solution
MSE:315


2
1
f x, y  fˆx, y

|  |  x, y 
35
Image Restoration by
Gaussian Graphical Model
Original Image
LBP
MSE:260
6 September, 2005
Degraded Image
Mean Field Method
MSE: 1529
MSE: 565
Exact Solution
TAP
MSE:248
GBP
MSE:236
SPDSA2005 (Roma)
MSE 
MSE:236


2
1
f x, y  fˆx, y

|  |  x, y 
36
Image Restoration by ˆ , ˆ   arg max Pg  ,  
 , 
Gaussian Graphical
Model
  40
  40
MSE 


2
1
f x, y  fˆx, y

|  |  x, y 
6 September, 2005
MSE
̂
̂
ln Pg ˆ , ˆ 
MF
611
0.000263
26.918
-5.13083
LBP
327
0.000611
36.302
-5.19201
TAP
320
0.000674
37.170
-5.20265
GBP
315
0.000758
37.909
-5.21172
Exact
315
0.000759
37.919
-5.21444
MSE
̂
̂
ln Pg ˆ , ˆ 
MF
565
0.000293
26.353
-5.09121
LBP
260
0.000574
33.998
-5.15241
TAP
248
0.000610
34.475
-5.16297
GBP
236
0.000652
34.971
-5.17256
Exact
236
0.000652
34.975
-5.17528
SPDSA2005 (Roma)
37
Image Restoration by Gaussian Graphical
Model and Conventional Filters
  40
MSE
GBP
MSE 

611
LBP
327
TAP
320
GBP
315
Exact
315
Lowpass
Filter
(3x3)
388
(5x5)
413
Median
Filter
(3x3)
486
(5x5)
445
Wiener
Filter
(3x3)
864
(5x5)
548

2
1
f x, y  fˆx, y

|  |  x, y 
6 September, 2005
MF
MSE
(3x3) Lowpass
(5x5) Median
SPDSA2005 (Roma)
(5x5) Wiener
38
Image Restoration by Gaussian Graphical
Model and Conventional Filters
  40
MSE
GBP
MSE 

565
LBP
260
TAP
248
GBP
236
Exact
236
Lowpass
Filter
(3x3)
241
(5x5)
224
Median
Filter
(3x3)
331
(5x5)
244
Wiener
Filter
(3x3)
703
(5x5)
372

2
1
f x, y  fˆx, y

|  |  x, y 
6 September, 2005
MF
MSE
(5x5) Lowpass
(5x5) Median
SPDSA2005 (Roma)
(5x5) Wiener
39
Contents
1.
2.
3.
4.
5.
Introduction
Loopy Belief Propagation
Generalized Belief Propagation
Probabilistic Image Processing
Concluding Remarks
6 September, 2005
SPDSA2005 (Roma)
40
Summary
Generalized Belief Propagation for
Gaussian Graphical Model
Accuracy of Generalized Belief
Propagation
Derivation of TAP Free Energy for
Gaussian Graphical Model by
Perturbation Expansion of Bethe
Approximation
6 September, 2005
SPDSA2005 (Roma)
41
Future Problem
Hyperparameter Estimation by TAP Free Energy is
better than by Loopy Belief Propagation.
Effectiveness of Higher Order Terms of TAP Free
Energy for Hyperparameter Estimation by means
of Marginal Likelihood in Bayesian Image Analysis.
TAP Free
Energy


F mi , Vi , Vij 

Mean Field Free Energy


1
1
2
2




   i Vi  mi  g i    ij Vi  V j  mi  m j
i 2
ijN 2
 
 1
 1
2
4
  1  ln 2Vi     ij ViV j  O  ij
2
 2 ijN
i 
6 September, 2005
SPDSA2005 (Roma)
42
