Physical Fluctuomatics
11th Probabilistic image processing by means of
physical models
Kazuyuki Tanaka
Graduate School of Information Sciences, Tohoku University
[email protected]
http://www.smapip.is.tohoku.ac.jp/~kazu/
Physical Fluctuomatics (Tohoku
University)
1
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing
by Probabilistic Models, Morikita Publishing Co.,
Ltd., 2006 (in Japanese) , Chapter 7.
Kazuyuki Tanaka: Statistical-mechanical approach to
image processing (Topical Review), Journal of Physics
A: Mathematical and General, vol.35, no.37, pp.R81R150, 2002.
Muneki Yasuda, Shun Kataoka and Kazuyuki
Tanaka: Computer Vision and Image Processing 3
(Edited by Y. Yagi and H. Saito): Chapter 6.
Stochastic Image Processing, pp.137-179, Advanced
Communication Media CO., Ltd., December 2010 (in
Japanese).
Physical Fluctuomatics (Tohoku
University)
2
Contents
1.
2.
3.
4.
Probabilistic Image Processing
Loopy Belief Propagation
Statistical Learning Algorithm
Summary
Physical Fluctuomatics (Tohoku University)
3
Image Restoration by Probabilistic Model
Noise
Transmission
Original
Image
Degraded
Image
Posterior



Pr{Original Image | Degraded Image }
Assumption 1: The degraded
image is randomly generated
from the original image by
according to the degradation
process.
Assumption 2: The original
image is randomly generated
by according to the prior
probability.
Prior
Likelihood


 


Pr{Degraded Image | Original Image }Pr{Original Image }

Pr{DegradedIm age}

Marginal Likelihood
Bayes Formula
Physical Fluctuomatics (Tohoku
University)
4
Image Restoration by Probabilistic Model
The original images and degraded images are represented
by f = (f1,f2,…,f|V|) and g = (g1,g2,…,g|V|), respectively.
Original
Degraded
Image
Image

ri  ( xi , yi )
Position Vector
i
i
of Pixel i
fi: Light Intensity of Pixel i
gi: Light Intensity of Pixel i
in Original Image
in Degraded Image
Physical Fluctuomatics (Tohoku
University)
5
Probabilistic Modeling of
Image Restoration Random Field of Original Image F  (F , F ,..., F
1
2
|V |
State Vector of Original Image f  ( f1 , f 2 ,..., f |V | )
Posterior



Pr{Original Image | Degraded Image }
Prior
Likelihood


 


 Pr{Degraded Image | Original Image }Pr{Original Image }
Assumption 2: The original image is generated according to a
prior probability. Prior Probability consists of a product of
functions defined on the neighbouring pixels.
Pr{Original Image f }
 Pr{F  f } 
 ( f i , f j )
ijE
i
j
Random Fields
Product over All the Nearest Neighbour Pairs of Pixels
Physical Fluctuomatics (Tohoku
University)
6
)
Probabilistic Modeling of Image
Random Field of Original
Restoration
Image F  ( F1 , F2 ,..., F|V | )
Random Field of Degraded Image G  (G1 , G2 ,..., G|V | )
Posterior



Pr{Original Image | Degraded Image }
Prior
Likelihood


 


 Pr{Degraded Image | Original Image }Pr{Original Image }
Assumption 1: A given degraded image is obtained from the
original image by changing the state of each pixel to another
state by the same probability, independently of the other pixels.
State Vector of Original Image f  ( f1 , f 2 ,..., f |V | )
State Vector of Degraded Image g  ( g1 , g 2 ,..., g |V | )
Pr{Degraded Image g | Original Image f }
 Pr{G  g | F  f }    ( gi , fi )
Random Fields
iV
Physical Fluctuomatics (Tohoku
University)
gi
gi
or
fi
fi
7
Bayesian Image Analysis
PrG  g F  f 
PrF  f 
f
Prior Probability
Original
Image
Degradation Process
Posterior Probability
PrF  f G  g 
V：Set of All
the pixels
E：Set of all
the nearest
neighbour
pairs of pixels
g
g
Degraded
Image
PrG  g F  f PrF  f 
PrG  g




    ( f i , g i )    ( f i , f j ) 
 iV
 {i , j}E

Image processing is reduced to calculations of
averages, variances and co-variances in the
posterior probability.
Physical Fluctuomatics (Tohoku
University)
8
Estimation of Original Image
We have some choices to estimate the restored image from
posterior probability.
In each choice, the computational time is generally
exponential order of the number of pixels.
(1) fˆ  arg max Pr{F  z | G  g}
z
(2) fˆi  arg max Pr{Fi  zi | G  g}
zi
Maximum A Posteriori (MAP)
estimation
Maximum posterior
marginal (MPM) estimation
Pr{Fi  f i | G  g} 
 Pr{F 
f | G  g}
f \ fi


ˆ  arg min   z Pr{F  z | G  g}dz 2
f
i  i
(3) i
i
Thresholded Posterior
Mean (TPM) estimation
Physical Fluctuomatics (Tohoku
University)
9
Prior Probability of Image Processing
 1
1
2

PrF  f  
exp      fi  f j  
Z PR ( )
 2 {i , j}E

Digital Images
(fi=0,1,…,q1)
Z PR ( ) 
 1
2





exp


z

z



j
 2  i

z1  0 z1  0
z|V |  0
{i , j }E


q 1 q 1
q 1
Sampling of Markov Chain Monte Carlo Method
 2
 1
E: Set of all the
 4
nearest-neighbour
pairs of pixels
Paramagnetic
V: Set of all the pixels
Ferromagnetic
Fluctuations increase near α=1.76….
Physical Fluctuomatics (Tohoku
University)
10
Bayesian Image Analysis by
Gaussian Graphical Model
Prior
Probability
V:Set of all
the pixels
 1
1
2

PrF  f |   
exp      f i  f j  
Z PR ( )
 2 {i , j}E

 1
2

Z PR ( )      exp     zi  z j  dz1dz2  dz|V |
 

 2 {i , j}E

 
Patterns are generated by MCMC.
  0.0001
E:Set of all the
nearest-neighbour
pairs of pixels

  0.0005
  0.0030
Markov Chain Monte Carlo Method
Physical Fluctuomatics (Tohoku
University)
11
Degradation Process for
Gaussian Graphical Model
Degradation Process is assumed to be the
additive white Gaussian noise.
PrG  g F  f ,    
iV
f i , gi   ,
1


exp  2  f i  g i 2 
 2

2 2
1
V: Set of all
the pixels
Original Image f
Gaussian Noise n
Degraded Image g
Degraded image is obtained by
adding a white Gaussian noise
to the original image.

g i  f i ~ N 0,  2
Histogram of Gaussian Random Numbers
Physical Fluctuomatics (Tohoku
University)
12

Bayesian Image Analysis
PrF  f 
f
PrG  g F  f 
Prior Probability
Degradation Process
Original
Image
Posterior Probability
PrF  f G  g 
V：Set of All
the pixels
E：Set of all
the nearest
neighbour
pairs of pixels
g
g
Degraded
Image
PrG  g F  f PrF  f 
PrG  g




    ( f i , g i )    ( f i , f j ) 
 iV
 {i , j}E

Image processing is reduced to calculations of
averages, variances and co-variances in the
posterior probability.
Physical Fluctuomatics (Tohoku
University)
13
Bayesian Image Analysis
PrF  f 
f
PrG  g F  f 
g
g
Degradation Process
Prior Probability Original
画素
Image
Posterior Probability
PrF  f G  g 
fˆi  PrFi  f i G  g 
Degraded
Image
PrG  g F  f PrF  f 
PrG  g
 PrF 
f \ fi
Marginal Posterior Probability of Each Pixel
Physical Fluctuomatics (Tohoku
University)
f G  g
Computational
Complexity
14
Posterior Probability for Image
Processing by Bayesian Analysis
PrF  f G  g,  ,   
 1
1
1
2
2
exp      f i  gi      f i  f j  
Z Posterior g,  ,  
2 {i , j}E
 2 iV

 1
1
2
2
Z Posterior g,  ,       exp     zi  g i     zi  z j  
2 {i , j}E
z1  0 z 2  0
z|V |  0
 2 iV

q 1 q 1
q 1

Additive White Gaussian Noise

1
PrF  f G  g 
{i , j} f i , f j

Z Posterior ( g ,  ,  ) {i , j}E

{i , j} f i , f j


1
 1
2
 exp    f i  g i    f j  g j
4
 4
Physical Fluctuomatics (Tohoku
University)

2
1
2


1
  fi  f j
2
15
2 

Bayesian Network and Posterior
Probability of Image Processing
Bayesian Network
Posterior Probability
PrF  f G  g

PrG  g F  f PrF  f 
PrG  g
Probabilistic Model expressed in terms of
Graphical Representations with Cycles
PrF  f G  g 
1
  f , f 
Z Posterior {i , j}E
{i , j }
i
j
V: Set of all the pixels
E: Set of all the nearest neighbour pairs of pixels
Physical Fluctuomatics (Tohoku
University)
16
Markov Random Fields
PrF  f G  g 
  f , f 
1
Z Posterior {i , j}E
{i , j }
i
j
PrFi  f i F1  f1 ,  , Fi 1  f i 1 , Fi 1  f i 1 , , FL  f L , G  g
 f , f 

PrF  f G  g


 P x x
 PrF  z G  g   z , f 
{i , j }
i
j
ji
i
{i , j }
zi
(V , E )
zi
i
i
j
j  i

j
ji
Markov Random
Fields
i
(V , E )
∂i: Set of all the nearest neighbour pixels of the pixel i
Physical Fluctuomatics (Tohoku
University)
17
Contents
1.
2.
3.
4.
Probabilistic Image Processing
Loopy Belief Propagation
Statistical Learning Algorithm
Summary
Physical Fluctuomatics (Tohoku University)
18
Loopy Belief Propagation
Probabilistic Model expressed in terms of
Graphical Representations with Cycles
1


Pr F  f G  g 
{i , j} f i , f j

Z Posterior {i , j}E


V: Set of all the pixels
E: Set of all the nearest neighbour pairs of pixels
PrF1  f1 G  g 
 PrF  f G  g
f \ f1
3
4
1
5
2
M 21  f1 M 31  f1 M 41  f1 M 51  f1 

 M 21 z1 M 31 z1 M 41 z1 M 51 z1 
z1
Physical Fluctuomatics (Tohoku
University)
19
Loopy Belief Propagation
M 12  f 2  
  z , f M z M z M z 
{1, 2}
1
31
2
41
1
51
1
1
z1
  z , z M z M z M z 
{1, 2}
z1
1
2
31
1
41
1
51
1
z2
3
4
1
2
Simultaneous fixed point equations
to determine the messages
5
 
  
M  M
Physical Fluctuomatics (Tohoku
University)
20
Belief Propagation Algorithm for Image Processing
Step 1: Solve the simultaneous fixed point equations
for messages by using iterative method
q 1
i
j
M i j  f j  
  z , f  M z 
zi  0
q 1 q 1
{i , j }
i
k i
j
i
k i
   z , z  M z 
zi 0 z j 0
{i , j }
i
k i
j
i
k i
(j  i, i  V , f i  {0,1,2, ,q  1})
i
PrFi  f i G  g 
Step 2: Substitute the messages to approximate
expressions of marginal probabilities for each pixel and
compute estimated image so as to maximize the
approximate marginal probability at each pixel.
M  f 
k i
i
k i
q 1
 M  f 
z i  0 k i
k i
i
(i  V , f i  {0 ,1,2, ,q  1})
fˆi  arg
Physical Fluctuomatics (Tohoku
University)
max
z i  0 ,1,, q 1
Pi  zi 
21
Belief Propagation
in Probabilistic Image Processing
Physical Fluctuomatics (Tohoku
University)
22
Contents
1.
2.
3.
4.
Probabilistic Image Processing
Loopy Belief Propagation
Statistical Learning Algorithm
Summary
Physical Fluctuomatics (Tohoku University)
23
Statistical Estimation of Hyperparameters
Hyperparameters ,  are determined so as to
maximize the marginal likelihood Pr{G=g|,} with
respect to , .
(ˆ , ˆ )  arg max Pr{G  g |  ,  }
 ,  
Pr{G  g |  ,  }   Pr{F  z, G  g |  ,  }
z
  Pr{G  g | F  z,  } Pr{F  z |  }
z
y
V
x
Original Image
Pr{F  f |  }
f
Degraded Image
Pr{G  g | F  f ,  }
g
g
Marginalized with respect to F
Pr{G  g |  ,  }
Physical Fluctuomatics (Tohoku
University)
Marginal Likelihood
24
Maximum Likelihood in Signal Processing
Original Image
=Parameter




f 



f1 

f2 
 

f|V | 
Incomplete
Data
Degraded Image
=Data
 ˆ

Hyperparameter


 g1  ˆ ,   arg max Pr G  g  , 
 ,  


 g2 
g
  Pr{G  g |  ,  }   Pr{G  g | F  z,  } Pr{F  z |  }


z
g 
 |V | 
Marginal Likelihood
Extremum Condition
  Pr{G  g |  ,  } 
  Pr{G  g |  ,  } 
 0, 
0






 ˆ ,  ˆ

 ˆ ,  ˆ
Physical Fluctuomatics (Tohoku
University)
25
Maximum Likelihood and EM algorithm
Original Image
=Parameter
Incomplete
Data
Degraded Image
=Data
Pr{G  g |  ,  }   Pr{G  g | F  z,  } Pr{F  z |  }
z

Marginal Likelihood
 f1 
 
 f2 
f  

 
f 
 |V | 

 g1 


 g 2  Q ,   ,  
Q -function
g
    PF  f G  g,  ,  ln PF  f , G  g  ,  


z
g 
 |V | 
 Q  ,   ,  
0





   ,   
Hyperparameter
E Step : Calculate Q  ,   (t ),  (t ) 
 Q  ,   ,  
0




   ,   
M Step : Update
α (t  1) , (t  1) 
 arg max Q  ,   (t ),  (t )    PrG  g  ,  
( ,  )
Expectation Maximization
(EM) algorithm provide us
one of Extremum Points of
Marginal Likelihood.



  PrG  g  ,  
 0, 
0




ˆ
ˆ
 ˆ ,   

 ˆ ,   
Extemum Condition
Physical Fluctuomatics (Tohoku
University)
26
Maximum Likelihood and EM algorithm
Physical Fluctuomatics (Tohoku
University)
27
Bayesian Image Analysis
by Exact Solution of Gaussian Graphical Model
f i , gi   ,
Posterior Probability
 1
Pr{F  f | G  g,  ,  }  exp   2
 2
 f
iV
 gi 
2
i
1
2
    f i  f j  
2 {i , j}E

Average of the posterior probability
can be calculated by using the multidimensional Gauss integral Formula
fˆ   z Pr{F  z | G  g,  ,  } dz

I
I   C
2
|V|x|V| matrix
g
V:Set of all
the pixels
E:Set of all the
nearest-neghbour
pairs of pixels
4

i C j   1
0

i  j 
{i, j}  E 
otherwise 
Multi-Dimensional Gaussian Integral Formula
Physical Fluctuomatics (Tohoku
University)
28
Bayesian Image Analysis
by Exact Solution of Gaussian Graphical Model
Iteration Procedure in Gaussian Graphical Model
 1

 t  12 C
1
C
T

 t  
Tr

g
g
2
2
|V |

|
V
|
I   t  1 t  1 C
I   t  1 t  1 C



g
1

1
 t  12 I
1
 t  12  t  14 C 2
 t  
Tr

g
gT
|V |
I   t  1 t  12 C | V | I   t  1 t  12 C 2

m (t ) 

I
I   (t ) (t ) 2 C
g


f̂
fˆi (t )  arg min zi  mi (t ) 2
zi
t 
0.0008
0.001
0.0006
0.0004
0.0002
g
0
0
20
40
60
 100t 
80
Physical Fluctuomatics (Tohoku
University)
f̂
29
Image Restorations
by Exact Solution and Loopy Belief
Propagation of Gaussian Graphical Model
 t  1, t  1  arg max Q ,  t , t , g .
 , 
Exact
ˆ Exact  37.624
ˆ Exact  0.000713

0.001
0.0008
0.0006
fˆ  hˆ ,ˆ , g 
0.0004
0.0002
0
0
20
40
60
80
ˆ LBP  36.335
ˆ LBP  0.000600
Physical Fluctuomatics (Tohoku
University)
100

Loopy Belief Propagation
30
Image Restorations by
Gaussian Graphical Model
Original Image
Degraded Image Belief Propagation
MSE: 1512
Lowpass Filter
MSE: 411
MSE:325
Wiener Filter
Median Filter
MSE: 545
MSE: 447
Physical Fluctuomatics (Tohoku
University)
Exact
MSE:315
MSE 

1
 fi  fˆi
| V | iV
31

2
Image Restoration by Gaussian
Graphical Model
Original Image
Degraded Image
Belief Propagation
MSE: 1529
MSE: 260
Lowpass Filter
Wiener Filter
Median Filter
MSE: 224
MSE: 372
MSE: 244
Physical Fluctuomatics (Tohoku
University)
Exact
MSE236
32
Image Restoration by Discrete
Gaussian Graphical Model
Original Image
Q=4
(t)
ˆ LBP  1.05224
ˆ LBP  1.20519
Degraded Image
=1
MSE:0.98893
SNR:1.03496 (dB)
Belief Propagation
MSE: 0.36011
SNR: 5.42228 (dB)
(t)
Physical Fluctuomatics (Tohoku
University)
33
Image Restoration by Discrete
Gaussian Graphical Model
Original Image
Q=4
(t)
ˆ LBP  1.02044
ˆ LBP  1.20294
Degraded Image
=1
MSE: 0.98893
SNR: 1.07221 (dB)
Belief Propagation
MSE: 0.28796
SNR: 6.43047 (dB)
(t)
Physical Fluctuomatics (Tohoku
University)
34
Image Restoration by Discrete
Gaussian Graphical Model
Original Image
Q=8
(t)
ˆ LBP  0.96380
ˆ LBP  0.66033
Degraded Image
=1
MSE:0.98893
SNR:1.89127 (dB)
Belief Propagation
(t)
MSE: 0.37697
SNR: 6.07987 (dB)
Physical Fluctuomatics (Tohoku
University)
35
Image Restoration by Discrete
Gaussian Graphical Model
Original Image
Q=8
(t)
Degraded Image
ˆ LBP  0.85195
ˆ LBP  0.62324
=1
MSE:0.98893
SNR:0.93517 (dB)
Belief Propagation
(t)
MSE: 0.32060
SNR: 5.82715 (dB)
Physical Fluctuomatics (Tohoku
University)
36
Contents
1.
2.
3.
4.
Probabilistic Image Processing
Loopy Belief Propagation
Statistical Learning Algorithm
Summary
Physical Fluctuomatics (Tohoku University)
37
Summary
Image Processing
Bayesian Network for Image Processing
Design of Belief Propagation Algorithm for Image
Processing
Physical Fluctuomatics (Tohoku
University)
38
Practice 10-1
For random fields F=(F1,F2,…,F|V|)T and G=(G1,G2,…,G|V|)T and state vectors
f=(f1,f2,…,f|V|)T and g=(g1,g2,…,g|V|)T, we consider the following posterior probability
distribution:
PrF  f G  g 
  f , f 
1
Z Posterior {i , j}E
{i , j }
i
j
Here ZPosterior is a normalization constant and E is the set of all the nearest-neighbour
pairs of nodes. Prove that
PrF  f G  g
PrFi  f i F1  f1 , , Fi 1  f i 1 , Fi 1  f i 1 , , FL  f L , G  g 
 PrF  z G  g
and
PrF  f G  g

 {i, j}  fi , f j 
zi
ji
 PrF  z G  g   z , f 
{i , j }
zi
zi
i
j
ji
where ∂i denotes the set of all the nearest neighbour pixels of the pixel i.
Physical Fluctuomatics (Tohoku
University)
39
Practice 10-2
Make a program that generate a degraded image
g=(g1,g2,…,g|V|) from a given image f =(f1,f2,…,f|V|) in
which each state variable fi takes integers between 0 and
q1 by according to the following conditional probability
distribution of the additive white Gaussian noise:
PrG  g F  f   
iV
 1
2
exp   2  f i  gi  
 2

2 2
1
Give some numerical
experiments for =20 and 40.
Original Image f
Gaussian Noise n
Degraded
Image g
Histogram of Gaussian Random Numbers
Physical Fluctuomatics (Tohoku
University)
40
Practice 10-3
Make a program for estimating q-valued original images f
=(f1,f2,…,f|V|) from a given degraded image g =(g1,g2,…,g|V|) by using
a belief propagation algorithm for the following posterior probability
distribution for q valued images:
PrF  f G  g 
1
  f , f 
Z Posterior ijB
ij
i
j
fi {0,1,2,, q  1}
1
1
 1
2
2
2
 ij  f i , f j   exp     f i  g i     f j  g j     f i  f j  
8
2
 8

Z Posterior 
q 1 q 1
q 1
1
1
 1
2
2
2







exp


z

g


z

g


z

z





i
i
j
j
i
j
8
8
2


z1  0 z 2  0
z|V |  0
Give some numerical experiments.
K. Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita
Publishing Co., Ltd., 2006 (in Japanese) , Chapter 8 .
M. Yasuda, S. Kataoka and K. Tanaka: Computer Vision and Image Processing 3
(Edited by Y. Yagi and H. Saito): Chapter 6. Stochastic Image Processing, pp.137-179,
Advanced Communication Media CO., Ltd., December 2010 (in Japanese).
Physical Fluctuomatics (Tohoku
University)
41
Practice 10-4
Make a program for estimating original images f =(f1,f2,…,f|V|) from
a given degraded image g =(g1,g2,…,g|V|) by using a belief
propagation algorithm for the following posterior probability
distribution based on the Gaussian graphical model:
PrF  f G  g 
1
  f , f 
Z Posterior {i , j}E
{i , j }
i
j
fi  (,)
1
1
 1
2
2
2
{i , j}  f i , f j   exp     f i  g i     f j  g j     f i  f j  
8
2
 8

1
1
 1
2
2
2







exp


z

g


z

g


z

z

   8 i i 8 j j 2 i j dz1dz2 dz|V |
   
Z Posterior  
 

Give some numerical experiments.
Kazuyuki Tanaka: Introduction of Image Processing
by Probabilistic Models, Morikita Publishing Co., Ltd.,
2006 (in Japanese) , Chapter 8 and Appendix G.
Physical Fluctuomatics (Tohoku
University)
42