Machine Learning for
Adaptive Bilateral Filtering
I. Frosio1, K. Egiazarian1,2, and K. Pulli1,3
1
NVIDIA, USA
2 Tampere University of Technology, Finland
3 Light, USA
Motivation
(denoising)
void denoise(float *img){
…
for (int y = 0; y < ys; y++){
for (int x = 0; x < xs; x++){
img(y*xs+x) = …
}
}
…
}
Motivation:
(Gaussian filter)
t(x)
d(x)
Bilateral filter
Motivation
(bilateral filter)
t(x)
d(x)
C. Tomasi and R. Manduchi,
Bilateral filtering for gray and
color images, ICCV, 1998.
Motivation:
(choice of the parameters)
sr
sd
sd
d(x)
Motivation
(use intuition?)
 σd scales with resolution
 σr scaled with grey level dynamics
 possible automatic design of parameter values
[BF, Tomasi and Manduchi, ICCV, 1998]
 image noise std σn
 σd = [1.5, 2.1], independently from σn
 σr = k·σn
[BF, Zhang, Gunturk, TIP, 2008]
sr
sd
sd
 local signal variance σs2(x,y)
 σd = [1.5, 2.1], independently from σn
 σr(x,y) = σn2 / aσs(x,y)
[ABF, Qi et al., AMR , 2013]
sr
sd
Motivation
(use machine learning!)
Framework (adaptive denoising)
3 features
q = [q0, q1, q2, q3, q4, q5]
6 unknowns
sd(x,y)
sr(x,y)
Framework (learning)
Training images {tj}j=1..N
Adaptive bilateral filter, q
Noise model
(AWGN)
Local image
features, fx,y
Image quality model
(PSNR)
i(x,y)
Entropy features
Flat
Gradient
Texture
Edges
0.0 bit
6.0 bits
1.0 bit
5.6 bits
Shannon’s entropy
i(x,y)
Entropy features
Gradient
Texture
Edges
ei
0.0 bit
6.0 bits
1.0 bit
5.6 bits
eg
0.0 bit
0.0 bit
1.2 bit
5.5 bits
||grad[i(x,y)]||
Flat
eg
ei
Entropy features
Framework (complete)
Training images {tj}j=1..N
Noisy image
Adaptive bilateral filter, q
Logistic functions
Local image features fx,y
Noise model
(AWGN)
Filtered image
sd(x,y)
EABF
Local image
features fx,y
sr(x,y)
Image quality model
(PSNR)
Results - PSNR
BF
BF [Zhang] ABF [Qi]
EABF
BF
BF [Zhang] ABF [Qi]
EABF
sd(x,y)
optimal
1.8
1.8
our
sd(x,y)
optimal
1.8
1.8
our
sr(x,y)
optimal
2sn
sn2/(0.3ss)
our
sr(x,y)
optimal
2sn
sn2/(0.3ss)
our
sn = 5
36.13
36.06
36
36.27
sn = 5
37.69
37.5
37.2
37.81
sn = 10
31.45
31.4
31.44
31.1
sn = 10
34
33.76
33.66
34.37
sn = 20
27.11
27.09
27.36
26.4
sn = 20
30.31
29.64
30.17
31.11
sn = 30
25.07
25
25.32
24.68
sn = 30
28.2
27.12
28.2
29.24
sn = 40
23.94
23.69
24.11
23.76
sn = 40
26.86
25.46
26.83
27.78
sn = 5
36.29
36.04
35.92
36.4
sn = 5
38.17
37.86
37.64
38.45
sn = 10
32.53
32.17
32.03
32.81
sn = 10
34.64
34.09
34.18
35.17
sn = 20
28.96
28.48
28.75
29.51
sn = 20
31.2
30.02
31.05
32.08
sn = 30
26.96
26.31
26.98
27.62
sn = 30
29.37
27.78
29.24
30.21
sn = 40
25.63
24.72
25.68
26.17
sn = 40
28.2
25.97
27.83
28.77
sn = 5
36.5
36.07
35.93
36.54
sn = 5
37.81
37.74
37.37
37.86
sn = 10
32.64
32.23
32.15
32.78
sn = 10
34.75
34.31
34.22
34.98
sn = 20
29.32
28.81
29.17
29.65
sn = 20
31.27
30.4
31.22
31.8
sn = 30
27.71
26.84
27.68
28.01
sn = 30
29.14
27.85
29.3
29.75
sn = 40
26.66
25.33
26.53
26.82
sn = 40
27.79
25.95
27.71
28.3
Results - PSNR
sd(x,y)
BF
[Zhang]
optimal
1.8
sr(x,y)
optimal
2sn
sn = 5
37.1
36.88
36.68
37.22
sn = 10
33.33
32.99
32.95
33.53
sn = 20
29.69
29.07
29.62
30.09
sn = 30
27.74
26.82
27.79
28.25
sn = 40
26.51
25.19
26.44
26.93
+1.01dB
sn = 5… 40
30.88
30.19
30.7
BF
average
ABF
[Qi]
1.8
EABF
our
sn2/(0.3ss) our
+0.51dB
31.21
Results – Image quality
Ground truth
Noisy
20.11 dB
ABF [Qi]
31.05 dB
EABF
32.08 dB
BF [Zhang]
30.02 dB
Machine learning vs. intuition: sd(x,y), sr(x,y)
sn = 20
BF [Zhang et al.]
ABF [Qi et al.]
EABF
[0.6, 2.6]
sd(x,y)
1.8
1.8
[20, 110]
sr(x,y)
2sn
[71, 85]
Machine learning vs. intuition: sd = sd (sn)
Flat area
Edge area
Machine learning vs. intuition: sr = sr (sn)
Flat area
Edge area
Conclusion
Learning optimal parameter
modulation
strategies
through Machine Learning is
feasible.
Modulation strategies are
complicated…
PSNR 
… But effective.
Conclusion
Training images {tj}j=1..N
Adaptive bilateral filter, q
Noise model
(AWGN)
Local image
features, fx,y
Image quality model
(PSNR)
Conclusion
Your training images
Adaptive bilateral filter, q
Noise model
(AWGN)
Local image
features, fx,y
Image quality model
(PSNR)
Conclusion
Your training images
Adaptive bilateral filter, q
Your noise model
Local image
features, fx,y
Image quality model
(PSNR)
Conclusion
Your training images
Adaptive bilateral filter, q
Your noise model
Your local
features, fx,y
Image quality model
(PSNR)
Conclusion
Your training images
Adaptive bilateral filter, q
Your noise model
Your local
features, fx,y
Your image quality
model, Q
Conclusion
Your training images
A different adaptive filter, q
Your noise model
Your local
features, fx,y
Your image quality
model, Q
Conclusion
A general
FRAMEWORK based
on MACHINE
LEARNING for the
development of
ADAPTIVE FILTERS