Image Pyramids
Image Processing - IDC
Image Representation
• Gaussian pyramids
• Laplacian Pyramids
• Wavelet Pyramids
• Applications
Image features at different resolutions require filters at
different scales.
Edges (derivatives):
f(x)
f (x)
Image Pyramids
Image pyramids
Image Pyramid = Hierarchical representation of an image
Low
Resolution
No details in image (blurred image)
low frequencies
High
Resolution
Details in image low+high frequencies
A collection of images at different resolutions.
• Gaussian Pyramids
• Laplacian Pyramids
• Wavelet/QMF
Image Pyramid
Image pyramids
Low resolution
• Gaussian Pyramids
• Laplacian Pyramids
• Wavelet/QMF
High resolution
Image Pyramid
Frequency Domain
Image Blurring = low pass filtering
Low resolution
High resolution
~
=
=
*
Gaussian Pyramid
*
Low resolution
=
*
=
*
High resolution
=
The Gaussian weights
Gaussian Pyramid
Level M
1X1
Level 1
2n-1 X 2n-1
Level 0
2n X 2n
w-2
w-1
w0
w-1
w-2 w-1 w0
w-1 w-2
w-2
The Gaussian weights
For a = 0.4 most similar to a Gauusian filter
Burt & Adelson (1981)
g = [0.05 0.25 0.4 0.25 0.05]
Normalized: Σwi = 1
low_pass_filter = g’ * g =
Symmetry: wi = w-i
Unimodal: wi ≥ wj
for 0 < i < j
Equal Contribution: for all j
c
b
a
b
Σwj+2i = constant
0.0025
0.0125
0.0200
0.0125
0.0025
0.0125
0.0625
0.1000
0.0625
0.0125
0.0200
0.1000
0.1600
0.1000
0.0200
0.0125
0.0625
0.1000
0.0625
0.0125
0.0025
0.0125
0.0200
0.0125
0.0025
c
0.2
c
b a
b c
0.15
0.1
a + 2b + 2c = 1
a + 2c = 2b
0.05
0
5
a > 0.25
b = 0.25
c = 0.25 - a/2
Gaussian Pyramid Computational Aspects
Memory:
2NX2N (1 + 1/4 + 1/16 + ... ) = 2NX2N * 4/3
Computation:
}
i times
Example:
=
*
g
5
4
3
g
3
2
2
1
1
Convolution and subsampling in
matrix notation (1D case)
r
r
F = Uf
transformed image
Level i can be computed with a single convolution
with filter: hi = g * g * g * .....
h2 =
4
Vectorized image
Basis vecrors (Fourier, Wavelet, etc)
U1 =
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
4
6
4
1
0
Image pyramids
• Gaussian Pyramid
• Laplacian Pyramid
• Wavelet/QMF
- Next pyramid level
U2 =
1
4
6
4
1
0
0
0
0
0
1
4
6
4
1
0
0
0
0
0
1
4
6
4
0
0
0
0
0
0
1
4
- The combined effect of the two pyramid levels
U2 * U1 =
1
4
10
20
31 40
0
0
0
0
1
4
10 20
44
40
31
31
0
0
0
0
0
0
0
0
1
4
0
0
0
0
0
0
0
0
0
0
20 10
40
4
44 40
10 20
0
1
31
31
0
0
0
0
0
0
0
20
10
4
1
0
0
0
40
1
0
44
40
30
16
4
0
4 10
20
25
16
4
0
from: B.Freeman
Laplacian Pyramid
What does blurring take away?
G0, G1, .... = the levels of a Gaussian Pyramid.
Predict level Gl from level Gl+1 by Expanding Gl+1 to obtain G’l
Gl+1
Expand
Reduce
Gl
G’l
Denote by Ll the error in prediction:
Ll = Gl - G’l
original
L0, L1, .... = the levels of a Laplacian Pyramid.
What does blurring take away?
What does blurring take away?
smoothed – original
smoothed (5x5 Gaussian)
Laplacian Pyramid
Frequency
Domain
Gaussian
Pyramid
Laplacian
Pyramid
Laplacian
Pyramid
Gaussian
Pyramid
expa
nd
-
=
-
=
-
=
-
=
-
=
-
=
ex
pa
nd
ex
pa
nd
Reconstruction of the original image
from the Laplacian Pyramid
Gl = Ll + G’l
Laplacian
Pyramid
expand
=
+
expand
+
=
expand
+
=
=
from: B.Freeman
Laplacian Pyramid Computational Aspects
Image pyramids
Memory:
2NX2N (1 + 1/4 + 1/16 + ... ) = 2NX2N * 4/3
However coefficients are highly compressable.
Computation:
Li can be computed from G0 with a single convolution
with filter: ki = hi-1 - hi
-
=
hi-1
hi
k1
k2
ki
k3
• Gaussian Pyramid
• Laplacian Pyramid
• Wavelet/QMF
Original
Image
What is a good representation
for image analysis?
Space-Frequency Tiling
Freq.
Standard basis
• Fourier transform domain tells you
“what” (textural properties), but not
“where”.
Spatial
– In space, this representation is too spread out.
• Pixel domain representation tells you
“where” (pixel location), but not
“what”.
Freq.
Fourier basis
– In space, this representation is too localized
• Want an image representation that gives
you a local description of image
events—what is happening where.
Spatial
Freq.
Waveletb basis
– That representation might be “just right”.
Spatial
Wavelets/QMF’s
r
r
F = Uf
• The simplest Wavelet transform: The
Haar transform:
U=
1
1
1
-1
• The inverse transform for Haar:
inv(U) =
0.5000
Apply this over multiple
spatial positions
U=
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
1 -1
0
0
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
0
1 -1
0
0.5000
0.5000 -0.5000
Note: the transform is orthogonal !
The High Frequencies
U=
The Low Frequencies
U=
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
0
0
0
0
0
0
0
0
1
1
0
0
0
0
0
0
1
1
1 -1
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
1 -1
0
0
0
0
0
0
0
1 -1
0
0
0
0
0
1 -1
0
The Inverse Transform
0
Wavelet bands are split recursively
U=
0.5000 0.5000
0
0
0
0 0.5000 0.5000
0
0
0
0 0.5000
0
0
0
0
0
0.5000 -0.5000
0
0
0
0 0.5000 -0.5000
0
0
0
0 0.5000
0
0
0
0
0
0
0
0
0
0.5000
0 0.5000
0
0
0
0
-0.5000
0 0.5000
0
0
0
0
0
0
0.5000
0
0
0
0
0
0
-0.5000
H
L
L
H
H
L
Wavelet Decomposition in 2D
Horizontal high pass
Apply the wavelet transform
separable in both dimensions
Horizontal high pass,
vertical high pass
Horizontal high pass,
vertical low-pass
Frequency domain
Horizontal low pass
Horizontal low pass,
vertical high-pass
• Splitting can be applied recursively:
Horizontal low pass,
Vertical low-pass
The Wavelet coefficients
Fourier transform
=
Gaussian pyramid
*
=
*
pixel image
Fourier bases
are global:
each transform
coefficient
depends on all
pixel locations.
Fourier
transform
pixel domain
image
From: B. Freeman
From: B. Freeman
Wavelet (QMF) transform
Laplacian pyramid
=
*
=
pixel image
Wavelet
pyramid
Laplacian
pyramid
Overcomplete representation.
Low-pass filters, sampled
appropriately for their blur.
Gaussian
pyramid
Overcomplete representation.
Transformed pixels represent
bandpassed image information.
From: B. Freeman
*
Ortho-normal
transform (like
Fourier transform),
but with localized
basis functions.
From: B. Freeman
pixel image
Image Pyramids - Comparison
Image pyramid levels = Filter then sample.
Filters:
Image Pyramids as Linear Transforms
Transform
Basis
Characteristics
Standard
Fourier
Sines+Cosines
Not localized in space
Localized in Frequency
Wavelet
Pyramid
Wavelet Filters
Localized in space
Localized in Frequency
Gaussian Pyramid
Laplacian Pyramid
Localized in space
Not localized in Frequency
Delta
Delta
Fourier
Wavelet
Wavelet Pyramid
space
Applications
•
•
•
•
space
space
Very useful property:
Low entropy of Wavelet bands
Compression
Noise removal
Multi-scale Pattern Matrching
Image Stitching
Applications:
- Compression
- Noise removal
Wavelet Shrinkage
Denoising
Image statistics (or, mathematically,
how can you tell image from noise?)
Noisy image
y = x+n
n ~ N (0, σ )
From: B. Freeman
Clean image
Image domain histogram
From: B. Freeman
From: B. Freeman
Wavelet domain image histogram
Image domain noise histogram
From: B. Freeman
From: B. Freeman
Noise-corrupted full-freq and bandpass images
Wavelet domain noise histogram
But want the bandpass image
histogram to look like this
From: B. Freeman
Bayesian MAP estimator for clean
bandpass coefficient values
Using Bayes theorem
Constant w.r.t.
parameters x.
Let x = bandpassed image value before adding noise.
Let y = noise-corrupted observation.
By Bayes theorem P(x|y) = k P(y|x) P(x)
argmaxx P(x|y) = P(y|x) P(x) / P(y)
The parameters you
want to estimate
Likelihood
function
What you observe
Prior probability
P(x)
maximum
y
P(y|x)
P(y|x)
P(x|y)
P(x|y)
From: B. Freeman
From: B. Freeman
Bayesian MAP estimator for clean
bandpass coefficient values
Bayesian MAP estimator for clean
bandpass coefficient values
y
y
P(y|x)
maximum
maximum
P(y|x)
P(x|y)
P(x|y)
From: B. Freeman
From: B. Freeman
MAP estimate, x̂ , as function of
observed coefficient value, y
Wavelet Shrinkage Pipe-line
x̂
Mappingfunctions
functions
Mapping
y
Transform
Transform
xiw
W
W
yiw
y
Inverse
Inverse
Transform
Transform
WTT
W
)
x
From: B. Freeman
Noise removal results
Simoncelli and Adelson, Noise Removal via
Bayesian Wavelet Coring
From: B. Freeman
More results
More results
Hierarchical Pattern Matching
• Two methods for searching a target
pattern over many scales.
Image Mosaicing
Transformation
Image Blending
Blending
Alpha Blending
When splining two images, transition from one image to
the other should behave:
High Frequencies
Middle Frequencies
Low Frequencies
Multiresolution Spline - Example
Left Image
Right Image
Left + Right
Narrow Transition
Wide Transition
(Burt & Adelson)
Multiresolustion Spline - Using Laplacian Pyramid
Multiresolution Spline - Example
Left Image
Multiresolution Spline - Example
Right Image
Left + Right
Narrow Transition
Wide Transition
Multiresolution Spline
Original - Left
Original - Right
Glued
Splined
(Burt & Adelson)
Multiresolution Spline - Example
The colored version
© prof. dmartin