Thank you for participating in
and contributing to our minisymposium on
“Locally Adaptive Patch-based
Image and Video Restoration”
Session I: Today (Mon) 10:30 – 1:00
Session II: Wed
Same Time, Same Room
Milanfar et al. EE Dept, UCSC
1
Local Adaptivity +
Patch-Based Approaches
• State of the Art Performance
A Convergence of Ideas
Extremely Popular
Milanfar et al. EE Dept, UCSC
2
Patch-based methods have become so
popular in fact ….
Patchy the Pirate
Milanfar et al. EE Dept, UCSC
3
Multi-dimensional Kernel
Regression for Video Processing
and Reconstruction
Peyman Milanfar*
EE Department
University of California, Santa Cruz
*Joint work with Hiro Takeda (UCSC),
Mattan Protter and Michael Elad (Technion),
Peter van Beek (Sharp Labs of America)
SIAM Imaging Science Meeting, July 7, 2008
Milanfar et al. EE Dept, UCSC
4
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC
5
Summary
• Motivation:
– Existing methods make strong assumptions
about signal and noise models.
– Develop “universal”, robust methods based on
adaptive nonparametric statistics
• Goal:
– Develop the adaptive Kernel Regression
framework for a wide class of problems,
including video processing; producing
algorithms competitive with state of the art.
Milanfar et al. EE Dept, UCSC
6
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC
7
Kernel Regression Framework
• The data model
A sample
Zero-mean, i.i.d noise (No other assump.)
The sampling position
The number of samples
The regression function
• The specific form of
may remain unspecified for now.
Milanfar et al. EE Dept, UCSC
8
Local Approximation in KR
• The data model
• Local representation (N-terms Taylor expansion)
• Note
Unknowns
– With a polynomial basis, we only need to estimate the first unknown,
– Other localized representations are also possible, and may be advantageous.
Milanfar et al. EE Dept, UCSC
9
Optimization Problem
• We have a local representation with respect to each sample:
• Minimization
This term give
the estimated
pixel value at x.
N+1 terms
The regression
order
The choice of the
kernel function is
open, e.g. Gaussian.
Milanfar et al. EE Dept, UCSC 10
Locally Linear Estimator
• The optimization yields a pointwise estimator:
Kernel
function
The smoothing
parameter
The weighted linear
combinations of the
given data
Equivalent
kernel function
The regression
order
• The bias and variance are related to the regression order and
the smoothing parameter:
– Large N  small bias and large variance
– Large h  large bias and small variance
Milanfar et al. EE Dept, UCSC 11
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel
(AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 12
(2D) Data-Adaptive Kernels
Classic kernel
Data-adapted kernel
• Take not only spatial distances, but also radiometric
distances (pixel value differences) into account
• Data-adaptive kernel function
• Yields locally non-linear estimators
Milanfar et al. EE Dept, UCSC 13
Simplest Case: Bilateral Kernels
Spatial
kernel
Low noise case
Radiometric
kernel
=
.
=
.
=
.
Milanfar et al. EE Dept, UCSC 14
Better: Steering Kernel Method
Local dominant
orientation estimate
based on local
gradient covariance
H. Takeda, S. Farsiu, P. Milanfar, “Kernel Regression for Image Processing and Reconstruction”,
IEEE Transactions on Image Processing, Vol. 16, No. 2, pp. 349-366, February 2007.
Milanfar et al. EE Dept, UCSC 16
Steering Kernel
Steering matrix
Local dominant orientation estimation
• Kernel adapted to locally dominant structure
• The steering matrices scale, elongate, and rotate the kernel
footprints locally.
Elongate
Rotate
Scale
Milanfar et al. EE Dept, UCSC 17
Steering Kernel (Low Noise)
• Kernel weights and footprints:
Steering kernel as
a function of xi
with x held fixed
Low noise case
Weights
Footprints
Steering kernel
as a function of
x with xi
and
Hi held fixed
Milanfar et al. EE Dept, UCSC 19
Steering Kernel (High Noise)
• Kernel weights and footprints:
Steering kernel as
a function of xi
with x held fixed
Weights
Footprints
Steering kernel
as a function of
x with xi
and
Hi held fixed
• Steering approach
provides stable
weights even in the
presence of
significant noise.
High noise case
Milanfar et al. EE Dept, UCSC 20
Some Related (0th-order)
Methods
• Non-Local Means (NLM)
–
A. Buades, B. Coll, and J. M. Morel. “A review of image denoising algorithms, with a new one.”
Multiscale Modeling & Simulation, 4(2):490-530, 2005.
• Optimal Spatial Adaptation (OSA)
–
C. Kervrann, J. Boulanger “Optimal spatial adaptation for patch-based image denoising.” IEEE Trans.
on Image Processing, 15(10):2866-2878, Oct 2006.
SKR
NLM
OSA
2
2
4
4
6
6
8
8
10
10
12
12
14
14
16
16
18
18
20
20
2
4
6
8
10
12
14
16
18
20
2
4
6
8
10
12
14
16
18
20
Milanfar et al. EE Dept, UCSC 21
Adaptive Kernels for Interpolation
• When there are missing pixels:
– We cannot have the radiometric distance.
– Using a “pilot” estimate, fill the missing pixels:
• Classic kernel regression
• Cubic or bilinear interpolation
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
Milanfar et al. EE Dept, UCSC 22
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 23
Kernel Regression in 3-D
• Setup is similar to 2-D, but…..
• Data samples come from
various (nearby) frames
• Signal “structure” is now in 3-D
• We can perform
– Denoising
– Spatial Interpolation
– Frame rate upconversion
– Space-time super-resolution
Spatial
gradients
Temporal
gradients
Milanfar et al. EE Dept, UCSC 24
Kernel Regression in 3-D Cont.
• Two ways to proceed
– Adaptive Implicit-Motion Steering Kernel (AIMS)
• Roughly warp the data to “neutralize” large motions
• Implicitly capture sub-pixel motions in 3-D Kernel
– Motion-Aligned Steering Kernel (MASK)
• Estimate motion with subpixel accuracy
• Accurately warp the kernel (instead of the data)
Milanfar et al. EE Dept, UCSC 25
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 26
AIMS Kernel in 3-D
• Steering kernel visualization examples
A plane structure
Steering kernel weights
Isosurface
A tube structure
Milanfar et al. EE Dept, UCSC 27
AIMS Motion Compensation
• Large displacements make
orientation estimation difficult.
The local kernel
effectively spread
along the local
motion trajectory.
The local kernel
effectively spread
along the local
motion trajectory.
• By neutralizing the large
displacement, the steering kernel
can effectively spread again.
The local kernel
after motion
compensation.
Shift
down
Shift
up
Small motions
Large motions
Important: The compensation
does not require subpixel accurate motion estimation, nor does it require interpolation
Milanfar et al. EE Dept, UCSC 28
AIMS Contains Implicit Motion
“Small” motion vector
Space-time gradients
of roughly compensated data
(Eigenvalues of C)
Optical flow equation
Assuming the patch moves
with approximate uniformity
Homogeneous Optical Flow Vector
Milanfar et al. EE Dept, UCSC 29
AIMS Summary
• AIMS is a two-tiered approach.
1. Neutralize whole-pixel motions.
2. 3-D SKR with implicit subpixel motion
information
Steering matrices
estimated from the motion
compensated data in 3-D.
Milanfar et al. EE Dept, UCSC 30
Foreman Example
Input video
(QCIF: 144 x 176 x 28)
Lanczos
(frame-by-frame upscaling)
AIMS
Factor of 2 upscaling
Milanfar et al. EE Dept, UCSC 32
Spatial Upscaling Example
Input (200 x 200)
Upscaled image by AIMS
(multi-frame, 5 frames), 400x400
Milanfar et al. EE Dept, UCSC 33
Spatiotemporal Upscaling
Input video
(200 x 200 x 20)
Spatiotemporal
classic kernel
regression
(400 x 400 x 40)
Single frame
steering kernel
regression
(400 x 400 x 20)
AIMS
regression
(400 x 400 x 40)
Milanfar et al. EE Dept, UCSC 34
Outline
• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions
Milanfar et al. EE Dept, UCSC 35
Motion-Aligned Steering Kernel
• Motion is explicitly estimated to subpixel accuracy
• Kernel weights are aligned with the local motion
vectors using warping/shearing
• The warped kernel acts directly on the data
– Handles large and/or complex motions
Accurate, explicit motion estimates
“2-D motion-steered”
(spatial) kernel
1-D (temporal) kernel
Milanfar et al. EE Dept, UCSC 36
Intuition Behind the MASK
2-D “motion-steered” (spatial) kernel
1-D (temporal) kernel
Milanfar et al. EE Dept, UCSC 37
The Shapes of MASK
• Spreads along spatial orientations and local motion vectors.
Local data
Slices of
MASK kernels
Milanfar et al. EE Dept, UCSC 38
A Comparison of AIMS and
MASK
• Spin Calendar video
Input video
(200 x 200 x 20)
AIMS
(400 x 400 x 40)
MASK
(400 x 400 x 40)
Milanfar et al. EE Dept, UCSC 40
A Comparison of AIMS and
MASK
• Foreman video
Input video
(QCIF: 144 x 176 x 28)
AIMS + BTV deblurring
(CIF: 288 x 352 x 28)
MASK + BTV deblurring
(CIF: 288 x 352 x 28)
Milanfar et al. EE Dept, UCSC 41
Conclusions
• We extended the 2-D kernel regression framework to 3-D.
– Illustrated 2 distinct approaches
• AIMS: Avoids subpixel motion estimation, needs comp. for large motions
• MASK: Needs subpixel motion estimation, deals directly with large motions
– Which is better? Depends on the application.
• The overall 3-D SKR framework is simultaneously well-suited
for spatial, temporal, and spatiotemporal
– upscaling, denoising, blocking artifact removal, superresolution
– not only in video but in general 3-D data sets.
• Future work
– Integration of deblurring directly in the 3-D framework
– Computational complexity
Milanfar et al. EE Dept, UCSC
42