Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
Compressed-Sensing Recovery of
Images and Video
Using Multihypothesis Predictions
MH
Predictions
Results
Chen Chen, Eric W. Tramel, and James E. Fowler
Conclusions
Department of Electrical & Computer Engineering
Geosystems Research Institute
Mississippi State University, MS USA
November 2011
CS Overview
Images and
Video CS
Using MH
Predictions
Compressed Sensing (CS)
Background
For x ∈ RN and a set of linear projections, Φ ∈ RM×N
where M ≪ N; i.e.,
y = Φx
MH
Predictions
x can be reconstructed from y by solving solving
Chen,
Tramel, &
Fowler
Results
min ||x||1
Conclusions
x
s.t. Φx = y
Complications
For N large and dense Φ:
Explicit storage of Φ becomes impractical
High computational complexity for reconstruction
Block Compressed Sensing
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Block Compressed Sensing (BCS)
Image partitioned into small blocks (B × B)
yj = ΦB xj
Background
MH
Predictions
ΦB : MB × B2 ,
xj : block j of image
Results
Conclusions
Smooth Projected Landweber (SPL)
BCS-SPL (Gan DSP2007, Mun & Fowler ICIP2009)
Iterative-thresholding reconstruction
Wiener filter to smooth blocky artifacts
Simple, fast
Multiscale BCS-SPL
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Conclusions
Multiscale BCS-SPL
MS-BCS-SPL (Fowler et al. EUSIPCO2011)
Assumes wavelet-domain sampling with blocks;
i.e., BCS is deployed within each subband of DWT
Different subrate at each decomposition level
Reconstruction uses similar SPL procedure
Residual Reconstruction
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
Residual Reconstruction (RR)
Prediction: x̃
Residual: r = x − x̃
Residual projection:
MH
Predictions
Results
q = Φr = Φ(x − x̃) = y − Φx̃
Conclusions
Residual r tends to be more compressible than x
Reconstruction:
x̂ = x̃ + Reconstruct(q, Φ)
where Reconstruct(·) is some suitable CS recovery
Residual Reconstruction
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
The Prediction Problem
We want to find x̃ so that x̃ ≈ x; i.e.,
x̃ = arg min kx − pk22
p∈P(xref )
MH
Predictions
but:x is unknown at reconstruction
Results
Use some initial reconstruction x̂ instead:
Conclusions
x̃ = arg min kx̂ − pk22
p∈P(xref )
Alternate approach—cast problem into
measurement domain:
x̃ = arg min ky − Φpk22
p∈P(xref )
MH Prediction for Video
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
Video-Frame Reconstruction
Form prediction of each frame in a video sequence
Block-based sampling in BCS permits ME/MC on
blocks in measurement domain
Multi-pass recovery: use recovered “key” frames
as references for multi-hypothesis (MH) prediction
of subsequent frames
MH Prediction for Video
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Linear Combination of Hypotheses
Prediction for block x̃t,i is linear combination of
hypothesis blocks, Ht,i :
x̃t,i = Ht,i w∗
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
Estimating w∗ : an ill-posed problem
Distance-Weighted ℓ2 (Tikhonov) Regularization
ŵ = arg min kyt,i − ΦHt,i wk22 + λkΓwk22
w
ℓ1 -Regularization (Do et al. ICIP2009, Prades-Nebot
et al. PCS2009)
ŵ = arg min kyt,i − ΦHt,i wk22 + λkwk1
w
MH Prediction for Video
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
Tikhonov Regularization—Distance Weighting
Bias regularization toward solutions with
projections close to measurements:


kyt,i − Φh1 k2
0


..
Γ=

.
0
kyt,i − ΦhK k2
Experimental results show distance-weighted ℓ2
regularization yields better predictions than ℓ1
regularization
MH-BCS-SPL for Images
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Subblock-Based MH Prediction
S1
S2
S3
S4
Background
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
b
MH
Predictions
current subblock
hypothesis subblock
search window
(c)
(b)
(a)
S1 S2
S1 0
0 S2
0
S3 S4
0
0
S3 0
0
0
0
0
0
0 S4
zero-padding
B
B × B blocks divided into b × b subblocks
Minimum subblock size, b = 21 B
Maximum subblock size, b = B
Multiple hypotheses drawn from spatial
surrounding area of a subblock
Predictions found for subblocks—zero padded to
block size
MH-BCS-SPL for Images
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
Algorithm
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
INITIALIZATION: B = 32, b = 16, w = 32
x̄ = BCS-SPL(y, Φ, Ψ, B)
repeat
x̃i = MH Prediction(x̄, y, Φ, b, w, B)
r = y − Φx̃i
r̂ = BCS-SPL(r, Φ, Ψ, B)
x̂i = x̃i + r̂
if update criterion satisfied then
b ← b × 2, w ← w × 2
end if
Update x̄ ← x̂i
i=i+1
until stopping criterion satisfied
x̄: initial reconstruction
Φ: measurement operator, Ψ: sparsity transform
MH-MS-BCS-SPL
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
MH-MS-BCS-SPL
Combines MS-BCS-SPL with MH predictions
carried out in wavelet domain
Block size Bl = [16, 32, 64]
Initial subblock size bl = 18 Bl
MH-MS-BCS-SPL
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
MH Predictions for
Video
MH Predictions for
Images
Results
Conclusions
Algorithm
1: repeat
2:
ẋi = Ωx̄
3:
for 1 ≤ l ≤ L do
4:
for each subband θ ∈ {H, V, D} do
˜ẋi (θ) = MH Prediction(ẋi (θ), y(θ), Φ, Bl , bl , w)
5:
6:
end for
7:
end for
8:
x̃i = Ω−1˜ẋi
9:
x̂i = x̃i + BCS-SPL(y − Φx̃i , Φ, Ψ, {Bl })
10:
if update criterion satisfied then
11:
for 1 ≤ l ≤ L do
12:
bl ← bl × 2
13:
end for
14:
w←w×2
15:
end if
16:
Update x̄ ← x̂i
17: until stopping criterion satisfied
Video Frame Recovery Performance
Bidirectional Recovery of Foreman Frame
Images and
Video CS
Using MH
Predictions
38
36
Chen,
Tramel, &
Fowler
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
Recovery PSNR (dB)
34
Background
32
30
28
26
24
0.1
0.2
0.3
0.4
Subrate (M/N)
RR w/ MH-TIK
RR w/ MH-GPSR
BCS-SPL
0.5
Video Frame Recovery Performance
Bidirectional Recovery of Football Frame
Images and
Video CS
Using MH
Predictions
32
Chen,
Tramel, &
Fowler
MH
Predictions
Results
Results for Video
Results for Images
Recovery PSNR (dB)
Background
30
28
Conclusions
26
24
0.1
0.2
0.3
0.4
Subrate (M/N)
RR w/ MH-TIK
RR w/ MH-GPSR
BCS-SPL
0.5
Image Recovery Performance, Lenna
Images and
Video CS
Using MH
Predictions
40
Chen,
Tramel, &
Fowler
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
36
Recovery PSNR (dB)
Background
38
34
32
30
28
0.1
0.2
0.3
0.4
0.5
Subrate (M/N)
MH-MS
MH
MS
BCS-SPL
TV
Image Recovery Performance, Barbara
Images and
Video CS
Using MH
Predictions
38
36
Chen,
Tramel, &
Fowler
MH
Predictions
Results
Results for Video
Results for Images
Recovery PSNR (dB)
Background
34
32
30
28
Conclusions
26
24
22
0.1
0.2
0.3
0.4
0.5
Subrate (M/N)
MH-MS
MH
MS
BCS-SPL
TV
Visual Comparison—Barbara
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
Visual Comparison
Barbara Recovery Inset, S = 0.1
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
BCS-SPL
TV
MS-BCS-SPL
Visual Comparison
Barbara Inset, S = 0.1
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
MS-GPSR
MH-MS-BCS-SPL
MH-BCS-SPL
Results for Images
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Results for Video
Results for Images
Conclusions
Runtime Comparison
Conclusions
Images and
Video CS
Using MH
Predictions
Chen,
Tramel, &
Fowler
Background
MH
Predictions
Results
Conclusions
Conclusions
MH predictions use a distance-weighted Tikhonov
regularization to find the best linear combination of
hypotheses
Multiple predictions were used to create a
measurement domain residual of the signal to be
recovered, which is generally more compressible
than the original signal
MH predictions work both for video and still images
and the performance improvement is significant
M ATLAB code
http://www.ece.msstate.edu/˜fowler/BCSSPL/