A Weighted Average of Sparse Representations
is Better than the Sparsest One Alone
Michael Elad and Irad Yavneh
SIAM Conference on Imaging Science ’08
Presented by Dehong Liu
ECE, Duke University
July 24, 2009
Outline
•
•
•
•
•
Motivation
A mixture of sparse representations
Experiments and results
Analysis
Conclusion
Motivation
• Noise removal problem
y=x+v, in which y is a measurement signal, x is the clean signal, v
is assumed to be zero mean iid Gaussian.
• Sparse representation
x=D, in which DRnm, n<m,  is a sparse vector.
• Compressive sensing problem
• Orthogonal Matching Pursuit (OMP)
Sparsest representation
• Question:
“Does this mean that other competitive and slightly inferior sparse
representations are meaningless?”
A mixture of sparse representations
• How to generate a set of sparse representations?
– Randomized OMP
• How to fuse these sparse representations?
– A plain averaging
OMP algorithm
Randomized OMP
Experiments and results
Model:
• y=x+v=D+v
• D: 100x200 random dictionary with entries
drawn from N(0,1), and then with columns
normalized;
• : a random representations with k=10 nonzeros chosen at random and with values
drawn from N(0,1);
• v: white Gaussian noise with entries drawn
from N(0,1);
• Noise threshold in OMP algorithm T=100(??);
• Run the OMP once, and the RandOMP 1000
times.
Observations
350
Random-OMP cardinalities
OMP cardinality
150
300
Random-OMP error
OMP error
Histogram
Histogram
250
100
50
200
150
100
50
0
0
10
20
Candinality
30
0
85
40
200
150
100
0.3
Noise Attenuation
Random-OMP denoising
OMP denoising
250
Histogram
105
0.35
300
Random-OMP denoising
OMP denoising
0.25
0.2
0.15
0.1
50
0
0
90
95
100
Representation Error
0.1
0.2
0.3
Noise Attenuation
0.4
0.05
0
5
10
Cardinality
15
20
Sparse vector reconstruction
3
Averaged Rep.
Original Rep.
OMP Rep.
2
value
1
0
-1
-2
-3
0
50
100
index
150
200
The average representation over 1000 RandOMP representations is
not sparse at all.
Denoising factor based on 1000 experiments
Run RandOMP 100 times for each experiment.
Denoising factor=
0.5
RandOMP Denoising Factor
0.45
OMP versus RandOMP results
Mean Point
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
0.1
0.2
0.3
0.4
OMP Denoising Factor
0.5
Performance with different parameters
Analysis
“
”
The RandOMP is an approximation of the Minimum-Mean-Squared-Error (MMSE) estimate.
Comparison
0.5
1. Emp. Oracle
2. Theor. Oracle
3. Emp. MMSE
4. Theor. MMSE
5. Emp. MAP
6. Theor. MAP
7. OMP
8. RandOMP
Relative Mean-Squared-Error
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
0.5
1

1.5
2
The above results correspond to a 20x30 dictionary. Parameters: True support=3,
x=1, Averaged over 1000 experiments.
Conclusion
• The paper shows that averaging several
sparse representations for a signal lead to
better denoising, as it approximates the
MMSE estimator.