INFONET, GIST
Journal Club (2015. 11. 5)
Support recovery with orthogonal matching pursuit in
the presence of noise.
Authors:
Jian Wang
Publication:
IEEE transactions on signal processing,
vol. 63, no. 21, Nov. 2015.
Jang Jehyuk
Speaker:
Short summary:
OMP is one of greedy algorithms for finding a sparse solution in underdetermined linear
simultaneous equations. It is popular by its simple principle.
There are many literatures for the performance analysis of OMP, especially for the sufficient
conditions for successful working. Whereas, relatively few literatures deal with necessary
conditions, and I haven’t found the literatures which dictate error rate of OMP.
In this paper, Wang provides the sufficient and necessary conditions of OMP. In addition,
Wang’s article provides upper bound for recovery error characterized by SNR and magnitudes of
the sparse vector. Sufficient conditions have been analyzed in many other papers, therefore, it is
not a novel. The necessary condition resembles some empirical results in prior works as well.
The sufficient and necessary conditions have a restriction on restricted isometry constant (RIC)
of sensing (system) matrix. However, in my opinion, the bound for recovery error seems a
completely fresh result since it doesn’t require any restriction on the RIC. The restriction can be
an obstacle in adopting the conditions into our practical models, where it is hard to calculate RIC
since it requires intractable computations.
Unfortunately, I’ve found defects in the proof of the bound for recovery error. Regardless of
whether the author intentionally missed it or not, the corrected bound requires a strong restriction
on the RIC even more than the sufficient and necessary conditions. I’ve tried to find a nontrivial
bound without any restriction on the RIC based on the proof of Wang’s paper, but I failed. In
conclusion, a nontrivial bound without the restriction on RIC cannot be found unless there is a
novel alternative tricks in the proof.
In this presentation, I first list the results of Wang’s paper, and then I focus on the proof for the
recovery rate and the defects.
I. PRELIMINARIES
A. Notations
Let   {1, 2,
S . xS 
ΦS 
S
m S
, n} . For S   , T \ S is the set of all elements contained in T but not in
represents a restriction of the vector x to the elements with indices in S .
is a submatrix of Φ that only contains columns indexed by S . If Φ S is full
Φ†S   ΦTS ΦS  ΦTS
1
column rank, then
is the pseudoinverse of Φ S . span(ΦS ) represents the
P   I - PS
span of columns in Φ S . S
is the projection onto the orthogonal complement of
span(ΦS ) , where I denotes the identity matrix. i denotes the i th column of Φ .
B. Sensing model and restricted isometry property
Consider following noisy CS model,
y = Φx + v
where, y 
m1
, Φ
mM
M 1
, x
(1)
, and m  M . Suppose that x is sparse with support
T with cardinality T  K .
Wang’s paper presents sufficient and necessary conditions for the recovery of unknown x
for given y and Φ . The bounds in conditions are function of RIC, SNR , and MAR , where
SNR:=
Φx
v
2
2
2
2
and MAR :
2
K min jT x j
x
is the minimum-to-average ratio of the input signal.
2
2
I therefore introduce the definition of restricted isometry property, which indicates
orthornomality of Φ when mapping sparse vectors.
Definition(Restricted isometry property)
Consider the model (1). If there exists a constant  K   0,1 such that, for every K -sparse
vector x K ,
(1   K ) x K
2
2
 Φx K
2
2
 (1   K ) x K
2
2
(2)
Then, Φ is said to satisfy the RIP of order K with the RIC  K .
2
One of major goal in compressed sensing theory is finding a good sensing matrix Φ with
smaller restricted isometry constant(RIC)  K . Note that computation of  K is NP-hard
problem, therefore, any analysis having restriction on  K would not be adoptable in practical
since no one do know exact value of RIC of their sensing matrix unless the intractable
computation is done.
C. Orthogonal matching pursuit
Now I introduce the pseudo code of OMP algorithm.
Table I, The OMP algorithm
Input
Φ , y , and sparsity K .
Initialize
iteration counter
k 0,
Estimated support T   ,
0
0
and residual vector r = y
While
k  K do
k  k 1
Identify
t k  arg max i , r k 1
Enlarge
T k  T k 1  t k
Estimate
i \T k 1
xk  arg min y - Φu
u:supp(u )=T k
Update r  y - Φx
k
2
k
End
Output
the estimated
T k and vector x K
In short, at each iteration OMP compares correlations between y and i , the maximum
argument is chosen as one element of estimated support, and then it removes the estimated
ingredient from y .
3
II. MAIN RESULTS
A. Sufficient and necessary conditions
A sufficient condition for the exact recovery with OMP in the presence of noise is presented in
Wang’s paper.
Theorem 3.1 (Sufficient condition) :
Suppose that the measurement matrix Φ satisfies the RIP with  K 1 
1
. Then
K 1
OMP performs the exact support recovery of any K -sparse signal x from its noisy
measurements y = Φx + v , provided that
SNR 
 
1
2 K (1   K 1 )


K  1  K 1  MAR
(3)
Besides, a necessary condition is also presented, and Wang claims that other related or
previous works haven’t provided the necessary condition.
Theorem 3.2 (Necessary condition) :
If one wishes to accurately recover the support of any K -sparse signal x from its noisy
measurements y = Φx + v with OMP, then the SNR should satisfy
SNR 
1 
K (1   K 1 )

K  K 1  MAR
(4)
In Theorem 3.2, note that the denominator in the right hand side of (4) should be positive to be
a nontrivial bound since
 K 1 
SNR is nonnegative. In other words, it should be assumed that
1
. The paper doesn’t refer about this.
K
Both sufficient and necessary conditions provided in Wang’s paper have restrictions on  K 1 ,
therefore, it is hard to exploit the conditions into our practical problems since the computation of
RIC is intractable.
B. Upper bound for recovery error rate
Theorem 4.1 :
4
Let  : max i , jT xi
x j . Then if SNR   2 2K3/2 , OMP recovers the support of K
-sparse signal x from its noisy measurements y = Φx + v with error rate
error  C 2 21/2K
(5)
where C is a constant.
Theorem 4.1 claims that recovery error rate would be bounded to a proportion of RIC if the
bound is nontrivial. Remarks in Wang’s paper claim that C is reasonably small and the
bound is reasonable (nontrivial) by explaining based on some examples. In specific, the author
claims that the small C  12 is obtained when   1 ,  2 K  0.001 , and K is sufficiently
large. In addition, the remarks insist the bound is reasonable from the fact that error  0 when
SNR   and Φ is an orthonormal matrix, i. e.,  2 K  0 . However, I think the remarks are
not sufficient to say the bound is a good or nontrivial. Indeed, I numerically evaluate the right
hand side of (5) based on the proof, where C is provided in a implicit form, for varying  2 K
when K  12 ,   1 , and SNR   . Following Fig. 1 shows the result.
K=12/ =1/ 0< <(1/49)
2.8
upper bound of error rate(Theorem4.1)
2.6
X: 0.02041
Y: 2.635
2.4
2.2
2
1.8
1.6
1.4
1.2
X: 0.001377
Y: 0.9969
1
X: 1e-220.8
Y: 0.9167 0
0.005
0.01
0.015
0.02
0.025

Fig.
1
Numerical evaluations for the upperbound of recovery error rate
This evaluation represents that the bound is nontrivial only approximately for  2 K  0.0014 .
Otherwise, the bound doesn’t provide any insight since we already know error  1 .
5
III. DEFECTS IN PROVIDED PROOF
In this section, I intoduce sketch of proof for Theorem 4.1 not to make you understand, but to
point out the defects, which result in constaint on  2 K even bitter than the constraint in
Theorem 3.1 and Theorem 3.2.
Before proceeding, I introduce notations and definitions used in proof. For notational
simplicity, let  :  2 K . At the k th iteration of OMP ( 0  k  K ), let  k : T \ T k denote the
set of missed detection of support indices. For given constant   (0,1] , let k denote the
subset of  k corresponding to the  K  largest elements (in magnitude) of x  k . Also, let xk
denote the  K  th largest element (in magnitude) in x  k . The author fixes    1/ 2 . If
k
k
k
k
 K    , then set    and x  0 . Since OMP totally runs K iterations before
stopping, the error rate of the support recovery can be given by
error :
T K \T
T
K

T

K
(6)
K
The recovery error rate is obtained from lower and upper bounds for energy of the residue
vector at last iteration, i. e., bounds for energy of r K
2
2
. The lower bound can be simply
obtained by
rK
2
2
 y - Φx K
2
2
 Φ( x - x K ) + v
(a)
2
2
2
2
2
2
 (1   ) Φ(x - x K )  (1/   1) v
(b )
 (1   )(1   ) x - x
(c)
 (1  2 ) x - x K
2
2
 (1  2 ) x  K
2
2
2
 (1/   1) v
 (1/   1) v
 (1  2 ) (x - x K )  K
(d )
K 2
2
2
2
2
2
2
2
2
 (1  2 )   xmin  (1/   1) v
K
2
(7)
2
 (1/   1) v
 (1/   1) v
2
2
2
6
where ( a ) uses the fact that u + v 2  (1  p) u 2  (1/ p  1) v
2
2
2
2
for any u, v , and p  0 ,
(b) follows from the RIP, ( c ) is because    1/ 2 and (1   )(1   2 )  (1   )2  1  2 , and
(d ) is due to the fact x K is supported on T K and hence xKK  xTK\T K  0 .
Here, defect
is
that
1  2
should be nonnegative. The author assumes that
1  2  1  2 1/2  0 but doesn’t refer. This assumption leads to a restriction on RIC of
1
 2K .
4
In consequent, lower bound of (7) is trivial. It is not for worth as a meaningful bound, but just
for deriving the term  K . An upper bound for r K
2
2
derived from now on therefore should be
meaningful. For this purpose, the author observes the energy behavior in residue vector for every
iteration as presented by following proposition.
Proposition 4.3:
For any 0  k  K   1/2 K  , the residual of OMP satisfies
rk
2
2
2
 r k 1  (1  7 1/2 ) xk1/2
2
2
(8)
Proof is omitted here. Proof of Proposition 4.3 provided in Wang’s paper also has same
kind of defects in proof of Theorem 4.1. By similar reasons for defect in (7),
1
  2 K is
36
assumed.
Now I turn to obtaining the upper bound for r K
T  {1,
2
2
. Without loss of generality, we assume that
, K } and that the elements of {xi }iK1 are in descending order of their magnitudes.
Then from the definition of xk we have that for any k  0 , k   K   K ,
xk  xk   K 
(9)
By applying Proposition 4.3, the upper bound of energy is obtained by
7
rK
2
2
K 1

 r0   rk
2
2
(a)
k 0
 y 2
2
 y 2
2
(b )
 y 2

k 1
K   K 

k 1
 y 2
2
K   K 

k 1
K


2
r
k   K  1
2
 r k 1
k 2
2
2

 r k 1
(1  7 ) xk
K   K 
2
2
2
2

2
(10)
(1  7 ) xk   K 
(1  7 ) xk
2
2
where ( a ) uses the facts that  K   1 and that the energy of residual of the OMP algorithm is
always non-increasing with the number of iterations (i. e. r k
2
2
2
 r k 1 , k  0 ), and (b) is
2
from (32) .
In common with the case in (7), it should be assumed that 1  7  1  7 1/2  0 , i.e.,  2 K 
1
,
49
which is missed in the statement of Theorem 4.1.
In conclusion, Theorem 4.1 also has a restriction on the RIC, even stricter than the restrictions
in sufficient and necessary conditions. I tried to modify the proof to remove the restriction while
following the idea of Wang’s proof, but I failed. The key in Wang’s proof for a nontrivial bound
is Proposition 4.3. Without employing Proposition 4.3, we can remove the restriction, but the
derived bound would be trivial. Meanwhile, Wang fixed    1/ 2 , but it is not mandatory since
we can tighten the bound and loosen the restriction on the RIC by letting  be a variable of
SNR . In specific, when we choose  
2K 


SNR  1  1 / SNR , then it follows that if
1
1    K 2 , then
and SNR 
1  2K
1    2 K 
error 
(1   )C
B
1    SNR
(11)
where
C
SNR 
1
1

(12)
and
8
B

1    2 K 
1    K  2 SNR
1   1   

2
(13)
Evaluation of the right hand side of (11) is presented by following figure,
K=12/ =1/Noiseless/Note that 1/(2K+1)=0.04
upper bound of error rate(modified version of Theorem 4.1)
1.4
X: 0.05
Y: 1.105
X: 0.04
Y: 1.083
1.2
1
0.8
X: 0.0204
Y: 0.5614
0.6
0.4
0.2
X: 0.0013
Y: 0.03636
0
X: 0.0001 0
Y: 0.0028
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04
0.045
0.05

Fig.
2
Numerical evaluation of the modified bound for error rate
9