Just Relax ❦ Convex Programming Methods for Subset Selection and Sparse Approximation Joel A. Tropp <[email protected]> The University of Texas at Austin 1 Subset Selection ❦ ❧ Work in finite-dimensional inner-product space Cd ❧ Let {ϕω : ω ∈ Ω} be a dictionary of unit-norm elementary signals ❧ Suppose s is an arbitrary input signal from Cd ❧ Let τ be a fixed, positive threshold ❧ The subset selection problem is to solve min c ∈ CΩ X s− ω∈Ω 2 cω ϕω + τ 2 kck0 2 ❧ Problem arose in statistics more than 50 years ago ❧ Reference: [Miller 2002] Just Relax 2 Applications ❦ ❧ ❧ ❧ ❧ ❧ ❧ ❧ ❧ Linear regression Lossy compression of audio, images and video De-noising functions Detection and estimation of superimposed signals Regularization of linear inverse problems Approximation of functions by low-cost surrogates Sparse pre-conditioners for conjugate gradient solvers ... Just Relax 3 Convex Relaxation ❦ Subset selection is combinatorial min c ∈ CΩ X s− ω∈Ω 2 cω ϕω + τ 2 kck0 2 ❧ References: [Natarajan 1995, Davis et al. 1997] Replace with a convex program min b ∈ CΩ 2 X 1 bω ϕω + γ kbk1 s− ω∈Ω 2 2 ❧ Can be solved in polynomial time with standard software ❧ Reference: [Chen et al. 1999] Just Relax 4 Why an `1 penalty? ❦ `0 quasi-norm Just Relax `1 norm `2 norm 5 Why an `1 penalty? ❦ `0 quasi-norm Just Relax `1 norm `2 norm 6 Why two different forms? ❦ Subset Selection min c ∈ CΩ X s− ω∈Ω 2 cω ϕω + τ 2 kck0 2 Convex Relaxation min b ∈ CΩ Just Relax 2 X 1 s − b ϕ ω ω + γ kbk1 ω∈Ω 2 2 7 Explanation, Part I ❦ ❧ If the dictionary is orthonormal, the `0 problem has an analytic solution ❧ Compute inner products between signal and dictionary cω = hs, ϕω i ❧ Apply hard threshold operator with cutoff τ to each coefficient Just Relax 8 Explanation, Part II ❦ ❧ If the dictionary is orthonormal, the `1 problem has an analytic solution ❧ Compute inner products between signal and dictionary bω = hs, ϕω i ❧ Apply soft threshold operator with cutoff γ to each coefficient Just Relax 9 The Coherence Parameter ❦ Insight: Subset selection is easy provided that the dictionary is nearly orthonormal. ❧ [Donoho–Huo 2001] introduces the coherence parameter def µ = max λ6=ω |hϕλ, ϕω i| ❧ Related to packing radius of dictionary, viewed as subset of Pd−1(C) √ ❧ Possible to have |Ω| = d and µ = 1/ d 2 Just Relax 10 An Incoherent Dictionary ❦ 1 1/√d Impulses Just Relax Complex Exponentials 11 Result for Subset Selection ❦ Theorem A. Fix an input signal and a threshold τ . Suppose that ❧ copt solves the subset selection problem with threshold τ ; ❧ copt contains no more than 13 µ−1 nonzero components; and ❧ b? solves the convex relaxation with γ = 2 τ . Then it follows that ❧ ❧ ❧ ❧ copt(ω) = 0 implies b?(ω) = 0; |b?(ω) − copt(ω)| ≤ 3 τ for each ω; in particular, b?(ω) 6= 0 so long as |copt(ω)| > 3 τ ; and the relaxation has a unique solution. Just Relax 12 Error-Constrained Sparse Approximation ❦ ❧ Suppose s is an arbitrary input signal from Cd ❧ Let ε be a fixed, positive error tolerance ❧ The error-constrained sparse approximation problem is min c ∈ CΩ kck0 subject to ω∈Ω cω ϕω ≤ ε ω∈Ω bω ϕ ω ≤ δ X s− 2 ❧ Its convex relaxation is min b ∈ CΩ Just Relax kbk1 subject to X s− 2 13 Result for Sparse Approximation ❦ Theorem B. Fix an input signal, and let m ≤ 31 µ−1. Suppose that ❧ copt solves the sparse approximation problem with tolerance ε; ❧ copt contains no more than m nonzero components; and √ ❧ b? solves the convex relaxation with tolerance δ = ε 1 + 6 m. Then it follows that ❧ copt(ω) = 0 implies p b?(ω) = 0; ❧ kb? − coptk2 ≤ δ 3/2; and ❧ the relaxation has a unique solution. [Donoho et al. 2004] contains related results. Just Relax 14 For more information. . . ❦ Just Relax: Convex Programming Methods for Subset Selection and Sparse Approximation Available from <http://www.ices.utexas.edu/~jtropp/> or write to <[email protected]> Other Work. . . ❧ ❧ ❧ ❧ Greedy and iterative algorithms for sparse approximation Other types of sparse approximation Construction of packings in Grassmannian manifolds Matrix nearness and inverse eigenvalue problems Just Relax 15
© Copyright 2026 Paperzz