Convex regularization of hybrid
discrete-continuous inverse problems
Christian Clason1
Karl Kunisch2
1 Faculty of Mathematics, Universität Duisburg-Essen
2 Institute of Mathematics and Scientific Computing, Karl-Franzens-Universität Graz
Workshop Recent Developments in Inverse Problems
WIAS, Berlin, September 18, 2015
Overview General approach Multi-bang penalty Numerical examples
1 / 34
Motivation: discrete optimization
L0 penalty
Z
kuk0 :=
Ω
|u(x)|0 dx
|t|0 :=
0
1
if t = 0
if t 6= 0
Lebesgue measure of support of u
popular in sparse optimization
binary penalty
combinatorial optimization
difficulty: non-smooth, non-convex, not lower-semicontinuous
not a norm
not coercive
Overview General approach Multi-bang penalty Numerical examples
2 / 34
Binary penalties
min F(u) + G(u)
u
F(u) tracking or discrepancy term
1
G(u) sparsity penalty
[Ito, Kunisch 2012]
G(u) =
α
kuk2L2 + βkuk0
2
u(x) = 0 almost everywhere
separate penalization of support (β), magnitude (α)
α > 0 necessary
Overview General approach Multi-bang penalty Numerical examples
3 / 34
Binary penalties
min F(u) + G(u)
u
2
G(u) multi-bang penalty
Z
G(u) =
Ω
[Clason, Kunisch 2013]
d
Y
α
|u(x)|2 + β
|u(x) – ui |0 dx
2
i=1
u(x) ∈ {u1 , . . . , ud } almost everywhere
motivation: discrete control (voltages, velocities, materials)
β > 0 penalizes free arc where u(x) 6= ui
α > 0 penalizes magnitude of u(x) = ui
Overview General approach Multi-bang penalty Numerical examples
3 / 34
Binary penalties
min F(u) + G(u)
u
3
G(u) switching penalty, u = (u1 , u2 )
G(u) =
ZT
0
[Clason, Ito, Kunisch 2014]
α
|u(t)|22 + β|u1 (t)u2 (t)|0 dt
2
u1 (t)u2 (t) = 0 almost everywhere
β > 0 penalizes free arc where u1 6= 0 and u2 6= 0
α > 0 penalizes magnitude of active ui
Overview General approach Multi-bang penalty Numerical examples
3 / 34
1
Overview
2 General approach
Convex relaxation
Moreau–Yosida regularization
Semismooth Newton method
3 Multi-bang penalty
Optimality system
Structure of solution
Numerical solution
4 Numerical examples
Overview General approach Multi-bang penalty Numerical examples
4 / 34
Convex relaxation: motivation
f : R → R differentiable:
derivative:
f (u + h) – f (u)
h
h→0
f 0 (u) = lim
f (u)
geometrically:
f 0 (u) tangent slope
ū
Fermat principle:
f (ū) = minu f (u) ⇒
f (ū) + hf 0 (ū), ui
u
f 0 (ū)
=0
calculus for f 0
Overview General approach Multi-bang penalty Numerical examples
5 / 34
Convex relaxation: motivation
f (ū) + hf 0 (ū; 1), ui
f : R → R not differentiable, convex:
f (u)
directional derivative:
f 0 (u; h) = lim+
t→0
f (u + th) – f (u)
t
ū
but: for all h,
f 0 (ū; h) 6= 0
Overview General approach Multi-bang penalty Numerical examples
u
f (ū) + hf 0 (ū; –1), ui
6 / 34
Convex relaxation: motivation
f : R → R not differentiable, convex:
subdifferential:
∂f (u) = u∗ : hu∗ , hi 6 f 0 (u; h)
geometrically:
∂f (u) set of tangent slopes
f (u)
∂f (ū)
ū
f (ū) + h0, ui
u
Fermat principle:
f (ū) = minu f (u) ⇒ 0 ∈ ∂f (ū)
calculus for ∂f
Overview General approach Multi-bang penalty Numerical examples
6 / 34
Fenchel duality
V Banach space, V ∗ dual space
f : V → R := R ∪ {+∞} convex,
subdifferential
∂f (v̄) = v ∗ ∈ V ∗ : hv ∗ , v – v̄iV ∗ ,V 6 f (v) – f (v̄)
for all v ∈ V
Fenchel conjugate (always convex)
f ∗ : V ∗ → R,
f ∗ (v ∗ ) = sup hv ∗ , viV ∗ ,V – f (v)
v∈V
“convex inverse function theorem”:
v ∗ ∈ ∂f (v)
⇔
v ∈ ∂f ∗ (v ∗ )
Overview General approach Multi-bang penalty Numerical examples
7 / 34
Fenchel duality: application
F(ū) + G(ū) = min F(u) + G(u)
u
1
Fermat principle:
2
sum rule:
0 ∈ ∂ (F(ū) + G(ū))
0 ∈ ∂F(ū) + ∂G(ū), i.e., there is p̄ ∈ V ∗ with
–p̄ ∈ ∂F(ū)
p̄ ∈ ∂G(ū)
3
Fenchel duality:
–p̄ ∈ ∂F(ū)
ū ∈ ∂G∗ (p̄)
Overview General approach Multi-bang penalty Numerical examples
8 / 34
Fenchel duality: example
G : V → R,
v 7→ kvkV :
G∗ : V ∗ → R,
G : V → R,
v ∗ 7→ δ{k·kV ∗ 61} (v ∗ ) :=
0 if kv ∗ kV ∗ 6 1
∞ else
v 7→ δ{k·kV 61} (v):
∂G(v̄) = v ∗ ∈ V ∗ : hv ∗ , v – v̄iV ∗ ,V 6 0
for all
kvkV 6 1
norm minimization equivalent to smooth optimization with
constraints
Overview General approach Multi-bang penalty Numerical examples
9 / 34
Convex relaxation
Consider F convex, G non-convex
J(ū) := F(ū) + G(ū) = min F(u) + G(u)
u
Optimality(?) conditions:
–p̄ ∈ ∂F(ū)
ū ∈ ∂G∗ (p̄)
Fenchel conjugate always convex, lower semi-continuous
well-defined, solution ū exists (minimizes F(u) + G∗∗ (u))
but: ū in general not minimizer of J
sub-optimal
Overview General approach Multi-bang penalty Numerical examples
10 / 34
Smoothing
G non-convex
subdifferential ∂G∗ set-valued
local smoothing
assume u, p ∈ L2 Hilbert space: consider for γ > 0
Proximal mapping
proxγG∗ (p) = arg min G∗ (w) +
w
1
kw – pk2
2γ
single-valued, Lipschitz continuous
coincides with resolvent (Id +γ∂G∗ )–1 (p)
Overview General approach Multi-bang penalty Numerical examples
11 / 34
Smoothing
Proximal mapping
proxγG∗ (p) = arg min G∗ (w) +
w
1
kw – pk2
2γ
Complementarity formulation of u ∈ ∂G∗ (p)
u=
1
(p + γu) – proxγG∗ (p + γu)
γ
equivalent for every γ > 0
single-valued, Lipschitz continuous, implicit
Overview General approach Multi-bang penalty Numerical examples
11 / 34
Smoothing
Proximal mapping
proxγG∗ (p) = arg min G∗ (w) +
w
1
kw – pk2
2γ
Moreau–Yosida regularization of u ∈ ∂G∗ (p)
u=
1
p – proxγG∗ (p) =: ∂G∗γ (p)
γ
not equivalent, but ∂G∗γ (p) → ∂G∗ (p) as γ → 0
single-valued, Lipschitz continuous, explicit
nonsmooth operator equation, Newton method
Overview General approach Multi-bang penalty Numerical examples
11 / 34
Smoothing: example
G∗ : V ∗ → R,
p 7→ δ{k·kV ∗ 61} (p):
Proximal mapping:
proxγG∗ (p) = proj{k·kV ∗ 61} (p)
Complementarity formulation: soft-shrinkage
Moreau–Yosida regularization (V ∗ = L∞ (Ω)):
∂G∗γ (p) =
1
max(0, p – 1)) + min(0, p + 1)
γ
(max, min pointwise almost everywhere)
Overview General approach Multi-bang penalty Numerical examples
12 / 34
Generalized Newton method
Consider Banach spaces X, Y, mapping F : X → Y
Newton method for F(x) = 0
choose x 0 ∈ X (close to solution x ∗ )
for k = 0, 1, . . .
1
2
choose Mk ∈ L(X, Y) invertible
solve for sk :
Mk sk = –F(x k )
3
set x k+1 = x k + sk
Overview General approach Multi-bang penalty Numerical examples
13 / 34
Generalized Newton method
Newton method for F(x) = 0
choose x 0 ∈ X (close to solution x ∗ )
for k = 0, 1, . . .
1
2
choose Mk ∈ L(X, Y) invertible
solve for sk :
Mk sk = –F(x k )
3
set x k+1 = x k + sk
convergence, i.e., kx k – x ∗ kX → 0?
superlinear convergence, i.e.,
kx k+1 – x ∗ kX
→ 0?
kx k – x ∗ kX
Overview General approach Multi-bang penalty Numerical examples
13 / 34
Convergence of Newton method
Set dk = x k – x ∗
k
∗
k
∗
kx k+1 – x ∗ kX = kM–1
k (F(x + d ) – F(x ) – Mk d )kX
superlinear convergence if
1
regularity condition
kM–1
k kL(Y,X) 6 C
2
for all k
approximation condition
kF(x ∗ + dk ) – F(x ∗ ) – Mk dk kY
=0
kdk kX
kd k kX →0
lim
Overview General approach Multi-bang penalty Numerical examples
14 / 34
Semismooth Newton method
Goal: define Newton derivative Mk = DN F such that
x k+1 = x k – DN F(x k )–1 F(x k )
converges superlinearly for F(x) = 0 nonsmooth
Rn : F Lipschitz
DN F from Clarke subdifferential
[Mifflin 1977, Kummer 1992, Qi/Sun 1993]
function space: no Clarke subdifferential
define DN F via approximation condition
[Chen/Nashed/Qi 2000, Hintermüller/Ito/Kunisch 2002]
f : RN → R semismooth superposition operator
F : Lp (Ω) → Lq (Ω) semismooth iff p > q
[Ulbrich 2002/03/11, Schiela 2008]
Overview General approach Multi-bang penalty Numerical examples
15 / 34
Semismooth functions: example
f : R → R,
t 7→ max(0, t)


t<0
{0}
DN f (t) ∈ ∂C f (t) = {1}
t>0


[0, 1] t = 0
F : Lp (Ω) → Lq (Ω),
u(x) 7→ max(0, u(x)),
[DN F(u)h](x) =
p>q
0
u(x) < 0
h(x) u(x) > 0
Moreau–Yosida regularization semismooth
Overview General approach Multi-bang penalty Numerical examples
16 / 34
Numerical solution: summary
For nonconvex G :
L2 (Ω)
Z
→ R,
G(u) =
Ω
g(u(x)) dx,
1
compute Fenchel conjugate g∗ (q)
2
compute subdifferential ∂g∗ (q)
3
compute proximal mapping proxγg∗ (q)
4
compute Moreau–Yosida regularization ∂g∗γ (q),
Newton derivative DN ∂g∗γ (q)
5
semismooth Newton method, continuation in γ for
superposition operator ∂G∗γ (p)(x) = ∂g∗γ (p(x))
Overview General approach Multi-bang penalty Numerical examples
17 / 34
1
Overview
2 General approach
Convex relaxation
Moreau–Yosida regularization
Semismooth Newton method
3 Multi-bang penalty
Optimality system
Structure of solution
Numerical solution
4 Numerical examples
Overview General approach Multi-bang penalty Numerical examples
18 / 34
Formulation







1
α
min ky – yδ k2L2 + kuk2L2 + β
2
2
u∈L (Ω) 2
s. t. Ay = u,
u1 < · · · < ud
Z Y
d
Ω i=1
|u(x) – ui |0 dx
u1 6 u(x) 6 ud
given parameter values
(d > 2)
yδ ∈ L2 (Ω) noisy data
A : V → V ∗ isomorphism for Hilbert space V ,→ L2 (Ω) ,→ V ∗
(e.g., elliptic differential operator with boundary conditions)
F(u) = 21 kA–1 u – yδ k2L2 convex, Fréchet-differentiable
Overview General approach Multi-bang penalty Numerical examples
19 / 34
Fenchel conjugate
d
g : R → R,
∗
g : R → R,
Y
α 2
v +β
|v – ui |0 + δ[u1 ,ud ] (v)
v→
7
2
i=1
q 7→ sup q v – g(v)
v
Case differentiation: sup attained at v̄,
qui – α u2 v̄ = ui ,
∗
g (q) = 1 2 2 i
v̄ 6= ui ,
2α q – β
16i6d
16i6d
Overview General approach Multi-bang penalty Numerical examples
20 / 34
Fenchel conjugate
∗
g (q) =
qui – α2 u2i
1 2
2α q – β
q ∈ Qi ,
q ∈ Q0
16i6d
p
α
Q1 := q : q – αu1 < 2αβ ∧ q < (u1 + u2 )
2
p
α
α
Qi := q : |q – αui | < 2αβ ∧ (ui–1 + ui ) < q < (ui + ui+1 )
2
2
p
α
Qd := q : q – αud > 2αβ ∧ (ud + ud–1 ) < q
2
p
Q0 := q : |q – αuj | > 2αβ for all j ∧ αu1 < q < αud
Overview General approach Multi-bang penalty Numerical examples
21 / 34
Fenchel conjugate
∗
g (q) =
qui – α2 u2i
1 2
2α q – β
q ∈ Qi ,
q ∈ Q0
16i6d
continuous, piecewise differentiable:


{ui }



 1 q
α
∂g∗ (q) =

[u
i , ui+1 ]



min u , 1 q , max u , 1 q
i α
i α
q ∈ Qi , 1 6 i < d
q ∈ Q0
q ∈ Qi ∩ Qi+1 , 1 6 i < d
q ∈ Qi ∩ Q0 , 1 6 i 6 d
(no explicit dependence on β!)
Overview General approach Multi-bang penalty Numerical examples
21 / 34
Optimality system
p̄ = S∗ (yδ – Sū)
ū ∈ ∂G∗ (p̄)


{ui }



 1 p̄(x)
α
=

[u
i , ui+1 ]



min u , 1 p̄(x) , max u , 1 p̄(x)
i α
i α
p̄(x) ∈ Qi
p̄(x) ∈ Q0
p̄(x) ∈ Qi ∩ Qi+1
p̄(x) ∈ Qi ∩ Q0
S : u 7→ y parameter-to-observation mapping,
S∗ adjoint
necessary conditions for minu F(u) + G∗∗ (u) (convex, l.s.c.)
unique solution (ū, p̄) ∈ L2 (Ω) × L2 (Ω)
Overview General approach Multi-bang penalty Numerical examples
22 / 34
Structure of solution
Ω=A∪F∪S
multi-bang arc A =
d
[
{x : ū(x) = ui }
i=1
free arc
singular arc
F = x : ū(x) = α1 p̄(x) 6= ui
S = x : ū(x) ∈/ ui , α1 p̄(x)
Overview General approach Multi-bang penalty Numerical examples
23 / 34
Generalized multi-bang principle
if β sufficiently large: Q0 = ∅, free arc
F ⊂ {p̄(x) ∈ Q0 } = ∅
singular arc corresponds to set-valued subdifferential:
d–1
d
[
[
S = p̄(x) ∈
(Qi ∩ Qi+1 ) ∪
(Qi ∩ Q0 )
⊂ p̄(x) ∈
i=1
α
i=1
2 (ui + ui+1 ), αui –
p
p
2αβ, αui + 2αβ
for suitable A, p̄(x) constant implies [A∗ p̄](x) = [yδ – ȳ](x) = 0
|{x : ȳ(x) = yδ (x)}| = 0 ⇒ ū ∈ {u1 , . . . ud } a. e., true multi-bang
Overview General approach Multi-bang penalty Numerical examples
24 / 34
(Sub)optimality
duality gap for non-convex G:
G(ū) + G∗ (p̄) – hp̄, ūi 6 β|S|
(pointwise gap of β where ∂g∗ (p̄(x)) set-valued)
in general: ū sub-optimal:
J(ū) 6 J(u) + β|S|
but: ū true multi-bang
|S| = 0
for all u
ū optimal
Overview General approach Multi-bang penalty Numerical examples
25 / 34
Moreau–Yosida regularization

ui



 1

α+γ q
p
∂g∗γ (q) = 1 
q
–
(αu
+
2αβ)
i

γ


1
α
γ q – 2 (ui + ui+1 )
q ∈ Qγi
q ∈ Qγ0
q ∈ Qγi0
q ∈ Qγi,i+1
p
Qγi = q : |q – (α + γ)ui | < 2αβ ∧
2γ
2γ
α
α
u
+
1
+
u
<
q
<
1
+
u
+
u
i–1
i
i
i+1
2
α
2
α
p
Qγ0 = q : |q – (α + γ)uj | > 2αβ ∧ (α + γ)u1 < q < (α + γ)ud
2γ
α
Qγi,i+1 = q : α2
1 + 2γ
u
+
u
6
q
6
u
+
1
+
u
i
i+1
i
i+1
α
2
α
p
p
γ
Qi0 = q : 2αβ 6 q – (α + γ)ui 6 1 + αγ
2αβ
Overview General approach Multi-bang penalty Numerical examples
26 / 34
Regularized optimality system
pγ = S∗ (yδ – Suγ )
uγ = ∂G∗γ (pγ )
∂G∗γ maximal monotone
unique solution (uγ , pγ )
(uγ , pγ ) * (ū, p̄) as γ → 0
∂g∗γ Lipschitz continuous, piecewise C 1 , norm gap V ,→ L2 (Ω)
semismooth Newton method
Overview General approach Multi-bang penalty Numerical examples
27 / 34
Regularized optimality system
A∗ pγ = yδ – yγ
Ayγ = G∗γ (pγ )
∂G∗γ maximal monotone
unique solution (uγ , pγ )
(uγ , pγ ) * (ū, p̄) as γ → 0
∂g∗γ Lipschitz continuous, piecewise C 1 , norm gap V ,→ L2 (Ω)
semismooth Newton method
introduce yγ = Suγ , eliminate uγ = G∗γ (pγ )
Overview General approach Multi-bang penalty Numerical examples
27 / 34
Semismooth Newton method
∗
Id
A∗
A p + y – yδ
δp
=–
A DN G∗γ (p)
Ay – G∗γ (p)
δy
 1

 α+γ δp(x)
∗
[DN Gγ (p)δp](x) = γ1 δp(x)


0
p(x) ∈ Qγ0
p(x) ∈ Qγi,i+1
else
symmetric, but: local convergence
continuation in γ → 0
backtracking line search based on residual norm
only number of sets Qi depends on d
linear complexity
Overview General approach Multi-bang penalty Numerical examples
28 / 34
Example: linear inverse problem
Ω = [0, 1]2 , A = –∆
u† (x) = u1 + u2 χ{x:(x1 –0.45)2 +(x2 –0.55)2 <0.1} (x)
+ (u3 – u2 )χ{x:(x1 –0.4)2 +(x2 –0.6)2 <0.02} (x)
d = 3, u1 = 0, u2 = 0.1, u3 ∈ {0.2, 0.12}
yδ = y† + ξ,
ξ ∈ N y† , δky† k∞
finite element discretization: uniform grid, 256 × 256 nodes
α = 5 · 10–5 , β = 10–1 (no free arc)
terminate at γ < 10–12
Overview General approach Multi-bang penalty Numerical examples
29 / 34
Numerical example: u3 = 0.2
Figure: exact parameter u†
Overview General approach Multi-bang penalty Numerical examples
30 / 34
Numerical example: u3 = 0.2
Figure: reconstruction uδ ,
δ = 0.1
Overview General approach Multi-bang penalty Numerical examples
30 / 34
Numerical example: u3 = 0.2
Figure: reconstruction uδ ,
δ = 0.5
Overview General approach Multi-bang penalty Numerical examples
30 / 34
Numerical example: u3 = 0.12
Figure: exact parameter u†
Overview General approach Multi-bang penalty Numerical examples
31 / 34
Numerical example: u3 = 0.12
Figure: reconstruction uδ ,
δ = 0.1
Overview General approach Multi-bang penalty Numerical examples
31 / 34
Numerical example: u3 = 0.12
Figure: reconstruction uδ ,
δ = 0.5
Overview General approach Multi-bang penalty Numerical examples
31 / 34
Example: nonlinear inverse problem
S : u 7→ y solving
–∆y + uy = f
F nonconvex, but (modified) approach applicable
numerical example:
Ω = [0, 1]2 , f ≡ 1
u† (x) = u1 + u2 χ{x:(x1 –0.45)2 +(x2 –0.55)2 <0.1} (x)
+ (u3 – u2 )χ{x:(x1 –0.4)2 +(x2 –0.6)2 <0.02} (x)
yδ = S(u† ) + ξ
α = 3 · 10–5 ,
β = ∞ (formal),
γ → 10–12
Overview General approach Multi-bang penalty Numerical examples
32 / 34
Numerical example: nonlinear problem
Figure: noisy data y δ
Overview General approach Multi-bang penalty Numerical examples
33 / 34
Numerical example: nonlinear problem
Figure: exact parameter u†
Overview General approach Multi-bang penalty Numerical examples
33 / 34
Numerical example: nonlinear problem
Figure: reconstruction uδ
Overview General approach Multi-bang penalty Numerical examples
33 / 34
Conclusion
Convex relaxation of discrete regularization:
well-posed primal-dual optimality system
solution optimal under general assumptions
linear complexity in number of parameter values
efficient numerical solution (superlinear convergence)
Outlook:
regularization properties, parameter choice
nonlinear inverse problems: EIT
combination with BV regularization
other hybrid discrete–continuous problems
Preprint, MATLAB/Python codes:
http://www.uni-due.de/mathematik/agclason/clason_pub.php
Overview General approach Multi-bang penalty Numerical examples
34 / 34