Total Variation
Numerical Schemes
Wrap up approximate formulations of subgradient relation
Martin Burger
Cetraro, September 2008
1
Total Variation
Numerical Schemes
Primal Approximation
Primal Fixed Point
Dual Approximation
Dual Fixed Point
Dual Fixed Point for Primal Relation
Martin Burger
Cetraro, September 2008
2
Total Variation
1. Fixed point methods
Matrix form
Martin Burger
Cetraro, September 2008
3
Total Variation
Fixed Point Schemes I
Primal Gradient Method Based on approximation of F:
Fixed-point approach for first optimality equation
Martin Burger
Cetraro, September 2008
4
Total Variation
5
Fixed Point Schemes I
Primal Gradient Method Based on Approximation, Rudin-OsherFatemi 89
+ easy to implement, efficient iteration steps
+ global convergence (descent method for variational problem)
- dependent on approximation
- slow convergence
- severe step size restrictions (explicit approximation of
differential operator)
Martin Burger
Cetraro, September 2008
Total Variation
6
Fixed Point Schemes I
Primal Gradient Method Based on Approximation, Rudin-OsherFatemi 89
Special case of fixed point methods with choice
Martin Burger
Cetraro, September 2008
Total Variation
7
Fixed Point Schemes I
Primal Gradient Method Based on Approximation, Rudin-OsherFatemi 89
+ easy to implement, efficient iteration steps
+ global convergence (descent method for variational problem)
- dependent on approximation
- slow convergence
- severe step size restrictions (explicit approximation of
differential operator)
Martin Burger
Cetraro, September 2008
Total Variation
Fixed Point Schemes II
Dual Gradient Projection Method
Dual methods eliminate also u
Martin Burger
Cetraro, September 2008
8
Total Variation
9
Fixed Point Schemes II
Dual Gradient Projection Method
Do gradient step on the quadrativc functional and project back
to constraint set M
Note: can be interpreted as a scheme where the first equation
is always satisfied, i.e. special case with
Martin Burger
Cetraro, September 2008
Total Variation
10
Fixed Point Schemes II
Dual Gradient Projection Method, Chambolle 05, Chan et al 08, Aujol
08
+ easy to implement, efficient iteration steps
+ global convergence (descent method for dual problem)
+ no approximation necessary
- slow convergence
- needs inversion of A*A, hence good for ROF, bad for inverse
problems
Obvious generalization for last point: use preconditioning of
A*A
Martin Burger
Cetraro, September 2008
Total Variation
Fixed Point Schemes II
Chambolle‘s Method
Dual method, again eliminates u
Martin Burger
Cetraro, September 2008
11
Total Variation
12
Fixed Point Schemes III
Chambolle‘s Method
Complicated derivation from dual minimization problem in
original paper
Note: can be interpreted as a scheme where the first equation
is always satisfied, in addition using dual fixed point form for
the primal subgradient relation, i.e. special case with
Martin Burger
Cetraro, September 2008
Total Variation
13
Fixed Point Schemes III
Chambolle‘s Method, Chambolle 04
+ easy to implement, efficient iteration steps
+ global convergence (descent method for dual problem)
+ no approximation necessary
- slow convergence
- needs inversion of A*A, hence good for ROF, bad for inverse
problems
Obvious generalization for last point: use preconditioning of
A*A
Martin Burger
Cetraro, September 2008
Total Variation
14
Fixed Point Schemes IV
Inexact Uzawa method, Zhu and Chan 08
Primal gradient descent, dual projected gradient ascent in the
reduced Lagrangian (for u and w)
Coincides with dual gradient projection if A = I and appropriate
choice of damping parameter q
Martin Burger
Cetraro, September 2008
Total Variation
Fixed Point Schemes V
Primal Lagged Diffusivity with Approximation, Vogel et al 95-97
Approximate smoothed primal optimality condition
Semi-implicit treatment of differential operator
Martin Burger
Cetraro, September 2008
15
Total Variation
Fixed Point Schemes V
Special case with choice
Martin Burger
Cetraro, September 2008
16
Total Variation
17
Fixed Point Schemes V
Primal Lagged Diffusivity with Approximation
+ acceptable step-size restrictions
+ global convergence (descent method for variational problem)
- dependent on approximation
- still slow convergence
- differential equation with changing parameter to be solved in
each step
Martin Burger
Cetraro, September 2008
Total Variation
2. Thesholding methods
C is damping matrix, possible perturbation
T is thresholding operator
Martin Burger
Cetraro, September 2008
18
Total Variation
Thresholding Methods I
Primal Thresholding Method, Daubechies-Defrise-DeMol 03
Only used for D = -I
Introduce
Martin Burger
Cetraro, September 2008
19
Total Variation
Thresholding Methods I
Primal Thresholding Method, Daubechies-Defrise-DeMol 03
Hence, special case with
Martin Burger
Cetraro, September 2008
20
Total Variation
Thresholding Scheme I
Primal Thresholding Method
+ easy to implement, efficient iteration steps if D= - I
- slow convergence
- cannot be generalized to cases where D is not invertible
Martin Burger
Cetraro, September 2008
21
Total Variation
Thresholding Methods II
Alternating Minimization, Yin et al 08, Amat-Pedregal 08
Use quadratic penalty for the gradient constraing (MoreauYosida regularization)
Alternate minimization with respect to the variables
Martin Burger
Cetraro, September 2008
22
Total Variation
Thresholding Methods II
Alternating Minimization, Yin et al 08, Amat-Pedregal 08
Use quadratic penalty for the gradient constraing (MoreauYosida regularization)
Alternate minimization with respect to the variables
Martin Burger
Cetraro, September 2008
23
Total Variation
24
Thresholding Methods II
Alternating Minimization, Yin et al 08, Amat-Pedregal 08
Introduce
Hence, special case with
Martin Burger
Cetraro, September 2008
Total Variation
25
Thresholding Scheme II
Primal Thresholding Method
+ efficient iteration steps if D*D and A*A can be jointly inverted
easily (e.g. by FFT)
+ treats differential operator implicitely, no severe stability
bounds
-Linear convergence
- Smoothes the regularization functional
Martin Burger
Cetraro, September 2008
Total Variation
26
Thresholding Methods III
Split Bregman, Goldstein-Osher 08
Original motivation from Bregman iteration, can be rewritten as
Martin Burger
Cetraro, September 2008
Total Variation
27
Thresholding Methods III
Split Bregman, Goldstein-Osher 08
Difficult to solve directly, hence subiteration with thresholding
After renumbering
Martin Burger
Cetraro, September 2008
Total Variation
28
Thresholding Scheme III
Split Bregman, Goldstein-Osher 08
+ efficient iteration steps if D*D and A*A can be jointly inverted
easily (e.g. by FFT)
+ treats differential operator implicitely, no severe stability
bounds
+ does not need smoothing
- Linear convergence
Martin Burger
Cetraro, September 2008
Total Variation
2. Newton type methods
Matrix form
Martin Burger
Cetraro, September 2008
29
Total Variation
30
Newton-type Methods
Primal or dual
+ fast local convergence
- global convergence difficult
- dependent on approximation (Newton-matrix degenerates)
- needs inversion of large Newton matrix
Good choice with efficient preconditioning for linear system in
each iteration step
Martin Burger
Cetraro, September 2008