Continuous and Discrete
Optimization Methods
in Computer Vision
Daniel Cremers
Department of Computer Science
University of Bonn
Oxford, August 16 2007
Segmentation by Energy Minimization
Given an image
, minimize the energy
boundary length
Blake, Zisserman ’87, Mumford, Shah ’89,
Ising ’27, Potts ’59, Geman, Geman ‘84
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
2
Continuous Versus Discrete Optimization
Min. cut
Max. flow
Ford, Fulkerson ’59
Greig et al. ’89
Osher, Sethian, J. of Comp. Phys. ’88
Daniel Cremers
Boykov, Kolmogorov ’02
Continuous and Discrete Optimization in Computer Vision
3
Level Set Formulation of Mumford-Shah
Chan & Vese ’99
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
4
Level Set Segmentation in Feature Space
efficient coarse-to-fine scheme
Brox, Weickert ’04, ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
5
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
Convex 3D reconstruction
Markerless human motion capture
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
6
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Cremers CVPR ’03, Cremers & Soatto IJCV ’05
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
7
Motion
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
8
Motion-based Image Segmentation
1. assumption: Intensity of moving points is preserved.
Problem underconstrained (aperture problem).
2. assumption: Velocity in each region is a
Gaussian-distributed random variable
Segmentation
&
Motion
Minimize
Cremers CVPR ’03, Cremers & Soatto IJCV ’05
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
9
Motion Competition
Motion Estimation
Daniel Cremers
Segmentation
Continuous and Discrete Optimization in Computer Vision
10
Motion Competition via Level Sets
Cremers, Yuille, ’04
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
11
Motion Competition via Level Sets
What is moving?
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
12
Motion Competition via Level Sets
Cremers, Yuille, ’04
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
13
Motion Competition via Level Sets
Cremers, Soatto, IJCV ’05: Piecewise Parametric Motion
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
14
Motion Competition via Graph Cuts
piecewise constant
piecewise affine
Schoenemann & Cremers, DAGM ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
up to 30 fps
15
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Wedel, Schoenemann, Brox, Cremers DAGM ’07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
16
WarpCut: Obstacle Segmentation
Segmentation of obstacles in monocular video
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
17
WarpCut: Obstacle Segmentation
Local scale changes depend on distance from the camera.
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
18
WarpCut: Obstacle Segmentation
Competing hypotheses: Obstacle or road?
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
19
WarpCut: Obstacle Segmentation
Wedel, Schoenemann, Brox, Cremers DAGM ’07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
20
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
21
Statistical Priors for Image Segmentation
initial contour
no prior
with prior
with prior
with prior
scale invariance
Cremers, Kohlberger, Schnörr, ECCV 2002
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
22
Statistical Learning of Familiar Shapes
Cremers, Osher, Soatto, IJCV ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
Rosenblatt ’56,
Parzen ’62
23
Statistical Learning of Implicit Shapes
Without statistical prior
With statistical prior
Cremers, Osher, Soatto, IJCV ’06
Sequence data courtesy of Alessandro Bissacco.
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
24
Statistical Priors for Image Segmentation
Purely geometric prior
Nonparametric prior
Cremers, Osher, Soatto, IJCV ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
25
Dynamical Models for Implicit Shapes
Training sequence
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
26
Dynamical Models for Implicit Shapes
1. Dimensionality reduction by PCA (Leventon et al. ’00, Tsai et al. ’01):
mean
eigen modes
2. Autoregressive model for the shape vectors:
3. Synthesized sequence of shape vectors and embedding functions:
mean
transition matrices
Gaussian noise
Cremers, IEEE Trans. on PAMI ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
27
Statistically Sampled Embedding Functions
Cremers, IEEE Trans. on PAMI ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
28
Dynamical Models for Implicit Shapes
1. Low-dim. representation via PCA (Leventon et al. ’00, Tsai et al. ’01):
2. Autoregressive model for the shape coefficients:
3. Synthesize shape vectors and embedding surfaces:
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
29
Dynamical Models for Implicit Shapes
1. Low-dim. representation via PCA (Leventon et al. ’00, Tsai et al. ’01):
2. Autoregressive model for the shape coefficients:
3. Synthesize shape vectors and embedding surfaces:
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
30
Dynamical Priors for Level Set Segmentation
Bayesian Aposteriori Maximization:
Image formation model
(color, texture, motion,…)
Dynamical shape model
Optimization by gradient descent:
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
31
Dynamical Priors for Level Set Segmentation
Input sequence with 90% uniform noise
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
32
Dynamical Priors for Level Set Segmentation
Cremers, IEEE Trans. on PAMI ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
33
Dynamical Priors for Level Set Segmentation
Pure deformation model
Cremers, IEEE Trans. on PAMI ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
34
Dynamical Priors for Level Set Segmentation
Model of joint deformation and transformation
Cremers, IEEE Trans. on PAMI ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
35
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Schoenemann & Cremers ICCV ‘07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
36
Limitations of Previous Approaches
• Local optimization
• Simple geometric distances
• Little generalization
• Requires large training sets
• Point
correspondence,
• Missing parts,
• Stretching/shrinking,...
• Combinatorial problem
Goal: Combinatorial solution for joint segmentation and alignment
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
37
The Product Space of Image and Template
Each node of the graph represents an image pixel and a point on the template.
All template points S are allowed to stretch over at most K image pixels.
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
38
The Product Space of Image and Template
Each node of the graph represents an image pixel and a point on the template.
All template points S are allowed to stretch over at most K image pixels.
A cycle in this graph defines an assignment of template points to image pixels.
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
39
Segmentation & Alignment by Energy Min.
Data term
Alignment with template
Favors strong gradients.
Daniel Cremers
Stretching cost
Favors minimal stretching / shrinking.
Continuous and Discrete Optimization in Computer Vision
40
Segmentation with a Single Template
Template
Segmentation results
Optimization by Minimum Ratio Cycles on GPU: ~15 sec / frame
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
41
Rotational Invariance
No rotational invariance
Daniel Cremers
Rotational invariance
Continuous and Discrete Optimization in Computer Vision
42
Tracking
Input sequence
Daniel Cremers
Globally optimal segmentations
Continuous and Discrete Optimization in Computer Vision
43
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Kolev, Klodt, Brox, Esedoglu & Cremers EMMCVPR ‘07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
44
3D Reconstruction
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
45
Probabilistic Formulation
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
46
Probabilistic Formulation
Kolev, Brox, Cremers DAGM ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
47
Continuous Convex Formulation
Introduce binary variable:
weighted TV norm
Relax binary constraint:
Convex functional optimized over a convex set.
Solution of binary-valued problem by thresholding.
Global minima by gradient descent (or SOR).
Kolev, Klodt, Brox, Esedoglu & Cremers EMMCVPR ‘07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
48
Multiview 3D Reconstruction
Kolev, Klodt, Brox, Esedoglu & Cremers EMMCVPR ‘07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
49
Comparison
Level sets
Convex formulation
Graph cuts
Continuous
Continuous
Discrete
Local optima
Global optima*
Global optima*
Memory limitations
Metrication errors
* for certain cost functionals only
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
50
Metrication Errors in Discrete Energies
Synthetic data:
blank out data term
in upper octant
Discrete graph cut optimization
Continuous convex optimization
Kolev, Klodt, Brox, Esedoglu & Cremers EMMCVPR ‘07
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
51
Overview
Motion segmentation
Obstacle segmentation
Prior shape knowledge
Shape priors in global optimization
3D reconstruction: a convex formulation
Markerless human motion capture
Brox et al. DAGM 2005
Brox, Rosenhahn, Cremers & Seidel, ECCV ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
52
Human Motion Tracking
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
53
Human Motion Tracking
Brox et al. DAGM 2005
Brox, Rosenhahn, Cremers & Seidel, ECCV ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
54
Coupled Segmentation and 3D Tracking
Brox et al. DAGM 2005
Brox, Rosenhahn, Cremers & Seidel, ECCV ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
55
3D Tracking with Statistical Priors
T. Brox, B. Rosenhahn, U. Kersting, D. Cremers, DAGM ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
56
Human Motion Tracking
T. Brox, B. Rosenhahn, U. Kersting, D. Cremers, DAGM ’06
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
57
Summary
Motion competition
Obstacle segmentation Dynamical shape priors
3D reconstruction
Human motion tracking
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
58
Announcement: IPAM Workshop
Boykov, Cremers, Darbon, Ishikawa, Kolmogorov, Osher:
IPAM Workshop on “Graph Cuts and Related Discrete or
Continuous Optimization Problems”, Feb. 25-29 2008
https://www.ipam.ucla.edu/programs/gc2008/
Daniel Cremers
Continuous and Discrete Optimization in Computer Vision
59