Representing Object-level Knowledge
for Segmentation and Image Parsing:
Epitome Priors and Bayesian
supervised clustering approaches
Jonathan Warrell
[email protected]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
1
Talk Outline
• Talk Outline
– A Typology of Segmentation and Parsing Problems
– Supervised Parsing using Epitome Priors
• Scene Parsing
• Face Parsing
– Object-level knowledge transfer for Unsupervised
Parsing
• Segmentation, Unsupervised Parsing and object-level knowledge
• Bayesian Supervised Clustering
– Summary
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
2
A Typology of Segmentation and Parsing
Problems: Over-segmentation
• Over-segmentation
a
Normalized Cut (NC)
Ren and Malik [ICCV 2003]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
Minimum Spanning Tree (FH)
Felzenszwalb & Huttenlocher
[IJCV 2004]
3
A Typology of Segmentation and Parsing
Problems: Segmentation as Grouping
• Segmentation as Perceptual Grouping
a
Sample images with 3 human segmentations
Martin et al [ICCV 2001]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
4
A Typology of Segmentation and Parsing
Problems: Segmentation as Grouping
•
Sampling from a Posterior over Segmentations
a
Original Image, DDMCMC result, human
segmentation
Tu and Zhu [PAMI 2002]
Oxford Brookes Seminar
Thursday 3rd September, 2009
Original Image, Result
Ren and Malik [ICCV 2003]
University College London
5
A Typology of Segmentation and Parsing
Problems: Segmentation as Grouping
•
Creating a Hierarchy of Regions
a
Original Image, UMC map, Segmentation at different thresholds
Arabelaez et al [CVPR 2009]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
6
A Typology of Segmentation and Parsing
Problems: Single Object Segmentation
• Supervised Object Segmentation
a
Bounding Box + Segmentation results on ETHZ shape database
Gu et al [CVPR 2009]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
7
A Typology of Segmentation and Parsing
Problems: Single Object Segmentation
• Supervised Scene Parsing
TextonBoost (Shotton et al [IJCV 2009])
a
Multiscale CRF (He and Zemel [CVPR 2004])
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
8
A Typology of Segmentation and Parsing
Problems: Object Discovery
• Co-Segmentation / Object Discovery
a
Russell et al [CVPR 2006]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
9
A Typology of Segmentation and Parsing
Problems: General Unsupervised Parsing
• General Unsupervised Parsing
a
Li et al [CVPR 2009]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
10
A Typology of Segmentation and Parsing
Problems: Summary
• Summary
a
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
11
A Typology of Segmentation and Parsing
Problems: Summary
• Summary
a
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
12
A Typology of Segmentation and Parsing
Problems: Summary
• Summary
a
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
13
Talk Outline
• Talk Outline
– A Typology of Segmentation and Parsing Problems
– Supervised Parsing using Epitome Priors
• Scene Parsing
• Face Parsing
– Object-level knowledge transfer for Unsupervised
Parsing
• Segmentation, Unsupervised Parsing and object-level knowledge
• Bayesian Supervised Clustering
– Summary
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
14
Supervised Parsing Using Epitome Priors:
A Simple Scene Parsing Pipeline
• A Simple Scene Parsing Pipeline
Local Classifier
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
Integration of prior/
contextual knowledge
15
Supervised Parsing Using Epitome Priors:
A Simple Scene Parsing Pipeline
• A Simple Scene Parsing Pipeline
Local Classifier
Integration of prior/
contextual knowledge
• Includes:
• Necessary for:
• Likely configurations of objects
• Disambiguation
• General properties of the label field e.g.
• Correction of classifier errors
stationarity/non-stationarity, symmetry
• Object shape
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
16
Supervised Parsing Using Epitome Priors:
Local MRF/CRF approaches
• Local MRF/CRF approaches
Geman and Geman, PAMI, 1986
Kumar and Herbert, ICCV, 2003
Shotton et al, ECCV, 2006
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
17
Supervised Parsing Using Epitome Priors:
Local MRF/CRF approaches
• Local MRF/CRF approaches
+ Few Parameters
+ Efficient Inference (via
alpha expansion)
Geman and Geman, PAMI, 1986
Kumar and Herbert, ICCV, 2003
Shotton et al, ECCV, 2006
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
18
Supervised Parsing Using Epitome Priors:
Local MRF/CRF approaches
• Local MRF/CRF approaches
+ Few Parameters
+ Efficient Inference (via
alpha expansion)
Geman and Geman, PAMI, 1986
Kumar and Herbert, ICCV, 2003
Shotton et al, ECCV, 2006
Oxford Brookes Seminar
Thursday 3rd September, 2009
– Limited representation
ability (e.g. more than
2 objects, object
shape)
University College London
19
Supervised Parsing Using Epitome Priors:
Larger Clique Size CRF approaches
• Larger Clique Size CRF approaches
He et al, CVPR, 2004
Kohli et al, CVPR, 2007
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
20
Supervised Parsing Using Epitome Priors:
Larger Clique Size CRF approaches
• Larger Clique Size CRF approaches
+ Rich representation
He et al, CVPR, 2004
Kohli et al, CVPR, 2007
Oxford Brookes Seminar
Thursday 3rd September, 2009
– Sampling required
during training and
inference
University College London
21
Supervised Parsing Using Epitome Priors:
Larger Clique Size CRF approaches
• Larger Clique Size CRF approaches
+ Efficient algorithms for
inference (alpha
expansion, s-t cut)
He et al, CVPR, 2004
Kohli et al, CVPR, 2007
Oxford Brookes Seminar
Thursday 3rd September, 2009
– Constrained
representation ability
University College London
22
Supervised Parsing Using Epitome Priors:
Directed Models
• Directed Models
Domke et al, CVPR, 2008
Feng et al, PAMI, 2002
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
23
Supervised Parsing Using Epitome Priors:
Directed Models
• Directed Models
+ Efficient algorithms for
training and inference
+/– Mild impositions on
representation
Domke et al, CVPR, 2008
Feng et al, PAMI, 2002
Oxford Brookes Seminar
Thursday 3rd September, 2009
– Large increase in
parameters needed for
a non-stationary
distribution
University College London
24
Supervised Parsing Using Epitome Priors:
Summary of problems
• Summary of problems
– Trade-off between 1) representational ability and 2)
efficiency of training and inference algorithms
– Desirability of modeling non-stationary distributions
with limited parameters
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
25
Supervised Parsing Using Epitome Priors:
Summary of problems
• Summary of problems
– Trade-off between 1) representational ability and 2)
efficiency of training and inference algorithms
– Desirability of modeling non-stationary distributions
with limited parameters
• Advantages of Epitomes
– Parameterization is compact
– Efficient training and inference
– Non-stationary distributions easily modeled
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
26
Epitomic Analysis of Appearance and Shape
Jojic, Frey and Kannan, 2003
• Epitome as a model of Image Patches
Epitome Parameters:{α, μ, σ}
Jojic, Frey and Kannan,
Epitomic Analysis of Appearance and Shape
[ICCV 2003]
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
27
Epitomes over Label Patches
• (Discrete) Epitomes over Label Patches
Epitome Parameters:{α, θ}
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
28
Generating from a Mixture of Multinomials
θ:
α:
h
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
29
Generating from a Mixture of Multinomials
θ:
α:
h=2
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
30
Generating from a Mixture of Multinomials
θ:
α:
h=2
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
31
Generating from a Mixture of Multinomials
θ:
α:
h=1
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
32
Generating from a Mixture of Multinomials
θ:
α:
h=1
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
33
Generating from a Mixture of Multinomials
θ:
α:
h=1
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
34
Generating from a Mixture of Multinomials
θ:
α:
h=1
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
35
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
36
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=3
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
37
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=3
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
38
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=4
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
39
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=4
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
40
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=6
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
41
Generating from an
Epitomized Mixture of Multinomials
θ:
α:
h=6
…
l:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
42
Epitomic Analysis of Appearance and Shape
Jojic, Frey and Kannan, 2003
• Epitome as model of a whole image
{α, μ, σ}
z:
…
x:
…
See Jojic, Frey and Kannan, Epitomic Analysis of Appearance
and Shape [ICCV 2003] for full model.
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
43
The Epitome Tree (an epitomized TSBN)
• The Epitome Tree (an epitomized Tree Structured Belief
Network)
α
h1
θ1
h2,1
h2,2
θ2
h3,1
l1
l2
h3,2
h3,3
h3,4
θ3
…
For Tree Structured Belief Networks, see Feng, Williams and Felderhof. Combining
Belief Networks and Neural Networks for Scene Segmentation. PAMI, 2002
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
44
CRF Model with Epitome Potentials for
Inference
• Models trained generatively, using E-M
• CRF model used for inference, with Epitome potentials
Epitome Tree
Epitomized Mixture
of Multinomials
Alternative training: Contrastive Divergence (see Hinton, Training Products of
Experts by Contrastive Divergence, Neural Comp. 2002.)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
45
Epitomized Priors: Results on Corel
• Epitomized Priors: Results on Corel (CVPR 2009)
Image
Oxford Brookes Seminar
Thursday 3rd September, 2009
Ground Truth
TextonBoost TextonBoost +
Unary
epitome tree
University College London
46
Epitomized Priors: Results continued
• Results: comparing models on Corel and Sowerby datasets
• Comparison with TextonBoost on Corel
[1] Shotton, Winn, Rother and Criminisi. TextonBoost: Joint Appearance, Shape and
Context Modeling for Multi-class Object Recognition and Segmentation. ECCV, 2006
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
47
Supervised Parsing Using Epitome Priors:
Face Parsing
• Face Parsing
Labeled Faces in the Wild (Huang et al, 2007)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
48
Supervised Parsing Using Epitome Priors:
Face Parsing Models
Mixture of Multinomials
Epitome Model with Spatial Weighting
Epitome Model
Oxford Brookes Seminar
Thursday 3rd September, 2009
Epitome Tree (samples from prior)
University College London
49
Supervised Parsing Using Epitome Priors:
Face Parsing Results
• Face Parsing Results (ICIP 2009)
Comparing: A) original image, B) unary classifier, C) weighted epitome,
D) epitome tree, E) ground truth
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
50
Supervised Parsing Using Epitome Priors:
Face Parsing Results
• Face Parsing Results
Overall pixels correct:
F-measure on individual classes:
[1] Shotton, Winn, Rother and Criminisi. TextonBoost: Joint Appearance, Shape and
Context Modeling for Multi-class Object Recognition and Segmentation. ECCV, 2006
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
51
Talk Outline
• Talk Outline
– A Typology of Segmentation and Parsing Problems
– Supervised Parsing using Epitome Priors
• Scene Parsing
• Face Parsing
– Object-level knowledge transfer for Unsupervised
Parsing
• Segmentation, Unsupervised Parsing and object-level knowledge
• Bayesian Supervised Clustering
– Summary
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
52
Object-level knowledge transfer for
Unsupervised Parsing
• Object-level knowledge for Segmentation?
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
53
Object-level knowledge transfer for
Unsupervised Parsing
• Object-level knowledge for Segmentation?
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
54
Object-level knowledge transfer for
Unsupervised Parsing
• Object-level knowledge for Co-segmentation/Object Discovery?
?
?
?
(Indoors)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
55
Object-level knowledge transfer for
Unsupervised Parsing
• Object-level knowledge for Co-segmentation/Object Discovery?
?
?
?
(Jungle)
(Indoors)
(Country)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
56
Object-level knowledge transfer for
Unsupervised Parsing: Problem Summary
• Problem Summary
– Can we use abstract object knowledge to help with
segmentation and unsupervised parsing tasks, where
the test objects may be different to those seen in
training?
– Humans seem to be capable of this!
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
57
Object-level knowledge transfer for
Unsupervised Parsing: Proposal
• Proposal
– Treat segmentation as a supervised clustering problem
Ren and Malik [ICCV 2003]
Oxford Brookes Seminar
Thursday 3rd September, 2009
‘Face Clustering’, using PLDA model
see Prince et al [ICCV 2007]
Bayesian Hierarchical Clustering, Heller and
Ghahramani [ICML 2005]
University College London
58
Object-level knowledge transfer for
Unsupervised Parsing: Bayesian Clustering
• Bayesian Supervised Clustering
x1:
x11 x12 x31
y1:
[1
1
2
…
…
x2:
3 2 … ]
y2: [1
2
1
3 4 … ]
…
xT:
Model
Learn: Pr(x|y), Pr(y)
yT:
[
?
…
]
For a test point, seek:
argmax Pr(y)Pr(x|y)
y
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
59
Summary
• Summary
– Segmentation vs. Parsing: multiple ways of combining
top-down and bottom-up information in a family of
related tasks
– Supervised Scene and Face parsing using Epitome
Priors
– Object-level knowledge transfer for Unsupervised
Parsing (Bayesian supervised Clustering)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
60
Any Questions?
• Any Questions?
• Details of the papers can be found at
http://web4.cs.ucl.ac.uk/research/vis/pvl
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
61
Object-level knowledge transfer for
Unsupervised Parsing: Mixture of Gaussians
Example
• Mixture of Gaussians Example
– Introduce a hidden variable, hn, associated with each
observation, taking values 1…C (N<C)
– Let: P(xn|hn = c) = G(x|μc,Σc)
– Generative process now becomes:
• Sample y: [1 1 2 3 3 2 2 1 3 … ] (from Pr(y)) (max value, K)
• Sample h: [4 4 5 2 2 5 5 4 2 … ] (from Pr(h))
• Sample x: [x1 x2 x3 x4 x5 x6 x7 x8 x9 ... ] (from Pr(x|h))
– Likelihood model is:
(function ind(i,j) returns the vector index of the jth entry of y with value i)
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
62
Object-level knowledge transfer for
Unsupervised Parsing: Factor Analysis and
PLDA examples
• Factor Analysis Extension
– Same as Mixture of Gaussians, but now with a
continuous h:
• Probabilistic Linear Discriminant Analysis
– Introduce an additional within-cluster hidden variable, w:
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
63
Object-level knowledge transfer for
Unsupervised Parsing: Prior and Inference
• The prior on y
– Prefer a certain number of clusters?
– Smoothness constraints? / Tree-based Region Prior?
– Image consistency constraints?
• Inference
–
–
–
–
Pairwise merging (within/between images)
Tree-based merging
Random subset selection
MCMC search
Oxford Brookes Seminar
Thursday 3rd September, 2009
University College London
64