Qualifying Exam:
Contour Grouping
Vida Movahedi
Supervisor: James Elder
Supervisory Committee:
Minas Spetsakis, Jeff Edmonds
York University
Summer 2009
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Introduction
• Segmentation
Partition an image into regions, each corresponding to
an object or entity
• Figure-Ground segmentation
Segmentation Methods
• Regional Segmentation
– Use regional info, optimize labelling of regional tokens,
e.g. clustering
– Depending on uniformity in object region
• Active Contour Models
– Use regional (external) & boundary (internal) info,
optimize deformation of model
– Sensitivity to initialization, too smooth
• Contour Grouping
– Use boundary info (& regional info), optimize grouping
of contour fragments
Problem Definition
• Input:
Color image
• Goal:
Figure-ground segmentation
• Method:
Contour Grouping
• Other available info: None
- No motion, stereo or video information
- No user interactions
- No assumptions on object types, shapes, color, etc.
- No assumptions on background or lighting conditions
Challenges
• High-dimensional data space, lots of information,
many cues
• Unknown cue integration
• Global optimization in a non-convex
multidimensional space
• Camera, imaging, quantization noise
• Clutter in natural scenes
• Occluded or overlapping objects
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Steps
Pre-processing
Image
Edge
Line /Curve
Detection
Approximation
Grouping Algorithm
Learned
Parameters or
Distributions
Saliency
Optimization
Computations
Algorithm
Figure/Ground
Segmentation
Pre-processing
Image Edge Map  Line Map  Contour
Gestalt Cues
How is grouping done in human vision?
• Proximity
• Similarity
– Brightness
– Contrast
• Good continuation
– Parallelism
– Co-circularity
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Grouping Methods
• Local Heuristic methods
– Defining a heuristic cost for contour
hypotheses, find the optimal one
• Local Probabilistic methods
– Find posterior probability of contour
hypotheses given cues, find the optimal one
• Global methods
– An extra step of calculating global saliencies
based on local measures
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Local & Heuristic
Example: Ratio Contour Method
(Wang et. al, PAMI’05)
• Detected/ virtual fragments
• Contour cost= curvature & gap per unit length
• Graph model
• Alternate cycle
Local & Heuristic
Example: Ratio Contour Method
(Wang et. al, PAMI’05)
• Edge/ Link costs
 [ (t )  
w(e) 
2
(t )]dt
B (e)
1 if v(t ) is on a virtual fragment
0 if v(t ) is on a real fragment
 (t )  
l (e) 
 dt
B (e)
• Ratio Contour Algorithm

(C ) 

eC
eC
w(e)
l ( e)
Sample Results for RC method
Image
Image
RC
RC
(from Wang et al., PAMI’05)
(from Stahl & Wang, TIP’07)
RRC
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Local & Probabilistic
(Elder et al., PAMI’03)
• Bayesian Rule:
p( H | D) 
p( D | H ). p( H )
1

1
p( D)
1  LP 
L
p( D | H )
p( H )
,P 
p( D | H )
p(H )
• Contour saliency= posterior probability of contour
c  (t1 ,..., t n 1 )  C ,  i {1...N }, i {1...n}
c*  arg max p(c  C | D)
c
• Assumptions:
– Markov Chain Assumption
– Independence of evidence from cues
– Comparing contours of same length
p(c  C | D)   pio

ti c
p

( ti ,t j )c
c
ij
Local & Probabilistic
(Elder et al., PAMI’03)
• Graph Model


o
c

log  p(c  C | D)    log( p )   log( p )    w (vi )   w (eij ) 
 v  P




ti c
( ti ,t j )c
( vi ,v j )Pc
i c

o
i
• Node weight
wo (vi )   log( pio ), i  1..N
• Link weight
wc (eij )   log( pijc ), i, j  1..N
• Shortest path/cycle
• Approximate search
c
ij
vi
eij
vj
Sample Results for Probabilistic Methods
(from Estrada & Elder- CVPRW’06)
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Global Model
Local weights
Global weights
Global Saliency
• Edge/Link Affinity
Based on collinearity, proximity, etc.
• Edge/ Link Saliency
Relative number of closed random
walks which visit that edge/link
(Mahamud et al., PAMI’03)
• Shown to be relevant to the
eigenvalues and eigenvectors of
the affinity matrix
• Grouping based on global saliency
Some Results of the Untangling method
(from Zhu; Song; Shi- ICCV’07)
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Evaluation
• Empirical discrepancy methods
The output of algorithms is compared with a
reference segmentation or ground truth
• Requirements
– A ground truth dataset
– An error measure
SOD: Salient Object Dataset
• Based on Berkeley Segmentation Dataset (BSD)
• 300 images, randomly showing 818
segmentations (half of BSD) to each of 7 subjects
• 12,110 object boundaries obtained
1
1
1
1
1
Region-based Error Measures
• Example
RIM ( A, B)  1 

RA  RB
RA  RB
| RA |  RA  RB
RA  RB

| RB |  RA  RB
RA  RB
• Not sensitive to some large shape features
(e.g., spikes, wiggles)
Boundary-based Error
Measures
d B (a)  min d (a, b) , a  A
bB
SDB ( A, B)  {d B (a), a  A}
h( A, B)  max( SDB ( A, B))  max min d (a, b)
aA
H ( A, B)  max h( A, B), h( B, A)
bB
• Not sensitive to object topology and some
large shape features (e.g., loop-backs,
wiggles)
Mixed Error Measures
• Example
1 1
MM ( A, B)  
2  N fn
N fn
1
dA( pj ) 

N fp
j 1


d
(
q
)

B
k 
k 1

N fp
pj, j=1..Nfn are pixels in the
false negative region (RB-RA)
qk, k=1..Nfp are pixels in the
false positive region (RA-RB)
• Not sensitive to some large shape features. Does
not respect ordering along contours.
Contour Mapping Measure
• Based upon a matching between
all points on the two boundaries
• Monotonically non-decreasing
• Allowing one-to-one, many-toone, and one-to-many matching
• Error= average distance between
matched pairs
• Dynamic Programming
Contour Mapping Distance=7.73
Contents
• Introduction
• Preliminary Concepts
– Pre-processing
– Gestalt cues
• Methods
– Local & Heuristic
– Local & Probabilistic
– Global Saliency
• Evaluation
• Conclusion & open problems
Conclusion & Open Problems
• Cue selection and combination
• Grouping Model
–
Global saliency
–
Probabilistic models
• Optimization Algorithms
• Hierarchical and multi-scale algorithms
• Quantitative evaluation