A Global Model and Algorithm
for Finding the Curves in an
Image
Pedro F. Felzenszwalb
Department of Computer Science
University of Chicago
Joint work with David McAllester from TTI-Chicago
Introduction
•
Find a global interpretation of an image in terms of a
small set of smooth curves
-
Think of “parsing” the image into curves
•
Related to image segmentation, where we want to
explain the whole image as a set of regions
•
Step towards parsing the image into objects
-
Simpler version of the problem
Can combine curves to build objects (grammar)
Applications
Edge detection
Grouping
Roads in satellite images
Arteries in MRI
Discrete curves
•
•
Curve is a sequence of short oriented segments
Fixed number of segments leaving each pixel
-
Corresponding to different orientations
segments leaving p
curve
Let s be a segment, S be the set of all segments
Basic idea
•
•
Test for curve presence at each segment
Ys: output of a local filter associated with segment s
-
Noisy indication that some curve goes through s
Image gradient perpendicular to s, etc.
•
Look for a global explanation of Ys in terms of a set
of curves H = (C1,C2,...,Ck)
•
Each curve “explains” filter responses along its path,
other responses explained by background model
•
Each curve should be smooth, n should be small
Model for scenes with one curve
•
•
•
Consider scene with one (unknown) curve X
Our goal is to estimate X using the filter responses
Assume X is chosen according to a Markov process
so that curves tend to be smooth
C = (s1 , . . . , sn )
1 !
P (X = C) =
Φ(si , si+1 )
Z
probability that unknown curve equals C
Data model
•
Assume value of Ys depends only on wether or not
segment is part of X
P (Ys |s ∈ X) = Pf (Ys )
•
P (Ys |s !∈ X) = Pb (Ys )
Assume filter responses are independent when
conditioned on location of true curve
!
!
P (Ys , s ∈ S|X = C) =
Pf (Ys )
Pb (Ys )
s∈C
s"∈C
probability observing all filter responses
MAP estimation
•
Best guess for X,
C ∗ = arg max P (X = C|Ys , s ∈ S)
C
•
Using Bayes law, etc.
! Pf (Ys )
C ∗ = arg max P (X = C)
C
Pb (Ys )
s∈C
•
Find smooth curve that goes through segments that
look more like foreground than background
•
Can solve using dynamic programming...
Multiple curves
•
•
First idea: find best curve, then second best curve...
Problem: algorithm is biased towards explaining the
whole image at once
Two circles, or figure eight?
•
Need to explicitly model scenes with multiple objects
Multiple curves
•
•
•
Let X=(C1,C2,...,Ck) be the set of curves in an image
•
Best guess for X is set of curves H maximizing...
Assume P (|X| = k) ∝ ak with 0 < a < 1
Assume each curve is smooth, drawn from same
Markov process P(C) as before
P (X = H|Ys , s ∈ S) ∝
!
C∈H
"
aP (C)
!
s∈C
#
Pf (Ys )
!
s∈Hb
Pb (Ys )
Combinatorial optimization
wb (s) = − log Pb (Ys )
wf (s) = − log Pf (Ys )
t(si , si+1 ) = − log Φ(si , si+1 )
A = − log a
Find: H = (C1 , . . . , Ck )
Minimizing: cost(H) =
!
(w(C) + A) +
C∈H
cost per curve
w(C) =
!
wf (si ) +
!
t(si , si+1 )
!
wb (s)
s∈Hb
cost of background
Minimum weight set cover problem!
•
Pick a set of curves and background segments that
cover all of S
•
Cost of picking a curve: w(C) + A
Cost of picking background segment: wb(s)
Minimize sum of costs...
Greedy approximation algorithm:
Sequentially select curve or background segment with
minimum cost per element covered
Finding an optimal curve
We need to find a curve with
minimum cost per length
cost(C)
|C|
cost(C) = w(C) + A
w(C) =
!
wf (si ) +
!
t(si , si+1 )
Algorithm idea
•
•
•
•
Q: queue of curves to be extended
Generate all curves of length 1 and put in Q
Repeat
-
Take curve C from Q
Generate one segment extensions of C, put in Q
Stop if no extension of curve in Q can be better than a
curve generated so far
Speedup...
•
Say A and B are two curves of same length ending in
same segment s
•
Only keep curve with smaller weight
This is always ok
Keep track of
-
T[s,l]: cost of best curve ending at s with length l
P[s,l]: previous segment of best curves
Best curves obtained by tracing back using P
Stopping criteria
•
Say C is prefix of optimal curve O, then:
cost(O)
w(C)
≥
|O|
|C|
•
Stop when every curve in Q has weight density larger
than cost density of best curve so far
•
Expand curves in weight density order
-
Small for every prefix of the optimal curve
Quickly generate optimal curve and stop
Results
More results
Subpixel curves
Berkeley segmentation benchmark
Edge detection evaluation
•
•
•
•
Compare detected edges to “ground truth”
Precision:
-
fraction of detected edges that are true edges
Recall:
-
fraction of true edges detected
Vary detection threshold to get precision/recall curve
BSD results
Future work
•
•
Improve curve model
-
Better data model, and different curve types
Move from curves to objects
-
Use minimum cover algorithm, but instead of
curves select “objects”
-
Define objects in terms of context free grammar
(curves are a simple example)
-
Build objects like we build curves by sequentially
extending smaller objects in best-first order