Object Detection Using 2D Spatial Ordering Constraints
Yan Li†
Yanghai Tsin∗
Yakup Genc∗
†
∗
ECE Department
Carnegie Mellon University
{yanli,tk}@cs.cmu.edu
Takeo Kanade†
Real-time Vision and Modeling
Siemens Corporate Research
{yanghai.tsin,yakup.genc}@siemens.com
Abstract
have partly solved the camera pose problem by handling
the shape distortion using invariants in a transformed image space. Scale-invariant [13], even affine-invariant [21]
feature detectors have been introduced in the literature.
Despite all these progress, one remaining issue of computational detection systems is the limited discriminative
power of local features. This can be caused by the design of
a feature descriptor, the sampling resolution and dynamic
range of an imaging device, the signal to noise ratio of an
image, or visual ambiguities that are abundant in natural
scenes. One way to improve a detection system is to resolve
the above limitations and develop better feature descriptors,
which remains an active research topic in the computer vision community. In this paper, we take an alternative approach, namely, combining existing feature representations
with some spatial constraints to resolve local visual ambiguities. A single feature can be confused with other features in
a local scale. However, the ambiguity is less and less likely
if we consider the feature in the context of growing neighborhoods. We consider two levels of feature scales in this
work, a local cluster of features in a small scale, and all the
valid features on the object in a global scale.
When accurate 3D models and camera poses are not
known, it is impossible to predict the relative locations of
features with respect to each other in an arbitrary view. This
is the practical difficulty of bringing spatial constraints into
consideration. Instead of enforcing these strict metric constraints, we utilize a set of ordering constraints which are
also powerful enough to handle the object detection task.
Object detection is challenging partly due to the limited discriminative power of local feature descriptors. We
amend this limitation by incorporating spatial constraints
among neighboring features. We propose a two-step algorithm. First, a feature together with its spatial neighbors
form a flexible feature template. Two feature templates can
be compared more informatively than two individual features without knowing the 3D object model. A large portion
of false matches can be excluded after the first step. In a second global matching step, object detection is formulated as
a graph matching problem. A model graph is constructed by
applying Delaunay triangulation on the surviving features.
The best matching graph in an input image is computed by
finding the maximum a posterior (MAP) estimate of a binary
Markov Random Field with triangular maximal clique. The
optimization is solved by the max-product algorithm (a.k.a.
belief propagation). Experiments on both rigid and nonrigid objects demonstrate the generality and efficacy of the
proposed methods.
1. Introduction
We consider the problem of detecting both rigid and
nonrigid 3D objects in cluttered background with unknown
camera poses. For each object of interest, a single image is
taken against a clean background. Reliable features in the
image are detected and stored as an object model. When a
new image is given, the goal is to identify all visible features
and assemble them into a representation of the object in the
new view. This type of object detection system is desirable
for a wide variety of applications, such as user interface,
tracking, security, surveillance and robot navigation. However, this problem is challenging due to the unknown 3D
model and camera pose, occlusion, visual distraction and
visual ambiguity.
Recent advances in feature detection [13, 8, 18, 21, 10]
1.1. Related Work
The performances of feature detectors are compared in
a recent paper by Mikolajczyk and Schmid [14]. The SIFT
feature descriptor [13] is found to be among the best. We
use SIFT in this research, but our method can also adopt
other advanced feature detectors, e.g., [8, 18, 21, 10].
The importance of spatial configuration has long been
recognized in computer vision research. Umeyama [20]
1
gave an approximate solution to the weighted graph matching problem by eigendecomposition. Amit and Kong [2]
adopted decomposable graph for coding the structure of an
object. Cross and Hancock [4] adopted the Delaunay triangulation to build a model graph. Our proposed method
differs from the above work in that it is designed to automatically detect objects in cluttered scenes without any metric
constraint on the spatial configuration.
More recently, several groups of researchers [1, 5, 15]
proposed part-based object recognition algorithms by representing objects as flexible constellation of rigid parts. Our
goal is different in that our program detects specific objects
in cluttered background and finds a dense set of matched
features, instead of a few highly discriminative parts.
Rothganger et al. [17] explicitly reconstructed 3D object
models composed of small surface patches. The spatial relationship is represented by a subspace constraint. Extending their method to non-affine cameras or non-rigid objects
is not trivial. Jung and Lacroix [9] consider local groups
of interest points for robust matching. Group matches are
based on local affine transformation and intensity correlation. In [6], spatial constraints are represented in the form
of affine dissimilarities between neighboring features. A region grow approach is used to form a “group of aggregated
matches” (GAM). Our choice of the two feature scales enables both flexible local matching and strong global constraints. The sidedness constraint is applied in Ferrari et al.
[7] in a voting scheme. We also adopt sidedness but use it
in constraining a global optimization problem.
Our paper also takes advantage of the recent progress in
energy function minimization methods, especially the belief
propagation [12, 23] algorithms. These methods make it
possible to find a strong solution to a generally NP-hard
combinatorial optimization problem when there are a large
number of variables.
1.2. Terminology and Notations
At a modeling stage, we take a picture of the object of interest against a clean background and call the image a model
image. Any picture within which the object is to be identified is termed an input image. A feature is an image patch
that can be robustly identified despite viewpoint and illumination changes. A feature descriptor is an abstraction of
the feature appearance and it can be compared with other
descriptors of the same type to measure feature similarity.
The set of features detected in the model image is I and
J in the input image. An individual feature in the model
image is denoted using letter i, while a feature in an input
image is denoted using letter j. Different features in the
same image are distinguished using subscriptions, e.g., i1 ,
jn . A neighborhood of feature is N (·), and feature correspondence is a function f (·), i.e., jn = f (im ).
2. Local Matching using Angular
Ordering Constraints
2.1. Feature Representation
In this study we adopt Lowe’s scale invariant feature
transform (SIFT) [13] for detecting features. The features
are detected as extrema in a scale space (u, v, σ), where
(u, v) are the pixel coordinates and σ is the scale dimension. The scale space is constructed by convolving an input
image with a bank of difference-of-Gaussian (DoG) filters
with increasingly larger scales.
For each detected SIFT feature, a local gradient histogram is computed at 8 orientations over a 4x4 grid of spatial locations, giving a 128-dimension vector. Therefore,
each feature has a position, orientation, and scale within the
object model as well as a feature vector describing its appearance.
2.2. Flexible Feature Template
The process begins with feature detection in both images. Initial feature matching can be established by finding the most similar match in the feature descriptor space.
Due to visual ambiguity, however, this distance measurement alone is usually not good enough for exact feature correspondence.
Our solution in a local scale is motivated by the very
successful template matching method in appearance-based
vision, such as those in stereo, optical flow, structure from
motion, tracking and recognition. In a template matching algorithm, there are an intensity part and a geometry
part. The geometry determines the exact correspondence
between two points in two templates, while distance in intensity values determines their similarity. In our object detection problem, we can similarly group a specific feature i
together with its spatial neighbors N (i) and form a template
in a loose sense, where correspondences among features are
subject to some constraints, but not in a strict parametric
form. We call such a group of features a flexible feature
template Ti = {i, N (i)}. In the template matching analogy,
the “intensity” is equivalent to the SIFT feature descriptor.
However, the geometry that determines the correspondence
between features is generally unknown, due to lack of 3D
object models and camera poses, or due to nonrigid object
deformation. A question to be answered is how to match
two flexible templates in the absence of such important information.
We define the neighbors of a feature in a model image
as its m nearest neighbors. In an input image we define the
neighbors of a feature to be its n nearest neighbors. To be
conservative, we choose n = 1.5m to allow some modeling
j 4’
j1
i1
i2
j2
i
j
i5
j5
i3
i4
(a)
2.3. Flexible Feature Template Matching
j 1’
j’
j 2’
j3
(b)
(c)
Figure 1. Flexible templates matching. (a) A flexible
template defined in a reference view. (b) The matching
pattern in the input image. (c) A pattern with similar
local appearance, but different neighboring features.
errors. Our first assumption under such a choice of neighborhood is:
Assumption 1 If ik ∈ N (i), f (ik ) ∈ N (f (i)).
Under Assumption 1, one solution to computing the
matching score between two feature templates might be to
enumerate all possible correspondences, compute the total
feature distance under each correspondence, and pick the
smallest one. However, this approach is not only costly but
also unnecessary. Not all correspondences are physically
possible. Some feature correspondences require transparent surfaces or object surface topology change that involves
self-intersection. Most objects in computer vision study are
piecewise smooth. Thus some very general constraints can
be added to limit the search space of possible feature correspondences, yet strong enough to identify the features. One
of such constraints is based on the local angular order.
For a flexible feature template Ti , we build a local polar
coordinate with the origin anchored at i. Furthermore, we
define an cyclic angular order of its neighbors N (i) based
on their polar angles. Our second assumption is to suggest
the most likely feature correspondences.
Assumption 2 The angular orders of most flexible feature
templates are preserved from all viewpoints.
Figure 1 gives an example of angular order preserving
template matching. Figure 1(a) illustrates a feature i and its
spatial 5-nearest neighbors i1 -i5 . Figure 1(b) represents a
nonrigidly transformed version of the template in (a), while
Figure 1(c) shows a non-matching template. Although feature i matches feature j 0 well, the ordering of the neighboring features j10 , j20 and j40 is wrong and there are missing/outlier features. If both feature templates in (b) and (c)
are present in an input image, it is more likely that feature i
corresponds to j instead of j 0 , even if individually j 0 looks
slightly more similar to i than j does.
Next we quantify the matching score between two flexible feature templates. Let Ti = {i, i1 , i2 , . . . , im } and
Tj = {j, j1 , j2 , . . . , jn } be two templates that are centered
at i and j correspondingly. Denote j = f (i) as a angular
order preserving mapping between the two templates. By
angular order preservation we mean that,
1. For any i1 6= i2 , f (i1 ) 6= f (i2 );
2. For any triple of features (i1 , i2 , i3 ), if they appear in a
counter-clockwise order in Ti , the corresponding features f (i1 ), f (i2 ), f (i3 ) should appear in the same order in Tj .
Denote all angular order preserving mappings as a set F.
The distance between the two templates is defined by
Rij = d(i, j) + min
f ∈F
m
X
d (ik , f (ik ))
(1)
k=1
where d is the distance of two SIFT feature descriptors.
Due to feature detector errors or occlusion, some neighboring features may not be observed. To cope with this situation, we add an auxiliary feature ja in Tj , representing
absence of a feature. Features mapped to ja do not need
to obey either the uniqueness constraint (1) or the angular order constraint (2), but they are associated with a fixed
penalty (a constant distance for d(in , ja ), any in ). That is,
absence of a feature is allowed at the cost of a penalty. Notice that by capping the matching errors between individual
features with a fixed penalty, we avoid the possibility of infinite influence of outlier patterns, thus bringing robustness
to the matching process.
Now we are ready to explain our flexible template matching algorithm. First, we need to select candidate feature
correspondences, i.e., centers of the templates. Candidate
feature correspondences are established by finding the most
similar feature in the feature descriptor space. Specifically,
we find the best match of each model feature i in J and the
best match of each input feature j in I. Only mutually-best
matches are accepted. In practice, this approach helps to
eliminate many false matches from the start. Once the center feature correspondences are established, flexible templates can be built by finding their designated k nearest
neighbors. In our experiments we fix k = 5 for the model
image, and k = 8 for an input image.
Second, we match two corresponding feature templates
Ti and Tj by dynamic programming. We start by studying
the neighborhood features of the two templates, N (i) =
{i1 , i2 , . . . , im } and N (j) = {j1 , j2 , . . . , jn }. In the case
that both N (i) and N (j) are angular-order sorted, an
important observation is that an order preserving correspondence is also angular order preserving. For example,
{f (i1 ) = j2 , f (i2 ) = j4 , f (i3 ) = j5 } is order preserving
but {f (i1 ) = j4 , f (i2 ) = j2 , f (i3 ) = j5 } is not. This property also holds for all cyclic permutations of N (j). As a
result, the template matching cost in Eqn. (1) can be computed as follows,
• Enumerate all the cyclic permutations of N (j).
• For each cyclic permutation, find the minimum matching cost among all order preserving correspondences.
This is similar to the intra-scanline dynamic programming stereo algorithm [16] and can be solved using the
methods therein.
• Find the minimum cost among all cyclic permutations
and add center feature distance d(i, j) as the minimum
matching cost Rij .
After the minimum matching costs for all pairs of flexible feature templates are computed, false matches can be
excluded by thresholding. In our experiments we use a fixed
threshold that is empirically determined.
3. Global Topology Constraint
Although flexible template matching filters out a majority of background features and outliers, there are in general
still some false matches which are very similar in appearance and happen to satisfy the angular ordering constraint.
In order to make the detection more robust, global placement of the matched features must be considered. This motivates us to use global topology constraint to detect false
matches.
Assumption 3 The sidedness constraint of any triangle in
a model graph is preserved in input images.
The preservation of sidedness implies local planarity of
graphs, i.e., folding and edge crossing is not permitted locally. In this sense, the matched graph should remain planar
in any view.
3.2 A Bayesian Formulation for Graph Matching
In detection, we wish to find the best labeling (true or
false match) for each matched feature in the input image, as
well as to preserve the spatial configuration constructed in
the model image. The key insight in this process is: if a feature match is correct, it should collaborate with its neighboring matches to form a locally planar graph which is topologically consistent with the reference graph. In addition, the
local subgraphs should coordinate with each other to evolve
into a maximal clique with respect to the reference graph.
Such a spatial interaction can be modeled in a Markov
Random Field (MRF) framwork. The feature labeling is
then reduced to a maximum a posterior MRF problem. We
model the feature labeling as a binary field on the reference
graph, denoted by L. The MAP estimate is the configuration with maximum probability given features F = {I, J }:
L∗ = arg max P (L|F )
(2)
L
where I and J now represent the matched features obtained
from flexible feature template matching. Bayes rule then
implies
L∗ = arg max P (F |L)P (L)
(3)
L
3.1 Spatial Configuration Modeling
A natural way to express the spatial configuration of features is to use a graph G = (V, E) where the vertices
V = (i1 , i2 , . . . , in ) correspond to the surviving features
after the first step. The matching problem is that of finding the best assignment (true or false match) of the features in the input image, where the quality of an assignment
depends both on the local evidence of individual features
and on agreement of the placement with the global topology. We establish the edges between the vertices by Delaunay triangulation. Our graph is different from the previous
work [4, 20] in that 1) vertices in our formulation encode
abstracted appearance information; 2) edges encode spatial
ordering of features besides connectivity. To be specific,
the ordering is the sidedness of three non-collinear features,
i.e., whether feature i3 is on the left half plane or right half
plane when we travel from i1 to i2 . Our last assumption
helps to propagate this ordering to the input images,
Intuitively, the estimation problem is formulated using a
likelihood term that enforces fidelity to the measurements
and a prior term that embodies assumptions about the spatial variation of the data. The likelihood P (F |L) is defined
by
P (F |L)
=
=
¡
¢
1 Y
exp − γ(i, li , F )
K
i∈I
Ã
!
X
1
exp −
γ(i, li , F )
K
i∈I
where γ(i, li , F ) is the matching cost of feature i given
observation F , K is a normalization constant. li is a binary variable which indicates whether feature i is a correct
match:
½
1
if i is a correct match
li =
(4)
0
otherwise
Let C be the set of maximal cliques 1 . The prior term can be
written as
Y
¡
¢
exp − ϕi1 i2 i3 (li1 , li2 , li3 )
P (L) ∝
(i1 ,i2 ,i3 )∈C
Φ i1
i1
i1
i2
Ψ i1i2i3
i3
i3
(a)
(b)
Φ i2
i2
Φ i3
where ϕi1 i2 i3 (li1 , li2 , li3 ) is the clique function of triangle whose nodes are (i1 , i2 , i3 ). Now the MAP problem in
Eqn. 3 becomes
max P (L|F ) = max P (F |L)P (L)
L
L
(
Y
¡
¢
exp − γ(i, li , F )
∝ max
L
(5)
i∈I

¢
exp − ϕi1 i2 i3 (li1 , li2 , li3 )

(i1 ,i2 ,i3 )∈C
Y
Y
∝ max
Ψi1 i2 i3 (li1 , li2 , li3 )
Φi (li )
Y
¡
L
(i1 ,i2 ,i3 )∈C
i
where
Φi (li ) =
Ψi1 i2 i3 (li1 , li2 , li3 ) =
¡
¢
exp − γ(i, li , F )
¡
¢
exp − ϕi1 i2 i3 (li1 , li2 , li3 )
are local evidence potential and clique potential, respectively. The first term ensures that the recovered correspondences are faithful to the data, while the second term encodes our prior assumption that local graphical planarity
should be preserved. A graphical depiction of this model
is shown in Fig 2(a). The filled-in circles represent the observed image nodes, while the empty circles represent the
“hidden” labeling nodes lf . Note that the clique potential
in our model differs from its counterparts in [19] and [11]
in that we model the spatial interaction of three nodes instead of two neighboring nodes in a pairwise Markov random field [23].
3.3 Implementation
The computation of the global optimal solution to the
energy function in Eqn. 5 is NP-hard because we need to
examine all the possible labeling configurations and compute their energy. The large number of nodes in the graph
also calls for faster algorithms like belief propagation (a.k.a
sum-product) [23] or graph cut [11]. However, a careful investigation shows that the energy function in Eqn. 5 is not
regular or graph-representable [11]. Therefore, standard st-cut algorithm cannot be applied in this case.
We propose to use the max-product algorithm for factor
graph [12] to solve the MAP-MRF problem. Although the
1 A maximal clique is a triangle in the Delauney triangulation in our
case.
Figure 2. (a) MRF; (b) The corresponding factor
graph.
max-product (or sum-product for marginal distribution estimation) algorithm is an approximate inference algorithm
which cannot guarantee the global optimal solution, it has
been successfully applied in many Bayesian inference problems in vision, bioinformatics, and error-correcting coding [12, 22]. In Fig 2(b) we illustrate the equivalent factor
graph to the MRF. Note that we introduce factor functions
Φ(·) of a single variable if they are attached to a single “hidden” node, and factor functions Ψ(·, ·, ·) of three variables
if they link three “hidden” nodes. Yedidia et al. [23] show
that the belief propagation algorithm is precisely mathematically equivalent at every iteration to the max-product algorithm by converting a factor graph into a pairwise MRF.
However, a factor graph representation is preferred in our
case because each node in the graph is physically meaningful and the message passing rule can be derived in a straightforward way.
3.3.1 The Max-Product Algorithm
Let mi→Φ (li ) and mi→Ψ (li ) denote the message sent from
node i to its neighboring function nodes; let mΦ→i (li ) and
mΨ→i (li ) denote the message sent from function nodes to
node i. The message passing performed by the max-product
algorithm can be expressed as follows:
1. Initialize all the messages m(li ) as unit messages.
2. For t = 1 : N , update the messages iteratively
variable to local function
Y (t)
(t+1)
mi→Φi (li ) ←−
mΨ→i (li )
Ψ∈N (i)

(t+1)
mi1 →Ψi i i (li1 ) ←− 
1 2 3

Y
(t)
mΨ→i1 (li1 )
Ψ∈N (i1 )\{Ψi1 i2 i3 }
(t)
mΦi
1 →i1
(li1 )
·
We use 4i1 i2 i3 and 4f (i1 )f (i2 )f (i3 ) to denote two
matched triangles. The clique potential is defined by
local function to variable
(t+1)
mΦi →i (li ) ←− Φi (li )
³
(t+1)
mΨi i i →i1 (li1 ) ←− max Ψi1 i2 i3 (li1 , li2 , li3 )
li2 ,li3
1 2 3
(t)
mi2 →Ψi
(t)
1 i2 i3
(li2 )mi3 →Ψi
1 i2 i3
´
(li3 )
where N (i) is the neighboring function nodes of i.
3. Compute the beliefs and MAP
µi (li )
= κΦi (li )
Y
mΨ→i (li )
Ψ∈N (i)
liM AP
=
arg max µi (li )
li
Notice that the variable to local function message
mi→Φi (li ) is not involved in the MAP computation explicitly. We list it here for the purpose of completeness.
3.3.2 Model Local Evidence
We model the local evidence as a robust function of the flexible template matching distance:
(
2
Rij
if Rij ≤ θ
2
1+R
γ(i, 1, F ) =
ij
α
otherwise
(
2
θ
if Rij ≥ θ
2
1+Rij
γ(i, 0, F ) =
α
otherwise
where Rij is the distance defined in Eqn. 1, θ is a predeθ2
fined threshold, and α = 1+θ
2 is the robust parameter. Our
robust function is similar to the Geman and McClure function [3] except that we make a truncation at the threshold
θ. Local evidence is defined in such a way that γ(i, li , F )
has converse behaviors for li = 0 and li = 1, i.e., when i
is labeled as a correct match, the local evidence favors the
feature with small matching cost; while when i is labeled as
a mismatch, the local evidence favors the feature with large
matching cost.
3.3.3 Model Clique Potential
The clique potential models the spatial interaction among
neighboring features. For objects with small deformations,
we assume that the sidedness of the vertices in a triangle
is preserved. The sidedness of a triple (i1 , i2 , i3 ) describes
their orientation in 2D, i.e., whether the vertices occur in
a clockwise (or counter-clockwise) order. As suggested by
Ferrari et al. [7], the sidedness constraint is valid for both
coplanar or non-coplanar triples and can be used to detect
false matches. Such a constraint can be determined by eval−
→ −
→ →
−
uating the sign of the scalar ( i1 × i2 ) · i3 .
ϕ
(l , l , l ) =
i1 i2 i3 i1 i2 i3
λ
if Sign(4i1 i2 i3 ) = Sign(4f (i1 )f (i2 )f (i3 ) )




and li1 = li2 = li3 = 1;

5λ
if li1 or li2 or li3 is 0;


20λ if Sign(4i1 i2 i3 ) 6= Sign(4f (i1 )f (i2 )f (i3 ) )



and li1 = li2 = li3 = 1
where λ is a parameter to measure the consistency. We favor topologically consistent matches, while enforce strong
penalty for matches which violate the sidedness constraint.
We also apply less penalty for an ambiguous matching
clique (one or more vertices are labeled as mismatches in
one triangle).
4. Experimental Results
Feature Template Matching: Our first example is to show
the process of finding mutually best local matches and flexible template matching. In Figure 3, the green dots are the
features which have found their mutually best matches, i.e.,
possible template centers. Although significant feature clusters on the background have been removed (we do not show
the original SIFT features here because they are very dense),
we see quite a lot of errors in the putative matches. This can
be observed partly from the green dots scattered outside of
the object region in the input image. After flexible template matching, the surviving feature template centers are
shown in red dots. We can see that flexible template matching effectively gets rid of all except one (on the border of
the “algorithms book”) false correspondences in this case.
In Figure 4 we show details of the template matching
process. We demonstrate the matching of a matched pair A
and A0 , and a mismatched pair B and B 0 (which are shown
in red crosses). For corresponding features in the model
view and input view, their 5- and 8-nearest neighbors are
shown to the right of the figure. Feature A finds very similar surrounding patches in A0 that are also angular order
preserving. However, B is supported very weakly by its
neighbors. As a result, B is detected as a false match.
Object Detection in Cluttered Scenes: Next we show results of the global matching step. The surviving features after the first step are subject to a global topology verification
procedure in which the max-product algorithm is performed
to search for the maximal subgraph in the input view. Figure 5 shows some object detection results in highly cluttered
scenes. For each input view, we show the matched graph
on the model view. Red lines in the image signal wrong
matches left over by the flexible template matching. They
are detected because their removal will give a maximal subgraph that is also faithful to the data.
B
A
A’
B’
Figure 3. Flexible feature template matching. Green
dots: mutually most similar features. Red dots: correspondences found by template matching.
Figure 5. Detection in cluttered background.
TA
TA’
TB
TB’
Figure 4. Matching by dynamic programming. Solid
lines: good matches; dashed lines: weak matches.
Non-rigid Object Detection: The proposed object detection framework can also be applied to non-rigid objects. In
Figure 6 we show the detection result of a magazine. We
manually introduce severe non-rigid distortions in the test
views which do not follow any explicit transformation. It
can be seen that our algorithm has successfully captured the
object shape even under a cluttered background, with severe
distortion and partial occlusion.
Object with Repetitive Patterns: In Figure 8, we show
a very challenging sequence in which the interested object
has repetitive patterns and is occluded by another object.
We show the object detection results at different scales and
orientations.
An Assumption Violation Case: Finally Figure 7 shows an
example of object detection when our 2D spatial ordering
assumptions are violated. The scene consists of a marker
pen in front of a textured background. Due to the picket-
and-fence effect, the ordering constraints of some features
are obviously violated. For example, the pen is to the left of
the “multiple view geometry book” in the model view, while
to the right of the same book in the test view. Features highlighted by two circles will have their local ordering constraint violated. However, our program is robust enough to
detect these violations and show them in red lines. Meanwhile, the rest of the scene is accurately detected.
Five parameters need to be set in our algorithm: one
threshold for flexible template matching, m and n for spatial
nearest neighbors selection, θ and λ for local evidence and
clique potential modeling. We choose these parameters empirically and all the experiments shown here use the same
set of parameters. The max-product algorithm has shown
to be very efficient for the graph matching problem. For a
graph with hundreds of nodes, it takes only 10 to 20 iterations for the beliefs to converge. We have tested our algorithm on a variety of objects and extensive experiments
show that the proposed method is able to detect object in
cluttered scenes effectively and efficiently. Our system currently runs at 2 frame-per-second on 320x240 images.
5. Conclusion
We have presented a two-step framework for general 3D
object detection in cluttered scenes with unknown camera
poses. We demonstrated that false matches between features are progressively detected by data evidence and 2D
ordering constraints in both a local and a global scale. Ex-
Figure 6. Nonrigid object detection.
Figure 8. Detecting objects with repetitive patterns.
[7] V. Ferrari, T. Tuytelaars, and L. Van Gool. Integrating multiple model
views for object recognition. In CVPR, 2004.
[8] W. T. Freeman and E. H. Adelson. The design and use of steerable
filters. PAMI, 13(9):891–906, 1991.
[9] I. K. Jung and S. Lacroix. A robust interest point matching algorithm.
In ICCV, 2001.
[10] T. Kadir and M. Brady. Scale, saliency and image description. IJCV,
45(2):83–105, 2001.
Figure 7. An assumption violation case.
periments on various objects have shown great promise in
applying the proposed methods to real world applications.
In our future work, we would like to increase the usability of the proposed method. We are interested in developing
real time object detection systems. Currently the most time
consuming part of our program is SIFT feature detection.
We plan to investigate other feature detectors and efficient
algorithms.
Acknowledgement
We would like to thank Prof. David Lowe for kindly
providing the SIFT source code.
References
[1] S. Agarwal and D. Roth. Learning a sparse representation for object
detection. In ECCV, pages 113–127, 2002.
[2] Y. Amit, D. Geman, and K. Wilder. Joint induction of shape features
and tree classifiers. PAMI, 19(11):1300–1305, 1997.
[3] M. J. Black and A. Rangarajan. On the unification of line processes,
outlier rejection, and robust statistics with applications in early vision. IJCV, 19(1):57–91, 1996.
[4] A. Cross and E. Hancock. Graph matching with a dual-step EM
algorithm. PAMI, 20(11):1236–1253, 1998.
[5] R. Fergus, P. Perona, and Z. Zisserman. Object class recognition by
unsupervised scale-invariant learning. In CVPR, 2003.
[6] V. Ferrari, T. Tuytelaars, and L. Van Gool. Wide baseline multiple
view correspondence. In CVPR, pages 718–725, 2003.
[11] V. Kolmogorov and R. Zabih. What energy functions can be minimized via graph cuts. PAMI, 26(2):147–159, 2004.
[12] F. R. Kschischang, B. J. Frey, and H.-A. Loeliger. Factor graphs
and the sum-product algorithm. IEEE Transactions on Information
Theory, 47(2):498–519, 2001.
[13] D. G. Lowe. Distinctive image features from scale-invariant keypoints. IJCV, 60(2):91–110, 2004.
[14] K. Mikolajczyk and C. Schmid. A performance evaluation of local
descriptors. In CVPR, 2003.
[15] P. Moreels, M. Maire, and P. Perona. Recognition by probabilistic
hypothesis construction. In ECCV, 2004.
[16] Y. Ohta and T. Kanade. Stereo by intra- and inter-scanline search
using dynamic programming. PAMI, 7(2):139–154, 1985.
[17] F. Rothganger, S. Lazebnik, C. Schmid, and J. Ponce. 3D object
modeling and recognition using affine-invariant patches and multiview spatial constraints. In CVPR, 2003.
[18] F. Schaffalitzky and A. Zisserman. Multi-view matching for unordered image sets, or How do I organize my holiday snaps? In
ECCV, 2002.
[19] J. Sun, N. N. Zheng, and H. Y. Shum. Stereo matching using belief
propagation. PAMI, 25(7):787–800, 2003.
[20] S. Umeyama. An eigendecomposition approach to weighted graph
matching problems. PAMI, 10(5):695–703, 1988.
[21] L. Van Gool, T. Moons, and D. Ungureanu. Affine photometric invariants for planar intensity patterns. In ECCV, pages 642–651, 1996.
[22] M. J. Wainwright and M. I. Jordan. Graphical models, exponential
families, and variational inference. Technical Report 69, Department
of Statistics, University of California, Berkeley, 2003.
[23] J. S. Yedidia, W. T. Freeman, and Y. Weiss. Understanding belief propagation and its generalizations. Technical Report 2001-22,
MERL, 2001.