Estimating Human Body Configurations
using Shape Context Matching
Greg Mori and Jitendra Malik
University of California
Berkeley
Computer Vision Group
Problem
University of California
Berkeley
Computer Vision Group
Approach: Exemplar-based Matching
• Set of stored exemplars with hand-labeled
keypoints
• Obtain sample points
• Deformable matching to exemplars:
– Shape context matching to get correspondences
– Kinematic chain deformation model
• Estimate 3D body configuration
University of California
Berkeley
Computer Vision Group
Comparing Pointsets
University of California
Berkeley
Computer Vision Group
Shape Context
Count the number of points
inside each bin, e.g.:
Count = 4
...
Count = 10
 Compact representation
of distribution of points
relative to each point
University of California
Berkeley
Computer Vision Group
Shape Context
University of California
Berkeley
Computer Vision Group
Comparing Shape Contexts
Compute matching costs using
Chi Squared distance:
Recover correspondences by
solving linear assignment
problem with costs Cij
[Jonker & Volgenant 1987]
University of California
Berkeley
Computer Vision Group
Deformable Matching
• Kinematic chain-based
deformation model
• Use iterations of
correspondence and
deformation
• Keypoints on exemplars
are deformed to locations
on query image
University of California
Berkeley
Computer Vision Group
University of California
Berkeley
Computer Vision Group
Problem
University of California
Berkeley
Computer Vision Group
Estimate 3D Body Configuration
[Taylor ’00]
• Known:
–
–
–
–
Relative lengths of body segments
(x,y) Image locations of keypoints
“closer endpoint” labels for each segment
Scaled orthographic camera model
• Solve for 3D locations of keypoints up to some scale
factor
– Scale factor can be estimated automatically
University of California
Berkeley
Computer Vision Group
Solving for Foreshortening
l  (X
2
University of California
Berkeley
1 X 2)
2
 (Y Y )  (Z  Z )
1
2
1
2
2
2
dZ  l 2  ((u1  u2 ) 2  (u1  u2 ) 2 Computer
) / s 2 Vision Group
Choosing Scale
s
University of California
Berkeley
l
((u1  u2 )  (v1  v2Computer
) ) Vision Group
2
2
Results
University of California
Berkeley
Computer Vision Group
University of California
Berkeley
Computer Vision Group
Multiple Exemplars
• Parts-based approach
– Use a combination of keypoints/whole limbs from different
exemplars
– Reduces the number of exemplars needed
• Compute a matching cost for each limb from every
exemplar
• Compute pairwise “consistency” costs for
neighbouring limbs
• Use dynamic programming to find best K
configurations
University of California
Berkeley
Computer Vision Group
Parts-based Approach
University of California
Berkeley
Computer Vision Group
Tracking by Repeated Finding
University of California
Berkeley
Computer Vision Group