KLT tracker & triangulation
Class 6
Read Shi and Tomasi’s paper on good features to track
http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf
Optional:
Lucas-Kanade 20 Years On http://www.ri.cmu.edu/projects/project_515.html
Feature matching vs. tracking
Image-to-image correspondences are key to
passive triangulation-based 3D reconstruction
Extract features independently
and then match by comparing
descriptors
Extract features in first images
and then try to find same
feature back in next view
What is a good feature?
Feature point extraction
• Approximate SSD for small displacement Δ
• Find points for which the following is maximum
maximize smallest eigenvalue of M
SIFT features
• Scale-space DoG maxima
• Verify minimum contrast and “cornerness”
• Orientation from dominant gradient
0
2
• Descriptor based on gradient distributions
0
2
Feature tracking
• Identify features and track them
over video
• Small difference between frames
• potential large difference overall
• Standard approach:
KLT (Kanade-Lukas-Tomasi)
Intermezzo: optical flow
• Brightness constancy assumption
(small motion)
• 1D example
possibility for iterative refinement
Intermezzo: optical flow
• Brightness constancy assumption
(small motion)
• 2D example
the “aperture” problem
(1 constraint)
(2 unknowns)
?
isophote I(t+1)=I
isophote I(t)=I
Intermezzo: optical flow
• How to deal with aperture problem?
(3 constraints if color gradients are different)
Assume neighbors have same displacement
Lucas-Kanade
Assume neighbors have same displacement
least-squares:
Alternative derivation
• Compute translation assuming it is small
differentiate:
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
Revisiting the small motion assumption
• Is this motion small enough?
• Probably not—it’s much larger than one pixel
(2nd order terms dominate)
• How might we solve this problem?
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Reduce the resolution!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
Coarse-to-fine optical flow
estimation
slides from
Bradsky and Thrun
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image It-1
u=10 pixels
Gaussian pyramid of image It-1
image I
Gaussian pyramid of image I
Coarse-to-fine optical flow
estimation
slides from
Bradsky and Thrun
run iterative L-K
warp & upsample
run iterative L-K
.
.
.
image IJt-1
Gaussian pyramid of image It-1
image I
Gaussian pyramid of image I
Good feature to track
• Tracking
• Use same window in feature selection as for tracking itself
maximize minimal eigenvalue of M
Strategy:
• Look for strong well distributed features, typically
few hundreds
• initialize and then track, renew feature when too
many are lost
Example
Simple displacement is sufficient between consecutive
frames, but not to compare to reference template
Example
Synthetic example
Good features to keep tracking
Perform affine alignment between first and last frame
Stop tracking features with too large errors
Live demo
• OpenCV (try it out!)
LKdemo
Triangulation
L2
C1
m1
M
L1
Triangulation
m2
C2
- calibration
- correspondences
Triangulation
• Backprojection
• Triangulation
Iterative least-squares
• Maximum Likelihood Triangulation
Backprojection
• Represent point as intersection of row and column
• Condition for solution?
Useful presentation for deriving and understanding multiple view geometry
(notice 3D planes are linear in 2D point coordinates)
Next class: epipolar geometry