CSSE463: Image Recognition
 This week
Day 30
Today: motion vectors and tracking
 Friday: Project workday.
First status report due. Please email by 11
am a brief summary of:

 what
you did
 what you didn’t and the technical challenges (lack
of time isn’t an excuse)
 what you will do next week.

Questions?
What is image flow?

Notice that we can
take partial
derivatives with
respect to x, y, and
time.
Image flow equations

Goal: to find where each pixel in frame t moves in frame t+Dt



Assume:







E.g. for 2 adjacent frames, Dt = 1
That is, Dx, Dy are unknown
Illumination of object doesn’t change
Distances of object from camera or lighting don’t change
Each small intensity neighborhood can be observed in consecutive
frames: f(x,y,t)f(x+Dx, y+Dy, t+Dt) for some Dx, Dy (the correct motion
vector).
Compute a Taylor-series expansion around a point in (x,y,t)
coordinates.
Gives edge gradient and temporal gradient
Solve for (Dx, Dy)
See answers to first quiz question now
Limitations


Assumptions don’t always hold in real-world
images.
Doesn’t give a unique solution for flow

Sometimes motion is ambiguous

Look at last question on last quiz.

“Live demo”

http://www.cs.mcgill.ca/~chundt/coherence/
Aperture problem

When we only see a small part of the image, motion can
be ambiguous

Lesson: By constraining ourselves to a small
neighborhood, we lose the “big picture”

Solutions:

Assume pixels belong to same object and have same motion

Propagate constraints between pixels




Can still take many frames to converge
Doesn‘t work at occlusions (hidden parts of object)
Sensitive to outliers
Only track motion of “interesting points”
Q1
Interest point tracking
Idea: corners and other easily-identifiable
points can be tracked, and their motion is
unambiguous.
 Technique for calculating a set of interest
points:


Loop over image:
 Calculate
an interest score for a small
neighborhood around each pixel
 If the score is above threshold, then add it to a
point set to be returned.
Q2
Interest scores

One solution:




Take minimum variance of those found in 4 directions:
(3x3 shown, but would use bigger window)
Another related texture-based operator is based on
directions with significant edge gradient.
Could also detect corners using Kirsch operators (similar
to other edge operators).
Tracking interest points

Search for them directly using a template.
Choose one with highest correlation
 Just need to search in small neighborhood if
frame rate is high enough
 Shapiro’s Alg. 9.3

Review: Correlation

Dot product with template at
each point.

Normalize each to avoid false
matches with bright regions

Matlab libraries exist to extract frames (still
images) from video for processing.
Computing trajectories



Problem: what if unlabeled moving points belong to
multiple trajectories?
Idea: want to maximize smoothness, defined as change
in direction and change of speed.
Given two points pt, pt+1, thought to be related, calculate
vt = pt+1 – pt. Then:
 Vt 1  Vt
S t  w
 Vt 1 Vt
direction
 2 Vt 1 Vt

  (1  w)

 Vt 1  Vt






Average speed
compared to
geom. mean
Computing trajectories
 Vt 1  Vt
S t  w
 Vt 1 Vt

 2 Vt 1 Vt

  (1  w)

 Vt 1  Vt






Then the “Greedy-Exchange” algorithm is used
to choose which points belong to which
trajectory.


Operates by creating trajectories and exchanging
points if it would increase total smoothness.
Local optimum.
Another approach
The Kalman filter is a probabilistic model
that combines noisy measurements with
the expected trajectory of the object. It
works even with occlusion.
 See
http://www.cs.unc.edu/~welch/kalman/ for
a start.
