Stereo Matching Using Dynamic
Programming
Jim Rehg
CS 4495/7495 Computer Vision
Lecture 4
Mon Sept 2, 2002
Correspondence

It is fundamentally ambiguous, even with stereo
constraints
Ordering constraint…
…and its failure
2
J. M. Rehg © 2002
Search Over Correspondences
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Three cases:



Sequential – cost of match
Occluded – cost of no match
Disoccluded – cost of no match
3
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Start
Left scanline
Right scanline
Dis-occluded Pixels
Dynamic programming
yields the optimal path
through grid. This is
the best set of
matches that satisfy
the ordering constraint
End
4
J. M. Rehg © 2002
Dynamic Programming
i 1
1
12
 22
1
1
j2
i2
2
i 3
3
3
3
Ct 1
Ct
Ct 1
 32
2
2
Principle of Optimality for an n-stage assignment problem:
Ct ( j )  arg max i  ijCt 1 (i)
5
J. M. Rehg © 2002
Dynamic Programming
i 1
1
1
bt (2)  2
1
j2
i2
2
i 3
3
3
3
Ct 1
Ct
Ct 1
2
2
Principle of Optimality for an n-stage assignment problem:
Ct ( j )  arg max i  ijCt 1 (i)
6
J. M. Rehg © 2002
Dynamic Programming
i 1
1
13
1
1
i2
2
 23
2
2
i 3
3
 33
Ct 1
3
j 3
Ct
3
Ct 1
Principle of Optimality for an n-stage assignment problem:
Ct ( j )  arg max i  ijCt 1 (i)
7
J. M. Rehg © 2002
Dynamic Programming
i 1
1
1
1
i2
2
2
2
i 3
3
bt (3)  1
Ct 1
3
j 3
Ct
3
Ct 1
Principle of Optimality for an n-stage assignment problem:
Ct ( j )  arg max i  ijCt 1 (i)
8
J. M. Rehg © 2002
Dynamic Programming
i 1
i2
i 3
1
2
3
bt (1)  1
bt (2)  2
bt (3)  1
Ct 1
1
1
2
2
3
3
Ct
Ct 1
j 1
Principle of Optimality for an n-stage assignment problem:
Ct ( j )  arg max i  ijCt 1 (i)
9
J. M. Rehg © 2002
Dynamic Programming
i 1
1
1
1
i2
2
2
2
i 3
3
3
3
CT 2
CT 1
CT
sT*  2
CT*  CT (2)
Back-chaining recovers the optimal path and its cost:
sT*  arg max j CT ( j ), CT*  CT (sT* ), sT* 1  bT (sT* ), ...
10
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
11
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
12
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
13
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
14
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
15
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
16
J. M. Rehg © 2002
Stereo Matching with Dynamic
Programming
Occluded Pixels
Left scanline
Right scanline
Dis-occluded Pixels
Scan across grid
computing optimal cost
for each node given its
upper-left neighbors.
Backtrack from the
terminal to get the
optimal path.
Terminal
17
J. M. Rehg © 2002
Computing Correspondence

Another approach is to match edges rather than
windows of pixels:

Which method is better?


Edges tend to fail in dense texture (outdoors)
Correlation tends to fail in smooth featureless areas
18
J. M. Rehg © 2002
Computing Correspondences

Both methods fail for smooth surfaces

There is currently no good solution to the
correspondence problem
19
J. M. Rehg © 2002