Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Lecture 18:
Generalized Stereo
Key idea: Any two images showing an overlapping
view of the world can be treated as a stereo pair...
Generalized Stereo:
Epipolar Geometry
... we just have to figure out how the
two views are related.
Some of the most “beautiful” math in vision concerns
describing how multiple views are related, geometrically.
Robert Collins
CSE486, Penn State
Recall: Epipolar Constraint
Robert Collins
CSE486, Penn State
P
Important Stereo Vision Concept:
Given a point in the left image, we don’t have to
search the whole right image for a corresponding point.
Review : Simple Stereo System
z
y
Left camera
z
y
Right camera
x
The “epipolar constraint” reduces the search space
to a one-dimensional line.
Epipolar line
Tx
depth
x
Same Y Coord!
baseline
Equation relating
depth and disparity
disparity
Robert Collins
CSE486, Penn State
Review: Epipolar Constraint
Robert Collins
CSE486, Penn State
x1
y1
General Stereo
z1
C1
C2
O1
y2
z2
O2
x2
In general, the cameras may be related by
an arbitrary transformation (R,T)
Corresponding features are constrained to
lie along conjugate epipolar lines (on the
same row in the case of our simple setup).
Epipolar Matrix
In general, intrinsic camera parameters
may be different, and even unknown
Fundamental Matrix
1
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Epipolar Geometry
P
pl
EPIPOLAR
GEOMETRY
el
Ol
pr
er
Or
Epipoles:
Epipoles:
• el: left image of Or
• er: right image of Ol
Epipolar plane:
• Three points: Ol,Or, and P define an epipolar plane
Epipolar lines and epipolar constraint:
• Intersections of epipolar plane with the image planes
• Corresponding points are on “conjugate”
conjugate” epipolar lines
The following slides are from Dr.Camps, PSU
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
BORING!!!
EPIPOLAR
GEOMETRY
Let’s try again...
Robert Collins
CSE486, Penn State
Epipolar Geometry
Robert Collins
CSE486, Penn State
A Visualization
Would would Pinhead’s
eye look like close up?
Rays to Points in Scene
Tie threads on
to the pins
and connect
focal point
to scene
points
Now what would this look like to a second observer?
2
Robert Collins
CSE486, Penn State
Rays Seen from Second Observer
Robert Collins
CSE486, Penn State
Rays Seen by the First Viewer
epipolar lines
Q
a
T
I
r
e
u
F
ic
k
(
L
Z
W
)
e
e
d
e
d
t
F
n
T
im
e
™
a
d
o
n
d
e
c
o
s
e
e
epipolar lines
a
m
t
p
r
h
is
e
s
s
o
r
p
ic
t
u
r
e
.
epipole
epipole
Image 2
Robert Collins
CSE486, Penn State
Image 1
Epipolar Geometry
image1
image1
Epipole : location of cam1
as seen by cam2.
Epipolar Geometry
image1
Epipolar Geometry
image 2
Epipole : location of cam2
as seen by cam1.
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
image 2
Corresponding points
lie on conjugate epipolar lines
Robert Collins
CSE486, Penn State
Epipole not Necessarily in Image
image 2
epipolar lines
Conjugate epipolar lines induce
a generalized 1D “scan-line” ordering
on the images (analogous to traditional
scan line ordering of rows in an image)
Q
a
T
I
r
e
u
F
ic
k
(
L
Z
W
)
e
e
d
e
d
t
F
n
T
im
e
™
d
o
a
n
d
e
c
o
s
e
e
a
m
t
p
r
h
is
e
s
s
o
r
p
ic
t
u
r
e
.
epipole
Image 2
3
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Epipolar Geometry
Epipolar Constraint:
P
pl
Ol
pl
pr
er
el
P
Or
pr
er
el
Ol
Or
Epipoles:
Epipoles:
• el: left image of Or
• er: right image of Ol
Given Epipoles:
Epipoles:
• el: left image of Or
• er: right image of Ol
Epipolar plane:
• Three points: Ol,Or, and P define an epipolar plane
Given pl:
•consider its epipolar line: pl el
•find epipolar plane: Ol,pl,el
•intersect the epipolar plane with the right image plane
•search for pr on the right epipolar line
Epipolar lines and epipolar constraint:
• Intersections of epipolar plane with the image planes
• Corresponding points are on “conjugate”
conjugate” epipolar lines
The following slides are from Dr.Camps, PSU
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Essential Matrix
Pl
P
pl
el
Ol
T
Essential Matrix
Pl
Pr
pr
er
P
pl
Or
el
Ol
T
Pr
pr
er
Or
Does this look familiar? Recall world to camera
transformation by (R,T). Here, we are transforming
from camera to camera.
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Essential Matrix
Epipolar constraint: Pl, T and Pl - T are coplanar:
P
Pl
Pr
pl
Ol
el
T
pr
er
Essential Matrix
Epipolar constraint: Pl, T and Pl - T are coplanar:
P
Pl
Pr
pl
Or
Ol
el
T
pr
er
Or
4
Robert Collins
CSE486, Penn State
Vector Product as a Matrix
Multiplication
Robert Collins
CSE486, Penn State
Essential Matrix
Epipolar constraint: Pl, T and Pl - T are coplanar:
P
Pl
Pr
pl
el
Ol
T
pr
er
Or
S has rank 2 ; it depends only on T
Robert Collins
CSE486, Penn State
Robert Collins
CSE486, Penn State
Essential Matrix
Essential Matrix Properties
Epipolar constraint: Pl, T and Pl - T are coplanar:
P
Pl
Pr
pl
Ol
Essential Matrix:
el
T
pr
er
Or
• has rank 2
• depends only on the EXTRINSIC Parameters (R & T)
We will discuss more of the wonderful
properties of this matrix next time...
5