3D Vision - Stereo Vision
Projection
  We
cannot
calculate
unique
3D
informa2on
with
a
single
camera
even
if
it
is
calibrated
because
mul2ple
3D
points
could
be
projected
to
the
same
image
pixel
point.
  However,
it
is
possible
to
unambiguously
project
3D
points
in
the
physical
world
to
points
in
the
image
in
pixel
coordinates.
2/14/11
ECEn
631
2
OpenCV Projection Function
void projectPoints( const Mat& objectPoints, const Mat& rvec, const Mat& tvec,
const Mat& cameraMatrix, const Mat& distCoeffs,
vector<Point2f>& imagePoints);
  Provide
a
list
of
3D
points
in
the
object’s
own
local
coordinate
system
and
rota2on
and
transla2on
matrices
that
relate
the
two
coordinate
systems
or
  a
list
of
3D
points
in
the
camera
coordinate
system
and
matrices
of
0s.
  The
output
is
a
list
of
points
in
the
image
pixel
coordinate
system.
2/14/11
ECEn
631
3
3D Pose Estimation
– a known object
  We
can
es2mate
the
posi2ons
of
known
objects
in
3D
  POSIT
‐
Pose
from
Orthography
and
Scaling
with
Itera2on
(not
very
accurate)
  Es2mate
the
posi2on
(T)
and
orienta2on
R
of
a
3D
object
whose
exact
dimensions
are
known.
  Need
at
least
four
points
on
the
image
that
are
the
projec2ons
of
at
least
four
non‐coplanar
points
on
the
object
surface.
2/14/11
ECEn
631
4
OpenCV POSIT Function
void cvPOSIT(CvPOSITObject* posit_object, CvPoint2D32f* image_points,
double focal_length, CvTermCriteria criteria, float* rotation_matrix,
float* translation_vector);
  Extra
coplanar
points
(more
than
3
points
on
a
planar
surface)
do
not
help
(actually
hurt)
the
algorithm’s
performance.
  Extra
non‐coplanar
points
will
help
the
algorithm.
  The
func2on
outputs
the
rota2on
and
transla2on
matrices.
They
can
be
used
to
determine
object
pose
(moving
object)
or
camera
pose
(moving
camera)
2/14/11
ECEn
631
5
OpenCV solvePnP Function
void solvePnP( const Mat& objectPoints, const Mat& imagePoints,
const Mat& cameraMatrix, const Mat& distCoeffs, Mat& rvec, Mat& tvec,
bool useExtrinsicGuess=false );
  Given
a
set
of
3D
object
points
in
the
object
coordinate
system
and
a
set
of
2D
corresponding
image
points
in
image
coordinates,
the
object
pose
can
be
es2mated
if
the
intrinsic
and
distor2on
parameters
are
available.
  The
func2on
outputs
the
rota2on
(3
rota2on
angles)
and
transla2on
vectors.
They
can
be
used
to
determine
object
pose
(moving
object)
or
camera
pose
(moving
camera)
2/14/11
ECEn
631
6
2/14/11
ECEn
631
7
Stereo Vision
In the absence of any noise or uncertainty, given two images taken
from calibrated cameras, one can in principle recover camera pose
and position of the points in space
Five Steps:
•  Camera Calibration to get intrinsic, extrinsic, distortion parameters
•  Undistort both images
•  Rectify both images
•  Correspondence of points from left and right images – disparity map
•  Reconstruction of 3D coordinates of the points using triangulation
2/14/11
ECEn
631
8
Simple Stereo System – real image
P(x,y,z)
h+x
h-x (x<0)
Cl
Cr
f
Pl
2/14/11
xl=0
h
ECEn
631
x=0
h
xr=0
Pr
9
pl
h+ x
=−
f
z
pr h − x
=
f
z
2hf
z( pr − pl ) = 2hf ⇒ z =
p r − pl
€
€
€
2/14/11
ECEn
631
10
Simple Stereo System – virtual image
P(x,y,z)
Z-f
Cl
f
xl
x’l
x’r
T+xl-xr
Ol
2/14/11
T
ECEn
631
xr
Cr
Or
11
Depth is inversely proportional to disparity
2/14/11
ECEn
631
12
Nonlinear relationship between depth and disparity
When object is far away
(small disparity) small
disparity differences
change the depth a lot.
When object is close (large disparity)
small disparity differences do not
change the depth by much.
High depth resolution.
2/14/11
ECEn
631
13
Mismatch
P(x,y,z)
P(x’,y’,z’)
Cl
xl
xr
x’l
x’r
Cr
f
Ol
T
Or
A mismatch will give a false result
2/14/11
ECEn
631
14
Canonical Configuration
C’
C
Optical Axes are parallel
The baseline is aligned to the horizontal coordinates axis
Horizontal lines (rows) of the two images are aligned perfectly.
2/14/11
ECEn
631
15
xl
yl
(cx, cy)
OpenCV use this
coordinate system.
Left camera is the
reference frame.
Z
ol
XC
YC
yr
Images are undistorted.
2/14/11
(cx, cy)
T
Pixel coordinates are
relative to the upper left
corner of the image.
or
ECEn
631
xr
16
General Configuration
General configuration is
more real.
Need to mathematically
align the two cameras
into one viewing plane
so that pixel rows are
aligned.
2/14/11
ol
T
or
ECEn
631
17
Canonical Configuration
Less overlapping area than general configuration
2/14/11
ECEn
631
18
General Configuration – small angle
2/14/11
ECEn
631
19
General Configuration – large angle
More overlapping area than canonical configuration
Rows are no longer aligned.
2/14/11
ECEn
631
20
Stereo Camera Parameters
Intrinsic parameters: – 2-D geometry
(Ox ,Oy) is the image center in pixels
Sx and Sy are the pixel size in pixels per mm or inch
f is the focal length in mm or inches
σ is the aspect ratio Sy/Sx (usually 1.0 for a square pixel camera)
o x , o y , f x = fs x , f y = fs y
Extrinsic parameters: – 3-D geometry
Rotation and translation between 2 cameras
T=[Tx Ty Tz] T
Distortion parameters:
Tied to 2-D geometry
k1, k2 , k 3 , p1, p2
2/14/11
ECEn
631
21
Correspondence → Disparity Map → Reconstruction
If stereo system geometry is known, 3-D scene can be
reconstructed.
What do we need? Intrinsic and extrinsic parameters
Intrinsic: o x , o y , f x = fs x , f y = fs y
Extrinsic: rigid transformation (rotation and translation)
between two camera frames (it was between C
and W for single camera)
€
2/14/11
ECEn
631
22