Computer Vision: Calibration and Reconstruction
Raquel Urtasun
TTI Chicago
Feb 7, 2013
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
1 / 56
What did we see in class last week?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
2 / 56
Panoramas
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
3 / 56
Today’s Readings
Chapter 6 and 11 of Szeliski’s book
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
4 / 56
Let’s look at camera calibration
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
5 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Focal length
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Focal length
Different scaling factors for row pixels and column pixels
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Focal length
Different scaling factors for row pixels and column pixels
Skew factor
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Focal length
Different scaling factors for row pixels and column pixels
Skew factor
Lens distortion (pin-cushion effect)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Camera Calibration
Find the quantities internal to the camera that affect the imaging process as well
as the position of the camera with respect to the world
Rotation and translation
Position of image center in the image
Focal length
Different scaling factors for row pixels and column pixels
Skew factor
Lens distortion (pin-cushion effect)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
6 / 56
Why do we need calibration?
Have good reconstruction
Interact with the 3D world
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
7 / 56
Camera and Calibration Target
Most methods assume that we have a known 3D target in the scene
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
8 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
We know positions of pattern corners only with respect to a coordinate
system of the target
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
We know positions of pattern corners only with respect to a coordinate
system of the target
We position camera in front of target and find images of corners
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
We know positions of pattern corners only with respect to a coordinate
system of the target
We position camera in front of target and find images of corners
We obtain equations that describe imaging and contain internal parameters
of camera
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
We know positions of pattern corners only with respect to a coordinate
system of the target
We position camera in front of target and find images of corners
We obtain equations that describe imaging and contain internal parameters
of camera
We also find position and orientation of camera with respect to target
(camera pose)
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Most common used Procedure
Many algorithms!
Calibration target: 2 planes at right angle with checkerboard patterns (Tsai
grid)
We know positions of pattern corners only with respect to a coordinate
system of the target
We position camera in front of target and find images of corners
We obtain equations that describe imaging and contain internal parameters
of camera
We also find position and orientation of camera with respect to target
(camera pose)
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
9 / 56
Obtaining 2D-3D correspondences
Canny edge detection
Straight line fitting to detect linked edges. How?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
10 / 56
Obtaining 2D-3D correspondences
Canny edge detection
Straight line fitting to detect linked edges. How?
Intersect the lines to obtain the image corners
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
10 / 56
Obtaining 2D-3D correspondences
Canny edge detection
Straight line fitting to detect linked edges. How?
Intersect the lines to obtain the image corners
Matching image corners and 3D target checkerboard corners (sometimes
using manual interaction)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
10 / 56
Obtaining 2D-3D correspondences
Canny edge detection
Straight line fitting to detect linked edges. How?
Intersect the lines to obtain the image corners
Matching image corners and 3D target checkerboard corners (sometimes
using manual interaction)
We get pairs (image point)–(world point), (xi , yi ) → (Xi , Yi , Zi ).
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
10 / 56
Obtaining 2D-3D correspondences
Canny edge detection
Straight line fitting to detect linked edges. How?
Intersect the lines to obtain the image corners
Matching image corners and 3D target checkerboard corners (sometimes
using manual interaction)
We get pairs (image point)–(world point), (xi , yi ) → (Xi , Yi , Zi ).
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
10 / 56
Calibration
Estimate matrix P using 2D-3D correspondences
Estimate K and (R, t) from P
P = K · R · [I3×3 | t]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
11 / 56
Calibration
Estimate matrix P using 2D-3D correspondences
Estimate K and (R, t) from P
P = K · R · [I3×3 | t]
Left 3x3 submatrix of P is product of upper-triangular matrix and
orthogonal matrix
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
11 / 56
Calibration
Estimate matrix P using 2D-3D correspondences
Estimate K and (R, t) from P
P = K · R · [I3×3 | t]
Left 3x3 submatrix of P is product of upper-triangular matrix and
orthogonal matrix
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
11 / 56
Inherent Constraints
The visual angle between any pair of 2D points must be the same as the
angle between their corresponding 3D points.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
12 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
This is called the Direct Linear Transform (DLT)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
This is called the Direct Linear Transform (DLT)
How many unknowns? how many correspondences?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
This is called the Direct Linear Transform (DLT)
How many unknowns? how many correspondences?
11 or 12 unknowns, 6 correspondences
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
This is called the Direct Linear Transform (DLT)
How many unknowns? how many correspondences?
11 or 12 unknowns, 6 correspondences
Are we done?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Direct Linear Transform
Simplest to form a set of linear equations (analog to the 2D case)
xi
=
yi
=
p00 Xi
p20 Xi
p10 Xi
p20 Xi
+ p01 Yi
+ p21 Yi
+ p11 Yi
+ p21 Yi
+ p02 Zi
+ p22 Yi
+ p12 Zi
+ p22 Yi
+ p03
+ p23
+ p13
+ p23
with (xi , yi ), the measured 2D points, and (Xi , Yi , Zi ) are known 3D
locations
This can be solve in a linear fashion for P. How?
Similar to the homography case.
This is called the Direct Linear Transform (DLT)
How many unknowns? how many correspondences?
11 or 12 unknowns, 6 correspondences
Are we done?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
13 / 56
Finding Camera Translation
Once P is recovered, how do I obtain the other matrices?
t is the null vector of matrix P as
Pt = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
14 / 56
Finding Camera Translation
Once P is recovered, how do I obtain the other matrices?
t is the null vector of matrix P as
Pt = 0
t is the unit singular vector of P corresponding to the smallest singular value
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
14 / 56
Finding Camera Translation
Once P is recovered, how do I obtain the other matrices?
t is the null vector of matrix P as
Pt = 0
t is the unit singular vector of P corresponding to the smallest singular value
The last column of V, where
P = UDVT
is the SVD of P
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
14 / 56
Finding Camera Translation
Once P is recovered, how do I obtain the other matrices?
t is the null vector of matrix P as
Pt = 0
t is the unit singular vector of P corresponding to the smallest singular value
The last column of V, where
P = UDVT
is the SVD of P
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
14 / 56
Finding Camera Orientation and Internal Parameters
Left 3 × 3 submatrix M of P is of form
M = KR
where K is an upper triangular matrix, and R is an orthogonal matrix
Any non-singular square matrix M can be decomposed into the product of
an upper-triangular matrix K and an orthogonal matrix R using the RQ
factorization
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
15 / 56
Finding Camera Orientation and Internal Parameters
Left 3 × 3 submatrix M of P is of form
M = KR
where K is an upper triangular matrix, and R is an orthogonal matrix
Any non-singular square matrix M can be decomposed into the product of
an upper-triangular matrix K and an orthogonal matrix R using the RQ
factorization
Similar to QR factorization but order of 2 matrices is reversed
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
15 / 56
Finding Camera Orientation and Internal Parameters
Left 3 × 3 submatrix M of P is of form
M = KR
where K is an upper triangular matrix, and R is an orthogonal matrix
Any non-singular square matrix M can be decomposed into the product of
an upper-triangular matrix K and an orthogonal matrix R using the RQ
factorization
Similar to QR factorization but order of 2 matrices is reversed
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
15 / 56
RQ Factorization
Define the matrices


 0
1 0 0
c
Rx = 0 c −s  , Ry =  0
−s 0
0 s c

 00
c
0 s0
1 0  , Rz =  s 00
0 c0
0
−s 00
c 00
0

0
0
1
We can compute
c =−
Raquel Urtasun (TTI-C)
M33
2 + M 2 )1/2
(M32
33
Computer Vision
s=
M32
2 + M 2 )1/2
(M32
33
Feb 7, 2013
16 / 56
RQ Factorization
Define the matrices


 0
1 0 0
c
Rx = 0 c −s  , Ry =  0
−s 0
0 s c

 00
c
0 s0
1 0  , Rz =  s 00
0 c0
0
−s 00
c 00
0

0
0
1
We can compute
c =−
M33
2 + M 2 )1/2
(M32
33
s=
M32
2 + M 2 )1/2
(M32
33
Multiply M by Rx . The resulting term at (3, 2) is zero because of the values
selected for c and s
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
16 / 56
RQ Factorization
Define the matrices


 0
1 0 0
c
Rx = 0 c −s  , Ry =  0
−s 0
0 s c

 00
c
0 s0
1 0  , Rz =  s 00
0 c0
0
−s 00
c 00
0

0
0
1
We can compute
c =−
M33
2 + M 2 )1/2
(M32
33
s=
M32
2 + M 2 )1/2
(M32
33
Multiply M by Rx . The resulting term at (3, 2) is zero because of the values
selected for c and s
Multiply the resulting matrix by Ry , after selecting c and s so that the
resulting term at position (3, 1) is set to zero
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
16 / 56
RQ Factorization
Define the matrices


 0
1 0 0
c
Rx = 0 c −s  , Ry =  0
−s 0
0 s c

 00
c
0 s0
1 0  , Rz =  s 00
0 c0
0
−s 00
c 00
0

0
0
1
We can compute
c =−
M33
2 + M 2 )1/2
(M32
33
s=
M32
2 + M 2 )1/2
(M32
33
Multiply M by Rx . The resulting term at (3, 2) is zero because of the values
selected for c and s
Multiply the resulting matrix by Ry , after selecting c and s so that the
resulting term at position (3, 1) is set to zero
Multiply the resulting matrix by Rz , after selecting c and s so that the
resulting term at position (2, 1) is set to zero
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
16 / 56
RQ Factorization
Define the matrices


 0
1 0 0
c
Rx = 0 c −s  , Ry =  0
−s 0
0 s c

 00
c
0 s0
1 0  , Rz =  s 00
0 c0
0
−s 00
c 00
0

0
0
1
We can compute
c =−
M33
2 + M 2 )1/2
(M32
33
s=
M32
2 + M 2 )1/2
(M32
33
Multiply M by Rx . The resulting term at (3, 2) is zero because of the values
selected for c and s
Multiply the resulting matrix by Ry , after selecting c and s so that the
resulting term at position (3, 1) is set to zero
Multiply the resulting matrix by Rz , after selecting c and s so that the
resulting term at position (2, 1) is set to zero
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
16 / 56
RQ Factorization
Why does this algorithm work?
MRx Ry Rz = K
Thus we have
T T
M = KRT
z Ry Rx = KR
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
17 / 56
RQ Factorization
Why does this algorithm work?
MRx Ry Rz = K
Thus we have
T T
M = KRT
z Ry Rx = KR
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
17 / 56
Improved computation of P
We have equations involving homgeneous coordinates
xi = PXi
Thus xi and Xi just have to be proportional
xi × PXi = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
18 / 56
Improved computation of P
We have equations involving homgeneous coordinates
xi = PXi
Thus xi and Xi just have to be proportional
xi × PXi = 0
T
T
Let pT
1 , p2 , p3 be the row vectors of P
 T 
p1 Xi

PXi = pT
2 Xi
pT
X
3 i
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
18 / 56
Improved computation of P
We have equations involving homgeneous coordinates
xi = PXi
Thus xi and Xi just have to be proportional
xi × PXi = 0
T
T
Let pT
1 , p2 , p3 be the row vectors of P
 T 
p1 Xi

PXi = pT
2 Xi
pT
X
3 i
Therefore
Raquel Urtasun (TTI-C)

T
vi pT
3 Xi − w i p 2 Xi
T

xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

Computer Vision
Feb 7, 2013
18 / 56
Improved computation of P
We have equations involving homgeneous coordinates
xi = PXi
Thus xi and Xi just have to be proportional
xi × PXi = 0
T
T
Let pT
1 , p2 , p3 be the row vectors of P
 T 
p1 Xi

PXi = pT
2 Xi
pT
X
3 i
Therefore
Raquel Urtasun (TTI-C)

T
vi pT
3 Xi − w i p 2 Xi
T

xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

Computer Vision
Feb 7, 2013
18 / 56
Improved computation of P
We have

T
vi pT
3 Xi − wi p2 Xi
T
=0
xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

We can thus write

04
 w i XT
i
−vi XT
i
Raquel Urtasun (TTI-C)
−wi XT
i
04
ui XT
i
 
vi X T
p1
i


p2  = 0
−ui XT
i
p3
04
Computer Vision
Feb 7, 2013
19 / 56
Improved computation of P
We have

T
vi pT
3 Xi − wi p2 Xi
T
=0
xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

We can thus write

04
 w i XT
i
−vi XT
i
−wi XT
i
04
ui XT
i
 
vi X T
p1
i


p2  = 0
−ui XT
i
p3
04
Third row can be obtained from sum of ui times first row −vi times second
row
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
19 / 56
Improved computation of P
We have

T
vi pT
3 Xi − wi p2 Xi
T
=0
xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

We can thus write

04
 w i XT
i
−vi XT
i
−wi XT
i
04
ui XT
i
 
vi X T
p1
i


p2  = 0
−ui XT
i
p3
04
Third row can be obtained from sum of ui times first row −vi times second
row
2 independent equations in 11 unknowns (ignoring scale)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
19 / 56
Improved computation of P
We have

T
vi pT
3 Xi − wi p2 Xi
T
=0
xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

We can thus write

04
 w i XT
i
−vi XT
i
−wi XT
i
04
ui XT
i
 
vi X T
p1
i


p2  = 0
−ui XT
i
p3
04
Third row can be obtained from sum of ui times first row −vi times second
row
2 independent equations in 11 unknowns (ignoring scale)
With 6 correspondences, we get enough equations to compute matrix P
Ap = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
19 / 56
Improved computation of P
We have

T
vi pT
3 Xi − wi p2 Xi
T
=0
xi × PXi = wi pT
3 Xi − ui p2 Xi
T
T
ui p3 Xi − vi p2 Xi

We can thus write

04
 w i XT
i
−vi XT
i
−wi XT
i
04
ui XT
i
 
vi X T
p1
i


p2  = 0
−ui XT
i
p3
04
Third row can be obtained from sum of ui times first row −vi times second
row
2 independent equations in 11 unknowns (ignoring scale)
With 6 correspondences, we get enough equations to compute matrix P
Ap = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
19 / 56
Solving the system
Ap = 0
Minimize ||Ap|| with the constraint ||p|| = 1
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
20 / 56
Solving the system
Ap = 0
Minimize ||Ap|| with the constraint ||p|| = 1
P is the unit singular vector of A corresponding to the smallest singular value
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
20 / 56
Solving the system
Ap = 0
Minimize ||Ap|| with the constraint ||p|| = 1
P is the unit singular vector of A corresponding to the smallest singular value
The last column of V, where
A = UDVT
is the SVD of A
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
20 / 56
Solving the system
Ap = 0
Minimize ||Ap|| with the constraint ||p|| = 1
P is the unit singular vector of A corresponding to the smallest singular value
The last column of V, where
A = UDVT
is the SVD of A
Called Direct Linear Transformation (DLT)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
20 / 56
Solving the system
Ap = 0
Minimize ||Ap|| with the constraint ||p|| = 1
P is the unit singular vector of A corresponding to the smallest singular value
The last column of V, where
A = UDVT
is the SVD of A
Called Direct Linear Transformation (DLT)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
20 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Find P using DLT
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Find P using DLT
P
Use as initialization for nonlinear minimization i ρ(xi , PXi ), with ρ a
robust estimator
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Find P using DLT
P
Use as initialization for nonlinear minimization i ρ(xi , PXi ), with ρ a
robust estimator
Use Levenberg-Marquardt iterative minimization
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Find P using DLT
P
Use as initialization for nonlinear minimization i ρ(xi , PXi ), with ρ a
robust estimator
Use Levenberg-Marquardt iterative minimization
See 6.2.2. in Szeliski’s book
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Improving the P estimation
In most applications we have prior knowledge about some of the parameters
of K, e.g., pixels are squared, skew is small, optical center near the center of
the image
Use this constraints and frame the problem as a minimization
Find P using DLT
P
Use as initialization for nonlinear minimization i ρ(xi , PXi ), with ρ a
robust estimator
Use Levenberg-Marquardt iterative minimization
See 6.2.2. in Szeliski’s book
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
21 / 56
Radial Distorsion
We have assumed that lines are imaged as lines
Significant error for cheap optics and for short focal lengths
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
22 / 56
Radial Distorsion Correction
We can write the correction as
2
xc − x0
= L(r )(x − x0 )
yc − y0
= L(r )(y − y0 )
4
with L(r ) = 1 + κ1 r + κ2 r and r 2 = (x − x0 )2 + (y − y0 )2
We thus minimize the following function
X
f (κ1 , κ2 ) =
(xi0 − xci )2 + (yi0 − yci )2
i
using lines known to be straight, with (x 0 , y 0 ) the radial projection of (x, y )
on straight line
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
23 / 56
Radial Distorsion Correction
We can write the correction as
2
xc − x0
= L(r )(x − x0 )
yc − y0
= L(r )(y − y0 )
4
with L(r ) = 1 + κ1 r + κ2 r and r 2 = (x − x0 )2 + (y − y0 )2
We thus minimize the following function
X
f (κ1 , κ2 ) =
(xi0 − xci )2 + (yi0 − yci )2
i
using lines known to be straight, with (x 0 , y 0 ) the radial projection of (x, y )
on straight line
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
23 / 56
Let’s look at 3D alignment
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
24 / 56
Motivation
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
25 / 56
3D Alignment
Before we were looking at aligning 2D points, or 2D to 3D points.
Now let’s do it in 3D
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
26 / 56
3D Alignment
Before we were looking at aligning 2D points, or 2D to 3D points.
Now let’s do it in 3D
Given two corresponding point sets {xi , x0i }, in the case of rigid (Euclidean)
motion we want R and t that minimizes
X
ER3D =
||x0i − Rxi − t||22
i
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
26 / 56
3D Alignment
Before we were looking at aligning 2D points, or 2D to 3D points.
Now let’s do it in 3D
Given two corresponding point sets {xi , x0i }, in the case of rigid (Euclidean)
motion we want R and t that minimizes
X
ER3D =
||x0i − Rxi − t||22
i
This is call the absolute orientation problem
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
26 / 56
3D Alignment
Before we were looking at aligning 2D points, or 2D to 3D points.
Now let’s do it in 3D
Given two corresponding point sets {xi , x0i }, in the case of rigid (Euclidean)
motion we want R and t that minimizes
X
ER3D =
||x0i − Rxi − t||22
i
This is call the absolute orientation problem
Is this easy to do?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
26 / 56
3D Alignment
Before we were looking at aligning 2D points, or 2D to 3D points.
Now let’s do it in 3D
Given two corresponding point sets {xi , x0i }, in the case of rigid (Euclidean)
motion we want R and t that minimizes
X
ER3D =
||x0i − Rxi − t||22
i
This is call the absolute orientation problem
Is this easy to do?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
26 / 56
Key Idea
If the correct correspondences are known, the correct relative
rotation/translation can be calculated in closed form
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
27 / 56
Center of Mass
The centroids of the two point clouds c and c 0 can be used to estimate the
translation
N
N
1 X 0
1 X
xi
µ0 =
xi
µ=
N
N
i=1
i=1
Subtract the corresponding center of mass from every point in the two sets,
Raquel Urtasun (TTI-C)
x̄i
= xi − µ
x̄0i
= x0i − µ0
Computer Vision
Feb 7, 2013
28 / 56
Center of Mass
The centroids of the two point clouds c and c 0 can be used to estimate the
translation
N
N
1 X 0
1 X
xi
µ0 =
xi
µ=
N
N
i=1
i=1
Subtract the corresponding center of mass from every point in the two sets,
Raquel Urtasun (TTI-C)
x̄i
= xi − µ
x̄0i
= x0i − µ0
Computer Vision
Feb 7, 2013
28 / 56
Solving for the Rotation
Let W =
PN
i=1
x̄0i x̄T
i
We can compute the SVD of W
W = USVT
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
29 / 56
Solving for the Rotation
Let W =
PN
i=1
x̄0i x̄T
i
We can compute the SVD of W
W = USVT
If the rank is 3 we have a unique solution
Raquel Urtasun (TTI-C)
R =
UVT
t =
µ − Rµ0
Computer Vision
Feb 7, 2013
29 / 56
Solving for the Rotation
Let W =
PN
i=1
x̄0i x̄T
i
We can compute the SVD of W
W = USVT
If the rank is 3 we have a unique solution
R =
UVT
t =
µ − Rµ0
What if we don’t have correspondences?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
29 / 56
Solving for the Rotation
Let W =
PN
i=1
x̄0i x̄T
i
We can compute the SVD of W
W = USVT
If the rank is 3 we have a unique solution
R =
UVT
t =
µ − Rµ0
What if we don’t have correspondences?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
29 / 56
ICP
If the correspondences are not know, it is not possible to obtain the global
solution
Iterated Closest Points (ICP) [Besl & McKay 92]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
30 / 56
ICP
If the correspondences are not know, it is not possible to obtain the global
solution
Iterated Closest Points (ICP) [Besl & McKay 92]
Iterative algorithm that alternates between solving for correspondences and
solving for (R, t)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
30 / 56
ICP
If the correspondences are not know, it is not possible to obtain the global
solution
Iterated Closest Points (ICP) [Besl & McKay 92]
Iterative algorithm that alternates between solving for correspondences and
solving for (R, t)
Does it remained you of any algorithm you have seen before?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
30 / 56
ICP
If the correspondences are not know, it is not possible to obtain the global
solution
Iterated Closest Points (ICP) [Besl & McKay 92]
Iterative algorithm that alternates between solving for correspondences and
solving for (R, t)
Does it remained you of any algorithm you have seen before?
Converges to right solution if starting positions are ”close enough”
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
30 / 56
ICP
If the correspondences are not know, it is not possible to obtain the global
solution
Iterated Closest Points (ICP) [Besl & McKay 92]
Iterative algorithm that alternates between solving for correspondences and
solving for (R, t)
Does it remained you of any algorithm you have seen before?
Converges to right solution if starting positions are ”close enough”
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
30 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
31 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
31 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
31 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
31 / 56
Performance of Variants
Speed
Stability (local minima)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
32 / 56
Performance of Variants
Speed
Stability (local minima)
Tolerance wrt. noise and/or outliers
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
32 / 56
Performance of Variants
Speed
Stability (local minima)
Tolerance wrt. noise and/or outliers
Basin of convergence (maximum initial misalignment)
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
32 / 56
Performance of Variants
Speed
Stability (local minima)
Tolerance wrt. noise and/or outliers
Basin of convergence (maximum initial misalignment)
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
32 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
33 / 56
ICP Variants
Use all points
Uniform sub-sampling
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
34 / 56
ICP Variants
Use all points
Uniform sub-sampling
Random sampling
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
34 / 56
ICP Variants
Use all points
Uniform sub-sampling
Random sampling
Normal-space sampling distributed as uniformly as possible
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
34 / 56
ICP Variants
Use all points
Uniform sub-sampling
Random sampling
Normal-space sampling distributed as uniformly as possible
Feature based Sampling
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
34 / 56
ICP Variants
Use all points
Uniform sub-sampling
Random sampling
Normal-space sampling distributed as uniformly as possible
Feature based Sampling
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
34 / 56
Normal-based sampling
Ensure that samples have normals distributed as uniformly as possible
better for mostly-smooth areas with sparse features
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
35 / 56
Feature-based sampling
try to find important points
decrease the number of correspondences
higher efficiency and higher accuracy
requires preprocessing
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
36 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
37 / 56
Selection vs. Weighting
Could achieve same effect with weighting
Hard to guarantee that enough samples of important features except at high
sampling rates
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
38 / 56
Selection vs. Weighting
Could achieve same effect with weighting
Hard to guarantee that enough samples of important features except at high
sampling rates
Weighting strategies turned out to be dependent on the data.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
38 / 56
Selection vs. Weighting
Could achieve same effect with weighting
Hard to guarantee that enough samples of important features except at high
sampling rates
Weighting strategies turned out to be dependent on the data.
Preprocessing / run-time cost tradeoff (how to find the correct weights?)
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
38 / 56
Selection vs. Weighting
Could achieve same effect with weighting
Hard to guarantee that enough samples of important features except at high
sampling rates
Weighting strategies turned out to be dependent on the data.
Preprocessing / run-time cost tradeoff (how to find the correct weights?)
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
38 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
39 / 56
Data association
Has greatest effect on convergence and speed
Closest point: stable, but slow and requires preprocessing
Normal shooting: slightly better than closest point for smooth structures,
worse for noisy or complex structures
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
40 / 56
Data association
Has greatest effect on convergence and speed
Closest point: stable, but slow and requires preprocessing
Normal shooting: slightly better than closest point for smooth structures,
worse for noisy or complex structures
Point to plane: lets flat regions slide along each other
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
40 / 56
Data association
Has greatest effect on convergence and speed
Closest point: stable, but slow and requires preprocessing
Normal shooting: slightly better than closest point for smooth structures,
worse for noisy or complex structures
Point to plane: lets flat regions slide along each other
Projection: much faster but slightly worst alignments
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
40 / 56
Data association
Has greatest effect on convergence and speed
Closest point: stable, but slow and requires preprocessing
Normal shooting: slightly better than closest point for smooth structures,
worse for noisy or complex structures
Point to plane: lets flat regions slide along each other
Projection: much faster but slightly worst alignments
Closest compatible point: normals, colors
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
40 / 56
Data association
Has greatest effect on convergence and speed
Closest point: stable, but slow and requires preprocessing
Normal shooting: slightly better than closest point for smooth structures,
worse for noisy or complex structures
Point to plane: lets flat regions slide along each other
Projection: much faster but slightly worst alignments
Closest compatible point: normals, colors
[Source: W. Burgard]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
40 / 56
ICP Variants
Point subsets (from one or both point sets)
Weighting the correspondences
Data association
Rejecting certain (outlier) point pairs, e.g., Trimmed ICP rejects a %
Where do these 3D points come from?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
41 / 56
Let’s look into stereo reconstruction
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
42 / 56
Stereo
Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923
[Source: N. Snavely]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
43 / 56
Stereo
Stereo matching is the process of taking two or more images and
estimating a 3D model of the scene by finding matching pixels in the
images and converting their 2D positions into 3D depths
We perceived depth based on the difference in appearance of the right and
left eye.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
44 / 56
Stereo
Stereo matching is the process of taking two or more images and
estimating a 3D model of the scene by finding matching pixels in the
images and converting their 2D positions into 3D depths
We perceived depth based on the difference in appearance of the right and
left eye.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
44 / 56
Stereo
Given two images from different viewpoints
The depth is proportional to the inverse of the disparity
How can we compute the depth of each point in the image?
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
45 / 56
Stereo
Given two images from different viewpoints
The depth is proportional to the inverse of the disparity
How can we compute the depth of each point in the image?
Based on how much each pixel moves between the two images
[Source: N. Snavely]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
45 / 56
Stereo
Given two images from different viewpoints
The depth is proportional to the inverse of the disparity
How can we compute the depth of each point in the image?
Based on how much each pixel moves between the two images
[Source: N. Snavely]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
45 / 56
Epipolar Geometry
Pixel in one image x0 projects to an epipolar line segment in the other
image
The segment is bounded at one end by the projection of the original viewing
ray at infinity p∞ and at the other end by the projection of the original
camera center c0 into the second camera, which is known as the epipole e1 .
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
46 / 56
Epipolar Geometry
Pixel in one image x0 projects to an epipolar line segment in the other
image
The segment is bounded at one end by the projection of the original viewing
ray at infinity p∞ and at the other end by the projection of the original
camera center c0 into the second camera, which is known as the epipole e1 .
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
46 / 56
Epipolar Geometry
If we project the epipolar line in the second image back into the first, we get
another line (segment), this time bounded by the other corresponding
epipole e0
Extending both line segments to infinity, we get a pair of corresponding
epipolar lines, which are the intersection of the two image planes with the
epipolar plane that passes through both camera centers c0 and c1 as well
as the point of interest p
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
47 / 56
Epipolar Geometry
If we project the epipolar line in the second image back into the first, we get
another line (segment), this time bounded by the other corresponding
epipole e0
Extending both line segments to infinity, we get a pair of corresponding
epipolar lines, which are the intersection of the two image planes with the
epipolar plane that passes through both camera centers c0 and c1 as well
as the point of interest p
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
47 / 56
Rectification
The epipolar geometry depends on the relative pose and calibration of the
cameras
This can be computed using the fundamental matrix
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
48 / 56
Rectification
The epipolar geometry depends on the relative pose and calibration of the
cameras
This can be computed using the fundamental matrix
Once this is computed, we can use the epipolar lines to restrict the search
space to a 1D search
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
48 / 56
Rectification
The epipolar geometry depends on the relative pose and calibration of the
cameras
This can be computed using the fundamental matrix
Once this is computed, we can use the epipolar lines to restrict the search
space to a 1D search
Rectification, the process of transforming the image so that the search is
along horizontal line
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
48 / 56
Rectification
The epipolar geometry depends on the relative pose and calibration of the
cameras
This can be computed using the fundamental matrix
Once this is computed, we can use the epipolar lines to restrict the search
space to a 1D search
Rectification, the process of transforming the image so that the search is
along horizontal line
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
48 / 56
Epipolar Geometry
!"#"$%&'(
%#)!*(
!"#$%&#'%
!"($%&#'%
!"#$%&'()*$+',-./)0$12$'$,./)32$4#/%5#6-'3$-/'6*3'76($+'&)/'$
8'!+,-!.(*-)/)#$,'%/9$
The disparity for pixel (x1 , y1 ) is (x2 − x1 ) if the images are rectified
This is a one dimensional search for each pixel
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
49 / 56
Epipolar Geometry
!"#"$%&'(
%#)!*(
!"#$%&#'%
!"($%&#'%
!"#$%&'()*$+',-./)0$12$'$,./)32$4#/%5#6-'3$-/'6*3'76($+'&)/'$
8'!+,-!.(*-)/)#$,'%/9$
The disparity for pixel (x1 , y1 ) is (x2 − x1 ) if the images are rectified
This is a one dimensional search for each pixel
Very challenging to estimate the correspondences
[Source: N. Snavely]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
49 / 56
Epipolar Geometry
!"#"$%&'(
%#)!*(
!"#$%&#'%
!"($%&#'%
!"#$%&'()*$+',-./)0$12$'$,./)32$4#/%5#6-'3$-/'6*3'76($+'&)/'$
8'!+,-!.(*-)/)#$,'%/9$
The disparity for pixel (x1 , y1 ) is (x2 − x1 ) if the images are rectified
This is a one dimensional search for each pixel
Very challenging to estimate the correspondences
[Source: N. Snavely]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
49 / 56
Fundamental Matrix
Projective geometry depends only on the cameras internal parameters and
relative pose of cameras
Fundamental matrix F encapsulates this geometry
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
50 / 56
Fundamental Matrix
Projective geometry depends only on the cameras internal parameters and
relative pose of cameras
Fundamental matrix F encapsulates this geometry
For any pair of points correspoding in both images
xT
0 Fx1 = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
50 / 56
Fundamental Matrix
Projective geometry depends only on the cameras internal parameters and
relative pose of cameras
Fundamental matrix F encapsulates this geometry
For any pair of points correspoding in both images
xT
0 Fx1 = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
50 / 56
Epipolar Plane
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
51 / 56
Pensils of Epipolar Lines
[Source: Ramani]
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
52 / 56
Computation of Fundamental Matrix
F can be computed from correspondences between image points alone
No knowledge of camera internal parameters required
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
53 / 56
Computation of Fundamental Matrix
F can be computed from correspondences between image points alone
No knowledge of camera internal parameters required
No knowledge of relative pose required
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
53 / 56
Computation of Fundamental Matrix
F can be computed from correspondences between image points alone
No knowledge of camera internal parameters required
No knowledge of relative pose required
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
53 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Find epipole e0 as image of C in camera P0 , e0 = P0 C
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Find epipole e0 as image of C in camera P0 , e0 = P0 C
Find epipolar line l0 from e0 to x0 in P0 as function of x
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Find epipole e0 as image of C in camera P0 , e0 = P0 C
Find epipolar line l0 from e0 to x0 in P0 as function of x
The fundamental matrix F is defined l0 = Fx
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Find epipole e0 as image of C in camera P0 , e0 = P0 C
Find epipolar line l0 from e0 to x0 in P0 as function of x
The fundamental matrix F is defined l0 = Fx
T
x0 belongs to l0 , so x0 l0 = 0, so
T
x0 Fx = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Fundamental Matrix and Projective Geometry
Take x in camera P and find scene point X on ray of x in camera P
Find the image x0 of X in camera P0
Find epipole e0 as image of C in camera P0 , e0 = P0 C
Find epipolar line l0 from e0 to x0 in P0 as function of x
The fundamental matrix F is defined l0 = Fx
T
x0 belongs to l0 , so x0 l0 = 0, so
T
x0 Fx = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
54 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Image of this point in camera P0 is x0 = P0 X = P0 P+ x
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Image of this point in camera P0 is x0 = P0 X = P0 P+ x
Image of C in camera P0 is epipole e0 = P0 C
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Image of this point in camera P0 is x0 = P0 X = P0 P+ x
Image of C in camera P0 is epipole e0 = P0 C
Epipolar line of x in P0 is
l0 = (e0 ) × (P0 P+ x)
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Image of this point in camera P0 is x0 = P0 X = P0 P+ x
Image of C in camera P0 is epipole e0 = P0 C
Epipolar line of x in P0 is
l0 = (e0 ) × (P0 P+ x)
l0 = Fx defines the fundamental matrix
F = (P0 C) × (P0 P+ )
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Finding the Fundamental Matrix from known Projections
Take x in camera P and find one scene point on ray from C to x
Point X = P+ x satisfies x = PX with P+ = PT (PPT )−1 so
PX = PPT (PPT )−1 x = x
Image of this point in camera P0 is x0 = P0 X = P0 P+ x
Image of C in camera P0 is epipole e0 = P0 C
Epipolar line of x in P0 is
l0 = (e0 ) × (P0 P+ x)
l0 = Fx defines the fundamental matrix
F = (P0 C) × (P0 P+ )
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
55 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Epipole e0 is left null space of F, and e is right null space.
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Epipole e0 is left null space of F, and e is right null space.
T
T
All epipolar lines l0 contains epipole e 0 , so e0 l0 = 0, i.e. e0 Fx = 0 for all x,
T
therefore e0 F = 0
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Epipole e0 is left null space of F, and e is right null space.
T
T
All epipolar lines l0 contains epipole e 0 , so e0 l0 = 0, i.e. e0 Fx = 0 for all x,
T
therefore e0 F = 0
F is of rank 2 because F = e0 × (P0 P+ ) and e0 × is of rank 2
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Epipole e0 is left null space of F, and e is right null space.
T
T
All epipolar lines l0 contains epipole e 0 , so e0 l0 = 0, i.e. e0 Fx = 0 for all x,
T
therefore e0 F = 0
F is of rank 2 because F = e0 × (P0 P+ ) and e0 × is of rank 2
F has 7 degrees of freedom, there are 9 elements, but scaling is not
important and det(F) = 0 removes one degree of freedom
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56
Properties of the fundamental matrix
T
Matrix 3 × 3 since x0 Fx = 0
Let F be the fundamental matrix of camera pair (P, P0 ), the fundamental
matrix of camera pair (P0 , P) is F0 = F T
T
T
This is true since xT F0 x0 = 0 implies x0 F0 x = 0, so F0 = FT
Epipolar line of x is l0 = Fx
Epipolar line of x0 is l = FT x0
Epipole e0 is left null space of F, and e is right null space.
T
T
All epipolar lines l0 contains epipole e 0 , so e0 l0 = 0, i.e. e0 Fx = 0 for all x,
T
therefore e0 F = 0
F is of rank 2 because F = e0 × (P0 P+ ) and e0 × is of rank 2
F has 7 degrees of freedom, there are 9 elements, but scaling is not
important and det(F) = 0 removes one degree of freedom
Raquel Urtasun (TTI-C)
Computer Vision
Feb 7, 2013
56 / 56