Camera calibration
Digital Visual Effects, Spring 2007
Yung-Yu Chuang
2007/4/17
with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes
Announcements
• Project #2 is due next Tuesday before the class
Outline
•
•
•
•
Camera projection models
Camera calibration (tools)
Nonlinear least square methods
Bundle adjustment
Camera projection models
Pinhole camera
Pinhole camera model
(X,Y,Z)
P
origin
p
(x,y)
principal point
(optical
center)
Pinhole camera model
principal
point
 x   fX   f
    
 y  ~  fY    0
1  Z  0
    
0
f
0
X
0 0  
Y 

0 0  
Z
1 0 
1
fX
x
Z
fY
y
Z
Pinhole camera model
principal
point
 x   fX   f
    
 y  ~  fY    0
1  Z  0
    
0
f
0
X
0 1 0 0 0 
Y 



0  0 1 0 0   
Z
1 0 0 1 0 
1
Principal point offset
principal
point
intrinsic matrix
only related to
camera projection
x ~ KI 0X
 x   fX   f
    
 y  ~  fY    0
1  Z  0
    
0
f
0
X
x0  1 0 0 0 
Y 



y 0  0 1 0 0   
Z
1  0 0 1 0 
1
Intrinsic matrix
Is this form of K good enough?
f
K   0
 0
• non-square pixels (digital video)
• skew
 fa
• radial distortion

K0
 0
0
f
0
s
f
0
x0 
y0 
1 
x0 

y0 
1 
Distortion
No distortion
Pin cushion
Barrel
• Radial distortion of the image
– Caused by imperfect lenses
– Deviations are most noticeable for rays that pass
through the edge of the lens
Camera rotation and translation
 X '
X
 
 
 Y '   R 33  Y   t
 Z'
Z
 
 
 x  f
  
 y ~ 0
1 0
  
0
f
0
X
 
x0 
Y 

y0 R t  
Z
 
1 
1
 
x ~ KR t X
extrinsic matrix
Two kinds of parameters
• internal or intrinsic parameters such as focal
length, optical center, aspect ratio:
what kind of camera?
• external or extrinsic (pose) parameters
including rotation and translation:
where is the camera?
Other projection models
Orthographic projection
• Special case of perspective projection
– Distance from the COP to the PP is infinite
Image
World
– Also called “parallel projection”: (x, y, z) → (x, y)
Other types of projections
• Scaled orthographic
– Also called “weak perspective”
• Affine projection
– Also called “paraperspective”
Fun with perspective
Perspective cues
Perspective cues
Fun with perspective
Ames room
Forced perspective in LOTR
Camera calibration
Camera calibration
• Estimate both intrinsic and extrinsic parameters
• Mainly, two categories:
1. Photometric calibration: uses reference objects
with known geometry
2. Self calibration: only assumes static scene, e.g.
structure from motion
Camera calibration approaches
1. linear regression (least squares)
2. nonlinear optimization
3. multiple planar patterns
Chromaglyphs (HP research)
Linear regression
x ~ KR t X  MX
Linear regression
• Directly estimate 11 unknowns in the M matrix
using known 3D points (Xi,Yi,Zi) and measured
feature positions (ui,vi)
Linear regression
Linear regression
Linear regression
Solve for Projection Matrix M using least-square
techniques
Normal equation
Given an overdetermined system
Ax  b
the normal equation is that which minimizes the
sum of the square differences between left and
right sides
A Ax  A b
T
Why?
T
Normal equation
E (x)  Ax  b 
2
 a11 ... a1m 
 b1 
 :



x
:
:


1


 


 :
:  :    : 


 


x
:
:
:
m




 
an1 ... anm 
bn 
nxm, n equations, m variables
Normal equation
m






  a1 j x j   b1 

a
x

 1 j j   b   j 1




1
j 1

  

 m :  :  m :





Ax  b    aij x j    bi      aij x j   bi 
j 1
    j 1



 :

:
m :    


  anj x j  bn   m
  anj x j   bn 
 j 1


 j 1


m
m




2
E (x)  Ax  b      aij x j   bi 
i 1 


 j 1
n
2
Normal equation
m




2
E (x)  Ax  b      aij x j   bi 
i 1 


 j 1
n
2
m




E
0
  2   aij x j   bi ai1
x1
i 1 


 j 1
n
n
m
n
i 1
j 1
i 1
 2 ai1  aij x j  2 ai1bi
E
T
T
T
T
0
 2( A Ax  A b) → A Ax  A b
x
Normal equation
Ax  b 
T
 Ax  b  Ax  b 
T
T
 Ax   b Ax  b 
 x T A T  b T Ax  b 
2
 x T A T Ax  b T Ax  x T A Tb  b Tb


T


T
 x A Ax  A b x  A b x  b b
T
T
T
E
T
T
 2A Ax  2A b
x
T
T
Linear regression
• Advantages:
– All specifics of the camera summarized in one matrix
– Can predict where any world point will map to in the
image
• Disadvantages:
– Doesn’t tell us about particular parameters
– Mixes up internal and external parameters
• pose specific: move the camera and everything breaks
Nonlinear optimization
• A probabilistic view of least square
• Feature measurement equations
• Likelihood of M given {(ui,vi)}
Optimal estimation
• Log likelihood of M given {(ui,vi)}
• It is a least square problem (but not necessarily
linear least square)
• How do we minimize C?
Optimal estimation
• Non-linear regression (least squares), because
the relations between ûi and ui are non-linear
functions M
unknown parameters
We could have terms like f cos  in this
u  uˆ ~ u  KR t X
known constant
• We can use Levenberg-Marquardt method to
minimize it
A popular calibration tool
Multi-plane calibration
Advantage
Images courtesy Jean-Yves Bouguet, Intel Corp.
• Only requires a plane
• Don’t have to know positions/orientations
• Good code available online!
–
Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
–
Matlab version by Jean-Yves Bouget:
http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
–
Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
Step 1: data acquisition
Step 2: specify corner order
Step 3: corner extraction
Step 3: corner extraction
Step 4: minimize projection error
Step 4: camera calibration
Step 4: camera calibration
Step 5: refinement
Nonlinear least square methods
Least square fitting
number of data points
number of parameters
Linear least square fitting
y
model parameters
y(t )  M ( x, t )  x0  x1t
t
f i ( x)  yi  M ( x, ti )
residual
prediction
M ( x, t )  x0  x1t  x2t is linear, too.
3
Nonlinear least square fitting
Function minimization
Least square is related to function minimization.
It is very hard to solve in general. Here, we only consider
a simpler problem of finding local minimum.
Function minimization
Quadratic functions
Approximate the function with
a quadratic function within
a small neighborhood
Quadratic functions
A is positive definite.
All eigenvalues
are positive.
Fall all x,
xTAx>0.
A is singular
negative definite
A is indefinite
Function minimization
Descent methods
Descent direction
Steepest descent method
the decrease of F(x) per
unit along h direction
→
hsd is a descent direction because hTsd F’(x)=- F’(x)2<0
Line search
Find  so that
 ( )  F(x 0  αh)
is minumum
 ( ) F(x 0  αh)
0



F x

 h T F ' ( x 0  αh )
x 
h  F' (x 0 )
Line search
h T F ' ( x 0  αh )  0
h  F' (x 0 )
h T F ' ( x 0  αh )

 h T F ' ( x 0 )  α F '' ( x 0 ) T h
 h Th  αh T Hh  0
T
h h
 T
h Hh

Steepest descent method
isocontour
gradient
Steepest descent method
It has good performance in the initial stage of the iterative
process. Converge very slow with a linear rate.
Newton’s method
→
→
→
→
It has good performance in the final stage of the iterative
process, where x is close to x*.
Hybrid method
This needs to calculate second-order derivative which
might not be available.