What is the Space of Camera Responses?
Michael Grossberg and Shree Nayar
CAVE Lab, Columbia University
IEEE CVPR Conference
June 2003, Madison, USA
Partially funded by NSF ITR Award, DARPA/ONR MURI
The Camera Response
0
255
Scene
Radiance
Linear Function
(Optical Attenuation)
Image
Irradiance
Non-Linear
Camera
Response
Image
Intensity
L
s
E
f
B
Impact of Camera Response
Accurate scene radiance required for
Color Constancy
Photometric Stereo
Measuring BRDF
from Images
Creating Accurate High
Dynamic Range Images
Shape from Shading
Inverse Rendering
Response Model for Recovery
Charts:
Known Reflectance
[Sawchuk, 77
Chang and Reid, 96]
Multiple Images:
Changing Exposure
[Debevec and Malik, 1997,
Mann, 2000,
Mann and Picard, 1995,
Mitsunaga and Nayar, 1999,
Tsin et al., 2001]
Model Required for
Interpolation
Breaking Ambiguities
[Grossberg, Nayar, 2002]
Response Normalization and Monotonicity
Saturation
Level
Camera
Response
Normalize
0
Dark Current
Level
f
Intensity B
Intensity B
1
0
Irradiance E 1
Irradiance E
Monotonicity Key Property:
Makes response invertible
Space of Response Functions
The space of all functions with f (0)  0
Let B1  f (1)
Space of theoretical
response functions
B1
0
1
Inequalities: f ( E )  0
Cone of monotonic functions
Linear Model of Response
f
f  f 0  c1h1  c2 h2
h
h2
f0
f0
h1
• M-order linear approximation model:
f ( E )  f 0  n1 cn hn ( E )
M
base response parameters of model
basis functions
Choosing a Basis
• Possible basis h1, h2, …
f 0 ( E )  E, hn ( E )  E  E n 1
Intensity
Polynomial basis
h4
h3
h2
Irradiance
h1
Trigonometric basis
• Which basis is best?
– Depends on which
response functions occur
h3
Intensity
f 0 ( E )  E, hn ( E )  sin( nE )
h1
h4
h2
Irradiance
Database of Response Functions (DoRF)
Collected 201 response curves from:
Film
Positive, negative, consumer, professional, color, b/w
Agfacolor Futura
Agfachrome RX-II
Fuji F125
Fuji FDIC
Kodak Advanced
Kodak Gold
…
CCDs
Kodak's KAI
and KAF series
…
Digital/Video
Sony DC 950
Canon Optura
Gamma curves
…
Sample curves
Kodak Ektachrome-100plus Green
Cannon Optura
Kodak DCS 315 Green
0.9
Sony DXC-950
1
Kodak Ektachrome-64 Green
Agfachrome CTPrecisa100 Green
Agfacolor Futura 100 Green
0.7
Agfacolor HDC 100 plus Green
0.6
Agfacolor Ultra 050 plus Green
0.5
Normalized Brightn
Agfapan APX 025
0.4
Agfa Scala 200x
Fuji F400 Green
0.3
Fuji F125 Green
0.2
Kodak Max Zoom 800 Green
gamma curve, g =0.6
gamma curve, g =1.0
g
0 gamma curve, =1.4
1
gamma curve, g =1.8
0.1
Kodak KAI0372 CCD
Kodak KAF2001 CCD
Intensity
0.8
Agfachrome RSX2 050 Blue
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Irradiance
Evaluate Bases Using DoRF:
Good basis provides good approximation
with few parameters
Empirical Model of Response (EMoR)
Build Basis using DoRF
•175 training curves, 26 testing curves
•Apply PCA to DoRF f0, h1, h2, ….
•99.5% of energy in first 3 dimensions
Response
Principal Components
0.8
Normalized
Intensity
Intensity
1
0.6
0.4
0.2
0
h4
0.04
h2
Normalized
0
-0.04
h3
h1
-0.08
0
0.2
0.4
0.6
Irradiance
0.8
1
0
0.2
0.4
0.6
Irradiance
0.8
1
Energy
100
98
96
94
92
90
88
86
84
82
Percent
of Energy
80
1 2 3 4 5 6 7 8 9 10
Percent
Mean Curve
Response
Principal Components
Log Model of Response (Log EMoR)
basis functions
base response
Log basis:
f0 ( E )
hn ( E ) c n
M
f (E)  e
 n 1 (e
)
•Log model
•Generalizes gamma curves
parameters of model
Log EMoR
•Apply PCA to Log DoRF
•99.6% of energy in first 3 dimensions
Principal Components
0.6
0.4
0.2
0
h4
0.04
0.8
Intensity
Intensity
1
h2
0
h3
h1
-0.04
-0.08
0
0.2
0.4
0.6
Irradiance
0.8
1
0
Energy
Percent
Mean Curve
0.2
0.4
0.6
Irradiance
0.8
1
100
98
96
94
92
90
88
86
84
82
80
1 2 3 4 5 6 7 8 9 10
Principal Components
Monotonic Approximation
Monotonicity:
Least Squares error:
arg min || f  f 0   cn hn ||2
( f 0   cn hn )( E )  0
c
 Linear inequalities in cn
Quadratic Programming
2 Principal Components
6
Gamma Curves
Other DoRF Curves
Derivative at Unity
Gamma = 2.5
Monotonic functions
5
4
Gamma = 0.2
3
2
1
0
Derivative at Unity
-1
-5
0
5
10
15
20
25 30 35
Derivative at Origin
40
45
EMoR/Log EMoR Model Evaluation
1
2
3
4
5
6
7
8
9
10
11
Accuracy:
6.8 bits
Mean RMSE Mean Disparity
4.12E-02
9.07E-02
1.94E-02
4.77E-02
8.87E-03
2.60E-02
4.15E-03
1.51E-02
2.82E-03
1.05E-02
1.91E-03
8.09E-03
1.58E-03
7.46E-03
1.15E-03
4.96E-03
9.10E-04
4.30E-03
7.60E-04
3.95E-03
6.02E-04
3.12E-03
9.0 bits
8.3 bits
Log EMoR Model
Parameters
Parameters
EMoR Model
1
2
3
4
5
6
7
8
9
10
11
Accuracy:
6.8 bits
Mean RMSE Mean Disparity
7.04E-02
1.11E-01
1.71E-02
3.44E-02
8.83E-03
1.85E-02
4.99E-03
1.11E-02
3.44E-03
8.14E-03
2.80E-03
6.58E-03
2.52E-03
5.77E-03
1.67E-03
4.12E-03
1.36E-03
3.36E-03
1.79E-03
4.14E-03
9.54E-04
2.49E-03
8.4 bits
8.6 bits
Models Compared
RMSE Error
Parameters
Model 1
2
3
4
Gamma 3.46E-02 N. A.
N. A.
N. A.
Polynomial 7.37E-02 3.29E-02 1.71E-02 1.06E-02
Trigonometric 6.83E-02 3.91E-02 2.58E-02 1.89E-02
EMoR 4.00E-02 1.73E-02 6.27E-03 2.54E-03
Accuracy: 4.8 bits
5.9 bits
5.3 bits
5
6
7
N. A.
N. A.
N. A.
6.93E-03 4.95E-03 3.65E-03
1.44E-02 1.16E-02 9.46E-03
1.77E-03 1.07E-03 9.55E-04
7.3 bits
Disparity Error
Parameters
Model 1
2
3
4
5
6
7
Gamma 2.52E-01 N. A.
N. A.
N. A.
N. A.
N. A.
N. A.
Polynomial 4.22E-01 2.90E-01 2.12E-01 1.51E-01 1.16E-01 8.96E-02 7.55E-02
Trigonometric 4.12E-01 3.30E-01 2.74E-01 2.32E-01 2.01E-01 1.77E-01 1.57E-01
EMoR 3.04E-01 1.38E-01 8.54E-02 4.90E-02 3.15E-02 2.25E-02 1.86E-02
Accuracy: 2.0 bits
2.2 bits
1.9 bits
3.9 bits
Response from Sparse Samples
Camera Response
Normalized Intensity
1
0.8
0.6
Brightness
0.4
Normalized
0.2
0
0
Monotonic EMoR
Monotonic polynomial
EMoR
Polynomial
Chart values used for fit
Other chart values
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Normalized Irradiance
1
Response from Multiple Images
Inverse Camera Response
Normalized Intensity
1
0.8
Monotonic EMoR
Mitsunaga-Nayar (Polynomial)
Debevec-Malik l = 8 (Log space)
Debevec-Malik l = 32 (Log space)
Debevec-Malik l = 128 (Log space)
Data from chart
0.6
0.4
0.2
0
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
Normalized Irradiance
1
Summary
• Determined Space of Response functions
– Intersection of cone and plane
• Linear and Log approximation models
– Generalized previous models
• Database of Response Functions (DoRF)
– Evaluate models
• Empirical Model of Response (EMoR)
– Superior model of camera response based on DoRF
• DoRF and EMoR available for download
from www.cs.columbia.edu/CAVE