Modeling the Space of Camera
Response Functions
IEEE Transactions on pattern analysis and machine intelligence,
vol. 26, no. 10, October 2004
Michael D. Grossberg and Shree K. Nayar
School of Electrical Engineering and Computer Science
Kyungpook National Univ.
Abstract
‹ Camera
response function
– Relating scene radiance to image intensity
– Collected a diverse database of real world camera
response functions (DoRF)
– Analyzed the properties that all camera responses
share
– To create a low-parameter empirical model of
response (EMoR)
– Interpolated the complete response function of a
camera from a small number of measurements
2 / 25
1. Scene radiance to image intensity
‹ Computer
vision algorithms
– Required precise measurements of scene radiance
– Examples of algorithms
• Color constancy
• High-dynamic range images
• Photometric stereo
• BRDF from images
‹ Goal
of this work
– Mapping from scene radiance to image intensity
3 / 25
– Including several complex factors
• Vignetting, lens fall-off, fixed pattern noise, shot noise, dark
current, read noise
• Film camera
– Photosensitive response of the film
– Film developing process and scan
– Two basic transformations
Fig. 1. Function s models the optics of the image system.
The mapping f of image irradiance to image intensity is called the
camera response function. B = f (E )
4 / 25
– Previous method
• Using multiple images of a scene taken with different
exposures
– Required many parameter
• Priori assumptions
– Mann and Picard describe the recovery of the parameters
» A gamma curve
f (E) = α + β E r
– Imposed smoothness constraints
– General approximation model of polynomials
5 / 25
– Seeking answers to the following fundamental
question
• What is the space of possible camera response function?
– Lie within a convex set
» Intersection of a hyperplane and a positive cone
• Which camera response functions within this space arise in
practice?
– Compiled a database of response functions (DoRF)
– Represent the variety of response functions
– Total of 201 real-world response functions
• What is a good model for response functions?
– Wide gamut of response functions with a very small number of
parameters
6 / 25
2. What is the theoretical space of camera
response functions ?
‹ Theoretical
space
– Help to build a model of camera response function
– First assumption
• Response function f is the same at each pixel
– Second assumption
• Range of our camera’s response goes from BMIN to BMAX
• Normalized response
BMIN = 0, BMAX = 1
7 / 25
– Third assumption
• Normalized response function is monotonic
WRF := { f | f (0) = 0, f (1) = 1, and f monotonically increasing}
All response vectors must lie in the
hyperplane W1 . Any response function
can be expressed as f = f 0 + h where
f 0 is some base response function.
Fig. 2. Visualize the theoretical space. Vectors that satisfy
f (1) = 1 lie on the hyperplane W1 .
• Hyperplane WRF
WRF = W1 ∩ ∧
(1)
8 / 25
3. Approximation models for the
response function
‹ Given
approximation models
– Choosing the number of parameters M
• Better approximation as large M
– The simple approach
• Parameterizing WRF uses (1)
M
f 0 ( E ) + ∑ cn hn ( E )
(2)
n =1
where c1 ,......, cM : coefficients or parameters of the model
hn : basis vector
• Gamma functions or log-space least-squares solutions
9 / 25
4. Real-world response functions
‹ Collected
a diverse set of response curves
– Film response curves
• Designed to produce attractive images
• Digital cameras often emulate these curves
• Published by manufacturers
– Digital camera response curves
• Unwilling to provide the responses
– Designed to gamma curves
• Many camera manufacturers
10 / 25
– A few of database
Fig. 3. Examples from our database of 201 real-world response functions.
The database includes photographic films, digital cameras, CCDs,
and synthetic gamma curves
11 / 25
5. An empirical model of response
‹ DoRF
database
– Training set of 175 response curves
– Testing set with 26 curves
‹ Finding
a low-dimensional approximation
– Goal
• Determining a base function f 0 and a basis {h1 , h2 ,...., hM }
• Small RMS error for the empirical data from DoRF
– Principal component analysis (PCA)
• Mean curve is also a response function
12 / 25
– Performing PCA
f0
Fig. 4. (a) The mean of 175 camera response.
(b) Four eigenvectors corresponding to the largest four eigenvalues.
(c) A plot showing the percentages of the energies.
Associated with the eigenvalues.
13 / 25
– Covariance operator
• Covariance matrix Cm,n of training set
(C f )( E ) =
1
∫ c ( E , x ) f ( x ) dx
0
N
c ( E1 , E 2 ) = ∑ ( g n ( E1 ) − f 0 ( E1 ))( g n ( E 2 ) − f 0 ( E 2 ))
n =1
(3)
N
C m , n = ∑ ( g p ( E n ) − f 0 ( E n ))( g p ( E m ) − f 0 ( E m ))
n =1
where
f ( En ) = Bn
En : sampling of response curve
• The largest M eigenvalues of the matrix
14 / 25
– EMoR approximation
~
f = f 0 + Hc
where
(4)
~
f : EMoR approximation
H : Eigenvectors , [h1.....hM ]
T
c : H ( f − f0 )
f ( En ) = Bn
– PCA in log-space
f ( E ) ≈ (e
f log, 0 ( E )
M
) ∏ (e
n =1
hlog,n ( E ) cn
)
(5)
• Simpler technique of finding a linear basis directly in WRF
• Very sensitive for small irradiance values
15 / 25
6. Imposing monotonicity
‹ Functions
in the theoretical space
– Must be monotonic
– Derivative is positive
• Discrete derivative matrix D
~
D f mon ≥ 0
~ mon
f
= f 0 + Hcˆ
where cˆ = arg min c || Hc − f − f 0 ||2
• Constraint
DHcˆ ≥ − f 0
• Using quadratic programming
– Finding ĉ
(6)
16 / 25
– To visualize both the space of response function WRF
and how the DoRF curves appear
Fig. 6. The span of the eigenvectors associated with the two largest
eigenvalues.
17 / 25
7. Evaluating the EMoR
‹ Evaluation
– Various approximation models
• High cost of using many parameters
– EMoR method
• The number of parameters M=1,3,5,7 and 9
Fig. 7. A qualitative illustration
of how the fit of the EMoR model
improves with the numbers of
model parameters.
18 / 25
– Extensive quantitative
Table 1. Approximations for M=11
Table 2. Various approximation models average
19 / 25
Table 3. Maximum disparity between curves
Table 4. Evaluation of the performance of the Log version
• Log-space does not minimize the least square error
– Advantageous for gamma response
20 / 25
8. Camera response from
sparse samples
‹ Estimated
a camera’s response function
– Macbeth chart
• Six patches from white through gray to black
• No guarantee that other method
Fig. 8. The response curve of
a Nikon 990 camera
21 / 25
– Condition
E1 ,.....E6
• Corresponding intensity values B1 ,.....B6
• Six normalized irradiance values
• Three coefficients for the EMoR model
• EMoR basis function h1 , h2 , h3
– Evaluated at
E1 ,.....E6
– EMoR model
• Accurated reconstructions of response curves from very few
samples
22 / 25
9. Response from multiple images
‹ Arbitrary
static scene
– Different exposures
−1
f
=g
– Recovered the inverse response function
M
g ( B) = g 0 ( B) + ∑ cn hninv ( B)
• Satisfied g ( Ba ) = k g ( Bb )
n
Fig. 9. Nikon 990 Coolpix camera using exposures
e, 2e, and 4e. These images were used to recover the
inverse response of the camera
23 / 25
– Using least-squares techniques
– Inverse camera response
Fig. 10. Inverse response curves recovered from the images
Fig. 9 using the monotonic EMoR model
24 / 25
10. Conclusion
‹ Recovered
camera response curves
– Provided a simple model which takes us from image
intensity to scene radiance
– Created database of 201 real-world response functions
– Using PCA
– Showed two example
• A few patches on a reflectance chart
• Arbitrary scene taken with different exposures
– Provided an accurate and efficient low-parameter
model
25 / 25