Multispectral Imaging Development at ENST
Francis Schmitt, Hans Brettel, Jon Yngve Hardeberg
Signal and Image Processing Department, CNRS URA 820
École Nationale Supérieure des Télécommunications
46 rue Barrault, F-75634 Paris Cedex 13, France
Abstract
We present the development at ENST of a multispectral imaging system. Various methods
have been implemented and tested for the characterisation of spectral camera sensitivity, the
optimal choice of filters and the reconstruction of the spectral reflectance of the imaged surfaces.
We first used a 4k-linear array camera with a set of large band filters. We are now experimenting
with a cooled 1k x 1k single chip camera coupled with a liquid crystal tunable filter.
Keywords: multispectral acquisition system, sensitivity characterization, filter selection,
spectral reflectance reconstruction
1
Introduction
Multispectral imaging systems are developping rapidly because of their strong potential in
many domains of application, such as remote sensing, astronomy, physics, museum, cosmetics,
medicine, high-accuracy colour printing, computer graphics... They provide information about
a number of spectral bands, from three components per pixel as for colour images up to one
thousand bands for hyperspectral images. Hyperspectral image acquisition systems remains
complex, expensive and difficult to manage due to the huge amount of data to be stored and
processed. For high-end multimedia applications we have considered a more affordable approach based on digital imaging techniques in which a reduced set of chromatic filters are used
with a CCD camera. This was a natural extension of the high quality digital color image system
that we developed for the analysis of paintings, based on a Kodak Eikonix 1412 line-scan CCD
camera (4k-linear array) with a set of large band filters [1, 2, 3].
The multispectral image acquisition system we are aiming at is inherently device independent; we seek to record data representing the spectral reflectance of the surface imaged in each
pixel of the scene, independently of the spectral characteristics of the acquisition system and of
the illuminant.
To obtain device independent measurements of high quality, it is necessary to specify the
spectral characteristics of the components involved in the image acquisition process. The first
step is to process to a spectral characterisation of the camera sensitivity. Then we have to
choose optimally a reduced set of colour filters which will provide the best spectral reflectance
reconstruction according to a given criterion. The set of filters being fixed, the last step consists
of estimating from the camera responses the spectral reflectances of the actual surfaces which
are imaged. These three steps are successively presented in the next sections. They remain valid
with the new and more flexible acquisition system which we are now developing and which is
based on a cooled 1k x 1k single chip camera coupled with a liquid crystal tunable filter. This
system is describd in the last section.
2
Spectral characterisation of the acquisition system
The main components involved in the image acquisition process are depicted in Figure 1. We
denote the spectral radiance of the illuminant by lR (), the spectral reflectance of the object
surface imaged in a pixel by r(), the spectral transmittance of the optical systems in front of
the detector array by o(), the spectral transmittance of an optical colour filter by k (), and
the spectral sensitivity of the CCD array by a(). In this section we first consider an unfiltered
monochrome camera and will omit the spectral transmittance k () in the calculations.
Illumination
lR(λ)
o(λ)
φk(λ)
a(λ)
r(λ)
Optical path
Colour filter
Observed object
Sensor
Figure 1: Schematic view of the image acquisition process.
Supposing a linear optoelectronic transfer function of the acquisition system, the camera
response c to an image pixel is then equal to
c=
Z max
min
lR ()r()o()a() d =
Z max
min
r()! () d
(1)
where ! () = lR ()o()a() denotes the system unknowns. The assumption of system linearity is based on the fact that the CCD sensor is inherently a linear device. However, for real
acquisition systems this assumption may not hold, due for example to electronic amplification
non-linearities or stray light in the camera. Then, appropriate nonlinear corrections may be
necessary [4].
By uniformly sampling the spectra at N wavelength intervals, we can rewrite Equation (1)
as a scalar product in matrix notation, c = t !; where ! = [! (1) ! (2 ) : : : ! (N )]t and
= [r(1 ) r(2 ) : : : r(N )]t .
r
r
Direct spectrophotometric measurements with monochromatic light require expensive equipment. We opted for the popular indirect approach where the vector ! describing the system
unknowns is estimated from the camera responses cp to a set of P samples with known reflectances p [4, 5]. Denoting the sampled spectral reflectances of all the patches as the matrix
= [ 1 2 : : : P ], the camera response P = [c1c2 : : : cP ]t to these P samples is then given by
R rr
r
r
c
cP = Rt!:
(2)
~
Several methods have been proposed for the estimation ! of the camera characteristics ! . The
simple system inversion by:
!~ = (RRt), RcP = (Rt),cP ;
1
R
(3)
R
,
where ( t) denotes the Moore-Penrose pseudo-inverse of t yields very large errors in the
presence of noise. Instead, the Principal Eigenvector method should be used, the noise sensitivity of the system inversion being reduced by only taking into account the singular vectors
corresponding to the most significant singular values. A Singular Value Decomposition (SVD)
is then applied to the (P N ) matrix t of the spectral reflectances of the observed patches:
t ; where
t =
and are (P P ) and (N N ) unitary matrices repectively,
is
a (P N ) matrix with general diagonal entry the singular values wi, i = 1 : : : R, and zeros
elsewhere, R being the rank of t .
and being unitary matrices, it can easily be verified
,
t ,
t
,
that ( ) =
, where
has a general diagonal entry equal to wi,1 , i = 1 : : : R, and
zeros elsewhere.
R UWV
R
U
R
V
R U
W
VW U
W
V
Camera spectral
sensitivity functions
Munsell patches
Sensitivities under illuminant A
1
0.9
0.8
Blue
Green
Red
Sensitivity
0.7
0.6
Evaluation
0.5
0.4
0.3
0.2
0.1
0
400
450
500
550
600
Wavelength, λ, [nm]
650
700
Sensitivities under illuminant A
1
3 reflectances from the finnish Munsell database
0.9
0.9
0.8
Camera
model
0.7
Reflectance r(λ)
.1
.4
.7
:
2.5R 7/6
2.5G 6/12
2.5B 6/8
0.6
0.5
0.4
.8
.3
.1
:
.6
.6
.1
:
0.8
0.6
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0
Blue
Green
Red
0.7
Methods:
PI, PE
Sensitivity
1
0
400
400
450
500
550
600
650
Wavelength, λ [nm]
700
750
450
500
550
600
Wavelength, λ, [nm]
650
700
800
Munsell
spectral reflectances
Acquisition
simulation
Camera
responses
Spectral
characterisation
Estimated spectral
sensitivity functions
Figure 2: Simulation of the acquisition system to evaluate spectral characterisation methods.
R
It is well known that the singular values of a matrix of spectral reflectances such as t are
strongly decreasing, and by consequence that reflectance spectra can be described accurately by
a quite small number of parameters. By taking into account only the first r < R singular values
in the system inversion, the spectral sensitivity may thus be estimated by :
!~ = VW r ,UtcP ;
( )
(4)
W
where (r), has now a general diagonal entry equal to wi,1 , i = 1 : : : r, (r < R).
We have performed a simulation to evaluate the spectral characterisation methods for three
RGB channels and to determine the optimal number r of principal eigenvectors that should be
retained, depending on the level of noise [6]. The scheme of the simulation is shown in Figure 2.
It is based on the Eikonix camera with its built-in filter barrel. We have considered the spectral
reflectances of 1269 matte Munsell colour chips for , and the effects of the quantization noise
for the spectral characterisation of the camera with three RGB filters and CIE illuminant A.
The results are shown in Figure 3. We see on the left that, for a given amount of noise (8 bit
quantisation), too few and too many principal eigenvectors (PE) cause high estimation errors.
The optimal number of PE’s is defined by the curve minimum. As the amount of noise increases,
fewer principal eigenvectors should be used for the spectral sensitivity estimation (center) and
if the noise is too high, good results cannot be obtained because excessive RMS errors (right).
Examples of the estimated camera sensitivity (with RGB filters) are shown in Fig. 4 by using
r = 10 and r = 20 PE’s and 8 bits quantisation noise. We observe that the estimation results
are severely deteriorated when using too many PE’s.
R
PE, quantising using 8 bits
70
0.35
Optimal number of principal eigenvectors
Blue
Green
Red
RMS estimation error
0.3
0.25
0.2
0.15
0.1
0.05
0
0
10
20
30
40
Number of principal eigenvectors
50
0.4
Blue
Green
Red
60
50
40
30
20
10
60
0
Blue
Green
Red
0.35
Optimal RMS estimation error
0.4
0.3
0.25
0.2
0.15
0.1
0.05
2
4
6
8
10
12
Number of bits for quantisation
14
16
0
2
4
6
8
10
12
Number of bits for quantisation
14
16
Figure 3: (left) RMS sensitivity estimation error for different number of PE’s with 8 bit quantisation noise ;(center) optimal number of PE’s for different levels of acquisition noise and (right)
resulting RMS estimation error.
8 bits, 20 PE’s
1.2
1
1
0.8
0.8
Sensitivity
Sensitivity
8 bits, 10 PE’s
1.2
0.6
0.4
0.6
0.4
0.2
0.2
0
0
−0.2
400
450
500
550
600
Wavelength, λ, [nm]
650
700
−0.2
400
450
500
550
600
Wavelength, λ, [nm]
650
700
Figure 4: 3 channel sensitivity estimation with 10 (left) and 20 (right) PE’s, 8 bit quantisation.
The use of 1269 reflectance spectra from the matte Munsell album is possible for simulation
but is a severe limitation for practical applications. However it is well known that the effective
10bits, 20 optimally chosen reflectances, PE(10)
10bits, 20 MacBeth reflectances, PE(10)
1.2
1
1
1
0.8
0.8
0.8
0.6
0.4
0.2
Sensitivity
1.2
Sensitivity
Sensitivity
10 bits, 10 PE’s
1.2
0.6
0.4
0.2
0.6
0.4
0.2
0
0
0
−0.2
400
−0.2
400
−0.2
400
450
500
550
600
Wavelength, λ, [nm]
650
700
450
500
550
600
Wavelength, λ, [nm]
650
700
450
500
550
600
Wavelength, λ, [nm]
650
700
Figure 5: Camera sensitivity estimation with 10 PE’s and 10 bit quantisation noise on (left) 1269
Munsell reflectances, (center) 20 optimally chosen reflectances, (right) 20 Macbeth reflectances.
dimension of reflectance spectra is much lower, e.g. 18 principal components suffice to represent
99 % of the Munsell atlas in cumulative energy. In order to reduce drastically the number
P of target patches, we have proposed the following iterative method for the selection of a
representative set of p samples s1 ; s2 ; : : : ; p [5]. First s1 is chosen as the white of maximum
norm. Next, we select s2 as maximizing the ratio of the smallest to the largest singular value
of the matrix [ s1 s2 ]. Further sample spectra are added according to the same rule: at the pth
iteration the sample sp , which maximizes the ratio of the smallest to the largest singular value
of [ s1 s2 : : : sp,1 p ], is chosen. This optimized constructive approach provides satisfactory
results for the spectral characterisation as shown in Figure 5, where we can compare the camera
sensitivity estimation with a 10 bit quantization noise for the full set of 1269 Munsell color
patches (left) and a reduced set of 20 targets chosen with the above method (center). The
relative increase of the RMS estimation error is only 20 %. On the right we present the result
obtained with the Macbeth ColorChecker, this colour chart being extensively used for colour
calibration tasks. We notice now a slight degradation in the estimation which warns us of the
importance of the color target selection.
rr
3
rr
r
r r
r
r r
r
r
Choice of the filters
The quality of the acquisition system depends heavily on the number and on the choice of the
color filters which will be used. We have proposed a solution where they are chosen from a set
of readily available filters [4]. This choice is optimized by maximizing the orthogonality of the
camera channels when applied to reflectances which is highly representative of the statistical
spectral properties of the objects that are to be imaged in a particular application. The filters are
selected sequentially to maximize their degree of orthogonality after projection into the vector
space spanned by the most significant reflectance eigenvectors. Although this approach remains
suboptimal, it avoids the heavy computation cost required with an exhaustive search.
We have performed a simulation to evaluate the complete multispectral image acquisition
system. We used D65 as the scanning illuminant, the Eikonix CCD camera spectral characteristics, filters chosen from a set of 37 Wratten, Hoffman, and Schott filters, and the spectral
reflectances of a colour chart of 64 pure pigments used in oil painting and provided to us by the
National Gallery in London [4]. The results for the case of a selection of seven filters are shown
in Figure 6.
Spectral transmittance of the 7 filters
Spectral sensitivity of the 7 channels
Spectral reconstruction using 7 filters
0.8
0.9
1
1
2
3
4
5
6
7
0.7
0.8
0.6
Sensitivity, θ(λ)
Transmittance, φ(λ)
0.7
0.5
0.6
0.5
0.4
0.4
0.3
0.3
Emerald green
Ultramarine
Red ochre
Mercuric Iodide
Reconstructions
0.9
0.8
0.7
Reflectance
1
0.6
0.5
0.4
0.3
0.2
0.2
0.2
0.1
0.1
0
400
450
500
550
600
650
Wavelength, λ, [nm]
700
0
750
0.1
400
450
500
550
600
650
Wavelength, λ, [nm]
700
750
0
400
450
500
550
600
650
Wavelength, λ [nm]
700
Figure 6: Optimized selection of seven filters : (left) Spectral transmittance; (center) Spectral
channel sensitivity, the numbers denote the sequence in which the filters have been chosen;
(right) Reconstruction of four spectral reflectances from the camera responses using the reconstruction operator 1.
Q
4
Spectral reflectance estimation from camera responses
The spectral characteristics ! () of the unfiltered camera being determined, we may now model
the camera responses ck , k = 1 : : : K to an unknown reflectance r(), using a set of K chroR max
matic filters with known spectral transmittances k (): ck = min
r()k ()! ()d: The vect
tor K = [c1c2 : : : cK ] representing the response to all K filters may be described using matrix
notation as
c
cK = tr;
(5)
where is the known N -line, K -column matrix of filter transmittances multiplied by the camera characteristics, that is = [kn ] = [k (n )! (n )].
The problem of the estimation of a spectral reflectance from the camera responses K is
often formulated as finding a matrix that reconstructs the spectrum from the measurement
vector (see Figure 7):
~r
Q
c
~r = QcK :
(6)
0.9
0.9
Emerald green
Ultramarine
Red ochre
Mercuric Iodide
Reflectance r(λ)
0.7
0.5
0.4
0.3
0.7
Q
0.2
0.6
0.5
0.4
0.3
0.2
0.1
0
Emerald green
Ultramarine
Red ochre
Mercuric Iodide
0.8
Digital
CCD camera
0.6
Reflectance r(λ)
0.8
0.1
400
450
500
550
600
650
Wavelength, λ [nm]
700
0
750
Observed spectral
reflectances
Multi-channel
image
400
450
500
550
600
650
Wavelength, λ [nm]
700
750
Spectral
reconstruction Reconstructed spectral
operator
reflectances
Figure 7: Spectral reflectance reconstruction with a multi-channel image acquisition system.
As other authors, we can take advantage of apriori knowledges on the spectral reflectances
that are to be imaged, by assuming that the reflectance in each pixel is a linear combination
of a known set of P smooth reflectance functions: =
; with
= [ 1 2 : : : P ] the matrix
of the P known reflectances and = [a1a2 : : : aP ]t a vector of coefficients. We have defined in
a
r
r Ra
R rr
r
750
Q
r ~r) between
[7] another reconstruction operator 1 that minimizes the Euclidian distance dE ( ;
the original spectrum and the reconstructed spectrum = 1 K :
r
~r Q c
Q = RRt(tRRt), :
1
1
(7)
To evaluate the quality of spectral reflectance reconstruction we use the mean and maximum RMS errors, that is, the Euclidian distance between original and reconstructed spectral
reflectances. In Table 1, the RMS spectral reconstruction errors using different number of filters are reported. As expected, we see that the mean reconstruction error decreases when an
increasing number of filters are used. In Figure 6-(right) we show some examples of spectral
reflectance reconstruction by using the seven filter selection.
Number of filters
Mean RMS error (100)
Max RMS error (100)
3
3.57
8.79
4
2.39
6.77
5
1.78
5.38
6
1.32
4.93
7
1.11
6.16
8
0.87
3.23
9
0.57
1.74
10
0.56
1.84
11
0.36
1.22
12
0.30
1.05
Table 1: Comparison of the RMS spectral reconstruction error for varying number of filters using the
reconstruction operator 1 .
Q
5
New development and conclusion
In our experiments done with the Eikonix line-scan CCD camera, the multispectral acquisition
process was prohibitively slow and the noise was a constant problem. We decided to develope a new experimental multispectral camera by assembling a PCO SensiCam monochrome
CCD camera (http://www.pco.de/) and a CRI VariSpec Liquid Crystal Tunable Filter (LCTF)
(http://www.cri-inc.com/). The camera has a 1280 1024 single chip and a dynamic range of
12 bit. It is cooled to ,12 C . Its exposure times is from 1 ms to 1000 s, and it operates at a
12.5 MHz readout frequency. Its spectral sensitivity function, as given by the manufacturer, is
reported in Figure 8-right. The camera is controlled from a PC via a PCI-board.
The peak wavelength of the tunable LCTF filter can be controlled electronically from an
external controller unit, or from a computer via a RS-232 interface, in the range [400 nm,
720 nm]. Figure 8-left shows the measured transmittances for 9 peak wavelength positions.
The transmittances have more or less a Gaussian-like shape , except for the wide-band setups at peak wavelengths > 650 nm. There is an additional infra-red filter present, the spectral
transmittances being cut at the red end of the spectrum. For peak wavelengths 440 nm, the
filters have an unwanted secondary peak at long wavelengths. The average Full-Width-at-HalfMaximum (FWHM) bandwidth is not constant and varies from 15 to 80 nm. These artifacts
have to be carefully compensated for in a device-independant multispectral acquisition system.
The LCTF technology offers a reasonably wide field of view (7 from the normal axis) but
nevertheless, the limitation in field-of-view is a parameter that has also to be treated with care
for imaging applications.
The tunable filter and the CCD camera are both controlled by a PC under Win-NT. This
”SpectraCam” computer is linked via Ethernet to the campus-wide Local Area Network (LAN)
and the Internet. A modular client-server software architecture has been developed. The client
parts are implemented in Java and allow for interactive multispectral acquisition across the Web.
In conclusion the development of a multispectral image acquisition system requires a precise
control of its spectral characteristics. The characterisation of camera sensitivity, the selection
of the filters and the reconstruction of the spectral reflectances from the camera responses have
CRI VariSpec LCTF 50346
Sony ICX085AL CCD Image Sensor
1
0.16
0.9
CCD spectral sensitivity
Transmittance
0.14
0.12
0.1
0.08
0.06
0.04
0.02
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
400
450
500
550 600 650
Wavelength [nm]
700
750
400
450
500
550 600 650
Wavelength [nm]
700
750
Figure 8: (left) Spectral transmittances of the CRI VariSpec Model VIS2 LCTF filter; (right)
Spectral sensitivity of the PCO SensiCam Model 370 KL camera (from http://www.pco.de/).
been carefully studied, mostly by simulation based on a previous system. We are currently
extending these approaches with the new system under development. We expect from it much
easier experiments with more precise measurements and more robust results.
References
[1] F. Schmitt, H. Maı̂tre, and Y. Wu, “First progress report: tasks 2.4 (Development / procurement of basic software routines) and 3.3 (Spectrophotometric characterization of paintings)
— Vasari project.,” Tech. Rep. 2649, CEE ESPRIT II, Jan. 1990.
[2] Laboratoire de Recherche des musées de France, LRMF, “Corot 1796-1875, 85 œuvres du
Musée du Louvre - Analyse scientifique.” CD-ROM AV 500041, Collection Art & Science,
1996.
[3] F. Schmitt, “High quality digital color images,” in Proceedings of the 5th International
Conference on High Technology: Imaging Science and Technology, Evolution and Promise,
(Chiba, Japan), pp. 55–62, Sept. 1996.
[4] H. Maı̂tre, F. Schmitt, J.-P. Crettez, Y. Wu, and J. Y. Hardeberg, “Spectrophotometric image
analysis of fine art paintings,” in Proceedings of IS&T and SID’s 4th Color Imaging Conference: Color Science, Systems and Applications, (Scottsdale, Arizona), pp. 50–53, Nov.
1996.
[5] J. Y. Hardeberg, H. Brettel, and F. Schmitt, “Spectral characterisation of electronic cameras,” in Electronic Imaging: Processing, Printing, and Publishing in Color, vol. 3409 of
SPIE Proceedings, (Zürich, Switzerland), pp. 100–109, May 1998.
[6] J. Y. Hardeberg, Acquisition and reproduction of colour images: colorimetric and multispectral approaches. Ph.D dissertation, enst, École Nationale Supérieure des Télécommunications, ENST, Paris, France, 1999.
[7] J. Y. Hardeberg, F. Schmitt, H. Brettel, J.-P. Crettez, and H. Maı̂tre, “Multispectral image
acquisition and simulation of illuminant changes,” in Colour Imaging: Vision and Technology (L. W. MacDonald and R. Luo, eds.), Wiley, 1999.