IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
409
Jointly Registering Images in Domain and Range by
Piecewise Linear Comparametric Analysis
Frank M. Candocia, Member, IEEE
Abstract—This paper describes an approach whereby comparametric analysis is used in jointly registering image pairs in
their domain and range, i.e., in their spatial coordinates and pixel
values, respectively. This is accomplished by approximating a
camera’s nonlinear comparametric function with a constrained
piecewise linear one. The optimal fitting of this approximation to
comparagram data is then used in a re-parameterized version of
the camera’s comparametric function to estimate the exposure difference between images. Doing this allows the inherently nonlinear
problem of joint domain and range registration to be performed
using a computationally attractive least squares formalism. The
paper first presents the range registration process and then
describes the strategy for performing the joint registration. The
models used allow for the pair-wise registration of images taken
from a camera that can automatically adjust its exposure as well
as tilt, pan, rotate and zoom about its optical center. Results
concerning the joint registration as well as range-only registration
are provided to demonstrate the method’s effectiveness.
Index Terms—Comparagram, comparametric function, domain
registration, image registration, joint registration, least squares,
photoquantity, piecewise linear, range registration.
I. INTRODUCTION
R
ESEARCH into the registration of images has been
ongoing for several decades [1]. Results of this research
have been applied in fields such as photogrammetry [2],
computer vision [3] and medical imaging [4] and to many
problems such as image superresolution [5], [6], depth extraction [7], video compression [8], [9], video indexing [10]
and the construction of large mosaics [11]. This large body
of research, however, has been slanted toward domain-only
registration; that is, registration between image functions’
spatial coordinates. A reading of comprehensive survey papers
on image registration and the references they list provides
such evidence [1], [4]. These registration approaches have
typically assumed a specific geometric relationship between
images’ spatial coordinates, for example translational, affine or
perspective relationships have been common [11]–[13]. Also
assumed have been more complex relationships involving the
computation of the optical flow field [14] or the modeling of
lens distortion [15]. More recently, there is a growing trend
Manuscript received June 27, 2002; revised January 23, 2003. This work was
supported in part by the Telecommunications and Information Technology Institute of Florida International University under Grant BA01-20. The associate
editor coordinating the review of this manuscript and approving it for publication was Prof. Bruno Carpentieri.
The author is with the Department of Electrical and Computer Engineering,
Florida International University, Miami, FL 33174 USA (e-mail: [email protected]).
Digital Object Identifier 10.1109/TIP.2003.811497
in optical imaging involving range-only registration; that is,
registration between image functions’ pixel values [16]–[20].
This research seeks to construct high dynamic range maps of
an imaged scene from several images captured at different
optical settings. In this way, the typical imaged scene resolution
of 8 bits/pixel can be significantly increased and the multiple
captured images can be combined into a single picture where
all objects are now properly exposed [21], [22]. The registration
of optical images in both domain and range has been an area
that has seemingly received less attention from the research
community. In medical imaging, the difficulty of accurately
modeling intensity relationships between inter-modality images
has led to variational approaches for performing the registration
[23], [24]. With optical images, this modeling has been less
complicated. As such, approaches that have drawn from the
aforementioned range-only registration research have utilized
spatially varying optical filters on cameras so as to obtain
multiple measurements of scene points under differing optical
settings [25], [26]. By characterizing these filters responses, a
mechanism has been created that can provide a greater dynamic
scene description and this, in turn, can be used to create image
mosaics with increased dynamic range.
The work presented in this paper is also concerned with registering images where scene points have been multiply imaged at
different optical settings. However, it registers captured images
from a camera needing no special lenses or attachments. That
is, as today’s typical cameras automatically adjust their exposure depending on the scene imaged and/or ambient lighting and
this effect on the image produced can be accurately modeled, it
should be accounted for during the registration process if indeed
this is warranted. Now, there are situations where this change
can be neglected, for example when the amount of variation is
practically less than camera’s noise power or when brightness
independent features are used in the registration process. But
when this variation in pixel intensity is no longer negligible, as
in luminance-based, i.e., featureless, registration approaches, it
should be accounted for during registration.
To the author’s knowledge, the lone work that has contributed
to this type of joint registration has been that of Mann [19],
[27]. This range registration work has established models that
can be used to describe the nonuniform biasing of pixels that
an image undergoes when a camera’s exposure settings are automatically adjusted in response to the amount of light sensed.
The nonuniform biasing is due to the nonlinear dynamic range
compression that an image sensor’s output voltage undergoes
and which is subsequently quantized to form a pixel. This range
compression is described by the camera’s response function or
can, alternately, be described by its associated comparametric
1057-7149/03$17.00 © 2003 IEEE
410
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
function [19]. Owing to the comparametric function’s nonlinear
nature, exposure estimation between images is typically a nonlinear estimation problem given the pixel samples between two
images. When images are also to be registered in their spatial
coordinates, all but the simplest of domain models also result
in a nonlinear parameter estimation problem. Any effective linearization or procedure whereby the registration problem can be
solved using a least squares formalism would present a computationally attractive alternative to this aforementioned nonlinear
problem—whether it be in domain registration, range registration or both.
The approach taken herein will perform the registration of
a set of images by pair-wise registering adjacent images in
the sequence and then utilizing the group structure of both the
domain and range models to relate each image to a common
domain and range reference. For each image pair, the joint
estimation of parameters of a homographic (perspective) mapping in domain and a comparametric mapping in range is
performed. The range model utilizes a piecewise linearization
of the associated nonlinear comparametric function and this is
incorporated into a domain model that is approximated using
a series expansion. This results in a straightforward procedure
where now, images can be registered jointly in domain and
range using linear least squares and this can be done using
any camera response function.
In the remainder of this paper, Section II will address the
camera response function and how this is used in registering two
images in range. A piecewise linear approximation will be presented so as to solve this range registration using least squares
and some range registration examples will be shown. In Section III, the established range model will be incorporated into
an optimization procedure whereby both the domain and range
model parameters can be determined using least squares computations. Results utilizing this framework on a real video sequence will be described in Section IV. The paper will conclude
in Section V with a brief summary and paper-related comments.
II. CAMERA RESPONSE FUNCTION AND
REGISTRATION PROCESS
THE
RANGE
The formation of each observed pixel value of an image is a
process whereby the integration of photons over a finite period
of time (the camera’s exposure time) occurs over a finite sensor
area. This integration over time and space results in a quantifiable unit regarded as a photoquantity [19] and will be denoted
by . The photoquantity is then subject to intrinsic sensor noise
and finally goes through a dynamic range compression for limiting the range of possible values to a finite interval which is
then quantized to form a pixel (typically 8 bits/pixel). We can
express the formation of an image pixel then as
related to the final formation of the pixel (which includes quantization, electronics and/or image compression related noise).
Also,
and represents the two-dimensional sensor’s
spatial coordinates.
Notice that the photoquantity lies in the interval
where
signifies that no light has been absorbed. The
is a monotonically increasing
camera response function
function that is “normalized” such that over the domain of ,
, hence the pixel value varies in direct relation
with the measured photoquantity. If we now capture an image
over the same field of view but where its exposure time is
different from that of , the photoquantity measured (per pixel)
in is directly proportional to that measured in , i.e., is
now expressed as
(2)
when there is an increase in exposure time in
and where
with a decrease in exposure time
image relative to ,
when the exposure times are equivalent.
from to and
To register images in range, one must estimate the exposure
difference between images typically using only the observed
and
, differ only in
pixel values. Now, let two images,
exposure. From (1) and (2) we see the only difference between
these two images is parameter , which as mentioned, quantifies the exposure difference we seek to estimate. By normalizing
,
, we
all pixel values in and such that
can model our camera using any appropriate unicomparametric
function described by Mann. The estimation process then involves the computation of a comparagram, i.e., a matrix used to
relate the pixel values of two differently exposed images of the
same subject matter, from pixel values of and followed by
the solving for in the unicomparametric equation that provides
an optimal fit to the comparagram data [19].
A. Piecewise Linear Modeling
can be
Any nonlinear unicomparametric function
.
modeled with a constrained piecewise linear function
Notice here that and are variables representing the normalized (to interval [0,1)) range, i.e., pixel values, of two images
and that the explicit relation to the spatial coordinates and
photoquantity upon which images and depend is purposely
represents
omitted for the time being. Also note that
the dependence of on the value of as well as the exposure
parameter . With that said, a piecewise function is defined
,
as
with linear segments over the interval
..
.
(3)
(1)
is the photoquantigraphic measure at spatial coorwhere
represents the intrinsic noise
dinate of the planar sensor,
is the nonlinear camera response funcrelated to the sensor,
is noise
tion performing the dynamic range compression and
,
is a linear segment
where
represents
prespecified knots in
,
. That
and
and
are the knots at the interval’s endpoints and ,
is,
are the
interior knots. In order that
CANDOCIA: JOINTLY REGISTERING IMAGES IN DOMAIN AND RANGE BY PIECEWISE LINEAR COMPARAMETRIC ANALYSIS
best approximates
in this work, certain constraints
are employed. First, the endpoints of adjacent linear segments
must be equal. This is satisfied by the constraint
411
where note that
(4)
and which accounts for the endpoints
where
“interior” linear segments. Second, the constraints
of the
and
to be
on the two unaccounted endpoints
enforced follow from the unicomparametric equations that are
and
.
being approximated here, namely that
,
This leads to simple constraints in our model where
and
,
. Using the constraints in
, the “ -intercepts” in each linear
(4) together with
becomes a function of the slopes
and knots
segment
defining each previous linear segment:
(10)
. That is, for each of the constrained linear
and
the partial derivative of this function with
segments in
depends on the linear segment of
respect to any parameter
interest. Also, note that the equivalence in (9) must hold for all
. This being the case, (9) can be written
where
in the form of a linear system of equations
is the vector of unknown parameters
is an
matrix and is
to estimate and
length vector. Please note that for each value of ,
an
, (9) can equivalently be expressed as
(5)
, the summation in (5) is not performed
Notice that when
in linear
since would range from 1 to 0. This leads to
as it should. When the constraint
is
segment
is seen to be a function of slope
included, slope parameter
and the knots . We can therefore write
parameters
(6)
. This makes
dewhere
since the knots are fixed.
pend only on parameters
Thus, we have chosen to parameterize the model in terms of
, the slopes of the first (
) linear segments
, and we can express each linear segment ,
of
, in terms of these free parameters as
(11)
Now substituting (3), whose constrained linear segments are described using (7), into (11) followed by simplification, we can
determine the values in and . We find that each element
in vector
is
(7)
.
pairs of samples ( ,
),
,
Given
from the comparagram of our two input images, we choose the
best-fit piecewise linear function to be the one that minimizes
the cost function
(12)
(8)
more compactly, we write
In order to express
where
with
unTo minimize this cost function with respect to the
we evaluate
,
known parameters
, to get a system of
simultaneous equations
unknowns:
in
(9)
otherwise
(13)
412
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
and
with
otherwise
where
that fall in the interval
with
,
(14)
is the number of samples
. Also,
;
otherwise
and
(15)
where
data pairs (
Once and are constructed from the
can be solved using, for example, least squares.
Fig. 1. Two images to register in range. There is a difference in exposure of
k = 16 in (b) relative to (a).
for values of near 0 and near 1 and inclusion of such values
from noise in the average unjustly biases the final estimate. The
last point to make regarding (17) is that the integral is easily
evaluated using any simple numerical integration technique. We
have used Simpson’s rule in our calculations [28].
C. Range Registration Example and Results
,
(16)
),
B. Exposure Difference Estimation
One can notice that the piecewise linear model established in
(3) is not parameterized with respect to our parameter of interest,
, which quantifies the exposure difference we seek between
images and . Instead, the estimate of , which will be denoted
, is obtained as the following average:
(17)
is the unicomparaA few points are now addressed. First,
metric function that has been re-expressed so that is a function of and now serves as the parameter, i.e., we’ve written
. however has been replaced with its piecewise linear
which in turn is a function of , knot loapproximation
and slopes
so that
in (17)
cations
. Second, this inrepresents
tegral arose from initially considering the minimization of the
cost function
(18)
with respect to which simply results in
. Rather than obtaining an average over some finite set of
samples of (which naturally leads to the question, “How many
samples of is enough?”), we can integrate over the entire range
, and forego the previous issue. Third,
of , i.e., the interval
note that continually ranges between [0,1) but constants and
in (17) are typically chosen such that is slightly greater than 0
is very sensitive
and is slightly less than 1. This is because
It is now instructive to illustrate range registration results
using a simple example before proceeding to the joint registration work of the paper. In these results, a SONY DCRTV510
camcorder was fixed on a tripod and a series of 5 images were
captured. The field of view in each of these images was the same
but each image was captured at an exposure level twice that of
the preceding image. Fig. 1 illustrates the first and last images
of the captured sequence. Notice that, because of the exposure
difference, it’s much easier to make out objects outside of the
window in Fig. 1(a) when compared to Fig. 1(b). Conversely, the
interior of the room is better seen in the longer exposed image
of Fig. 1(b) when compared to Fig. 1(a).
The range model between images that was utilized was the
“preferred” unicomparametric equation given in [19] which relates the pixel values between images and by
. By plotting the “preferred”
unicomparametric equation for different values of , we can appreciate the nonlinear relationship between pixel values of two
images that differ in exposure. This is illustrated in the comparagram of Fig. 2. Note that parameters and are fixed for
a given camera. Once these are estimated for any camera, they
remain fixed. We found values for our camera of
and
. To solve for the exposure difference between
the two images in Fig. 1, we created a comparagram between
the pixel values in these images and then used an
linear
segment model in (3) with equispaced knots in order to populate matrix and vector , as described in Section II-A so that
of the first
we could solve for the slopes
linear segments using least squares. The was then estimated
and the true value
using (17). The estimated value was
was known to be 16. We compared this value (obtained using
linear least squares) to a nonlinear least squares optimization
rather than the pieceusing the unicomparametric equation
. Here, we also found
wise linear approximation to it,
. Fig. 3 illustrates the comparagram plot of the data of
and
have
our two images of Fig. 1 and the best fit
been superimposed on this plot.
The values listed in Table I describe the estimates obtained
by registering the image in Fig. 1(a) with the other four images
CANDOCIA: JOINTLY REGISTERING IMAGES IN DOMAIN AND RANGE BY PIECEWISE LINEAR COMPARAMETRIC ANALYSIS
413
^ = 13:9
Fig. 4. Range estimated images computed from Fig. 1(a). (a) Using k
obtained from g (f ). (b) Using the a priori known true k = 16.
TABLE I
COMPARISON BETWEEN TRUE AND ESTIMATED VALUES
THE CAPTURED SEQUENCE
Fig. 2. Plot of “preferred” unicomparametric function
exposure values k (with a, c fixed).
g (f )
OF
k
FROM
for different
utilized does a good job of bringing Fig. 1(a) into range correspondence with Fig. 1(b). This is seen in Fig. 4(a) where the
image of Fig. 1(a) has had its exposure adjusted using the “pre.
ferred” unicomparametric model with the estimated
For comparative purposes, Fig. 4(b) illustrates the exposure-ad. Nojusted image of Fig. 1(a) using the a priori known
tice that the resulting images are practically indistinguishable
and now closely resemble the captured image in Fig. 1(b).
D. An Alternate Form
To facilitate the derivation of the joint registration in the next
section, we will re-express (3) into a more compact form. Before doing this, it is useful to define a set of orthogonal “basis
images” in range which, in essence, indicate those spatial coordinates whose pixel values are within a range interval of interest:
for all
(19)
otherwise
Fig. 3. Comparagram and comparametric functions for the images in
Fig. 1. Comparagram data is indicated by shading in the figure where darker
shading indicates a higher density of samples. Superimposed are the estimated
piecewise linear comparametric function g (f ) indicated as a solid line and
the nonlinear comparametric function g (f ) indicated as a dashed line.
in our five image “range sequence.” The results are seen to be
strikingly similar regardless of whether the unicomparametric
model or its piecewise linear approximation was used. Notice
that the “preferred” model being utilized approximates the camcorder’s response function well but it is not a highly accurate
model as the comparagram’s data indicates a saturation of image
pixel values (here normalized to the interval [0,1)) at around
0.93. This is the reason for the error difference between the true
and estimated values of in Table I. Nonetheless, the model
and where the dependence of image on the
for
photoquantity and spatial coordinates [that have purposely
been omitted in (3)–(7)] have been explicitly written for the sake
of clarity in defining (19). Also, recall the ’s in (19) represent
the knot locations into which is partitioned. Using (19) and
on and
dropping, once again, the dependencies of and
for notational simplicity we can express the piecewise linear
function of (3) as
(20)
where represents multiplication for any spatial coordinate in
our image; otherwise, it would represent element-wise matrix
multiplication if the set of spatial coordinates of our image is
414
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
viewed as a matrix to be operated on. By substituting (7) into
(20) and simplifying, we arrive at a compact form for
in terms of the free parameters
:
and using the pseudo-perspective
model described in [13], we can express the spatial displacement as
(21)
where the ’s are nonlinearly related to the eight parameters
describing the perspective mapping. Also, note
and
where
(28)
(22)
is approximated as the time differ. By approxience between frames and will be denoted
mating (27) with the piecewise linear model of (21) and utilizing
the approximations mentioned above, (26) can be rewritten as
III. JOINT DOMAIN AND RANGE REGISTRATION
A. Models Used and Their Group Structure
The process of jointly estimating the domain and range transformations between two images that is presented here utilizes
the eight-parameter homographic, or perspective, mapping
(23)
(29)
(24)
on and and that of
where the dependence of , , and
and on have been dropped for notational conciseness.
Note that
in the domain and the single parameter range mapping
which describes the change in exposure between images. The
group structure of these domain and range models allows for a
simple computing of the composition of joint domain and range
transformations, specifically that
(25)
,
where
, and scalar
and we also have
.
Thus, knowledge of transformations between consecutive pairs
of images suffices to create a composite image.
B. Joint Optimization
and
be two images, captured at different
Let
times, that are related by a rotation, panning, tilting, zooming
and/or exposure change, all about the camera’s optical center
(we can equivalently have arbitrary camera motion along with
exposure change when imaging a planar surface). The objective
is to jointly determine the domain and range transformations
that register with . To this end, the cost function to be minimized is
(30)
can be approximated with the difand that
ference operator in the -direction. A similar argument
. The camera response function implemented
holds for
here is the “preferred” function described in [19] which is
and for which the partial derivative of
in (30) is
.
We now list all free parameters to estimate in a vector and
paramperform the minimization of with respect to the
eters in
. To do so, we find the
partial derivative of in (29) with respect to each of the
parameters in and equate each partial derivative to 0. This reequations in as many unknowns:
sults in a system of
(31)
where
(32)
(26)
with
and now
and where
where is minimum at that
both images are registered in domain and range, respectively.
Let us define
so that
denotes the domain-registered image.
We can then see that
(27)
is now the requirement for a jointly registered image.
Using a first order Maclaurin series expansion we can
write
,
, and
,
. We
have that
,
,
. The elements
are shown in (33)–(35) at the bottom
in the submatrices of
of the next page. Note that since
, we write to denote elements that aren’t necessary to fill in. Also, the denotes summation over all and . After estimating from (31)
using least squares, we must relate the parameters
to the eight parameters of the domain’s perspective transformation. This portion is accomplished as described in [13]. Also,
of our piecewise linear
we must relate the slopes
model to the exposure estimate . This is done as described in
Section II-B.
CANDOCIA: JOINTLY REGISTERING IMAGES IN DOMAIN AND RANGE BY PIECEWISE LINEAR COMPARAMETRIC ANALYSIS
C. Registration Process
As stated earlier, the adjacent images in a sequence will be
registered pair-wise and then all of them will be brought to a
final domain and range reference using our models’ group structure as described in Section III-A. So, given two images and
, we proceed to iteratively solve for the true parameters of our
415
domain and range models, i.e., the , , and in (23) and
(24). At each iteration step of the pair-wise registration, we first
solve for vector in (31) and then we relate these approximate
model parameters to the true parameters of our models as described in Sections II and III. To begin the iterations, we have
used phase correlation [29] to estimate the translational component between and . That is, initially, we set to the identity
(33)
and
..
.
..
.
..
.
(34)
and
(35)
416
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
Fig. 5. Sequence of images to register in both domain and range. The camera panned across the room from left to right yielding the time-ordered images (a)–(s).
Image (i) in the figure denotes the one whose domain and range coordinates are used as the reference for the composite image created in Fig. 6.
matrix (no rotation or scaling), to the translation estimate from
the phase correlation, and to zero (no perspective transformation). Also, we initially assume no exposure difference between
images so is set to 1.
Using these initial values we solve for a new set of model
parameters and, using (25), we update our true parameter
estimates. This iteration process continues until the percent
decrease among successive iterations for the sum of squared
and
is
differences between the parameter transformed
smaller than some preset value. We have not used a hierarchical
or multistage approach to performing the registration in this
paper as done in many other registration approaches, but this
approach is certainly amenable to that. Finally, once all adjacent
images of our sequence are pair-wise registered, we choose
one of the images in the sequence to serve as the “global”
domain and range reference. Thus, using (25), we can use the
local (pair-wise) domain and range transformations computed
to determine the global transformations that take us to our
desired coordinates in domain as well as range for each image
in the sequence considered.
IV. JOINT REGISTRATION EXPERIMENTAL RESULTS
To test the approach presented, a sequence consisting of 19
images, each of size 240 320, was used. These images are
depicted in Fig. 5. Please note that the camera, its parameters
CANDOCIA: JOINTLY REGISTERING IMAGES IN DOMAIN AND RANGE BY PIECEWISE LINEAR COMPARAMETRIC ANALYSIS
417
TABLE II
REGISTRATION DATA FOR THE IMAGE SEQUENCE IN FIG. 5
and the models used were the same as for the experiment described in Section II-C. Also, this sequence was chosen to illustrate the algorithm’s performance in the presence of reasonable
model errors. That is, as the camera was hand-held and panned
roughly about its optical center, a reasonable approximation for
relating the spatial coordinates between images in the sequence
is the homographic transformation. In range, our use of Mann’s
“preferred” comparametric equation also yields a reasonable approximation to the camera’s actual response—please refer to
Fig. 3 for an example. The sequence in question is of a room
containing a sliding door with a brightly lit exterior. The camera
begins inside the room and proceeds to pan across the sliding
door from left to right, i.e., the time-ordered sequence begins
with image (a) of Fig. 5 and continues until image (s). Because
the inside of the room is dark relative to the brightly lit exterior,
the camera automatically initially adjusts its exposure to capture the interior of the room in the top left image. As the camera
begins to pan across the sliding door, the exposure time is automatically diminished so that now objects outside the sliding
door can be discerned. However, this now leads to a “darkening”
or underexposure of objects inside the room.
Table II illustrates data obtained from the registration of the
image pairs in this sequence. Note that the table contains results using domain-only registration as well as joint domain and
range registration. The domain-only registration is easily performed within the framework presented by simply fixing in
simply
(27). That is, as an exposure isn’t being estimated,
represents one of the two images being registered. As there are
no parameters being associated with , any derivatives associated with it are zero and the resulting optimization is modified accordingly. Please note that image pair 1 referred to in
Table II consists of images (a) and (b) of Fig. 5, image pair 2
refers to images (b) and (c), and so on. In both registration approaches, the mean absolute difference (MAD) between pixels
of the registered image pairs is shown. Note that for all pairs, the
MAD for domain-only registration is greater than when the joint
registration is performed. This is intuitively satisfying because
we expect a better match between pixels in the case when exposure differences—that we know exist in this sequence—are
accounted for. This exposure difference, denoted by , is also
listed in the table. This value describes that additional exposure
into range correspondence with
required to bring image
image . For example, consider image pair 1. In order to register
image (b) in Fig. 5 with image (a), we need to adjust the exposure of image (b) by a factor of 1.37, i.e., image (b) would have
needed to be exposed 1.37 times more than it was in order to
have it registered in range with image (a). In general, we can see
that, as our camera panned across the sliding door, the exposure
times between adjacent images continued diminishing, just as
we would expect. We can see that, using our joint model’s group
structure of (25), to bring image (s) in our sequence into range
correspondence with image (a), we would need an exposure 653
times greater than that used in capturing image (s) (just multiply
all the values of found in Table II). The additional accuracy
in registration from the algorithm does come at a computational
price. Notice that the domain-only registration process requires
an average of about 5 seconds per image pair. This is in contrast to the joint registration process whose registration time per
image pair varied between approximately one to two minutes in
time. The algorithms were written and run using the Matlab 6.0
R12 package in the Windows 2000 environment on a Dell 330
Precision PC with a 1.5 GHz Pentium IV processor.
To visually illustrate the results of the presented approach,
the jointly registered sequence in Fig. 5 is shown in Fig. 6. Note
that the domain and range reference being utilized is that of
image (i) in Fig. 5. The gray lines superimposed on the figure are
for illustrating the domain mapping that was computed for each
image and how each of these images where overlain to produce
the final composite. We can also notice that since these images
were also registered in range, the images whose interior is darker
(those captured after our reference image was) have now had
their exposures synthetically increased so that they can match
with that of the reference image. These now appear brighter in
the final composite. Accordingly, the images captured before the
418
IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 4, APRIL 2003
Fig. 6. Composite image resulting from the joint registration of the sequence in Fig. 5. The gray lines illustrate the mappings performed to obtain domain
registration. The global reference used, both for domain and for range registration, is image (i) depicted in Fig. 5.
reference image was (those with greater exposure times than the
reference) have had their exposures decreased to match with that
of the reference image. These images appear darker in the final
composite.
reviewers for providing helpful comments that were used in
preparing the final manuscript.
REFERENCES
V. SUMMARY AND CONCLUSIONS
An approach has been described whereby the joint registration of images in domain as well as range is accomplished
using piecewise linear comparametric analysis. The attractiveness of this formalism lies in its ability to convert an
inherently nonlinear estimation problem in both domain and
range into one involving linear estimation. In both the domain and range models, an intermediate set of parameters
is solved for using least squares and these are then easily
related to the true parameters we seek to estimate. The method
for estimating the exposure difference between images used
a constrained piecewise linear model for the camera’s comparametric function. The accuracy with which the piecewise
linear function approximates the comparametric equation depends on the number of segments one selects to use. We
have empirically found that six equispaced segments provides
good results while keeping the computational complexity low.
Results indicate that the estimation of the exposure using the
linear least squares approach presented here is as effective as
estimation using nonlinear least squares with the original comparametric function. Experimental results have been conducted
over many sets of images and illustrate that this procedure
works well.
ACKNOWLEDGMENT
The author would like to thank the Telecommunications
and Information Technology Institute at Florida International
University for supporting this work as well as the anonymous
[1] L. G. Brown, “A survey of image registration techniques,” ACM
Comput. Surv., vol. 24, no. 4, pp. 325–376, Dec. 1992.
[2] E. M. Mikhail, J. S. Bethel, and J. C. McGlone, Introduction to Modern
Photogrammetry. New York: Wiley, 2001.
[3] L. G. Shapiro and G. C. Stockman, Computer Vision. Englewood
Cliffs, NJ: Prentice-Hall, 2001.
[4] J. B. A. Maintz and M. A. Viergever, “A survey of medical image registration,” Med. Image Anal., vol. 2, no. 1, pp. 1–37, 1998.
[5] R. Schultz and R. Stevenson, “Extraction of high-resolution frames from
video sequences,” IEEE Trans. Image Processing, vol. 5, pp. 996–1011,
June 1996.
[6] M. C. Chiang and T. E. Boult, “Efficient image warping and super-resolution,” in Proc. IEEE Workshop on Applications of Computer Vision
(WACV ’96), 1996, pp. 56–61.
[7] U. Dhond and J. Aggarwal, “Structure from stereo—A review,” IEEE
Trans. Syst., Man, Cybern., vol. 19, no. 6, pp. 1489–1510, 1989.
[8] S. Baron and W. Wilson, “MPEG Overview,” SMPTE J., pp. 391–394,
June 1994.
[9] M. Irani, S. Hsu, and P. Anandan, “Video compression using mosaic
representations,” Signal Process.: Image Commun., vol. 7, pp. 529–552,
1995.
[10] H. S. Sawhney and S. Ayer, “Compact representation of videos through
dominant multiple motion estimation,” IEEE Trans. Pattern Anal. Machine Intell., vol. 18, no. 8, pp. 814–830, 1996.
[11] H. Y. Shum and R. Szeliski, “Systems and experiment paper: Construction of panoramic image mosaics with global and local alignment,” Int.
J. Comput. Vis., vol. 36, no. 2, pp. 101–130, 2000.
[12] M. Irani and S. Peleg, “Improving resolution by image registration,”
Comput. Vis., Graph., Image Process.: Graph. Models Image Process.,
vol. 53, no. 3, pp. 231–239, 1991.
[13] S. Mann and R. Picard, “Video orbits of the projective group: A simple
approach to featureless estimation of parameters,” IEEE Trans. Image
Processing, vol. 6, pp. 1281–1295, Sept. 1997.
[14] S. Baker and T. Kanade, “Superresolution optical flow,” Robotics
Institute, Carnegie Mellon Univ., Pittsburgh, PA, Tech. Rep.
CMU-RI-TR-99-36, 1999.
[15] H. S. Sawhney and R. Kumar, “True multi-image alignment and its applications to mosaicing and lens distortion correction,” IEEE Trans. Pattern Anal. Machine Intell., vol. 21, pp. 235–243, Mar. 1999.
CANDOCIA: JOINTLY REGISTERING IMAGES IN DOMAIN AND RANGE BY PIECEWISE LINEAR COMPARAMETRIC ANALYSIS
[16] S. Mann and R. W. Picard, “On being ‘undigital’ with digital cameras:
Extending dynamic range by combining differently exposed pictures,”
in Proc. of IS&T 48th Annual Conference, May 1995, pp. 422–428.
[17] P. E. Debevec and J. Malik, “Recovering high dynamic range radiance
maps from photographs,” in Proc. SIGGRAPH, 1997, pp. 369–378.
[18] T. Mitsunaga and S. Nayar, “Radiometric self calibration,” in Proc. IEEE
Conf. Computer Vision and Pattern Recognition, 1999, pp. 374–380.
[19] S. Mann, “Comparametric equations with practical applications in
quantigraphic image processing,” IEEE Trans. Image Processing, vol.
9, pp. 1389–1406, Aug. 2000.
[20] S. Mann and R. Mann, “Quantigraphic imaging: Estimating the camera
response and exposures from differently exposed images,” in Proc. IEEE
Conf. Computer Vision and Pattern Recognition, 2001, pp. 842–849.
[21] R. Fattal, D. Lischinski, and M. Werman, “Gradient domain high
dynamic range compression,” ACM Trans. Graph., vol. 21, no. 3, pp.
249–256, 2002.
[22] S. K. Nayar and T. Mitsunaga, “High dynamic range imaging: Spatially
varying pixel exposures,” in Proc. IEEE Conf. on Computer Vision and
Pattern Recognition, vol. 1, June 2000, pp. 472–479.
[23] C. ChefD’Hotel, G. Hermilloso, and O. Faugeras, “A variational approach to multi-modal image matching,” in Proc. IEEE Workshop on
Variational and Level Set Methods in Computer Vision, 2001, pp. 21–28.
[24] A. Roche, A. Guimond, J. Meunier, and N. Ayache, “Multimodal elastic
matching of brain images,” in Proc. of 6th Eur. Conf. on Computer Vision, Dublin, Ireland, June 2000.
[25] Y. Y. Schechner and S. K. Nayar, “Generalized mosaicing,” in Proc.
IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, 2001, pp.
17–24.
[26] M. Aggarwal and N. Ahuja, “High dynamic range panoramic imaging,”
in Proc. IEEE Int. Conf. Computer Vision, vol. 1, 2001, pp. 2–9.
[27] S. Mann, “Pencigraphy’ with AGC: Joint parameter estimation in both
domain and range of functions in same orbit of the Projective-Wyckoff
Group,” in IEEE Int. Conf. Image Processing (ICIP ‘96), vol. 3, Los
Alamitos, CA, 1996, pp. 193–196.
419
[28] E. Kreyszig, Advanced Engineering Mathematics, 7th ed. New York:
Wiley, 1993.
[29] B. S. Reddy and B. N. Chatterji, “An FFT-based technique for translation, rotation and scale-invariant image registration,” IEEE Trans. Image
Processing, vol. 5, pp. 1266–1271, Aug. 1996.
Frank M. Candocia (M’94) received the B.S. and
M.S. degrees in electrical engineering in 1991 and
1993, respectively, from Florida International University, Miami. In 1998, he received the Ph.D. degree
in electrical and computer engineering from the University of Florida, Gainesville, where his work focused on optical image superresolution. During his
doctoral studies, he also investigated superresolution
of synthetic aperture radar imagery at MIT Lincoln
Labs.
From 1998 to 2000, he was a Senior Systems
Engineer for Raytheon Systems Company, Bedford, MA, where he was
responsible for the design and implementation of the search and acquisition
portion of the National Missile Defense program’s X-band radar. He is
currently an Assistant Professor with the Department of Electrical and
Computer Engineering, Florida International University, and is the founder
of the image understanding lab (http://iul.eng.fiu.edu). His research interests
include all phases of the image understanding problem as well as the tools
needed toward their eventual solution.
Dr. Candocia has been the recipient of several scholarships and fellowships
including a University of Florida graduate minority fellowship and a Motorola
fellowship.