Change detection using mean-shift and outlier-distance
metrics
Joshua Zollwega , Ariel Schlamm,a David B. Gillis,b and David Messinger,a
a Rochester
Institute of Technology, Center for Imaging Science, Digital Imaging and Remote
Sensing Laboratory, 54 Lomb Memorial Drive, Rochester, NY, 14623, USA
b U.S.
Naval Research Lab, 4555 Overlook Ave SW, Washington, DC 20375
ABSTRACT
Change detection with application to wide-area search seeks to identify where interesting activity has occurred
between two images. Since there are many different classes of change, one metric may miss a particular type of
change. Therefore, it is potentially beneficial to select metrics with complementary properties. With this idea
in mind, a new change detection scheme was created using mean-shift and outlier-distance metrics. Using these
metrics in combination should identify and characterize change more completely than either individually. An
algorithm using both metrics was developed and tested using registered sets of multispectral imagery.
Keywords: multispectral, change detection
1. INTRODUCTION
Change detection in remote sensing is the process of identifying the differences between two or more images. In
the simplest case, two perfectly registered, single band images can be subtracted pixel-by-pixel. High values in
the result correspond to changes and low values to static regions. In remote sensing applications with spectral
imagery, this method is not practical in most situations, but has laid the basis for change detection in spectral
imagery. A typical approach to spectral image change detection involves some type of statistical prediction used
to bring images collected at different times and under different conditions into the same domain. Schaum and
Stocker (2004) predict what one image should look like at the later time by using methods based on covariance
equalization followed by the chronochrome algorithm or matched change detection to determine whether a
change occurred. Chronochrome and matched change detection are very sensitive to misregistration errors,
but using covariance equalization as the predictor reduces the number of false alarms caused by misregistration
error.1, 2 Meola and Eismann (2009) extend standard hyperspectral change detection routines to include multiple
reference scenes when predicting what the detection image may look like. The reference images used were taken
under varying illumination conditions in order to better describe what the detection image could look like if an
illumination change took place. This is important because it may help distinguish illumination changes from
interesting changes in the detection process.3 In general, two or more images are registered and the difference
between the two images is estimated on a pixel-by-pixel basis in order to detect change between spectral images.
Pixel-based subtraction methods are sensitive to some amount of misregistration error. A thorough review of
statistical anomalous change detection algorithms commonly applied to spectral imagery can be found in Theiler
(2008).4
Often change detection, and spectral image analysis in general, considers each pixel on an individual basis.
When considering the task of large area search, this can be daunting for a visual analyst and computationally
expensive. Recently, many algorithms for complexity estimation,5, 6 anomaly detection,7 interest quantification,8
and even change detection9–12 have taken a different approach. These algorithms quantify the spectral distributions of spatially local tiles in terms of volume,5, 6 dimension,13–15 and other measurable attributes. This
information is recorded for each tile in an image and used to decide whether anomalies or multiple material
classes exist within the particular tile relative to the other tiles in the scene. For change detection, these metrics
can simply be subtracted for corresponding tiles and used to indicate the relative amount of change in the tile.
Further author information: Joshua Zollweg: [email protected]
Tile-based approaches to change detection are more robust to misregistration error than pixel-based approaches.
Measuring specific attributes of the spectral distribution can also be more robust against slight illumination
differences as the whole distribution moves in the same fashion under this type of change. These methods are
well suited to the task of large area search as the result produces a simple map that can be used to cue an analyst
and also easily executed in a parallel processing environment.
Small scale, or anomalous changes, can occur between two images taken at any time interval. If significant
time has passed between the collection of the images, large scale changes may also occur. Large scale changes
are many pixel changes that can significantly effect the size, shape, and location of the spectral distribution.
An example large scale change is a tile containing a forest in one image that has been leveled into a parking
lot in the second image. Anomalous change detection algorithms often miss large scale changes as a result
of the prediction step because these changes often manifest in the background distribution of the data. Postclassification or cluster-level change detection16, 17 is one approach to detecting both large and small scale changes
with a single algorithm.
The methodology introduced in Section 2 uses the spatial tile-based approach to estimated metrics based
on the spectral distribution of the data. In order to quantify large and small scale changes, two metrics that
quantify these changes individually are developed in Section 3. Neither of these metrics rely on prediction or
statistical assumptions about the data distribution from one image to the next. The multispectral data used in
this analysis is described in Section 4. Results of the algorithm against these data are presented in Section 5.
2. METHODOLOGY
Figure 1. Generalized change detection methodology used in this analysis.
The change detection methodology shown in Figure 1 was developed to assist an analyst searching for change
between two images.9, 10 The general methodology requires two registered images as input and produces a scaled
detection map where tile brightness corresponds to the amount of change between the two images. Metrics are
estimated for each image tile and subtracted with the corresponding metric from another image to produce the
change metric. Each tile is treated independently of the rest of the image. The metric scores relate to the relative
amount of change across the image and may not be easily comparable to a different image from a different sensor.
The metrics are scaled by the maximum score in the image to ensure all of the scores are between zero and one.
For this research, two metrics which quantify different types of change are used, resulting in two detection maps.
These can be used individually or used in conjunction with a single band of the image to produce a false color
map where the brightness indicates the degree of change and the color indicates the type of change. Combining
a tiling approach with estimating distribution metrics make this methodology more robust to misregistration
error and even slight illumination differences between the images.
3. APPROACH
The Shift-Outlier (ShOut) algorithm is a two-metric approach to change detection, consisting of mean-shift and
outlier distances, that measures large and small scale changes between two images. These distances are illustrated
in Figure 2. The mean-shift distance is calculated by estimating the distance between the distribution means of
two registered image tiles. Mean-shift is a measure of broad, large scale changes within a tile. The magnitude
of spectral difference and spatial proportion of changed pixels in a tile from the originals are the two factors
that affect how much the mean has shifted. However, if changes involve a small number of pixels relative to the
number of pixels in a tile, the mean may not shift significantly and the change may not be detected. To avoid
missing these small-scale changes, the outlier-distance is used as a complimentary metric. For corresponding
tiles, the distance in spectral space from the most distant pixel vector to the distribution mean is calculated.
The difference between these two outlier distances is used as the outlier-distance metric. With outlier-distance,
a large response is possible even when only one pixel has changed between the image tiles. Though a small
number of spectrally different pixels may not move the mean, they may be outliers with respect to the rest of the
distribution. However, it is possible for a single pixel to be changed such that it is not further than the original
most outlying pixel. In this case, a change may not be detected. Each of these metrics produce an individual
detection map. However, the strength in this approach comes from combining these complimentary metrics into
a single, false color detection map
Figure 2. Notional example of ShOut metrics.
4. DATA
Initial application of this change detection methodology to large area search was done on 4-band Quickbird
images of the Indonesia coastline before and after the tsunami of 2004, shown in Figure 3. The tsunami caused
easily visible, significant damage to the coastline and some inland areas. Additionally, there are many examples
of more subtle and small scale changes in the images. No precise ground truth is available, but visual inspection
of the high spatial resolution data is done to locate and evaluate changes. Additionally, WorldView 2 (WV2)
data of the Rochester Institute of Technology (RIT) campus taken in June and September, 2010 was provided
by DigitalGlobe as part of the 8-Band Challenge. The RIT campus was desired as the location because known
construction and changes have taken place on and around the campus during that time frame, providing some
amount of ground truth. However, many of the areas under construction had not been completed during the
3 months between collections. The area of the provided data used is shown in Figure 4. The two images were
taken under different viewing and illumination configurations, resulting in glint off the Genesee River and every
small pond in the June image. The Rochester International Airport is located in the upper left and the RIT
campus is in the lower left. The area below campus includes some agriculture. Between the RIT campus and
the airport is a large public park and golf course. The right side of the image is mostly residential, suburban,
and urban areas. This imagery has examples of large and small scale changes.
(a) Before
(b) After
Figure 3. Indonesia before (a) and after (b) the 2004 tsunami.
5. RESULTS
The general methodology and the ShOut algorithm are applied to two multispectral image pairs. As stated
earlier, each metric is estimated for the individual image tiles, in this case 50 × 50 pixel tiles. The tiles are
scored independently and scaled such that the tile with the most significant change in each algorithm is given a
score of 1. The scores are used to produce a scaled brightness map where the tile brightness corresponds to the
magnitude of the metric score. The results for each metric are presented individually and as false color imagery.
This is shown in Figure 5 for the Quickbird imagery of Indonesia. In the false color display, a single band
from the multispectral image is used in the green channel. The red and blue channels are the scaled brightness
maps corresponding to the mean-shift and outlier-distance metrics, respectively. For this reason, the color and
brightness of a particular tile indicates the degree and type of change. A dark or green tile is only showing the
background image, meaning no change has occurred. A primarily red tile indicates a change detected in the
mean-shift channel, or a large scale change. A blue tile indicates a small scale or anomalous change has occurred.
A magenta tile indicates a change has been detected by both metrics. The tile brightness corresponds to the
magnitude of the change metric and therefore the degree of the change measured by the particular metric(s).
A brighter tile has a more significant change than a darker tile. Because each metric is measuring different
phenomenology and different types of change between the spectral distributions, a significant change in one
metric is not necessarily a significant change in another metric.
Figure 5(a) shows the mean-shift detection plane for the Quickbird imagery of Indonesia. In general, most
of these changes occur just inland where vegetation areas were destroyed. The tile brightness, or amount of
change, tapers off as distance inland increases because less damage occurred in these regions. The water in the
bay areas does have some mean-shift change, though significantly less than the land. This is because much of the
sand, vegetation, and dirt was likely swept into the bay as the water subsided, changing the water properties and
constituents slightly. On the other hand, the outlier-distance detection plane in Figure 5(b) highlights the coast,
where the most significant damage occurred. This is because these tiles have a combination of water and land
(a) June 2010
(b) September 2010
Figure 4. True color image of WV2 data taken over the Rochester International Airport and RIT campus in June (a) and
September (b).
damage in them. The water spectra changed slightly from the damage, but the land spectra changed significantly.
Most of the completely inland and open water tiles are dark. Figure 5(c) shows the false color visualization of
both detection planes at once, easily indicating both type and severity of the change. Note that the line of bright
tiles across the bottom of every image is due to zero-padding around the image after georectification.
(a) Mean-Shift
(b) Outlier-Distance
(c) ShOut
Figure 5. Mean-shift (a), outlier-distance (b), and ShOut (c) results against the Quickbird imagery of Indonesia.
Figure 6(a-e) shows a region from the upper right hand side of the image. This is an area which is partially
in a cloud shadow. Though this change is not necessarily of interest to an analyst, it is a strong spectral change
that manifests in both detection planes, appearing magenta in the false color map. Because those tiles in shadow
are significantly darker, the mean obviously shifts significantly. However, as a result of the mean shifting so
much due to the illumination, the distance to the outlying pixel also increases. Figure 7(a-e) shows one of the
brightest responses in the mean-shift detection plane. This is an area containing a large scale change due to
clouds in one image. There is no response in the outlier-shift detection plane over these tiles because no true
small scale changes have occurred.
(a) Before
(b) After
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 6. Area showing cloud shadow effects on ShOut metrics.
(a) Before
(b) After
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 7. Area showing cloud effects on ShOut metrics.
(a) Before
(b) After
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 8. Area over ocean containing buoys and a boat before the tsunami and none after.
In the lower left hand portion of the image before the tsunami, there is a line of buoys in the water near a
boat. After the tsunami, the boat is no longer in that area and all of the buoys are washed away. This region
is shown in Figure 8. This is a change that is not obvious by looking just at the true color imagery and might
be easily missed by an analyst. Each buoy is just a few pixels and do not significantly effect the location of
the mean. As a result, none of these changes are detected by the mean-shift metric. These buoys are detected
by the outlier-shift metric because they are significantly far away from the mean, which is dominated by the
background water spectra. The boat is brighter than the buoys and larger in terms of pixels, resulting in the
tile containing the boat to be brighter than those only containing buoys.
The region in Figure 9 shows a damaged coastal region. These changes are immediately obvious to the
unaided eye, however both large and small scale changes occur in this area. The water reflectance changes due
to the addition of land particles after the tsunami recedes, causing a subtle, large scale change that manifests
in the mean-shift metric but not at all in the outlier-shift metric. The coastline predominantly undergoes small
scale changes and is seen in the outlier-shift detection plane. The inlet from the ocean has both large and small
scale change. The large scale change in the mean-shift metric is due to the different water and land reflectance
in the area. The small scale change in the outlier-shift metric is likely caused by the destruction of the farming
pools. The vegetation area remains mostly unchanged and is easily visible in the false color, ShOut detection
map, which highlights the attention to the coast and inlet areas that underwent the most significant damage.
(a) Before
(b) After
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 9. Severely damaged coastal area.
The false color result of applying the ShOut methodology using 50 × 50 pixel tiles to the WV2 imagery
collected over the RIT campus in June and September, 2010, is shown in Figure 10. This imagery is a more
complicated test for many reasons. WV2 is an 8-band sensor, providing more spectral information about the
materials on the ground. The scene itself is much more complex. Overall, about half of the WV2 image area is
detected as having some sort of change, mostly small scale. Most of this image is of a heavily populated area
that constantly undergoes a significant amount of small scale change, shown in the blue channel of the false color
image. While this seems like a large proportion of the image, most of this area corresponds to the developed
regions of the image where construction and roadway paving/sealing are typically done during the short summer
season in Rochester, NY. Additionally, this is a heavily travelled area including many main roads. Car locations
on the roads and in parking lots are expected small scale changes and regularly occurring. The outlier metric
also detects many small scale changes including tents and swimming pools in backyards. Both the mean and
(a) Mean-Shift
(b) Outlier-Distance
(c) ShOut
Figure 10. Change detection results for WV2 imagery over the RIT campus.
outlier shift metrics find a large portion of the image to have little to no change over the Genesee Valley Park in
the center of the image. The handful of tiles within the park that are detected are small scale changes found by
the outlier metric that correspond to vehicles in a parking lot and a small structure in the park.
The red channel of the false color image in Figure 10(c) shows the mean shift metric results or large scale
changes. The strongest of these changes are due to the different illumination and collection geometries between
June and September. The June image was taken in the direct reflection direction, causing the Genesee River,
manmade ponds, and highly reflective roofs to be very bright. These significantly effect the mean of an image tile
and as a result manifest in the mean-shift detection plane. The other large scale changes are mostly found over
large areas of vegetation. The lower valued large scale changes are of vegetation that is green in both images,
such as the park. This change is likely due to slight differences in reflectance from varying degrees of vegetation
health and moisture content over the summer. An agricultural area in the lower left corner of the image in
Figure 10 is not planted in the June image, but lush and green in the September image. The subset of this
area in Figure 11 shows the high response from the mean shift metric over these fields. This area is next to two
small commercial complexes which have undergone small scale changes due to paving and cars in the parking
lot, which are detected by the outlier shift metric. Note that the single tile completely contained within the
roof of the building has not been detected because the roof of the building has not changed beyond illumination
conditions.
(a) June
(b) September
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 11. Change detection results for an area containing agricultural fields and business complexes.
This scene includes the Rochester International Airport. A subset of this area, shown in Figure 12, is centered
over one of the terminals. This area contains very little large scale change, demonstrated by low response in the
mean shift detection plane. Five tiles over the terminals appear blue in the false color map, corresponding to
small scale changes. These are due to airplanes being parked in one gate location in one image and different
airplanes in different locations in the second image. The brightest blue tile corresponds to a truck parked near
the terminal in the June image, not an airplane. Because the outlier-distance metric does not depend on the
spatial size of the change, the brightness does not correspond to the number of pixels that changed. This truck
is more spectrally different from the background of the tile than the airplanes, possibly due to the illumination
conditions in the June image, causing it to be the strongest change detected in this small area.
(a) June
(b) September
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 12. Change detection results for the Rochester International Airport Terminal.
This particular location was chosen for the collect because a significant amount of known change occurred
in and around the RIT campus. In June, RIT was hosting the Empire State Games qualification rounds. As a
result, a large field near the main entrance of campus, shown in Figure 13, was full of tents in June, but empty
in the September image. This area is detected strongly by the outlier metric but also found using the mean shift
metric, indicated by the slight purple cast of the area in Figure 13(e). This change manifests in the mean shift
detection plane because there are enough tent pixels in each tile to sufficiently change the mean spectrum. The
significant spectral reflectance difference between the white tents and vegetation background cause high scores
in the outlier-distance metric.
(a) June
(b) September
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 13. Change detection results for an area of the RIT campus holding the Empire State Games in June, 2010.
Many areas of active construction were also taking place during the summer. One construction site in
particular, shown in Figure 14, is located on the western edge of campus. It is clear from the true color imagery
that a small parking lot was replaced by a large parking lot. This is detected by both the mean-shift and outlierdistance metrics as the background has changed from mostly vegetation to paved roadway and from many parked
cars to few. In the lower right area, a few small buildings were constructed that are mostly detected by the
outlier-distance metric because of their extreme spectral difference and slightly detected by the mean-shift metric
because the number of pixels is substantial enough to effect the tiles’ spectral mean. Some of the construction
in this area was not completed by the September collection. As a result, some areas were detected as having no
change because the spectral distribution of both the early and late construction phases are very similar.
(a) June
(b) September
(c) Mean-Shift
(d) Outlier-Distance
(e) ShOut
Figure 14. Change detection results for an area of the RIT campus that underwent construction during the summer of
2010.
6. CONCLUSION
Typical approaches to change detection are based on standard spectral analysis assumptions, such as multivariate
normality, and pixel level differencing. Although these methods historically provide adequate results, as high
spatial resolution, multispectral sensors capture large area scenes (such as the DigitalGlobe WV2), the standard,
statistics-based approaches are not optimal. Presented here is a a general change detection methodology based
on image tiles and the ShOut change detection algorithm. Using image tiles and distribution based metrics
makes this methodology more robust against misregistration errors and some amount of illumination effects.
This algorithm limits the assumptions placed on the spectral distribution and estimates two data driven metrics:
mean-shift and outlier-distance. These metrics can be classified as indicators for large scale or small scale change,
respectively. For each image tile, the two metrics are estimated based on the spectral distribution of the tile.
These metrics are fused with the original image data to create a false color image where the brightness and
color of an image tile corresponds to the magnitude and type of change that occurred between the images. The
strength in this methodology for large area search comes from the complimentary metrics that can immediately
indicate to an analyst the degree and type of change that has occurred between the two images.
REFERENCES
[1] Schaum, A. P. and Stocker, A., “Hyperspectral change detection and supervised matched filtering based
on covariance equalization,” Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral
Imagery X 5425(1), 77–90, SPIE (2004). [DOI:10.1117/12.544026].
[2] Schaum, A. and Stocker, A., “Advanced algorithms for autonomous hyperspectral change detection,”
in [Applied Imagery Pattern Recognition Workshop, 2004. Proceedings. 33rd], 33–38, IEEE (Oct 2004).
[DOI:10.1109/AIPR.2004.10].
[3] Meola, J. and Eismann, M. T., “Methods for multi-temporal change detection,” Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV 7334(1), 73340L, SPIE (2009).
[DOI:10.1117/12.816121].
[4] Theiler, J., “Quantitative comparison of quadratic covariance-based anomalous change detectors,” Appl.
Opt. 47(28), F12–F26 (2008). [DOI:10.1364/AO.47.000F12].
[5] Messinger, D., Ziemann, A., Basener, W., and Schlamm, A., “Spectral image complexity estimated through
local convex hull volume,” 2nd Workshop on Hyperspectral Image and Signal Processing: Evolution in
Remote Sensing (June 2010).
[6] Messinger, D., Ziemann, A., Basener, B., and Schlamm, A., “A metric of spectral image complexity with
application to large area search,” submitted to TGARS (2010).
[7] Schlamm, A. and Messinger, D., “A euclidean distance transformation for improved anomaly detection in
spectral imagery,” Western New York Image Processing Workshop, IEEE (November 2010).
[8] Schlamm, A., Messinger, D., and Basener, W., “Interest segmentation of hyperspectral imagery,” 2nd
Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (June 2010).
[9] Schlamm, A., Messinger, D., and Basener, W., “Change detection in spectral imagery using point density
estimation,” submitted to JARS (November 2009).
[10] Ziemann, A., Messinger, D., and Basener, W., “Iterative convex hull volume estimation in hyperspectral
imagery for change detection,” Proc. SPIE 7695 (2010). [DOI:10.1117/12.850122].
[11] Albano, J. A., Messinger, D. W., Schlamm, A., and Basener, W. F., “Automatic clustering of multispectral
imagery by maximization of the graph modularity,” Proc. SPIE 8048 (April 2011).
[12] Mercovich, R., Harkin, A., and Messinger, D., “Utilizing the graph modularity to blind cluster multispectral
satellite imagery,” Western New York Image Processing Workshop, IEEE (November 2010).
[13] Schlamm, A., Messinger, D., and Basener, W., “Geometric estimation of the inherent dimensionality of a single and multi-material clusters hyperspectral imagery,” JARS 3, 033527 (Feb 2009).
[DOI:10.1117/1.3133323].
[14] Resmini, R. G., “A tool for the nonparametric characterization of the geometry of spectra in hyperspace,”
Proc. SPIE 7334, 73341S, SPIE (2009). [DOI: 10.1117/12.819312].
[15] Schlamm, A., Resmini, R. G., Messinger, D., and Basener, W., “A comparison study of dimension estimation
algorithms,” Proc. SPIE 7695 (2010).
[16] Canty, M. J., [Image Analysis, Classification, and Change Detection in Remote Sensing ], CRC Press, 2nd ed.
(2010).
[17] Mercovich, R. A., Harkin, A. A., and Messinger, D. W., “Automatic clustering of multispectral imagery by
maximization of the graph modularity,” Proc. SPIE 8048 (April 2011).