Image Representation Alternatives for the Analysis of
Satellite Image Time Series
Corneliu Octavian Dumitru, Gottfried Schwarz, and Mihai Datcu
Remote Sensing Technology Institute, German Aerospace Center (DLR), 82234 Wessling, Germany
[email protected], [email protected], [email protected]
Abstract— Current satellite images and image time series
provide us with detailed information about the state of our planet
as well as about our technical infrastructure and human activities. These images allow us to learn more about local, regional,
and global phenomena and events, including - if interpreted
properly - their causes and effects. In particular, image time series provide specific information about the dynamics of many
processes implicitly contained in our images that need to be unearthed and investigated in detail. A traditional approach toward
this aim is to start with pixel-level or patch-level data analysis for
pixel-based image analysis, followed, if necessary, by subsequent
feature extraction, clustering, classification and semantic labelling in order to generate various types of change maps on different representation levels. The classification step can be supported
by interactive human intervention, or by automated machine
learning strategies to identify higher level objects and their spatial and temporal relationships. The detected relationships can
then be formulated as parameterized rule sets that create higherlevel descriptor sets of the content of the selected images and of
additional external data such as thematic maps or typical dynamics descriptions. As an innovative extension of this traditional
concept, we propose a highly automated approach for application-adapted image content exploration and knowledge extraction. The reason for this strategy is the additional amount and
the precision of semantic relationships and details that we can
assign to an image time series once we know the final application
field and how to embed and access image content within
knowledge graphs.
for each temporal cube layer and perform manually
controlled data analyses versus time.

The approach is adapted to a number of typical predefined individual use cases for application-dependent
knowledge representation and mining. This adaptation
allows systematic case-specific compact representations of image features and knowledge content for the
characterization of satellite image time series of selected types. For instance, the actually selected classification and image clustering parameters are then most appropriate for the given instrument and image type, the
current application field and geographical region, as
well as typical cases of dynamic developments. In addition, we optimize our knowledge retrieval capabilities by adapting them also to the selected type and operating mode of remote sensing instruments (e.g., coherence or polarization data of SAR (synthetic aperture
radar) images, or hyperspectral bands of optical instruments). As new instruments with innovative capabilities are constantly being added to the portfolio of
remote sensing data sources, we also foresee a repeated
updating and validation of the available use cases.
Then we can use a combination of multi-source images
to retrieve their maximum image content.

We include external knowledge about the imaged surface areas, use cases and typical dynamics descriptions
from publicly available databases (e.g., digital elevation models, spectral reflectance data, instrument response and data acquisition parameters, existing thematic surface maps, statistical data form national archives, vegetation cycle descriptions, and local climate
data and weather reports). This allows us to make reasonable basic assumptions about the target area characteristics, their potential interpretation and reliability,
and supports a consistent knowledge extraction. These
data will enter the classification and labelling process
during further image analysis.

In order to analyze and interpret our data on a higher
hierarchical knowledge level, we conceive a carefully
conceived graph structure (“knowledge graph”) for
knowledge representation and analysis. This important
key connectivity component describes knowledge by
links, weights, and labels and allows for innovative
“graph mining”, an information representation, handling, and manipulation technology that has not yet be-
Keywords—Classification maps; graphs; information content;
SAR; semantics.
I.
INTRODUCTION
The automated extraction of the full information content
contained in a satellite image time series is a task for future
generations of researchers. Yet, what we propose here is the
design, implementation and validation of a carefully conceived
and validated knowledge extraction approach for sequences of
satellite images that differs from existing conventional image
analysis and knowledge retrieval mainly in four domains:

We arrange candidate time series images into stacks of
carefully co-registered images (so-called image cubes).
As the temporal recording intervals between image acquisitions may be irregular, we cannot apply uncontrolled transforms along the time axis. Instead, we apply conventional two-dimensional image analysis steps
come a standard approach for the interpretation of satellite images. Publications of knowledge graphs as described starting from [1] to [2] and [3] show a rapidly
evolving application potential for image interpretation
tasks. Therefore, we are confident to obtain a new type
of image time series content descriptor by our proposed graph technique.
II.
PROPOSED APPROACH
The proposed layout of our image interpretation approach
is shown in Fig. 1. It depicts the six main columns of hierarchical data and actionable information representation (from
left to right): (a) Original satellite images together with external data; (b) Local multi-resolution patches cut out from full
scenes yielding technically more easily manageable subscenes; (c) Feature extraction results comprising traditional
texture and / or spectral features as well as advanced deep
learning features resulting in content-oriented image descriptors; (d) Strongly compacted clustering results (i.e., application-oriented categories) from automated or supervised
classification; (e) Semantic labels annotating sub-scenes and
objects; (f) Knowledge organized in graphs that can be interpreted at least semi-automatically.
Fig. 1. Proposed structure of our knowledge extraction and mining approach.
This layout assures that one can analyze and compare the
content of single or of time series images on different representation and evidence levels. The parameter settings of the data
analysis steps are controlled by pre-defined use cases and applications (see the top row of Fig. 1). The results of the
knowledge extraction are provided by different graphs and
messages (see the bottom row of Fig. 1).
A. Selection of use cases applications
The diversity of applications that can be considered for the
proposed approach are rather broad and include, for instance,
coastal environmental monitoring (sea level, tides and wave
direction), land cover / land use changes, disaster monitoring,
forest management, ice monitoring, monitoring of active volcanoes, waste deposit site management, traffic monitoring,
vegetation monitoring, urban sprawl, soil moisture dynamics,
etc.
The approach proposed here is focused on a target set of
applications that covers areas from all over the world. In the
following, we describe two applications (use cases), namely
monitoring coastal environments and rapid mapping. For the
first use case, we select the Wadden Sea (in the Netherlands),
and the Danube Delta (in Romania) which are internationally
recognized protected areas as UNESCO Natural Heritage sites.
For the second use case, we select the area around Sendai (in
Japan) affected by a tsunami in March 2011 and the Elbe river
(in Germany) affected by floods in June 2013.
B. Applied datasets
An important aspect to be addressed is the creation of a reference dataset for test and validation of the approach. We already semantically annotated an initial SAR dataset resulting in
a semantic catalogue of hundreds of semantic labels grouped in
a 3-level hierarchical scheme [4]. This annotated database can
be considered as our initial ground truth dataset [5].
Our applied dataset contains: a) thirty Sentinel-1A images
(acquired from 2015 to 2016) that cover the Danube Delta and
surrounding areas and twenty one Sentinel-1A images (acquired between 2014-2016) that cover the Wadden Sea and the
Dutch Delta for monitoring coastal environments; b) nine TerraSAR-X images (acquired in 2010 and in 2011) that cover the
Sendai area for rapid mapping.
C. Multi-scale patch cutting
In this module, image patches are cut and used further in
the feature extraction module to generate individual or combined descriptors. For an optimal number of patch levels and
patch sizes for each level [6], [7], [8], we need to consider the
following parameters: selected application, instrument type,
image parameters (e.g., resolution, pixel spacing), and type of
features to be used (e.g., minimum patch size accepted by a
feature extraction method). Based on our first tests, we observed that for eight general categories [4] (Water bodies, Natural vegetation, Settlements, Bare ground, Transport, Military
facilities, Agriculture, and Industrial production areas) the
accuracy of the classification after multi-scale patch cutting
remains almost the same; however, the volume of the data to be
handled by this method is reduced by a factor of nearly 10.
D. Feature extraction
Our first feature extraction option is an enhanced rotationinvariant feature extractor. This allows us to compare linear
features independently of their actual orientation. This can be
accomplished rather simply by re-arranging the elements of the
extracted feature vectors, where Gabor filter bank [15] results
are sorted by magnitude.
An alternative feature extraction step is a spectral transformation of the image, often being preferred for multipolarization SAR images.
A third feature extraction option is to use deep learning for
satellite images [9]. Deep learning is exploited for the extraction of characteristic patterns and physics-related discoveries
from images and / or other data sources representing still undiscovered information content features. Following a training
phase, one can access the final results (that may not refer directly to a physical quantity) together with intermediate results
at selected layers of the neural net (that may refer to typical
target area characteristics). We assume that some network layers refer to specific image content details.
E. Classification
For image time series, we apply an active learning classification approach based on a cascaded learning method [10]. The
procedure is using a Support Vector Machine and is summarized in [11]. The advantages of cascaded learning are: 1) the
reduction of the volume of data to be classified from one level
(one patch size) to another level (another patch size) is made by
an active learning procedure that discards the content of the
data that do not contain the desired category; 2) the user can
select, based on the application and the size of the objects within each patch, up to which level the data shall be classified and
annotated.
F. Annotation and semantics
Publicly available semantic annotation schemes for satellite
images are mostly vegetation-oriented and have not been developed for high-resolution images, while the situation is less
critical for medium-resolution images where we already have,
for instance, a number of object-based categories [12]. Therefore, we already developed a hierarchical semantic annotation
scheme for high-resolution SAR images, with 3 hierarchical
levels and with a total of 150 categories [4]. Interestingly, the
uppermost level 3 categories describe details of man-made
infrastructure, while the categories describing natural environments do not have level 3 refinements. For medium-resolution
SAR images (e.g., Sentinel-1 images) our proposed annotation
scheme will not cover the very detailed level.
G. Knowledge graphs
The scientific intent of knowledge graphs is to unambiguously link various digital data contained in a database with
high-level (semantic) formulations, for instance, to summarize
the information content contained in digital image time series
reduced to high-level output messages. A fictitious example in
the near future could be a text output like “The traffic jam on
the M4 has dissolved” after automatically interpreting a sequence of airborne motorway pictures. A detailed scientific
paper dealing with knowledge graphs for remote sensing images has been published by [13].
Here, the scientific goal of our knowledge graphs is to select image data combined with additional information and to
generate from them higher-level interpretation results. The
linking can be understood as an upwards translation of binary
data into content-related information. In contrast, current applications of linked data are mostly limited to the integration of
geographical data with query systems.
H. Output analysis tools
Our output analysis tools mainly comprise classification
maps, data analytics results, performance measures, and image
content information.
Classification change maps of each dataset can be generated for all image time series (e.g., natural disaster scenarios);
these maps will be used to analyze the temporal evolution of
the affected areas (see Fig. 2). Then, the whole examined area
can be studied in order to detect any changes (in the case of a
disaster), or to see the distribution of the retrieved categories in
an image and to present these results in different charts (see
Fig. 3). The classification accuracy of the extracted features
will be evaluated by computing characteristic metrics (e.g.,
precision/recall, ROC, etc. [14]) detailing the classification
results of the actual features and the corresponding reference
data (see Fig. 4). Finally, the information content of each newly
collected image will be analyzed based on knowledge graphs,
in order to detect any anomalies or changes between the new
data and given reference data from the database. In future, the
output results can be a message (e.g., text, voice, etc.) that one
or more semantic labels were changed (e.g., Mixed forest to
Bare soil); hints to changes (e.g., less vegetation derived from
analyzed image data); high level information (e.g., a construction site was finished, or agricultural land was turned into an
industrial area).
III.
TYPICAL RESULTS
Typical output results of the proposed approach can be seen
in Figs. 2, 3 and 4. More results can be found in [5].
Airport runways Aquaculture
Bridges Channels
Debris
Flooded areas
Industrial buildings
Medium‐density residential areas
Mountains
Ocean
Ploughed agricultural land Shores
Fig.2. Comparative time series classification maps based on data taken prior
to the Sendai tsunami (left column), one day after the March 11, 2011
tsunami (center-left column), two months after the tsunami (center-right
column), and three months after the tsunami (right column).
Fig. 4. Attainable classification accuracies by Gabor feature vectors for a
selected number of categories from our dataset.
ACKNOWLEDGMENT
The selection of the first use case and protected areas were
made jointly with partners from the H2020 ECOPOTENTIAL
project.
REFERENCES
[1]
[2]
[3]
[4]
[5]
[6]
Fig. 3.a. Quantitative analysis of the tsunami-affected areas.
[7]
[8]
[9]
[10]
Fig. 3.b. Diversity of categories identified from the Danube Delta.
[11]
IV. VALIDATION ASPECTS
Our approach was tested with different applications covering different geographical areas. One area of special interest is
the Danube Delta in Romania.
At the mouth of the Danube, the alluvial discharge decreases every year. This makes it interesting to monitor the evolution of the alluvial discharge and to investigate its impact on
the Danube Delta through the years. The data can be combined
with other types of information, such as risk maps needed by
shipping traffic and by local authorities to protect the human
settlements.
[12]
[13]
[14]
[15]
D.J. Cook, L.B. Holder (eds.), “Mining Graph Data”, John Wiley &
Sons, 2007.
D. Sullyvan, 2012. Available: http://searchengineland.com/googlelaunches-knowledge-graph-121585.
S. Réjichi, F. Chaabane, F. Tupin, "Expert Knowledge-Based Method
for Satellite Image Time Series Analysis and Interpretation," IEEE
JSTARS, 8(5), pp. 2138-2150, 2015.
C.O. Dumitru, G. Schwarz, and M. Datcu, “Land Cover Semantic
Annotation Derived from High Resolution SAR Images“, IEEE
JSTARS, 9(6), pp. 2215-2232, 2016.
C.O. Dumitru, G. Schwarz, and M. Datcu, “SAR Image Land Cover
Datasets for Classification Benchmarking“, IEEE JSTARS, under
review, 2017.
P. Blanchart, M. Ferecatu and M. Datcu, "Cascaded Active Learning for
Object Retrieval Using Multiscale Coarse to Fine Analysis", in Proc. of
ICIP, Brussels, Belgium, pp. 2793-2796, 2011.
S. Cui and M. Datcu, “Cascaded Active Learning for Evolution Pattern
Extraction from SAR Image Time Series”, in Proc. of MultiTemp,
Alberta, Canada, pp. 1-4, 2013.
Q. Liu, R. Hang, H. Song, and Z. Li, “Learning Multi-Scale Deep
Features for High-Resolution Satellite Image Classification”, CoRR,
2016.
D. Marmanis, K. Schindler, J. D. Wegner, S. Galliani, M. Datcu, and U.
Stilla, “Classification with an Edge: Improving Semantic Image
Segmentation with Boundary Detection“, in Proc. of Computer Vision
and Pattern Recognition, 2016.Available: arXiv:1612.01337.
P. Blanchart and M. Datcu, “A Semi-Supervised Algorithm for AutoAnnotation and Unknown Structures Discovery in Satellite Image
Databases”, IEEE JSTARS, 3(4), pp. 698-717, 2010.
C.O. Dumitru, S. Cui, and M. Datcu, “Validation of Cascaded Active
Learning for TerraSAR-X Images”, in Proc. of IIM 2015, Bucharest,
Romania, 2015. Available: http://elib.dlr.de/100474/.
H. Taubenböck, M. Klotz, M. Wurm, J. Schmieder, B. Wagner, M.
Wooster, T. Esch, S. Dech, “Delineation of Central Business Districts in
Mega City Regions Using Remotely Sensed Data“, Remote Sensing of
Environment, 136, pp. 386-401, 2013.
S. Réjichi, F. Chaabane, F. Tupin, "Expert Knowledge-Based Method
for Satellite Image Time Series Analysis and Interpretation," IEEE
JSTARS, 8(5), pp. 2138-2150, 2015.
I. Witten, E. Frank, and M. Hall, “Data Mining: Practical Machine
Learning Tools and Techniques”, Morgan Kaufmann Publishers, 2011.
B. Manjunath, and W. Ma, “Texture Features for Browsing and
Retrieval of Image Data”, IEEE PAMI, 18(8), pp. 837-842, 1996.