AI and Cultural Heritage
Semantic
Classification
of Byzantine Icons
Paraskevi Tzouveli, Nikos Simou, Giorgios Stamou, and Stefanos Kollias,
National Technical University of Athens
R
ecent advances in Web technologies that ensure quick, effective information distribution have created the ideal terrain for spreading vari-
ous cultures through cultural-content presentation. Nowadays, we are witThis system uses
nessing the publication of a huge amount of cultural information on the Web.
fuzzy description
In the past, people gathered cultural information from physical objects (such as
books, sculptures, statues, and paintings).
Now, digital collections have become common practice for most cultural-content
providers.
However, as the amount of the Web’s cultural content grows, search and retrieval
procedures for that content become increasingly difficult. Moreover, Web users need
more efficient ways to access huge amounts
of content. So, researchers have proposed sophisticated browsing and viewing technologies, raising the need for detailed metadata
that effectively describes the cultural content. Several annotation standards have been
developed and implemented, and Semantic
Web technologies provide a solution for the
semantic description of collections on the
Web.1 Unfortunately, the semantic annotation of cultural content is time consuming
and expensive, making it one of the main
difficulties of cultural-content publication.
Therefore, the need for automatic or semi­
automatic analysis and classification of cultural assets has emerged.
logics and patterns
to automatically
determine the sacred
figure depicted
in an icon.
March/April 2009
To meet this need, researchers have proposed using image analysis methods (for
some examples, see the sidebar on the next
page). However, these methods use domain
knowledge for only low-level analysis. Also,
two main difficulties have arisen in using
image analysis to automate annotation and
classification of cultural digital assets. The
first is the failure of semantic image segmentation and image analysis algorithms in
some real-world conditions. This is due to
the high variability of content and environmental parameters (luminance and so on),
which makes the problem complex. The second difficulty is the extensive and vague nature of domain knowledge (at least for some
cultural content), which complicates formal
knowledge representation and reasoning.
However, some cultural domains are appropriate for automatic analysis and classification methods. Byzantine icon art is one
of them. The predefined image content and
the low variability of the image characteristics support the successful application of image analysis methods. Consequently, we’ve
developed a system that exploits traditional
1541-1672/09/$25.00 © 2009 IEEE
Published by the IEEE Computer Society
35
AI and Cultural Heritage
Related Work on Image Analysis for Cultural Heritage
T
he July 2008 IEEE Signal Processing Magazine presented some particularly interesting applications of
image analysis to the cultural-heritage domain. Alejandro Ribes and his colleagues described problems concerning the design of multispectral cameras as well as the
analysis of the Mona Lisa in the context of the digitization
of paintings.1 Anna Pelagotti and her colleagues proposed
a novel multispectral digital-imaging technique for clustering image regions in a set of images containing similar
characteristics when exposed to electromagnetic radiation. 2 Their technique provides material localization and
identification over a painting’s entire surface. C. Richard
Johnson and his colleagues described how three research
groups developed systems that identify artists on the basis
iconographic guidelines to automatically analyze icons.
Our system detects a set of concepts and properties depicted in the icon, according to which it recognizes a sacred figure’s face, thus providing semantic classification.
Byzantine Iconography
Byzantine art refers to the art of Eastern Orthodox states
that were concurrent with the Byzantine Empire. Certain
artistic traditions born in the Byzantine Empire, particularly regarding icon painting and church architecture, have
continued in Greece, Bulgaria, Russia, and other Eastern
Orthodox countries up to the present. The icons usually
depict sacred figures from the Christian faith, such as Jesus, the Virgin Mary, the apostles, and the saints.
At the beginning of the 17th century, iconography manuals represented the most important source of inspiration
for icon painters. The 16th-century monk Dionysios from
Fourna is credited as the author of the prototypical manual, Interpretation of the Byzantine Art. 2 In it, he defines
the rules and iconographic patterns that painters still use
to create icons.
Byzantine iconography follows a unique convention of
painting. The artistic language of Byzantine painters is
characterized by apparent simplicity, overemphasized flatness, unreal and symbolic colors, lack of perspective, and
strange proportions. The sacred figures are set beyond real
time and real space through the use of gold backgrounds.
The most important figure in an icon is depicted frontally,
focusing on the eyes, facial expression, and hands. Clarity
is the rule, not only in describing shapes but also in arranging figures in balanced compositions so that the narrative’s
actions are clear. The figures have large eyes, enlarged ears,
long, thin noses, and small mouths, each painted following
specific rules.
When depicting various figures, the artist is obliged to
observe the manual’s rules regarding posture, hair and
beard style, apparel, and other attributes. Specific sacred
36
of brushwork characteristics that art historians use for authentication. 3 Finally, Howard Leung and his colleagues reported on the preservation of ancient Chinese calligraphy. 4
References
1. A. Ribes et al., “Studying That Smile,” IEEE Signal Processing
Magazine, vol. 25, no. 4, 2008, pp. 14–26.
2. A. Pelagotti et al., “Multispectral Imaging of Paintings,” IEEE
Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 27–37.
3. C.R. Johnson et al., “Image Processing for Artist Identification,” IEEE Signal Processing Magazine, vol. 25, no. 4, 2008,
pp. 37–48.
4. H. Leung, S.T.S. Wong, and H.H.S. Ip, “In the Name of Art,” IEEE
Signal Processing Magazine, vol. 25, no. 4, 2008, pp. 49–54.
figures of the Christian faith are defined by specific facial
characteristics. For example, according to Dionysios, Jesus
is illustrated as a young figure with long, straight, thick,
black hair and a short, straight, thick, black beard.
According to these rules, the area in which the sacred
figure will be painted is separated into equal segments. The
first segment contains the head, focusing on the eyes and
facial expression. This segment contains the sacred figure’s most distinguishable features—for example, hair and
beard style. The remaining segments contain the sacred figure’s body, focusing on the hands and arms. The body can
be depicted in different poses—for example, standing, sitting, from the head to the waist, and from the head to the
thorax.
The head should be elliptical. The major axis is along
the vertical part of the head and is equal to four times the
height of the nose (H). (According to Dionysios, the nose
serves as the metric for producing harmonious facial and
body proportions.) The minor axis is along the horizontal
and is equal to 3H. The head segment can be separated into
four equal vertical sections of height H. The hair is in the
first part, the forehead is in the second, the nose is in the
third, and the area between the mustache and the chin is in
the fourth.
A Byzantine painter applies four base colors in layers on
the face segment. The darking is a dark color for the base
of the face segment and the darkest shadows. The proplasmos is a less-dark color for lighter shadows. The sarkoma
is for flesh, and the lightening is for highlights, containing
a good amount of white and being light enough to show off
against the base color. The absolute value of these colors
differs from icon to icon.
From Art Manual to Semantic Representation
Although the knowledge in Dionysios’s manual concerns
vague concepts such as “long hair,” “young face,” and so
on, it’s quite strict and formally described. Consequently,
www.computer.org/intelligent
IEEE INTELLIGENT SYSTEMS
Sacred-figure
we can create an ontological represenrecognition
Feature
tation of this knowledge using OWL.
extraction
In this way, the ontology’s axiomatic
Semantic
skeleton will provide the terminology
segmentation
and restrictions for Byzantine icons.
Semantic
interpretation
The only problem is that even a
Byzantine icon
well-tuned image analysis algorithm
analysis
can’t produce perfect results for a
Byzantine
wide set of data with different charicon
acteristics. Moreover, many Byzandatabase
tine icons might be misinterpreted
Assertional
owing to their age, further affecting
knowledge
the results. Finally, the vague nature
Reasoning
of terms such as distance, color, and
engine
length introduces fuzziness in the catTerminological
knowledge
egorization of objects. For example,
sometimes a sacred figure represents
Knowledge representation
a young face, but only to a certain
and reasoning
degree. In such cases, standard description logics (DLs) are insufficient
because they can’t deal with vague,
uncertain information. To handle Figure 1. The Byzantine icon classification system’s architecture. The analysis
such problems, researchers have pro- module extracts information about the icon, and the knowledge representation and
posed fuzzy DLs, based on fuzzy sets reasoning module uses this information to infer implicit knowledge that categorizes
and fuzzy logic, which are suitable for the icon.
handling imprecise information. 3
To classify the important facial features of Byzantine a large set of heuristics that facilitated the design of the
icons and detect the sacred figures having specific charac- subsystem modules and helped us select the most apteristics, we combine image analysis algorithms with the propriate semantic segmentation and feature extraction
expressive power of fuzzy DLs. Using the rules and pat- methods.
terns described in Dionysios’s manual, we constructed a
fuzzy knowledge base that’s populated by the results of im- Semantic segmentation. This module divides the icon into
age analysis of the digitized paintings. Using fuzzy reason- regions visualizing the icon’s physical objects and their
ing and on the basis of the defined terminology, our system parts.
The first step of semantic segmentation is face detection.
can classify the figures in Byzantine icons.
For this, we adopted a fast method based on support vector decision functions.4 This method first uses integral imThe Icon Classification System
Figure 1 shows the general architecture of our Byzantine aging for very fast feature evaluation based on Haar basis
icon classification system. It consists of the Byzantine icon functions. Then, it uses a simple, efficient classifier based
analysis subsystem and the knowledge representation and on the AdaBoost learning algorithm to select a small number of critical visual features from a very large set of poreasoning subsystem.
tential features. Finally, this method combines classifiers
in a “cascade,” quickly discarding the image’s background
Byzantine Icon Analysis
This subsystem performs the image processing and analy- regions while concentrating computation on promising
sis tasks necessary to detect a set of primitive concepts and face-like regions. An example of this method’s results is
properties (such as face and long hair) and form assertional the green rectangle in the first image of the face detection
knowledge that semantically describes the icon’s content. It box in Figure 2 (see next page).
The next step is localization of the eyes. Beconsists of modules for semantic segmentation, feature excause the nose’s height should be equal to the distraction, and semantic interpretation.
The first step of system development was the interpre- tance of one retina from the other, that distance is a
tation, with the aid of Byzantine iconography experts, critical parameter for segmentation. The semanticof Dionysios’s manual. This procedure provided us with segmentation module performs eye localization with the
March/april 2009
www.computer.org/intelligent
37
AI and Cultural Heritage
Face detection
extracts the face
and its basic
components
(nose and eyes).
Face component
detection
detects the most
important components
of the face (beard,
moustache, cheek,
forehead, and hair,
respectively).
Base color
analysis extracts
the base color
layers (darking,
proplasmos.
sarkoma, and
lightening).
s1
Semantic
interpretation
gives meaning to
specific features,
properties, and
relationships of
face components
with the aid
of fuzzy
partitions.
Membership
Feature extraction
automatically extracts
features such as
distance, color,
and length for
each face
component and
for some
segments
of the face
components.
1
0.7
0.3
0
s2
s3
s4
s5
BelowEarsHair
AboveEarsHair
2.8
5
Hairlength
Fuzzy partitioning
Image description
s: Face
s1: Hair
s2: Forehead
s3: Cheek
s4: Mustache
s5: Beard
(s, s1): hasComponent
...
Length(s1) = 2.8*H
...
darking(s1) = 88%
...
Formal assertions
s: Face
s1: Hair
s2: Forehead
s3: Cheek
s4: Mustache
s5: Beard
(s, s1): hasComponent
...
s1: AboveEarsHair = 0.7
s1: BelowEarsHair = 0.3
...
Figure 2. Image analysis. First, the algorithm detects the sacred figure’s face region,
eyes, and nose. Then, it extracts the hair, forehead, cheek, mustache, and beard
parts together with the face’s base color layers. Further analysis of the extracted
parts provides information about characteristic features. Finally, the algorithm
produces a semantic interpretation for each of these features, together with formal
assertions.
aid of geometric information from the eyes and the surrounding area. 5 We use a training set of sacred-figure
eyes to create vector maps, and we use principal components analysis to derive eigenvectors for these maps.
For a candidate face segment, the algorithm 5 projects
the length and its angle maps on the spaces spanned
by the eigenvectors that were evaluated during training. In that way, we use the similarity of the projection
38
weights and model eyes to determine an eye’s presence and position
(indicated by the red and green dots
in the second image in the face detection box of Figure 2).
After eye localization, the module detects the nose. Let H denote the
distance between the eyes and consequently the nose’s length. According to Dionysios, the nose starts on a
small horizontal line whose center is
a distance of H/5 below the midpoint
between the eyes. We use a Hough
transform to find this line. Therefore,
we determine the nose’s ending line
by applying a Hough transform at a
distance H down vertically from the
starting line.
The computation of H leads to the
detection of the face’s other main components—the hair, forehead, cheek,
mustache, and beard—because they are
well defined in Dionysios’s manual.
The module detects the four base
colors by applying an Otsu segmentation algorithm 6 to a restricted area
of the face segment (see the base
color analysis box in Figure 2), determining a range of intensity values
for the pixels.
To segment each part of the face, we
use H together with Dionysios’s proportion guidelines to define the seeds
for graph-cut segmentation.7 The cost
functions used as soft constraints incorporate boundary and part information for each of the five segmented
parts. To achieve segmentation of
each part, we compute the minimum
cut that’s the global optimum among
all the segmentations satisfying the
constraints (see the face-componentdetection box in Figure 2).
Feature extraction. This module automatically extracts features from the segmented parts, 8 providing
additional information regarding each part’s length,
color, shape, and texture. This information constitutes
the characteristic values of the features, providing an
image description set (see the image-description section of the feature extraction box in Figure 2).
Dionysios assigns four features to hair:
www.computer.org/intelligent
IEEE INTELLIGENT SYSTEMS
• color (dark, gray, or white),
• density (thin or thick),
• length (above the ears or below the ears, and whether the
figure has a braid on the left, right, or both shoulders), and
• form (straight, wild, or curly).
The feature extraction module categorizes hair color (see
Figure 2, feature extraction, segment s1) using the intervals
of Otsu segmentation. It estimates the density and form by
combining a Canny edge detector and a morphological fi lter and by evaluating the mean value of the curvatures of
the detected edges. It fi nds the hair’s length using H together with Dionysios’s proportion guidelines.
Next, the module determines information about the sacred figure’s age by computing the
number of wrinkles on the forehead
(s2). Using the Hough transform, the
module defi nes the number of vertical and horizontal lines in the forehead, which represent the wrinkles.
A young sacred figure has zero or few
wrinkles, whereas an older sacred figure has many.
The cheek (s3) provides additional
information about age. According to
Dionysios, a small number of wrinkles on the larger (left or right) part
of the cheek indicates a young sacred
figure. Conversely, an older sacred figure’s cheek will have a greater number
of wrinkles (and therefore a greater proportion of the colors
used for shadows).
Finally, the module defi nes the feature sets for the mustache (s4) and beard (s5). Using techniques similar to those
for hair analysis, it analyzes the mustache’s color and shape.
It also extracts information on the beard’s length (short, not
very long, long, down to the waist, or down to the knees),
color (fair or dark), form (straight, wild, or curly) and shape
(sharp, circle-like, circle, wide, or fluffy).
can express an assertion in a fuzzy manner, with the aid of
fuzzy set theory. This procedure is critical for the knowledge representation system.
Suppose that a specific feature’s values lie in the interval X. A fuzzy set A of X associates each value x of X
to a real number in the interval [0, 1], with the value of
fA(X) at x representing the grade of membership of x in
A. So, we defi ne an appropriate fuzzy partition (a set of
fuzzy sets) of X and connect each fuzzy set of the partition with a subconcept of the terminology, defi ned by that
fuzzy set’s property. For example, having extracted the
value 2.8H for the length of hair (s1), we determine that
s1 has a membership value of 0.7 for AboveEarsHair and
0.3 for BelowEarsHair (see the graph in the semanticinterpretation box in Figure 2). The
defi nition of the fuzzy partitions is
based on the heuristics extracted from
Dionysios’s manual.
the module determines
information about
Knowledge representation
and reasoning
This subsystem consists of terminological and assertional knowledge and a
reasoning engine.
These types of knowledge are
the basic components of a knowledge-based system based on DLs, 9 a
structured knowledge-representation
formalism with decidable-reasoning
algorithms. DLs have become popular,
especially because of their use in the Semantic Web (as in
OWL DL, for example). DLs represent a domain’s important notions as concept and role descriptions. To do this,
DLs use a set of concept and role constructors on the basic elements of a domain-specific alphabet. This alphabet
consists of a set of individuals (objects) constituting the domain, a set of atomic concepts describing the individuals,
and a set of atomic roles that relate the individuals. The
concept and role constructors that are used indicate the expressive power and the name of the specific DL. Here, we
use SHIN, an expressive subset of OWL DL that employs
concept negation, intersection, and union; existential and
universal quantifiers; transitive and inverse roles; role hierarchy; and number restrictions.
the sacred figure’s age
by computing the
number of wrinkles
on the forehead.
Semantic interpretation. This module represents the extracted information in terms of the sacred-figure ontology.
Representing the face and its main components is
straightforward: we simply write assertions such as s1:
Hair (see the semantic-interpretation box in Figure 2).
However, each component’s properties are vague. For example, you can’t always directly determine on the basis of
length whether the sacred figure has long hair. This difficulty is due to the vagueness of the defi nition of length
and the imprecision introduced by feature extraction (see,
for example, the result of the segmentation of the hair in
the feature extraction box in Figure 2). Nevertheless, we
March/april 2009
Terminological and assertional knowledge. The terminological knowledge is an ontological representation of the
knowledge in Dionysios’s manual. The assertional knowledge is a formal set of assertions describing a specific icon
in terms of the terminological knowledge. For example, the
terminology contains axiomatic knowledge such as “a Jesus
Face is depicted as a young face with long, straight, thick,
www.computer.org/intelligent
39
AI and Cultural Heritage
black hair and a short, straight, thick, black beard,” whereas
“the face depicted in the icon is a face with long hair in some
degree” is a relative assertion.
The knowledge base has two main components. The
terminological component (TBox) describes the relevant
notions of the application domain by stating properties of
concepts and roles and their interrelations. TBox is actually a set of concept inclusion axioms of the form C v D,
where D is a superconcept of C, and concept equivalence
axioms of the form C ≡ D, where C is equivalent to D.
The assertional component (ABox) describes a concrete
world by stating properties of individuals and their specific interrelations. To deal with the vagueness introduced
in our case, we use f-SHIN, a fuzzy extension of SHIN. 3 In
f-SHIN, ABox is a finite set of fuzzy assertions of the form
〈a : C⋈n〉, 〈(a, b) : R⋈n〉, where ⋈ stands for ≥, >, ≤, or
<, for a, b ∈ I. Intuitively, a fuzzy assertion of the form 〈a
: C ≥ n〉 means that the membership degree of a to concept
C is at least equal to n. (A formal definition of the syntax
and semantics of fuzzy DLs appears elsewhere. 3,10)
In the Byzantine-icon-analysis domain, the individuals
are the segments that the semantic-segmentation module
extracted. The DL concepts classifying the individuals are
determined by the properties measured as features during
feature extraction (for example, BlackHair and AboveEarsHair). The DL roles mainly describe partonomic
(mereological) hierarchies and spatial relations (for example, the axiom isLeftOf– = isRightOf indicates that
the role isLeftOf is the inverse of isRightOf, and the
axioms Trans(isLeftOf) and Trans(isAboveOf) indicate
that isLeftOf and isRightOF are transitive).
The terminology’s main concepts are Figure, Face, and
FaceComponent. They’re connected with the partonomic
role hasPart and its subroles hasSegment and hasComponent, to indicate whether a segment is a subsegment of
another segment and so on. Using these roles, we describe
the partonomic hierarchy as follows:
Face ≡ ∀hasComponent.FaceComponent
Figure ≡ ∃hasSegment.Face
FaceComponent v Beard t Cheek t Forehead t Hair
t Mustache
We describe relevant assertions as follows:
〈s:Face ≥ 1.0〉
〈s1:Hair ≥ 1.0〉
〈(s, s1):hasSegment ≥ 1.0〉
40
…
The taxonomy is constructed using the properties measured as features during feature extraction. Here are some
examples of terminological axioms defining the concept
hierarchy:
AboveEarsHair v Hair
OldCheek v Cheek
OldForehead v Forehead
HairyBeard v Beard
Restrictions concerning the taxonomy mainly describe disjointness and exhaustiveness, forming axioms such as
AboveEarsHair v ¬BelowEarsHair
Hair v AboveEarsHair t BelowEarsHair
Using these concepts, we define more expressive descriptions that specify some specific face characteristics on the
basis of Dionysios’s manual. For example, we define a
young man’s face as
YoungMansFace ≡ ∃hasComponent.BlackHair u
∃hasComponent.(NotVeryLongBeard u BlackBeard)
u ∃hasComponent.YoungForehead u ∃hasPart.
YoungCheek u ∃hasPart.BlackMustache
We similarly define concepts specifying the characteristics of some popular sacred figures. For example, we define
JesusFace as
JesusFace ≡ YoungMansFace u ∃hasComponent.
(BraidOnTheLeftShoulder u HairyHair
u StraightHair) u ∃hasComponent.
(CircleLikeBeard u StraightBeard)
The reasoning engine. This module uses the terminological knowledge and the assertions to recognize the sacred
figure. Our system uses FiRE, the Fuzzy Reasoning Engine
(www.image.ece.ntua.gr/~nsimou/FiRE), which fully supports f-SHIN.
Figure 3 illustrates how the engine performs reasoning
and classification for recognizing Jesus as the sacred figure.
We use FiRE’s greatest lower bound (GLB) reasoning service, which evaluates an individual’s greatest possible degree
of participation in a concept. The concepts of interest in this
case, YoungMansFace and JesusFace, only include conjunc-
www.computer.org/intelligent
IEEE INTELLIGENT SYSTEMS
YoungMansFace � �hasComponent.
BlackHair � hasComponent.
�.(NotVeryLongBeard � BlackBeard)
� �hasComponent.YoungForehead
� �hasPart.YoungCheek � �hasPart.
BlackMoustache
s1
s2
s3
s4
s5
JesusFace � YoungMansFace �
�hasComponent.(BraidOnTheLeftShoulder
� HairyHair � StraightHair)
� �hasComponent.(CircleLikeBeard
� StraightBeard)
JesusFace
TBox
Reasoning
engine
s: Face
s1: Hair …
(s, s1): hasComponent …
s1: BraidOnTheLeftShoulder = 0.78
s1: HairyHair = 0.96
s1: StraightHair = 0.86
s1: BlackHair = 0.88
s2: YoungForehead = 0.82
s3: YoungCheek = 0.87
s4: StraightMoustache = 1.0
s4: BlackMoustache = 0.83
s5: CircleLikeBeard = 0.79
s5: StraightBeard = 0.92
s5: NotVeryLongBeard = 1.0
s5: BlackBeard = 0.84
ABox
Knowledge
representation
and reasoning
Figure 3. A reasoning example. The extracted information from image analysis constitutes the assertional component (ABox) of
the knowledge base; the terminological component (TBox) is defined on the basis of Fourna’s specification. These components
form the input to the fuzzy reasoning engine, which infers information about the icon.
tions, so we use the fuzzy intersection operator for the minimum. So, the GLB of segment s (the face) in YoungMansFace
is 0.82, whereas the GLB of segment s in JesusFace is 0.78.
In other words, the extracted features indicate that the face
depicted is YoungMansFace to a degree of 0.82 and is JesusFace to a degree of 0.78.
Results
We evaluated our system on a database, provided by the
Mount Sinai Foundation in Greece, containing 2,000 digitized Byzantine icons dating back to the 13th century. The
icons depict 50 different characters; according to Dionysios, each character has specific facial features that makes
him or her distinguishable.
Evaluation of the Byzantine-icon-analysis subsystem produced promising results. The subsystem’s mean response
time was approximately 15 seconds on a typical PC.
In the semantic-segmentation module, the face detection
submodule reached 80 percent accuracy. In most cases, the
failure occurred in icons with a destroyed face area. If the
submodule detected the face, it almost always detected the
March/april 2009
eyes and nose. The base-color-analysis submodule defined
the color model used by the icon’s artist, which the feature extraction module later used. Using the nose’s height,
which can be easily determined, and taking into account
the manual’s rules, the face-component-detection submodule estimated the positions of the background and foreground pixels for every part of the face. These positions
constituted the input to the graph-cut algorithm. This submodule achieved 96 percent accuracy.
The feature extraction module extracted features for
every icon segment. Then, using the fuzzy partitions, the
semantic-interpretation module interpreted each segment’s
specific features and properties and the relationship of face
parts. Thus, that module determined each feature’s degree
of membership in a specific class, thereby creating an image
description.
To evaluate overall system performance, we used
precision and recall. Table 1 (on the next page) presents results for 20 of the sacred-figure classes. The
more distinctive facial features the sacred figure in a
class contained, the better our method performed (for
www.computer.org/intelligent
41
AI and Cultural Heritage
Table 1. Evaluation of the icon classification system.
Class
No. of icons in class
No. of images classified
No. of images
correctly classified
Precision
Recall
Jesus
70
67
58
0.87
0.83
Virgin Mary
60
54
43
0.80
0.72
Peter
50
44
38
0.86
0.76
Paul
50
44
39
0.89
0.78
Katerina
40
35
30
0.86
0.75
Ioannis
40
37
29
0.78
0.73
Luke
40
33
28
0.85
0.70
Andrew
40
33
29
0.88
0.73
Stefanos
30
27
24
0.89
0.80
Konstantinos
40
36
30
0.83
0.75
Dimitrios
40
36
31
0.86
0.78
Georgios
40
37
32
0.86
0.80
Eleni
40
37
31
0.84
0.78
Pelegia
20
17
15
0.88
0.75
Nikolaos
40
38
33
0.87
0.83
Basileios
40
37
31
0.84
0.78
Antonios
30
27
20
0.74
0.67
Eythimios
25
23
18
0.78
0.72
Thomas
35
32
26
0.81
0.74
Minas
20
17
14
0.82
0.70
example, for the Jesus class). Conversely, our method
performed worse on figures with similar facial features,
such as women.
sify paintings on the basis of unmixed colors and dense
textures.
Acknowledgments
W
e’re developing a version of our system that takes
into account information about clothes associated
with specific types of figures (such as apostles, kings, angels, and monks) and clothing accessories (kerchiefs and
so on) and nonclothing accessories (books, crosses, and so
on) associated with specific sacred figures. For example,
the system could incorporate the knowledge that icons of
the Virgin Mary depict her wearing a kerchief with three
stars. Such information might help in classifying figures
with similar facial features.
Furthermore, an integrated version of our system could
eventually date icons and attribute them to a particular
artist or artistic school, on the basis of features specific
to each artistic period. Such information can help classify
an icon even if the specific sacred figure is difficult to recognize. We’re also investigating applying our method to
other art movements such as impressionism, trying to clas-
42
Our research has been partly supported by the MORFES (extraction
and modeling of figure in Byzantine paintings of Mount Sinai monuments using measures) project, funded by the EU and the Greek Secretariat for Research and Development. We thank the Mount Sinai
Foundation for providing access to the rich data set and especially
archeologist Dimitris Kalomirakis for his participation in knowledge extraction and his continuing support during this research.
Finally, we thank the anonymous reviewers for their constructive
comments.
References
1.T. Berners-Lee, J. Hendler, and O. Lassila, “The Semantic
Web,” Scientific American, May 2001, pp. 34–43.
2.Dionysios tou ek Phourna, Hermeneia tis zografikis technis
(interpretation of the Byzantine art), B. Kirschbaum, 1909.
3.G. Stoilos et al., “Reasoning with Very Expressive Fuzzy De­­
scription Logics,” J. Artificial Intelligence Research, vol. 30,
no. 5, 2007, pp. 273–320.
4.P. Viola and M.J. Jones, “Robust Real-Time Face Detection,”
Int’l J. Computer Vision, vol. 57, no. 2, 2004, pp. 137–152.
5.S. Asteriadis et al., “An Eye Detection Algorithm Using Pixel
to Edge Information,” Proc. 2nd IEEE-EURASIP Int’l Symp.
www.computer.org/intelligent
IEEE INTELLIGENT SYSTEMS
Th e A u t h o r s
Control, Communications, and Sig­
nal Processing (ISCCSP 06), 2006;
www.eurasip.org/Proceedings/Ext/
ISCCSP2006/defevent/papers/cr1124.
pdf.
6.N.A. Otsu, “A Threshold Selection
Method from Gray-Level Histograms,”
IEEE Trans. Systems, Man, and Cybernetics, vol. 1, no. 9, 1979, pp. 62–66.
7.Y. Boykov and M.-P. Jolly, “Interactive
Graph Cuts for Optimal Boundary and
Region Segmentation of Objects in N-D
Images,” Proc. Int’l Conf. Computer
Vision (ICCV 01), vol. 1, IEEE Press,
2001, pp. 105–112.
8.R.C. Gonzalez and R.E. Woods, Digital Image Processing, 3rd ed., Prentice
Hall, 2008.
9.F. Baader et al., The Description Logic
Handbook: Theory, Implementation
and Applications, Cambridge Univ.
Press, 2002.
10.G. Stoilos et al., “Uncertainty and the
Semantic Web,” IEEE Intelligent Systems, vol. 21, no. 5, 2006, pp. 84–87.
For more information on this or any other
computing topic, please visit our Digital Library at www.computer.org/csdl.
Paraskevi Tzouveli is pursuing her PhD in image and video analysis at the National Technical University of Athens. Her research interests include image and video analysis, information retrieval, knowledge manipulation, e-learning, and digital cultural heritage
systems. Tzouveli received her Diploma in electrical and computer engineering from the
National Technical University of Athens. Contact her at [email protected].
Nikos Simou is a PhD student in the National Technical University of Athens Depart-
ment of Electrical and Computer Engineering and a member of the Image, Video, and
Multimedia Systems Laboratory’s Knowledge Technologies Group. His research interests
include knowledge representation under certain and uncertain or imprecise knowledge
based on crisp and fuzzy description logics, ontologies, and the implementation and optimization of reasoning systems and applications for the Semantic Web. Contact him at
[email protected].
Giorgios Stamou is a lecturer in the National Technical University of Athens School of
Electrical and Computer Engineering and a member of the university’s Image, Video, and
Multimedia Systems Laboratory, heading the lab’s Knowledge Technologies Group. His
main research interests include knowledge representation and reasoning, theory of uncertainty, machine learning, semantic interoperability, and digital archives. Contact him at
[email protected].
Stefanos Kollias is a professor in the National Technical University of Athens (NTUA)
School of Electrical and Computer Engineering and the director of the Image, Video, and
Multimedia Systems Laboratory. His research interests include image and video processing, analysis, retrieval, multimedia systems, computer graphics and virtual reality, artificial
intelligence, neural networks, human-computer interaction, and digital libraries. Kollias
received his PhD in signal processing from NTUA. He’s a member of the European Commission expert group in digital libraries and of the EC committee on semantic interoper­
ability, leading NTUA participation in DL projects, EDLnet, VideoActive, Europeana
Connect, Europeana Network 1.0, and EU Screen. Contact him at [email protected].
AdvertiSER Information
March/April 2009 • IEEE Intelligent Systems
Advertising Personnel
Marion Delaney
IEEE Media,
Advertising Dir.
Phone: +1 415 863 4717
Email: md.ieeemedia@
ieee.org
Marian Anderson
Sr. Advertising
Coordinator
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: manderson@
computer.org
Sandy Brown
Sr. Business
Development Mgr.
Phone: +1 714 821 8380
Fax: +1 714 821 4010
Email: sb.ieeemedia@
ieee.org
March/april 2009
Advertising Sales
Representatives
Product:
Recruitment:
Midwest/Southwest
Darcy Giovingo
Phone: +1 847 498 4520
Fax: +1 847 498 5911
Email: [email protected]
Mid Atlantic
Lisa Rinaldo
Phone: +1 732 772 0160
Fax: +1 732 772 0164
Email: lr.ieeemedia@
ieee.org
Northwest/Southern CA
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: tm.ieeemedia@
ieee.org
New England
John Restchack
Phone: +1 212 419 7578
Fax: +1 212 419 7589
Email: [email protected]
Japan
Tim Matteson
Phone: +1 310 836 4064
Fax: +1 310 836 4067
Email: [email protected]
US West
Lynne Stickrod
Phone: +1 415 931 9782
Fax: +1 415 931 9782
Email: [email protected]
Southeast
Thomas M. Flynn
Phone: +1 770 645 2944
Fax: +1 770 993 4423
Email: flynntom@
mindspring.com
Europe
Hilary Turnbull
Phone: +44 1875 825700
Fax: +44 1875 825701
Email: impress@
impressmedia.com
Europe
Sven Anacker
Phone: +49 202 27169 11
Fax: +49 202 27169 20
Email: sanacker@
intermediapartners.de
www.computer.org/intelligent
US East
Joseph M. Donnelly
Phone: +1 732 526 7119
Email: [email protected]
US Central
Darcy Giovingo
Phone: +1 847 498 4520
Fax: +1 847 498 5911
Email: [email protected]
43