IEEE MELECON 2006, May 16-19, Benalmádena (Málaga), Spain
Detection, Tracking and Classification of Road Signs in
Adverse Conditions
George K. Siogkas
Evangelos S. Dermatas
Dept. of Electrical Engineering & Computer Technology
University of Patras
Patras, Greece
Email: [email protected]
Dept. of Electrical Engineering & Computer Technology
University of Patras
Patras, Greece
Email: [email protected]
rectangles are used. The shape and colors of a sign define its
significance along with the ideogram that it contains.
Abstract—In this paper a complete automatic system for roadsign detection, tracking, and classification is presented and
evaluated. The processing of video frames in the L*a*b color
space improves significantly the sign detection rate by
processing the same frame in different normalized color
spaces. The tracking module reduces significantly the
processing time by transferring the sign detection information
in the next frames and processing different radii signs in
parallel. The proposed system is evaluated in normal, raining
and night driving conditions. In a total number of 266 roadsign recordings, the complete system track and recognize
successfully 216. The main source of system fault appears in
city night driving due to the presence of a great number of
light sources.
I.
The development of a system that can robustly detect and
classify road signs (in other words perform RSR) in real
time, has twofold benefits; on one hand it can be used in
driver assistance systems (DAS), that help the driver focus
more on the navigation of the vehicle by providing the
information given by the signs. On the other hand RSR
systems can, in the future, be embedded in fully autonomous
vehicles. Of course, in order for these systems to be
functional, they must have a number of characteristics. First
of all, they must be resilient to any change in lighting or
weather conditions. They also need to be able to recognize
partially occluded signs, as well as signs that are either
rotated, or not exactly perpendicular to the camera axis.
Finally these systems must be robust to the deterioration of
some signs’ color, usually due to their age and bad weather
conditions.
INTRODUCTION
The area of road sign recognition (RSR) has attracted the
attention of many researchers over the past decade. Its
importance lies mainly on the vast amount of car accidents
that happen each year all over the world, caused by the
drivers’ inability to process all the visual information they
receive while driving. Usually, the most important
information is provided by the road signs placed in the
drivers’ visual field. Road signs are designed to assist the
drivers in their effort to navigate their vehicle to their
destination efficiently and safely. They can be divided into
three main groups: danger proclamation signs, traffic
regulation signs and informational signs. Signs that belong to
the first group are placed to warn drivers of the dangers that
exist ahead on the road, so they can anticipate them. The
second group comprises signs that inform the drivers of the
special obligations, restrictions or prohibitions they should
conform to. The signs of the third group provide information
that assists the driver in the navigation task, such as
junctions, distances etc. Road signs are designed in a way
that helps the drivers to spot them easily in natural scenes.
This is achieved by selecting colors and shapes that
differentiate the signs from the background. Consequently,
the main colors that are used are red, blue, yellow and green,
with black or white ideograms. The shapes of the signs are
symmetrical. Triangles, circles, octagons, diamonds and
1-4244-0088-0/06/$20.00 ©2006 IEEE
The first important decision when dealing with the
problem of road sign detection (RSD) is whether to use color
images. In the implementations where grayscale images are
processed, the detection is based solely on morphological
features, such as symmetry in [1-3], distance transforms from
templates generated offline in [4], and border detection using
pyramidal structures as in [5-7]. A more sophisticated
method of sign detection based on genetic algorithms is
proposed in [8]. In color based RSD, the color space plays an
important role. In [9-11] the popular RGB color-space, or
variations based on it, such as relations between the color
coefficients are used. However, the RGB space is not
optimized for problems such as RSC, because it is
susceptible to lighting changes. Thus, color spaces more
immune to such changes are preferred: the HSI is used in
[12-18], or the LUV space in [19]. After color segmentation,
road signs are detected using circular and triangular shapes
[12], neural networks [13,18], genetic algorithms [15,16].
Simulated annealing is used together with genetic algorithms
in [17].
Among the most important tracking algorithms for road
signs, the authors of [18] use Kalman filters, Kalman-Bucy
537
filtering is used in [4], and a motion-model plus temporal
information propagation is proposed in [10]. For RSC the
most popular methods are based on template matching, either
by cross-correlation [12,17] or by matching pursuit filters
[14]. Various type of neural networks for RSC are
implemented in [11,13,16,19], and a Bayesian generative
modeling is described in [10].
The system processes a given frame, first by converting it to
L*a*b format and then by performing color segmentation.
Then, the segmented frame is passed on to the detection and
tracking modules. The centers and radii of all the detected
road signs are used in the tracking stage for the next frame
and are used to crop the signs for classification.
A. ColorSpace Selection - Color Segmentation
The L*a*b color space possesses a number of important
features: it is more immune to lighting changes, the a* (red green) and b* (yellow - blue) channels represent perceptual
color differences and are linearly spaced.
In this paper a complete system for road-sign detection,
tracking, and classification is proposed. The system is
founded upon a mixture of widely used methods, like the
symmetry detection of [21], and partly modified ones, such
as the modified Otsu threshold of [20], used here for color
segmentation. Moreover, a number of novel features are
introduced, such as the use of L*a*b color space for the color
image processing stage, and sign tracking in multiple frames
by examining previously selected sub-windows of each
frame.
The lighting changes in an image affect mostly the L
(Luminosity) channel of the L*a*b space. Thus, an
examination of the L channel of an image provides
information on the lighting conditions in which it was
acquired. This attribute can be used in the segmentation
process, as the signs appear lighter than the mean luminosity
in dark scenes, and darker than the mean luminosity in welllit scenes. This is especially useful in night driving
conditions, as it filters out much unneeded information.
The structure of the paper is as follows: in section II the
road-sign detection module is presented, while in section III
and IV the tracking and classification modules are discussed.
In section V the experimental results are presented, followed
by a short discussion.
II.
The simple, efficient, and fast Otsu’s thresholding
algorithm as proposed in [20] is used to transform the L*
channel to a binary image denoted as Lbo. Four binary
images are estimated by bisecting the positive and negative
part of a* and b* channels, to acquire four chromatic
subspaces and the negative subspaces are multiplied by -1 to
ensure positive definition for all channels. The four
chromatic subspaces are then transformed to binary images
by using the Otsu algorithm.
ROAD SIGN DETECTION
The structure of the proposed road-sign detection,
tracking and classification system is shown in fig. 1.
The four color-based regions-of-interest (ROIs) are
estimated by the intersection of the corresponding binary
image and the Lbo, if the mean luminosity of the frame is
lower than 40, otherwise the four ROIs are identical to the
corresponding binary images. The total segmented area is
defined by the union of the four ROIs.
B. Symmetry Detection
The four chromatic coefficients are scanned for
symmetrical shapes, using the fast radial symmetry detection
method of [21]. This method has been used in several
applications [5-7] due to its computational efficiency and its
relevance to the RSD problem. Depended on the types of
road signs are detected, some or all chromatic coefficients of
the segmented image are used. In the proposed system the
symmetry detection method is optimized for circular shapes,
but different symmetrical shapes can also be derived by
adjusting the radial strictness factor Į. This means that every
road sign in the image can be detected by this method, as
long as it remains in the image frame after its segmentation.
The symmetry detection algorithm scans for shapes of one or
more given radii. In order to blindly detect every existing
road sign, a large set of radii is used, thus the processing time
increases enormously. The computations can be accelerated
by taking into account the: (a) capability to perform
computations in parallel architecture, as the calculations
needed for the symmetry detection can be performed
Figure 1. Flowchart of the proposed system.
538
are defined, circular red, triangular red, circular blue and
rectangular blue.
independently for each radius and, (b) information acquired
from previous frames. The second approach has been
implemented in the proposed system reducing both the
number of radii and the size of the searching areas. A more
detailed presentation of this module is given in the section
discussing the tracking process.
The classification rule is based on the normalized crosscorrelation (NCC) of the cropped road signs with the
reference templates for all color channels. The classification
process for the circular blue signs uses only the NCC for the
b* channel, while for all the other categories the sum of NCC
is used for classification.
C. Center Localization – Shape Determination
The center of the detected symmetrical shape is
performed by a simple circle fitting. This procedure also
gives an approximation of the shape’s radius. The error
inserted in this process by the assumption that all shapes are
circular is not considered to be important for the overall
method. Once the center is localized, the sign image is
cropped, fill its binary image and perform cross-correlation
based template matching with the road sign shapes templates
(i.e. circle, triangle, octagon, square). At the end of this
stage, the approximate location, radius and exact shape of the
sign have been estimated.
III.
ROAD SIGN TRACKING
The tracking module has been designed to minimize the
computational effort and the tracking errors: The center
coordinates, for all symmetrical shapes of a chosen small
radius (e.g. 10 pixels), identified as a potential road sign, are
passed to the module that processes the next frame. This
module now performs symmetry detection and circle fitting
as in the RSD module for a sub-window of specified size and
centered in the coordinates given by the localization
procedure. Thus, once having detected a road sign centered
in (x,y), the algorithm scans the next frame in an area around
(x,y), for a symmetrical shape with a radius of the nearest
integer value above the one calculated for the previously
detected sign. This procedure is repeated for every sign
detected in the previous frame, regardless if it was its first
appearance or if it was tracked from an earlier frame. An
obvious flaw of this procedure is the possibility of the
temporary loss of visibility of the tracked sign (either total,
or partial) in one or more consecutive frames, which results
in tracking failure. An efficient, parallel, without
backtracking process in previous frames method is used to
resolve this problem, by choosing more than one radii in the
detection process (e.g. 10, 15 and 20 pixels), thus the sign is
relocated at the following frames. This technique slightly
increases the computational effort but recovers missing road
signs by processing more informative frames.
IV.
Figure 2. Detection and tracking (from left to right). (a) Image with
detected road-signs, (b) blue chromatic subspace,(c) red chromatic
subspace with denoted sub-window for sign tracking, (d) blue symmetry
detection for radius of 12 pixels, (e) red symmetry detection for radius of
the sign detected in the previous frame, and (f) centers and radii of the
detected signs.
V.
EXPERIMENTS
Our experimental data consisted of 50 video clips,
acquired from a video camera positioning into a moving
vehicle in two different adverse environments and, 75 video
clips in a typical condition. Two different video cameras, the
SONY HC85 and the SONY DCR-TRV60E were used to
record road-signs in the city of Patras and the suburbs using
PAL non-interlaced video (frames of 720x576 pixels) at 25
fps. The first set of video clips was shot at noon under shiny
weather and was 9 minutes long. The second set of clips was
taken in a rainy morning, with light rainfall, and was 2
minutes and 15 seconds long. The third one was shot at night
with good weather conditions in the city of Patras and was 3
minutes long. The vehicle was moving with fluctuating
speeds and in the first clip there were some parts in which
ROAD SIGN CLASSIFICATION
The proposed system detects and tracks the road signs in
the frames sequence taken from a moving vehicle. With a
successful detection and tracking procedure, every road-sign
in the driver’s visual field is cropped, resized to 64x64 pixels
and passed on to the RSC module.
The number of potential template matches can be
significantly reduced by processing the information of sign
color and shape. More specific, four categories of road-signs
539
[3]
the lighting conditions were adverse, due to the sun’s
position relative to the camera. A total number of 266 signs
were recorded from 21375 frames. The proposed system was
evaluated using the complete set of 139 road-signs used in
the European Union.
[4]
[5]
[6]
[7]
[8]
[9]
[10]
Figure 3. Sign detection in one normal and three adverse conditions.
In table 1 the recognition rate for the classification
module using only one frame, the ROIs located by the sign
detection module, is shown for normal, night driving and
raining conditions. In night recordings, the detection module
locates faulty signs due to the presence of a great number of
light sources.
TABLE I.
[11]
[12]
[13]
ROAD-SIGN RECOGNITION RATE USING ONE FRAME
[14]
TYPE OF ENVIRONMENT
Normal
43.92%
(343/781)
Raining
43.75%
Night driving
(7/16)
6.92%
(11/159)
[15]
The total correct detection rate for all recordings was
95.3% using simultaneous search in three radii 10, 15 and 20
pixels when a single frame is processed. The proposed
tracking method increases the sign-detection and recognition
rate to 81.2% classifying correctly 216 out of 266 signs.
[16]
[17]
The proposed system usually faults to detect the
triangular signs especially in low light and raining conditions
and faulty signs are detected in city night driving.
[18]
[19]
REFERENCES
[1]
[2]
Sergio Escalera and Petia Radeva, “Fast greyscale road sign model
matching and recognition”, Recent Advances in Artificial Intelligence
Research and Development, J. Vitria et al. (Eds.) IOS Press, 2004,
pp.69-76.
G. Loy and N. Barnes, “Fast Shape-based Road Sign Detection for a
Driver Assistance System”, Proceedings of International Conference
on Intelligent Robots and Systems (IROS), 2004.
[20]
[21]
540
Xavier Baró and Jordi Vitrià, “Fast Traffic Sign Detection on
greyscale images”, Congrés Català d’Intelligència Artificial
(CCIA’2004).
D.M. Gavrila, V. Philomin, “Real-time object detection for smart
vehicles”, International Conference on Computer Vision, IEEE
September , 1999.
]. H. Austerirmeier, U. Buker, B. Merstching, S. Zimmermann,
“Analysis of traffic scenes using the hierarchical structure code”,
International Workshop on Structural and Syntactic Pattern
Recognition August, 1992.
B. Besserer, S. Estable, B. Ulmer, D. Reichqardt, “Shape
classification for traffic sign recognition”, First International
Workshop on Intelligent Autonomous Vehicles, IFAC April, 1993.
U. Buker, B. Mertsching, “Parallel evaluation of hierarchical image
databases”, Journal of Parallel and Distributed Computing 31 (2),
1995, pp. 141–152.
Y. Aoyagi, T. Asakura, “A study on traffic sign recognition in scene
image using genetic algorithms and neural networks”, 22nd
International Conference on Industrial Electronics, Control, and
Instrumentation, IEEE August, 1996.
Rozenn Dahyot, Pierre Charbonnier, Fabrice Heitz, “Unsupervised
Statistical Detection of Changing Objects in Camera-In-Motion
Video”, ICIP-2001 International Conference on Image ProcessingThessaloniki, Greece - October 7-10, 2001.
Claus Bahlmann, Ying Zhu, Visvanathan Ramesh, Martin Pellkofer,
and Thorsten Koehler. A System for Traffic Sign Detection,
Tracking, and Recognition Using Color, Shape, and Motion
Information. In IEEE Intelligent Vehicles Symposium (IV 2005), Las
Vegas, NV, June 2005.
A. de la Escalera, L. Moreno, M.A. Salichs, J.Ma. Armingol, “Road
Traffic Sign Detection and Classification”, IEEE Transactions on
Industrial Electronics 44 (6), 1997, pp. 848–859.
G. Piccioli, E. de Micheli, P. Parodia, M. Campani, “Robust method
for road sign detection and recognition”, Image and Vision
Computing 14 (3), 1996, pp. 209–223.
C.Y. Fang, C.S. Fuh, P.S. Yen, S. Cherng, S.W. Chen, “An automatic
road sign recognition system based on a computational model of
human recognition processing”, Elsevier Computer Vision and Image
Understanding 96, 2004, pp. 237–268.
S.-H. Hsu, C.-L. Huang, ”Road sign detection and recognition using
matching pursuit method”, Elsevier Image and Vision Computing 19,
2001, pp. 119–129.
A. de la Escalera, J. M. Armingol, M.A. Salichs, “Traffic Sign
Detection for Driver Support Systems”, 3rd International Conference
on Field and Service Robotics, Espoo, Finland, June 11-13, 2001.
A. de la Escalera, J. M. Armingol, M. Mata, “Traffic Sign
Recognition and Analysis for Intelligent Vehicles”, Image and Vision
Computing, Vol. 11, N. 3, 2003, pp. 247-258.
A. de la Escalera, J. M. Armingol, F. Rodriguez, J. M. Pastor,
“Visual sign information extraction and identification by deformable
models for intelligent vehicles”, IEEE Transactions on Intelligent
Transportation Systems, Vol. 5, N. 2, 2004, pp. 57-68.
Chiung-Yao Fang, Sei-Wang Chen and Chiou-Shann Fuh, “RoadSign Detection and Tracking”, IEEE Transactions on Vehicular
Technology, Vol. 52, No. 5, September 2003, pp. 1329- 1341.
D.S. Kang, N.C. Griswold, N. Kehtarnavaz, “An invariant traffic
sign recognition system based on sequentialcolor processing and
geometrical transformation”, Proceedings of the IEEE Southwest
Symposium on Image Analysis and Interpretation, Dallas, TX, USA,
April 1994, pp. 88-93.
N. Otsu, “A threshold selection method from gray-level histogram,”
IEEE Transactions on System Man Cybernetics, Vol. SMC-9, No. 1,
1979, pp. 62-66.
Gareth Loy and Alexander Zelinsky, “Fast Radial Symmetry for
Detecting Points of Interest”, IEEE Transactions on Pattern Analysis
and Machine Intelligence, Vol. 25, No. 8, August 2003, pp. 959-973.