Robust Road Sign Segmentation
Chung-Hsin Liao 1, Hiromitsu Hama 2
1
Department of Electronic Engineering, ChengShiu University
No.840,ChengCing Rd., Niaosong Township, Kaohsiung County, Taiwan, R.O.C, [email protected]
2
Graduate School of Engineering, Osaka City University, Japan
3-3-138, Sugimoto, Sumiyoshiku, Osaka 558-8585, Japan, [email protected]
information necessary for successful driving-they
describe the current traffic situation, define right-ofThis paper deals with the robust segmentation of road way, prohibit or permit certain directions, warn about
signs under bad illumination conditions including risky factors and also help drivers with navigation.
night-time. The road sign recognition system is very Recently, many techniques have been developed to
important for safe driving. The proposed segmentation segment and recognize road signs. Many of them
system is developed using relative similarity with the adopted the road sign segmentation using color and
two-way connectivity. Here, we focus mainly on how shape information. For example, Piccioli et al. [1] use
to obtain the optimum threshold to each illumination color and a priori information to limit the possible
condition. The proposed system works on various locations of signs in the image. They then extract
types of road signs such as circular, equilateral edges and look for circular or triangular regions before
triangle, pentagon and diamond-shapes. The final goal applying a cross-correlation technique for recognizing
of our proposed system indicates to the driver the the signs. In [2], a redness measure is used to locate
t
op,y
i
e
l
d,a
n
d“
do n
ote
n
t
e
r
”s
i
gn
s
.Th
i
ss
t
e
pi
s
presence of a sign in advance so that some incorrect s
followed
by
edge
detection
and
shape
analysis
to
human decisions may be avoided. The effectiveness of
the proposed method is demonstrated using 100 identify the sign. In their approach to detecting stop
images of road signs under various illumination signs, Yuille and his colleagues [3] correct for the
conditions. The result is encouraging and color of the ambient illumination, locate the
demonstrates the ability and efficiency of our boundaries of the signs and map the sign into a front
algorithm in achieving the segmentation task. to parallel position before reading the sign. Another
According to our experimental results, the average paper that describes actively controlling the camera
for sign detection is [4]. They predict the location of
accuracy rate for segmentation is 98%.
the sign and point the camera for a closer view. Signs
are recognized and their contents are read by template
1. INTRODUCTION
matching.
The road sign recognition system is an essential A decision tree method is used in [5] to detect and
modu
l
e of‘
Dr
i
v
e
rSu
ppor
tSy
s
t
e
m(
DSS)
’
,whi
c
h recognize signs without using color. Detection is
should help human drivers to predict and avoid based on shape using local orientations of image edges
dangerous situation in traffic environment. The and hierarchical templates. The results are sent to a
objective of DSS is to increase the safety, efficiency decision tree which either labels the regions by sign
and comfort of driving. Among them, driving safety is type or rejects them. A method to detect speed limit
the most important and the major concern. So, the signs is given in [6]. It is based on first using color to
primary goal of such a system is to ensure the traffic locate candidate signs, followed by a multi-resolution
safety, which may be also used as a support tool for application of templates that look for circular regions.
vehicle guidance and navigation. The increase in Another approach is taken by Fleischer et al., [7], who
traffic accidents is becoming a serious social problem use a model-based, top down approach. Predictions
with the recent rapid traffic increase. In many cases, are made of locations in which signs may appear and
t
h
edr
i
v
e
r
’
sc
a
r
e
l
e
s
s
n
e
s
si
st
h
epr
i
ma
r
yf
a
c
t
oroft
r
a
f
f
i
c shape, size, and color are used to specify edge models
accident, and the DSS is demanded for supporting for the signs in the 3D world. Signs that are found are
dr
i
v
e
r
’
ss
a
f
e
t
y
.To be a
bl
et
oh
e
l
pt
h
e dr
i
v
e
ri
n tracked through subsequent images using a Kalman
decision-making, DSS must understand the current filter. Shaposhnikov et al., [8], make use of color
traffic situation. Therefore, it should create and segmentation using the CIECAM97 color appearance
maintain the model of its neighbourhood. Because of model. They then use histograms of oriented edge
the dominant role of visual information for the human elements to determine the shape of the sign followed
driver, computer vision methods are often used in by location of the center of the sign. Escalera et al., [9],
semiautonomous and autonomous vehicle prototypes also start with color matching, which they follow with
for the creation of such model. Road signs carry much corner detection in which they look for corners in
Abstract
specific relationships that correspond to triangular,
rectangular, or circular signs.
Extracted sign regions are matched with templates
using a Markov [9-11] model, and the match with
highest rank is regarded as the recognition result.
Until now, many approaches have been proposed to
develop road sign segmentation and recognition
s
y
s
t
e
msun
de
rg
ood c
on
di
t
i
on
sbu
tt
h
e
yc
a
n’
tg
i
v
e
successful results under bad conditions, for example,
at night-time. In this paper, we propose a robust road
sign segmentation system under all illumination
conditions. Although the segmentation is not easy to
be successful at night-time, we focus on such difficult
conditions as shown in figure 1. This paper applies
adaptive thresholds for robust segmentation. Needless
to say, segmentation is not so easy under outdoor
scenes. There are many degrees of freedom that Fig. 1: Difficulties: (a) reflections, (b) not controlled lighting, (c)
shadows, (d) night-time.
caused the segmentation to be difficult.

occlusion and the presence of other objects
with the same plane,

lighting conditions are changeable and not
controllable, producing bright spots and shadows,

color and shape of signs depending their age,
physical conditions and scratches.
To overcome these difficulties, the proposed algorithm
is used both of color and shape information, and
furthermore relative similarity with the two-way
connectivity method. Though the experiments with
100 road signs taken under various illumination
conditions including night-time, it is shown that the
system can give 98% segmentation rate.
The remainder of this system is organized as follows.
In Section 2, we present segmentation method. The
flow diagram for the proposed system is described in
this section. In Section 3, the experimental results and
successful rates are shown. Finally, conclusions and
future works are discussed in Section 4.
2. SEGMENTATION METHOD
Our proposed method is a combination of color feature
based and region based segmentation method.
Segmentation is an important part of any automated
image recognition systems. To achieve region
segmentation, it is needed to set the color distance also
called the threshold first. It depends on decision of
suitable thresholds whether or not to segment the
required region successfully. Color distance is used as
a measurement of color similarity where pixels
satisfying a certain degree of color homogeneity are
grouped to form a cluster.
Here, we propose a robust segmentation method using
relative similarity with the two-way connectivity
method. This paper focuses on how to decide
adaptively the threshold with K-nearest connectivity
for relative similarity using color information. After
noise removal, the interested regions are divided up
from outliers using shape information. The
segmentation is done in the RGB space. The flow
diagram is shown in figure 2.
Fig. 2: Flow diagram.
Most of previous road sign segmentation approaches
used the fixed thresholds or absolute similarity, so
called bounding-box method. For example, the region
segmentation was developed using the thresholds over
gray level images and color images but these methods
were very sensitive to the threshold values. This
method is one of the simplest ways of color image
segmentation. The circular road signs include only
three colors: red, blue and white. For example, in the
case of bounding-box for red region detection, the
thresholds for R, G and B components may be set to
40 R 100 , 10 G 85 ,and 15 B 75 , respectively.
Example is shown in figure 3. The segmentation is
completed for figure 3(a), but not for figure 3(b).
Although we may change the bounding-box so that
can be successful in this failure case, the new box
c
a
n
’
tg
r
a
n
t
e
et
os
ol
v
eot
h
e
ri
npu
ti
ma
g
e
s
.Th
i
sme
a
n
s
,
the bounding-box should be always changed according
to an input image.
Fig. 3: Road sign segmentation using a bounding-box (RGB color
space): (a-1) and (b-1) input image, (a-2) and (b-2) segmented result
using the bounding-box, (a-3) segmented result (successful) and (b3) segmented result (unsuccessful).
Fig. 4: Lattice seed points: (a) lattice seed points in an image, and
(b) complete segmentation by seed region growing (threshold=16).
Fig. 5: Examples of K-nearest connectivity links: (a) one-way and
two-way similarities, (b) K=1, (c) K=2, (d) K=3, (e) K=4, (f) K=5,
(g) K=6, (h) K=7, and (i) K=8 ((b)～(i) threshold=2).
For these reasons, the proposed segmentation
algorithm is developed with a relative similarity
method instead of an absolute one. Relative to filling
algorithm of connected regions, there are many
techniques seed filling, scan line filling algorithms and
so on. They can be applicable to obtaining connected
regions by relative similarity. Among them, the seed
filling algorithm is adopted, and the seed region
growing method here. The main idea is to compare the
relation of the seed pixel (central pixel) and
neighbourhood pixels, that is, they belong to the same
objects. The choice of criterion is very important and
critical. The seed points are assigned at every lattice
point as shown in figure 4. If the lattice is small
enough, then the result would include at least one
region which should be subdivided. The assumption
using suitable intervals can be solved with the option
of splitting such regions. On the other hand, it takes
expensive computational cost. Here we treat three
main types of connectivity: the traditional 4-connected
(N=4), 8-connected (N=8), and K-nearest connectivity
(K=4～8), and two types of similarity: one-way and
two-way. Figure 5 (a) shows a schematic illustration
of the concept of relative similarity where the links are
neighborhoods.
In this figure, one way arrow (A→B) means that A is
similar to B, but B is not similar to A. The two way
arrow (A
C) means that A and C are similar to each
other. In this case, the link is made only between A
and C. According to this way, connected regions are
made, and it is called the two-way method. Figure 5
(e) and (f) show that there is a link between the central
two points ④ and ③ when K=5, and no link when
K=4. In image pattern recognition, thresholding is an
extremely useful technique, because of its powerful
ability to extract characters or objects from their
backgrounds. A real world outdoor scene includes
many objects, such as cars, persons, visual
information signs, buildings and so on, which
composed of a variety of shapes and colors under
various lighting conditions. In analyzing such scene
images, the results after thresholding must be invariant
to these conditions.
The segmentation method used adaptive thresholds
[12-16] for relative similarity [17-19]. More precisely,
the same color region means the uniform color region
under at a certain similarity. The similarity value for
two consecutive pixels is calculated by Eq.(1) that is
defined in RGB space. Here, it is very important how
to decide the optimum threshold. We will find the
stable region for deciding the threshold. It means that
t
h
er
e
g
i
on c
a
n’
tdr
a
ma
t
i
c
a
l
l
yc
h
a
ng
ea
l
t
h
oug
ht
h
e
threshold is gradually changing in a wide range. Here,
this method can be developed to decide the suitable
threshold automatically for various images. These
aspects are shown in figure 6. In this figure, when the
threshold is 14, there are more than three connected
regions. The areas corresponding to L1, L2 and L3 are
100, 125 and 84 pixels, respectively. When the
threshold increases to 16, there is only one region. It
means that the three regions are combined into one
object. In other words, L1, L2 and L3 belong to a
common region. If the threshold increases, the region
area becomes larger. Although the area size can be in
the predetermined range, there may be sometimes a
few error cases. To avoid this kind of problems, this
system uses the margin to the upper and lower
boundaries of the threshold range. Here, the margin 
is set as 5 to obtain the stable area. In figure 6, the
deciding of the optimal thresholds is explained with a
red ring color region of a road sign. According to this
figure, the stable area is in the range of 310 to 400
pixels and the stable region is between 16 and 36.
Then the optimal threshold is decided a value between
21 and 31, for example, Threshold=26. Here the
margin  is 5, then 16+ =21, and 36- =31. Here,
used road signs are taken from the distance between
30m to 40m. Here, the area and ratio parameter for
circular, equilateral-triangle, pentagon and diamondshapes using 20 test images. Using these training data,
the predetermined areas for uniform color regions of
various road sign types are described as shown in table
1. In this paper, relative similarity with 4-connectivity
is adopted to decide the suitable threshold.

distance  R i R j
2
2
2 
G i G j 

B i B j 
,
K-nearest connectivity. The K-nearest (K=8) and 8connected (N=8) are the same. It can be also seen that
K-nearest (K=4) is the best because it gives the widest
range of suitable thresholds that is the mean, minimum
and maximum in the case of K=4 are larger than those
of other K values. The reason to choose (K=4) is that
the wider range of the optimum thresholds gives the
more stable uniform region. Firstly, the optimum
threshold is automatically decided using relative
similarity with the two-way K-nearest connectivity,
here K=4. Using the temporary decided threshold, the
system performs the labelling process. The ROI is
examined using its area and ratio of width and height.
For example, the red ring of road signs is 314～400
pixels at the distance 30m ～ 40m, that already
predefined in table 1. If there is no suitable region at
this stage, the system goes back to the former stage
and checks again and changing the threshold. After
extracting the ROI, the system removes noise and then
completes the segmentation.
(1)
where, Ri, Gi, Bi : the R, G and B components of the
current interest pixel Xi,
Rj, Gj, Bj: the R, G and B components of the
neighborhood pixel Xj.
Fig. 7: The suitable threshold valuation.
3. EXPERIMENT RESULTS
Fig. 6: Decision of the optimum thresholds.
TABLE 1: AREA AND RATION PARAMETER:
Moreover,
the
thresholdings
using
various
connectivity methods are implemented and the results
are compared as shown in figure 7. It includes the
results of 4-connectivity, 8-connectivity and two-way.
To confirm the effectiveness of the proposed method,
the images of road signs were taken under various
illumination conditions by resolution CCD camera.
For the images taken from 30m to 40m distances, the
red ring region area is in the range of 300 to 400 pixels
according to our experimental results. Using the
relationship of the red ring region area and the
distance, the system can be easily extended to images
taken from any distance. In this paper, special
attention is paid to robustness and flexibility under
bad conditions. Some examples of bad conditions are
shown. The access rates for segmentation are
expressed in table 2.
This system was implemented using on Visual Basic
Ver.6 and Halcon Ver.6 and executed on a personal
computer with Pentium 4 processor 2.66 GHz.
Experimental data used here are circular, equilateral
triangle, pentagon and diamond-shapes road signs.
The processing time for deciding the adaptive
threshold is approximately 2.2 seconds. The total time
for segmentation is 2.8 sec. This system decided
automatically adaptive thresholds and then segmented
input images almost perfectly. In this system, there
were a few failure cases in segmentation because of
occlusion and worse illumination conditions that are
shown in figure 8.
REFERENCES
[1] Piccioli, G., De Micheli, E., Parodi, P., and
Ca
mpa
n
i
,M.
,“
A Robu
s
tMe
t
h
odf
orRoa
dSi
g
n
De
t
e
c
t
i
on a
n
d Re
c
ogn
i
t
i
on
,
”I
ma
g
ea
n
d Vi
s
i
on
Computing, vol. 14, no. 3, pp. 209-223, 1996.
[2] L.Es
t
e
v
e
za
n
d N.Ke
ht
a
r
na
v
a
z
,“
Ar
e
a
l
-time
histographic approach to road sign r
e
c
ogn
i
t
i
on
,
”
in Proc. IEEE Southwest Symp. Image Analysis
Interpretation, pp. 94–100, 1996.
[3] Yuille, A. L., Snow, D., and Nitzberg, M.
Si
gnf
i
n
de
r
,“
Us
i
ngCol
ort
oDe
t
e
c
t
,Loc
a
l
i
z
ea
n
d
I
d
e
n
t
i
f
y
I
n
f
o
r
m
a
t
i
o
n
a
l
S
i
g
n
s
,
”
i
n
I
n
t
.
C
o
n
f
.
o
n
4. CONCLUSIONS AND DISCUSSIONS
Computer Vision (ICCV98), Bombay, India, pp.
628-633, 1998.
We have presented an approach for road sign
segmentation under various illumination conditions [4] Miura, Jun, Tsuyoshi, Kanda, and Shirai,
Yos
h
i
a
k
i
,“
AnAc
t
i
v
eVi
s
i
onSy
s
t
e
mf
orRe
a
l
with outdoor scenes. This paper focused differing
T
i
m
e
T
r
a
f
f
i
c
S
i
g
n
R
e
c
o
g
n
i
t
i
o
n
,
”
I
E
E
E
C
o
n
f
.
o
n
from previous approaches, in dealing with many areas
Intelligent Transportation Systems, Dearborn, MI,
of freedoms. The main objective was emphasized to
pp. 52-57, 2000.
the problem of lighting changes and shadows. In this
[5]
Paclik, Pavel and Novov
i
c
ova
,J
a
n
a
,”
Roa
dSi
g
n
system, a robust road sign segmentation algorithm was
Cl
a
s
s
i
f
i
c
a
t
i
on Wi
t
h
ou
tCol
or I
n
f
or
ma
t
i
on
,
”i
n
proposed using adaptive thresholds for relative
Proc. of the 6th Conf. of Advanced School of
similarity with K-nearest connectivity, where K=4.
Imaging and Computing, Lommel, Belgium, 2000.
The system was developed successfully under various
[6]
Se
k
a
n
i
n
a
,Luk
a
sa
n
dTor
r
e
s
e
n,J
i
m,“
De
t
e
c
t
i
onof
illumination conditions such as daytime, evening and
N
o
r
w
e
g
i
a
n
S
p
e
e
d
L
i
m
i
t
S
i
g
n
s
,
”
i
n
P
r
o
c
.
o
ft
h
e
night-time. In this paper, segmentation was completed
16th
European
Simulation
Multiconference
98 road signs among 100 road sign images.
(ESM-2002), Darmstadt, Germany, pp. 337-340,
The future work is to develop a robust segmentation
2002.
system under more occluded conditions, and bad
,a
ndRa
t
h
,T.M.
,“
3Dweather such as snow and rainy days and so on. The [7] Fleischer, K, Nagel, H.-H.
Model-Based-Vision for Innercity Driving
next step of our system is to recognize road signs and
Sc
e
n
e
s
,
” I
EEE I
n
t
e
l
l
i
g
e
n
t Ve
h
i
c
l
e
s Sy
mp.
transmit only the necessary information to the driver
(IV'2002),
Versailles,
France,
pp.
477-482,
2002.
by the way of voice or vibrations that may not disturb
in his driving. This system will be intended to [8] Shaposhnikov, Lubov N., Podladchikova,
Alexander V., Golovan, Natalia, Shevtsova,
c
on
s
i
de
rt
h
edr
i
v
e
r
’
sg
a
z
i
ng u
s
i
ng e
y
ec
a
me
r
aa
n
d
A.
,
Hong
,Kun
bi
n
,a
n
d Ga
o,Xi
a
oh
on
g
,“
Roa
d
when the driver dose not pay attention to the necessary
Sign
Recognition
by
Single
Positioning
of
Spaceroad environments, the system will alert to the driver
Va
r
i
a
n
tSe
n
s
orWi
n
dow,
”i
nPr
oc
.oft
h
e15t
hI
n
t
.
in time.
Conf. on Vision Interface, Calgary, Canada, pp.
213-217, 2002.
[9] A.del
aEs
c
a
l
e
r
aa
n
dL.Mor
e
n
o,“
Roa
dt
r
a
f
f
i
c
s
i
gn de
t
e
c
t
i
on a
n
dc
l
a
s
s
i
f
i
c
a
t
i
on
,
”I
EEE Tr
a
n
s
.
Industrial Electromagnetism, vol. 44, pp. 847–
859,
1997.
[10] M.La
l
on
dea
n
dY.Li
,“
De
t
e
c
t
i
onofr
oa
ds
i
g
n
s
using color i
n
de
x
i
n
g
,
” Ce
n
t
r
e de Re
c
h
e
r
c
h
e
Informatique de Montreal, Montreal, QC, Canada,
CRIM-IT-95, pp. 12–49, 1995.
[11] D.
S.Ka
n
g
,N.
C.Gr
i
s
wol
d,N.Ke
h
t
a
r
n
a
v
a
z
,“
An
invariant traffic sign recognition system based on
sequential color processing and geometrical
transf
or
ma
t
i
on
,
”i
nPr
oc
.oft
h
eI
EEESou
t
hwe
s
t
Symp. on Image Analysis and Interpretation,
Dallas, TX, pp.88-93, April, 1994.
[12] Thi Thi Zin, C. Liao, T. Kaneko, Y. Yanagihara,
Fig. 8: Examples of failure cases in segmentation: (a) occlusion, (b)
H.Ha
ma
,“
Re
c
og
ni
t
i
onofRoa
dEn
v
i
r
onme
n
tby
very dirty, (c) very dark and deteriorate, (d) only a part of region
Using Color Information from Moving Video
was affected by strongly sunlight.
I
ma
g
e
s
,
”I
nPr
oc
.7t
hI
ma
geMe
di
aPr
oc
e
s
s
i
n
g
Symp. (IMPS 2002), pp.109-110, November,
2002.
TABLE 2: SEGMENTATION RESULTS:
[13] Thi Thi Zin, C. Liao, T. Kaneko, H. Hama, Pyke
Ti
n
,“
Robus
tRoa
d Si
g
n Re
c
og
nition byUsing
Multi-f
e
a
t
u
r
eTe
mpl
a
t
eMa
t
c
h
i
ng
,
”I
nPr
oc
.2n
d
Int. Conf. on Computer Applications, Yangon,
Myanmar, pp.474-481, January, 2004.
[14] C.Li
a
o,T.Ka
n
e
k
o,H.Ha
ma
,“
Ex
t
r
a
c
t
i
onof
Road Traffic Signs using Similarity of Adjacent
Col
orI
n
f
or
ma
t
i
on
,
”In Proc. 4th Int. Conf. On
Advanced Mechatronic-Toward Evolutionary
Fusion of It and Mechatronics, Asahikawa Japan,
pp.103-108, 2004.
[15] C.Li
a
o,H.Ha
ma
,“
Ex
t
r
a
c
t
i
onofRoa
dTr
a
f
f
i
c
Signs using Similarity of Adjacent Color
I
n
f
or
ma
t
i
on
,
”I
n Pr
oc
. 36t
hI
n
t
. Conf. On
Stochastic System Theory and ITS Applications,
Tokyo Denki University, Saitama, Japan, pp.8-13,
2004.
[16] C.Li
a
o,H.Ha
ma
,“
Re
c
ogn
i
t
i
onofRoa
dTr
a
f
f
i
c
Si
gn
s
,
”I
n Pr
oc
.9t
hI
ma
g
e Me
di
a Pr
oc
e
s
s
i
ng
Symp. (IMPS 2004), pp.109-110, November,
2004.
[17] C. Liao, Thi Thi Zin,T. Kaneko, H. Hama,
“
Robu
s
tSe
gme
n
t
a
t
i
on of Roa
d Tr
a
f
f
i
c Si
gn
s
Us
i
ng Ada
pt
i
v
e Th
r
e
s
h
ol
ds
”t
h
eI
n
s
t
i
t
ut
e of
Electronics, Information and Communication
Engineers Electronics Express, Vol. 2, No. 14,
pp.423-428, July, 2005.
[18] C.Li
a
o,“
Robus
tSe
gme
ntation of road traffic
s
i
gn
sUs
i
ngRe
l
a
t
i
v
eSi
mi
l
a
r
i
t
y
,
”I
nPr
oc
.10t
h
Image Media Processing Symp. (IMPS 2005),
pp.111-112, November, 2005.
[19] C.Li
a
o,“
Ada
pt
i
v
eRe
g
i
onCol
orSe
gme
n
t
a
t
i
on
f
orRoa
d Si
g
ns
,
”I
n Pr
oc
.37t
hI
n
t
.Con
f
.On
Stochastic System Theory and ITS Applications,
Japan, pp.13-14, October, 2005.
中文摘要
本研究在探討最合適門檻値自動對道路交通標識分割。在數位影像中大多數仍使用絶對門檻値,進行
對影像分割,在各種撮影状態下,很容易受雜音影響。本研究中所使用的 K-nearest connectivity 可
提供非常穩定的影像分割結果。此方法利用相對性性質,並接合了交通標識形顔色形状資訊,以便能正
確對道路交通標識進行分割。在實驗中以晴天,傍晩及夜間,共 100 張照片可自動取得最合適門檻値並
且不需用手動方法自動獲得 98%正確的分割率。