A Multi-Classifier Combination Strategy
for the Recognition of Handwritten Cursive Words
Brigitte PLESSIS, Anne SICSU,Laurent HEUTTE,Eric MENU,
Eric LECOLINET, Olivier DEBON, Jean-Vincent MOREAU
MATRA CAP Systemes- Departement Lecture Automatique
3,av.duCentre- 78052 Saint-Quentin-en-Yvelines
-Abstract
-
The paper describes a recognition scheme for reading
handwritten cursive words using three word recognition
techniques. It particularlyfocuses on the implementation
used to combine the three techniques based on a
comparative sru& of different strategies. The first
holistic recognition technique derives a global encoding
of the word. The other techniques both rely on the
segmentatiorr of the word into letters, bur diger in the
character ClassiJierthey use. Theformer runs a statistical
linear classifier, and the latter runs a neural network with
a different representation of the input data. The testing,
comparison, and combination studies have been
perfornied on word images from mail provided by the
USPS. The top choice recognition rates achieved so far
correspond to 88 %, 76 %, 6.5 % with respect to lexicon
sizes of 10, 100, and IO00 words.
IL Recognition schemes
11.1 Holistic word recognition
The rationale is to identify a word by the only use of its
most visible components. A more or less fuzzy but
overall content of the word can thus be derived. This
method first extracts favoured features which correspond
to obvious, irregularities of the handwritten word signal.
It then compares its "signature of irregularities" to a
lexicon, which stands for the contextual information.
The feature extraction stage highly relies upon the prior
detection of a lower case area which becomes a
reference for deciding on the feature type. Vertical
strokes within the lower case area are first extracted and
provide a rough idea of the word number of letters.
Extensions which are above or below the lower case area
are then detected. Horizontal srrokes within the lower
case area are lastly considered when pairing parallely on
top and bottom and when linking two vertical strokes by
top. This three types of strokes are combined into a first
encoding of the word. Loops are also meaningful features
in a word, but their occurrence is more dependent on the
writing style and can be mere flourishes. They are
therefore detected cautiously and combined into a
I. Introduction
The state-of-the art in handwritten cursive word
recognition often puts forward research work on the
development of complete word recognition systems. The
techniques proposed usually belong to 3 main categories :
- holistic, where the word is identified in its whole from
basic structural or temporal features [5,7,8].
- based on segmentation followed by recognition, where
the word is decomposed into its letters, which are
subsequently identified and matched against a list of
candidate words [l,4.81.
- based on segmentation driven by the recognition,
where the three above processings are performed at the
same time [ 2, 3,6].
Most authors generally agree that a unique recognition
method can not cope with the wide variety of cursive
writing styles, and that high recognition performances are
likely to be achieved by rather combining different
techniques. Complementary techniques, which complete
one another in their strengths and weaknesses, should
642
0-8186-4960-7/93
$3.00 0 1993 IEEE
ideally perform better than a unique one. In this paper, we
present some comparative studies which led us to choose
a final strategy for combining three different word
recognition
techniques. They involve a holistic
recognition scheme, and two segment-and-recognize ones
which differ in the character classifier they use. The
comparative studies were performed on address word
images provided by the USPS and tested .over three
different lexicon sizes (10-100-1000), the lexicon being
still necessary to constrain the vocabulary range. The
paper first describes the three techniques. It then presents
three combination strategies and analyses their
performance. End to end results are finally provided.
IL2.b. Recognition
separate encoding. Figure 1 shows an example of a word
global encoding.
The lexicon is encoded on a letter basis with the chosen
features which allows for an homogeneous matching. The
two different encodings are compared in turn to each
words in the lexicon, through dynamic programming thus
yielding two different rankings of the lexicon. The two
lists are then merged into a unique one where the loop
based ranking is only used to strengthen or to correct the
stroke based rankings.
The recognition function identifies each graphem image
received as input. Hence, 72 such classes have been
retained which cover both the variety of graphems
output by our segmentation and variances in writing
styles. Each graphem image is First transformed into a
feature vector of 164components. They belong to
8 different families of features, thus providing a wide
range of identification clues. The first family is based on
intersections with horizontal and vertical straight lines,
the second one computes moments of black pixels.
Measurements on the graphem profiles and on its
horizontal and vertical projections count for 3 other sets.
Another family of features includes top, bottom, left and
right extrema of the graphems. Structural features such as
end points, X and Y junctions derived on the graphem
skeleton are concatenated to the vector, as well as
features describing the convexity of the graphem
boundaries computed from its polygonization transform.
The last set is derived from global concavity
measurements. The character classifier relies on linear
discrimination where the supervised learning stage has
been performed on 26 000 graphem samples.
11.2SegmendRecognize with statistical classifier
Another natural recognition technique is to perform a
precise analysis of the word content by identifying each
one of its letters. Our current implementation of such an
analytical reading process is however sequential.
Segmentation, recognition and contextual word
identification are invoked in turn.
IL2.a Character segmentation
This sequential implementation, although being direct,
must cope with the lack of contextual information when
the process starts. As a consequence the segmentation
does not exactly extract characters but an over-set which
also includes broken or connected characters. We call
them graphems. The graphem segmentation is guided by
prior assumptions on the location of likely separations
between cursive letters. However the variety of writing
styles does not make these assumptions always true,
which explains for over or under segmentation problems.
More precisely, the segmentation invokes two
competitive processes: link detection and prerecognition. The First one finds the locations of links
between characters, and the last one, counter-balances
these detections to avoid over segmenting. The prerecognition corresponds to a rough identification of the
component whose role is to decide whether the
component is a character or not. The links detection
assumes that most of the ligatures between letters are
lower links which correspond to the minima of the upper
outline of the word, and that ligatures following letters
"o", "v", "w" are upper links which correspond to the
maxima of the lower outline of the word. Figure2
provides an example of word segmentation into
graphems. .
II.2.c. Contextual word identification
This stage derives all the combinations of graphem
solutions into words which are identical or close to the
words of the lexicon. The matching itself relies on an
Edit Distance which has been extended to cope with our
segmentation behaviour. Namely, the elementary
transforms have been completed with additional
transforms such as the merging ones to transform broken
letter sequences into a letter, and the splitting ones to
separate merged graphems into letters. Transform
transitional costs were learned.
11.3SegmendRecognize with neural classifier
This analytical technique uses the previous segmentation
and the resulting graphems are input to a neural network
classifier. This one has a modular decomposition into
three sub-networks which handle separately graphem
with upper, lower or no extension.
Each network directly takes as input the graphem image
and processes it through a two-stage multi-layer
perceptron trained with the gradient back propagation
algorithm. The first stage scrutinizes the central third of
the image to output a decision upon which a dedicated
sub-network should be activated to yield the final
recognition results, among 24 possible classes. The
network architectures are based cm local connectivity and
weight sharing techniques. The contextual word
r
Figure 1
Figure 2
643
recognition stage corresponds to the previous one but for
the matching costs which were learned accordingly.
I
Top SO Holistic/
I
Top S Holistic/
III.Combination strategies
111.1
Holistic driven recognition
In this scheme, the holistic method is simply used as an
input filter for the analytical segment/recognizemethods,
as shows the diagram
m.
Table 1 : Method complementarity
111.2
A -
Parallel processing and word level fusion
In this strategy the three word recognition techniques are
in parallel so as to output independent lists of solutions.
The three lists of solutions are then merged to provide a
final unique list (see Figure5.a.). The drawback is that
each contextual analysis module yields word, recognition
costs which are not homogeneous from one set to the
other one. Therefore the first step of the fusion mes to
level the techniques by modifying the words rank with
respect to their costs as follows : the first solution of each
list is assigned the rank 1 and the following solutions are
assigned a rank corresponding to the difference between
their cost and the cost of the first solution. The second
step of the fusion consists in comparing and reorganizing
the 3 lists assuming that if the frrst solution of one list is
"far" from the second one, its probalitility to be the
correct solution is high. The results of this strategy were
compared to the following strategy results (table 3).
Figure 3
The rationale is that the holistic method can provide a list
of words whose general "shape' is similar to the shape of
the input word, but can not select the correct word among
this list. Conversely analytical methods can better
discriminate two "close" words thanks to their higher
precision.
The approach performance has been investigated with the
following test. For different top acceptance choices,
words that were well recognized (OK) by the holistic
method (respectively analytical methods) and badly
recognized (BAD) by the analytical methods
(respectively holistic method) were recorded: (see
table 1- Analytical stands for statistical or neural). Lower
acceptance choices were kept for the holistic method to
maintain a fair balance with the better performances of
the analytical ones.
The.conclusion is that holistic and analytical methods
seem to be complementary whatever the lexicon size is.
Although the holistic method describes words "roughly"
and provides less reliable solutions than the analytical
ones, this method is the only one which overcomes
segmentation weaknesses.Therefore, if a word can not be
well segmented, processing it through the analytical
algorithms will not decide which word from a holistic
filtered list of word solutions is the best one. In parallel,
if a word is '*flat"and has no significant extensions but
can be easily segmented, it will not be selected in the
filtered list of word solutions, whereas it would have
been found without this filtering. To conclude, this fusion
scheme is not suited to the methods presented here.
111.3
Semi-parallel processing and word fusion
This strategy first combines the graphem solutions of the
two segment/recognize techniques to yield a unique
analytical set of solutions. It then merges these word
solutions to the holistic ones (see Figure 5.b.).
The fusion at the graphem level has 2 interests. The 2
graphem classifiers are independent since they do no
extract the same features on the input graphem. Hence
the confidences of common solutions are raised and
inversely reduced when coming from only one of the 2
methods. At the word level, 2 methods are merged
instead of 3, thus minimising the difficulties of
combining hetereogeneous solutions. Regarding
computation time, a substantial gain is also obtained by
removing one of the contextual analysis.
The input of the graphem fusion algorithm are two lists
of 5 graphem candidates with associated confidences
which are normalized. If the statistical classifier does not
output any solution because of classification failure, only
644
i
the neural list is kept. In other cases, the two lists contain
at least one solution. A reducing coefficient is applied to
the neural solutions, since the statistical recognizer
performs better than the neural one. The two lists are
merged using a Bor& Count which further emphasizes
the redundancy between the two methods by rising the
costs of the common solutions.
Table 2 shows the top choice rates obtained on a test base
of 505 address word images provided by the USPS.
These cursive words are either city, state, street, personal
or firm words. The results put forward the improvement
gained by merging the two segmenthecognize
approaches at the graphem level.
I
I
I
I
I
I
I
I
I
100
58.6
66.4
<
.., ,
PAP
Word fusion
-
0
Final word
Solutions
Figure 5 : Word Fusion Parallel (a) and Semi
Parallel (b) processing
In this paper, we presented a multi-classifier recognition
module for the reading of handwritten cursive words. The
module is based on the use of three word recognition
techniques whose complementarity has been tested
through different combination strategies. The final
strategy choice provides top choice recognition rates of
88 %, 76 9% and 65 % for lexicons of size 10, 100 and
loo0 respectively. The high level of recognition rates
obtained for the other highest choices indicates that much
is still to expect of additional tuning of the fusion scheme
so as to improve the top choice rates.
VI.References
1000
42.5
I
I
P
P
~I
I
-
(1992 -Advanced Technoloa Conference Volume 1)
[l]M.Chen, A.Jundu, J.Zhou and S.Srihari - (SUNY Buffalo)
Off-line handwitten word recognition using HMM - p 563
[Z]J.Favata and S.Shihari+XJNY Buffalo) Recognition of
general handwitten words using a hypothesis generation and
reduction methodology - p237
[3]A.Gillies-(Enviromental Research Institute of Michigan)
Cursive word recognition using HMM - p 557
[4]M.Gillous, M.Lerous-(French Post Technical Research
Center) Recognition of cursive script amounts on postal cheques
- p 545
[j]V.Govindaraju, D.Wand and S.Srihari - (SUNY Buffalo)
Using temporal information in off-line word recognition - p 529
[6]F.Kimura, S.Tsuruoka, M.Shridhar and Z.Chen - (Mie
Universib) Context directed handwitten word recognition for
postal service applications - p 199
[7]S.Madhvanath and V.Govindaraju-(SUNY Buffalo) Using
holistic features in handwitten word recognition - p 183
[8]B.Plessis, A.Sicsu, E.Menu and JV. Moreau - (MATRA
MS2i) Isolated handwitten word recognition for contex-1
address reading - p 579
53
The third strategy performed tetter than the two others
and was therefore retained. Improvements on this
strategy can even be foreseen especially for the graphem
iusion where cross-use of the confusion mamx of each
individual classifier should better rake advantage of their
complementarity. Parallel improvements of the third
strategy at the segmentation and the contextual analysis
were subsequently performed and now yields top choice
recognition rates shown in Table 4.
LA
e'Lx
V. Summary
10
77
81.3
--. . ,
I
Solulions
Final word
80.7
64.6
53
Fusion at the graphem level allows to take into account
redundancies between the two lists of solutions derived
by each classifier (statistical and neural) thus ensuring
higher confidences for the solutions when they appear in
the two lists and lower confidences when they appear in
only one list. Therefore the merged list input to the
contextual analysis stage contains propositions with more
discriminating confidences than the ones of each
individual approach. Table 3 compares the results of the
two previous combination strategies at the system end,
thus including the holistic technique.
Lexicon size
Strategy 1
Strategy 2
I
0
I Graphem fusion
Lexicon size Statistical Neural
10
75.7
62.7
100
59
I 43.5
1000
28.7
38.8
Anrlylrrrl MelhOdO
I
I
TaMe 4 : Improved strategy 3
645