Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
Context Dependent Class Language Model based on Word
Co-occurrence Matrix in LSA Framework for Speech Recognition
Welly Naptali∗ , Masatoshi Tsuchiya∗∗ , Seiichi Nakagawa∗
Toyohashi University of Technology
Department of Information and Computer Sciences∗ , Information and Media Center∗∗
1-1, Hibarigaoka, Tempaku-cho, Toyohashi, Aichi, 441-8580
JAPAN
{naptali, nakagawa}@slp.ics.tut.ac.jp, {tsuchiya}@imc.tut.ac.jp
Abstract: We address the issue of data sparseness problem in language model (LM). Using class LM is one way
to avoid this problem. In class LM, infrequent words are supported by more frequent words in the same class.
This paper investigates a class LM based on LSA. A word-document matrix is usually used to represent a corpus in
LSA framework. However, this matrix ignores word order in the sentence. We propose several word co-occurrence
matrices that keep word order. Together with these matrices, we define a context dependent class (CDC) LM which
distinguishes classes according to their context in the sentences. Experiments on Wall Street Journal (WSJ) corpus
show that the word co-occurrence matrix works better than word-document matrix. Furthermore, the CDC achieves
better perplexity than the traditional class LM based on LSA.
Key–Words: LSA, Language model, Word co-occurrence matrix
1
Introduction
corpus is not large enough, word which occurs few
times will have unreliable probability known as a data
sparseness problem. This problem is often solved by
using a good smoothing technique, but only partially.
A class LM is another way to solve the problem by
mapping words into classes. Where frequent words
give more confidence to infrequent words in the same
class. However, this procedure results in an LM with
less parameters and makes LM hard to recognize different histories. We need a class LM without loosing
its ability to recognize different histories.
A speech recognition task is to convert sound wave
into the corresponding word sequence. For a given
acoustic input A, the corresponding word sequence
Ŵ is a word sequence W that has maximum posterior probability P (W |A) as shown by the following
equation:
Ŵ = arg max(log PA (A|W ) + λ log PL (W )), (1)
W
where PA (A|W ) is the probability of A given W
based on acoustic model and PL (W ) is the probability of W based on language model (LM), and λ is
a scaling factor. LM helps an acoustic model in reducing the search space and solving the ambiguity by
assigning probability to word sequences. Most automatic speech recognition systems use n-gram LM because of its simple and quite powerful method. It is
based on an assumption that the current word depends
on only n − 1 preceding words. In case of trigram
(n = 3), LM gives the following probability to a word
sequence W = w1 , w2 , ..., wN :
PT RIGRAM (W ) =
N
Y
P (wi |wi−2 , wi−1 ).
Recently, Latent Semantic Analysis (LSA), which
originally comes from information retrieval, has been
used in language modeling to map a discrete word
into a continuous vector space (LSA space). J.R. Bellegarda [1] combines the global constraint given by
LSA with the local constraint of an n-gram language
model. The same approach is used in [2] but using
Neural Network (NN) as an estimator. Gaussian mixture model (GMM) could be also trained on this LSA
space [3]. Instead of a word-document matrix, a wordphrase matrix is used in [4] as a representation of a
corpus. Their model shows better performance than
the traditional class LM based on mutual information.
However, their model is limited to only the class for
the previous word. Our work is somewhat similar to
theirs with several extensions.
(2)
i=1
The probability is usually trained from a very large
collection of text (corpus). In practice, it is not possible to provide such large corpora. If the training
ISSN: 1790-5109
LSA is usually used together with a worddocument matrix to represent the corpus, where the
275
ISBN: 978-960-474-028-4
Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
order of words is ignored. This is why LSA is also
called ”bag-of-word” method. However, the word order is too important to be ignored. We will show that
LSA could also extract the hidden (latent) information
behind this word order. With a word co-occurrence
matrix, LSA maps a word into a different point in a
continuous vector space according to the word’s position in the sentence. Then a clustering is applied
on each LSA space to get word classes for different
word position. Finally, we propose a context dependent class (CDC) LM that can maintain its ability to
recognize different histories.
This paper divided into the following sections.
Section 2 gives a brief review about LSA. Section 3
introduces our proposed method; context dependent
class language model. Section 4 describes how to
build the matrix representation to get the projection
matrices from words to the LSA vector space. Section
5 reports the experiments of the proposed model. The
conclusions and future works are described in Section
6.
2
Figure 1: Bigram Matrix
project words and matrix V is used to project documents into the LSA space.
To project a word into the LSA space, let V be
a size of a vocabulary. Then each word in the vocabulary can be mapped into the l-dimensional vector
space according to the following equation:
ui = XT ci (1 ≤ i ≤ V ),
where X is a projection matrix with V × l dimension,
that is, corresponds to matrix U or V in (4), and ci is
a discrete vector of word wi , where the i-th element
of the vector is set to 1 and all other V − 1 elements
are set to 0. Since ui is a vector which representing
word wi , any familiar clustering method could be applied. Then the corresponding word wi is mapped to
the class Ci and the probability can be approximated
according to a class-based n-gram LM [5],
Latent Semantic Analysis
LSA extracts semantic relations from a corpus, and
maps them on an l-dimension vector space. Discrete
indexed words are projected into an LSA space by applying singular value decomposition (SVD) to a matrix which representing a corpus. In the original LSA,
this representation matrix is a word-document matrix.
This is a matrix where its cell aij contains occurrence
word wi in document dj , in other words, the rows
of matrix corresponding with words and the columns
corresponding with documents.
Let A be a representation matrix with M × N dimension. SVD decomposes matrix A into three other
matrices U, S, and V
T
AM ×N = UM ×k Sk×k Vk×N
,
PCLASS (wi |wi−n+1 , ..., wi−1 )
= P (Ci |Ci−n+1 , ..., Ci−1 )P (wi |Ci ).
(6)
3
(4)
where l k and  is the best least square fit approximation to A. The rows of matrix U is corresponding with the rows of matrix A, and rows of
matrix V is corresponding with the columns of matrix A. These LSA matrices are then used to project
words/documents into an l-dimension LSA space. In
case of word-document matrix, matrix U is used to
ISSN: 1790-5109
Context Dependent Class Language Model
Because we use the word co-occurrence matrix to represent the corpus, as a result LSA gives a different
projection matrix for different word position. For instance, bigram matrix is decomposed into matrix U
that projects the current word wi , and matrix V is a
projection matrix for the 1st preceding word wi−1 . So
instead of calculating probability using Equation (6),
we need another formulation that could support different classes. The idea is similar to multi-class composite n-gram in [7]. A simple modification on Equation
(6), we define a context dependent class (CDC) language model as
(3)
where k = min(M, N ). Because the solution’s dimensionality is too large for computing resources and
the original matrix A is presumed to be noisy, LSA
matrices (U and V matrix) dimension is set to smaller
than the original
T
ÂM ×N = UM ×l Sl×l Vl×N
,
(5)
PCDC (wi |wi−n+1 , ..., wi−1 )
= P (C(wi , Xi )|C(wi−n+1 , Xi−n+1 ), ..., C(wi−1 , Xi−1 ))
×P (wi |C(wi , Xi ),
(7)
276
ISBN: 978-960-474-028-4
Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
matrix is defined as a matrix where each row represents a current word wi , each column in the first n
columns represents the 2nd preceding word wi−2 , and
each column in the second n columns represents the
1st preceding word wi−1 .
Each cell aij , for the first n columns (1 ≤ j ≤
n), is a co-occurrence frequency when the word wj
occurs as the 2nd preceding word of word wi . For
the second n columns, each cell aij (n + 1 ≤ j ≤
2n) is a co-occurrence frequency of word sequence
wj wi . The resulting SVD matrix V consists of two
different parts. The first n rows are used to project
the 2nd preceding word into an LSA space, and the
next n rows are used to project the 1st preceding word.
Matrix U is used to project a current word. In this
case, the CDC is calculated as follows:
Figure 2: Trigram Matrix
where C(wi , Xi ) is a class of word wi based on the
projection matrix Xi . For an unseen n-gram class, we
applied class backoff to a lower context class.
In [6], language models that model different aspects have been successfully combined with an ngram language model. Here, the statistical n-gram
language model is used to capture the local constraint
using linear interpolation
PL ≈ αPCDC + (1 − α)PN GRAM ,
PCDC (wi |wi−2 , wi−1 )
= P (C(wi , U)|C(wi−2 , V1 ), C(wi−1 , V2 ))
×P (wi |C(wi , U)),
(10)


V1


where V =  −− .
V2
(8)
where α is a weight constant.
4
Matrix Representation
Originally, LSA uses a word-document matrix to represent a corpus. This matrix ignores word order in
the sentence. In this paper, we tried to keep the order by using co-occurrence of word-word matrix. We
propose three kinds of representation matrices, they
are bigram matrix, trigram matrix, and 1-r distance
bigram matrix.
4.1
4.3
Different with bigram or trigram matrix, in this matrix
we tried to collect the information about the previous
word in general by accumulating the co-occurrence of
r-distance bigram words. So the column in 1-r distance bigram matrix represents the preceding words
of wi−r , ..., wi−1 in general as illustrated by Figure 1.
Each cell aij is the accumulation of cooccurrence word wi as a current word with wj appearing from the 1st preceding word to the rth preceding
word. The resulting SVD matrix U is used to project
a current word wi into an LSA space. While matrix
V is used to projecting the preceding words. In this
case, Equation (7) becomes
Bigram Matrix
Bigram matrix is a matrix representation where each
row represents a current word wi , and each column
represents the 1st preceding word wi−1 as illustrated
by Figure 1. Each cell aij is a co-occurrence frequency of word sequence wj wi in the corpus. The
resulting SVD matrix U is used to project a current
word into an LSA space. While matrix V is used to
project the 1st preceding word. In this case, the CDC
in Equation (7) becomes
PCDC (wi |wi−1 )
= P (C(wi , U)|C(wi−1 , V))P (wi |C(wi , U)).
(9)
4.2
PCDC (wi |wi−n+1 , ..., wi−1 )
= P (C(wi , U)|C(wi−n+1 , V), ..., C(wi−1 , V))
×P (wi |C(wi , U)).
(11)
Because the matrix V contains information about all
the preceding words, not only the 1st or 2nd preceding
word in bigram or trigram matrix case, the CDC context could be extended into n-gram context without
increasing cost to calculate the matrix.
Trigram Matrix
Figure 2 illustrates the trigram matrix. Unlike the trigram matrix defined in [4], in this paper the two previous words will not be seen as a phrase, but will be put
as independent words in the column. By doing this,
we made the matrix dimension even smaller. Trigram
ISSN: 1790-5109
1-r distance Bigram Matrix
277
ISBN: 978-960-474-028-4
Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
Figure 3: Bigram model on increasing LSA
dimension
Figure 4: Trigram model on increasing LSA
dimension
5
5.2
Experiments
5.1
Setup
To show the performance of word co-occurrence matrix against word-document matrix, we conducted experiment on these matrices where the probability is
calculated with class-based n-gram LM (6). It means
that word co-occurrence matrix will only use its U
matrix after applying LSA. The models are worddocument matrix (TD) as a baseline, bigram matrix
(B), trigram matrix (T), 1-2 distance bigram matrix
(12DB), 1-3 distance bigram matrix (13DB), 1-4 distance bigram matrix (14DB). For bigram model, the
result is given by Figure 3. It shows that by keeping word order in matrix representation gives improvement on perplexity about 3.62%-12.72% relative. Similar for trigram model (Figure 4), although
the trigram matrix gives worse perplexity on 20 dimensions, overall results show that keeping rhe word
order could improve performance. LSA extracts latent
information that lies within the word order in the word
co-occurrence matrix effectively. Up to this point, the
usage of word co-occurrence matrix is not optimum
yet. We only used matrix U in the model. In the following experiment, we incorporated all LSA matrices
of word co-occurrence matrix in CDC LM.
We consider several CDC models; they are CDC
with bigram matrix (CDC-B), CDC with trigram matrix (CDC-T), CDC with 1-2 distance bigram matrix
(CDC-12DB), CDC with 1-3 distance bigram matrix
(CDC-13DB), CDC with 1-4 distance bigram matrix
(CDC-14DB). As a baseline, a traditional class-based
LM with word-document matrix (TD) is used. Using
a fixed 2000 classes, we varied the LSA dimension
and show the results in Figure 5 for bigram model and
Figure 6 for trigram model. In bigram model, we can
see all word co-occurrence matrices give better result
than the baseline word-document matrix about 7.88%13.48% relative. In trigram model, at 20 dimensions,
The data taken from WSJ corpus from year 1987 to
year 1989 consists of 37 million words. It is divided
into training and test set. The vocabulary is used
ARPA’s official ”20o.nvp” (20k most common WSJ
words, non-verbalized punctuation). By inserting a
beginning sentence, an end sentence, and an OOV
symbols, the vocabulary size is 19,982 words. More
detail is given by Table 1.
Table 1: Experimental data statistics
#Word
OOV Rate
Training Set 36,754,891
0.0236
Test Set
336,096
0.0243
As a baseline, LSA with word-document matrix
is used. The matrix representation was decomposed
and reduced using SVDLIBC 1 with Lanczos method.
The LSA dimension was varied from 20, 50, 100,
and 200 dimensions. The clustering was conducted
by Gmeans2 using K-means algorithm with Euclidean
distance and various numbers of classes from 500,
1000, 2000, and 4000. The models are evaluated using perplexity, defined by the following equation:
1
P erplexity = 2− N log2 PL (W ) .
(12)
For an interpolation model, Katz backoff word-based
trigram language model is used. Build using HTK
Language Model toolkit[8], the perplexity of a conventional word-based trigram LM was 111.54. We
maximize the interpolation weight α from 0.1 to 0.9
with step size 0.1.
1
2
http://tedlab.mit.edu/∼dr/SVDLIBC
http://www.cs.utexas.edu/users/dml/Software/gmeans.html
ISSN: 1790-5109
Results
278
ISBN: 978-960-474-028-4
Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
Figure 5: Bigram model of CDC on increasing LSA
dimension
Figure 7: Interpolated bigram model of CDC with
word-based trigram on incrasing
LSA dimension
Figure 6: Trigram model of CDC on increasing LSA
dimension
Figure 8: Interplated trigram model of CDC with
word-based trigram on incrasing
LSA dimension
the trigram matrix gives worse performance but after
that the performance is increasing. While 1-r distance
bigram matrix gives better performance in any dimension. Next, we interpolated the model with wordbased trigram and show the results in Figure 7 and
Figure 8. Although the interpolation gives larger impact to the baseline, but the word co-occurrence matrix model still gives better perplexity about 0.6%1.68% relative. This is caused by keeping the word
order on word co-occurrence matrix. So a combination with word-based trigram which has strong local
constraint, will impact more on a model which only
captures global constraint.
In the next experiment, we tried to show the
model’s behaviour on increasing the number of clusters. We set a fixed 200 dimensions LSA space and
various numbers from 500 to 4000. The results are
given by Figure 9 for bigram model and Figure 10 for
trigram model. The interpolation model’s results are
also given in Figure 11 and Figure 12. In these figures
we can also see that the proposed word co-occurrence
matrix has better peformance than the original wordISSN: 1790-5109
document matrix. These results also indicate that
CDC performs better than the traditional class-based
language model.
6
Conclusion
In this paper, we demonstrated that word cooccurrence matrix has better performance than the traditional word-document matrix in LSA framework.
One of the reason is because the proposed word cooccurrence matrix keeps word order unlike worddocument matrix. We also showed that the CDC
LM performs better than the traditional class-based ngram LM based on LSA.
In future works, there are still many things that
can be improved, such as using another distance in
the clustering or changing the clustering method itself. We are also looking forward to use another word
extraction method, such as Probabilistic LSA (PLSA)
or Latent Dirichlet Allocation (LDA).
279
ISBN: 978-960-474-028-4
Proceedings of the 8th WSEAS International Conference on APPLIED COMPUTER SCIENCE (ACS'08)
Figure 9: Bigram model of CDC on increasing cluster
size
Figure 11: Interpolated bigram model of CDC with
word-based trigram on increasing
cluster size
Figure 10: Trigram model of CDC on increasing
cluster size
Figure 12: Interplated trigram model of CDC with
word-based trigram on increasing
cluster size
Acknowledgements: The research was supported by
Global COE program ”Frontier of Intelligent Sensing” and MEXT. The views expressed are not necessarily endorsed by the sponsors.
[5] P. Brown, V. Pietra, P. deSouza, J. Lai, and R. Mercer.
Class-based n-gram models of natural language. volume 18, pages 467–479. Computational Linguistics,
1992.
[6] S. Broman and M. Kurimo. Methods for combining
language models in speech recognition. pages 1317–
1320. Interspeech, September 2005.
[7] H. Yamamoto, S. Isogai, and Y. Sagisaka. Multiclass composite n-gram language model for spoken
language processing using multiple word clusters. In
ACL 2001: Proceedings of the 39th Annual Meeting
on Association for Computational Linguistics, pages
531–538. Association for Computational Linguistics,
2001.
[8] S. Yung, G. Evermann, M. Gales, T. Hain, D. Kershaw, G. Moore, J. Odell, D. Ollason, D. Povey,
V. Valtchev, , and P. Woodland. The HTK book (for
HTK Version 3.3). Cambridge, 2005.
References:
[1] J. Bellegarda. Latent semantic mapping. In IEEE
Signal Processing Magazine, pages 70–80, September 2005.
[2] Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin.
A neural probabilistic language model. In Journal of
Machine Learning Research, volume 3, pages 1137–
1155, 2003.
[3] M. Afify, O. Siohan, and R. Sarikaya. Gaussian mixture language models for speech recognition. volume 4, pages 29–32. ICASSP, April 2007.
[4] S. Terashima, K. Takeda, and F. Itakura. A linear
space representation of language probability through
svd of n-gram matrix. In Electronics and Communications in Japan (Part III: Fundamental Electronic
Science), volume 86, pages 61–70, 2003.
ISSN: 1790-5109
280
ISBN: 978-960-474-028-4