Using Mutual Information
Technique in Cross Language
Information Retrieval
Syandra Sari
Trisakti University
[email protected]
Mirna Adriani
University of Indonesia
[email protected]
ICADL 2008
Bali, 3 Sept 2008
Outline
1.
2.
3.
4.
5.
6.
7.
8.
Introduction
CLIR
Related Research
Our Work
Mutual Information & QE
Experiment
Result and Analysis
Conclusion
Syandra Sari
2
Introduction
• The explosive growth of the World Wide
Web
• Multilingual world in the Internet.
Stimulated the CLIR research area
Syandra Sari
3
CLIR
• The challenge in this area is to
overcome the language barrier
between the query and the document
collection.
• needs to transform queries and
documents into a common
representation, so that monolingual IR
techniques can be applied
Syandra Sari
4
CLIR
Two approaches in CLIR:
• translate the query into the language of
the documents or
• translate the documents into the language
of the query.
Syandra Sari
5
CLIR
Techniques for translation process:
1.Machine translation
2.Bilingual dictionary
3.Parallel or comparable corpus
Syandra Sari
6
Related Research
• Yiming Yang et. al (1998):
– Created a corpus-based term-equivalence matrix
extracted automatically from bilingual corpora
– English-Chinese CLIR and English-Japanese CLIR
• Lavrenko et. al (2002):
– Applied language model for CLIR using parallel corpus
– Chinese-English CLIR
• Martin Braschler (2004):
– Used similarity thesauri for query translation.
– Some European language CLIR (English, Italian,
German, French, Spanish)
Syandra Sari
7
CLIR for Indonesian Language
• In our earlier study (Hayurani et.al, 2006):
– Indonesian-English CLIR
– Machine translation 84.82% of monolingual
performance
– Bilingual dictionary 51.98% of monolingual
performance
– Parallel corpus (using bilingual dictionary) 8% of monolingual performance
Syandra Sari
8
Our Work
• is to do the translation process for
Indonesia-English CLIR using
– parallel corpus
• Pseudo translation
• Mutual Information technique
– Dictionary
– Machine translations
• is aimed at evaluating language resources
and tools available for Indonesian-English
Syandra Sari
9
pair
Mutual Information
• In monolingual IR, was used for finding word
association. (Church, 1990)
• In our work:
• For measuring the association degree between
Indonesian and English word pair.
• The word pair that has highest mutual
information value is considered to be the best
word pair.
Syandra Sari
10
Mutual Information
• Mutual information of two points (words), x and
y, is defined to be:
P(x, y)
I(x, y) = log2
P(x)P(y)
• P(x) is the occurrence probability of word x
• P(y) is the occurrence probability of word y
• P(x,y) is the probability that the words x and y
occur together
• I(x,y) is mutual information value
Syandra Sari
11
Mutual Information
• In (Church, 1990) and (Myung-Gill, 1999) mutual
information value is computed based on word cooccurrence statistics and can be define as follows:
N * f ( x, y )
I ( x, y ) = log 2
f ( x) f ( y )
• f(x) is the number of documents containing x in a corpus;
• f(y) is the number of documents containing y in the corpus;
• f(x,y) is the number of documents containing both x and y
in the corpus;
• N is the number of itemsSyandra
or words
in the corpus
Sari
12
Mutual Information
• We adapted the formula for Indonesian-English
parallel corpus
I(x, y) =
D(x, y)
log2
*(NIndonesian + NEnglish )
(D(x) +1)*(D( y) +1)
Syandra Sari
13
Mutual Information
• D(x) is the number of Indonesian documents
that contain Indonesian word x (exclusive);
• D(y) is the number of English documents that
contain English word y (exclusive);
• D(x,y) is the number of Indonesian-English
document pairs that contain Indonesian word x
and English word y;
• NIndonesian is the number of items or words in
the Indonesian corpus;
• NEnglish is the number of items or words in the
English corpus
Syandra Sari
14
Query Expansion
• is process of adding words found in a
certain number of top English documents
retrieved into the query
• We used language model formula in
choosing the best words to be added to
the query
Syandra Sari
15
Experiment (1)
• Query
– 50 Indonesian queries from CLEF 2006
• Merek Nestle Merek-merek apa yang
dipasarkan oleh Nestle di seluruh dunia
• Nestlé Brands What brands are marketed
by Nestlé around the world
• Collection
– 169.478 English documents from CLEF 2006
(Glasgow Herald 1995, Los Angeles Time 1994)
Syandra Sari
16
Experiment (2)
Building Indonesian-English Parallel Corpus
BUILDING PARALLEL CORPUS
Machine
Translation
Indonesian
corpus
English
corpus
INDONESIAN-ENGLISH
PARALLEL CORPUS
Syandra Sari
17
Experiment (3)
Indonesian-English Parallel Corpus
Indonesian
Queries
English
Queries
Building BILINGUAL LIST WORD
using MUTUAL INFORMATION
& Translation Process
Syandra Sari
IRS
Relevant
Document
18
Experiment (4)
Queries were also translated using:
• Bilingual dictionary
• Machine translations:
– Toggletext
– Transtool
• Pseudo-translation based on parallel
corpus
Syandra Sari
19
Result and Analysis
• The Mean Average
Precision (MAP) of
– monolingual English
queries,
– the Indonesian queries
translated using
• bilingual dictionary
• Toggletext machine
translation
• Transtool machine
translation
Technique
MAP
Monolingual
0,3242
Dictionary
0,1685
(-48,02%)
M. Translation
(Toggletext)
0,2750
(-15,18%)
M. Translation
(Transtool)
0,2529
(-22,00%)
Syandra Sari
20
Result and Analysis
The Mean Average Precision (MAP) of the Indonesian
queries translated using parallel corpus : pseudotranslation and mutual information technique
Technique
MAP
Parallel-Pseudo Translation (PT)
0.2245 (-30.75%)
Parallel-Mutual Information(MI)
0.1085 (-66.53%)
Parallel- Mutual Information with
0.1357 (-58.14%)
query expansion (MI-QE)
Syandra Sari
21
Result and Analysis
• English word
from the
highest value
of MI
(first rank)
• 110 word from
260 word
Indonesian word
English word
from MI
(first rank)
merek
trademark
keadaan
circumstance
pengangguran
unemployment
visa
visa
bahaya
danger
Syandra Sari
22
Result &
Analysis
English word from
Indonesian
MI 1st rank to 5th
word
rank
africa
• English word
from 2nd to
5th value of perawatan
MI
• 51 word
bijaksana
main
Correct
English
word
Rank
afrikan, african,
kwazulu, AFRICA,
gatsha
AFRICA 4
aftercare, surgery,
TREATMENT,
carefree, arthritis
TREAT
MENT
discreet, PRUDENT, PRU
wiser, wisdom,
DENT
indiscreet
romp, overplay,
PLAY
PLAY playground,
nut
Syandra Sari
3
2
3
23
Result and Analysis
• Example of
Indonesian
words get
wrong
English
words as
translation
• 44 word
English word
from MI (first
Indonesian word
to fifth rank)
relentless,
unsuspected,
pengaruh
mindful, favor,
abscond
latch, lug,
lubricant,
produksi
rapacity, sewage
conniption, petal,
ulna, grantee,
universitas
Syandra Sari
24
packager
Result and Analysis
• There are 55 Indonesian words did not
get best / correct English words because
of the stemming process such as:
1. Some Indonesian words get antonym word in
English
• Example: “adil” was translated into “unjustice” using MI
• (“unjustice” is antonym of “justice”).
2. Some Indonesian words get related word in
English.
Example: “matahari” was translated into “sunlight”, “solar”
using MI (“sunlight”,Syandra
“solar”
Sari are related to “sun”).
25
Result and Analysis
3. Some Indonesian words don’t get the best
translation.
Example: “putri” was translated into “daughter” using MI
the best translation for “putri” in English is “princess.
4. Some Indonesian words get English word that is a
translation for other variation of the Indonesian
words.
Example: “acara” was translated into “lawyer” using MI
“Lawyer” means “pengacara” in Indonesia.
Sari
26
“Pengacara” is one ofSyandra
variation
for word “acara”.
Result and Analysis
• There are 9 phrases but only 3 phrases get
correct translation in English
Indonesian phrase
In English
bahan bakar
energi atom
gerhana matahari
gempa bumi
lintas alam
perdana menteri
sidang pengadilan
uang sekolah
undang-undang dasar
fuel
atomic energy
solar eclipse
earthquake
cross country
prime ministry
trial
tuition
constitution
Syandra Sari
Using Mutual Information
firewood explosive
atomic energy
solar eclipse
quake
undergo straightaway
morihiro
unjustice convocation
tuition
basic invitation
27
Conclusion
• We find that mutual information
– could rank the words in good order based on value of
mutual information to get the best translation, however
sometimes it gives the wrong or incorrect translation.
• Machine translation techniques are the best
translation method so far
• Based on the result of our evaluation, there is still
room for improvement to explore parallel corpus
in our future work.
Syandra Sari
28
Future Research
• explore better techniques in finding
bilingual word pairs than mutual
information technique.
• Apply better query expansion technique to
improve the result
Syandra Sari
29
Syandra Sari
30