An Instance-Based Approach for Pinpointing Answers
in Chinese Question Answering
Ang Sun,
Minghu Jiang,
Yanjun Ma
Lab. of Computational Linguistics, Dept. of Chinese Language, Tsinghua University, Beijing, 100084, China
Email: [email protected]
Abstract
In this paper, we propose an instance-based approach
for pinpointing answers in Chinese Question Answering
(QA). We take the view that, for a particular class, the
strategy of answering new questions can be learned from
the already solved ones. To confirm the feasibility of this
new approach, we use question answer pairs (QA pair)
as our learning instances and Maximum Entropy Model
(MEM) as a Machine Learning (ML) technique. The
experiment conducted on the class-LOC_ COUNTRY and
OBJ_LANGUAGE achieves good performance.
1. Introduction
The QA system differs from traditional Information
Retrieval (IR) system in two aspects: one lies in its input
and the other in its output. Its input is not the keywords
used by traditional IR system, but is questions posed in
natural language; its output is not a list of documents or
web pages, but is the correct answers to the questions.
In order to find correct answers, the QA system, is
often built as a framework which consists of three
sequential phases:
Question Analysis: The main target of this phase is to
identify the question types and to select the query words
for retrieving information.
Information Retrieval (IR): This phase utilizes the
acquired query words to retrieve information related to
the question from the Internet or document. Its returned
results is a set of answer candidate snippets.
Answering Extraction: The main task of this phase is
to extract correct answers for the question. It first
extracts answer candidates, then ranks, often scores the
candidates and finally provides the top 5 answers.
According to different standards of answer forms, the
output of this phase can be a paragraph, a sentence, a
phrase or a word.
For example, given Question1: 世界上人口最少的国
家是哪个？(What country in the world has the smallest
population?) Based on the analysis of linguistic features
and semantic information of the question, the Question
Analysis phase first recognizes its question type as
LOC_COUNTRY, and then extracts “世界(the world)
人 口 (population), 最 少 (smallest), 国 家 (country)” as
the query words. Secondly, using the query words, IR
retrieves documents and returns a set of snippets related
to the country which has the smallest population. Thirdly,
Answer Extraction phase extracts and scores the
candidates according to word-overlap information,
similarity between the question and answer sentence and
so forth, and finally gives the correct answer “梵蒂冈
(Vatican City)”.
Placing great emphasis on answer extraction, two
approaches are often applied by QA researchers. One is
rule-based answer extraction, a conventional natural
language processing technique. This approach relies
mainly on its question type patterns and answer patterns.
By recognizing the question type, the corresponding
answer patterns are triggered to extract answers. Take
“FALCON”, a QA system devised by S. Harabagiu et al
[8] for example, for the purpose of processing
DEFINETION questions, FALCON identifies the
question types by its matching results of a set of
linguistic question patterns related to definition and then
one answer pattern would be applied to extract answer.
These linguistic patterns are effective in pinpointing
answers. However, its drawback is obvious. The most
significant one is that it is very costly and time
consuming to manually devise useful linguistic patterns.
Moreover, the interference of rules also adds to the
system’s inefficiency and inaccuracy in finding answers.
Another approach is based on Machine Learning, a
comparative new one in QA research. By applying ML
techniques to training data, researchers [2, 3, 6] can train
models that are able to estimate the probability P (Q, C ) ,
which indicates the candidate’s capability of answering a
question. For example, Echihabi et al [2] developed a
noisy channel model for computing this distance. They
trained a probabilistic model for estimating the
conditional probability P (Q, S A ) . Using the parameters
learned from the model, they can find the sentence
S i ∈ Σ and an answer in it Ai , j by searching for the
S iAi , j
that maximizes the conditional probability
P (Q | S iAi , j ) .
Inspired by the good performance of Lita’s
instance-based approach and other studies which
integrating ML techniques to QA, we apply ML to
Chinese QA answer extraction. Although some
pioneering studies [1, 4] have exploited ML in Chinese
QA, classifying question types and calculating the
confidence score are their main research scope. At the
same time, QA pair is out of their research scope. Thus,
using strategies learned from instances to pinpoint
answers is a challenge for us because there is no
off-the-shelf Chinese QA system constructed on the basis
of QA pairs.
As our approach is a tentative one, we first only focus
on processing factoid (fact-based) questions in this paper.
And according to the requirement of TREC, we set the
correct answer word as our system’s output.
In section 2, we first describe our instance-based
classifier, and then briefly introduce MEM and its
combination with QA. In section 3, we display the
process of generating training samples. Section 4 is our
experiment and section 5 is our conclusion.
2. An instance-based classifier for pinpointing answers
We regard the problem of pinpointing answers for
factoid questions as a classification one: classify the
candidate answers into positive or negative. As the
answer of factoid question is small in size, often
distributes within a sentence, we first classify the
candidate sentences into positive or negative, and then
we classify the words of the positive sentence into
positive or negative. Finally the word with positive tags
will be extracted as correct answer.
For example, given a group of answer candidate
sentences of Question 1:
S1 : 梵蒂冈是世界上人口最少的国家(Vatican City
has the smallest population in the world.)
S 2 : 梵蒂冈：地球上女性最少的国家(Vatican City:
Country with the smallest females on earth.)
S i :…
Our classifier first recognizes S1 as positive sentence
for answering question 1, and S 2 as negative. Secondly,
it recognizes “梵蒂冈” as positive answer word in S1 ,
and other words as negative ones. Finally, our system can
provides “梵蒂冈” as the answer word for question 1.
In order to construct such classifier, we use QA pair as
our instances for learning. Each QA pair consists of one
question and its correct answer. For each question, its
correct answer sentence will be labeled as positive, and
incorrect ones as negative. For each correct answer
sentence, the correct answer word will be labeled as
positive, and incorrect ones as negative.
After extracting features concerning question sentence,
correct answer sentence and correct answer word, we
apply MEM as a ML technique to train our classifier.
The probability model is defined over X × Y , where
X is a set of input symbols, or "histories", and Y is a
set of class labels. As for QA, X is a set of features
extracted from QA pair, and Y reflects the accuracy of
the answer, e.g. Y = 1 means the answer is correct
while Y = 0 means incorrect. Thus, we can regard QA
as a classification problem. A sample ( x, y ) is a pair of
input x = {x1 , L , x m }( x i ∈ X ) and output y ∈ Y .
The model’s probability of a history x together with
a label y is defined as:
p( x, y ) = πµ
k
∏α
f j ( x, y )
j
.
(1)
j =1
Where π is a normalization constant, { µ , α 1 , L α k }
are the positive model parameters and { f 1 L f k } are
known as “features”, where f j ( x , y ) ∈ {0 ,1} . Note that
each parameter α j corresponds to a feature f j .
Given data ( x (1) , y (1) ), L , ( x n , y ( n ) )( x i ∈ X , y i ∈ Y ) ,
let U ( x ( k ) × { y ( k ) }) = { < ~x 1 , ~y 1 > , L , < ~x m , ~y m > } .
k
This means that we enumerate all pairs of an input
sample and label and represent them as < ~xi , ~yi > using
index i (1 ≤ i ≤ m) .
The feature function f j is often defined as follows.
⎧1 if x, y satisfy certain condition
fi = ⎨
.
else
⎩0
(2)
The Maximum Entropy Principle is to find a model
H ( p) = −
∑ p( x | y) log( p( x | y))
which
means
a
x∈X , y∈Y
probability model
that maximizes entropy
p( x | y )
H ( p) .
The constraints are given by:
~
Ef i = Ef i ,
1≤ j ≤ k .
(3)
Where the model’s feature expectation is:
∑ p ( x, y ) f
Ef j =
j ( x,
y) .
(4)
x∈X , y∈Y
And the observed feature expectation is:
~
Ef j =
n
∑ ~p( x , y ) f
i
i
j ( xi ,
yi ) .
(5)
i =1
p ( x i , y i ) denotes the observed probability of
Where ~
( x i , y i ) in the training data. Thus the constraints force
the model to match its feature expectations with those
observed in the training data. In practice, X is very
large and the model’s expectation Ef j can not be
computed directly, so the following approximation is
used:
~
Ef j =
n
∑ ~p( x ) p( y
i
i , xi ) f j ( xi ,
yi ) .
(6)
i =1
p ( xi ) is the observed probability of the history
Where ~
xi in the training set.
3. Generating training samples
QA pair is our resources for generating training
samples.
For each QA pair, we first generate POS of the
question sentence and answer sentence. And we
represent them as POS_Q and POS_A_S.
e.g., POS_Q of question 1: {世界/n 上/m 人口/n
最/d 少/a 的/u 国家/n 是/v 哪个/r ？/w}
POS_A_S of S1 : {梵蒂冈/ns 是/v 世界/n 上/m
人口/n 最/d 少/a 的/u 国家/n 。/w}
Secondly, we extract query words (QW) from POS_Q.
We established three standards for extracting QW. One is
selecting all content words such as name entities
(locations, organizations, person names…), nouns, verbs,
adjectives and so on. Another is selectively extract
cardinal number words, ordinal number words and
measure words, for those do not immediately follow an
interrogative word in the sequence of a question sentence
are selected, otherwise are not selected. For example,
given POS_Q of Question 2: {第一/m 次/q 工业革命
/l 首 先 /d 发 生 /v 在 /p 哪 /r 一 个 /m 国 家
/n ？ /w} (The first Industrial Revolution first took
place in which country?) “第一次” will be selected while
“一个” will not be selected. The last is to remove the
interrogative words and all the stop list words(e.g.是,的,
有,在…).
So, the QW of Question 2 is {第一 次 工业革命 首
先 发生 国家}
Thirdly, we generate POS_A which denotes POS of
the answer word. POS_A is important because it enable
our classifier to learn to distinguish correct answer word
from incorrect words.
e.g., POS_A of question 1 is {/ns}
Finally, for each question’s answer candidate
sentences, we compute the result of three functions:
f1 : The matching result of QW and POS_S (POS of
the candidate sentence), TRUE if POS_S matches more
than x% of the content words, where x% is a
empirical value and can be adjusted in the experiment. If
lower than x% , then f1 =FALSE.
f 2 : The matching result of POS_A and POS_S. TRUE
if there is one POS in POS_S matches POS_A, otherwise
f 2 =FALSE.
f 3 : The matching result of Interrogative Words (IW)
and POS_S. FALSE if there is one POS in POS_S
matches IW of POS_Q, otherwise f 3 =TRUE.
While f1 and f 2 aim at pinpointing answers, f 3 ’s
main function is to filter question sentences whose value
of f1 is more likely to be TRUE. If we do not consider
f 3 , many question sentences in the Internet and
document may be recognized as positive samples, thus it
may mislead our classifier.
The algorithm for generate training sample:
1. Given a question q and its correct answer sentence
s.
2. Use ICTCLAS (a free resource from Institute of
Computing Technology, Chinese Academy of
Sciences) to get POS_Q and POS_A_S, generate QW,
POS_A and IW.
3. Input QW to IR Engine, preprocess the candidate
sentences returned by the Engine.
4. Compute f1 , f 2 and f 3 .
5. Generate sample ( x ( i ) , y ( i ) ) , x (i ) contains the
result obtained by 4, and y (i ) is the label 0 or 1.
6. For each question q, execute 1-5.
4. Experiments
In this section, we first show our IR engine, training
and testing data, then we describe how we conduct
experiment on the class LOC_COUNTRY and
OBJ_LANGUAGE
4.1 IR Engine
Our IR engine is called Web Search, devised by our
lab. It can automatically acquire information of 100
WebPages’ snippet returned by Google, Baidu and
Tianwang.
For each question, we input its QW to Web Search,
and the engine returns 100 WebPages’ snippet which are
related to the question and may contain the correct
answer.
After preprocessing the acquired snippets, by filtering
useless information and by cutting them into sentences,
we get a group of sentences with number ranging from
200 to 400. These answer candidate sentences are our
resources for extracting answers.
4.2 Training and Testing Data
We select 500 questions both from HIT-IRLab QA
question instances (a free resource from IRLab of Harbin
Institute of Technology) and from the Internet. Then we
manually find and collate their answers by searching on
the Internet. Table 1 and Table 2 show the details of our
training and testing data, respectively.
4.3 Experiment
4.3.1 Training and testing
Training Phase:
For each QA pair belonging to the training data, we
first generate its sample ( x ( i ) , y ( i ) ) . Then we apply it to
MEM to train our answer classifier.
Testing phase:
For each question q, we first generate its sample
( x ( i ) , y ( i ) ) . Then we apply it to our classifier to classify
the answer candidates.
Table 1: Training data
Class
LOC_COUNTRY
OBJ_LANGUAGE
Number
of QA
pairs
200
100
Number of
WebPages’
snippet
20000
10000
Number of
answer
candidate
sentences
40000
20000
Table 2: Testing data
Class
LOC_COUNTRY
OBJ_LANGUAGE
Number
of QA
pairs
100
100
Number of
WebPages’
snippet
10000
10000
Number of
answer
candidate
sentences
20000
20000
4.3.2 Experimental results
We use precision, recall and F-measure to evaluate the
output of the experiment.
The Baseline is computed only by the 3 functions, not
by our classifier: It regards a sample as positive if all the
three functions are TRUE, negative if one feature is
FALSE.
Number of correctly classified candidates
Precision =
Total number of candidates
Number of correctly classified Positive Examples
Recall =
Number of Positive Examples
( β 2 + 1) ⋅ Recall ⋅ Precision
β 2 ⋅ Recall + Precision
( β = 1 in all experiments)
Table 3 shows the results on class LOC_COUNTRY.
Table 4 shows the results on class OBJ_LANGUAGE.
Fβ -score =
Table 3: Test Result on Class LOC_COUNTRY
Experiment
Precision
Recall
F-measure
Baseline
55.38%
14.42%
22.88%
Classifier
68.42%
87.56%
76.82%
Table 4: Test Result on Class OBJ_LANGUAGE
Experiment
Precision
Recall
F-measure
Baseline
28.75%
9.42%
14.19%
Classifier
31.18%
66.27%
42.37%
The baseline tells us that if we only rely on the three
functions to estimate whether a sample is correct or not
the performance is not ideal. Compared to the low
performance of baseline, the F-measure shows that our
instance-based classifier achieves good performance.
The precision of LOC_COUNTRY and OBJ_
LANGUAGE increases by 13.04% and 2.43%, and the
recall increases by 73.14% and 56.85%, which means
that our classifier’s ability to recognize positive and
negative samples are both enhanced. Also, it indicates
that the training samples we used is more helpful in
estimating positive samples than negative ones.
5. Conclusions
This paper takes the view that the strategy of
answering new questions can be learned from the solved
ones. Our experimental results confirm the feasibility of
this instance-based approach. Meanwhile, it also
indicates that applying this approach to more classes and
extracting more helpful features is our future work.
Acknowledgement
This work was supported by National Natural Science
Key Foundation of China (No. 60331010). Tsinghua
University 985 research fund; the excellent young
teacher program, Ministry of Education, China; and State
Key Lab of Pattern Recognition open fund, Chinese
Academy of Sciences.
References
[1] Wu M., Zheng X., Duan M., et al. Question
Answering By Pattern Matching, Web-Proofing,
Semantic Form Proofing. http://trec.nist.gov/pubs/
trec12/t12_proceedings.htm
[2] Echihabi A. and Marcu D. “A Noisy-Channel
Approach to Question Answering”, Proc. of
ACL-2003, pp. 16-23 (2003).
[3] Sasaki Y. “Question Answering as Question-Biased
Term Extraction: A New Approach toward
Multilingual QA”. Proc. of ACL 2005, pp.215-222.
[4] You L., Zhou Y. Q., Huang X. J., et al. “A
maximum entropy model based confidence scoring
algorithm for QA”. Journal of Software, 2005,16(8):
pp. 1407−1414.
[5] Lau R., Rosenfeld R., and Roukos S. “Adaptive
Language Modeling Using the Maximum Entropy
Principle”. Proceedings of the Human Language
Technology Workshop, 1993. pp. 108-113. ARPA.
[6] Clarke C., Cormack G., Kemkes G., et al. Statistical
selection of exact answers. 2003. TREC.
[7] Lucian V. L. and Carbonell J. “Instance-based
question answering: a data-driven approach”.
Proceedings of the 42nd Annual Meeting of the
Association for Computational Linguistics 2004, pp.
21-26.
[8] Harabagiu S., Moldovan D., Pasca M., et al.
FALCON:
Boosting
Knowledge
for
Answer
Engines TREC-9 2000 NIST Special Publication
500-249: pp. 479http://trec.nist.gov/pubs/trec12/t12 pro
-ceedings.html
[9]
Zhang Q., Chen Q. Study of Web-Oriented Chinese
Q&A System for IT Domain. Master thesis of
Tsinghua. 2003