0LQLQJ$QVZHUVLQ*HUPDQ:HE
3DJHV
*QWHU1HXPDQQ)HL\X;X
/DQJXDJH 7HFKQRORJ\/DE
*HUPDQ5HVHDUFK&HQWHUIRU$UWLILFLDO,QWHOOLJHQFH
').,6DDUEUFNHQ
6RXUFH*1HXPDQQ:,+DOLID[&$
0RWLYDWLRQ)URP6HDUFK(QJLQHV
WR $QVZHU(QJLQHV
8VHU4XHU\
)URP.H\:UGVWR:K&ODXVH
FRQWH[WXDO
6HDUFK (QJLQHV
4$F\FOHV
$QVZHU(QJLQHV
6KLIWPRUH ÄLQWHUSUHWDWLRQ
HIIRUW³WRPDFKLQHV
8VHUVWLOOVFDUULHVWKH
PDMRUHIIRUWV LQ
XQGHUVWDQGLQJ
:$*WKH :HE
:HEEDVHGDQVZHUH[WUDFWLRQV\VWHPIRUH[WUDFWLQJVHQWHQFHDQG
H[DFWDQVZHUVIURP*HUPDQ(QJOLVKZHESDJHV
6KDOORZTXHVWLRQDQVZHULQJDSSURDFK
1DPHG (QWLW\ 1(UHFRJQLWLRQ DQGPRUSKRV\QWDFWLF
„
DQDO\VLVDVOLQJXLVWLFSUHSURFHVVLQJ
1(EDVHGLQYHUWHGLQGH[IRUSDUDJUDSKVHQWHQFH DQG
„
DQVZHUVHOHFWLRQ 1(EDVHG ÅWH[W]RRPLQJ´
7RNHQ1(RYHUODSDSSURDFKIRUH[DFWDQVZHUVWULQJV
„
$SSURDFKFDQEHVFDOHGXSIRUSURFHVVLQJGLIIHUHQW
TXHVWLRQVW\SHVIDFWRLGOLVWWHPSODWHTXHVWLRQV
(YDOXDWLRQXVLQJTXHVWLRQDQVZHUSDLUVH[WUDFWHGIURPD
SRSXODU*HUPDQTXL]ERRN
4XHU\)RUPXODWLRQ
:KR
won the Nobel Prize 2000 in chemistry?
will be expressed as:
7 : WRN  7 : WRN
 7 : '1( 7 : WRN
 7 :
, 
,
,
,








9 : won 9 : Nobel Prize 9 : 2000  9 : chemistry 9 :


?  
31(
Expected answer type (EAT) as NE-type with value ?
Also possible:
multiple EATs (for template-based questions)
:$*µV&RQWURO)ORZ
Keywords
Query
formulation
Google
NEq& EAT
Relevant
documents
Shallow NL
processing
Stream of sentences (docPOS,
tokens, lexems, NE)
Paragraph
filtering
{<si,sj,sNE-eat>
Inverted
NE index
NE
ranking
|
query-compatible}
Sentence
Equi-class
ranking
Sentence/
short answer
1(5DQNLQJ
6HDUFKIRU DOOSRVVLEOH 1(V LQWKH 1*RRJOHUHWULHYHGGRFXPHQWV
'
1
FRYHUHGE\WKH 1(JUDPPDUVRIRXUVKDOORZ1/3V\VWHP
1RWHRXU1(UHFRJQL]HUDOVRSHUIRUPV 1(UHIHUHQFHUHVROXWLRQ
HJ0DUWLQ0DULHWWD&RUS
0DULHWWD
NE-weighting schema
❏
(DFK 1(JHWV DZHLJKWDFFRUGLQJ WR
❏
1(VPHQWLRQHGYHU\RIWHQ LQ
‡
$EVROXWHIUHTXHQF\ RI1(LQ'
‡
'RFXPHQWIUHTXHQF\ RI1(
KLJKE\ *RRJOHUHFHLYH KLJK
‡
'RFXPHQWUDQNLQJGHWHUPLQHG
UDQN
E\ *RRJOH
GLIIHUHQWGRFXPHQWV UDQNHG
1
Æ
$QVZHUH[WUDFWLRQZLOOEH
HIIHFWHGE\UHGXQGDQF\ DQG
PXOWLSOHGRFXPHQWV
3DUDJUDSK6HOHFWLRQ
1RWHHDFKGRFXPHQWLV DOLVWRIVHQWHQFHV DVHQWHQFHLV D
VHTXHQFH RIWRNHQV DQG1(¶VDVLGHQWLILHGE\ RXU 1/3V\VWHP
6WHS 6WHS &RPSXWH DQLQYHUWHG 1(LQGH[ OLVWVXEFODVVLILHGE\W\SHV
&RPSXWH DQLQYHUWHG 1(LQGH[ OLVWVXEFODVVLILHGE\W\SHV
„
„
(DFKIRXQG 1(KDVDSRLQWHULQWR DOOVHQWHQFHV LQ'
LQZKLFKLW
(DFKIRXQG 1(KDVDSRLQWHULQWR DOOVHQWHQFHV LQ' 1LQZKLFKLW
1
RFFXUV
RFFXUV
6WHS 6WHS )RUHDFK 1( ZKLFKLV W\SHFRPSDWLEOHZLWK ($7
)RUHDFK 1( ZKLFKLV W\SHFRPSDWLEOHZLWK ($7
„
„
([WUDFW DVHW RISDUDJUDSKFDQGLGDWHV ^V V V
V V
V V
V V `
([WUDFW DVHW RISDUDJUDSKFDQGLGDWHV ^V V V 1(V V 1(
V V 1(
V V `
„
„
6HOHFWWKHRQHZLWKKLJKHVWQXPEHU RIPDWFKLQJ TXHU\NH\ZRUGV 6HOHFWWKHRQHZLWKKLJKHVWQXPEHU RIPDWFKLQJ TXHU\NH\ZRUGV 1(
1(
1(
WKHLU GLVWDQFHIURP 1(
WKHLU GLVWDQFHIURP 1(
Robustness:
For relevant sentences: no need to contain NEEAT (but a referring general NP);
Hence, can tolerate NE-gaps caused by the NE-recognizer, and helps to guide NPreference resolution
6HQWHQFH&ODVV5DQNLQJ
&ROOHFW DOOVHQWHQFHVIURPDOOVHOHFWHGSDUDJUDSKVZLWKVDPH UDQN
LQWR DFODVV
Q
= ( RO7RNHQ +
(&
RO1(
) * (1 . 5 +
∑
L
U
1(
=1
( ($7 ) L
Q
)
5DQNLQJ
QXPEHU RIRYHUODSSLQJHOHPHQWV WRNHQV1(
EHWZHHQ
12($7
TXHU\ DQGVHQWHQFH
ZHLJKWHG UDQNRIRFFXUHQFLHV RILQVWDQFHV RI1(
($7
&DQGLGDWHV
6HOHFWIURP 1EHVW&ODVVHVDVHQWHQFH DQGDQLQVWDQFHV RI1(
($7
([DPSOH4XHVWLRQ$QVZHU 3DLU
:HOFKHV3VHXGRQ\PQDKP1RUPD-HDQ%DNHUDQ"
:KLFKSVHXGRQ\PGLG1RUPD-HDQ%DNHUXVH"
<eclass rank=’7.254032855348923’>
<sentence url=’http://www.beatlesfan.de/alle.htm’>
0DULO\Q0RQURHZDUGHU.XHQVWOHUQDPH YRQ1RUPD-HDQ0RUWHQVRQ
DXFKEHNDQQWDOV1RUPD-HDQ%DNHU
</sentence>
<exact-answer type =’PERSON’>
<name rank=’0.6029282321968642’>
0DULO\Q0RQURH
</name>
<name rank=’0.024088195477597402’>
1RUPD-HDQ0RUWHQVRQ
</name>
</exact-answer></eclass>
4XHU\DQG*RRJOH5HWULHYDO
:HU IORJ]XP
HUVWHQ 0DODOOHLQ
XQGQRQVWRSYRQ
1HZ<RUNQDFK
3DULVEHUGHQ
$WODQWLN"
:KRIOHZIRUWKHILUVWWLPH
DORQHDQG QRQVWRS IURP1HZ
<RUNWR3DULVRYHUWKH$WODQWLFLQ
"
,QYHUWHG OLVWRILQVWDQFHV RI
H[SHFWHGDQVZHUW\SH
%HVW1(
($7
WLPHVIRXQG LQ
GLIIHUHQW
GRFXPHQWV LQFO
1(UHIHUHQFHV
5HIHUULQJ
GRFXPHQWV
%HVWVHQWHQFH DQGH[DFWDQVZHU
,QLWLDO([SHULPHQWV
&RUSXV
4XHU\DQVZHUSDLUVIURPSRSXODU *HUPDQTXL]HERRN
„
Š 4XHVWLRQVIURPPRUHWKDQ VXEMHFWDUHDVSKLORVRSK\QDWXUH
VFLHQFHKLVWRU\JHRJUDSK\FXOWXUHVSRUWHWF
Š 4XHVWLRQVIRUPDOL]HGLQGHSHQG RIDQ\ 4$V\VWHPLQPLQG
7ZR WHVWHQYLURQPHQWVUHVWULFWHG YVH[WHQGHG
„
&RPSDULVLRQ RI:$*UHVXOWVZLWK 0
WRS*RRJOH
VQLSSHWV
0HDVXUH
5HFDOOWKHSHUFHQWDJHRIWKHTXHVWLRQVDQVZHUHGFRUUHFWO\
„
FRPSDUHGWRDOOTXHVWLRQVRIWKHWHVWFRUSXV
0HDQ5HFLSURFDO5DQN055IRU1TXHVWLRQVDQGUHFLSURFDO
„
UDQNRIIRXQGDQVZHUIRUTXHVW,ò«RU
1
055
=
∑
L
=1
1
UDQN
1
L
7ZR7HVW(QYLURQPHQWV
5HVWULFWHG
„
SHUVRQUHODWHGDQGORFDWLRQUHODWHG
„
$VVXPH1(LQVWDQFHVRIH[SHFWHGDQVZHUW\SHVDUHFRYHUHGE\
RXU1(UHFRJQL]HU
Š
Š
Who became Israel’s first prime minister in 1949?
In which former soviet state was the nuclear power plant
Chernobyl?
([WHQGHG
„
ORFDWLRQUHODWHGRUJDQL]DWLRQUHODWHGSHUVRQUHODWHG
„
WKHH[SHFWHGDQVZHUW\SHRIDTXHVWLRQLVQRWDOZD\VFRYHUHGE\
RXU1(UHFRJQL]HU
Š
Š
What¶s the name of the organisation found in 1971, which helps
people all over the world?
Which boy group did the English singer Robbie Williams belong to
before his Solo Career?
$YHUDJH5HVXOWV RI5HVWULFWHG
7HVW
Person
Location
Average performance of the Google snippets
VQLSSHW
all questions
Excluding no
document cases
MRR
(N=5)
0.103
correct
top1
0.3
correct
top3
0.35
0.122
0.35
0.41
VQLSSHW
all questions
MRR
(N=5)
correct
top1
correct
top3
0.092
0.21
0.52
MRR
(N=5)
correct
top1
correct
top3
0,135
0.31
0.42
MRR
(N=5)
Recall
top1
Recall
top3
0,126
0.26
0.37
Average performance of WAG exact answers
exact
answer
MRR
(N=5)
correct
top1
correct
top3
all questions
0.212
0.45
0.55
excluding no
document cases
0.236
0.53
0.64
exact
answer
all questions
Average performance of WAG sentence answers
MRR
(N=5)
Recall
top1
Recall
top3
All questions
0,216
0.5
0.55
excluding zero
document cases
0,254
0.59
0.64
sentence
sentence
all questions
$YHUDJH5HVXOWV RI([WHQGHG 7HVW
Location
Person
Average performance of the Google snippets
VQLSSHW
all questions
MRR
(N=5)
Recall
Top1
Recall
top3
0,126
0.34
0.48
VQLSSHW
all questions
MRR
(N=5)
Recall
top1
Recall
top3
0.113
0.216
0.6
Average performance of WAG exact answers
exact
answer
all questions
MRR
(N=5)
Recall
top1
Recall
Top3
0,127
0.314
0.4
exact
answer
all questions
MRR
(N=5)
Recall
top1
Recall
top3
0.16
0.44
0.56
Average performance of WAG sentence answers
MRR
(N=5)
Recall
top1
Recall
Top3
sentence
all questions
0,154
all questions
Sentence
0.4
0.485
MRR
(N=5)
Recall
top1
Recall
top3
0,183
0.32
0.72
6XPPDU\RI(YDOXDWLRQ
*RRJOHLQ5HVWULFWHG
Metric
snippet
MRR (N=5)
0.0975
Recall top1
0.255
Recall top3
0.435
:$*LQ5HVWULFWHG
Metric
snippet
MRR (N=5)
0.136
Recall top1
0.310
Recall top3
0.61
:$*LQ([WHQGHG
Metric
exact answer
Sentence
MRR (N=5)
0.174
0.171
MRR (N=5)
0.126
0.166
Recall top1
0.38
0.38
Recall top1
0.29
0.365
Recall top3
0.48
0.46
Recall top3
0.445
0.61
Sentence
exact answer
Metric
*RRJOHLQ([WHQGHG
4XDOLWDWLYH$QDO\VLV
'RHVWKH4XL]%RRNSURYLGHDFWXDOO\YDOLGDQVZHUNH\V"
„
7UDGHRIIEHWZHHQ:$*DQVZHUVDQG4XL]ERRNDQVZHUNH\V
„
,QFRPSOHWHQHVVRIWKHSURYLGHGDQVZHUV
([DFW$QVZHUDQG6HQWHQFH$QVZHU
„
6HQWHQFHDQVZHULVPRUHUREXVWWKDQH[DFWDQVZHU
„
([DFWDQVZHUVSURYLGHPRUHFOHDUUHVSRQVHLQFDVHRIQRQ³ZHOO
IRUPHG´ VHQWHQFHV
5HGXQGDQF\DQG3RSXODU,QIRUPDWLRQ
0DMRULW\SROOLQJDORQHLVQRWVXIILFLHQWWRJXDUDQWHHWKHFRUUHFW
„
DQVZHU
0RVWSRSXODULQIRUPDWLRQZLOOEHUDQNHGKLJKHUWKDQWKHFRUUHFW
„
DQVZHULQVRPHFDVHV
$QDSKRUD5HVROXWLRQ
6HQWHQFHVPD[LPDOO\PDWFKHGZLWKTXHULHVFRQWDLQVRPHWLPHV
„
DQDSKRULFH[SUHVVLRQVLQVWHDGRIDQVZHUVWKHUHIRUH
3DUDJUDSKVFDQEHXVHGDVGLVFRXUVHFRQWH[WIRUDQDSKRU
UHVROXWLRQ
„
6XPPDU\
WKHOHVVRQZHOHDUQHG
:HEEDVHGDQVZHUH[WUDFWLRQ EHQHILWV IURP UREXVW
DQGHIILFLHQW/7WHFKQRORJ\
7KHEHWWHU1(UHFRJQLWLRQLVWKHEHWWHUWKHDQVZHU
H[WUDFWLRQSURFHVVLVZUWFRYHUDJHDQGILQH
JUDLQHGQHVV
8VLQJD1(DQFKRUHGSDUDJUDSKVHQWHQFHDQG
„
7DNLQJLQWRDFFRXQWWKHUHGXQGDQF\RQWKH:HE
„
3URFHVVLQJFRPSOH[OLVWWHPSODWHTXHVWLRQV
+DQGOLQJPXOWLSOHGRFXPHQWV
„
SKUDVHVHOHFWLRQVWUDWHJ\KHOSVLQ
)XWXUHZRUN&URVVODQJXDJH
7HPSODWHEDVHG 4$
&URVVODQJXDJHZHEEDVHG4$
„
*HUPDQTXHULHV(QJOLVKGRFXPHQWVDQGYLFHYHUVD
„
)LUVWSURWRW\SHIRU7UHFOLNHFRUSXVH[LVWV
1HXPDQQ6DFDOHDQX&OHI
:HEEDVHG'\QDPLF,QIRUPDWLRQ([WUDFWLRQ
„
6SHFLILFDWLRQ RIPXOWLVORWTXHVWLRQV
„
([WUDFWLRQ0HUJLQJ RISDUWLDOWHPSODWHV
Š 6LQJOHLQVWDQFHV
Š 0XOWLSOHLQVWDQFHV OLVWEDVHGWHPSODWHPHUJLQJ
)LUVWSURWRW\SHQRW\HWHYDOXDWHG
„
4XHU\6SHFLILFDWLRQ0XOWLSOH
([SHFWHG$QVZHU7\SHV
5DQNLQJDFFRUGLQJ WRDOO
SURFHVVHG 1(W\SHV
7HPSODWHH[WUDFWLRQWKURXJK
PHUJLQJ
6RIDU'RKDQGOH
6LQJOH,QVWDQFH
LQVLQJOHVHQWHQFH
1H[WVWHSPHUJLQJ
RISDUWLDOWHPSODWHV
'RQH±
7KDQN\RXIRU\RXUDWWHQWLRQ
:$*([DPSOH
3DUDJUDSKUHWULHYDOIRUWHPSODWHH[WUDFWLRQWHPSODWHPHUJLQJWHPSODWHPLQLQJ
„
VHQWHQFH!
'$7(! '$7(! &KHPLH&KHPLH1REHOSUHLV (OHNWULVFKOHLWHQGH
.XQVWVWRIIHNRHQQWHQ GLH7HFKQLN GHV-DKUKXQGHUWVEHVWLPPHQ'XUFK GLH
(QWGHFNXQJOHLWIDHKLJHU.XQVWVWRIIHKDEHQ GLHEHLGHQ 1DWLRQDOLW\!86
$PHULNDQHU1DWLRQDOLW\! 31!$ODQ+HHJHU 31! YRQGHU2UJDQL]DWLRQ!
8QLYHUVLWDHW YRQ.DOLIRUQLHQ2UJDQL]DWLRQ!
LQ/RFDWLRQ!6DQWD%DUEDUD/RFDWLRQ!
XQG31!$ODQ0DF'LDUPLG 31!YRQGHU2UJDQL]DWLRQ!8QLYHUVLWDHW YRQ
3HQQV\OYDQLD 2UJDQL]DWLRQ! XQGGHU1DWLRQDOLW\! -DSDQHU 1DWLRQDOLW\! 31!
+LGHNL6KLUDNDZD 31! YRQGHU2UJDQL]DWLRQ!8QLYHUVLWDHW 7VXNXED
2UJDQL]DWLRQ! $QZHQGXQJHQZLH/HXFKWGLRGHQRGHU$Q]HLJHQGLVSOD\V LQ
0RELOWHOHIRQHQPRHJOLFKJHPDFKW
VHQWHQFH!
>3HUVRQ1DPH$ODQ+HHJHU
2UJDQL]DWLRQ8QLYHUVLWDHW YRQ.DOLIRUQLHQ
2UJDQL]DWLRQ8QLYHUVLWDHW7VXNXED
/RFDWLRQ
/RFDWLRQ6DQWD%DUEDUD
6DQWD%DUEDUD
1DWLRQDOLW\86$PHULNDQHU@
>3HUVRQ1DPH$ODQ0DF'LDUPLG
2UJDQL]DWLRQ8QLYHUVLWDHW YRQ3HQQV\OYDQLD
/RFDWLRQ 6DQWD%DUEDUD
1DWLRQDOLW\86$PHULNDQHU@
>3HUVRQ1DPH+LGHNL6KLUDNDZD
1DWLRQDOLW\-DSDQHU@