QuestionAnswering
(andTextualEntailment)
Prof.SameerSingh
CS295:STATISTICALNLP
WINTER2017
March14,2017
BasedonslidesfromDanJurafsky,YejinChoi,StephenClark,DanKlein,Niranjan Balasubramanian,andeveryoneelsetheycopiedfrom.
Upcoming…
Homework
Summaries
Project
• Homework4wasdueonlastnight
• Lowestgradeofthehomeworks willbedropped
• Papersummariesduetonight
• Summary2graded
TA/Instructor
Evaluations
areavailable!
• Finalreportdueinaweek:March20,2017
• Instructionscomingsoon:ACLstyle,5pages(+references)
CS295:STATISTICALNLP(WINTER2017)
2
Outline
QuestionAnswering
IR-BasedQASystem
OtherExtensions
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
3
Outline
QuestionAnswering
IR-BasedQASystem
OtherExtensions
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
4
QuestionsinModernSystems
Factoidquestions
◦
◦
◦
◦
Whowrote“TheUniversalDeclarationofHumanRights”?
Howmanycaloriesarethereintwoslicesofapplepie?
Whatistheaverageageoftheonsetofautism?
WhereisAppleComputerbased?
Complex(narrative)questions:
◦ Inchildrenwithanacutefebrileillness,whatistheefficacyof
acetaminopheninreducingfever?
◦ WhatdoscholarsthinkaboutJefferson’spositionondealingwithpirates?
CS295:STATISTICALNLP(WINTER2017)
5
Commercialsystems:
mainlyfactoidquestions
WhereistheLouvreMuseumlocated?
InParis,France
What’stheabbreviation forlimitedpartnership?
L.P.
What arethenamesofOdin’sravens?
Huginn and
Muninn
What currencyisusedinChina?
Theyuan
Whatkindofnutsareusedinmarzipan?
almonds
WhatinstrumentdoesMaxRoachplay?
drums
WhatisthetelephonenumberforStanfordUniversity?
650-723-2300
CS295:STATISTICALNLP(WINTER2017)
6
ParadigmsforQA
IR-basedapproaches
◦ TREC;IBMWatson;Google
Knowledge-basedandHybridapproaches
◦ IBMWatson;AppleSiri;WolframAlpha;TrueKnowledgeEvi
CS295:STATISTICALNLP(WINTER2017)
7
Manyquestionscanalready
beansweredbywebsearch
CS295:STATISTICALNLP(WINTER2017)
8
IR-basedQuestionAnswering
CS295:STATISTICALNLP(WINTER2017)
9
IR-basedFactoidQA
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
10
Knowledge-basedQA(Siri)
Buildasemanticrepresentationofthequery
◦ Times,dates,locations,entities,numericquantities
Mapfromthissemanticstoquerystructureddataorresources
◦
◦
◦
◦
Geospatialdatabases
Ontologies(Wikipediainfoboxes,dbPedia,WordNet,Yago)
Restaurantreviewsourcesandreservationservices
Scientificdatabases
CS295:STATISTICALNLP(WINTER2017)
11
Hybridapproaches(Watson)
Buildashallowsemanticrepresentationofthequery
GenerateanswercandidatesusingIRmethods
◦ Augmentedwithontologiesandsemi-structureddata
Scoreeachcandidateusingricherknowledgesources
◦ Geospatialdatabases
◦ Temporalreasoning
◦ Taxonomicalclassification
CS295:STATISTICALNLP(WINTER2017)
12
IBM’sWatson
WILLIAMWILKINSON’S
“ANACCOUNTOFTHEPRINCIPALITIESOF
WALLACHIAANDMOLDOVIA”
INSPIREDTHISAUTHOR’S
MOSTFAMOUSNOVEL
BramStoker
WonJeopardy onFebruary16,2011!
CS295:STATISTICALNLP(WINTER2017)
13
MotivationforWatson
CS295:STATISTICALNLP(WINTER2017)
14
SingleSourceisnotSufficient
CS295:STATISTICALNLP(WINTER2017)
15
WatsonArchitecture
Questio
n
Evidenc
e
Sources
Answer
Sources
Primary
Search
Learned Models
help combine and
weigh the Evidence
Evidence
Retrieval
Candidate
Answer
Generation
Evidence
Scoring
Models
Models
Models
Models
Models
Models
Synthesis
Hypothesis and
Evidence Scoring
...
Merging
&
Ranking
CS295:STATISTICALNLP(WINTER2017)
Answer &
Confidence
16
WatsonPerformance
CS295:STATISTICALNLP(WINTER2017)
17
Outline
QuestionAnswering
IR-BasedQASystem
OtherExtensions
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
18
IR-basedFactoidQA
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
19
IR-basedFactoidQA
QUESTIONPROCESSING
◦ Detectquestiontype,answertype,focus,relations
◦ Formulatequeriestosendtoasearchengine
PASSAGERETRIEVAL
◦ Retrieverankeddocuments
◦ Breakintosuitablepassagesandrerank
ANSWERPROCESSING
◦ Extractcandidateanswers
◦ Rankcandidates
◦ usingevidencefromthetextandexternalsources
CS295:STATISTICALNLP(WINTER2017)
20
FactoidQ/A
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
21
QuestionProcessing
AnswerTypeDetection
◦ Decidethenamedentitytype(person,place)oftheanswer
QueryFormulation
◦ ChoosequerykeywordsfortheIRsystem
QuestionTypeclassification
◦ Isthisadefinitionquestion,amathquestion,alistquestion?
FocusDetection
◦ Findthequestionwordsthatarereplacedbytheanswer
RelationExtraction
◦ Findrelationsbetweenentitiesinthequestion
CS295:STATISTICALNLP(WINTER2017)
22
QuestionProcessing
They’rethetwostatesyoucouldbereenteringif
you’recrossingFlorida’snorthernborder
AnswerType:USstate
Query:twostates,border,Florida,north
Focus:thetwostates
Relations:borders(Florida,?x,north)
CS295:STATISTICALNLP(WINTER2017)
23
AnswerTypes:NamedEntities
WhofoundedVirginAirlines?
PERSON
WhatCanadiancityhasthelargestpopulation?
CITY
CS295:STATISTICALNLP(WINTER2017)
24
PartofAnswerTypeTaxonomy
city
country
state
reason
expression
LOCATION
definition
abbreviation
ABBREVIATION
DESCRIPTION
individual
food
ENTITY
HUMAN
NUMERIC
currency
animal
date
title
group
money
percent
distance
Li,Roth.LearningQuestionClassifiers.COLING(2002)
size
CS295:STATISTICALNLP(WINTER2017)
25
AnswerTypes
CS295:STATISTICALNLP(WINTER2017)
26
MoreAnswerTypes
CS295:STATISTICALNLP(WINTER2017)
27
AnswertypesinWatson
2500answertypesin20,000Jeopardyquestionsample
◦ Themostfrequent200answertypescover<50%ofdata
The40mostfrequentJeopardyanswertypes
he,country,city,man,film,state,she,author,group,here,company,
president,capital,star,novel,character,woman,river,island,king,
song,part,series,sport,singer,actor,play,team,show,actress,animal,
presidential,composer,musical,nation,book,title,leader,game
Ferrucci etal.BuildingWatson:AnOverviewofthe
DeepQA Project.AIMagazine.2010
CS295:STATISTICALNLP(WINTER2017)
28
AnswerTypeDetection
Regularexpression-basedrulescangetsomecases:
◦ Who{is|was|are|were}PERSON
Hand-writtenRules
MachineLearning
Otherrulesusethequestionheadword:
◦ (theheadwordofthefirstnounphraseafterthewh-word)
◦ WhichcityinChinahasmostforeignfinancialcompanies?
◦ WhatisthestateflowerofCalifornia?
Questionwordsandphrases
Part-of-speechtags
Parsefeatures(headwords)
NamedEntities
Semanticallyrelatedwords
CS295:STATISTICALNLP(WINTER2017)
29
FactoidQ/A
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
30
KeywordSelectionAlgorithm
1.Selectallnon-stopwordsinquotations
2.SelectallNNPwordsinrecognizednamedentities
3.Selectallcomplexnominals withtheiradjectivalmodifiers
4.Selectallothercomplexnominals
5.Selectallnounswiththeiradjectivalmodifiers
6.Selectallothernouns
7.Selectallverbs
8.Selectalladverbs
9.SelecttheQFWword(skippedinallprevioussteps)
10.Selectallotherwords
Moldovan,Harabagiu,Pasca,Mihalcea,Goodrum,Girju
andRus.TREC(1999)
CS295:STATISTICALNLP(WINTER2017)
31
Choosingkeywords
Whocoinedtheterm“cyberspace”inhisnovel“Neuromancer”?
1
4
1
4
7
cyberspace/1Neuromancer/1term/4novel/4coined/7
CS295:STATISTICALNLP(WINTER2017)
32
FactoidQ/A
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
33
PassageRetrieval
Step1
Step2
Step3
RetrievedocumentsusingIR
◦ querytermsaskeywords
Segmentthedocumentsintoshorterunits
◦ somethinglikeparagraphs
Passageranking
◦ Useanswertypetohelprerank passages
CS295:STATISTICALNLP(WINTER2017)
34
FeaturesforPassageRanking
Eitherinrule-basedclassifiersorwithsupervisedmachinelearning
NumberofNamedEntitiesoftherighttypeinpassage
Numberofquerywordsinpassage
NumberofquestionN-gramsalsoinpassage
Proximityofquerykeywordstoeachotherinpassage
Longestsequenceofquestionwords
Rankofthedocumentcontainingpassage
CS295:STATISTICALNLP(WINTER2017)
35
FactoidQ/A
Document
DocumentDocument
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Answer Type
Detection
CS295:STATISTICALNLP(WINTER2017)
36
AnswerExtraction
Runananswer-typenamed-entitytaggeronthepassages
◦ Eachanswertyperequiresanamed-entitytaggerthatdetectsit
◦ IfanswertypeisCITY,taggerhastotagCITY
◦ CanbefullNER,simpleregularexpressions,orhybrid
Returnthestringwiththerighttype:
◦ WhoistheprimeministerofIndia(PERSON)
◦ ManmohanSingh,PrimeMinisterofIndia,hadtoldleftleadersthat
thedealwouldnotberenegotiated.
◦ HowtallisMt.Everest?(LENGTH)
◦ TheofficialheightofMountEverestis29035feet
CS295:STATISTICALNLP(WINTER2017)
37
RankingCandidateAnswers
Butwhatiftherearemultiplecandidateanswers!
Q:WhowasQueenVictoria’ssecondson?
AnswerType:Person
• Passage:
TheMariebiscuitisnamedafterMarieAlexandrovna,
thedaughterofCzarAlexanderIIofRussiaandwifeof
Alfred,thesecondsonofQueenVictoriaandPrince
Albert
CS295:STATISTICALNLP(WINTER2017)
38
RankingCandidateAnswers
Butwhatiftherearemultiplecandidateanswers!
Q:WhowasQueenVictoria’ssecondson?
AnswerType:Person
• Passage:
TheMariebiscuitisnamedafterMarieAlexandrovna,
thedaughterofCzarAlexanderIIofRussiaandwifeof
Alfred,thesecondsonofQueenVictoriaandPrince
Albert
CS295:STATISTICALNLP(WINTER2017)
39
FeaturesforML
Answertypematch:Candidatecontainsaphrasewiththecorrectanswertype.
Patternmatch:Regularexpressionpatternmatchesthecandidate.
Questionkeywords:#ofquestionkeywordsinthecandidate.
Keyworddistance:Distanceinwordsbetweenthecandidateandquerykeywords
Noveltyfactor:Awordinthecandidateisnotinthequery.
Appositionfeatures:Thecandidateisanappositivetoquestionterms
Punctuationlocation:Thecandidateisimmediatelyfollowedbyacomma,period,
quotationmarks,semicolon,orexclamationmark.
Sequencesofquestionterms:Thelengthofthelongestsequenceofquestion
termsthatoccursinthecandidateanswer.
CS295:STATISTICALNLP(WINTER2017)
40
ScoringCandidatesinWatson
Eachcandidateanswergetsscoresfrom>50components
◦
◦
◦
◦
(fromunstructuredtext,semi-structuredtext,triplestores)
logicalform(parse)matchbetweenquestionandcandidate
passagesourcereliability
geospatiallocation
◦ Californiais”southwestofMontana”
◦ temporalrelationships
◦ taxonomicclassification
CS295:STATISTICALNLP(WINTER2017)
41
Outline
QuestionAnswering
IR-BasedQASystem
OtherExtensions
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
42
AskMSR
QuestionProcessing
1
2
Search
3
5
AnswerScoring
Dumais, Banko, Brill, Lin, Ng,SIGIR(2002)
4
AnswerExtraction
CS295:STATISTICALNLP(WINTER2017)
43
Step1:RewriteQueries
Intuition: Questions are often syntactically quite closeto sentences with the answer
• Where istheLouvreMuseumlocated?
• TheLouvreMuseumislocated in Paris
• Who createdthecharacterofScrooge?
• Charles DickenscreatedthecharacterofScrooge
CS295:STATISTICALNLP(WINTER2017)
44
FeedbackLoops:FALCON
CS295:STATISTICALNLP(WINTER2017)
45
AllenAIScienceChallenge
Whichobjectisthebestconductorofelectricity?
(A) awaxcrayon(B)aplasticspoon
(C)arubbereraser(D)anironnail
CS295:STATISTICALNLP(WINTER2017)
46
AllenAIScienceChallenge
Whichobjectisthebestconductorofelectricity?
(A) awaxcrayon(B)aplasticspoon
(C)arubbereraser(D)anironnail
CS295:STATISTICALNLP(WINTER2017)
47
AllenAIScienceChallenge
Fourthgradersareplanningaroller-skaterace.
Whichsurfacewouldbethebestforthisrace?
(A)gravel(B)sand(C)blacktop(D)grass
CS295:STATISTICALNLP(WINTER2017)
48
AllenAIScienceChallenge
Fourthgradersareplanningaroller-skaterace.
Whichsurfacewouldbethebestforthisrace?
(A)gravel(B)sand(C)blacktop (D)grass
§ Informationretrievalmethodsfail
§ Wordco-occurrencemethodsstruggle
Graders are commonly used in
the construction and maintenance
of dirt roads and gravel roads
Also strong correlations between:
grass « race
gravel « surface
CS295:STATISTICALNLP(WINTER2017)
49
AllenAIScienceChallenge
Astudentputstwoidenticalplantsinthesame
typeandamountofsoil.Shegivesthemthesame
amountofwater.Sheputsoneoftheseplantsnear
asunnywindowandtheotherinadarkroom.
Thisexperimenttestshowtheplantsrespondto
(A)light(B)air(C)water(D)soil
CS295:STATISTICALNLP(WINTER2017)
50
AllenAIScienceChallenge
Astudentputstwoidenticalplantsinthesame
typeandamountofsoil.Shegivesthemthesame
amountofwater.Sheputsoneoftheseplantsnear
asunnywindowandtheotherinadarkroom.
Thisexperimenttestshowtheplantsrespondto
(A)light (B)air(C)water(D)soil
Knowledge needed(forexample):
§ nearsunnywindow® receivelight
§ inadarkroom® nolight
§ testX’sresponsetoY
® compareX+YwithX+not(Y)
CS295:STATISTICALNLP(WINTER2017)
51
Outline
QuestionAnswering
IR-BasedQASystem
OtherExtensions
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
52
NaturalLanguage&Meaning
Variability
Ambiguity
expression
interpretation
Meaning
Language
CS295:STATISTICALNLP(WINTER2017)
53
InferencevsEntailment
Inference
Meaning
Representation
Natural Language
TextualEntailment
CS295:STATISTICALNLP(WINTER2017)
54
TextualEntailment
• Adirectionalrelationbetweentwotextfragments:
Text(t) andHypothesis(h):
t entails h (t Þ h) if
humans reading t will infer that h is most likely true
• Assuming“commonbackgroundknowledge”–
whichisindeedexpectedfromapplications
CS295:STATISTICALNLP(WINTER2017)
55
Example
https://cogcomp.cs.illinois.edu/page/resource_view/9
CS295:STATISTICALNLP(WINTER2017)
56
MoreSentencePairs
1. Some students came to school by car.
Some students came to school.
2. No students came to school by car.
Some students came to school.
3. John drove legally.
John drove.
4. John drove predictably.
John drove.
5. Legally, John could drive.
John drove.
CS295:STATISTICALNLP(WINTER2017)
57
EntailmentwithKnowledge
t entails h (t Þ h) if
humans reading t will infer that h is most likely true
Fortextualentailmenttoholdwerequire:
◦ text ANDknowledge Þ h,but
◦ knowledgeshouldnotentailh alone
Systemsare notsupposedtovalidateh’struthregardlessoft
(e.g.bysearchinghontheweb)
CS295:STATISTICALNLP(WINTER2017)
58
Example
TEXT: …WhilenooneaccusesMadonnaofdoinganythingillegal
inadoptingthe4-year-oldgirl,reportedlynamedMercy,thereare
questionsnonethelessabouthowMadonnaisabletonavigate
Malawi's18-to-24monthvettingperiodinjustamatterofdaysor
weeks…
HYPOTHESIS:
Madonnais50yearsold.
CS295:STATISTICALNLP(WINTER2017)
59
ThirdLabel:Contradictions
t contradicts h (t ^ h) if
humans reading t will find the relations/events
described by h to be highly unlikely given t.
CS295:STATISTICALNLP(WINTER2017)
60
Example
CS295:STATISTICALNLP(WINTER2017)
61
MoreExamples
CS295:STATISTICALNLP(WINTER2017)
62
Applications
Question
Answering
Information
Extraction
Machine
Translation
Information
Retrieval
Question
Who bought Overture?
Overture’s acquisition
by Yahoo
>>
Expected answer form
X bought Overture
Yahoo bought Overture
entails
hypothesized answer
text
Heabhorredthemen’sunctuousways.
Hedislikedthemen’sflatteringways.
Similaritybasedonwhetherdocumententailsthequery.
CS295:STATISTICALNLP(WINTER2017)
63