Question
Answering
What is Ques+on Answering? Dan Jurafsky Ques%on Answering One of the oldest NLP tasks (punched card systems in 1961) Ques%on:
What do worms eat?
worms
eat
Poten%al-Answers:
Worms eat grass
worms
eat
grass
what
Birds eat worms
birds
eat
2 Simmons, Klein, McConlogue. 1964. Indexing and Dependency Logic for Answering English Ques+ons. American Documenta+on 15:30, 196-­‐204 worms
Horses with worms eat grass
horses
with
worms
eat
grass
Grass is eaten by worms
worms
eat
grass
Dan Jurafsky Ques%on Answering: IBM’s Watson •  Won Jeopardy on February 16, 2011! WILLIAM WILKINSON’S
“AN ACCOUNT OF THE PRINCIPALITIES OF
WALLACHIA AND MOLDOVIA”
INSPIRED THIS AUTHOR’S
MOST FAMOUS NOVEL
3 Bram Stoker Dan Jurafsky Apple’s Siri 4 Dan Jurafsky Wolfram Alpha 5 Dan Jurafsky Types of Ques%ons in Modern Systems •  Factoid ques+ons • 
• 
• 
• 
Who wrote “The Universal Declara4on of Human Rights”? How many calories are there in two slices of apple pie? What is the average age of the onset of au4sm? Where is Apple Computer based? •  Complex (narra+ve) ques+ons: •  In children with an acute febrile illness, what is the efficacy of acetaminophen in reducing fever? •  What do scholars think about Jefferson’s posi4on on dealing with pirates? 6 Dan Jurafsky Commercial systems: mainly factoid ques%ons Where is the Louvre Museum located? In Paris, France What’s the abbrevia+on for limited partnership? L.P. What are the names of Odin’s ravens? Huginn and Muninn What currency is used in China? The yuan What kind of nuts are used in marzipan? almonds What instrument does Max Roach play? drums What is the telephone number for Stanford 650-­‐723-­‐2300 University? Dan Jurafsky Paradigms for QA •  IR-­‐based approaches •  TREC; IBM Watson; Google •  Knowledge-­‐based and Hybrid approaches •  IBM Watson; Apple Siri; Wolfram Alpha; True Knowledge Evi 8 Dan Jurafsky •  a 9 Many ques%ons can already be answered by web search Dan Jurafsky IR-­‐based Ques%on Answering •  a 10 Dan Jurafsky IR-­‐based Factoid QA Document
Document
Document
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Answer Type
Detection
11 Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Dan Jurafsky IR-­‐based Factoid QA •  QUESTION PROCESSING •  Detect ques+on type, answer type, focus, rela+ons •  Formulate queries to send to a search engine •  PASSAGE RETRIEVAL •  Retrieve ranked documents •  Break into suitable passages and rerank •  ANSWER PROCESSING •  Extract candidate answers •  Rank candidates •  using evidence from the text and external sources Dan Jurafsky Knowledge-­‐based approaches (Siri) •  Build a seman+c representa+on of the query •  Times, dates, loca+ons, en++es, numeric quan++es •  Map from this seman+cs to query structured data or resources • 
• 
• 
• 
13 Geospa+al databases Ontologies (Wikipedia infoboxes, dbPedia, WordNet, Yago) Restaurant review sources and reserva+on services Scien+fic databases Dan Jurafsky Hybrid approaches (IBM Watson) •  Build a shallow seman+c representa+on of the query •  Generate answer candidates using IR methods •  Augmented with ontologies and semi-­‐structured data •  Score each candidate using richer knowledge sources •  Geospa+al databases •  Temporal reasoning •  Taxonomical classifica+on 14 Question
Answering
What is Ques+on Answering? Question
Answering
Answer Types and Query Formula+on Dan Jurafsky Factoid Q/A Document
Document
Document
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Answer Type
Detection
17 Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Dan Jurafsky Ques%on Processing Things to extract from the ques%on •  Answer Type Detec+on •  Decide the named en%ty type (person, place) of the answer •  Query Formula+on •  Choose query keywords for the IR system •  Ques+on Type classifica+on •  Is this a defini+on ques+on, a math ques+on, a list ques+on? •  Focus Detec+on •  Find the ques+on words that are replaced by the answer •  Rela+on Extrac+on 18 •  Find rela+ons between en++es in the ques+on Dan Jurafsky Question Processing
They’re the two states you could be reentering if you’re crossing
Florida’s northern border
• 
• 
• 
• 
19 Answer Type: US state Query: two states, border, Florida, north Focus: the two states Rela+ons: borders(Florida, ?x, north) Dan Jurafsky Answer Type Detec%on: Named En%%es •  Who founded Virgin Airlines? •  PERSON •  What Canadian city has the largest popula4on? •  CITY. Dan Jurafsky Answer Type Taxonomy Xin Li, Dan Roth. 2002. Learning Ques+on Classifiers. COLING'02 •  6 coarse classes •  ABBEVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC •  50 finer classes •  LOCATION: city, country, mountain… •  HUMAN: group, individual, +tle, descrip+on •  ENTITY: animal, body, color, currency… 21 Dan Jurafsky Part of Li & Roth’s Answer Type Taxonomy city
country
state
reason
expression
LOCATION
definition
abbreviation
ABBREVIATION
DESCRIPTION
individual
food
ENTITY
NUMERIC
currency
animal
HUMAN
date
title
group
money
percent
distance
size
22 Dan Jurafsky Answer Types 23 Dan Jurafsky More Answer Types 24 Dan Jurafsky Answer types in Jeopardy Ferrucci et al. 2010. Building Watson: An Overview of the DeepQA Project. AI Magazine. Fall 2010. 59-­‐79. •  2500 answer types in 20,000 Jeopardy ques+on sample •  The most frequent 200 answer types cover < 50% of data •  The 40 most frequent Jeopardy answer types he, country, city, man, film, state, she, author, group, here, company, president, capital, star, novel, character, woman, river, island, king, song, part, series, sport, singer, actor, play, team, show, actress, animal, presiden+al, composer, musical, na+on, book, +tle, leader, game 25 Dan Jurafsky Answer Type Detec%on •  Hand-­‐wrioen rules •  Machine Learning •  Hybrids Dan Jurafsky Answer Type Detec%on •  Regular expression-­‐based rules can get some cases: •  Who {is|was|are|were} PERSON •  PERSON (YEAR – YEAR) •  Other rules use the ques%on headword: (the headword of the first noun phrase ater the wh-­‐word) •  Which city in China has the largest number of foreign financial companies? •  What is the state flower of California? Dan Jurafsky Answer Type Detec%on •  Most oten, we treat the problem as machine learning classifica+on •  Define a taxonomy of ques+on types •  Annotate training data for each ques+on type •  Train classifiers for each ques+on class using a rich set of features. •  features include those hand-­‐wrioen rules! 28 Dan Jurafsky Features for Answer Type Detec%on • 
• 
• 
• 
• 
29 Ques+on words and phrases Part-­‐of-­‐speech tags Parse features (headwords) Named En++es Seman+cally related words Dan Jurafsky Factoid Q/A Document
Document
Document
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Answer Type
Detection
30 Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Dan Jurafsky Keyword Selec%on Algorithm Dan Moldovan, Sanda Harabagiu, Marius Paca, Rada Mihalcea, Richard Goodrum, Roxana Girju and Vasile Rus. 1999. Proceedings of TREC-­‐8. 1. Select all non-­‐stop words in quota+ons 2. Select all NNP words in recognized named en++es 3. Select all complex nominals with their adjec+val modifiers 4. Select all other complex nominals 5. Select all nouns with their adjec+val modifiers 6. Select all other nouns 7. Select all verbs 8. Select all adverbs 9. Select the QFW word (skipped in all previous steps) 10. Select all other words Dan Jurafsky Choosing keywords from the query
Slide from Mihai Surdeanu Who coined the term “cyberspace” in his novel “Neuromancer”?
1
4
1
4
7
cyberspace/1 Neuromancer/1 term/4 novel/4 coined/7
32
Question
Answering
Answer Types and Query Formula+on Question
Answering
Passage Retrieval and Answer Extrac+on Dan Jurafsky Factoid Q/A Document
Document
Document
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Answer Type
Detection
35 Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Dan Jurafsky Passage Retrieval •  Step 1: IR engine retrieves documents using query terms •  Step 2: Segment the documents into shorter units •  something like paragraphs •  Step 3: Passage ranking •  Use answer type to help rerank passages 36 Dan Jurafsky Features for Passage Ranking Either in rule-­‐based classifiers or with supervised machine learning • 
• 
• 
• 
• 
• 
Number of Named En++es of the right type in passage Number of query words in passage Number of ques+on N-­‐grams also in passage Proximity of query keywords to each other in passage Longest sequence of ques+on words Rank of the document containing passage Dan Jurafsky Factoid Q/A Document
Document
Document
Document
Document Document
Answer
Indexing
Passage
Retrieval
Question
Processing
Question
Query
Formulation
Answer Type
Detection
38 Document
Retrieval
Docume
Docume
nt
Docume
nt
Docume
nt
Docume
nt
Relevant
nt
Docs
Passage
Retrieval
passages
Answer
Processing
Dan Jurafsky Answer Extrac%on •  Run an answer-­‐type named-­‐en+ty tagger on the passages •  Each answer type requires a named-­‐en+ty tagger that detects it •  If answer type is CITY, tagger has to tag CITY •  Can be full NER, simple regular expressions, or hybrid •  Return the string with the right type: •  Who is the prime minister of India (PERSON) Manmohan Singh, Prime Minister of India, had told
left leaders that the deal would not be renegotiated.!
•  How tall is Mt. Everest? (LENGTH) The official height of Mount Everest is 29035 feet!
Dan Jurafsky Ranking Candidate Answers •  But what if there are mul+ple candidate answers! Q: Who was Queen Victoria’s second son?!
•  Answer Type: Person •  Passage: The Marie biscuit is named ater Marie Alexandrovna, the daughter of Czar Alexander II of Russia and wife of Alfred, the second son of Queen Victoria and Prince Albert Dan Jurafsky Ranking Candidate Answers •  But what if there are mul+ple candidate answers! Q: Who was Queen Victoria’s second son?!
•  Answer Type: Person •  Passage: The Marie biscuit is named ater Marie Alexandrovna, the daughter of Czar Alexander II of Russia and wife of Alfred, the second son of Queen Victoria and Prince Albert Dan Jurafsky Use machine learning: Features for ranking candidate answers Answer type match: Candidate contains a phrase with the correct answer type. PaZern match: Regular expression paoern matches the candidate. Ques%on keywords: # of ques+on keywords in the candidate. Keyword distance: Distance in words between the candidate and query keywords Novelty factor: A word in the candidate is not in the query. Apposi%on features: The candidate is an apposi+ve to ques+on terms Punctua%on loca%on: The candidate is immediately followed by a comma, period, quota+on marks, semicolon, or exclama+on mark. Sequences of ques%on terms: The length of the longest sequence of ques+on terms that occurs in the candidate answer. Dan Jurafsky Candidate Answer scoring in IBM Watson •  Each candidate answer gets scores from >50 components •  (from unstructured text, semi-­‐structured text, triple stores) 43 •  logical form (parse) match between ques+on and candidate •  passage source reliability •  geospa+al loca+on •  California is ”southwest of Montana” •  temporal rela+onships •  taxonomic classifica+on Dan Jurafsky Common Evalua%on Metrics 1.  Accuracy (does answer match gold-­‐labeled answer?) 2.  Mean Reciprocal Rank •  For each query return a ranked list of M candidate answers. •  Query score is 1/Rank of the first correct answer N
•  If first answer is correct: 1 1
∑ rank
•  else if second answer is correct: ½ i
i=1
•  else if third answer is correct: ⅓, etc. •  Score is 0 if none of the M answers are correct •  Take the mean over all N queries 44 MRR =
N
Question
Answering
Passage Retrieval and Answer Extrac+on Question
Answering
Using Knowledge in QA Dan Jurafsky Rela%on Extrac%on •  Answers: Databases of Rela+ons •  born-­‐in(“Emma Goldman”, “June 27 1869”) •  author-­‐of(“Cao Xue Qin”, “Dream of the Red Chamber”) •  Draw from Wikipedia infoboxes, DBpedia, FreeBase, etc. •  Ques+ons: Extrac+ng Rela+ons in Ques+ons Whose granddaughter starred in E.T.? 47 (acted-in ?x “E.T.”)!
(granddaughter-of ?x ?y)!
Dan Jurafsky Temporal Reasoning •  Rela+on databases •  (and obituaries, biographical dic+onaries, etc.) •  IBM Watson ”In 1594 he took a job as a tax collector in Andalusia” Candidates: •  Thoreau is a bad answer (born in 1817) •  Cervantes is possible (was alive in 1594) 48 Dan Jurafsky Geospa%al knowledge (containment, direc%onality, borders) •  Beijing is a good answer for ”Asian city” •  California is ”southwest of Montana” •  geonames.org: 49 Dan Jurafsky Context and Conversa%on in Virtual Assistants like Siri •  Coreference helps resolve ambigui+es U: “Book a table at Il Fornaio at 7:00 with my mom” U: “Also send her an email reminder” •  Clarifica+on ques+ons: U: “Chicago pizza” S: “Did you mean pizza restaurants in Chicago or Chicago-­‐style pizza?” 50 Question
Answering
Using Knowledge in QA Question
Answering
Ques+on Answering in Watson (Deep QA) Dan Jurafsky Ques%on Answering: IBM’s Watson •  Won Jeopardy on February 16, 2011! WILLIAM WILKINSON’S
“AN ACCOUNT OF THE PRINCIPALITIES OF
WALLACHIA AND MOLDOVIA”
INSPIRED THIS AUTHOR’S
MOST FAMOUS NOVEL
53 Bram Stoker Dan Jurafsky The Architecture of Watson Question
(2) Candidate Answer Generation
From Text Resources
(1) Question
Processing
Focus Detection
Lexical
Answer Type
Detection
Question
Classification
Parsing
Document
and
Passsage
Retrieval
passages
Document titles
Anchor text
Document
Document
Document
Document
Document Document
From Structured Data
Relation
Retrieval
Named Entity
Tagging
DBPedia
Relation Extraction
Coreference
Answer
Extraction
Freebase
Candidate
Answer
Candidate
Answer
Candidate
Candidate
Answer
Candidate
Answer
Answer
Candidate
Answer
Candidate
Answer
Candidate
Candidate
Answer
Answer
Candidate
Answer
Candidate
Answer
Candidate
Answer
(3) Candidate
Answer
Scoring
Evidence
Retrieval
and scoring
Text
Time from
Evidence DBPedia
Sources
Answer
Type
Text
Evidence
Sources
Space from
Facebook
Candidate
Answer
+
Confidence
Candidate
Answer
+
Confidence
Candidate
Answer
+
Confidence
Candidate
Answer
+
Candidate
Confidence
Answer
+
Confidence
(4)
Confidence
Merging
and
Ranking
Merge
Equivalent
Answers
Logistic
Regression
Answer
Ranker
Answer
and
Confidence
Dan Jurafsky Stage 1: Ques%on Processing • 
• 
• 
• 
• 
• 
55 Parsing Named En+ty Tagging Rela+on Extrac+on Focus Answer Type Ques+on Classifica+on Named En)ty and Parse Focus Answer Type Rela)on Extrac)on GEO Poets and Poetry: He was a bank clerk in the Yukon before he published “Songs of a Sourdough” in 1907. YEAR COMPOSITION PERSON THEATRE: A new play based on this Sir Arthur Conan Doyle canine classic opened on the London stage in 2007. GEO authorof(focus,“Songs of a sourdough”) publish (e1, he, “Songs of a sourdough”) in (e2, e1, 1907) temporallink(publish(...), 1907) YEAR Dan Jurafsky Focus extrac%on •  Focus: the part of the ques+on that co-­‐refers with the answer •  Replace it with answer to find a suppor+ng passage. •  Extracted by hand-­‐wrioen rules 57 •  "Extract any noun phrase with determiner this” •  “Extrac+ng pronouns she, he, hers, him, “!
Dan Jurafsky Lexical Answer Type city
country
•  The seman+c class of the answer •  But for Jeopardy the TREC answer type taxonomy is insufficient state
reason
expression
LOCATION
definition
abbreviation
ABBREVIATION
DESCRIPTION
individual
food
ENTITY
animal
HUMAN
NUMERIC
currency
date
title
group
money
percent
distance
size
•  DeepQA team inves+gated 20,000 ques+ons •  100 named en++es only covered <50% of the ques+ons! •  Instead: Extract lots of words: 5,000 for those 20,000 ques+ons 58 Dan Jurafsky Lexical Answer Type •  Answer types extracted by hand-­‐wrioen rules •  Syntac+c headword of the focus. •  Words that are coreferent with the focus •  Jeopardy! category, if refers to compa+ble en+ty. 59 Poets and Poetry: He was a bank clerk in the Yukon before he published “Songs of a Sourdough” in 1907. Dan Jurafsky Rela%on Extrac%on in DeepQA •  For the most frequent 30 rela+ons: •  Hand-­‐wrioen regular expressions •  AuthorOf: •  Many paoerns such as one to deal with: •  a Mary Shelley tale, the Saroyan novel, Twain’s travel books, a1984 Tom Clancy thriller •  [Author] [Prose] •  For the rest: distant supervision 60 Dan Jurafsky Stage 2: Candidate Answer Genera%on 61 Dan Jurafsky Extrac%ng candidate answers from triple stores •  If we extracted a rela+on from the ques+on … he published “Songs of a sourdough” (author-of ?x “Songs of a sourdough”)!
•  We just query a triple store 62 •  Wikipedia infoboxes, DBpedia, FreeBase, etc. •  born-­‐in(“Emma Goldman”, “June 27 1869”) •  author-­‐of(“Cao Xue Qin”, “Dream of the Red Chamber”) •  author-­‐of(“Songs of a sourdough”, “Robert Service”) Dan Jurafsky Extrac%ng candidate answers from text: get documents/passages 1.  Do standard IR-­‐based QA to get documents Robert Redford and Paul Newman starred in this depression-­‐era griter flick. (2.0 Robert Redford) (2.0 Paul Newman) star depression era griter (1.5 flick) 63 Dan Jurafsky Extrac%ng answers from documents/
passages •  Useful fact: Jeopardy! answers are mostly the +tle of a Wikipedia document •  If the document is a Wikipedia ar+cle, just take the +tle •  If not, extract all noun phrases in the passage that are Wikipedia document +tles •  Or extract all anchor text <a>The Sting</a>!
64 Dan Jurafsky Stage 3: Candidate Answer Scoring •  Use lots of sources of evidence to score an answer •  more than 50 scorers •  Lexical answer type is a big one 65 •  Different in DeepQA than in pure IR factoid QA •  In pure IR factoid QA, answer type is used to strictly filter answers •  In DeepQA, answer type is just one of many pieces of evidence Dan Jurafsky Lexical Answer Type (LAT) for Scoring Candidates •  Given: •  candidate answer & lexical answer type •  Return a score: can answer can be a subclass of this answer type? •  Candidate: “difficulty swallowing” & LAT “condi4on” 1.  Check DBPedia, WordNet, etc •  difficulty swallowing -­‐> Dbpedia Dysphagia -­‐> WordNet Dysphagia •  condi4on-­‐> WordNet Condi4on 2.  Check if “Dysphagia” IS-­‐A “Condi+on” in WordNet •  Wordnet for dysphagia 66
Dan Jurafsky Rela%ons for scoring •  Q: This hockey defenseman ended his career on June 5, 2008 •  Passage: On June 5, 2008, Wesley announced his re+rement ater his 20th NHL season •  Ques+on and passage have very few keywords in common •  But both have the Dbpedia rela+on ActiveYearsEndDate() !
67 Dan Jurafsky Temporal Reasoning for Scoring Candidates •  Rela+on databases •  (and obituaries, biographical dic+onaries, etc.) •  IBM Watson ”In 1594 he took a job as a tax collector in Andalusia” Candidates: •  Thoreau is a bad answer (born in 1817) •  Cervantes is possible (was alive in 1594) 68 Dan Jurafsky Geospa%al knowledge (containment, direc%onality, borders) •  Beijing is a good answer for ”Asian city” •  California is ”southwest of Montana” •  geonames.org: 69 Dan Jurafsky Text-­‐retrieval-­‐based answer scorer Generate a query from the ques+on and retrieve passages Replace the focus in the ques+on with the candidate answer See how well it fits the passages. Robert Redford and Paul Newman starred in this depression-­‐era gri`er flick •  Robert Redford and Paul Newman starred in The S+ng • 
• 
• 
• 
70 Dan Jurafsky Stage 4: Answer Merging and Scoring •  Now we have a list candidate answers each with a score vector •  J.F.K [.5 .4 1.2 33 .35 … ]!
•  John F. Kennedy [.2 .56 5.3 2 …]!
•  Merge equivalent answers: J.F.K. and John F. Kennedy •  Use Wikipedia dic+onaries that list synonyms: •  JFK, John F. Kennedy, John Fitzgerald Kennedy, Senator John F. Kennedy, President Kennedy, Jack Kennedy •  Use stemming and other morphology 71 Dan Jurafsky Stage 4: Answer Scoring •  Build a classifier to take answers and a score vector and assign a probability •  Train on datasets of hand-­‐labeled correct and incorrect answers. 72 Question
Answering
Ques+on Answering in Watson (Deep QA)