Question Answering Tasks
with Embeddings
KERE Seminar
March 25 th , 2015
LU Yangyang @ SEKE Team
[email protected]
Outline
I.
Open QA with KB Triples
● W. Yih et al. Semantic parsing for single-relation question. ACL’14.
● A. Bordes et al. Open question answering with weakly supervised embedding
models. ECML’14.
● A. Bordes et al. Question Answering with Subgraph Embeddings. EMNLP’14.
II.
Answer Selection via Unstructured Text Understanding
● L. Yu et al. Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop).
● M. Iyyer et al. A Neural Network for Factoid Question Answering over Paragraphs.
EMNLP’14.
III. More QA tasks
● J. Weston et al. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
arXiv’15.
2
I. Open QA with KB Triples
 Introduction:
● Using a knowledge base to answer factoid questions
● Given a question q, the corresponding answer is given by a triple t from a KB.
 Related Work
● Information Retrieval based
● Semantic Parsing based
 References:
● Semantic parsing for single-relation question. ACL’14 (short paper)
● Wen-tau Yih, Xiaodong He, Christopher Meek
● Microsoft Research (Redmond)
Single-Relation QA, Convolutional NN based Semantic Model
● Open question answering with weakly supervised embedding models. ECML’14
● Antoine Borde, Jason Weston, and Nicolas Usunier
● CNRS，Google
Entity Answer Ranking, Bilinear Scoring Function, Negative Sampling
● Question Answering with Subgraph Embeddings. EMNLP’14
● Antoine Bordes, Jason Weston, and Sumit Chopra
● Facebook
3
Semantic parsing for single-relation question. ACL’14
 Introduction
● TASK: Answering single-relation factual questions
● A single-relation question is defined as a question composed of an entity
mention and a binary relation description, where the answer to this question
would be an entity that has the relation with the given entity.
● Decompose each question into an entity mention and a relation pattern
● Pattern-relation(KB) Mapping + mention-entity mapping
● determined by corresponding semantic similarity models
Semantic parsing for single-relation question (cont.)
 Convolutional Neural Network based Semantic Model (CNNSM)
● two CNNSMs: for pattern–relation and mention–entity pairs, respectively
word: a count vector of letter-trigrams
Semantic parsing for single-relation question (cont.)
 Model Training
● PARALEX:1.8M pairs of questions v.s. single-relation database queries
QUESTION: When were DVD players invented?
DB QUERY: be-invent-in(dvd-player,?)
● two parallel corpora derived from PARALEX
● pattern-relation, mention-entity
 Results
Open question answering with weakly supervised embedding models. ECML’14
 Introduction
● TASK: Answering simple factual questions on a broad range of topics
● Single KB triples: both the question and an answer (may be many)
Q: What is parrotsh's habitat?
KB: (parrotfish.e, live-in.r, southern-water.e)
A: southern-water.e
Q: What is the main language of Hong-Kong?
KB: (cantonese.e, be-major-language-in.r, hong-kong.e)
A: cantonese.e.
● Main Difficulties: come from lexical variability rather than from complex syntax
● Multiple answers per question >> Score function for Ranking
● The absence of a supervised training signal >> Weak supervision
● Automatically generating questions from KB triples --> Training data
● Question paraphrase dataset as supplement
● Mapping questions and answers into a same feature representation space
● Learning low-dimensional vector embeddings of words and of KB triples
● 800k words, 2.4M entities and 600k relationships
Open question answering with weakly supervised embedding models (cont.)
 Task Definition:
● Given a question q, the corresponding answer is given by a triple t from a KB.
● Learning a function S(·): score question- answer triple pairs (q, t)
 Training Data Preparation
● 𝓚: Knowledge Base
● REVERB: an open-source database( ≥ 14M triples, ≥ 2M entities, ≥ 600k
relationships)
● More noise but also more relation types than FREEBASE (~20k relationships)
● 𝓓: Question Generation
● 16 seed question patterns
● Generated questions:
● simplistic syntactic structure
● semantically invalid
● 𝓟: Question Paraphrases
● 18M question-paraphrase pairs, with 2.4M distinct questions
● only 55%: actual paraphrases
Open question answering with weakly supervised embedding models (cont.)
 Embedding-based Model
word embedding lookup table
● Question-KB Triple Scoring
the (sparse) binary representation of q
indicating absence or presence of words
the (sparse) binary representation of t indicating
absence or presence of entities and relationships
entities and relationships embedding lookup table
● Training by Ranking: Negative sampling
● a corrupted triple t’: pick another random triple ttmp from 𝒦, replace with 66% chance
each member of t by the corresponding element in ttmp.
● Multitask Training with Paraphrases Pairs
● Fine-tuning the Similarity between Embeddings
Open question answering with weakly supervised embedding models (cont.)
 Experiments
● Test Set: 37 questions from WIKIANSWERS >> 691 questions via paraphrases >> PARALEX:
48k candidate triples >> hand-labeled
● Reranking
● sorting by the score S(q, t) or Sft(q,t)
● Full Ranking
● for 691 questions: ranked all 14M
triples from REVERB
● Semantic Similarity Measure
● Evaluation on WebQuestions
● NL questions matched with answers corresponding to entities of Freebase
Question Answering with Subgraph Embeddings. EMNLP’14
 Extension version of ECML’14 paper
● Data Preparation
question-answer pairs
triples >> generated questions
distinct questions
question paraphrase clusters
Question Answering with Subgraph Embeddings (cont.)
 Embedding Questions and Answers
• W: a lookup table for words, entities and relations
• 𝜙 𝑞 : a sparse vector indicating the number of times
each word appears in the question q (usually 0 or 1)
• 𝜑 𝑎 :
• Single Entity
• Path Representation: 1- or 2-hops path
• Subgraph Representation:
path entities + the entire subgraph of entities
connected to the candidate answer entity
Question Answering with Subgraph Embeddings (cont.)
 Experiments:
● Objective Function: minimize
● Multitask Training of Embeddings:
● Inference:
● 𝒜(𝑞): the candidate answer
set
● C1: all triples from FREEBASE
involving the identified
entity in the question
● C2: add 2-hops entities,
their paths contain top 10
type related relations
Outline
I.
Open QA with KB Triples
● W. Yih et al. Semantic parsing for single-relation question. ACL’14.
● A. Bordes et al. Open question answering with weakly supervised embedding models.
ECML’14.
● A. Bordes et al. Question Answering with Subgraph Embeddings. EMNLP’14.
II.
Answer Selection via Unstructured Text Understanding
● L. Yu et al. Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop).
● M. Iyyer et al. A Neural Network for Factoid Question Answering over Paragraphs.
EMNLP’14.
III. More QA tasks:
● J. Weston et al. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
arXiv’15.
14
II. Answer Selection via Unstructured Text Understanding
 Introduction:
● Question – Answer : 1 – n
● Given the question q, select the correct answer(s)
● The question or answer may contains several sentences
● Dealing with the unstructured text directly
● Without KB triples matching
 References:
● Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop)
● Lei Yu, Karl Moritz Hermann, Phil Blunsom, Stephen Pulman
● University of Oxford, Google
TREC Answer Selection Dataset, Bilinear BoW & Bigram Model
● A Neural Network for Factoid Question Answering over Paragraphs. EMNLP’14
● Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, Hal Daume III
● University of Maryland, University of Colorado, Stanford University
Quiz Bowl, Recursive NN
Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop)
 Introduction
● Answer sentence selection:
the task of identifying sentences that contain the answer to a given question.
● Matching the semantic similarity between question and answer
● Prior work:
● syntactic matching of parse trees
minimal edit sequences between dependency parse trees
● including semantic features from resources such as WordNet
● Neural network-based distributional sentence models
● sentiment analysis, paraphrase detection and document classification
>> Learn to match questions with answers by considering their semantic
encoding
Deep Learning for Answer Sentence Selection (cont.)
 QA Matching Model
● Each question 𝑞𝑖 : associated with a list of answer sentences{𝑎𝑖1 , 𝑎𝑖2 , … , 𝑎𝑖𝑚 }
together with their judgements {𝑦𝑖1 , 𝑦𝑖2 , … , 𝑦𝑖𝑚 }
● Minimize the cross entropy of all labelled data QA pairs
● Sentence Model:
● Bag-of-words model: summing over all words
● Bigram model
Deep Learning for Answer Sentence Selection (cont.)
 Experiments
● TREC Answer Selection Dataset: TREC QA 8-13(1999-2004)
● Results:
A Neural Network for Factoid Question Answering over Paragraphs. EMNLP’14.
 Quiz Bowl
● A Task: Mapping raw text to a large set of well-known entities
● Questions: 4 ∼ 6 sentences
 Approach
● Sentence Representation: A sentence -- A dependency tree
● Sentence level – paragraph level: Averaging the representations of each
sentence seen so far
● Mapping questions to their corresponding answer entities
>> A multi-class classication task: Using a softmax layer
● Jointly learning answer and question representations
A Neural Network for Factoid Question Answering over Paragraphs (cont.)
 Experiments
● Dataset:
● Total: 46, 842 questions in 14 different categories
● Results:
Outline
I.
Open QA with KB Triples
● W. Yih et al. Semantic parsing for single-relation question. ACL’14.
● A. Bordes et al. Open question answering with weakly supervised embedding models.
ECML’14.
● A. Bordes et al. Question Answering with Subgraph Embeddings. EMNLP’14
II.
Answer Selection via Unstructured Text Understanding
● L. Yu et al. Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop).
● M. Iyyer et al. A Neural Network for Factoid Question Answering over Paragraphs.
EMNLP’14.
III. More QA tasks:
● J. Weston et al. Towards AI-Complete Question Answering: A Set of Prerequisite
Toy Tasks. arXiv’15.
21
III. More QA tasks
 References:
● Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
arXiv’15 (Mar 14th,2015)
● Jason Weston, Antoine Bordes, Sumit Chopra, Tomas Mikolov
● Facebook
 Introduction:
● One long-term goal of machine learning research is to produce methods that are
applicable to understanding and reasoning natural language, in particular
building an intelligent dialogue agent.
● To measure progress towards that goal
>> a set of proxy tasks that evaluate reading comprehension via QA
● in a similar way to how software testing is built in computer science
● provided 20 toy question answering tasks
● built with a unified underlying simulation of a physical world, akin to a classic
text adventure game
● proposed a framework for the goal of text understanding and reasoning
Towards AI-Complete Question Answering:
A Set of Prerequisite Toy Tasks
1. Basic Factoid QA with Single Supporting Fact
4. Two Argument Relations: Subject vs. Object
2. Factoid QA with Two Supporting Facts
5. Three Argument Relations
3. Factoid QA with Three Supporting Facts
6.Yes/No questions
Towards AI-Complete Question Answering:
A Set of Prerequisite Toy Tasks
7. Counting
11. Basic Coreference
12. Conjunction
8. Lists / Sets
13. Compound Coreference
9. Simple Negation
14. Time Manipulation
10. Indefinite Knowledge
Towards AI-Complete Question Answering:
A Set of Prerequisite Toy Tasks
15. Basic Deduction
18. Reasoning about Size
16. Basic Induction
19. Path Finding
17. Positional Reasoning
20. Reasoning about Agent’s Motivations
Towards AI-Complete Question Answering (cont.)
 Memory Networks
● a memory 𝒎 (an array of objects indexed by 𝒎𝑖 )
● four potentially learnable components 𝐼, 𝐺, 𝑂 and 𝑅
● The 𝑂 module produces output features by finding 𝑘 supporting memories given 𝑥
Towards AI-Complete Question Answering (cont.)
 Adaptive Memories (and Responses)
● Finding supporting memories: Add a special fact 𝑚𝜙
● Multiple Answers: Add a special word 𝑚𝜙
 Nonlinear Sentence Modeling
i is the position of the word in a sentence of length l
Towards AI-Complete Question Answering (cont.)
Summary
I.
Open QA with KB Triples
● W. Yih et al. Semantic parsing for single-relation question. ACL’14.
Entity Identification for Single-Relation Question.
Convolutional NN.
● A. Bordes et al. Open question answering with weakly supervised embedding models. ECML’14.
● A. Bordes et al. Question Answering with Subgraph Embeddings. EMNLP’14.
Triple Ranking for Simple Factoid Question.
Bilinear Matching Model via Subgraph Embeddings
II.
Answer Selection via Unstructured Text Understanding
● L. Yu et al. Deep Learning for Answer Sentence Selection. NIPS’14 (DL workshop).
Answer Sentence Ranking for Question Sentence.
Matching Model via BoW and Bigram Embeddings
● M. Iyyer et al. A Neural Network for Factoid Question Answering over Paragraphs. EMNLP’14.
Entity Selection for Question Paragraphs.
Recursive NN based on Dependency Tree.
III. More QA tasks
● J. Weston et al. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
arXiv’15.
20 QA Toy Tasks for Text Understanding and Reasoning.
(Adaptive) Memory Networks.
29
Q&A
Thank You For Listening!