Considerations for Evaluating
Models of Language Understanding
and Reasoning
Gabriel Recchia
University of Cambridge
• Background: The bAbI dataset
• Introducing the GABITS dataset
(http://nowin2d.com/gabits/)
Some history
Some history
(slide from Bordes, Weston, Chopra, Mikolov, Joulin & Bottou, 2015)
Training Set
Generating
Process
Test Set
Facebook’s bAbI dataset
(slide from Bordes, Weston, Chopra, Mikolov, Joulin & Bottou, 2015)
The bAbI dataset
(slide from Bordes, Weston, Chopra, Mikolov, Joulin & Bottou, 2015)
Introducing GABITS
The Grounded and bAbI-Inspired Task Set
Introducing GABITS
The Grounded and bAbI-Inspired Task Set
Each training instance consists of
– A narrative
– A group of questions and associated answers
– An image illustrating the state of the world at
every point when something changes state
– A symbolic representation of the state of the
world at every point when something changes
state (optional)
Narrative
1 The lamp is in the kitchen.
2 The ball is in the dining room.
3 Eve is in the hall.
4 Carol is in the kitchen.
5 Frank is in the hall.
6 Carol got the lamp.
7 Eve went to the kitchen.
8 Eve travelled to the billiard room.
9 Frank travelled to the kitchen.
10 Eve went to the kitchen.
11 Carol travelled to the billiard room.
12 Carol discarded the lamp.
13 Carol grabbed the lamp.
Questions
14 (T1.a) Who is in the kitchen?
Eve,Frank
15 (T2.a) Where is Eve? kitchen
16 (T3.a) What is Carol holding? lamp
17 (T12.3) How many objects is Carol holding?
one
18 (T3.b) Who is holding the lamp? Carol
19 (T3.b) Who is holding the ball? no one
20 (T4.a) What has Carol held?
lamp
21 (T4.b) Who has held the lamp? Carol
22 (T6) Who moved the lamp to the billiard room? Carol
23 (T7.a) Where has Eve been?
billiard room,hall,kitchen
24 (T7.a) Where has Frank been? hall,kitchen
25 (T7.c) Where has the lamp been?
billiard room,kitchen
26 (T7.b) Where has Eve not been? dining room
29 (T8.a) Who has been in the hall?Eve,Frank
30 (T8.c) What has been in the kitchen?
lamp
32 (T8.c) What has been in the billiard room?
lamp
33 (T8.b) Who has not been in the billiard room?
Frank
Questions (cont.)
27 (T12.8) How many people have been in the kitchen?
three
28 (T13.8) Have fewer than four people been in the kitchen? yes
31 (T13.8) Have fewer than three objects been in the kitchen?yes
37 (T11) Who has been in the hall or the dining room (but not both)? Eve,Frank
38 (T9.a) Who has been in the dining room or the kitchen (or both)?
Carol,Eve,Frank
42 (T13.9) Have more than five people been in the billiard room or the hall or
both? no
Visual representation of world
1 The lamp is in the kitchen.
2 The ball is in the dining room.
3 Eve is in the hall.
4 Carol is in the kitchen.
5 Frank is in the hall.
6 Carol got the lamp.
7 Eve moved to the kitchen.
Symbolic representation of world
time: 65
(Carol took the lamp)
agent0.name Eve
agent0.x 149
agent0.y 324
(agent0.room hall)
agent1.name Carol
agent1.x 284
agent1.y 414
(agent1.room kitchen)
agent2.name Frank
agent2.x 170
agent2.y 414
(agent2.room hall)
item0.name lamp
item0.x 278
item0.y 408
(item0.room kitchen)
(item0.owner Carol)
item1.name ball
item1.x 118
item1.y 52
(item1.room dining room)
(item1.owner null)
Advantages
self-contained: all or nearly all of the information
necessary to perform well at the task is present within
the training data
– It should be obviously possible for a human to
solve the task even if they do not speak the
language in which the task is rendered
Advantages
incremental and compositional:
questions build on each other
Who is in the hall?
Who has been in the hall?
Who has not been in the hall?
Who has been in the hall and the lounge?
How many people have been in the hall?
How many people have been in the hall and the lounge?
Advantages
wide-coverage: the tasks in the dataset correspond to
diverse abilities
For even wider coverage - even more tasks!
In recent years, there has been an increasing number of
papers with (mostly) self-contained tasks involving twoor three-dimensional spatial representations
• Contact me for our list so far!
• Or to let me know about more tasks to add to the
collection!