360.28
Representation of real-world event schemas during narrative perception
Christopher Baldassano, Uri Hasson, Kenneth Norman
Schema-specific
Representations
Hand-labeled events
Shared PCC
event patterns
1
4
2
3
Restaurant
Ac
ro
Measure
event
match
Full details: Baldassano et al. bioRxiv 2016, 10.1101/081018
Events
2
3
Airport
Time (minutes)
Airport:
Enter
Airport
Airport
Restaurant
Video
Seated at
Table
Security
Voxel 1
Voxel 1
Voxel 2
Voxel 2
Voxel 3
Voxel 3
Time
Ordering
Food
Boarding
Gate
Stimulus A
Stimulus B
Across modality
only
How I Met Your Mother
Seinfeld
Up in the Air
2mm voxels
1.5 sec TR
Within schema
3 minute stories
48 minutes total
Across modality
q<0.05
Posterior cingulate, anterior temporal, and lateral frontal
regions have representations of schemas that
generalize across stories and modalities
0.9
2 3 4
Events
0.3
Across
Subjects later freely recalled all stories in the scanner
Examples of errors:
"Robert Downey Jr. gets on the plane eventually. First he notices
this woman, he’s kind of looking around at the airport and he notices
a woman that wants to get a window seat—WAIT wrong movie."
"Yeah I don’t really remember much about what happened here with
the Seinfeld. Yeah it was sort of like a conversation between the two
characters... um, yeah I think they were at a restaurant? But I’m
not actually sure."
100%
My Cousin Vinny
0
Next Step:
Impact of Schemas on Memory
Time
Shame
1
-0.7
Even without stimulus labels, PCC can identify shared
temporal structure among stories in a schema
Predicting event transitions
Avg PCC event patterns
on held-out story
Food
Arrives
31 subjects
4
0.5
Within
Event 4
Finding
Seat
3
0.1
Time
Transferring Event Labels
The Big Bang Theory
Friends
3
0.03
Audio
The Santa Clause
Events
2
2
Event pattern
correlation
(3) Given multiple timecourses, align matching events
0.05
Within schema
1
0.7
1
Timepoint classification accuracy
Enter
Restaurant:
Restaurant
Even runs
Voxel 3
Hand-labeled events
from 7 stories
Event 3
p=0.03
(2) Given timecourse, segment into stable events
Posterior
cingulate
Stimuli
Event 2
3
Time
0.07
0.01
Event 1
1
Voxel 2
Voxel 3
3
0.09
Event match (r)
• Unsupervised temporal clustering of PCC activity
separates stories from different schemas
2
Events
2
Odd runs
• Event transitions in a held-out story can be predicted
from PCC activity
Compare within-schema
event match to null
model (permutation)
1
Airport stories
(Even runs)
Voxel 1
Voxel 1
Voxel 2
0
ss
(1) Given event voxel patterns, find matching events
Average patterns (1 subject)
• High-level brain regions (including posterior cingulate
cortex) have representations of schemas that
generalize across subjects, stories, and modalities
Restaurant stories
(Even runs)
Airport stories
(Odd runs)
Ways to fit the model:
1
Using fMRI data from subjects watching and listening to
stories that share schemas, we find:
Within
Restaurant stories
(Odd runs)
4
Shared PCC
event patterns
Events
3
2
Unsupervised event alignments on odd and even runs
Average patterns (N-1 subjects)
Voxels
1
Model latent event structure of narrative-driven brain activity
Event match (r)
Situation models are built from schematic templates
learned over a lifetime
Bower, Black, Turner 1979
Data-driven Alignment
Stimulus A
Understanding and remembering everyday experiences
requires maintaining situation models of ongoing events
Event Segmentation Model
Stimulus B
Summary
"Uh Shame…I I uh…I don’t remember which one that was sorry."
p=0.005
Predictions
Correct recall requires both the story schema and
story-specific details to be encoded and retrieved
50%
Using neural measures of encoding and reinstatement, predict:
Transfer accuracy
Scrambled-event null
0%
Restaurant stories
Airport stories
Average
Schema event structure can be predicted in
unlabeled stimuli using PCC activity patterns
Story Schema
Story Details
Behavior
Present
Present
Coherent, correct recall
Present
Absent
Within-schema event confusion
Absent
Present
Incoherent recall
Absent
Absent
No meaningful recall