bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
1
Discoveringeventstructureincontinuousnarrativeperceptionandmemory
2
ChristopherBaldassano,JaniceChen,AsiehZadbood,JonathanWPillow,UriHasson,KennethANorman
3
PrincetonUniversity,PrincetonNeuroscienceInstituteandDepartmentofPsychology
4
Contact:ChristopherBaldassano,[email protected]
5
6
Summary:Duringrealistic,continuousperception,humansautomaticallysegmentexperiencesinto
7
discreteevents.Usinganovelmodelofneuraleventdynamics,weinvestigatehowcorticalstructures
8
generateeventrepresentationsduringcontinuousnarratives,andhowtheseeventsarestoredand
9
retrievedfromlong-termmemory.Ourdata-drivenapproachenablesidentificationofeventboundaries
10
andeventcorrespondencesacrossdatasetswithouthuman-generatedstimulusannotations,and
11
revealsthatdifferentregionssegmentnarrativesatdifferenttimescales.Wealsoprovidethefirstdirect
12
evidencethatnarrativeeventboundariesinhigh-orderareas(overlappingthedefaultmodenetwork)
13
triggerencodingprocessesinthehippocampus,andthatthisencodingactivitypredictspattern
14
reinstatementduringrecall.Finally,wedemonstratethattheseareasrepresentabstract,multimodal
15
situationmodels,andshowanticipatoryeventreinstatementassubjectslistentoafamiliarnarrative.
16
Ourresultsprovidestrongevidencethatbrainactivityisnaturallystructuredintosemantically
17
meaningfulevents,whicharestoredinandretrievedfromlong-termmemory.
18
1
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
19
Introduction
20
Typically,perceptionandmemoryarestudiedinthecontextofdiscretepicturesorwords.Real-life
21
experience,however,consistsofacontinuousstreamofperceptualstimuli.Thebrainthereforeneeds
22
tostructureexperienceintounitsthatcanbeunderstoodandremembered:“themeaningfulsegments
23
ofone’slife,thecoherentunitsofone’spersonalhistory”(Beal&Weiss,2013).Althoughthisquestion
24
wasfirstinvestigateddecadesago(Newtson,Engquist,&Bois,1977),ageneral“eventsegmentation
25
theory”wasonlyproposedrecently(Zacks,Speer,Swallow,Braver,&Reynolds,2007).Theseandother
26
authorshavearguedthathumansimplicitlygenerateeventboundarieswhenevertheworldchangesina
27
surprisingway,orwhenconsecutivestimulihavedistincttemporalassociations(Schapiro,Rogers,
28
Cordova,Turk-Browne,&Botvinick,2013).
29
Twocriticaldimensionsofeventrepresentationshavenotyetbeendeeplyexplored.First,eventscanbe
30
definedatmultipletimescales.Whenreadingastory,wecouldchunkitintodiscreteunitsofindividual
31
words,sentences,paragraphs,orchapters,andmayneedtochunkinformationonmultipletimescales
32
inparallel.Arecenttheoryofcorticalinformationprocessingarguesforadistributedtopographical
33
hierarchyoftimescales,fromshortprocessingtimescales(10sto100sofmilliseconds)inearlysensory
34
regionstolongprocessingtimescales(10sto100sofseconds)inhigher-orderareas(broadly
35
overlappingthedefaultmodenetwork)(Hasson,Chen,&Honey,2015).Inthisview,“events”inlow-
36
levelsensorycortex(e.g.hearingasingleword)(VanRullen,2016)areprogressivelyintegratedintothe
37
minutes-longeventstypicallyreportedbyhumanobservers.
38
Second,howarereal-lifeexperiencesencodedintolong-termmemory?Behavioralexperimentsand
39
mathematicalmodelshavearguedthatlong-termmemoryreflectseventstructureduringencoding
40
(Ezzyat&Davachi,2011;Gershman,Radulescu,Norman,&Niv,2014;Sargentetal.,2013;Zacks,
41
Tversky,&Iyer,2001),suggestingthattheeventsegmentsgeneratedduringperceptionmayserveas
2
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
42
the“episodes”ofepisodicmemory.Thewell-acceptedideathatthehippocampusstores“snapshots”of
43
corticalactivityhasbeendevelopedwithdiscretememoranda(Danker,Tompary,&Davachi,2016),for
44
whichitisobviouswhensnapshotsshouldbetakenandwhatinformationtheyshouldcontain.
45
However,duringacontinuousstreamofinformationinareal-lifecontext,itisnotatallclearatwhich
46
timescale(e.g.words,sentences,situations)snapshotsshouldbetaken,andwhetherthesesnapshots
47
shouldbecontinuouslyupdatedduringeventsorencodedonlyafteraneventhascompleted.
48
Weproposethatthefulllifecycleofanevent,fromconstructiontolong-termstorage,canbedescribed
49
inaunifiedtheory,illustratedinFig.1.Eachbrainregionalongtheprocessinghierarchysegments
50
informationatitspreferredtimescale,beginningwithshorteventsinprimaryvisualandauditorycortex
51
andbuildingtomultimodal,abstractrepresentationsofthefeaturesofthecurrentevent(“situation
52
models”,Zwaan&Radvansky,1998)inlong-timescaleareas,includingdefaultmoderegionssuchasthe
53
angulargyrusandposteriormedialcortex.Ateventboundariesinlong-timescaleareas,thesituation
54
modelistransmittedtothehippocampus,whichcanlaterreinstatethesituationmodelinlong
55
timescaleregionsduringrecall,andfacilitaterecognitionofsimilareventsinthefuture.Thistheory
56
makesthefollowingpredictions:1)Eventsshouldbeidentifiableatdifferenttimescalesthroughoutthe
57
processingtimescalehierarchy,withsegmentationintoshorteventsinearlysensoryareasand
58
integrationintolongereventsinhigh-orderareas.2)Eventboundariesannotatedbyhumanobservers
59
shouldbemostrelatedtoneuraleventboundariesinlongtimescaleregions.3)Theendofaneventin
60
longtimescalecorticalregionsshouldtriggerthehippocampustoencodeinformationaboutthejust-
61
concludedeventintoepisodicmemory.4)Storedeventmemoriescanbereinstatedinlongtimescale
62
corticalregionsduringrecall,withstrongerreinstatementformorestrongly-encodedevents.5)Neural
63
patternsinlongtimescaleregionscorrespondtoabstractsituationmodels,whichrepresentthefeatures
64
ofthesituationregardlessofthewaythatthesituationisdescribed(e.g.amovieoraverbalnarrative).
3
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
65
6)Priormemoryforanarrativeshouldinfluencetheprocessingoffutureevents,leadingtoanticipatory
66
reinstatementinlongtimescaleregions.
67
Testingthiskindofintegratedtheoryisbeyondthereachofexistingapproachesthatrelyonhuman
68
annotatorstosegmentevents.Itrequiresidentifyinghowdifferentbrainareassegmentevents(possibly
69
atdifferenttimescales),andaligningeventsacrossdifferentdatasetswithdifferenttimings(e.g.tosee
70
whetherthesame“situationmodel”isbeingelicitedbyamovievs.averbalnarrative,oramovievs.
71
laterrecall).Stimulus-basedannotationsalsocannotaddressquestionssuchasanticipatory
72
reinstatement,inwhichanidenticalstimulusgeneratesdifferenteventsegmentationsindifferent
73
observers(dependingontheirpriorexperience).Thus,tosearchfortheneuralcorrelatesofevent
74
segmentation,wehavedevelopedanewdata-drivenanalysismethodthatallowsustoidentifyevents
75
directlyfromneuralactivitypatterns,acrossmultipletimescalesanddatasets.
76
Ouranalysisapproach(summarizedhere,anddescribedindetailintheMaterialsandMethods)starts
77
withtwosimpleassumptions:1)whileprocessingaparticularnarrativestimulus,observersprogress
78
throughaparticularsequenceofdiscreteeventrepresentations(hiddenstates),and2)eacheventhasa
79
distinct(observable)signature(amulti-voxelfMRIpattern)thatispresentthroughouttheevent.We
80
implementtheseassumptionsusingavariantofaHiddenMarkovModel(HMM).Fittingthemodelto
81
fMRIdata(e.g.whilewatchingamovie)entailssimultaneouslyestimatingwhenthetransitionsbetween
82
eventsoccurandalsothemeanneuralpatternforeachevent.Theoptimalnumberofeventsisselected
83
bysweepingoverarangeofvaluesandmaximizingthefitonheld-outdata.Whenapplyingthemodel
84
tomultipledatasetsthatexpressthesamenarrative(e.g.whilewatchingamovieandduringlaterverbal
85
recall),themodelisconstrainedtofindthesamesequenceofpatterns(becausetheeventsarethe
86
same),butthetimingofthetransitionsbetweenthepatternscanvary(e.g.sincethespokendescription
87
oftheeventsmightnottakeaslongastheoriginalevents).Forexample,ifthenumberofeventsisset
88
to10,themodelwillattempttoexplainbothdatasetsintermsofonemulti-voxelpatterntransitioning
4
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
89
toasecondandthenathird,andsoforth,wherethesetenpatternsarecommontobothdatasets,but
90
exactlywhenthepatternsswitchcanvaryacrossdatasets.
91
Thismodelallowsustotestthesixpredictionsofourunifiedtheorydescribedabove,followingevents
92
fromtheirinitialperceptioninsensorycortextotheirincorporationintolong-termmemory.Ourresults
93
providethefirstdirectevidencethatbrainactivityduringrealisticexperiencesisnaturallystructured
94
intosegmentedeventsacrossmultipletimescales,thateventrepresentationsinhigh-orderareasatthe
95
topoftheprocessinghierarchycontainhigh-levelsemanticsituationdescriptions,andthatthesehigh-
96
leveleventsarediscretelyencodedbylong-termmemorystructures.
5
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
97
98
Figure1:Theoryofeventsegmentationandmemory.(1)Duringperception,eventsareconstructedat
99
ahierarchyoftimescales,withshorteventsinearlysensoryregions(includingprimaryvisualcortexand
100
primaryauditorycortex)andlongeventsinhigh-levelregions(includingdefaultmoderegionssuchas
101
angulargyrusandposteriormedialcortex).(2)Putativeeventboundariesidentifiedbyhumanobservers
102
shouldcorrespondmostcloselytolongtimescaleeventsnearthetopofthehierarchy.(3)Attheendof
103
ahigh-levelevent,thesituationmodelisstoredintolong-termmemory,resultinginpost-boundary
104
encodingactivityinthehippocampus.(4)Episodiceventmemoriescanbereinstatedintohigh-level
105
corticalregionsduringrecall.(5)Sincesituationmodelsareabstractsemanticdescriptions,thesame
106
sequenceofhigh-leveleventscanbeactivatedbymultipleinputmodalitiesiftheydescribethesame
107
story.(6)Prioreventmemoriescanalsoinfluenceongoingprocessing,facilitatingpredictionof
108
upcomingeventsinrelatednarratives.Wetesteachofthesehypothesesusingadata-drivenevent
109
segmentationmodel,whichcanautomaticallyidentifytransitionsinneuralactivitypatternsanddetect
110
correspondencesinactivitypatternsacrossdatasets.
6
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
111
Results
112
AllofouranalysesarecarriedoutusingournewHMM-basedeventsegmentationmodel(summarized
113
above,anddescribedindetailintheEventSegmentationModelsubsectionofMaterialsandMethods),
114
whichcanautomaticallydiscovertheneuralsignaturesofeacheventanditstemporalboundariesina
115
particulardataset.Wevalidatedthismodelusingbothsyntheticdata(Supp.Fig.1)andnarrativedata
116
withcleareventbreaksbetweenstories(Supp.Fig.2),confirmingthatwecouldaccuratelyrecoverthe
117
numberofeventboundariesandtheirlocations(seeMaterialsandMethods).Wethenappliedthe
118
modeltotestsixpredictionsofourtheoryofeventperceptionandmemory.
7
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
119
Timescalesofcorticaleventsegmentation
120
Thefirstpredictionofourtheoryisthat“eventsshouldbeidentifiableatdifferenttimescales
121
throughouttheprocessingtimescalehierarchy,withsegmentationintoshorteventsinearlysensory
122
areasandintegrationintolongereventsinhigh-orderareas.”Wemeasuredtheextenttowhich
123
continuousstimulievokedtheeventstructurehypothesizedbyourmodel(periodswithstableevent
124
patternspunctuatedbyshiftsbetweenevents),andwhetherthetimescalesoftheseeventsvariedalong
125
thecorticalhierarchy.WetestedthemodelbyfittingittofMRIdatacollectedwhilesubjectswatcheda
126
50-minutemovie(Chen,Leong,Norman,&Hasson,2016),andthenassessinghowwellthelearned
127
eventstructureexplainedtheactivitypatternsofaheld-outsubject(bycomparingwithin-eventvs.
128
across-eventpatternsimilarity,withlargerwithin-vs.across-eventsimilarityindicatingbettermodelfit).
129
Notethatpreviousanalysesofthisdatasethaveshownthattheevokedactivityissimilaracross
130
subjects,justifyinganacross-subjectsdesign(Chen,Leong,etal.,2016).Wefoundthatessentiallyall
131
brainregionsthatrespondedconsistentlytothemovie(acrosssubjects)showedevidenceforevent-like
132
structure,andthattheoptimalnumberofeventsvariedacrossthecortex(Fig.2).Sensoryregionslike
133
visualcortexshowedfastertransitionsbetweenstableactivitypatterns,whilehigher-levelregionslike
134
theprecuneushadactivitypatternsthatoftenremainedconstantforoveraminutebeforetransitioning
135
toanewstablepattern(seeFig.2insets).Thistopographyofeventtimescalesisbroadlyconsistentwith
136
thatfoundinpreviouswork(Hassonetal.,2015)measuringsensitivitytotemporalscramblingofa
137
moviestimulus(seeSupp.Fig.3).
8
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
138
139
140
Figure2:Eventsegmentationmodelformovie-watchingdatarevealseventtimescales.Theevent
141
segmentationmodelidentifiestemporally-clusteredstructureinmovie-watchingdatathroughoutall
142
regionsofcortexwithhighintersubjectcorrelation.Theoptimalnumberofeventsvariedbyanorderof
143
magnitudeacrossdifferentregions,withalargenumberofshorteventsinsensorycortexandasmall
144
numberoflongeventsinhigh-levelcortex.Forexample,thetimepointcorrelationmatrixforaregionin
145
theprecuneusexhibitedcoarseblocksofcorrelatedpatterns,leadingtomodelfitswithasmallnumber
146
ofevents(whitesquares),whilearegioninvisualcortexwasbestmodeledwithalargernumberof
147
shortevents(notethatonly~3minutesofthe50minutestimulusareshown).Thesearchlightismasked
148
toincludeonlyregionswithintersubjectcorrelation>0.25,andvoxelwisethresholdedforgreater
149
within-thanacross-eventsimilarity,q<0.001.
150
9
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
151
Comparisontohuman-labeledeventboundaries
152
Oursecondpredictionisthat“eventboundariesannotatedbyhumanobserversshouldbemostrelated
153
toneuraleventboundariesinlongtimescaleregions.”Weaskedfourindependentraterstodividethe
154
movieinto“scenes”basedonmajorshiftsinthenarrative(suchasinlocation,topic,ortime).The
155
numberofeventboundariesidentifiedbytheobserversvariedbetween36and64,buttheboundaries
156
hadasignificantamountofoverlap,withanaveragepairwiseDice’scoefficientof0.63and20event
157
boundariesthatwerelabeledbyallfourraters.Wethenmeasured,foreachbrainsearchlight,what
158
fractionofitsneurally-definedboundarieswerecloseto(withinthreetimepointsof)ahuman-labeled
159
eventboundary.AsshowninFig.3,thisrevealedagradientfromearlysensorycortextohigh-levellong
160
timescaleregions.Earlyauditoryandvisualcortexexhibitedmanyneuralboundaries,somenear
161
boundariesmarkedbyahumanobserverbutalsoatmanyothertimesduringthemovie,likelydueto
162
shiftsinlow-level(butnothigh-level)features.Inlongtimescaleregions,suchasangulargyrusand
163
especiallyposteriormedialcortex,amajorityoftheneurally-identifiedeventboundariescorresponded
164
withaboundarymarkedbyahumanobserver.
10
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
165
166
Figure3:Neuraleventboundariesmatchhuman-labeledeventboundaries,especiallyinposterior
167
medialcortex.Comparingtheeventboundariesidentifiedbythemodeltohuman-labeledevent
168
boundaries,wefindthatsimilarityincreasesaswemovefromsensoryregionstohigh-levelregions.The
169
plotontherightcompareshuman-labeledeventboundariesfromallfourhumanobserverstoneural
170
eventboundariesforthreeexamplesearchlights(forseveralminutesofthemovie).Earlysensory
171
regionssuchasV1producealargenumberofboundariesthatarenotstronglypredictiveofahuman-
172
labeledevent.Longtimescaleregions,includingangulargyrusandespeciallyinsuperiorparietaland
173
posteriormedialcortex,haveamajorityoftheireventboundariesnearhuman-labeledboundaries.The
174
searchlightismaskedtoincludeonlyregionswithintersubjectcorrelation>0.25.
175
11
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
176
Relationshipbetweencorticaleventboundariesandhippocampalencoding
177
Thethirdpredictionofourtheoryisthat“theendofaneventinlongtimescalecorticalregionsshould
178
triggerthehippocampustoencodeinformationaboutthejust-concludedeventintoepisodicmemory.”
179
Priorworkhasshownthattheendofavideoclipisassociatedwithincreasedhippocampalactivity,and
180
themagnitudeoftheactivitypredictslatermemory(Ben-Yakov&Dudai,2011;Ben-Yakov,Eshel,&
181
Dudai,2013).Theseexperiments,however,haveusedonlyisolatedshortvideoclipswithclear
182
transitionsbetweenevents.Doneurally-definedeventboundariesinacontinuousmovie,evokedby
183
subtlertransitionsbetweenrelatedscenes,generatethesamekindofhippocampalsignature?Usinga
184
searchlightprocedure,weidentifiedeventboundarieswiththeHMMsegmentationmodelforeach
185
corticalareaacrossthetimescalehierarchy.Wethencomputedtheaveragehippocampalactivity
186
aroundtheeventboundariesofeachcorticalarea,todeterminewhetheracorticalboundarytendedto
187
triggerahippocampalresponse.Wefoundthateventboundariesinadistributedsetoflong(butnot
188
short)timescaleregions(includingposteriorcingulatecortexandbilateralangulargyrus)allshoweda
189
strongrelationshiptohippocampalactivity,withthehippocampalresponsetypicallypeakingwithin
190
severaltimepointsaftertheeventboundary(Fig.4).
12
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
191
192
Figure4:Hippocampalactivityincreasesatcortically-definedeventboundaries.Todeterminewhether
193
eventboundariesmayberelatedtolong-termmemoryencoding,weidentifyeventboundariesbased
194
onacorticalregionandthenmeasurehippocampalactivityaroundthoseboundaries.Inasetofhigh-
195
levelregions(includingbilateralangulargyrus)wefindthateventboundariesintheseregionsrobustly
196
predictincreasesinhippocampalactivity,whichtendstopeakjustaftertheeventboundary.The
197
searchlightismaskedtoincludeonlyregionswithintersubjectcorrelation>0.25,andvoxelwise
198
thresholdedforpost-boundaryhippocampalactivitygreaterthanpre-boundaryactivity,q<0.001.
199
13
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
200
Reinstatementofeventpatternsduringfreerecall
201
Wethentestedourfourthprediction,that“storedeventmemoriescanbereinstatedinlongtimescale
202
corticalregionsduringrecall,withstrongerreinstatementformorestrongly-encodedevents.”After
203
watchingthemovie,allsubjectsinthisdatasetwereaskedtoretellthestorytheyhadjustwatched
204
(withoutanycuesorstimulus).Wefocusedouranalysesonthehigh-levelregionsthatshowedastrong
205
relationshipwithhippocampalactivityinthepreviousanalysis(posteriorcingulateandangulargyrus),as
206
wellasearlyauditorycortexforcomparison.
207
Usingtheeventsegmentationmodel,wefirstestimatedthe(group-average)seriesofevent-specific
208
neuralpatternsevokedbythemovie,andthenattemptedtosegmenteachsubject’srecalldatainto
209
correspondingevents.Whenfittingthemodeltotherecalldata,weassumedthatthesameevent-
210
specificneuralpatternsseenduringthemovie-viewingwillbereinstatedduringthespokenrecall.
211
Analyzingthespokenrecalltranscriptionsrevealedthatsubjectsgenerallyrecalledtheeventsinthe
212
sameorderastheyappearedinthemovie(seetableS1inChen,Leong,etal.,2016).Therefore,the
213
modelwasconstrainedtousethesameorderofmulti-voxeleventpatternsforrecallthatithadlearned
214
fromthemovie-watchingdata.However,crucially,themodelwasallowedtolearndifferentevent
215
timingsfortherecalldatacomparedtothemoviedata–thisallowedustoaccommodatethefactthat
216
eventdurationsdifferedforfreerecallvs.movie-watching.
217
Foreachsubject,themodelattemptedtofindasharedsequenceoflatenteventpatternsthatwas
218
sharedbetweenthemovieandrecall,asshownintheexamplewith25eventsinFig.5a.Comparedto
219
thenullhypothesisthattherewasnosharedeventorderbetweenthemovieandrecall,wefound
220
significantmodelfitsinboththeposteriorcingulate(p=0.015)andtheangulargyrus(p=0.002),butnot
221
inlow-levelauditorycortex(p=0.277)(Fig.5b).Thisresultdemonstratesthatwecanidentifyshared
14
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
222
temporalstructurebetweenperceptionandrecallwithoutanyhumanannotations.Asimilarpatternof
223
resultscanbefoundregardlessofthenumberoflatenteventsused(seeSupp.Fig4).
224
Wethenassessedwhetherthehippocampalresponseevokedbytheendofaneventduringthe
225
encodingofthemovietomemorywaspredictiveofthelengthoftimeforwhichtheeventwasstrongly
226
reactivatedduringrecall.AsshowninFig.5c-d,wefoundthatencodingactivityandeventreactivation
227
werepositivelycorrelatedinbothangulargyrus(r=0.362,p=0.002)andtheposteriorcingulate(r=0.312,
228
p=0.042),butnotearlyauditorycortex(r=0.080,p=0.333).Notethattherewasnorelationshipbetween
229
thehippocampalactivityatthestartingboundaryofaneventandthatevent’slaterrecallintheangular
230
gyrus(r=-0.119,p=0.867;differencefromendingboundarycorrelationp=0.004)andonlyaweak,
231
nonsignificantrelationshipinposteriorcingulate(r=0.189,p=0.113;differencefromendingboundary
232
correlationp=0.274).
233
15
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
234
235
Figure5:Movie-watchingeventsarereactivatedduringindividualfreerecall,andreactivationis
236
relatedtohippocampalactivationatencodingeventboundaries.(a)Wecanobtainanestimated
237
correspondencebetweenmovie-watchingdataandfree-recalldatainindividualsubjectsbyidentifying
238
asharedsequenceofeventpatterns,shownhereforanexamplesubjectusingdatafromposterior
239
cingulatecortex.(b)Foreachregionofinterest,wetestedwhetherthemovieandrecalldatasharedan
240
orderedsequenceoflatentevents(relativetoanullmodelinwhichtheorderofeventswasshuffled
241
betweenmovieandrecall).Wefoundthatbothangulargyrus(blue)andposteriorcingulatecortex
242
(green)showedsignificantreactivationofeventpatterns,whileearlyauditorycortex(red)didnot.(c-d)
243
Eventswhoseoffsetdroveastronghippocampalresponseduringencoding(movie-watching)were
244
stronglyreactivatedforlongerfractionsoftherecallperiod,bothintheangulargyrusandtheposterior
245
cingulate.Errorbarsforeventpointsdenotes.e.m.acrosssubjects,anderrorbarsonthebest-fitline
246
indicate95%confidenceintervalsfrombootstrappedbest-fitlines.
16
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
247
Sharedeventstructureacrossmodalities
248
Ourfifthhypothesisisthat“neuralpatternsinlongtimescaleregionscorrespondtoabstractsituation
249
models,whichrepresentthefeaturesofthesituationregardlessofthewaythatthesituationis
250
described(e.g.amovieoraverbalnarrative).”Wetestedthishypothesisusingaseparatedataset
251
(Zadbood,Chen,Leong,Norman,&Hasson,2016),inwhichsomesubjectswatchedamovie(thefirst24
252
minutesofSherlock)whileothersubjectslistenedtoan18-minuteaudionarrationdescribingtheevents
253
thatoccurredinthemovie.Foreachcorticalsearchlight,wefirstsegmentedthemoviedataintoevents,
254
andthentestedwhetherthissamesequenceofeventsfromthemovie-watchingsubjectswaspresentin
255
theaudio-narrationsubjects.High-levelcorticalregionswithlongprocessingtimescalesincludingthe
256
angulargyrusandposteriormedialcortexshowedastronglysignificantcorrespondencebetweenthe
257
twomodalities,indicatingthatasimilarsequenceofeventpatternswasevokedbythemovieandaudio
258
narration(Fig.6),irrespectiveofthemodalityusedtodescribetheevents.Incontrast,thoughlow-level
259
auditorycortexwasreliablyactivatedbybothofthesestimuli,therewasnoabove-chancesimilarity
260
betweentheseriesofactivitypatternsevokedbythetwostimuli(movievs.verbaldescription),
261
presumablybecausethelowlevelauditoryfeaturesofthetwostimuliweremarkedlydifferent.
262
17
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
263
264
Figure6:Movie-watchingmodelgeneralizestoaudionarrationinhigh-levelcortex.Afteridentifyinga
265
seriesofeventpatternsinagroupofsubjectswhowatchedamovie,wetestedwhetherthissameseries
266
ofeventsoccurredinaseparategroupofsubjectswhoheardanaudionarrationofthesamestory.The
267
movieandaudiostimuliwerenotsynchronizedanddifferedintheirduration.Werestrictedour
268
searchlighttovoxelsthatrespondedtoboththemovieandaudiostimuli(havinghighISCwithineach
269
group).Movie-watchingeventpatternsinearlyauditorycortex(dottedline)didnotgeneralizetothe
270
activityevokedbyaudionarration,whileregionsincludingtheangulargyrusandposteriormedialcortex
271
exhibitedsharedeventstructureacrossthetwostimulusmodalities.Thesearchlightismaskedto
272
includeonlyregionswithintersubjectcorrelation>0.1inallconditions,andvoxelwisethresholdedfor
273
above-chancemovie-audiofit,q<10-5.
274
18
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
275
Anticipatoryreinstatementforafamiliarnarrative
276
Finally,wetestedoursixthprediction,that“priormemoryforanarrativeshouldinfluencethe
277
processingoffutureevents,leadingtoanticipatoryreinstatementinlongtimescaleregions.”Our
278
analysessofarhaveexamineddataonperceptionoronmemory,butineverydaylifewedrawonthese
279
twofunctionssimultaneously.Ourongoinginterpretationofeventscanbeinfluencedbyprior
280
knowledge;specifically,ifsubjectslisteningtotheaudioversionofanarrativehadalreadyseenthe
281
movieversion,theymayanticipateupcomingeventscomparedtosubjectsexperiencingthenarrative
282
forthefirsttime.Detectingthiskindofanticipationhasnotbeenpossiblewithpreviousapproachesthat
283
relyonstimulusannotations,sincethedifferencebetweenthetwogroupsisnotinthestimulus(which
284
isidentical)butratherinthetemporaldynamicsoftheircognitiveprocesses.
285
Wecanfitoureventsegmentationmodeltothethreeconditions(watchingthemovie,listeningtothe
286
narrationwithmemory,andlisteningtothenarrationwithoutmemory)simultaneously,lookingforthe
287
samesequenceofeventpatternsinallthreecases(withvaryingeventboundaries).Byanalyzingwhich
288
timepoints(acrossthethreeconditions)wereassignedtothesameevent,wecangenerateatimepoint
289
correspondenceindicating–foreachtimepointduringtheaudionarrationdatasets–whichtimepoints
290
ofthemoviearemoststronglyevoked(onaverage)inthemindofthelisteners.
291
Wesearchedforcorticalregionsalongthehierarchyoftimescaleshowinganticipation,inwhichthis
292
correspondenceforthememorygroupwasconsistentlyaheadofthecorrespondencefortheno-
293
memorygroup(relativetochance).AsshowninFig.7,wefoundanticipatoryeventreinstatementin
294
severalhigh-levelregionswithlongprocessingtimescales,includingtheangulargyrusandposterior
295
medialcortex,withthelargestleadingeffectsinthemedialfrontalcortex.Examiningthemovie-audio
296
correspondencesintheseregions,thememorygroupwasconsistentlyaheadoftheno-memorygroup,
297
indicatingthatforagiventimepointoftheaudionarrationthememorygrouphadevent
298
representationsthatcorrespondedtolatertimepointsinthemovie.
19
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
299
300
Figure7:Priormemoryshiftsmovie-audiocorrespondence.Theeventsegmentationmodelwasfit
301
simultaneouslytoadatafromagroupwatchingthemovie,thesamegrouplisteningtotheaudio
302
narrationafterhavingseenthemovie(“memory”),andaseparategrouplisteningtotheaudionarration
303
forthefirsttime(“nomemory”).Byexaminingwhichtimepointswereestimatedtofallwithinthesame
304
latentevent,weobtainedacorrespondencebetweentimepointsintheaudiodata(forbothgroups)and
305
timepointsinthemoviedata.Wefoundthatthecorrespondenceinbothgroupswasclosetothe
306
human-labeledcorrespondencebetweenthemovieandaudiostimuli(blackboxes).Insomeregions,
307
however,thememorycorrespondence(orange)significantlyledthenon-memorycorrespondence
308
(blue),witheventsfromthemovieappearingslightlyearlierforthememorygroup(indicatedbyan
309
upwardshiftonthecorrespondenceplots)despitethestimuliforthetwogroupsbeingidentical.This
310
suggeststhatcorticalregionsofthememorygroupwereanticipatingeventsinthenarrationbasedon
311
knowledgeofthemovie,withtheanticipationeffectincreasingfromposteriortoanteriorregions.The
312
searchlightismaskedtoincludeonlyregionswithintersubjectcorrelation>0.1inallconditions,and
313
voxelwisethresholdedforabove-chancedifferencesbetweenmemoryandnomemorygroups,q<0.05.
20
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
314
Discussion
315
Usingadata-driveneventsegmentationmodelthatcanidentifytemporalstructuredirectlyfromneural
316
measurements,wefoundthatactivitypatternsincorticalregionsincludingposteriormedialcortexand
317
theangulargyrusprocessnarrativesasasequenceofhigh-levelsemanticevents.Althoughnarratives
318
evokerapidshiftsbetweenstableactivitypatternsinmanycorticalregionsalongthetimescale
319
hierarchy,onlythesehigh-levelregionshaveeventrepresentationsthatarecloselyrelatedtohuman
320
annotations,predicthippocampalencoding,arereactivatedduringrecall,generalizeacrossmodalities,
321
andshowanticipatorycodingforfamiliarnarratives.
322
Eventsegmentationtheory
323
Ourresultsarethefirsttodemonstrateanumberofkeypredictionsofeventsegmentationtheory
324
(Zacksetal.,2007)directlyfromneuraldataofnaturalisticnarratives,withoutusingspecially-
325
constructedstimuliorsubjectivelabelingofwhereeventsshouldstartandend.Previousworkhas
326
shownthathand-labeledeventboundariesareassociatedwithunivariateactivityincreasesinanetwork
327
ofregionsoverlappingourhigh-levelareas(Ezzyat&Davachi,2011;Speer,Zacks,&Reynolds,2007;
328
Swallowetal.,2011;Whitneyetal.,2009;Zacks,Braver,etal.,2001;Zacks,Speer,Swallow,&Maley,
329
2010),butbymodelingfine-scalespatialactivitypatternswewereabletodetecttheseeventchanges
330
withoutanexternalreference.Thisallowedustoidentifyregionswithtemporaleventstructuresat
331
manydifferenttimescales,onlysomeofwhichmatchedhuman-labeledboundaries.Otheranalysesof
332
thesedatasetsalsofoundreactivationduringrecall(Chen,Leong,etal.,2016)andsharedevent
333
structureacrossmodalities(Zadboodetal.,2016);however,becausetheseotheranalysesdefined
334
eventsbasedonthenarrativeratherthanbrainactivity,theywereunabletoidentifydifferencesin
335
eventsegmentationacrossbrainareasoracrossgroupswithdifferentpriorknowledge.
21
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
336
Timescalesofperception
337
Thetopographyofeventtimescalesrevealedbyouranalysisprovidesconvergingevidenceforan
338
emergingviewofhowinformationisprocessedduringreal-lifeexperience(Hassonetal.,2015).The
339
“processmemoryframework”arguesthatperceptualstimuliareintegratedacrosslongerandlonger
340
timescalesalongahierarchyfromearlysensoryregionstoregionsinthedefaultmodenetwork.Usinga
341
varietyofexperimentalapproaches,includingfMRI,electrocorticography(ECoG),andsingle-unit
342
recording,thistopographyhaspreviouslybeenmappedeitherbytemporallyscramblingthestimulusat
343
differenttimescalestoseewhichregions’responsesaredisrupted(Hasson,Yang,Vallines,Heeger,&
344
Rubin,2008;Honeyetal.,2012;Lerner,Honey,Silbert,&Hasson,2011)orbyexaminingthepower
345
spectrumofintrinsicdynamicswithineachregion(Honeyetal.,2012;Murrayetal.,2014;Stephens,
346
Honey,&Hasson,2013).Ourmodelandresultsaddtothesefindings,bysuggestingthatallprocessing
347
regionsexhibitfastchangesateventboundaries,butthatthesefastchangesaremuchlessfrequentin
348
long-timescaleregionswhichaccumulateandsynthesizeinformationatthesituationmodellevel,since
349
theyexperiencelargeupdatesonlywhenthehigh-levelsituationmodelchanges.
350
Interactionsbetweenlongtimescalecorticalregionsandthehippocampus
351
Severallong-timescaleregions,includingposteriorcingulatecortexandtheangulargyrus,showed
352
effectsacrossmanyofourindependentanalyses.Theseareasareinvolvedinhigh-levelscene
353
processingtasksinvolvingmemoryandnavigation(Baldassano,Esteva,Beck,&Fei-Fei,2016),arepart
354
ofthe“generalrecollectionnetwork”withstronganatomicalandfunctionalconnectivitytothe
355
hippocampus(Rugg&Vilberg,2013),andarethecorecomponentsoftheposteriormedialmemory
356
system(Ranganath&Ritchey,2012),whichisthoughttorepresentandupdatearepresentationofthe
357
currentsituation(Johnson-Laird,1983;VanDijk&Kintsch,1983;Zwaan,Langston,&Graesser,1995;
358
Zwaan&Radvansky,1998).Sinceeventrepresentationsintheseregionsgeneralizedacrossmodalities
22
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
359
andbetweenperceptionandrecall,ourresultsprovidefurtherevidencethattheyencodehigh-level
360
situationdescriptions.
361
Priorwork,however,hasnotaddressedwhathappenstotherepresentationsintheseregionswhenthe
362
situationchanges.Behavioralexperimentshaveshownthatlong-termmemoryreflectseventstructure
363
duringencoding(Ezzyat&Davachi,2011;Sargentetal.,2013;Zacks,Tversky,etal.,2001),suggesting
364
thatsituationrepresentationsare“saved”intomemoryasdiscreteevents.Wehavedemonstratedthat
365
thehippocampalencodingactivitypreviouslyshowntobepresentattheendofmovieclips(Ben-Yakov
366
&Dudai,2011;Ben-Yakovetal.,2013)andatabruptswitchesbetweenstimuluscategoryandtask
367
(DuBrow&Davachi,2016)alsooccursatthemuchmoresubtletransitionsbetweenevents(definedby
368
patternshiftsinhigh-levelregions),providingevidencethateventboundariestriggerthestorageofthe
369
currentsituationrepresentationintolong-termmemory.Wehavealsoshownthatthispost-event
370
hippocampalactivityisrelatedtopatternreinstatementduringrecall,ashasbeenrecently
371
demonstratedfortheencodingofdiscreteitems(Dankeretal.,2016),therebysupportingtheviewthat
372
eventsarethenaturalunitsofepisodicmemoryduringeverydaylife.
373
Oureventsegmentationmodel
374
Temporallatentvariablemodelshavebeenlargelyabsentfromthefieldofhumanneuroscience,since
375
thevastmajorityofexperimentshaveatemporalstructurethatisdefinedaheadoftimebythe
376
experimenter.OnenotableexceptionistherecentworkofAndersonandcolleagues,whichhasused
377
HMM-basedmodelstodiscovertemporalstructureinneuralresponsesduringmathematicalproblem
378
solving(Anderson&Fincham,2014;Anderson,Lee,&Fincham,2014;Anderson,Pyke,&Fincham,
379
2016).Thesemodelsareusedtosegmentproblem-solvingoperations(performedinlessthan30
380
seconds)intoasmallnumberofcognitivelydistinctstagessuchasencoding,planning,solvingand
381
responding.Ourworkisthefirsttoshowthat(usingamodifiedHMMandanannealedfitting
23
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
382
procedure)thislatent-stateapproachcanbeextendedtomuchlongerexperimentalparadigmswitha
383
muchlargernumberoflatentstates.
384
Forfindingcorrespondencesbetweencontinuousdatasets,asinouranalysesofsharedstructure
385
betweenperceptionandrecallorperceptionunderdifferentmodalities,severalothertypesof
386
approaches(notbasedonHMMs)havebeenproposedinpsychologyandmachinelearning.Dynamic
387
timewarping(Kang&Wheatley,2015;Silbert,Honey,Simony,Poeppel,&Hasson,2014)locally
388
stretchesorcompressestwotimeseriestofindthebestmatch,andmorecomplexmethodssuchas
389
conditionalrandomfields(Zhuetal.,2015)allowforpartsofthematchtobeoutoforder.However,
390
thesemethodsdonotexplicitlymodeleventboundaries,andfutureworkwillberequiredtoinvestigate
391
whattypesofneuralcorrespondencesarewellmodeledbycontinuouswarpingversusevent-structured
392
models.
393
Perceptionandmemoryinthewild
394
Ourresultsprovideabridgebetweenthelargeliteratureonlong-termencodingofindividualitems
395
(suchaswordsorpictures)andstudiesofmemoryforreal-lifeexperience(Nielson,Smith,Sreekumar,
396
Dennis,&Sederberg,2015;Rissman,Chow,Reggente,&Wagner,2016).Sinceourapproachdoesnot
397
requireanexperimentaldesignwithrigidtiming,itopensthepossibilityofhavingsubjectsbemore
398
activelyandrealisticallyengagedinatask,allowingforthestudyofeventsgeneratedduringvirtual
399
realitynavigation(suchasspatialboundaries,Horner,Bisby,Wang,Bogus,&Burgess,2016)orwhile
400
holdingdialogueswithasimultaneously-scannedsubject(Hasson,Ghazanfar,Galantucci,Garrod,&
401
Keysers,2012).ThemodelalsoisnotfMRI-specific,andcouldbeappliedtoothertypesofneural
402
timeseriessuchaselectrocorticography(ECoG),electroencephalography(EEG),orfunctionalnear-
403
infraredspectroscopy(fNIRS),includingportablesystemsthatcouldallowexperimentstoberun
404
outsidethelab(Mckendricketal.,2016).
24
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
405
Conclusion
406
Usinganoveleventsegmentationmodelthatcanbefitdirectlytoneuroimagingdata,weshowedthat
407
neuralresponsestonaturalisticstimuliaretemporallyorganizedintodiscreteeventsatvarying
408
timescales.Inanetworkofhigh-levelassociationregions,wefoundthattheseeventswererelatedto
409
subjectiveeventannotationsbyhumanobservers,predictedhippocampalencoding,generalizedacross
410
modalitiesandbetweenperceptionandrecall,andshowedanticipatorycodingoffamiliarnarratives.
411
Ourresultsprovideanewframeworkforunderstandinghowcontinuousexperienceisaccumulated,
412
stored,andrecalled.
25
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
413
ExperimentalProcedures
414
415
Figure8:Eventsegmentationmodel.Ourhypothesisabouttheeventstructureofnarrativestimuliis
416
that,foraparticularstory,aseriesofdistincteventsoccursinafixedorder(acrossstimulusmodalities
417
andbetweenencodingandrecall),andthateacheventkhasasignatureneuralpatternmk.Toencode
418
thishypothesisinaquantitativemodel,weusedamodifiedHiddenMarkovModel(HMM)inwhichthe
419
latentstateforeachtimepointdenotestheeventtowhichthattimepointbelongs.Themodelstartsin
420
thefirstevent,andtheneverysuccessivetimepointeithercontinuesthecurrenteventorstartsthenext
421
event,withthefinaltimepointconstrainedtofinishinthefinaleventK.Allneuraldatapointsduring
422
eventkareassumedtobehighlycorrelated(Pearson’sr)withmk.
423
26
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
424
EventSegmentationModel
425
Ourmodelisbuiltontwohypotheses:1)whileprocessingnarrativestimuli,observersexperiencea
426
sequenceofdiscreteevents,and2)eacheventhasadistinctneuralsignature.Mathematically,agiven
427
subject(oraveragedgroupofsubjects)startsinevents1=1andendsineventsT=K,whereTisthetotal
428
numberoftimepointsandKisthetotalnumberofevents.Oneachtimepointthemodeleitherremains
429
inthecurrentstateoradvancestothenextstate,i.e.st+1∈{st,st+1}foralltimepointst.Eacheventhasa
430
signaturemeanactivitypatternmkacrossallVvoxelsinaregionofinterest,andtheobservedbrain
431
activitybtatanytimepointtisassumedtobehighlycorrelatedwithmk,asillustratedinFig.8.
432
Giventhesequenceofobservedbrainactivitiesbt,ourgoalistoinferboththeeventsignaturesmkand
433
theeventstructurest.Toaccomplishthis,wecastourmodelasavariantofaHiddenMarkovModel
434
(HMM).Thelatentstatesaretheeventsstthatevolveaccordingtoasimpletransitionmatrix,inwhich
435
allelementsarezeroexceptforthediagonal(correspondingtost+1=st)andtheadjacentoff-diagonal
436
(correspondingtost+1=st+1),andtheobservationmodelisanisotropicGaussian$ %& '& = ( =
437
438
variance.Notethat,duetothisz-scoring,thelogprobabilityofobservingbrainstatebtinaneventwith
439
signaturemkissimplyproportionaltothePearsoncorrelationbetweenbtandmkplusaconstantoffset.
440
TheHMMisfittotheneuraldatabyusinganannealedversionoftheBaum-Welchalgorithm,which
441
iteratesbetweenestimatingtheneuralsignaturesmkandthelatenteventstructurest.Giventhe
442
signatureestimatesmk,theeventestimatesp(st=k)canbecomputedusingtheforward-backward
443
algorithm.Giventheeventestimatesp(st=k),thesignaturesmkcanbecomputedastheweighted
444
average;< =
445
observationvarianceσ2as4 ∙ 0.98F whereiisthenumberofloopsofBaum-Welchcompletedsofar.We
)
*+, -
.
/
0
-1-
2 34 /2 56 --
4=
>4 ?< 34
4=
>4 ?<
,where7(9)denotesz-scoringaninputvectorxtohavezeromeanandunit
.Toencourageconvergencetoahigh-likelihoodsolution,weannealthe
27
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
446
stopthefittingprocedurewhenthelog-likelihoodbeginstodecrease,indicatingthattheobservation
447
variancehasbeguntodropbelowtheactualeventactivityvariance.Wecanalsofitthemodel
448
simultaneouslytomultipledatasets;oneachroundofBaum-Welch,weruntheforward-backward
449
algorithmoneachdatasetseparately,andthenaverageacrossalldatasetstocomputeasinglesetof
450
sharedsignaturesmk.
451
Afterfittingthemodelononesetofdata,wecanthenlookforthesamesequenceofeventsinanother
452
dataset.Usingthesignaturesmklearnedfromthefirstdataset,wesimplyperformasingleroundofthe
453
forward-backwardalgorithmtoobtaineventestimatesp(st=k)ontheseconddataset.Ifweexpectthe
454
datasetstohavesimilarnoiseproperties(e.g.bothdatasetsaregroup-averageddatafromthesame
455
numberofsubjects),wesettheobservationvariancetothefinalσ2obtainedwhilefittingthefirst
456
dataset.Whentransferringeventslearnedongroup-averageddatatoindividualsubjects,weestimate
457
thevarianceforeacheventacrosstheindividualsubjectsofthefirstdataset.
458
Theendstaterequirementofourmodel–thatallstatesshouldbevisited,andtheendstateshouldbe
459
symmetricaltoallotherstates–requiresextendingthetraditionalHMMbymodifyingtheobservation
460
probabilities$ %& '& = ( .First,weenforce'G = Hbyrequiringthat,onthefinaltimestep,onlythe
461
finalstateKcouldhavegeneratedthedata,bysetting$ %G 'G = ( = 0forall( ≠ H.Equivalently,we
462
canviewthisasamodificationofthebackwardspass,byinitializingthebackwardsmessageJ('G = ()
463
to1for( = Hand0otherwise.Second,wemustmodifythetransitionmatrixtoensurethatallvalid
464
eventsegmentations(whichstartatevent1andendateventK,andproceedmonotonicallythroughall
465
events)havethesamepriorprobability.Formally,weintroduceadummyabsorbingstateK+1towhich
466
stateKcantransition,ensuringthatthetransitionprobabilitiesforstateKareidenticaltothosefor
467
previousstates,andthenset$ %& '& = H + 1 = 0toensurethatthisstateisneveractuallyused.
28
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
468
Sincewedonotwanttoassumethateventswillhavethesamerelativelengthsacrossdifferentdatasets
469
(suchasamovieandaudio-narrationversionofthesamenarrative),wefixallstatestohavethesame
470
probabilityofstayinginthesamestate(st+1=st)versusjumpingtothenextstate(st+1=st+1).Notethat
471
thesharedprobabilityofjumpingtothenextstatecantakeanyvaluebetween0and1withnoeffecton
472
theresults(uptoanormalizationconstantinthelog-likelihood),sinceeveryvalideventsegmentation
473
willcontainexactlythesamenumberofjumps(K-1).
474
Ourmodelinducesaprioroverthelocationsoftheeventboundaries.Thereareatotalof
475
likelyplacementsoftheK-1eventboundaries,andthenumberofwaystohaveeventboundarykfallon
476
timepointtisthenumberofwaysthatk-1boundariescanbeplacedint-1timepointstimesthenumber
477
ofwaysthat(K-1)-(k-1)-1boundariescanbeplacedinT-ttimepoints.Therefore$ '& = (&'&N) = ( +
478
1 =
479
thedistributionoverboundarylocationsstartsatthisprior,andslowlyadjuststomatchtheevent
480
structureofthedata.
481
Themodelimplementationwasfirstverifiedusingsimulateddata.Anevent-structureddatasetwas
482
constructedwithV=10voxels,K=10events,andT=500timepoints.Theeventstructurewaschosentobe
483
eitheruniform(with50timepointsperevent),orthelengthofeacheventwassampled(fromfirstto
484
last)fromN(1,0.25)*(timepointsremaining)/(eventsremaining).Ameanpatternwasdrawnforeach
485
eventfromastandardnormaldistribution,andthesimulateddataforeachtimepointwasthesumof
486
theeventpatternforthattimepointplusrandomlydistributednoisewithzeromeanandvarying
487
standarddeviation.Thenoisydataweretheninputtotheeventsegmentationmodel,andwemeasured
488
thefractionoftheeventboundariesthatwereexactlyrecoveredfromthetrueunderlyingevent
489
structure.AsshowninSupp.Fig.1,wereabletorecoveramajorityoftheeventboundariesevenwhen
490
thenoiselevelwasaslargeasthesignaturepatternsthemselves.
4O0
6O0
PO4
QO6O0
P
QO0
equally
.AnexampleofthisdistributionisshowninSupp.Fig.5.Duringtheannealingprocess,
29
G
M/)
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
491
Implementationsofourmodel,alongwithsimulateddataexamples,areavailableongithubat
492
https://github.com/intelpni/brainiak(python)andathttps://github.com/cbaldassano/Event-
493
Segmentation(Matlab).
30
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
494
ExperimentalData
495
InterleavedStoriesdataset
496
Totestourmodelinadatasetwithclear,unambiguouseventboundaries,weuseddatafromsubjects
497
wholistenedtotwounrelatedaudionarratives(Chen,Chow,Norman,&Hasson,2015).
498
22subjects(allnativeEnglishspeakers)wererecruitedfromthePrincetoncommunity(9male,13
499
female,ages18-26).Allsubjectsprovidedinformedwrittenconsentpriortothestartofthestudyin
500
accordancewithexperimentalproceduresapprovedbythePrincetonUniversityInstitutionalReview
501
Board.Thestudywasapproximately2hourslongandsubjectsreceived$20perhourascompensation
502
fortheirtime.Datafrom3subjectswerediscardedduetofallingasleepduringthescan,and1dueto
503
problemswithaudiodelivery.
504
Inthisworkweuseddatafrom18subjectswholistenedtothetwoaudionarrativesinaninterleaved
505
fashion,withtheaudiostimulusswitchingbetweenthetwonarrativesapproximatelyevery60seconds
506
atnaturalparagraphbreaks.Thetotalstimuluslengthwasapproximately29minutes,duringwhich
507
therewere32storyswitches.Theaudiowasdeliveredviain-earheadphones.
508
Imagingdatawereacquiredona3Tfull-bodyscanner(SiemensSkyra)witha20-channelheadcoilusing
509
aT2*-weightedechoplanarimaging(EPI)pulsesequence(TR1500ms,TE28ms,flipangle64,whole-
510
braincoverage27slicesof4mmthickness,in-planeresolution3x3mm,FOV192x192mm).
511
PreprocessingwasperformedinFSL,includingslicetimecorrection,motioncorrection,linear
512
detrending,high-passfiltering(140scutoff),andcoregistrationandaffinetransformationofthe
513
functionalvolumestoatemplatebrain(MNI).Functionalimageswereresampledto3mmisotropic
514
voxelsforallanalyses.
31
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
515
Theanalysesinthispaperwerecarriedoutusingdatafromaposteriorcingulateregionofinterest,the
516
posteriormedialclusterinthe“dorsaldefaultmodenetwork”definedbywhole-brainrestingstate
517
connectivityclustering(Shirer,Ryali,Rykhlevskaia,Menon,&Greicius,2012).
518
SherlockRecalldataset
519
Ourprimarydatasetconsistedof17subjectswhowatchedthefirst50minutesofthefirstepisodeof
520
BBC’sSherlock,andwerethenaskedtofreelyrecalltheepisodeinthescannerwithoutcues(Chen,
521
Leong,etal.,2016).Subjectsvariedinthelengthandrichnessoftheirrecall,withtotalrecalltimes
522
rangingfrom11minutesto46minutes(andameanof22minutes).Imagingdatawasacquiredusinga
523
T2*-weightedechoplanarimaging(EPI)pulsesequence(TR1500ms,TE28ms,flipangle64,whole-
524
braincoverage27slicesof4mmthickness,in-planeresolution3x3mm,FOV192x192mm).
525
Werestrictedoursearchlightanalysestovoxelsthatwerereliablydrivenbythestimuli,measuredusing
526
intersubjectcorrelation(Hasson,Nir,Levy,Fuhrmann,&Malach,2004).Voxelswithacorrelationless
527
thanr=0.25duringmovie-watchingwereremovedbeforerunningthesearchlightanalysis.
528
Wedefinedthreeregionsofinterestbasedonpriorwork.Inadditiontotheposteriorcingulateregion
529
definedabove,wedefinedtheangulargyrusasareaPG(bothPGaandPGp)usingthemaximum
530
probabilitymapsfromacytoarchitectonicatlas(Eickhoffetal.,2005),andwedefinedearlyauditory
531
cortexasvoxelswithintheHeschl’sgyrusregion(Harvard-Oxfordcorticalatlas)withreliableintersubject
532
correlationduringanaudionarrative(“Pieman”,Simonyetal.,2016).
533
SherlockNarrativedataset
534
Toinvestigatecross-modaleventrepresentationsandtheimpactofpriormemory,weusedaseparate
535
datasetinwhichsubjectsexperiencedmultipleversionsofanarrative.Onegroupof17subjects
536
watchedthefirst24minutesofthefirstepisodeofSherlock(aportionofthesameepisodeusedinthe
32
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
537
SherlockRecalldataset),whileanothergroupof17subjects(whohadneverseentheepisodebefore)
538
listenedtoan18minuteaudiodescriptionoftheeventsduringthispartoftheepisode(takenfromthe
539
audiorecordingofonesubject’srecallintheSherlockRecalldataset).Thesubjectswhowatchedthe
540
episodethenlistenedtothesame18minuteaudiodescription.Thisyieldedthreesetsofdata,allbased
541
onthesamestory:watchingamovieoftheevents,listeningtoanaudionarrationoftheeventswithout
542
priormemory,andlisteningtoanaudionarrationoftheeventswithpriormemory.Imagingdatawas
543
acquiredusingthesamesequenceasinSherlockRecalldataset;seeZadboodetal.(2016)forfull
544
details.
545
AsintheSherlockRecallexperiment,weremovedallvoxelsthatwerenotreliablydrivenbythestimuli.
546
Onlyvoxelswithanintersubjectcorrelationofatleastr=0.1acrossallthreeconditionswereincludedin
547
searchlightanalyses.
548
Eventannotationsbyhumanobservers
549
Fourhumanobserversweregiventhevideofileforthe50-minuteSherlockstimulus,andgiventhe
550
followingdirections:“Writedownthetimesatwhichyoufeellikeanewsceneisstarting;theseare
551
pointsinthemoviewhenthereisamajorchangeintopic,location,time,etc.Each“scene”shouldbe
552
between10secondsand3minuteslong.Also,giveeachsceneashorttitle.”Thesimilarityamong
553
observerswasmeasuredusingDice’scoefficient(numberofmatchingboundariesdividedbymean
554
numberofboundaries,consideringboundarieswithinthreetimepointsofoneanothertomatch).
33
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
555
Findingeventstructureinnarratives
556
TovalidateoureventsegmentationmodelonrealfMRIdata,wefirstfitthemodeltogroup-averaged
557
PCCdatafromtheInterleavedStoriesexperiment.Inthisexperiment,weexpectthataneventboundary
558
shouldbegeneratedeverytimethestimulusswitchesstories,givingagroundtruthagainstwhichto
559
comparethemodel’ssegmentations.AsshowninSupp.Fig.2,ourmethodwashighlyeffectiveat
560
identifyingevents,withthemajorityoftheidentifiedboundariesfallingclosetoastoryswitch.
561
TheremainingsubsectionsoftheMaterialsandMethodsdescribehowthemodelwasusedtoobtain
562
eachoftheexperimentalresults,withsubsectiontitlescorrespondingtosubsectionsoftheResults.
563
Timescalesofcorticaleventsegmentation
564
Weappliedthemodelinasearchlighttothewhole-brainmovie-watchingdatafromtheSherlockRecall
565
study.Cubicalsearchlightswerescannedthroughoutthevolumeatastepsizeof3voxelsandwitha
566
sidelengthof7voxels.Foreachsearchlight,theeventsegmentationmodelwasappliedtogroup-
567
averageddatafromallbutonesubject.Wemeasuredtherobustnessoftheidentifiedboundariesby
568
testingwhethertheseboundariesexplainedthedataintheheld-outsubject.Wemeasuredthespatial
569
correlationbetweenallpairsoftimepointsthatwerefourtimepointsapart,andthenbinnedthese
570
correlationsaccordingtowhetherthepairoftimepointsfellwithinthesameeventorcrossedoveran
571
eventboundary.Theaveragedifferencebetweenthewithin-versusacross-eventcorrelationswasused
572
asanindexofhowwellthelearnedboundariescapturedthetemporalstructureoftheheld-outsubject.
573
Theanalysiswasrepeatedforeverypossibleheld-outsubject,andwithavaryingnumberofeventsfrom
574
K=10toK=120.Afteraveragingtheresultsacrosssubjects,thenumberofeventswiththebestwithin-
575
versusacross-eventcorrelationswaschosenastheoptimalnumberofeventsforthissearchlight.To
576
generateanulldistribution,thesameanalysiswasperformedexceptthattheeventboundarieswere
34
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
577
scrambledbeforecomputingthewithin-versusacross-eventcorrelation.Thisscramblingwasperformed
578
byreorderingtheeventswiththeirdurationsheldconstant,toensurethatthenulleventshadthesame
579
distributionofeventlengthsastherealevents.Thewithinversusacrossdifferencefortherealevents
580
comparedto1000nulleventswasusedtocomputeazvalue,whichwasconvertedtoapvalueusing
581
thenormaldistribution.ThepvalueswereBonferronicorrectedforthe12choicesofthenumberof
582
events,andthenthefalsediscoveryrateqwascomputedusingthesamecalculationasinAFNI(Cox,
583
1996).
584
Sincethetopographyoftheresultswassimilartopreviousworkontemporalreceptivewindows,we
585
comparedthemapoftheoptimalnumberofeventswiththeshortandmedium/longtimescalemaps
586
derivedbymeasuringinter-subjectcorrelationforintactversusscrambledmovies(Chen,Honey,etal.,
587
2016).Thehistogramoftheoptimalnumberofeventsforvoxelswascomputedwithineachofthe
588
timescalemaps.
589
Comparisontohuman-labeledeventboundaries
590
Tocomparetheneurally-definedeventboundariesthroughoutthecortextothehuman-labeledevent
591
boundaries,wecomputedthefractionoftheneuraleventboundariesthatwereclosetoahuman-
592
labeledboundaryforeachsearchlight.Wedefined“closeto”as“withinthreetimepoints,”sincethe
593
typicaluncertaintyinthemodelaboutexactlywhereaneuraleventswitchoccurredwasapproximately
594
threetimepoints.
35
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
595
Relationshipbetweencorticaleventboundariesandhippocampalencoding
596
Afterapplyingtheeventsegmentationmodelthroughoutthecortexasdescribedabove,wemeasured
597
whetherthedata-driveneventboundarieswererelatedtoactivityinthehippocampus.Foragiven
598
corticalsearchlight,weextractedawindowofmeanhippocampalactivityaroundeachofthe
599
searchlight’seventboundaries.Wethenaveragedthesewindowstogether,yieldingaprofileof
600
boundary-triggeredhippocampalresponseaccordingtothisregion’sboundaries.Toassesswhetherthe
601
hippocampusshowedasignificantincreaseinactivityrelatedtotheseeventboundaries,wemeasured
602
themeanhippocampalactivityforthe10timepointsfollowingtheeventboundaryminusthemean
603
activityforthe10timepointsprecedingtheeventboundary,andcomparedthisdifferencetothesame
604
calculationfortheshuffledeventboundaries(asdescribedabove).Thezvalueforthisdifferencewas
605
computedtoapvalue,andthentransformedtoafalsediscoveryrateq.
36
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
606
Reinstatementofeventpatternsduringfreerecall
607
Foreachregionofinterest,wefittheeventsegmentationmodelasdescribedabove(onthegroup-
608
averageddata).Wethentookthelearnedsequenceofeventsignaturesmkandrantheforward-
609
backwardalgorithmoneachindividualsubject’srecalldata.Wesetthevarianceofeachevent’s
610
observationmodelbycomputingthevariancewithineacheventinthemovie-watchingdataof
611
individualsubjects,poolingacrossbothtimepointsandsubjects.Wecomparedthelog-likelihoodofthe
612
fittotherecalldataagainstanullmodelinwhichtheeventsignatureswererandomlyre-ordered,and
613
computedthezvalueofthetruelog-likelihoodcomparedto100nullshuffles,thenconvertedtoap
614
value.Thisnullhypothesistestthereforeassessedwhethertherecallexhibitedorderedreactivationof
615
theeventsidentifiedduringmovie-watching.Theanalysiswasrunfor10eventsto60eventsinstepsof
616
5.
617
Weoperationalizedtheoverallreinstatementofaneventk,as
618
recalltimepointsoftheprobabilitythatthesubjectwasrecallingperceptualeventkatthattimepoint.
619
Wemeasuredwhetherthisper-eventre-activationduringrecallcouldbepredictedduringmovie-
620
watching,basedonthehippocampalresponseattheendoftheevent.Foreachsubject,wecomputed
621
thedifferencebetweenhippocampalactivityafterversusbeforetheeventboundaryasabove.Wethen
622
averagedtheeventre-activationandhippocampaloffsetresponseacrosssubjects,andmeasuredtheir
623
correlation.Toassesstherobustnessofthesecorrelations,weperformedabootstraptest,inwhichwe
624
resampledsubjects(withreplacement,yielding17subjectsasintheoriginaldataset)beforetakingthe
625
averageandcomputingthecorrelation.Thepvaluewasdefinedasthefractionof1000resamplesthat
626
yieldedcorrelationsgreaterthanzero.Forcomparisonpurposes,wealsoperformedthesameanalysis
627
butwithhippocampaldifferencesatthebeginningofeachevent,ratherthantheend.
37
& p(sT
= k);thatis,thesumacrossall
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
628
Sharedeventstructureacrossmodalities
629
Todeterminewhetheraudionarrationofastoryelicitedthesamesequenceofeventsasamovieofthat
630
story,weusedanapproachsimilartothatusedfordetectingreactivationatrecall.Afterfittingthe
631
eventsegmentationmodeltoasearchlightofmovie-watchingdatafromtheSherlockNarration
632
experiment,wetookthelearnedeventsignaturesmkandusedthemtoruntheforward-backward
633
algorithmontheaudionarrationdata.Sinceboththemovieandaudiodatawereaveragedatthegroup
634
level,theyshouldhavesimilarlevelsofnoise,andthereforewesimplyusedthefitmovievarianceσ2for
635
theobservationvariance.Asabove,wecomparedtoanullmodelinwhichtheorderoftheevent
636
signatureswasshuffledbeforefittingtothenarrationdata,whichyieldedazvaluethatwasconverted
637
toapvalueandthencorrectedtoafalsediscoveryrateq.
638
Anticipatoryreinstatementforafamiliarnarrative
639
Todeterminewhethermemorychangedtheeventcorrespondencebetweenthemovieandnarration,
640
wethenfitthesegmentationmodelsimultaneouslytogroup-averageddatafromthemovie-watching
641
condition,audionarrationno-memorycondition,andaudionarrationwithmemorycondition,yieldinga
642
sequenceofeventsineachconditionwiththesameneuralsignatures.Wecomputedthe
643
correspondencebetweenthemoviestatessm,tandtheaudiono-memorystatessanm,tasas$ '5,&0 =
644
'WX5,&- =
645
determineifthiscorrespondencewassignificantlydifferentbetweenthememoryandno-memory
646
conditions,wecreatednullgroupsbyaveragingtogetherarandomhalfoftheno-memorysubjectswith
647
arandomhalfofmemorysubjects,andthenaveragingtogethertheremainingsubjectsfromeach
648
group,yieldingtwogroup-averagedtimecourseswhosecorrespondencesshoulddifferonlybychance.
649
Forboththerealandnullcorrespondences,wecomputedthedifferencesbetweenthegroup
< $('5,&0
= () ∙ $('WX5,&- = (),andsimilarlyfortheaudiomemorystatessam,t.To
38
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
'5,&0 = 'WX5,&- − $ '5,&0 = 'W5,&- )* ,andcalculatedazvaluebased
650
correspondencesas
651
ontheresultsforrealversusnullgroups.Thiszvaluewasconvertedtoapvalueandthencorrectedtoa
652
falsediscoveryrateq.Forvisualization,wealsocomputedhowfarthememorycorrespondencewas
653
aheadoftheno-memorycorrespondenceasthemeanovert2ofthedifferenceintheexpectedvalues
654
&0 Z) $
&)
&*($
'5,&0 = 'WX5,&- −
&0 Z) $
'5,&0 = 'W5,&- .
39
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
655
Acknowledgements
656
WethankM.ChowforassistanceincollectingtheInterleavedStoriesdataset,M.C.Iordanforhelpin
657
portingthemodelimplementationtopython,andthemembersoftheHassonandNormanlabsfortheir
658
commentsandsupport.ThisworkwassupportedbyagrantfromIntelLabs(CAB),TheNational
659
InstitutesofHealth(R01-MH094480,UH,and2T32MH065214-11,KAN),theMcKnightFoundation
660
(JWP),NSFCAREERAwardIIS-1150186(JWP),andagrantfromtheSimonsCollaborationontheGlobal
661
Brain(JWP).
40
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
662
References
663
Anderson,J.R.,&Fincham,J.M.(2014).Discoveringthesequentialstructureofthought.Cognitive
664
665
Science,38(2),322–352.https://doi.org/10.1111/cogs.12068
Anderson,J.R.,Lee,H.S.,&Fincham,J.M.(2014).Discoveringthestructureofmathematicalproblem
666
667
solving.NeuroImage,97,163–177.https://doi.org/10.1016/j.neuroimage.2014.04.031
Anderson,J.R.,Pyke,A.A.,&Fincham,J.M.(2016).HiddenStagesofCognitionRevealedinPatternsof
668
669
BrainActivation.PsychologicalScience.https://doi.org/10.1177/0956797616654912
Baldassano,C.,Esteva,A.,Beck,D.M.,&Fei-Fei,L.(2016).Twodistinctsceneprocessingnetworks
670
671
connectingvisionandmemory(Vol.3).https://doi.org/10.1101/057406
Beal,D.J.,&Weiss,H.M.(2013).TheEpisodicStructureofLifeatWork.InA.B.Bakker&K.Daniels
672
(Eds.),ADayintheLifeofaHappyWorker(pp.8–24).NewYork,NY:PsychologyPress.
673
Ben-Yakov,A.,&Dudai,Y.(2011).Constructingrealisticengrams:poststimulusactivityofhippocampus
674
anddorsalstriatumpredictssubsequentepisodicmemory.TheJournalofNeuroscience :The
675
OfficialJournaloftheSocietyforNeuroscience,31(24),9032–42.
676
https://doi.org/10.1523/JNEUROSCI.0702-11.2011
677
Ben-Yakov,A.,Eshel,N.,&Dudai,Y.(2013).Hippocampalimmediatepoststimulusactivityinthe
678
encodingofconsecutivenaturalisticepisodes.JournalofExperimentalPsychology.General,142(4),
679
1255–63.https://doi.org/10.1037/a0033558
680
Chen,J.,Chow,M.,Norman,K.A.,&Hasson,U.(2015).Differentiationofneuralrepresentationsduring
681
682
processingofmultipleinformationstreams.InSocietyforNeuroscience.
Chen,J.,Honey,C.J.,Simony,E.,Arcaro,M.J.,Norman,K.A.,&Hasson,U.(2016).AccessingReal-Life
41
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
683
EpisodicInformationfromMinutesversusHoursEarlierModulatesHippocampalandHigh-Order
684
CorticalDynamics.CerebralCortex,26(8),3428–3441.https://doi.org/10.1093/cercor/bhv155
685
Chen,J.,Leong,Y.C.,Norman,K.A.,&Hasson,U.(2016).Sharedexperience,sharedmemory:a
686
commonstructureforbrainactivityduringnaturalisticrecall.https://doi.org/10.1101/035931
687
Cox,R.W.(1996).AFNI:softwareforanalysisandvisualizationoffunctionalmagneticresonance
688
neuroimages.ComputersandBiomedicalResearch,anInternationalJournal,29(3),162–73.
689
Retrievedfromhttp://www.ncbi.nlm.nih.gov/pubmed/8812068
690
Danker,J.F.,Tompary,A.,&Davachi,L.(2016).Trial-by-TrialHippocampalEncodingActivationPredicts
691
theFidelityofCorticalReinstatementDuringSubsequentRetrieval.CerebralCortex,bhw146.
692
https://doi.org/10.1093/cercor/bhw146
693
DuBrow,S.,&Davachi,L.(2016).Temporalbindingwithinandacrossevents.NeurobiologyofLearning
694
andMemory,134,107–114.https://doi.org/10.1016/j.nlm.2016.07.011
695
Eickhoff,S.B.,Stephan,K.E.,Mohlberg,H.,Grefkes,C.,Fink,G.R.,Amunts,K.,&Zilles,K.(2005).Anew
696
SPMtoolboxforcombiningprobabilisticcytoarchitectonicmapsandfunctionalimagingdata.
697
NeuroImage,25(4),1325–35.https://doi.org/10.1016/j.neuroimage.2004.12.034
698
Ezzyat,Y.,&Davachi,L.(2011).Whatconstitutesanepisodeinepisodicmemory?PsychologicalScience,
699
700
22(2),243–52.https://doi.org/10.1177/0956797610393742
Gershman,S.J.,Radulescu,A.,Norman,K.A.,&Niv,Y.(2014).StatisticalComputationsUnderlyingthe
701
DynamicsofMemoryUpdating.PLoSComputationalBiology,10(11),e1003939.
702
https://doi.org/10.1371/journal.pcbi.1003939
703
Hasson,U.,Chen,J.,&Honey,C.J.(2015).Hierarchicalprocessmemory:Memoryasanintegral
704
componentofinformationprocessing.TrendsinCognitiveSciences,19(6),304–313.
42
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
705
https://doi.org/10.1016/j.tics.2015.04.006
706
Hasson,U.,Ghazanfar,A.A.,Galantucci,B.,Garrod,S.,&Keysers,C.(2012).Brain-to-braincoupling:A
707
mechanismforcreatingandsharingasocialworld.TrendsinCognitiveSciences,16(2),114–121.
708
https://doi.org/10.1016/j.tics.2011.12.007
709
Hasson,U.,Nir,Y.,Levy,I.,Fuhrmann,G.,&Malach,R.(2004).Intersubjectsynchronizationofcortical
710
activityduringnaturalvision.Science(NewYork,N.Y.),303(5664),1634–40.
711
https://doi.org/10.1126/science.1089506
712
Hasson,U.,Yang,E.,Vallines,I.,Heeger,D.J.,&Rubin,N.(2008).AHierarchyofTemporalReceptive
713
WindowsinHumanCortex.JournalofNeuroscience,28(10),2539–2550.
714
https://doi.org/10.1523/JNEUROSCI.5487-07.2008
715
Honey,C.J.,Thesen,T.,Donner,T.H.,Silbert,L.J.,Carlson,C.E.,Devinsky,O.,…Hasson,U.(2012).Slow
716
CorticalDynamicsandtheAccumulationofInformationoverLongTimescales.Neuron,76(2),423–
717
434.https://doi.org/10.1016/j.neuron.2012.08.011
718
Horner,A.J.,Bisby,J.A.,Wang,A.,Bogus,K.,&Burgess,N.(2016).Theroleofspatialboundariesin
719
shapinglong-termeventrepresentations.Cognition,30,1–30.
720
https://doi.org/10.1016/j.cognition.2016.05.013
721
Johnson-Laird,P.N.(1983).Mentalmodels:Towardsacognitivescienceoflanguage,inference,and
722
723
consciousness.HarvardUniversityPress.
Kang,O.,&Wheatley,T.(2015).Pupildilationpatternsreflectthecontentsofconsciousness.
724
725
ConsciousnessandCognition,35(August),128–135.https://doi.org/10.1016/j.concog.2015.05.001
Lerner,Y.,Honey,C.J.,Silbert,L.J.,&Hasson,U.(2011).Topographicmappingofahierarchyof
726
temporalreceptivewindowsusinganarratedstory.TheJournalofNeuroscience :TheOfficial
43
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
727
JournaloftheSocietyforNeuroscience,31(8),2906–15.https://doi.org/10.1523/JNEUROSCI.3684-
728
10.2011
729
Mckendrick,R.,Parasuraman,R.,Murtza,R.,Formwalt,A.,Baccus,W.,Paczynski,M.,&Ayaz,H.(2016).
730
IntoTheWild:NeuroergonomicDifferentiationofHand-HeldandAugmentedRealityWearable
731
DisplaysDuringOutdoorNavigationwithFunctionalNearInfraredSpectroscopy.Frontiersin
732
HumanNeuroscience,10(216),1–15.https://doi.org/10.3389/fnhum.2016.00216
733
Murray,J.D.,Bernacchia,A.,Freedman,D.J.,Romo,R.,Wallis,J.D.,Cai,X.,…Wang,X.-J.(2014).A
734
hierarchyofintrinsictimescalesacrossprimatecortex.NatureNeuroscience,17(12),1661–3.
735
https://doi.org/10.1038/nn.3862
736
Newtson,D.,Engquist,G.A.,&Bois,J.(1977).Theobjectivebasisofbehaviorunits.Journalof
737
738
PersonalityandSocialPsychology,35(12),847–862.https://doi.org/10.1037/0022-3514.35.12.847
Nielson,D.M.,Smith,T.A.,Sreekumar,V.,Dennis,S.,&Sederberg,P.B.(2015).Humanhippocampus
739
representsspaceandtimeduringretrievalofreal-worldmemories.ProceedingsoftheNational
740
AcademyofSciencesoftheUnitedStatesofAmerica,112(35),11078–83.
741
https://doi.org/10.1073/pnas.1507104112
742
Ranganath,C.,&Ritchey,M.(2012).Twocorticalsystemsformemory-guidedbehaviour.Nature
743
744
Reviews.Neuroscience,13(10),713–26.https://doi.org/10.1038/nrn3338
Rissman,J.,Chow,T.E.,Reggente,N.,&Wagner,A.D.(2016).DecodingfMRISignaturesofReal-world
745
AutobiographicalMemoryRetrieval.JournalofCognitiveNeuroscience,28(4),604–620.
746
https://doi.org/10.1162/jocn_a_00920
747
Rugg,M.D.,&Vilberg,K.L.(2013).Brainnetworksunderlyingepisodicmemoryretrieval.Current
748
OpinioninNeurobiology,23(2),255–260.https://doi.org/10.1016/j.conb.2012.11.005
44
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
749
Sargent,J.Q.,Zacks,J.M.,Hambrick,D.Z.,Zacks,R.T.,Kurby,C.A.,Bailey,H.R.,…Beck,T.M.(2013).
750
Eventsegmentationabilityuniquelypredictseventmemory.Cognition,129(2),241–255.
751
https://doi.org/10.1016/j.cognition.2013.07.002
752
Schapiro,A.C.,Rogers,T.T.,Cordova,N.I.,Turk-Browne,N.B.,&Botvinick,M.M.(2013).Neural
753
representationsofeventsarisefromtemporalcommunitystructure.NatureNeuroscience,16(4),
754
486–92.https://doi.org/10.1038/nn.3331
755
Shirer,W.R.,Ryali,S.,Rykhlevskaia,E.,Menon,V.,&Greicius,M.D.(2012).Decodingsubject-driven
756
cognitivestateswithwhole-brainconnectivitypatterns.CerebralCortex,22(1),158–165.
757
https://doi.org/10.1093/cercor/bhr099
758
Silbert,L.J.,Honey,C.J.,Simony,E.,Poeppel,D.,&Hasson,U.(2014).Coupledneuralsystemsunderlie
759
theproductionandcomprehensionofnaturalisticnarrativespeech.ProceedingsoftheNational
760
AcademyofSciences,111(43),E4687–E4696.https://doi.org/10.1073/pnas.1323812111
761
Simony,E.,Honey,C.J.,Chen,J.,Lositsky,O.,Yeshurun,Y.,Wiesel,A.,&Hasson,U.(2016).Dynamical
762
reconfigurationofthedefaultmodenetworkduringnarrativecomprehension.Nature
763
Communications,7(May2015),1–13.https://doi.org/10.1038/ncomms12141
764
Speer,N.K.,Zacks,J.M.,&Reynolds,J.R.(2007).HumanBrainActivityTime-LockedtoNarrativeEvent
765
Boundaries.PsychologicalScience,18(5),449–455.https://doi.org/10.1111/j.1467-
766
9280.2007.01920.x
767
Stephens,G.J.,Honey,C.J.,&Hasson,U.(2013).Aplacefortime:thespatiotemporalstructureof
768
neuraldynamicsduringnaturalaudition.TheJournalofNeuroscience,110(9),2019–26.
769
https://doi.org/10.1152/jn.00268.2013
770
Swallow,K.M.,Barch,D.M.,Head,D.,Maley,C.J.,Holder,D.,&Zacks,J.M.(2011).ChangesinEvents
45
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
771
AlterHowPeopleRememberRecentInformation.JournalofCognitiveNeuroscience,23(5),1052–
772
1064.https://doi.org/10.1162/jocn.2010.21524
773
VanDijk,T.A.,&Kintsch,W.(1983).Strategiesofdiscoursecomprehension.NewYork:AcademicPress.
774
VanRullen,R.(2016).PerceptualCycles.TrendsinCognitiveSciences,15(12),1401.
775
776
https://doi.org/10.1016/j.tics.2016.07.006
Whitney,C.,Huber,W.,Klann,J.,Weis,S.,Krach,S.,&Kircher,T.(2009).Neuralcorrelatesofnarrative
777
shiftsduringauditorystorycomprehension.NeuroImage,47(1),360–366.
778
https://doi.org/10.1016/j.neuroimage.2009.04.037
779
Zacks,J.M.,Braver,T.S.,Sheridan,M.A.,Donaldson,D.I.,Snyder,A.Z.,Ollinger,J.M.,…Raichle,M.E.
780
(2001).Humanbrainactivitytime-lockedtoperceptualeventboundaries.NatureNeuroscience,
781
4(6),651–5.https://doi.org/10.1038/88486
782
Zacks,J.M.,Speer,N.K.,Swallow,K.M.,Braver,T.S.,&Reynolds,J.R.(2007).Eventperception:amind-
783
brainperspective.PsychologicalBulletin,133(2),273–293.https://doi.org/10.1037/0033-
784
2909.133.2.273
785
Zacks,J.M.,Speer,N.K.,Swallow,K.M.,&Maley,C.J.(2010).TheBrain’sCutting-RoomFloor:
786
SegmentationofNarrativeCinema.FrontiersinHumanNeuroscience,4(October),1–15.
787
https://doi.org/10.3389/fnhum.2010.00168
788
Zacks,J.M.,Tversky,B.,&Iyer,G.(2001).Perceiving,remembering,andcommunicatingstructurein
789
events.JournalofExperimentalPsychology.General,130(1),29–58.https://doi.org/10.1037/0096-
790
3445.130.1.29
791
Zadbood,A.,Chen,J.,Leong,Y.C.,Norman,K.A.,&Hasson,U.(2016).Howwetransmitmemoriesto
792
otherbrains:constructingsharedneuralrepresentationsviacommunication.bioRxiv.
46
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
793
Zhu,Y.,Kiros,R.,Zemel,R.,Salakhutdinov,R.,Urtasun,R.,Torralba,A.,&Fidler,S.(2015).Aligning
794
BooksandMovies:TowardsStory-likeVisualExplanationsbyWatchingMoviesandReadingBooks.
795
InInternationalConferenceonComputerVision(pp.19–27).
796
Zwaan,R.,Langston,M.,&Graesser,A.(1995).TheConstructionofSituationModelsinNarrative
797
Comprehension:AnEvent-IndexingModel.PsychologicalScience,6(5),292–297.
798
https://doi.org/10.1111/j.1467-9280.1995.tb00513.x
799
Zwaan,R.,&Radvansky,G.(1998).Situationmodelsinlanguagecomprehensionandmemory.
800
PsychologicalBulletin,123(2),162–185.https://doi.org/10.1037/0033-2909.123.2.162
801
802
47
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
803
804
805
806
807
SupplementaryFigures
808
48
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
809
810
SupplementaryFigure1:Theeventsegmentationmodelrecoverseventboundariesonsimulateddata.
811
Simulateddatawithadiscreteeventstructureobscuredbyvaryinglevelsofnoisewasinputtothe
812
segmentationmodel,withT=500,K=10,andV=10.Themodelsuccessfullyrecoversamajorityofthe
813
underlyingeventboundariesatlownoiselevels,andcanstillidentifyanabove-chancefractionof
814
boundariesevenathighnoiselevelsthatareaslargeasthedifferencesbetweentheeventpatterns.
815
Havingvariableeventlengthsleadstoonlyasmalllossinperformance,anddoesnotchangetheoverall
816
performancecurve.
817
49
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
818
819
SupplementaryFigure2:Theeventsegmentationmodelsuccessfullyidentifiesswitchesbetween
820
stories.Subjectslistenedtotwostories,whichwereinterleavedsuchthattheyalternatedbackand
821
forthabouteveryminute.UsingdatafromPCC,aneventsegmentationmodelwith34eventtransitions
822
showedthebestfittoheld-outsubjects(veryclosetotheactualnumberof32).Fittingthemodelwith
823
34transitions,themajority(20)werewithin3timepointsofastoryswitch.Anulldistributionwas
824
createdbypermutingtheorderoftheevents(preservingeventlengths);underthisnulldistributionthe
825
chanceofhavingthismanyeventboundariesclosetotruestoryswitcheswasp<0.001.
826
50
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
827
828
SupplementaryFigure3:Topographyofeventtimescalesbroadlymatchesthetopographyoftemporal
829
receptivewindows.(a)Theoptimalnumberofeventsduringmoviewatching(fromFig.2)was
830
comparedtothemapofvoxeltimescales(Chen,Honey,etal.,2016),whichwasdefinedbasedon
831
sensitivitytotemporalscramblingofamovie.Althoughderivedfromverydifferenttypesof
832
experimentaldata,thesetwoapproachesyieldsimilartopographies,withearlyvisualandauditory
833
regionsexhibitingalargenumberofeventsandhavingshorttimescales(orange),andhigher-level
834
regionshavingasmallnumberofeventsandmedium/longtimescales(blue).(b)Plottingthe
835
distributionsofthenumberofeventswithintheshortandmedium/longtimescalemasksconfirmsthat
836
mostregionswithasmallnumberofeventshavemedium/longtemporalreceptivewindows,which
837
mostregionswithalargenumberofeventshaveshorttemporalreceptivewindows.
838
51
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
839
840
SupplementaryFigure4:Movieandrecalldatashowmatchingeventstructureinhigh-levelregions
841
acrossarangeofsettingsforthenumberoflatentevents.TheresultsshowninFig.5bholdformost
842
choicesofthenumberoflatenteventsbetween10and40,withdecreasinggoodness-of-fitforlarger
843
numbersofevents.Notethatthebestfitswereachievedwithmodelshavingapproximately20-25
844
events,similartotheminimumnumberofhuman-labeledeventsthatwererecalledbythesubjects(24,
845
seetableS1inChen,Leong,etal.,2016).
52
bioRxiv preprint first posted online Oct. 14, 2016; doi: http://dx.doi.org/10.1101/081018. The copyright holder for this preprint (which was not
peer-reviewed) is the author/funder. It is made available under a CC-BY-NC-ND 4.0 International license.
846
847
SupplementaryFigure5:Priordistributionovereventboundaries.Oureventsegmentationmodel
848
definesauniformprioroverallpossibleeventsegmentationsinwhicheveryeventoccursforatleast
849
onetimepointandalleventsoccurinorder.Thisinducesapriordistributionovereventboundaries,
850
shownhereforT=500,K=10.Duringtheannealingprocess,thedistributionofboundariesstartsatthis
851
prior,whichallowsfora(highlyuncertain)firstestimateofthesignatureneuralpatternforeachevent.
852
Basedonthesepatterns,thelatenteventsforalltimepointsarerefit,andthenthepatternsare
853
recalculated.Theprocesscontinues,withthepatternvarianceslowlydecreasing,untiltheloglikelihood
854
reachesapeak.
855
53