1
CollaborativeInformationSeekingLabExperimentsDataset
ChiragShahandRobertoGonzalez-Ibanez
[email protected],[email protected]
INTRODUCTION
Thisisabriefdocumentationforthedatasetbeingsharedfromasetoflabexperimentsconductedby
ChiragShahandRobertoGonzalez-IbanezatRutgersUniversityin2010-2011.Areasonablecarehas
beentakentocleanupthedataandremoveanyidentifyinginformationabouttheparticipants,but
anyoneusingthisdatashouldassumeittobetheirresponsibilitytododuediligenceregardingthis.Any
publicationsandpresentationsresultingfromtheusageofthisdatamustciteand/oracknowledgethis
data.Ittookmonthstodesignthestudyandthetoolsusedfortheexperiments,andseveralmore
monthstocollectthedata.
Howtocitethisdata:
Shah,C.,&Gonzalez-Ibanez,R.(2017).CollaborativeInformationSeekingLabExperimentsDataset.
Availablefromhttp://infoseeking.org/data.php#cis2010
Relevantpublicationstolearnmoreabouttheseexperimentsandcite:
•
•
•
•
•
Shah,C.,andGonzalez-Ibanez,R.(2011).Evaluatingthesynergiceffectofcollaborationin
informationseeking.ProceedingsofACMSIGIR,pp.913-922.Beijing,China.
Shah,C.,Gonzalez-Ibanez,R.(2012).Spatialcontextincollaborativeinformation
seeking.JournalofInformationScience(JIS).38(4),333-349.
Gonzalez-Ibanez,R.,Haseki,M.,&Shah,C.(2012).Timeandspaceincollaborativeinformation
seeking:Theclashofeffectivenessanduniqueness.ProceedingsofAssociationforInformation
Science&Technology(ASIST)AnnualMeeting,Baltimore,MD.10pp.Availablefrom
http://onlinelibrary.wiley.com/doi/10.1002/meet.14504901080/epdf.
Gonzalez-Ibanez,R.,Shah,C.,&Cardova,N.R.(2011).Smile!Studyingexpressivityofhappiness
asasynergicfactorincollaborativeinformationseeking.ProceedingsofAssociationfor
InformationScience&Technology(ASIST)AnnualMeeting,NewOrleans,LA,10pp.Available
fromhttp://onlinelibrary.wiley.com/doi/10.1002/meet.2011.14504801171/epdf.
Gonzalez-Ibanez,R.,Haseki,M.,andShah,C.(2013).Let'ssearchtogether,butnottooclose!An
analysisofcommunicationandperformanceincollaborativeinformationseeking.Information
Processing&Management,49(5),1165-1179.
Format
ThedataisavailableasanSQLdump.Oneshouldbeabletoloaditupeasilyusing‘source’commandon
anSQLpromptorimportfunctionavailableinmostGUI-basedSQLclients.Therearefivetablesinthis
database:
•
•
users–containsinformationaboutuserID,teamID,andthecondition(1to8)inwhichagiven
teamwasrandomlyassignedto(seeTable1).
actions–containslogdataofvariousactionstheusersperformed.
2
•
•
•
pages–tablecontainingpagevisits.IfapagewasaSERP,thequeryisalsoextracted.Ifthepage
wassaved,‘bookmark’issetto1.
queries–tablecontainingqueries.
snippets–tablecontainingtextsnippetstheparticipantscollected.
Toolsandinstruments
TheprimarytoolusedforcollectingthisdatawasCoagmento,whichisavailableasafree,open-source
downloadabletoolfromhttp://coagmento.org.Informationaboutothertools(e.g.,emotionextraction)
andinstruments(e.g.,questionnaires)canbeobtainedbycontactingtheauthors.
METHOD
Weconductedalaboratorystudyinvolvingatotalof160participantsin80teams.Thissectiondescribes
thestudyprocedure,thesubjects,thesystem,thetask,andtheexperimentalconditions.
Participants
Participantsinthisstudywereaskedtosignupinpairswithsomeonewithwhomtheyhadprevious
experiencecollaborating.Inaddition,theywereinformedoftheircompensationforparticipatinginthe
study,whichconsistedof$10perpersonandthepossibilitytoobtainadditionalprizesiftheywere
amongthesixbestperformingteams(twoteamseachfor$50,$25,and$15perpersonadditionally)at
theendofthestudy.AllofthesewerestudentsfromRutgersUniversity,recruitedandrandomly
assignedtoeightexperimentalscenarios(Table1).
System
WedevelopedapluginfortheFirefoxwebbrowser,calledCoagmento,1whichprovidedappropriate
toolsandsupportfortheparticipantsworkinginvariousconditions.Ascreenshotofthispluginwithin
FirefoxisshowninFigure1.Asshown,thepluginincludedatoolbarandasidebar.Thetoolbarhadthe
followingbuttons:(1)Home–fortakingtheparticipanttoappropriatequestionnaires,(2)Bookmark–
forbookmarkingawebpage,(3)Snip–forcollectingasnippetusinghighlightedtextfromawebpage,
and(4)Editor–foraccessingasharededitorforwritingthereport(Figure2).
Thesidebarhadtwomajorcomponents:achat-boxandaresourcespanel.Thechat-boxallowedthe
collaboratorsinagiventeamtocommunicatewitheachother.Theresearcherconductingthestudyalso
usedittoprovideinstructionstotheparticipants.Theresourcespanelincludedtabsforbookmarks,
savedsnippets,andexecutedqueries.
Publiclyavailablefromhttp://coagmento.org.
1
3
Figure1:Asnapshotoftheexperimentalsystemwithpartsofitshownindetails.
Figure2:Coagmento'scollaborativeeditor.
4
Sessionworkflow
EachexperimentalsessionlastedlessthananhourandwasstructuredinsixpartsasdepictedinFigure
3anddescribedbelow.
1. Participantswereintroducedtothestudyandaskedtosignaconsentform.
2. Participantswatchedabrieftutorialinordertolearnthebasicfunctionalitiesrequiredduring
thetask.
3. Participantsindividuallyfilledoutasetofpre-taskquestionnaires.Inthecaseoftwo
participantsworkingatthesamecomputer(conditionlaterexplained),theparticipantswere
separatedforthisphase.
4. Participantsreadthetaskdescription(givenlater).
5. Eachparticipant/teamworkedforapproximately25minutesonthegiventaskthatincluded
searchingforrelevantinformation,andusingittocomposeareport.
6. Participantsfilledoutpost-taskquestionnaires.
Theresearcherconductingthestudycommunicatedwiththeparticipantsthroughthechat-boxat
differenttimesduringthestudyinstructingthemtostart/stopthetaskorfillinaquestionnaire.
Figure3:Studysessionworkflow.
Conditions
Variousconditions(C1toC8)fortheseexperimentsweredefinedbasedon(1)space(C1,C2,C3);(2)
time(C1-C7forsynchronous,C8forasynchronous);(3)modeofcommunication(C1-C5);(4)tools
availableforinformationsynthesis(C7vs.restoftheconditions);and(5)availabilityofemotional
feedback(C3vs.C6).
AllbutoneconditionusedthecollaborativeeditorshowninFigure2forsynthesizinginformation
(writingareport).ForC7,theparticipantsweresetupwithGmailaccountsthattheyusedforsending
theirpiecesordraftsofthereportsback-and-forth.
Wedevelopedatoolforextractingfacialemotionsusingawebcaminrealtime.Thisinformationwas
usedinC6.Inotherwords,inC6,aparticipantwasgivenareal-timeemotionalstatusfromhis/her
collaborator.
Fortheasynchronouscondition(C8),thecollaboratorswereinvitedtothelab,andafterexplainingthe
task,theyweresplitup.Oneofthemworkedonthetaskforabout30minutes,andthentheotherone
tookoverforanother30minutes.
AsummaryofalltheseconditionscanbefoundinTable1.Pleaserefertothepublicationslistedearlier
formoredetailsabouttheconditionsandtheexperimentalsetups.
5
Table1:Experimentalconditions.
Cond.
C1
C3
C4
C5
C6
Description
Co-locatedusingthesame
computer
Co-locatedusingdifferent
computers
Remotelylocated
Remotelylocated
Remotelylocated
Remotelylocated
C7
C8
Remotelylocated
Asynchronous
C2
Communication
Inperson
Informationsynthesistool
Collaborativeeditor
Inperson
Collaborativeeditor
Textchat
Textchat+audio
Textchat+video
Textchat+emotional
feedback
Textchat
Inperson(atthebeginning
andduringthehandoff)
Collaborativeeditor
Collaborativeeditor
Collaborativeeditor
Collaborativeeditor
Email
Collaborativeeditor
Task
Wechose“gulfoilspill”asthetopicforthisexperimentationsinceitwasquitepopularandrelevantat
thetimethestudywasbeingconducted.Ourpreliminaryinvestigations,includingafewpilotruns,
indicatedthattherewasahugeamountofmaterialonthistopic,andthattheparticipantswouldfindit
interestingandchallengingenoughasanexploratorysearchtask.Eachparticipantwasgiventhe
followingtaskdescription.
“Aleadingnewspaperhashiredyourteamtocreateacomprehensivereportonthecauses,effects,and
consequencesoftherecentgulfoilspill.Asapartofyourcontract,youarerequiredtocollectallthe
relevantinformationfromanyavailableonlinesourcesthatyoucanfind.
Topreparethisreport,searchandvisitanywebsitethatyouwantandlookforspecificaspectsasgiven
intheguidelinebelow.Asyoufindusefulinformation,highlightandsaverelevantsnippets.Makesure
youalsorateasnippettohelpyouinrankingthembasedontheirqualityandusefulness.Later,youcan
usethesesnippetstocompileyourreport,nolongerthan200lines,asinstructed.
Yourreportonthistopicshouldaddressthefollowingissues:descriptionofhowtheoilspilltookplace,
reactionsbyBPaswellasvariousgovernmentandotheragencies,impactoneconomyandlife(people
andanimals)inthegulf,attemptstofixtheleakingwellandtocleanthewaters,long-termimplications
andlessonslearned.”
Theparticipantssawthisdescriptiononthescreen(part4inthestudy),andwerealsogivenaprinted
copytorefertoduringtheirsession.