Iden%fyingSen%mentandEmo%on
inLowResourceLanguages
JuliaHirschbergandZixiaofan
(Brenda)Yang
ColumbiaUniversity
HumanitarianAssistanceandDisaster
Relief(HADR)inDARPALorelei
•  Indisastersitua%ons,interna%onalrespondersmay
notspeakthelanguageoftheareaindistressandmay
haveliIlereliableaccesstolocalinformants
–  7100+ac%velanguagesintheworld--hardtopredict
whichlanguageswillbeneedednext
•  44inBokoHaramarea(Hausa,Kanuri)~522languagesinallof
Nigeria
•  19inEbolaoutbreakareasinLiberia,SierraLeone,andGuinea
•  20+MayanlanguagesspokenbyCentralAmericanrefugee
children
–  Currentmethodsrequire3yearsand$10M’sperlanguage
(mostlytopreparetrainingcorpora)
•  Wouldrequire$70Band230Kperson-yearstohandleall
languages
Challenge
•  Howcanwedeveloplanguagetechnologies
quicklytohelpfirstrespondersunderstand
textandspeechinforma%onvitaltotheir
mission(socialmedia,hotlinemsgs,news
broadcasts)?
–  Triageinforma%onbyurgencyandsen%ment/
emo%on(anger,stress,fear,happiness)
–  Displayinforma%oninaformthatreliefworkers
caneasilyunderstand
OurGoal
•  Iden%fysen%mentandemo%oninwriIenand
spokendatatosharewithreliefworkers
–  Provideaddi%onal,extra-proposi%onalmeaning
•  Fearandstressofvic%ms
•  Happinessatsuccessofreliefefforts
•  Angeratreliefworkers
–  Method:Developwaystorecognizeandinterpret
sen%mentandemo%oninLRLsbytrainingonHigh
ResourceLanguagesandotherLRLs
ThreeMainPossibili%es
•  Canwerecognizeemo%onsrelevanttoLorelei
fromlabeledspeech(e.g.anger,stress,fear)
•  Cantext-trainedsen%mentsystemsbeusedto
labelunlabeledspeechtranscriptstotrain
sen%mentrecogni%oninspeech?
•  Cansystemstrainedonemo%on/sen%mentin
speechofonelanguagebeusedtorecognize
emo%on/sen%mentinanother?
Angervs.Neutral
•  Corpus:MandarinAffec%veSpeech
•  Language Mandarin
–  Neutralsentences(e.g.“Itwillraintonight.”)and
words(e.g.“train,”“apple”)
–  5basicemo%ons(neutral,anger,ela%on,panic,
sadness)simulatedby68students
•  Ourstudy:Anger:5100vs.Neutral:5100
FeatureExtrac%onUsingopenSMILE
•  Baselinefeatures(384)
•  ‘Standard’simplelow-levelacous%cfeatures
(e.g.,MFCC’s;max,minandmeanframeenergy)
•  ‘Unique’features(e.g.slopeandoffsetofalinear
approxima%onofMFCC1-12)
–  Largerfeatureset(6552)
•  MoreFunc%onalsandLow-LevelDescriptors
MachineLearningResults
•  Randomforest:(Scikit-learn)
– Traindecisiontreeclassifiersonvarious
sub-samplesofthetrainingsetusing384
featureset
– Usesaveragingtoimprovethepredic%ve
accuracyandcontrolover-fipng
•  WeightedF-measure:0.88(0.50
baseline);P=.88;R=.88
UsefulFeatures
•  Arithme%cmeanandmaxvalueofMFCC[1](mel
frequencycepstralcoefficients)
•  Theoffsetoflinearapproxima%onofroot-meansquareframeenergy
•  Arithme%cmeanandmaxvalueofMFCC[2]
•  Range,maxvalue,quadra%cerrorandstandard
devia%onofthe1storderdeltacoefficientofMFCC[1]
•  Offsetoflinearapproxima%onofMFCC[1]
•  Arithme%cmeanofroot-mean-squareframeenergy
English:Stressvs.Neutral
•  Corpus:SUSAS(Speechundersimulatedand
actualstress)
–  Neutralwords(e.g.“break”or“eight”)simulated
by9speakers
–  Stressproduceddoingsingletrackingtasks
–  Stress:630;Neutral:631
•  Classifica%onresultonrandomforestmodel:
WeightedF-measure:0.7031(.50baseline);P=.
70;R=.70
Mul%-labeledSemaineCorpus
•  Queen’sUBelfast,hIp://semaine-db.eu
•  Naturalinterac%onsinEnglishbetweenusersandan
’operator’simula%ngaSensi%veAr%ficialListener
(SAL)agent
•  SALagentexamples –  ‘Dotellmeallthedeliciousdetails.’
–  ‘Ohh....thatwouldbelovely.’
–  ‘Whatareyourweaknesses?’
–  ‘It'sallrubbish.’
•  Annota%onsby6-8ratersforeachconversa%on
–  Fullra%ngforvalence,ac%va%on,power,
expecta%on/an%cipa%on,intensity
–  Op%onalra%ngforbasicemo%ons:anger,happiness,
sadness,fear,contempt…
•  SolidSALpart:87conversa%ons,eachlas%ng
approximately5minutes
Examples
•  Valencescore:-0.8819
•  Valencescore:0.1125
•  Valencescore:0.5803
•  Valencescore:0.8308
ComparingHumanSen%mentLabels
toAutoma%cLabels
•  Ques%on:
–  Supposewehaveunlabeledspeech,canweannotate
transcriptsautoma%callywithasen%mentannota%onsystem
andusethoselabelsforunlabeledspeechinsteadofmanual
labels?
•  Method:
–  Segmenttranscriptsintosentencesandalignwithspeech
–  TurnSemainemanual,con%nuouspos/neglabelsintobinaryfor
useasgoldstandard
–  Labeltrainingtranscriptsentencesusingtext-trainedsen%ment
analyzertolabelposi%ve/nega%ve/neutral
–  Buildclassifierfromsen%ment-labeledspeechandcompareto
classifierbuiltusingmanualSemainespeechlabels
EnglishText-basedSen%mentAnalysis
•  Sen%mentdetec%onsystem(Rosenthal2014)
•  Features(lexical,syntac%c):
- 
- 
- 
- 
- 
Dic%onaryofAffectandLanguage(DAL)
WordNet3.0
Wik%onary
POStags
Top500n-gramfeatures
•  Outputlabel:posi%ve/nega%ve/neutral
Comparisonof
Sen%mentLabelsvs.ValenceScores
•  Examples:
–  Anywayhewouldprobablydoallthewrongshopping.
•  Sen%mentanalysisoutputlabel:NegaDve
•  Valencescore:-0.4420
–  Theremustbelot’sofhappythingsinyourlife.
•  Sen%mentanalysisoutputlabel:PosiDve
•  Valencescore:0.7451
–  *AndhowamIgoingtowrapallthepresents?
•  Sen%mentanalysisoutputlabel:Neutral
•  Valencescore:-0.4090
–  *Lifeisverybad,Idon’tsupposeyoursisanybeIer.
•  Sen%mentanalysisoutputlabel:PosiDve
•  Valencescore:-0.7500
Comparisonof
Sen%mentLabelsvs.ValenceScores
•  Sen%ment:Posi%ve 1301,Nega%ve:978,Neutral:1177
•  Distribu%onofsen%mentlabelsovervalencescores:
ResultsofSen%mentAnalysisof
Transcripts
•  Manuallyannotatedvalencescoresare
unbalanced:
–  2363sentenceswithposi%vescore(score>=0)
–  1093sentenceswithnega%vescore(score<0)
•  Set‘neutral’thresholdto0.118
–  1728sentenceswithposi%ve/nega%vescore
•  Precisionofsen%mentlabelsusingnew
threshold:
–  Posi%velabelprecision:57.88%
–  Nega%velabelprecision:60.22%
Experiments:
Sen%mentLabelsvs.ValenceScores
•  openSMILEbaseline(384)featureset
•  4speechexperiments:
-  Trainonsen%mentlabels;testonsen%mentlabels
-  *Trainonsen%mentlabels;teston(human)valence
scores
-  Trainon(human)valencescores;testonsen%ment
labels
-  *Trainon(human)valencescores;teston(human)
valencescores
•  10-foldcrossvalida%on;weightedf-measure
Experiments:
Sen%mentLabelsvs.ValenceScores
•  Unbalancedclassesintrainingdata:
–  Movingthresholdscoreforabalanceddivision
–  Upsampling
–  Downsampling
•  Machinelearningalgorithms:(Scikit-learn)
–  Linearmodels:Linearregression;Ridge;Lasso
–  Nearestneighborsmodel:KNN
–  Treemodel:Decisiontree
–  Ensemblemodels:Randomforest;AdaBoost
•  Unbalancedclassesintestdata:
–  Evalua%on:WeightedF-measure
Experiments:
Sen%mentLabelsvs.ValenceScores
•  Baseline:Majorityclass(posi%ve)
Trainon
SenDmentLabels
SemaineValenceScores
Teston
SenDment
Labels
Valence
Scores
SenDment
Labels
Valence
Scores
Baseline
0.4140
0.5526
0.4140
0.5526
RandomForest
0.5425
0.6111
0.4979
0.6897
•  Shouldimprovewhenweaddlexicalfeaturesto
acous%cones
Cross-LingualTraining
•  GivenacorpusofangerinEnglish,canwe
predictangerinMandarin?
•  GivenacorpusofangerinMandarin,canwe
predictangerinEnglish?
•  TrainonEnglishSemaine,testonMandarin
AffectCorpus:F1=0.56(cf.Mand/Mand0.88)
TrainonMandarinAffect,testonEnglish
Semaine:F1=0.62(cf.Eng/Eng:F1=.77)
ConclusionsandFutureWork
•  Wecandetectemo%onslikeangerandstress
fromlabeledMandarinandEnglishspeech
reasonablywell
•  Wecandetectemo%ons(e.g.anger)bytraining
ononelanguageandtes%ngonanotherwith
performanceabovethebaselines
•  WecandetectmanuallylabeledEnglish
emo%onslspeechfromtranscriptsautoma%cally
labeledwithsen%ment,alsowithpromising
results
•  Future:AppenLoreleiandBabellanguages
(Turkish,Mandarin,Uyghur)
–  Developtext-basedsen%mentdetectorscrosslinguallyforLRL
–  Detectsen%mentinAppentranscripts
–  Labelalignedspeech
–  Trainsen%mentmodelson”labeled”speech
–  DeepLearning
Thankyou!
Ques%ons?