Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Don't you love it? Sentiment Analysis with
Crowd Sourcing
Wouter van Atteveldt et al.
2017-02-20
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
About me
ˆ Wouter van Atteveldt (
ˆ
ˆ
ˆ
ˆ
ˆ
)
Political communication, VU Amsterdam
MSc Articial Intelligence (Edinburgh)
PhD AI & Communication Science (VU Amsterdam)
Research: Automatic Text Analysis, Data Analysis
http://vanatteveldt.com
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
What is sentiment analysis?
ˆ Measure evaluative meaning of (subjective) language
ˆ "This movie sucks" → negative
ˆ Applications e.g.
ˆ analysing hotel reviews
ˆ automatic stock market trading
ˆ early warning systems (for brands and countries)
ˆ Pang, B., & Lee, L. (2008). Opinion mining and
sentiment analysis. Foundations and trends in information
retrieval, 2(1-2), 1-135.
ˆ Liu, B. (2012). Sentiment analysis and opinion mining.
Synthesis lectures on human language technologies, 5(1),
1-167.
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Common approaches in sentiment analysis
ˆ Human annotation
ˆ Dictionaries of positive, negative terms
ˆ semi-automatic expansion of dictionaries
ˆ Machine learning
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Problems with sentiment analysis
ˆ Evaluations are inherently subjective
ˆ Evaluative language is creative and context-sensitive
ˆ (even more so than factual language)
ˆ Evaluation implies a relation, but most data/tools
undirected
ˆ
Source
likes/dislikes
target
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Problems with sentiment analysis
ˆ "Terrorist Attacks: 250 Innocent Massacred By ISIS"
ˆ "Saddam Hussein was executed by hanging"
ˆ "This car has better mpg than my old Volvo"
ˆ "Preacher who applauded Orlando mass killing asked to
relocate"
ˆ "Janeane Garafalo also was an interesting character ."
ˆ "Brexit means brexit"
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Sentiment analysis: what do I want
Task denition
Given a piece of text, what do the author and mentioned
actors think about all mentioned actors and issues
ˆ Preacher who applauded Orlando mass killing asked to
relocate
ˆ → preacher/+/killing
ˆ "This car has better mpg than my old Volvo"
ˆ → author/+/thiscar, author/-/volvo
ˆ "Terrorist Attacks: 250 Innocent Massacred By ISIS"
ˆ → ISIS/-/innocents, author/-/ISIS
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Crowd sourcing
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Crowd sourcing
ˆ Use anonymous/untrained people from Internet to
perform a task
ˆ Useful for simple tasks
ˆ Split bigger tasks into smaller steps
ˆ Can be very cheap (cents per unit)
ˆ Quality control using test questoins
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
CrowdFlower
ˆ Platform for easily distributing tasks
ˆ Other platforms exist, e.g. mturk
https://make.crowdflower.com/jobs/933225/editor
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Why use Crowd Sourcing for sentiment analysis?
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Why use Crowd Sourcing for sentiment analysis?
ˆ Task is easy to explain
ˆ Judgment is subjective anyway
ˆ Low cost means multiple codings per unit possible
ˆ Low cost means task/domain dependent analysis viable
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Previous work in social science
ˆ Ken Benoit et al,
, APSR
Crowd-sourced text analysis
2016
ˆ Martin Haselmayer et al,
Sentiment analysis of political
communication: combining a dictionary approach with
, QQ 2016
ˆ Richard Socher et al, Recursive
crowdcoding
Deep Models for
,
Semantic Compositionality Over a Sentiment Treebank
EMNLP 2013
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Benoit et al
ˆ Compare crowd coding to manual coding
ˆ Policy positions in party manifestoes
ˆ 18k sentences, 4-6 expert codings, 5-20 crowd codings
Benoit, K., Conway, D., Lauderdale, B. E., Laver, M., &
Mikhaylov, S. (2016). Crowd-sourced text analysis:
reproducible and agile production of political data. American
Political Science Review, 110(02), 278-295.
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Benoit et al
Errors when using multiple experts / crowd coders
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Benoit et al
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Haselmayer et al
ˆ Create German sentiment dictionary using codings
ˆ Applied to election coverage statements
ˆ 13k sentences, 130k codings (2k euro)
Haselmayer, M., & Jenny, M. (2016),
Sentiment analysis of
political communication: combining a dictionary approach
. Quality & Quantity, 1-24.
with crowdcoding
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Haselmayer et al
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Socher et al
ˆ Recursive Neural Network of sentiment
ˆ Crowd sourced 215k phrases in 12k sentences
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Socher et al
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Socher et al
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Existing work: summary
ˆ Promising results of crowd sourcing
ˆ 3 use cases / approaches for crowd results
ˆ Direct use
ˆ Build dictionary
ˆ Build statistical model
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Our Objectives
ˆ Build sentiment tools for Dutch, English
ˆ Compare 3 approaches
ˆ Evaluate accuracy, costs, generalizability
ICA Panel proposal: Automatic Sentiment Analysis for
Communication Research (Wouter van Atteveldt and Pablo
Barbera)
Using crowdsourcing for developing an attributed sentiment analysis
tool (Wouter van Atteveldt, Antske Fokkens, Isa Maks, Kevin van
Veenen, and Mariken van der Velden)
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Method
ˆ Tweets and newspaper sentences about Ukraine treaty
ˆ Coded manually and by CrowdFlower
ˆ Compared to sentiment dictionary and Coosto
ˆ (Coosto is a social media analytics company that does
sentiment analysis)
ˆ (n~200)
Kevin van Veenen (MA-thesis 2016), Methodologal study of
automatic sentiment analysis in political news
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Results
Method
All sentences:
Expert vs Dictionary
Expert vs Domain Dictionary
Expert vs Crowd
Tweets:
Expert vs. Coosto
Dictionary vs. Coosto
Domain dictionary vs Coosto
Crowd vs Coosto
Don't you love it? Sentiment Analysis with Crowd Sourcing
n Reliability Correlation
(kappa)
(rho)
139
158
190
.38
.58
.86
.48
.58
.86
132
104
109
132
.49
.63
.43
.52
.56
.68
.55
.59
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Accuracy by number of coders
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Costs
Task
N Cost codings / $
Expert coding
190 87.5
2.2
Crowd (15 codings) 450 10.78
42
Crowd (3 codings) 480 4.02
119
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Conclusion
ˆ Crowd Sourcing cheap and accurate
ˆ To-do:
ˆ Finish English task
ˆ Train machine learning / dictionary with crowd data
ˆ Error analysis / validation
ˆ Create easy tool for using results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Other work
Some other things I am working on:
ˆ AmCAT Text Analysis Toolkit
ˆ NLPipe Linguistic Processing
ˆ Clause analysis and Source Detection
ˆ Scraping Cantonese
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
AmCAT
ˆ Amsterdam Content Analysis Toolkit
ˆ Open source web-based text analysis tool
ˆ Automatic analysis, quantitative manual analysis
ˆ Multi-user, permissions per project
ˆ REST API, integration with R/python
ˆ Setup your own server or use ours
ˆ (experimental docker support)
ˆ http://wiki.amcat.nl, https://amcat.nl
ˆ http://github.com/amcat/amcat
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
NLPipe
ˆ Natural Language Pipelining
ˆ Easy docker-installable server for NLP
ˆ Lemmatizing, POS tagging, parsing
ˆ Integrated with R / Python
ˆ Modules for Dutch, English
ˆ http://github.com/vanatteveldt/nlpipe
Automatic Text Analysis Made Easy: Using AmCAT, NLPipe and R
to do corpus management, linguistic processing, and automatic text
Wouter van Atteveldt, Kasper Welbers, Antske Fokkens,
Nel Ruigrok, Martijn Bastiaan, Christian Stuart (ICA 2017)
analysis.
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Clause Analysis / RSyntax
ˆ Detect source use and clauses
ˆ Who does what to whom
ˆ http://vanatteveldt.com/2016-clause-analysis/
(In press, Political Analysis)
ˆ http://github.com/vanatteveldt/rsyntax
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Scraping Cantonese
ˆ With Chris Fei Shen
ˆ HK Discussion forums in Cantonese
ˆ No segmenter exists for Cantonese
ˆ Cannot do corpus analysis, machine learning etc.
ˆ Scrape discusshk.com, build segmenter from
trigrams/bigrams
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Scraping Cantonese
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Don't you love it? Sentiment Analysis with Crowd Sourcing
Other work
Wouter van Atteveldt et al.
Sentiment Analysis
Sentiment Analysis: Methods and Results
Other work
Conclusion
ˆ I like automatic text analysis :)
ˆ All tools, code available on github
ˆ Try it out and let me know!
Slides: http://vanatteveldt.com/cityu_seminar
Don't you love it? Sentiment Analysis with Crowd Sourcing
Wouter van Atteveldt et al.