Can Deep Learning solve the
Sentiment Analysis Problem?
Mark Cieliebak
Zurich University of Applied Sciences
Annual Meeting of SGAICO – Swiss Group for Artificial Intelligence and Cognitive Science
18.11.2014
Outline
1. What is sentiment analysis?
2. How good are "classical" approaches?
3. Does deep learning solve the problem?
18.11.2014
Mark Cieliebak
2
About Me
Mark Cieliebak
Institute of Applied Information Technology (InIT)
ZHAW, Winterthur
Email: [email protected], Website: www.zhaw.ch/~ciel
Open
Data
Software
Engineering
Research
Interests
Automated
Test
Generation
Text
Analytics
18.11.2014
Mark Cieliebak
3
What is Sentiment Analysis
"… WiFi Analytics is a free Android app that I find very
handy when it comes to troubleshooting and monitoring
a home network. "[1]
18.11.2014
Mark Cieliebak
4
Sample Application:
Social Media Monitoring
[7]
Text Analytics Components:
• Find relevant documents
• Hot topic Analysis
• Sentiment analysis
18.11.2014
Mark Cieliebak
5
Flavours of Sentiment Analysis
• Document Based
• Sentence Based
• Target-Specific
• Rating Prediction
18.11.2014
Mark Cieliebak
6
Classic Approaches to Sentiment Analysis
Corpus-Based
Rule-Based
Predicted
Label
[4]
[3]
18.11.2014
Mark Cieliebak
7
Simple Sentiment Analysis
Idea: Count number of positive and negative words
"This camera is great[+1]."
+1 (pos)
"I find it beautiful[+1] and good[+1]."
+2 (pos)
"It looks terrible[-1]."
-1 (neg)
"This car has a blue color."
0 (neu)
Use Sentiment-Dictionary:
18.11.2014
POSITIVE:
NEUTRAL:
NEGATIVE:
great
hello
bad
love
see
hate
nice
I
ugly
...
…
...
Mark Cieliebak
8
Sample Rules
• Detect Booster Words: "The car is really very expensive[-1 -1 -2] ."
• New Category "Mixed": "This car has an appealing[+1] design and
comfortable[+1] seats, but it is expensive[-1]. "
• Negation: Invert only score of words occuring after the negation:
"The car is appealing[+3] and I do not[*-1] find it expensive[-2]"
• I do not find the car expensive and it is appealing.
 Need to “understand” the sentence
18.11.2014
Mark Cieliebak
9
Linguistic Analysis
Sentence
Sentence
Noun
Phrase
Sentence
Verb Phrase
Verb
Adverb
Verb
Det.
I
Conj.
do
not
find
Noun
Phrase
Det
Noun
the
car
Adj.
expensive
Noun
Phrase
and
Verb Phrase
Det.
Verb
Participle
it
is
appealing
-> RULE: Invert scores of words being in the same phrases as negation.
“I do not find the car expensive[+2]
and it is appealing[+3].” → +5 (pos)
18.11.2014
Mark Cieliebak
10
Rule-Based Sentiment Analysis
Most Important Issues:
- Requires good hand-crafted rules
- Hard to transfer to new tasks
or languages
[5]
- Does not work well for texts with bad grammer (Twitter)
18.11.2014
Mark Cieliebak
11
Classic Approaches to Sentiment Analysis
Corpus-Based
Rule-Based
Predicted
Label
[4]
[3]
18.11.2014
Mark Cieliebak
12
Corpus-Based Sentiment Analysis
Predicted
Label
[4]
18.11.2014
Mark Cieliebak
13
Corpus-Based Sentiment Analysis
Annotated Corpus
Sentence
Polarity
This analysis is good.
Pos
It looks awful.
Neg
This car has a blue color.
Neu
This car has an appealing design, comfortable seats, but it is expensive.
Mix
This car has a very appealing design, comfortable seats, but it is really expensive.
Mix
This analysis is not good.
Neg
This car has an appealing design, comfortable seats and it is not expensive.
Mix
This movie was like a horror event.
Neg
This car is appealing and is not expensive.
Mix
...
...
18.11.2014
Mark Cieliebak
14
Sample Features for Tweets
• Word ngrams: presence or absence of contiguous sequences of 1, 2, 3,
and 4 tokens; noncontiguous ngrams
• POS: the number of occurrences of each part-of-speech tag
• Sentiment Lexica: each word annotated with tonality score (-1..0..+1)
• Negation: the number of negated contexts
• Punctuation: the number of contiguous sequences of exclamation marks,
question marks, and both exclamation and question marks
• Emoticons: presence or absence, last token is a positive or negative
emoticon;
• Hashtags: the number of hashtags;
• Elongated words: the number of words with one character repeated (e.g.
‘soooo’)
from: Mohammad et al., SemEval 2013
18.11.2014
Mark Cieliebak
15
Corpus-Based Sentiment Analysis
Most Important Issues:
- Requires large annotated corpora
- Depends on good features
[6]
18.11.2014
Mark Cieliebak
16
How good are Sentiment Analysis Tools?
18.11.2014
Mark Cieliebak
17
Quick Poll
"How good are state-of-the-art
sentiment analysis tools?"
• Short texts: 1-2 sentences from
Twitter, news, reviews etc.
• Three-class classification: positive,
negative, other
# 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑑𝑜𝑐𝑠
• Accuracy =
# 𝑑𝑜𝑐𝑠
18.11.2014
Accuracy Votes
<50%
50-60%
60-70%
70-80%
80-90%
>90%
Mark Cieliebak
21
Tool Accuracy
Accuracy
Avg.
0,8
Best Tool per Corpus
61%
0,7
Worst Tool per Corpus
40%
0,6
0,5
0,4
0,3
0,2
[14]
18.11.2014
Mark Cieliebak
22
Tool Accuracy
0,8
Accuracy
0,7
Best Tool per Corpus
Avg.
61%
Worst Tool per Corpus
40%
Overall Best Tool
59%
0,6
0,5
0,4
0,3
0,2
18.11.2014
Mark Cieliebak
23
Take-Home Lesson
Accuracy of best commercial tool on
arbitrary short texts is 59%
18.11.2014
Mark Cieliebak
24
Approaches to Sentiment Analysis
Corpus-Based
Rule-Based
Pr
edi
cte
d
La
bel
Deep Learning
[9]
[8]
18.11.2014
Mark Cieliebak
25
Deep Learning on Text
It's all about Word Vectors!
18.11.2014
Mark Cieliebak
26
Word2Vec
[9]
• Huge set of text samples
(billions of words)
• Extract dictionary
• Word-Matrix: k-dimensional
vector for each word (k typically
50-500)
• Word vector initialized randomly
• Train word vectors to predict
next words, given a sequence of
words from sample text
Major contributions by Bengio et al. 2003, Collobert&Weston 2008, Socher et al. 2011, Mikolov et al. 2013
18.11.2014
Mark Cieliebak
27
The Magic of Word Vectors
[10]
King
-
Man +
Woman ≈ Queen
Live Demo on 100b words from Google News dataset: http://radimrehurek.com/2014/02/word2vec-tutorial/
18.11.2014
Mark Cieliebak
28
Relations Learned by Word2Vec
[11]
18.11.2014
Mark Cieliebak
29
Using Word Vectors in NLP
Collobert et al., 2011:
• SENNA: Generic NLP System based on word vectors
• No manual feature engineering
• Solves many NLP-Tasks as good as benchmark systems
[12]
18.11.2014
Mark Cieliebak
30
Deep Learning and Sentiment
Maas et al., 2011
• Enrich word vectors with sentiment context
• Capture semantic of words (unsupervised) and sentiment (supervised) in
parallel, using multiple learning tasks
wonderful
amazing
terrible
awful
18.11.2014
Mark Cieliebak
31
Deep Learning and Sentiment
Socher et al. 2013:
• Word Vectors do not help for Sentiment Analysis
• Recursive Neural Tensor Networks
• Representing sentence structures as trees while adding sentiment
annotations at same time
• Restricted to single, well-structured sentences
•
18.11.2014
Mark Cieliebak
[13]
32
Deep Learning and Sentiment
Quoc and Mikolov, 2014:
• "Paragraph Vectors"
• Add context (sentence, paragraph, document) to word vectors during
training
• Improves many existing approaches
[9]
18.11.2014
Mark Cieliebak
33
Does Deep Learning solve the
Sentiment Analysis Problem?
18.11.2014
Mark Cieliebak
34
Conclusion: Deep Learning for Sentiment
• Small improvements, not revolution
• Very recent research, not yet "end of the story"
• SemEval 2015 will be benchmark
18.11.2014
Mark Cieliebak
35
Talk in Short!
1. Classic approaches are rule-based
or corpus-based
2. State-of-the-art tools classify 4
out of 10 docs wrong
3. Deep Learning does not need
hand-crafted features
4. Deep Learning improves
existing benchmarks
18.11.2014
Mark Cieliebak
36
Thank You!
[15]
Mark Cieliebak
Zurich University of Applied Sciences (ZHAW)
Winterthur, Switzerland
Email: [email protected], Website: www.zhaw.ch/~ciel
18.11.2014
Mark Cieliebak
37