Suggestion Analysis and Detection of
Explicit Suggestions
Sapna Negi, Paul Buitelaar
UNLP PhD Day: 25 November, 2014
From Sentiments to…
I am not a fan of
vintage themes.
This phone
was a bad
decision.
Nikon cameras
do wonder with
colors!
Suggestions
I wish this
phone came
in other
colors too
The window
ledgers need
cleaning.
I suggest to keep
a set of normal,
macro and
telephoto lens
Outline
1.  Introduction
2.  Suggestion Analysis
3.  Data and Experiments
4.  Conclusion
Introduction
Setiment Analysis: Identification of sentiments
Sentences
Sentiment Category
I love my new iPad.
positive
I did a big mistake by buying
this product.
negative
I bought this phone 2 months
back.
neutral
Opinion Mining ≈ Sentiment Analysis
Other information in opinionated text?
“WP7 is nice but its v.1 couldn't compete with latest version of
Android. Hopefully Microsoft will continue to iterate. Please
Microsoft, more apps needed on Marketplace! I can't keep
switching!...”
“This C4280 has filled the bill in those areas, however, for the
price, HP should have considered throwing in more features
and lowering the cost of printing….”
- 
view
e
r
e
h
t
nts
e
m
o
m
r
f
e
way r improv
a
e
k
-  Ta estions fo
Sugg
Other information in opinionated text?
"Breakfast was typical hotel fare, not especially exciting, but
fresh, varied and plentiful. Room-service fast and delicious,
great selection of food. Ate dinner at Georgetown - highly
recommended. There are plenty of places to eat in the area
around the hotel. Try the baguette-sandwiches from
Patisserie Paul in the tube-vaults.”
ers
da
n
a
s
p
Ti
dv
the
o
t
s
e
ic
fe
tom
s
u
c
llow
Applications
Applications
sked
a
y
l
t
i
c
Expli eviewers
r
from
Applications
For customers:
-  See which restaurant travellers prefer
-  See recommended shops
-  …..
-  Other suggestions
Applications
For customers:
-  See which restaurant travellers prefer
-  See recommended shops
-  …..
-  Other suggestions
For service providers/manufacturers:
-  Suggestions for improvements
-  Suggestions for new features
Suggestion Detection: Starting point
Task: Automatic identification of a suggestion bearing sentence
from given sentences.
Definition of Suggestion (Oxford Dictionary):
An idea or plan put forward for consideration.
Synonyms: Proposal, recommendaion, advice, hint, tip
Suggestion Detection: Starting point
Task: Automatic identification of a suggestion bearing
sentence from given sentences.
Definition of Suggestion (Oxford Dictionary):
for a
e
v
i
t
An idea or plan put forward for consideration. ighly subjec al task!
H
tation
comp
u
Synonyms: Proposal, recommendaion, advice, hint, tip
Related Work: Suggestion Detection
Identification of suggestion bearing text
Brun et al 2013: Suggestion extraction from product reviews
(F measure = 0.73, small evaluation dataset, unavailable)
Dong et al 2013: Suggestion tweet identification for products
(F measure = 0.69, dataset available)
Related Work: Suggestion Detection
Task: Identification of suggestion bearing text
Brun et al 2013: Suggestion extraction from product reviews
(F measure = 0.73, small evaluation dataset, unavailable)
Dong et al 2013: Suggestion tweet identification for products
(F measure = 0.69, dataset available)
- No clear definition of ‘suggestions’, inconsistent annotations
Related Work: Advice Detection
Task: Identification of advice revealing sentences in travel forums.
(Wicaksono et al, 2013)
-  F score = 0.75, Data available
-  Any information provided from a traveller is tagged as advice
-
No clear definition of ‘advice revealing’, inconsistent annotations
Sample Advice Labelling Task
Guess the Labels: advice/ non-advice
Id
Sentence
Label
1
We went there on 20th of April for a week
?
2
The weather was beautiful and not too hot.
?
3
Hotel was very comfortable and very nice
?
4
We used both credit card and ATMs in Turkey with no
problems using just commonsense on where and when to
use either .
?
5
If you have an iPad/Phone , there are a couple of apps
“Galileo Offline Maps” and “OffMaps” one of which may
meet your needs .
?
Sample Advice Labelling Task
Labels in the Advice Dataset
Id
Sentence
Label
1
We went there on 20th of April for a week
advice
2
The weather was beautiful and not too hot.
advice
3
Hotel was very comfortable and very nice
advice
4
We used both credit card and ATMs in Turkey with no
advice
problems using just commonsense on where and when
to use either .
5
If you have an iPad/Phone , there are a couple of apps advice
“Galileo Offline Maps” and “OffMaps” one of which may
meet your needs .
Contribution
-  Analysis of the task of Suggestion Detection
-  Prepare Gold Standard Dataset
-  Linguistic analysis of suggestion bearing sentences
-  Technique for suggestion detection
Suggestion Analysis
Dimensions of Suggestions in Opinion Mining
1) 
- 
- 
- 
- 
- 
Domain (Data Source, Product/Service Type)
Product Review: Brun et. al. 2013
Hotel Review
Weblogs/Discussion Forums: Wicaksono et al, 2013
Twitter: Dong et. al. 2013
…
2)
- 
- 
- 
Target
Provider/manufacturer: Brun et. al. 2013, Dong et. al. 2013
Consumers/customers:
Enquirer: Wicaksono et al, 2013
- 
…….
Dimensions of Suggestions
3) Level
-  Explicit:
I suggest to eat at the bakery next door.
A chest of drawers would be a useful addition to the room.
(Suggestions at Syntactic and Semantic level)
-  Implicit:
The bakery next door was really nice.
We struggled to fit in our stuff in the provided storage space.
(Suggestions at Pragmatic level)
Dimensions of Suggestions
3) Level
-  Explicit:
I suggest to eat at the bakery next door.
A chest of drawers would be a useful addition to the room.
(Suggestions at Syntactic and Semantic
level)
rk?
d Wo
e
t
a
l
Re
ar
e
l
c
t
No
-  Implicit:
The bakery next door was really nice.
We struggled to fit in our stuff in the provided storage space.
(Suggestions at Pragmatic level)
Data and Experiments
Objective
Objective:
Classification of sentences into suggestion and non-suggestions.
Values of Dimensions for Suggestion:
- Domain: Hotel Reviews
-  Target : Fellow customers
-  Level: Explicit
Less Subjective Annotations
Id
Sentence
Label
1
We went there on 20th of April for a week
no
2
The weather was beautiful and not too hot.
no
3
Hotel was very comfortable and very nice
no
4
We used both credit card and ATMs in Turkey with no
problems using just commonsense on where and
when to use either .
no
5
If you have an iPad/Phone , there are a couple of
apps “Galileo Offline Maps” and “OffMaps” one of
which may meet your needs .
yes
Data Preparation
- 
- 
- 
TripAdvisor Hotel reviews
330 reviews manually split into ~5300 sentences
Sparse dataset: 230 sentences (4.5% tagged as suggestions)
Re-tagged advice tgged sentences in the advice corpus, 2191
sentences
Process:
-  Labelled using crowdsourcing (CrowdFlower platform)
Quality Assurance:
-  Iteratively improved annotation guidelines.
-  Annotators restricted to native speakers and experienced workers
-  Required Test score: above 80% on gold questions
-  Minimum of 3 judgements
-  High confidence score of atleast 0.8 should be achieved
Data Analysis
Strategies to express suggestions
Strategy
Example
Obligations (with or without a
subject)
-  You should visit the bakery next door
-  Take rooms on side or rear
Condition + predicted outcome
If you can secure a special rate and spend
most of your time out, then the location will
compensate hotel’s downsides.
Recommend
I recommend visiting the Turkish restaurant
in Wilton Road called Mezze
Warn
Be careful while snorkling at Red Beach
Request
Please build one lounge that has AC
Prohibitions
Do not take a room at the ground floor.
Approach
Statistical Classification
Features:
-  Words in “suggestion clause” , customised stopword list
-  2, 3 length frequent sequences of part of speech tags of
suggestion clause
-  If Sentence begins with a verb base form (go,ask, do, visit…)
-  Modal Verbs (could, would….)
-  Suggestion verbs (recommend, suggest, advice,..)
-  Prohibition verbs (avoid,.., Do not)
-  Suggestion nouns (advice, suggestion, need,..)
Approach
Suggestion Clause:
-  Often 2 or more clauses in sentences
-  Only one expresses suggestion
-  Others add context
Example:
-  We found the dining options expensive, even for the chain and
considering the plethora of great places to eat within walking distance, I
recommend dining out.
Presence of:
-  second person pronoun (you)
-  base form of verb
-  Suggestion verb
-  modals
Evaluation
Train = TripAdvisor: 4839 sentences; Travel Forum: 220 suggestions
Test = TripAdvisor: 600 sentences
Training Dataset
Classifier
Precision
Recall
F Score
TripAdvisor
Naïve Bayes
0.65
0.46
0.54
TripAdvisor
SVM
0.88
0.51
0.60
TripAdvisor + Forum
Naïve Bayes
0.70
0.49
0.59
TripAdvisor + Forum
SVM
0.829
0.56
0.67
Conclusion
-  Positive Results, Work in progress
-  Suggestion theory, Work in progress
-  Benchmark dataset
Future Work
Short term:
-  Improve current results.
-  Domain independent classifier.
Long Term:
- Aspect Based Suggestion: Given the entity, find related suggestion
-  Implicit Suggestions
Suggestions?