On Enhancing the Label Propagation Algorithm
for Sentiment Analysis Using Active Learning
with an Artificial Oracle
Anis Yazidi, Hugo Lewi Hammer, Aleksander Bai, and Paal Engelstad(!)
Department of Computer Science,
Oslo and Akershus University College of Applied Sciences, Oslo, Norway
{anis.yazidi,hugo.hammer,aleksander.bai,paal.engelstad}@hioa.no
Abstract. A core component of Sentiment Analysis is the generation
of sentiment lists. Label propagation is equivocally one of the most used
approaches for generating sentiment lists based on annotated seed words
in a manual manner. Words which are situated many hops away from the
seed words tend to get low sentiment values. Such inherent property of
the Label Propagation algorithm poses a controversial challenge in sentiment analysis. In this paper, we propose an iterative approach based
on the theory of Active Learning [1] that attempts to remedy to this
problem without any need for additional manual labeling. Our algorithm
is bootstrapped with a limited amount of seeds. Then, at each iteration,
a fixed number of “informative words” are selected as new seeds for labeling according to different criteria that we will elucidate in the paper.
Subsequently, the Label Propagation is retrained in the next iteration
with the additional labeled seeds. A major contribution of this article
is that, unlike the theory of Active Learning that prompts the user for
additional labeling, we generate the additional seeds with an Artificial
Oracle. This is radically different from the main stream of Active Learning Theory that resorts to a human (user) as oracle for labeling those
additional seeds. Consequently, we relieve the user from the cumbersome
task of manual annotation while still achieving a high performance. The
lexicons were evaluated by classifying product and movie reviews. Most
of the generated sentiment lexicons using Active learning perform better
than the Label Propagation algorithm.
Keywords: Sentiment analysis · Label propagation · Active learning
1
Introduction
With the increasing amount of unstructured textual information available on
the Internet, sentiment analysis and opinion mining have recently gained a
groundswell of interest from the research community as well as among practitioners. In general terms, sentiment analysis attempts to automate the classification
of text materials as either expressing positive sentiment or negative sentiment.
Such classification is particularity interesting for making sense of huge amount
c Springer International Publishing Switzerland 2015
⃝
L. Rutkowski et al. (Eds.): ICAISC 2015, Part II, LNAI 9120, pp. 799–810, 2015.
DOI: 10.1007/978-3-319-19369-4_71
800
A. Yazidi et al.
of text information and extracting the ”word of mouth” from product reviews,
and political discussions etc.
Possessing beforehand a sentiment lexicon is a key element in the task of applying sentimental analysis on a phrase or document level. A sentiment lexicon
is merely composed of sentiment words and sentiment phrases (idioms) characterized by sentiment polarity, positive or negative, and by sentimental strength.
For example, the word ’excellent’ has positive polarity and high strength whereas
the word ’good’ is also positive but has a lower strength. Once a lexicon is built
and in place, a range of different approaches can be deployed to classify the
sentiment in a text as positive or negative. These approaches range from simply
computing the difference between the sum of the scores of the positive lexicon
and sum of the scores of the negative lexicon, and subsequently classifying the
sentiment in the text according to the sign of the difference.
In order to generate a sentiment lexicon, the most obvious and naive approach
involves manual generation. Nevertheless, the manual generation is tedious and
time consuming rendering it an impractical task.
Due to the difficulty of manual generation, a significant amount of research
has been dedicated to presenting approaches for automatically building sentiment lexicon. To alleviate the task of lexicon generation, the research community has suggested a myriad of semi-automatic schemes that falls mainly under
two families: dictionary-based family and corpus-based family. Both families are
semi-automatic because the underlying idea is to bootstrap the generation from
a short list of words (seed words), however they differ in the methodology for
iteratively building the lexicon.
The majority of dictionary-based approaches form a graph of the words in a
dictionary, where the words correspond to nodes and where relations between
the words (e.g. in terms of synonyms, antonyms and/or hyponyms) may form
the edges. A limited number of seed words are manually assigned a positive or a
negative sentiment value (or label), and an algorithm, such as the Label Propagation mechanism proposed in [2], is used to automatically assign sentiment
scores to the other non-seed words in the graph.
Label propagation is one of the most used semi-automatic methods for generating sentiment lists based on seed words. A major drawback of Label Propagation in these settings resides in the fact that words which are situated many hops
away from the seed words tend to get low sentiment values. Such phenomenon
poses a controversial challenge in sentiment analysis as words that have high
sentiment values can end up with a low score.
In this paper, we propose an iterative approach to Label Propagation that
is based on the principles of Active Learning theory. Our approach boosts the
propagation process and avoids having labels that get low scores only because
they merely are located many hops away from a seed word. Our work presents
a two-fold contribution. First, to the best of our knowledge, this is the first
reported work in lexicon generation that embraces the theory of Active Learning
as a tool for enhancing the Label Propagation process. Unlike Active Learning,
our approach does not require prompting the user to label some “informative
On Enhancing the Label Propagation Algorithm for Sentiment Analysis
801
examples”, but we will rather do it in an automatic manner using an Artificial
Oracle, thus alleviating the user from the burden of annotation. In fact, the
manual annotation of seeds is a cumbersome and time-consuming task that we
will attempt to minimize. Furthermore, different methods will be presented for
selection procedures of the seed words to be annotated.
Another latent motivation in this article is to investigate the potential of
generating lexicon in an automatic manner without any human intervention
or refinement. We try to achieve this by increasing the sources of information,
namely three different thesauruses, instead of solely relying on a single thesaurus
as commonly done in the literature. In fact, we suggest that by increasing the
number of thesauruses we can increase the quality of the generated lexicon.
Finally, while most of research in the field of sentiment analysis has been
centered on the English language, little work has been reported for smaller languages where there is a shortage of good sentiment lists. In this paper, we tackle
the problem of building sentiment lexicon for a smaller language, and use the
Norwegian language as an example.
1.1
Background Work
The underlying idea of a dictionary-based approach is to build the lexicon based
on a dictionary containing synonyms and antonyms, and possibly hyponyms. A
set of seed words is used to iteratively extend the list by resorting to the synonym and antonym structure. A word-graph is formed where a word corresponds
to a node and a relationship between two words (e.g. a synonym relationship)
corresponds to an edge. The intuitive idea behind the approach is that polarity
of a sentiment word is preserved by a synonym relationship and inverted by an
antonym relationship.
Dictionary-based approaches were introduced by Hu and Liu in their seminal
work [3]. They stop the generation when no more words can be added to the
list. Mohammad, Dunne and Dorr [4] used a rather subtle and elegant enhancement of Hu and Liu’s work [5,3] by exploiting the antonym-generating prefixes
and suffixes in order to include more words in the lexicon. In [6], the authors
constructed an undirected graph based on adjectives in WordNet [7] and define
distance between two words as the shortest path in WordNet. The polarity of
an adjective is then defined as the sign of the difference between its distance
from the word ”bad” and its distance from the word ”good”. While the strength
of the sentiment depends on the later quantity as well as the distance between
words ”bad” and ”good”.
Blair and his colleagues [8] employ a novel bootstrapping idea in order to
counter the effect of neutral words in lexicon generation and thus improve the
quality of the lexicon. The idea is to bootstrap the generation with neutral seed
in addition to a positive and a negative seed. The neutral seeds are used to
avoid positive and negative sentiment propagation through neutral words. The
latter work uses a modified version of the label propagation algorithm proposed
in [2]. The label propagation algorithms have an initial phase: where a score
+1 is assigned to positive seed words and a score −1 to negative seed words,
802
A. Yazidi et al.
and 0 to the rest of words obtained through bootstrapping, then the transfer of
score is performed from the seed to the rest of words using the simple idea that
the sentiment of a word is the average of its neighbor. Note that the scores are
updated in an iterative manner.
Rao and Ravichandran [9] used semi-supervised techniques based on two variants of the Mincut algorithm [10] in order to separate the positive words and
negative words in the graph generated by means of bootstrapping. In simple
words, Mincut algorithms [10] are used in graph theory in order to partition a
graph into two partitions minimizing the number of nodes possessing strong similarity being placed in different partitions. Rao and Ravichandran [9] employed
only the synonym relationship as a similarity metric between two nodes in the
graph. The results are encouraging and show some advantages of using Mincut
over label propagation.
In [11], Hassan and Radev use elements from the theory of random walks of
lexicon generation. The distance between two words in the graph is defined based
on the notion of hitting time. The hitting time h(i|S)is the average number hops
it takes for a node i to reach a node in the set S. A word w is classified as positive
if h(w|S+ ) > h(w|S− ) and vice versa, where S+ denotes the set of positive seed
words, and S− refers to the set of negative seed words.
Kim and Hovy [12] resorts to the Bayesian theory for assigning the most
probable label (here polarity) of a bootstrapped word.
In [13], a hybrid approach was proposed that combines both elements from
corpus based approaches and dictionary based approaches. In fact, the bootstrapping is done based on a dictionary while the score assigning is based on
corpus.
It is worth mentioning that another research direction for building lexicon
for foreign language is based on exploiting the well-developed English sentiment
lexicon. A representative work of such approaches is reported in [14].
1.2
Active Learning Theory
The merit of Active Learning [1] manifests in situations where there is an abundant amount of data to be labeled, while the amount of available labeled data
is scarce. Usually, in normal supervised learning situations, we have enough
labels in order to build a classifier that has good classification performance.
Thus, the theory of Active Learning attempts to solve this problem by proposing ways to intelligently choose very few informative data points to label. Then
the user/human is prompted to return labels for these carefully chosen data
points. In this sense, Active Learning idea identifies some data points whose labeling can present a high value for the classification and then prompts the user
to label them. A lot of work in the field of Active Learning is concerned with
algorithms for devising schemes for selecting the data points to be annotated.
Without loss of generality, we claim that the closest Active Learning approach
to our work [1] is “Query by committee” where the labels that the committee
disagrees most are chosen for manual labeling.
On Enhancing the Label Propagation Algorithm for Sentiment Analysis
803
In Active Learning, a selection procedure is used in order to select the informative examples to be labeled, and then the user is actively asked for annotation.
Thus, the user plays the role of an oracle for labeling.
In this article, the oracle is not a human but rather an Artificial Oracle (i.e,
algorithm). Therefore, we relieve the human user from the manual task of annotation. We use the term Artificial Oracle for describing the oracle in order to
emphasize the difference between the user as oracle and an algorithm as oracle. Despite that relying on the Artificial Oracle for annotating the informative
examples introduces some inaccuracy, we were still able to get good performance.
2
The Novel Approach
A major shortcoming of the Label Propagation algorithm [2] is that while the
seed values are propagated throughout the word graph of synonyms, the algorithm does not take into account that for each edge between two synonyms,
there is an introduction of inaccuracy. The source of inaccuracy is primarily
that the meaning of two synonyms is typically not overlapping. A synonym of
a word might carry a different and skewed meaning and sentiment compared to
the meaning and sentiment of the word. Thus, the sentiment value of a word
is not a 100% trustworthy indicator of the value that should be propagated to
its synonyms. Problems are typically observed in the middle area between the
positive and the negative seed words, i.e. typically several hops away from any
seed word. While other works have introduced fixes to this problem (e.g. by introducing neutral seed words as described above), we take a different approach
to the solution.
To account for this effect, we propose a novel approach that combines the
ideas from Label Propagation and the Active Learning, and that focuses on
the shortest path between a word and the seed words. This is described in the
following section.
2.1
Description of the Main Algorithm
Our Active Learning base algorithm consists of four phases, which are executed
iteratively for a fixed number of iterations:
– Running the label propagation
– Selecting the Top L lowest absolute values of the scores.
– Among the top Top L values, apply a Selection Algorithm in order to select
a subset of K informative words for automatic labeling
– Annotate the later subset of informative words using Annotation Based
Graph Voting Procedure
The formal algorithm, which puts all the pieces of this puzzle together, follows
in Algorithm 1.
The core elements of our algorithm are delineated in the following two sections.
In section 2.2, we will describe the Annotation Based Graph Voting (ABGV)
procedure and in section 2.3 the different variants of Selection Procedure.
804
A. Yazidi et al.
Algorithm 1. Main Algorithm
Input:
G = (V, E), is the graph built from the dictionaries.
S = {s ∈ V such that s is seed}
W = {w ∈ V such that w is not a seed word}.
Finite number of iterations N .
Output:
Score(S) = {score(w) such that w ∈ W}.
Method:
1: for iteration ≤ N do
2:
Apply Label Propagation on G
3:
Identify Z = {w ∈ W such that |score(w)| is among the Top L lowest values}.
′
4:
Apply a Selection Procedure to select a subset Z ⊂ Z.
5:
for each w in Z ′ do
6:
if Vp (w) ≥ VN (w) (Annotation Based Graph Voting Procedure) then
7:
score(w) = +5
8:
else
9:
score(w) = −5
10:
end if
11:
end for
12: end for
End Main Algorithm
2.2
The Annotation Based Graph Voting Procedure
The Annotation Based Graph Voting (ABGC) Procedure is responsible for annotating a word w that is selected by the Selection Procedure.
For a given word selected by the Selection Procedure to be a seed word, the
oracle annotates it by considering what we define as positive and negative votes
from all the seed words. To vote focuses on the shortest path between a word
and a seed word and accounts for the number of antonyms along that path.
We shall now define what is meant by the term vote.
Let V(s → w) be the vote of seed s to a word w.
– V(s → w) = +1, i.e, positive vote from seed s to word w, if one of the
following two cases takes place:
• if score(s) > 0 and number of antonyms along the shortest path from s
to w in the graph G is even.
• or score(s) < 0 and number of antonyms along the shortest path is odd.
– V(s → w) = −1, i.e, negative vote from seed s to word w, if one of the
following two cases is valid:
• score(s) < 0 and number of antonyms along the shortest path is even.
• or score(s) > 0 and and number of antonyms along the shortest path is
odd.
We define VP (w) as the total number of positive votes to word w, or formally
as the cardinality of the set {s ∈ S such that V(s → w) = +1}. Similarly, we
On Enhancing the Label Propagation Algorithm for Sentiment Analysis
805
define VN (w) as the total number of negative votes to word w, or formally as
the cardinality
! of the set {s ∈ S such that V(s → w) = −1}.
Vp (w) = s∈S I(V(s!→ w) = −1)
Similarly, VN (w) = w∈S I(V(s → w) = +1)
Where I(·) is the indicator function.
The idea behind the voting procedure is that an even number of antonyms
along the shortest path from a seed to a word would preserves the sentiment
polarity since antonym of antonym is synonym. Similarly, an odd number of
antonyms along the shortest path from a seed to a word would invert the sentiment polarity.
Artificial Oracle: The informed reader should note that a crucial element of our
approach is that we do not prompt the user (human) to label the words returned
by the Selection Procedure, but we rather develop a simple and intuitive method
for automatically annotating these words without human intervention using an
Artificial Oracle. The motivation behind the automatic annotation is based on
the concept of voting.
2.3
The Selection Procedure
The Selection Procedure is responsible for choosing at each iteration a set of
informative words to be annotated. This is achieved using a set of approaches
or variants that we shall describe in this section.
At each iteration, the Selection Procedure selects K words for automatic annotation from Z = {w ∈ W such that |score(w)| is among the Top L lowest values}.
However, it is difficult to judge whenever a word gets a score around 0, whether
it is neutral word or just far away from the seed. In order to counter this, we
develop range of different strategies to decide this, as outlined in the following.
Purity Based Selection. Low certainty is measured using the notion of Purity,
and captures the case where the number of positive votes is significantly larger
than the number negative votes or vice-versa. Let us define the entropy of a word
of w in the graph as:
H(w) = −(pP (w) log pP (w) + pN (w) log pN (w))
where pP (w) = VP (w)/(VP (w)+VN (w)) and pN (w) = VN (w)/(VP (w)+VN (w)).
The Purity is defined as P (w) = 1 − H(w).
Centrality Based Approach. We select the Top K most central words for
automatic annotation from L lowest values based on two definition of centrality:
– High betweenness centrality: captures the fact that a node lies in the path of
a high number of shortest pathes in the whole graph. Betweeness centrality
can be useful for many algorithms in order to locate nodes that are important
for vehiculating some sort of information between the different nodes in the
806
A. Yazidi et al.
graph, where we suppose that this information flows through the shortest
path.
– High Closeness Centrality: captures the concept that a node might be important locally within its own cluster since it is connected to many nodes in
the cluster. Thus, a node that is near to the center of local clusters will have
high centrality.
Weighted Centrality and Purity Based Approach. We define a weighted
function that is a compromise between two of the aforementioned Selection Procedures, namely high closeness centrality and high purity. At each iteration, the
Selection Procedure selects K words for automatic annotation from the Top L
lowest absolute values of the scores that have the following highest score:
β1 CC(w) + β2 P (w)
(1)
where CC(w) defines the closeness centrality while P (w) the purity, and β1 , β2
are normalized weights . Intuitively, we try to find a balance between choosing
the locally central nodes, and the nodes that have high purity, which might
indicate a strong sentiment strength.
Other strategies. In addition to strategies above we also consider the following
two simple strategies
– We select the nodes which are closest to zero among the lowest L scores
around 0, thus the nodes with the lowest value in absolute value in the
whole graph.
– We sample randomly a subset from the pool of the lowest L scores around 0.
3
3.1
Linguistic Resources and Benchmark Lexicons for
Evaluation
Building a Word Graph From Thesauruses
We built a large undirected graph of synonym and antonym relations between
words from the three Norwegian thesauruses [15]. The words were nodes in the
graph and synonym and antonym relations were edges. The full graph consisted
of a total of 6036 nodes (words) and 16475 edges.
For all lexicons generated from the word graph, we started with a set of 109
seed words (51 positive and 57 negative). The words were manually selected
where all used frequently in the Norwegian language and spanned different dimensions of positive (’happy’, ’clever’, ’intelligent’, ’love’ etc.) and negative sentiment (’lazy’, ’aggressive’, ’hopeless’, ’chaotic’ etc.).
On Enhancing the Label Propagation Algorithm for Sentiment Analysis
3.2
807
Benchmark Lexicons
We generated sentiment lexicons from the word graph using the Label Propagation algorithm [2] which is the most popular method to generate sentiment
lexicons.
As a complement to the graph-generated lexicons, we generated a benchmark sentiment lexicon by translating the well-known English sentiment lexicon
AFINN [16] to Norwegian using machine translation (Google translate). We also
generated a second lexicon by manually checking and correcting several different
errors from the machine translation.
3.3
Text Resources for Evaluation
We tested the quality of the created sentiment lexicons using 15118 product
reviews from the Norwegian online shopping sites www.komplett.no, mpx.no
and 4149 movie reviews from www.filmweb.no. Each product review contained
a rating from one to five, five being the best and the movie reviews a rating from
one to six, six being the best.
4
Evaluation of Novel Approach Against Benchmark
Lexicons
We generated a sentiment lexicon by applying our novel approach in Section 2
to the same word graph used for the Label Propagation benchmark lexicon, and
by using the same seed words.
For each lexicon, we computed the sentiment score of a review by simply
adding the score of each sentiment word in a sentiment lexicon together, which
is the most common way to do it [17]. If the sentiment shifter ’not’ (’ikke’) was
one or two words in front of a sentiment word, sentiment score was switched.
E.g. ’happy’ (’glad’) is given sentient score 0.8, while ’not happy’ (’ikke glad’)
is given score −0.8. Finally, the sum is divided by the number of words in the
review, giving us the final sentiment score for the review. We also considered
other sentiment shifter, e.g. ’never’ (’aldri’), and other distance between sentiment word and shifter, but our approach seems to be the best for such lexicon
approaches in Norwegian [18].
Classification performance. We evaluated the classification performance using average difference in absolute value between the true and predicted rating
based on sentiment lexicons. For details, see [15].
5
Results
This section presents the results of classification performance on product and
movie reviews for different sentiment lexicons. The results are shown in Tables 1
808
A. Yazidi et al.
Table 1. Classification performance for sentiment lexicons on komplett.no and mpx.no
product reviews. The number of words automatically annotated si 40. The columns
from left to right show the sentiment lexicon names, the number of words in the sentiment lexicons, the mean absolute error with standard deviation and the 95% confidence
intervals for mean absolute error.
N Mean (Stdev) 95% conf.int.
AFINN
6036
1.43 (1.41)
(1.4, 1.47)
6036
1.47 (1.39)
(1.44, 1.5)
AFINN M
ZERO
6036
1.49 (1.38) (1.46, 1.52)
BETWEENESS 6036
1.51 (1.42) (1.48, 1.54)
6036
1.52 (1.4) (1.49, 1.55)
PC 0.5 0.5
CLOSENESS 6036
1.53 (1.38)
(1.5, 1.56)
6036
1.56 (1.41)
(1.53, 1.6)
PC 0.7 0.3
PURITY
6036
1.57 (1.44)
(1.54, 1.6)
SAMPLING
6036
1.58 (1.46) (1.55, 1.61)
6036
1.61 (1.46) (1.58, 1.64)
PC 0.3 0.7
LABEL
6036
1.69 (1.51) (1.66, 1.72)
and 2. AFINN and AFINN M refer to the translated and manually adjusted
AFINN sentiment lists. LABEL refers to the Label propagation algorithm. ZERO
refers to closest to zero, PC refers to the weighting between purity and centrality.
Further CLOSENESS and BETWEENESS, refer to closeness and betweenness
centrality and finally SAMPLING refers to random sampling of nodes to be
annotated. Training and test sets were created by randomly adding an equal
amount of reviews to both sets. All sentiment lexicons were trained and tested
on the same training and test sets, making comparisons easier. This procedure
were also repeated several times and every time the results were in practice
identical to the results in Tables 1 and 2, documenting that the results are
independent of which reviews that were added to the training and test sets.
For both review types we observe some variations in classification performance
ranging from 1.43 to 1.69 for product reviews and from 1.87 to 2.08 for movie
reviews. Comparing Tables 1 and 2, we see that the classification performance is
poorer for movie reviews than for product reviews. It is known from the literature
that sentiment analysis of movie reviews is normally harder than product reviews
[17]. E.g. movie reviews typically contain a summary of the plot of the movie
that could contain many negative sentiment words (sad movie), but still the
movie can get an excellent rating.
We see that most of the active learning strategies performs better than the
Label propagation algorithm. Among the active learning approaches, we see that
the ZERO strategy performs the best indicating that detecting nodes that are
close to zero is an effective strategy to build reliable sentiment lists. The PURITY
strategy seems to perform poorer than the BETWEENESS and CLOSENESS
centrality strategies. The automatically generated sentiment lexicons are also
competitive to the translated AFINN lists. We also considered cases where the
number of automatically annotated words was 30 and 50 and the results were
similar to the results above.
On Enhancing the Label Propagation Algorithm for Sentiment Analysis
809
Table 2. Classification performance for sentiment lexicons on filmweb.no movie reviews. The number of words automatically annotated is 40. The columns from left to
right show the sentiment lexicon names, the number of words in the sentiment lexicons,
mean absolute error with standard deviation and 95% confidence intervals for mean
absolute error.
N Mean (Stdev) 95% conf.int.
ZERO
6036
1.87 (1.1) (1.82, 1.91)
6036
1.88 (1.14) (1.84, 1.93)
PC 0.7 0.3
SAMPLING
6036
1.94 (1.11) (1.89, 1.99)
6036
1.95 (1.13)
(1.9, 2)
PC 0.5 0.5
CLOSENESS 6036
1.96 (1.1)
(1.91, 2)
AFINN
6036
2.02 (1.12) (1.97, 2.07)
6036
2.04 (1.1) (1.99, 2.08)
PC 0.3 0.7
6036
2.04 (1.17) (1.99, 2.09)
AFINN M
LABEL
6036
2.05 (1.1)
(2, 2.1)
BETWEENESS 6036
2.07 (1.12) (2.02, 2.11)
PURITY
6036
2.08 (1.11) (2.04, 2.13)
6
Conclusion
In this paper, we have shown that an approach inspired by Active Learning
theory offer a new promising alternative for overcoming the limitations of Label
Propagation. In fact, we propose different methods for intelligently selecting seed
words for labeling. In addition, we alleviate the user from manual annotation
of the selected seed words by introducing an Artificial Oracle. Comprehensive
simulation results demonstrate the feasibility of our approach. As a future work,
we would like to compare Active Learning based on human annotation with our
Artificial Oracle based approach.
References
1. Settles, B.: Active learning. Synthesis Lectures on Artificial Intelligence and Machine Learning 6(1), 1–114 (2012)
2. Zhu, X., Ghahramani, Z.: Learning from labeled and unlabeled data with label
propagation. Technical report, Technical Report CMU-CALD-02-107, Carnegie
Mellon University (2002)
3. Hu, M., Liu, B.: Mining opinion features in customer reviews. In: Proceedings of
AAAI, pp. 755–760 (2004)
4. Mohammad, S., Dunne, C., Dorr, B.: Generating high-coverage semantic orientation lexicons from overtly marked words and a thesaurus. In: Proceedings of the
2009 Conference on Empirical Methods in Natural Language Processing, vol. 2,
pp. 599–608. Association for Computational Linguistics (2009)
5. Hu, M., Liu, B.: Mining and summarizing customer reviews. In: Proceedings of
the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD),
pp. 168–177 (2004)
6. Kamps, J., Marx, M., Mokken, R.J., De Rijke, M.: Using wordnet to measure
semantic orientations of adjectives (2004)
810
A. Yazidi et al.
7. Miller, G.A.: Wordnet: a lexical database for english. Communications of the
ACM 38(11), 39–41 (1995)
8. Blair-Goldensohn, S., Hannan, K., McDonald, R., Neylon, T., Reis, G.A., Reynar,
J.: Building a sentiment summarizer for local service reviews. In: WWW Workshop
on NLP in the Information Explosion Era, p. 14 (2008)
9. Rao, D., Ravichandran, D.: Proceedings of the 12th Conference of the European
Chapter of the Association for Computational Linguistics. In: Association for
Computational Linguistics, pp. 675–682 (2009)
10. Blum, A., Lafferty, J., Rwebangira, M.R., Reddy, R.: Semi-supervised learning using randomized mincuts. In: Proceedings of the twenty-first International
Conference on Machine Learning, p. 13. ACM (2004)
11. Hassan, A., Radev, D.: Identifying text polarity using random walks. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics,
pp. 395–403. Association for Computational Linguistics (2010)
12. Kim, S.M., Hovy, E.: Automatic identification of pro and con reasons in online
reviews. In: Proceedings of the COLING/ACL on Main Conference Poster Sessions,
pp. 483–490. Association for Computational Linguistics (2006)
13. Peng, W., Park, D.H.: Generate adjective sentiment dictionary for social media
sentiment analysis using constrained nonnegative matrix factorization. Urbana 51,
61801 (2004)
14. Hassan, A., Abu-Jbara, A., Jha, R., Radev, D.: Identifying the semantic orientation
of foreign words. In: Proceedings of the 49th Annual Meeting of the Association for
Computational Linguistics: Human Language Technologies: Short Papers-Volume
2, pp. 592–597. Association for Computational Linguistics (2011)
15. Hammer, H., Bai, A., Yazidi, A., Engelstad, P.: Building sentiment lexicons applying graph theory on information from three Norwegian thesauruses. In: Norweian
Informatics Conference (2014)
16. Nielsen, F.Å.: A new ANEW: Evaluation of a word list for sentiment analysis in
microblogs. CoRR abs/1103.2903 (2011)
17. Bing, L.: Web Data Mining. Exploring Hyperlinks, Contents, and Usage Data.
Springer (2011)
18. Hammer, H.L., Solberg, P.E., Øvrelid, L.: Sentiment classification of online political discussions: a comparison of a word-based and dependency-based method. In:
Proceedings of the 5th Workshop on Computational Approaches to Subjectivity,
Sentiment and Social Media Analysis, pp. 90–96. Association for Computational
Linguistics (2014)