Automated detection of offensive language behavior
on social networking sites
Baptist Vandersmissen
Promotoren: prof. dr. ir. Filip De Turck, dr. ir. Tim Wauters
Begeleiders: Philip Leroux, ir. Johannes Deleu, Joost Roelandts (Massive Media)
Masterproef ingediend tot het behalen van de academische graad van
Master in de ingenieurswetenschappen: computerwetenschappen
Vakgroep Informatietechnologie
Voorzitter: prof. dr. ir. Daniël De Zutter
Faculteit Ingenieurswetenschappen en Architectuur
Academiejaar 2011-2012
Automated detection of offensive language behavior
on social networking sites
Baptist Vandersmissen
Promotoren: prof. dr. ir. Filip De Turck, dr. ir. Tim Wauters
Begeleiders: Philip Leroux, ir. Johannes Deleu, Joost Roelandts (Massive Media)
Masterproef ingediend tot het behalen van de academische graad van
Master in de ingenieurswetenschappen: computerwetenschappen
Vakgroep Informatietechnologie
Voorzitter: prof. dr. ir. Daniël De Zutter
Faculteit Ingenieurswetenschappen en Architectuur
Academiejaar 2011-2012
Acknowledgments
First of all, I would like to express great gratitude to my supervisors Philip Leroux and
ir. Johannes Deleu for their many insights, continuous support and patience. Also dr.
Thomas Demeester should be noticed for his contributions to this study.
I thank Massive Media for providing me with data of Netlog.com
I also want to thank my dear friends at university, Ruben Verhack and Sacha Vanhecke for these marvelous and intense five years. As I want to thank my friends and
family in general for their support and contagious enthusiasm.
iii
Permission for Use of Content
“The author gives permission to make this master dissertation available for consultation
and to copy parts of this master dissertation for personal use. In the case of any other
use, the limitations of the copyright have to be respected, in particular with regard
to the obligation to state expressly the source when quoting results from this master
dissertation.”
Baptist Vandersmissen, June 2012
Automated detection of offensive language behavior on
social networking sites
Baptist Vandersmissen
Promotoren: prof. dr. ir. Filip De Turck, dr. ir. Tim Wauters
Begeleiders: Philip Leroux, ir. Johannes Deleu, Joost Roelandts (Massive Media)
Scriptie ingediend tot het behalen van de academische graad van
burgerlijk ingenieur in de computerwetenschappen
Vakgroep Informatietechnologie
Voorzitter: prof. dr. ir. Daniël De Zutter
Faculteit Ingenieurswetenschappen en Architectuur
Academiejaar 2011-2012
Resume
This study applies machine learning techniques to perform automated offensive language
detection. A corpus, originating from the Dutch distribution of the social network Netlog,
is used. An Information Retrieval system, enhanced with Rocchio Query Expansion,
is developed to effectively find offensive messages. Subsequently, a Naive Bayes and
Support Vector Machine classifier are implemented and trained on the gathered offensive
(and irrelevant) messages. To increase reliability, a third classifier is created that is based
on word lists. Since each classifier has its own strengths and weaknesses, a combination
of different methods is proposed. The combination of seperate classifiers outperforms all
other methods.
Keywords: offensive language detection, query expansion, text classification
Abstract
Social Networking Sites are booming as never before. Apart from the numerous new
opportunities that are provided, also hazards such as messages containing sexual harrasment or racist attacks have to be taken into account. Since manually monitoring and
analysing all messages seperately is unattainable, solutions using automated methods
are sought.
This study applies machine learning techniques to perform automated offensive language
detection. Offensive language can be defined as “expressing extreme subjectivity“ and
this study mainly focuses on two categories ’sexual’ and ’racist’. A corpus, originating
from the Dutch distribution of the social network Netlog, is used and contains over seven
million blog messages. We note that only a very small amount (approximately 0.85%)
of these blog messages can be defined as messages that contain abusive language.
Initially, the intention is to implement two supervised learning methods Naive Bayes
and Support Vector Machine. These methods base the classification of a message on
previous experiences, derived from a labeled training set. To build such training set
offensive messages should be efficiently extracted out of the corpus. In order to achieve
this, an information retrieval system, expanded with a query expansion technique, is
applied. A query containing offensive terms delivers offensive messages, however a more
efficient approach is considered by enhancing the query using Rocchio query expansion.
This study shows that using query expansion can effectively increase the amount of relevant messages retrieved.
These supervised classifiers are trained on the labeled set and afterwards their performance is tested on an independant validation set. The Naive Bayes classifier does
not perform well on the validation set and is therefore disregarded in the further analysis. Our Support Vector Machine implementation achieves results of approximately 69%
precision and 62% recall. However, these results are obtained by ignoring very small
messages, since SVM has difficulties classifying messages that do not contain much information.
To tackle the issues SVM suffers from a more reliable, but less dynamic method is
designed, based on word lists. This method, that is named a semantic classifier, obtains
ix
reasonable results on the validation set with a recall of 93%, but more importantly is
highly complementary with the SVM classifier. The classification of a message is eventually performed by choosing the correct classifier depending on the situation and context.
This method outperforms all others and achieves a precision of 100% combined with a
79% recall.
Notwithstanding the solid results of our classifier on the validation set, we note that
offensive language detection is a challenging domain that, as many text classification
problems, suffers from specific linguistic characteristics. Automated systems can be an
excellent aid, but human capacity to properly estimate the real meaning of a message
is, still, irreplaceable.
Samenvatting
Sociale netwerksites zijn brandend actueel en alomtegenwoordig. Ondanks de talloze
nieuwe mogelijkheden zijn er ook risico’s aan verbonden. Deze komen tot uiting in
de vorm van gebruikers die het medium trachten te misbruiken. Het verspreiden van
ongepaste, aanstootgevende of kwetsende boodschappen is een vaak voorkomend probleem op elk digitaal communicatie middel. Aangezien het niet realistisch is om elk bericht
manueel te controleren, worden er automatische oplossingen gezocht.
Deze studie probeert aan de hand van technieken gelinkt aan machinaal leren op automatische wijze kwetsende of aanstootgevende taal te detecteren. Beledigend taalgebruik kan gedefinieerd worden als ”het uiten van extreme subjectiviteit“. Deze studie
richt zich voornamelijk op berichten die enerzijds seksueel getinte, ongepaste inhoud of
anderzijds racistisch getinte boodschappen bevatten. Hiervoor wordt een berichtenverzameling gebruikt, afkomstig van de Nederlandstalige distributie van de sociale netwerksite
Netlog, dewelke meer dan zeven miljoen blogberichten telt. Er wordt echter opgemerkt
dat slechts 0.85% van deze berichten ook daadwerkelijk ongepaste taal bevat.
Initieel was het de bedoeling om twee gesuperviseerde leermethodes te ontwikkelen, die
enerzijds gebaseerd zijn op Naı̈ve Bayes en anderzijds op een Support Vector Machine.
Deze methoden baseren de classificatie van een bericht op vorige ervaringen, afgeleid van
een gekende trainingset. Een trainingset wordt opgebouwd aan de hand van positieve
en negatieve voorbeelden. Er moeten met andere woorden beschrijvende voorbeeldberichten verzameld worden uit de complete verzameling. Om dit te bekomen is er een
zoeksysteem geı̈mplementeerd in combinatie met een techniek die zoekopdrachten tracht
te verbeteren. Relevante berichten worden in de berichtenverzameling gezocht door het
invoeren van zoekopdrachten in het systeem. Deze zoekopdrachten worden op hun beurt
uitgebreid door Rocchio’s query expansie techniek. Aan de hand van het gebruik van
een query expansie techniek toont deze studie aan dat, op een efficiëntere manier meer
relevante berichten kunnen verkregen worden.
Hierna worden beide gesuperviseerde classifiers getraind op de gelabelde berichtenverzameling om dan vervolgens hun algemene prestaties te testen op een validatieverzameling.
Aangezien de Naı̈ve Bayes classifier eerder ondermaats presteert, laten we deze buiten
beschouwing. De Support Vector Machine behaalt hogere resultaten met een precisie
xi
van ongeveer 69% en een recall van 62%. Deze resultaten zijn echter bekomen door te
kleine berichten, die niet tenminste vijf verschillende woorden bezitten, te negeren. Dit
verbetert het resultaat sterk aangezien SVM moeilijk overweg kan met kleine berichten.
Om de verschillende problemen van de SVM methode aan te pakken is er ook een derde
methode ontwikkeld die berichten classificeert aan de hand van vooraf gedefinieerde
woordenlijsten. Deze semantische methode behaalt redelijke resultaten op de validatieverzameling met een zeer hoge recall van 93%, en is daarbovenop zeer complementair
met de SVM classifier. De uiteindelijke classificatie van een bericht is dan bepaald door
een combinatie van beide methoden. Afhankelijk van de situatie wordt voor de ene of
de andere classifier gekozen. Met dit systeem behalen we een precisie van 100% en een
recall van 79% op de validatieverzameling.
Desalniettemin de uitstekende resultaten op de validatieverzameling, merken we op dat
deze verzameling geen perfecte representatie van de gehele berichtenverzameling is. Het
detecteren van ongepaste taal is een uitdagend domein dat nog zeer sterk zal moeten
evolueren om de moeilijke en typische karakteristieken van taal te doorgronden.
Contents
Acknowledgments
iii
Preface
1
1 Introduction: What is Offensive
1.1 Introduction . . . . . . . . . . .
1.2 Goals of This Study . . . . . .
1.3 Approach . . . . . . . . . . . .
1.4 Applications . . . . . . . . . . .
1.4.1 Query Expansion . . . .
1.4.2 Text Categorisation . .
1.5 Challenges . . . . . . . . . . . .
1.6 Related Work . . . . . . . . . .
2 The
2.1
2.2
2.3
2.4
2.5
Language Detection?
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
. . . . . . . . . . . . . .
Dutch Netlog Corpus
Netlog . . . . . . . . . . . . . . . . . . . . .
Corpus Overview . . . . . . . . . . . . . . .
Offensive language . . . . . . . . . . . . . .
Creating Training and Validation set . . . .
2.4.1 Labels . . . . . . . . . . . . . . . . .
2.4.2 Validation set . . . . . . . . . . . . .
2.4.3 Training set . . . . . . . . . . . . . .
Informal Language and Multiple Languages
3 Methodology
3.1 Information Retrieval . . . .
3.1.1 Features . . . . . . . .
3.1.2 Evaluation . . . . . .
3.1.3 Spelling Correction . .
3.1.4 Stemming . . . . . . .
3.1.5 Part-of-Speech Tagger
3.1.6 Query Expansion . . .
3.1.7 Open Source Software
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
xiii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
3
5
6
6
6
7
7
8
.
.
.
.
.
.
.
.
11
11
11
12
13
13
14
15
16
.
.
.
.
.
.
.
.
17
17
18
18
19
19
20
20
26
3.2
Machine Learning Techniques . .
3.2.1 Data Representation . . .
3.2.2 Feature Selection . . . . .
3.2.3 Classification Techniques
Lexicon Based Text Classification
Evaluation . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
28
29
29
30
33
33
4 Design and Implementation
4.1 Information Retrieval . . . . . .
4.1.1 Query Expansion . . . . .
4.2 Offensive Language Detection . .
4.2.1 Naive Bayes . . . . . . . .
4.2.2 Support Vector Machine .
4.2.3 Semantic Methods . . . .
4.2.4 Combination of Methods
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
35
35
37
37
38
38
40
3.3
3.4
5 Results
5.1 Query Expansion . . . . . . . . . . . .
5.1.1 Rocchio Relevance Feedback .
5.1.2 Conclusion . . . . . . . . . . .
5.2 Offensive Language Detection . . . . .
5.2.1 Naive Bayes . . . . . . . . . . .
5.2.2 Support Vector Machine . . . .
5.2.3 Semantic Classifier . . . . . . .
5.2.4 Comparison of Single Methods
5.2.5 Combined Algorithm . . . . . .
5.2.6 Corpus Classification . . . . . .
5.2.7 Conclusion . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
43
44
49
49
49
50
56
58
60
61
62
6 Discussion
6.1 Main Findings . . . . . . .
6.1.1 Corpus . . . . . . .
6.1.2 Development . . . .
6.1.3 Experimental Study
6.2 Discussion . . . . . . . . . .
6.2.1 Domain-dependency
6.2.2 Implicit Language .
6.2.3 Conclusion . . . . .
6.3 Future Improvements . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
63
63
63
63
64
65
66
66
66
66
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7 Conclusion
69
7.1 Main Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Preface
“On average 37% of the people talk to people more online than they do in
real life.”
A survey of Badoo in the U.K., U.S. and Germany
Since the beginning of the internet it has always been a goal to create forms of
computer-mediated social interaction. The first type of social networking sites (SNS)
came in the form of online communities such as Theglobe.com and Tripod.com. They
mostly focused on creating interaction between different people by bringing them together in chat rooms. It is only in the late 1990s that many sites began to incorporate
more advanced features and user profiles became the central point of view [56]. This
somewhat newer form of social networking sites began to flourish with the rise of SixDegrees.com in 1997 [4]. Followed by many other such as Friendster and for example the
European Netlog, formerly known as Facebox. It is since then that the popularity of
social networking sites has rapidly been increasing up until today. Many of these pioneering sites such as SixDegrees.com and Friendster however died a silent death or lost
a great amount of users to the more modern SNS.
Social networking sites have become widespread around the global population. Nowadays a staggering 900 million users are monthly active on the networking site Facebook
1 . This means around one person on eight in this world has a facebook profile. Beside
Facebook there are off course many more social networking sites. Qzone for example is a
Chinese SNS and has a user base around 531 million [51]. Except from the most popular
main stream SNSs a numerous smaller niche social network sites exist. The majority of
these SNSs in their turn can also present increasing visitor numbers. Facebook is by far
the most popular SNS but for others the popularity strongly depends on the specific features and geographical location. Other cultures will for example expect more privacy etc.
Social networking sites have long been labeled as unsustainable and have been predicted
to defunct sooner than later. Often the reason to describe SNSs more as an internet hype
or bubble was because the user base mostly consists out of young and “unpredictable”
people. On top the SNSs did not produce any essential value except from staying in con1
Facebook Statistics, http://newsroom.fb.com/content/default.aspx?NewsAreaId=22, [2012-05-04]
1
Automated Detection of Offensive Language Behavior on Social Networking Sites
tact with old friends or sharing pictures. These predictions however have slowly faded
away as the importance and value of these sites continued to grow. Nowadays almost
all major companies are present on one or more social networking sites. Even marketing
strategies and other important decisions are often influenced by the social media.
The fact that a whole world is accessible behind a computer is one of the greatest
and key aspects of the internet. But at the same time creates a safe haven for criminals
and those with less nobel goals. It is obvious that these digital environment pose many
threads, especially for younger users. A study, done by ScanSafe, shows that up to 80%
of blogs contain offensive language [71]. Offensive language has spread into almost every
corner of online communities. Well known issues like bullying, harassment, exposure
to harmful content, sexual grooming and racist attacks are important problems that
directly and indirectly affect our mental health.
To be able to control such digital environments tremendous human efforts have to be
made. For example every twenty minutes 1,587,000 blog posts and 10,208,000 comments
are posted on Facebook 2 . It is here that this study will present an automatic method
to detect offensive or abusive language.
We begin with an introduction of what offensive language detection exactly entails in
chapter one. Moreover, we describe the challenges associated with this domain and our
approach to this study. Chapter two gives an overview of the Netlog data corpus we will
work with. In chapter three the available techniques that are interesting in regard to
this study are discussed. Which techniques we used and how they are implemented can
be found in chapter four, while the fifth chapter reports on the results of the experiments
and the evolution of the whole study. Chapter six discusses the main findings of this
study, and makes some proposals for future work. Finally Chapter seven summarizes
the conclusions.
2
Obsessed with Facebook, http://www.onlineschools.org/blog/facebook-obsession/, [2012-05-01]
2
Chapter 1
Introduction: What is Offensive
Language Detection?
1.1
Introduction
What is Automated Detection?
Automated means “operating with minimal human intervention; independent of external
control” 1 . Automated Detection is the operation of detecting matters with no or minimal human intervention in a controlled environment. Terms directly associated with
automated learning are machine learning and supervised learning. Machine learning, a part of the artificial intellegence branch, is a scientific discipline dealing with the
design and development of algorithms that allow machines to evolve behavior based on
experience. Supervised learning is the machine learning task of deducting a function
from labeled (supervised) training data [24].
What is Offensive Language Behavior?
Offensive
2
can be described as:
− causing anger or annoyance; ’offensive remarks’
− causing or capable of causing harm
− exhibiting lack of respect; rude and discourteous
In this study offensive language is defined as the propagation of offensive messages or
remarks that in some circumstances are inappropriate, exhibit a lack of respect towards
certain groups of people or are just rude in general. In literature offensive language
is often expressed as ’flame’. [2] defines flames as ’exhibiting extreme subjectivity’.
However, offensive language (flames) is a very vague and ambigious concept. In general
1
2
Definition on www.thefreedictionary.com, [2012-05-02]
idem
3
Automated Detection of Offensive Language Behavior on Social Networking Sites
people describe or experience certain events or messages in different ways according to
their own education, culture and personal experience. Therefore it is very important to
accurately define what we describe as being offensive language.
Our final goal is not only to detect offensive language but to be able to discover offensive language behavior. Offensive language behavior can be described as a person - a
user of a social networking site - repeatedly propagating offensive language in a certain
time interval.
Offensive language is not only a vague concept but is also exceptionally wide. Next
subjects can all be interpreted as offensive or at least cause certain nuisance:
− Messages containing unwanted advertisement or plain spam.
− Scammers trying to steal personal information.
− Sexually inappropriate messages or indecent proposals
− Racist messages: offending certain people or entire groups (aggression against
some culture, subgroup of the society, race or ideology in a tirade)
− ...
Because different categories often require different strategies, we decided to focus on just
two categories.
As stated above harassment, sexual grooming and racist attacks are important as
they are widespread and can do more harm to younger child than spam. Therefore, we
focus on the detection of sexually inappropriate and racist messages. However, we
intend to build a system that with some effort can be expanded in a flexible way.
What is a Social Networking Site?
In [4] a social networking site (SNS) is defined as: a web-based service that allows
individuals to
i Construct a public or semi-public profile within a bounded system.
ii Articulate a list of other users with whom they share a connection.
iii View and traverse their list of connections and those made by others within the
system.
The exact meaning of the above terms off course differ from site to site and are
relatively flexible. Holding this definition in mind several hundreds social networking
sites nowadays exist and are widespread around the global population.
For this study we made use of the Dutch Netlog corpus (cf. chapter 2).
4
Automated Detection of Offensive Language Behavior on Social Networking Sites
1.2
Goals of This Study
In this study we attempt to create a machine driven solution to be able to automatically
detect offensive language behavior on social networking sites. The final goal is thus to
develop a system that can automatically detect profiles/persons that extensively and deliberately post messages containing offensive language (cf. section 2.3). We can briefly
describe our study by stating we are dealing with a text classification problem. In the
next chapter (cf. chapter 2) we give an overview of what we mean with offensive language. As mentioned before we focus on two main categories being ’sexual’ and ’racist’
messages.
The main objective is however divided in multiple smaller parts. The first objective
focuses on finding and annotating a relevant subset of messages. As we deal with
a rather large message set, it is key to efficiently extract relevant messages. In our case
relevant messages are messages that contain offensive language. While searching for relevant messages we also intend to build a thesaurus 3 that can help us to build a classifier,
which is not based upon training samples, but on prefabricated word lists.
With this annotated subset of messages we will then try to build several supervised learning methods. We mainly focus on two well-known machine learning techniques being
Naive Bayes and Support Vector Machine. Since the more complex and advanced
nature of a Support Vector Machine, we hypothesize the Support Vector Machine will
outperform our Naive Bayes method. Therefore, we will use our Naive Bayes classifier as
a baseline to compare the eventual improvements. When finished building/implementing
our classifiers we need to be able to validate the results of these classifiers. In order to
do this we expand our existing training set4 with a set of irrelevant messages. We also
randomly select two thousand messages out of the whole message set to create a realistic
validation set. This validation set can be used to test the performance of our classifiers.
We then further try to improve the general performance by tweaking the available parameters and isolate determining factors.
A third goal is to not only take into account the message itself but to analyze the
reactions to a message. Sexually inappropriate or racist messages could cause a fuss,
possibly manifested in the reactions to that message. We do this by creating a third
classifier5 that is able to not only detect sexual or racist offensive language, but has the
ability to detect outrage 6 .
In our final goal we will compare the results of the different seperate classification methods. This research also checks if performance can be increased by combining several
3
In this context a thesaurus can be defined as a list of concepts that describe a certain offensive
category.
4
One set of sexually inappropriate messages and one set of racist messages.
5
A classifier that is build on prefab word lists.
6
In this study ’outrage’ is defined as ’profound indignation, anger, or resentment’
5
Automated Detection of Offensive Language Behavior on Social Networking Sites
seperate methods into one whole. We then choose our best performing classification
method and classify the whole data set to create a general overview. Due to time constraints and the fact that the decision if a user should be blocked depends on much more
than only posting offensive messages, the propagation from message level to profile level
has not been extensively developed. Our final results intend to sort users based on the
number of detected offensive messages. Netlog will have an overview of offensive and
non offensive messages per user, ranked according to a relevance score7 .
1.3
Approach
Our study can be seperated in two main parts. The first part is mainly oriented towards
finding relevant messages in the whole message corpus (cf. chapter 2 for a detailed
overview of the corpus). Because we intend to work with supervised learning methods
(cf. section 1.1), it is crucial to find as many relevant messages as possible to build
a complete training set. Query expansion is a well-known technique that tries to
enhance the search to find more relevant documents by expanding the query and adding
informative terms. From our point of view every message containing racist or sexually
inappropriate content is a relevant message. In the second part we pursue our final goal,
classification of an unknown message, by applying text classification techniques. Text
classification is the domain where documents should be assigned one or more predefined
classes (cf. chapter 3 for a more elaborated overview of these techniques).
1.4
Applications
The exponential growth of the World Wide Web makes it is essential that people can use
developing tools to better find, filter, and manage this electronic information. Document
retrieval, categorization, routing and filtering can all be formulated as classification
problems [29].
1.4.1
Query Expansion
The most well-known application using query expansion is by far Google search engine.
Google will do a numerous things to enhance your search results and increase the amount
of relevant pages. From the Google help pages we find:
− Words that share the same word stem as the word given by the user. For example, if
the user search query includes ’engineer’ the search appliance could add ’engineers’
to the query.
− Terms of one or more space-separated words that are synonymous or closely related
to the words given by the user. For example, if a user searches for ’FAQ’ the
appliance could add ’frequently asked questions’ to the query.
7
This score is based on the belief the classifier attached to the classification of a message to a certain
category.
6
Automated Detection of Offensive Language Behavior on Social Networking Sites
1.4.2
Text Categorisation
Text categorization is of great importance for many information organization and management tasks.
Spam Filtering
The Internet Security Threat Report recently stated that spam email traffic dropped
around 13%. Despite this positive evolution still 75% of all email traffic is identitfied
as spam 8 . Spam messages are annoying to most users, as they waste their time and
clutter their mailboxes. Text classification tries to differentiate regular emails from spam
messages.
Topic Spotting
Topic spotting tries to automatically determine the topic of a text. it is often applied to
structure large quantities of documents into a set of categories. For example, a collection
of news articles can be divided into different categories such as politics, regional, economy,
etc.
Language Identification
Text classification techniques can also be used to try to automatically determine the
language of a text.
1.5
Challenges
Since we are concerned with a text classification problem a number of challenges arise.
The main difficulty in the field of Computational Linguistics is ambiguity. Ambiguity
can occur in a semantic, lexical or syntactic way [58] and is a challenging issue in natural language processing (NLP) [50, 59]. Lexical ambiguity of a word (or phrase) implies
that this word has more than one meaning in the language to which it belongs. For
instance, the word “bow” has several distinct lexical definitions, including “the front of
the ship” and “the weapon to shoot arrows with”. Lexical ambiguity can be solved by
inferring the meaning of a word out of the context. In contrast with lexical ambiguity
there is semantic ambiguity that imposes a choice between any number of possible
interpretations. This form of ambiguity is closely related to vagueness [58]. When a
phrase can be parsed in two or more ways we speak of syntactic ambiguity. The
same sequence of words can then be matched to different grammatical structures. The
well-known sentence for example ’Flying planes can be dangerous’ can be interpreted as
’Is flying planes dangerous?’ or ’are flying planes dangerous?’. Again context can be of
crucial importance here.
8
Internet Security Threat Report instigated by Symantec.
http://www.symantec.com/threatreport, [2012-05-15]
7
Report can be found on
Automated Detection of Offensive Language Behavior on Social Networking Sites
Not only ambiguity but the fact we are working in a social network environment, with
a great deal of very young users, poses a challenge. This means the vast majority of the
text will not be in standard Dutch, but will most likely contain informal language and
different dialects. Numerous different spellings for one word make it harder to estimate
the real value of that word (cf. chapter 2). Informal language’s lack of grammatical
rules makes it more difficult to identify seperate sentences and analyze the context to
resolve ambiguity.
An utmost challenging problem is the detection of implicit messages. Implicit messages could for example contain irony, sarcasm, etc. Again, humans can use the context
to reveal if the writer of a message is using irony or not. But even among human beings
misinterpreted irony is a frequent source of conflicts. The same applies to humour, but
this even includes an extra difficulty. Evaluating the appropriateness of humour implies
determining if a joke crosses a certain non tolerable line. As exploring the boundaries
of the right for freedom of speech is not one of our goals, we will simply ignore the fact
whether or not one is using irony, humour, etc.
To end the list of challenges, domain-independence is one of the biggest problems
in machine learning and classification. Accuracy results for one certain corpus (e.g. blog
corpus) can be very high, but perform very poorly when applied to another kind of
corpus (e.g. biomedical corpus). Finding effective approaches to overcome this problem
is a valued research field [58].
1.6
Related Work
Relatively few articles specifically discuss the detection of offensive language. However,
many researchers are working on different kinds of opinion mining or sentiment detection.
Examples are Pang et al. [45], Gordon et al. [18], Yu and Hatzivassiloglou [73], Riloff
and Wiebe [52], Yi et al. [72] , Dave et al. [11] and Riloff et al. [53]. We mention these
articles because you could state that in many cases the detection of sentiment is for some
specific attributes similar to the detection of offensive language. Hence, the subjective
language detection is a task for which flame detection could be considered an offspring.
Fk yea I swear: Cursing and gender in a corpus of MySpace pages
Thelwall, in [65], studies offensive language on the online network site Myspace with a
specific focus on ’swearing’. In this article they investigate whether (strong) swearing is
more dominant based on gender. The methods used in this article do not rely on any
text classification method. They do show that swearing declines when a user gets older.
But they could not find an overall gender difference for the use of stronger swear words.
8
Automated Detection of Offensive Language Behavior on Social Networking Sites
Smokey: Automatic recognition of hostile messages.
Ellen Spertus proposes in [60] a flame recognition system that not only looks for insulting
words, but also for syntactic constructs that have a negative or condescending tendency.
In the first step the smokey software converts each parsed sentence into Lisp s-expressions
by using sed and awk scripts. Next, semantic rules are used to further process these sexpressions, resulting in a 47-element feature vector based on the syntax and semantics
of each sentence. Finally one feature vector per message is created by summing up the
vectors of each sentence. A message is then classified as flame or not by evaluating
its feature vector with rules, generated by Quinlan’s C4.5 decision-tree generator. The
feature based rules were generated using a training set of 720 messages , which were able
to correctly categorize 64% of the flames and 98% of the non- flames in a separate test
set of 460 messages.
Detecting flames and insults in text
[34] is a sentence level classification system that tries to distinguish flames and information by interpreting the basic meaning of a sentence. This is achieved by using a set of
rules and analysing the general semantic structure of a sentence. However, the system
is affected by the fact that insulting sentences can only be detected when containing
related words or phrases originated from a lexicon.
Filtering Offensive Language in Online Communities using Grammatical Relations
In [71] the focus is more on filtering techniques than on the actual detection of offensive
language. This articles makes use of simple word matching algorithm in combination
with an offensive word lexicon to detect the actual offensive words
Offensive Language Detection Using Multi-level Classification
[48] tries to develop automatic and intelligent software for flame detection. Flames is
an other word for offensive language including taunts, squalid phrases, etc. The article
states that we are increasenly confronted with abusive language in for example emails or
other texts. They intend to perform flame detection by extracting features at different
conceptual levels and applying multi-level classification. A message can be assigned to
either the class Okay or the class Flame. Results point out that a 3-level classification
gives a general accuracy around 96% with a flame precision of 96.6%. They used a total
number of 1525 messages of which 68% Okay and 32% Flame.
9
Chapter 2
The Dutch Netlog Corpus
As we are trying to detect offensive language behavior on social networking sites a resembling corpus is needed. Different corpora are available such as Ken Lang’s Newsgroups
data set [27], the Multi-Domain Sentiment corpus of Blitzer [3], movie reviews [45],
the Wall Street Journal corpus, etc. However, none of these corpora
(a) are real representations of the data on a social networking site, or
(b) are well suited for flame detection, as they’ve been constructed for opinion mining
purposes.
We were able to make use of the Dutch Netlog Corpus (DNC), especially constructed
for this study, containing a collection of blog messages and reactions on Netlog.
2.1
Netlog
Netlog is a European social networking site that was founded in the early 2000s by
Lorenz Bogaerts and Toon Coppens. Netlog, nowadays, is being developed by Massive
Media NV, located in Ghent. Over 96 million members1 spread over more than forty
languages frequently visit Netlog. With Dutch accounted for the ninth highest number
(over four million) of members. At the beginning Netlog focused on reaching a younger
target audience, but this has changed over the years. Apart from Netlog the company
also develops a fast growing online dating site, named Twoo.
We explicitly thank Massive//Media for supplying this corpus.
2.2
Corpus Overview
In this section we give a general overview of the Dutch Netlog Corpus. We received
approximately seven million blog messages from 800,201 different users. We have not
1
http://nl.netlog.com/go/about/statistics, [2012-05-09]
11
Automated Detection of Offensive Language Behavior on Social Networking Sites
received any information concerning a user’s age or other personal data. The corpus also
includes more than eleven million reactions to the different blog messages.
However, 71.9% of the blog messages does not contain any reaction. This means that
the eleven million reactions are spread over about two million blog messages. In other
words, a blog message belonging to the group containing reactions has on average six
reactions. This is an important stat because in a later stadium reactions to a message
are taken into account in the classification process (cf. chapter 4). Only a small subset
of the messages will thus eventually be able to benefit from the extra information that
reactions contain. Moreover, it is important to notice that a message containing offensive language should be detected as soon as possible, thus optimally nobody has been
able to react to such a message.
2.3
Offensive language
Because offensive language or flames are very subjective concepts, we hereby attempt to
clearly define these concepts.
We created two general categories of offensive language. The first category is named
racist, and contains the following types of abusive language:
− Extremism: These phrases target some religion or ideologies.
− Homophobia: Phrases are usually talking about homosexual sentiments.
− Provocative language: expressions that may cause anger or violence.
− Racism: These phrases intimidate race or ethnicity of individuals.
− References to handicaps: These phrases attack the reader using his / her shortcomings.
− Slurs: These phrases try to attack a culture or ethnicity in some way.
The second category is named sexual:
− Crude language: expressions that embarrass people, mostly because it refers to
sexual matters or excrement.
− Implicit/ambigious language: expressions that in an indirect way refer to sexual
matters.
− Indecent proposals: expressions that contain indecent proposals often related to
sex.
− Unrefined language: some expressions that lack polite manners and the speaker is
harsh and rude.
12
Automated Detection of Offensive Language Behavior on Social Networking Sites
Each time this study mentions the concepts flame detection or offensive language,
implicitly we refer to the above definitions. However, even with an elaborate definition,
the labeling of a message as being offensive or not is a nontrivial task. Determining when
a certain expression is rude, unrefined, indecent or provocative is a subjective decision.
2.4
Creating Training and Validation set
Since we intend to work with a supervised learning technique (cf. section 1.1), it is
key to annotate and label a vast subset of blog messages. Supervised learning methods
require an example (labeled) set to build experience, which they can then use to base
decisions of new unlabeled messages on. Therefore, a training set should be as complete
and descriptive as possible.
2.4.1
Labels
A message can be labeled in five different ways: ’sexual’, ’racist’, ’outrage’, ’irrelevant’
or ’unknown’.
Sexual
A message is labeled as ’sexual’ when it relies to the definition of offensive language.
Moreover, it should contain either crude or unrefined language, implicit / ambigious language or an indecent proposal. Because we work on a social networking
site, which has a vast number of young users, we have a rather conservative approach to
what can be described as being sexual and what not.
For example: ’goed geil wil camsex ’
Racist
A message is labeled as ’racist’ when it relies to the definition of offensive language.
The ’racist’ category contains more general abusive language than the name makes you
believe. We do not consider the definition of a racist message to describe only discrimination of a race or ethnicity, but we consider it to also include matters such as homophobia,
extremism, slurs, etc.
For example: ’Bloed moet vloeien, weg met die vrijheid van die Joden republiek! ’
Outrage
Outrage is a category, which is specifically designed for reactions, to indicate if the
writer of the message intends to express contempt. A regular blog message should not
be tagged as an ’outrage’ message. Although it is perfectly possible to annotate a set
of messages expressing outrage, it is not our primary goal to create a training set for
13
Automated Detection of Offensive Language Behavior on Social Networking Sites
outrage.
For example: ’Pfff,vuile seksist! Ik weiger hier aan mee te doen. [thumbs down] ’
Irrelevant
All blog messages that do not contain any form of offensive language are considered to
be ’irrelevant’.
For example: ’ik kan niet slapen :( ’
Unknown
Our last label is named ’unknown’ and is applicable when it is unclear if a message
contains offensive language or not. Messages labeled as ’unknown’ are ignored.
2.4.2
Validation set
To be able to properly evaluate the performance of our classification system a validation
set is required: A validation set (also known as a test set) is a part of a data set to
assess the performance of classification models that have been trained on a separate part
of the same data set 2 .
We created a validation set containing a total of two thousand messages. To create
a realistic image of the whole data set, we randomly selected these messages. Based on
how the validation set is labeled, we can determine what the positive/negative ratio
is in the whole set. The positive/negative ratio gives an indication of the percentage of
messages containing offensive languages in comparison with irrelevant messages.
Thus, we labeled two thousand messages that contain:
Label
Amount
Racist
Sexual
Irrelevant
2
15
1983
Percentage (%)
0.10
0.75
99.15
Table 2.1: Distribution of validation set.
We clearly see that the amount of relevant messages is very low compared to the
amount of irrelevant messages. According to the annotation of the test set, only a tiny
0.85% of the whole data set contains offensive language. This means that our whole data
set should contain around 59500 messages with flames of which 7000 racist messages
and 52500 sexual messages.
2
http://www2.statistics.com/resources/glossary/v/validset.php, [2012-05-10]
14
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 2.1: Distribution of messages containing offensive language.
2.4.3
Training set
A training set can be defined as a specific part of the data set that is characteristic for
the problem to be solved, and is used as input for learning algorithms3 . As mentioned
before, a training set should contain as many labeled samples as possible. Finding relevant messages is not a trivial task in a set that contains just over seven million messages.
Therefore, we decided to use an information retrieval (IR) system, combined with query
expansion (cf. chapter 3), to be able to easily search for relevant documents.
With the aid of such IR system we were able to build a training set containing following message distribution:
Label
Amount
Racist
Sexual
Irrelevant
Total
173
362
4755
5290
Percentage (%)
3.27
6.84
89.89
100.00
Table 2.2: Distribution of different classes in training set.
3
http://www.answers.com/topic/training-set, [2012-05-10]
15
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 2.2: Distribution of different classes in training set.
2.5
Informal Language and Multiple Languages
To conclude this chapter we briefly discuss the nature of the data we received from Netlog. As we stated multiple times, the presence of informal language makes the detection
of offensive language specific, or text classification in general, more difficult. To begin
with, every person has his or her own writing style and practices. Moreover internet is
known for its fast-changing pace and new trends, including the way people communicate
with each other.
Typical examples are the lack of spaces in words (eg. “ikhaatu”) and excessive use
of abbreviations (eg. “BFF”, “LOL”). Other examples are adding or removing characters , writing in dialect (eg. “kzien a geire”), using emoticons (eg. “:-)”, “:D”), using
words from a foreign language, etc. Endless variations and the complete lack of any
structural rule, except from ones personal mindset, makes it very hard to process chat
language. Since there is not one way to write something, which results in different variations that in essence all refer to one word, frequency distribution will not be as accurate
as with standard language. The fact that a classifier is unable to estimate the real value
of a word, hinders the final classification.
16
Chapter 3
Methodology
Text categorization (TC) (also known as text classification, or topic spotting) is the task
of automatically sorting a set of documents into categories (or classes, or topics) from
a predefined set. This task falls at the crossroads of information retrieval (IR) and machine learning (ML).
This study is divided in two main parts based on two different techniques. In the
first part we focus on collecting relevant messages from our data set. To do this we use,
adjust and extend an information retrieval system. We extend this IR system to increase
its overall efficiency, by applying a technique called query expansion.
The second part then focuses on building a classification system to perform text classification. We first start with implementing a baseline classifier to be able to measure
the general improvement and compare our results with a more advanced classifier. Our
baseline classifier implementation will be the relatively simple but widely used Naive
Bayes (NB) method. A second more advanced classifier will be an implementation of
a support vector machine (SVM). The first two classification methods are supervised
learning methods, which means they rely on a training set. In our last step we also
implement a third classifier that does not have the ability to learn. This classifier will
be based on prefab word lists and will, next to detecting offensive language, also have
the ability to detect outrage in a message. The detection of ’outrage’ is however only
used in the reactions to support a possible prior classification of a blog message.
3.1
Information Retrieval
To be able to search through a vast set of messages, the use of an information retrieval
system is essential. However, the term information retrieval is widely used and very
broad. We use the definition proposed by [36]: “Information Retrieval is finding material
(usually documents) of an unstructured nature that satifies an information need from
within large collections.”
17
Automated Detection of Offensive Language Behavior on Social Networking Sites
3.1.1
Features
The eventual goal of an IR system is to provide the user with a specific information need
gathered from within a large collection of data. Although numerous different types of IR
systems exist, depending on the specific context, they all comply to a general structure.
In this section we will very briefly discuss the basic functionality of such a system.
Since an IR system relies on a collection of “unstructured data”, grouped per document,
its first task is to virtually organize the different documents in the collection. This is
done by the creation of an index. How a specific index is created, heavily depends on
the type of data we are working with. An index can be defined as :“A list of words
and corresponding pointers” 1 . An index can thus be compared to a thesaurus where
every word refers to a list of pointers. These pointers represent the set of documents
containing that word.
The construction of an index makes it possible to identify a document based on its
content. After creating an index, the next objective is to be able to efficiently describe
an information need. An information need is mostly described by a summarizing
sentence that lists the most important keywords. This is called a query. A query is
handled by an information retrieval system by processing and selecting the (key)words
in the query. These words are then used to find the matching documents in the index.
3.1.2
Evaluation
The evaluation of an information retrieval system is based on the amount of retrieved
relevant documents. In this study we mainly focus on three important metrics:
Precision
Precision is the fraction of retrieved documents that are relevant:
Precision =
|{relevant documents} ∩ {retrieved documents}|
|{retrieved documents}|
Recall
Recall is the fraction of relevant documents that are retrieved:
Recall =
|{relevant documents} ∩ {retrieved documents}|
|{relevant documents}|
Average Precision (AP)
Average precision takes into account precision and recall simultaneously. This metric
is especially useful when retrieved documents are ranked. Better ranking systems will
1
http://www.answers.com/topic/index, [2012-05-11]
18
Automated Detection of Offensive Language Behavior on Social Networking Sites
yield a higher average precision as opposed to systems where relevant documents are
more scattered [74].
Z
Average Precision =
1
p(t)dr(t) =
0
n
X
p(i)∆r(i)
i=1
with ∆r(i) being the change in recall from i − 1 to i and p(i) the precision from 0 . . . i.
3.1.3
Spelling Correction
The general purpose of spelling correction focuses on resolving typographical errors such
as insertions, deletions, substitutions, and transpositions of letters that result in unknown words [8]. Unknown words are words that can not be found in a trusted lexicon.
However, 25 to over 50% of observed spelling mistakes are mispelled words resulting in
a valid, though unintended word [25]. For example, “I want you to be quite!” where
“quite” should be spelled as “quiet”. The correction of a mispelled word strongly depends
on the information available from the context. Multiple supervised and unsupervised
learning methods exist, some based on a lexicon, other based on machine learning techniques [16, 17].
A very basic approach towards spelling correction is combining a trusted lexicon with
the levenshtein distance. When we find an unknown word, we search a best match in
the trusted lexicon according to the levenshtein distance. This levenshtein (or edit) distance is based on the number of insertions, deletions or substitutions that is required to
transform one word to the other.
Since the description of the information need (query) in an IR system is from utmost
importance, spelling correction can be of significant meaning. Misspelled words can
change the entire focus of the query and consequently result in very few or no relevant
documents at all.
3.1.4
Stemming
Stemming is the process of reducing all words with the same root or stem to a common
form [33]. A stemmer will for example match the words “fishing”, “fished”, “fish”, and
“fisher” all to the same root word, “fish”. The most basic stemming algorithms only take
into account morphological information to find a word’s reduced form. More advanced
stemmers are language specific and try to use statistical and contextual information to
find a word’s root form [69, 46].
Although we can generally increase the recall by using a stemmer, it can also significantly reduce precision by retrieving too many documents that have been incorrectly
matched. When analyzing the results of applying stemming to a large number of queries,
we notice that for every query that is helped by the technique, one is hurt [46].
19
Automated Detection of Offensive Language Behavior on Social Networking Sites
The biggest problem that arises when using a stemmer is off course the fact that words
with a totally different meaning, sometimes are matched to the same root form. For
example stemming the Dutch words ’negeer’, ’negeren’ and ’neger ’ all result to the same
root ’neger ’. This is a typical example, which will greatly reduce precision as ’neger ’
can be defined as offensive language, while ’negeer ’ is definitely not.
3.1.5
Part-of-Speech Tagger
Part-of-speech tagging (or POS tagging) is a technique that assigns an appropriate and
context-related grammatical descriptor to words in a text. The early POS taggers distinguished words among eight different grammatical tags: noun, verb, particle, article,
pronoun, preposition, adverb and conjunction [41]. Nowadays, the Penn Treebank tag
set is often used as basis to build an english POS tagger. This tag set, which is extensively described in [38], contains thirty-six different tags to describe a word. Since very
few grammatical rules can be applied to more than one languages, a general part-ofspeech tagger does not exist. However, different part-of-speech taggers all more or less
operate according to the same procedure [41]:
− Tokenization: divides a text in seperate processing units, and removes unwanted
information.
− Ambiguity look-up: implies using a lexicon to tag known words and a guesser
for tokens not represented in the lexicon.
− Ambigiuty resolution or disambiguation: based on two information sources,
one out of possible multiple pos tags should be chosen.
Part-of-speech tagging can greatly affect the performance of an IR system [32]. Assigning a part-of-speech tag to a term in a query improves the overall description of the
information need, which is an essential part of information retrieval. In other words, a
user is able to discriminate between different senses in which a term is used. On top the
extra information that is provided by a pos tag can be used by a stemmer. This would,
for example solve the issue with ’negeer ’ and ’neger ’, where it is clear that ’negeer ’ (verb)
and ’neger ’ (noun) do not carry the same pos tag and thus should not be mapped to
the same root form.
3.1.6
Query Expansion
Users tend to have difficulties formulating an appropriate query when searching for information on the web. According to a study the average length of a search query is 2.6
words long [1]. Which is in most cases too vague and small to retrieve a large set of relevant documents. Good queries often presuppose knowledge of relevant documents and
on top users may not know how a query is used by a retrieval model. Query expansion
tries to enhance the search to find more relevant documents by expanding the query and
adding informative terms. Additional search terms will define a more specific query that
20
Automated Detection of Offensive Language Behavior on Social Networking Sites
is less ambiguous and will result in documents that reflect the underlying information
needed.
Methods existing to perform query expansion can be divided into two classes:
− Global methods are techniques for expanding or reformulating query terms independent of the query and results returned from it, so that changes in the query
wording will cause the new query to match other semantically similar terms.
− Local methods adjust a query relative to the documents that initially appear to
match the query [36].
A popular technique categorized as being a local method is relevance feedback.
This technique is based on the possible relevance of highly ranked documents. This
information is then used to further specify the query to effectively increase the results.
Relevance feedback can be exploited on a manual or automatic way.
3.1.6.1
Rocchio Classification
Rocchio Classification is a method stemmed from the SMART Information Retrieval
System around the year 1970. It makes use of the Vector Space Model. The algorithm
is based on the assumption that users have a general idea of which documents should be
denoted as relevant or irrelevant [36].
An ideal query has maximal similarity to the relevant documents and minimum similarity
to irrelevant documents. Suppose we have |Dr | relevant documents and |Dn | irrelevant
documents. Then
X
X
1
1
Q=
×
di −
×
dj
|Dr |
|Dn |
di ∈Dr
dj ∈Dn
is the ideal query. Clearly, the relevant documents are not known. Assuming the user defines which retrieved documents are relevant, a set of relevant documents Du (feedback )
and irrelevant documents Dv (remaining documents) are available. Then
Qi+1 = α × Qi +
X
X
β
γ
×
di −
×
dj
|Du |
|Dv |
di ∈Du
dj ∈Dv
is a modified query that shifts towards Q. With α, β, γ being tuning parameters and
Qi the original query [55].
Local methods such as the algorithm described above depend heavily on the relevance
and similarity of the documents first retrieved. It also depends on the documents that
are selected as being relevant, whether this happens by hand or automatical. When the
initial query retrieves only few relevant documents, a query drift can occur. Which will
lead the derived query much further away from the ideal query. A query drift occurs
when the expanded form of the query changes the underlying “intent” [75]. In general
Rocchio query expansion leads to rather large queries and will often increase recall at
the cost of precision [36].
21
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 3.1: Example of Rocchio classification. Obtained from [36].
3.1.6.2
Local Feedback
Local feedback is a local method that will consider the top m most highly ranked documents. The algorithm consists of following steps:
− Add the n-most frequent (non-stop word) terms in the m most highly ranked
documents to the query.
− Disregard information from the other documents.
Many other variants exist (based upon clustering terms) and recent years, many improvements have been obtained on the basis of local feedback. This includes re-ranking
the retrieved documents using automatically constructed fuzzy Boolean filters, clustering
the top-ranked documents and removing the singleton clusters, clustering the retrieved
documents and using the terms that best match the original query for expansion [9].
TREC (Text REtrieval Conference) has showed in 1996 that local feedback approaches are effective [68]. This technique has however an obvious drawback as it supports on blind feedback, assuming the top ranked documents being relevant. If a large
fraction of the top-ranked documents is actually irrelevant, then the words added to
the query are likely to be unrelated to the topic. Thus the effects of pseudo-feedback
strongly depend on the quality of the initial retrieval.
Remark 1. Implicit relevance Feedback
Document’s relevance is not based on direct user feedback but instead indirect sources
of evidence are used. Rather than completely relying on the top-ranked documents.
An example of an indirect source of evidence is an often clicked document. Implicit
feedback is less reliable than explicit feedback, but is more useful than blind relevance
feedback, because local feedback or blind relevance feedback contains no evidence of user
judgements [36].
22
Automated Detection of Offensive Language Behavior on Social Networking Sites
Remark 2. Relevance feedback does not always guarantee a successful query expansion.
− Misspellings. If the user spells a term in a different way to the way it is spelled in
any document in the collection, then relevance feedback is unlikely to be effective.
This is very likely when working with chat language. This is one of the reasons
that shows the importance of stemming and the use of a decent spelling corrector.
− Cross-language information retrieval: Documents (partly) in another language are not nearby in a vector space based on term distribution. This is, anew
the case with chat langauge as this is often influenced by foreign languages. On
the contrary, documents from the same language tend to cluster.
3.1.6.3
Phrasefinder
Global methods consider all documents in the set for query expansion. The basic idea
is that the global context of a concept can determine similarities between concepts. In
other words, related terms will co-occur with each other. This is based on the association
hypothesis stating that words related in a corpus tend to co-occur in the documents of
that corpus [70].
Concept and context can be defined in numerous ways. The simplest definition is the
one that states that all words are concepts (except for stop words) and the context
for that word is the set of all words that co-occur in the set of documents [47]. More
advanced definitions compare concepts with noun groups (containing multiple words)
and define context as a fixed set of words surrounding the concept. Words surrounding
the concept is called a window and has a typical size of one to three sentences. These
definitions are a result from the TREC-3 conference in 1994 and are used as a basis for
the phrasefinder technique.
Every concept (noun group) will be associated with a pseudo-document. The content
of this pseudo-document will be the words occurring in every window for that concept
in the documents. At first these pseudo-documents will be filtered by removing stop
words and words that occur too often or too rare. Then a database or thesaurus is automatically built with these pseudo-documents creating a concept database. To expand
a query a ranked list of concepts will be generated out of this database. A number of
concepts from this ranked list will be added to the query weighted approximately.
In general global methods are very robust techniques that improve information retrieval.
Some queries will however be degraded when adding irrelevant concepts. Another drawback in this case is that most general thesauri will be expensive in terms of disk space
and computer time to analyse the data and build the database [68].
23
Automated Detection of Offensive Language Behavior on Social Networking Sites
3.1.6.4
Latent Semantic Analysis
Latent semantic analysis (LSA) is a mathematical method for computer modeling and
simulation of the meaning of words and passages by analysis of representative corpora of
natural text [26]. Latent Semantic Indexing (LSI) decomposes a term into a vector in a
low-dimensional space. This is achieved using a technique called singular value decomposition. It is hoped that related terms which are orthogonal in the high-dimensional
space will have similar representations in the low-dimensional space, and as a result,
retrieval based on the reduced representations will be more effective.
The first step in the algorithm is to calculate the term-document frequency matrix.
The entries in the term-document matrix are then transformed using a “ltc” weighting.
This weighting takes the log of individual cell entries, multiplies each entry for a term
by the inverse document frequency weight of the term, and then normalizes the document length. The transformed term-document matrix is taken as input for the single
value decomposition algorithm. A best “reduced-dimension” approximation to this
matrix is then calculated which results in a reduced-dimension vector for each term and
each document. The cosine between term-term, document-document, or term-document
vectors is then used as the measure of similarity between them and thus can be used to
build a global thesaurus [12].
Despite its potential, retrieval results using latent semantic indexing so far have not
shown to be conclusively better than those of standard vector space retrieval systems
[70].
A serious problem with term clustering is that it cannot handle ambiguous terms. If
a query term has several meanings, term clustering will add terms related to different
meanings of the term and make the query even more ambiguous.
3.1.6.5
Local Context Analysis
Local context analysis attempts to combine local and global methods. Based on the
hypothesis stating that top-ranked documents tend to form several clusters [70]. And
the number of relevant documents that contain a query term is non zero for (almost)
every query term [70]. As in phrasefinder noun groups are used as concepts and concepts are selected based on co-occurrence with the query terms. Instead of choosing
the concepts out of the whole set of documents the top n ranked are considered. Only
the best passages out of this ranked list are taken into account. A passage is a text
window of fixed size. Using passages is more ideal as opposed to documents, which can
be very long and a co-occurring of concept at the beginning and a term at the end can
be meaningless. It is also more efficient to use smaller parts of a document because it
reduces the cost of processing.
The algorithm is divided in the following steps:
− Use a standard IR system to retrieve the top m passages.
24
Automated Detection of Offensive Language Behavior on Social Networking Sites
− Concepts in these passages are ranked according to the formula
Y
log(af(c, ti )) × idfc idfi
bel(Q, c) =
δ+
log(n)
ti ∈Q
where
af(c, ti ) =
Pj=n
j=1
ftij × fcj
idfi = max 1,
idfc = max 1,
log10
5
log10
N
Ni
N
Nc
!
!
5
and
c
f tij
f cj
N
Ni
Nc
δ
is
is
is
is
is
is
is
a concept.
the number of occurrences of ti in pj .
the number of occurrences of c in pj .
the number of passages in the collection.
the number of passages containing ti .
the number of passages containing c.
a low non zero value to avoid zero bel values.
This formula is a variation on the tf idf formula, which is used in most information retrieval systems. The af part rewards concepts frequently occurring with
query terms, idfc penalizes concepts frequently occurring in the collection. idfi
emphasizes infrequent concepts. Finally, multiplication is used to emphasize cooccurrence with all query terms [68].
− In the third step the top m concepts are chosen from the ranked list and added
to the query. Concepts are weighted in proportion to their ranks so that a higherranked concept is weighted more heavily than a lower-ranked concept. Concepts
are added to the query according to the following formulas:
Qnew = #WSUM(1.0 1.0 Qold wtQ0 )
Q0 = #WSUM(1.0 1.0 wt1 c1 wt2 c2 . . . wtm cm )
wti = 1.0 − 0.90i
m
And ci being the ith ranked concept. The default value for wt is 2.0. #WSUM
is an INQUERY operator to combine evidence from different parts of a query.
Specifically, it computes a weighted average of its operands.
Local context analysis has several advantages. Once the top ranked passages are retrieved query expansion performs fast. This technique also performs well with regard
to proximity constraints. Phrasefinder can for example add concepts to the query that
co-occur with all query terms, but still do not match proximity constraints.
25
Automated Detection of Offensive Language Behavior on Social Networking Sites
3.1.6.6
Lexical Semantic Algorithms
Lexical semantic algorithms make use of the semantics in a document to expand and
enhance a query. The basic idea of these algorithms is to expand the query focusing
on the original query terms and their synonyms, acronyms, etc. These methods do
not make use of any statistical data, but rely on an external lexicon/thesaurus. As a
consequence we are dealing with a language dependant method. The algorithm described
in [67] makes use of the online database WordNet to act as a thesaurus. WordNet is
a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped
into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets
are interlinked by means of conceptual-semantic and lexical relations [40]. One of the
biggest challenge in this algorithm is to properly choose which query terms to expand
based on the thesaurus.
3.1.6.7
Wikipedia as Knowledge Base
Different articles state that using Wikipedia as extra knowledge base to perform query
expansion can be useful [6, 30]. Wikipedia can be used to enhance our knowledge
about certain concepts. In [30], Li et al. run every query both on the target corpus as
on Wikipedia. Experiments pointed out that retrieval performance was superior with
Wikipedia-based query expansion than without. In [20] an algorithm is developed to
design an automatic concept graph by using Wikipedia. This approach is a statistical
approach and is language independent.
3.1.6.8
Manual Constructed Thesaurus
A manual constructed thesaurus is a method that speaks for itself. To apply this method
a thesaurus is manually constructed at first. To expand a query every query term is
matched against a thesaurus and the corresponding terms are then added to the query.
3.1.7
Open Source Software
The most known and popular information retrieval system is off course google’s search
engine. But beside google many open source IR systems exist. We list some freely
available and Java-based IR software. Different systems off course offer different advantages and drawbacks.
3.1.7.1
Lucene
Apache Lucene(TM) is a java-based full-featured text search engine library, originally
developed by Doug Cutting. Lucene is a very popular search framework, and is used
by multiple large companies such as IBM, Twitter, Apple, etc.2 Key features of Apache
Lucene are:
2
http://wiki.apache.org/lucene-java/PoweredBy, [2012-05-11]
26
Automated Detection of Offensive Language Behavior on Social Networking Sites
− Lucene’s core is based on the idea of a document containing fields of text. This
allows Lucene’s API to be independent of the file format and enables text from
PDFs, HTML, Microsoft Word, to OpenDocument documents to be indexed.
− Lucene is highly supported by a large community, which facilitates finding a solution for a certain problem.
− It enables scalability and high-performance indexing.
− Powerful, accurate and efficient search algorithms.
Lucene provides multiple ways to enhance a query:
− Wildcard Searches: It is possible to use single or multiple character wildcards to
broaden the search.
− Fuzzy Searches: A fuzzy search is based on the Levenshtein distance algorithm.
This makes it possible to search on slight variations of the original word.
− Boosting a Term: A term or phrase can be boosted by adding a caret followed by
a boost factor to the term. By boosting a term you can control the relevance of a
document.
In Lucene the relevance of a document is calculated taken into account following factors:
tfq
tfd
idft
numDocs
docF reqt
normq
normdt
boostt
coordqd
:
:
:
:
:
:
:
:
:
Term frequency in query q
Term frequency in document d
Inverse document frequency
Number of documents in index
Number of documents containing t
Norm of q
Square root of number of tokens in document d in the same field as term t
Boost factor for term t
Number of terms in both query q as document d
Number of terms in query q
(3.1)
This gives the following equation to calculate the relevance of a certain document:
!
X tfq × idft × tfd × boostt
scored =
× coordqd
normq × normdt
t
3.1.7.2
Terrier
Terrier, Terabyte Retriever, is a project that was initiated at the University of Glasgow in 2000. The goal of the project was to provide a flexible platform for the rapid
development of large-scale Information Retrieval applications. On top it provides a stateof-the-art test-bed for research and experimentation in the wider field of IR. The Terrier
27
Automated Detection of Offensive Language Behavior on Social Networking Sites
project explored novel, efficient and effective search methods for large scale document
collections, combining new and cutting-edge ideas from probabilistic theory, statistical
analysis, and data compression techniques [43].
Automatic pseudo-relevance feedback (cf. section 3.1.6) is included in the open source
engine. The method works by taking the top most informative terms from the top-ranked
documents of the query, and adding these new related terms into the query. The new
query is reweighted and rerun and as such providing a richer set of retrieved documents.
Terrier provides several term weighting models from the DFR framework, which are
useful for identifying informative terms from top-ranked documents. In addition Terrier
also includes well-established pseudo-relevance feedback (cf. section 3.1.6) techniques,
such as Rocchios method (cf. section 3.1.6.1). Automatic query expansion is highly
effective for many IR tasks.
3.1.7.3
Xapian
Xapian is a highly adaptable toolkit which allows developers to easily add advanced
indexing and search facilities to their own applications. It supports the Probabilistic
Information Retrieval model and also supports a rich set of boolean query operators.
Xapian supports following important features:
− Relevance feedback - given one or more documents, Xapian can suggest the
most relevant index terms to expand a query, suggest related documents, categorise
documents, etc.
− Supports stemming of search terms (e.g. a search for ”football” would match
documents which mention ”footballs” or ”footballer”). This helps to find relevant
documents which might otherwise be missed. Stemmers are currently included for
Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian,
Portuguese, Romanian, Russian, Spanish, Swedish, and Turkish.
− Synonyms are supported, both explicitly (e.g. ” cash”) and as an automatic form
of query expansion.
3.2
Machine Learning Techniques
Text classification can be performed in various ways. Nowadays, most popular and effective techniques are machine learninig techniques. Machine learning is a subdomain
of Artificial Intelligence that deals with algorithms, which can learn based on previous
experiences. In other words, these algorithms are based on previously labeled data, from
which they can infer certain properties and patterns. These properties can then be applied to classify new data samples. Machine learning techniques can be supervised, as
well as unsupervised or used by applying reinforcement learning.
28
Automated Detection of Offensive Language Behavior on Social Networking Sites
The major weakness these techniques involve is their domain-dependency, which means
that new properties and unseen patterns in data will very likely lead to falsely classified
data samples. As language can vary in infinite ways, rarely occuring patters are the rule
rather than the exception. Machine learning techniques are often limited to generalize
based on already known information.
3.2.1
Data Representation
Since we are dealing with textual data, it is vital this information is extracted and
represented in an efficient and complete manner. The simplest approach is using a bagof-words model. This is an unordered set of words (features), disregarding grammar
and even the exact position of the words [58]. Each distinct word corresponds to a feature, with the frequency of the word in the document as its value. Only words that do
not occur in a stop list are considered. Although a bag-of-words representation seems
to throw away significant information, it is a very simple, yet effective representation of
a document [22].
An other way of representing a textual document is by using N-grams. N-grams provide
the ability to identify n-word expressions. For example, ’I love you’ can be defined as
a 3-gram and provides a whole lot more information than the seperate words ’I’, ’love’
and ’you’. However, it is not recommended to use n-grams with more than three words.
Co-occurence of certain n-grams will be less likely to be found as the size of the n-grams
increases.
The extraction of words or concepts from a document can also be influenced by whether
stemming (cf. section 3.1.4) is applied. On top extra contextual information can be
added by applying a part-of-speech tagger (cf. section 3.1.5), detection of emoticons, excessive use of capital letters, etc.
3.2.2
Feature Selection
Feature selection (also known as subset selection or feature reduction) is the technique
of selecting a subset of relevant features. In a textual environment, effective feature
selection is essential to improve the efficiency of the learning task and increase general
accuracy [14]. Text classification in general and offensive language detection more specific contain positive (relevant) classes and a negative (irrelevant) class. In most cases
a relevant document is characterized by only a few specific features. Non-informative
features are often described as noize. This noise can be defined as features that do not
provide any extra information to further enhance the decision making proces. Eliminating noise and selecting only the most informative features is the key task of feature
selection.
The overall feature selection procedure works by giving each potential feature a score
according to a particular metric. Then the best k features are selected and only those
29
Automated Detection of Offensive Language Behavior on Social Networking Sites
features are used in the further text classification process [14]. Multiple feature selection
metrics exist and make use of different aspects of the data. In essence, they can be
divided into wrappers, filters, and embedded methods [19].
− Wrappers rank features according to their predictive power; this score is obtained
by utilizing a learning machine as a black box to determine most predictive subset
of features. A wrapper method often uses sequential stepwise selection to gradually
remove or add features at each step [62].
− Filters select subsets of features as a pre-processing step, not depending on the
chosen predictor. Filters perform the feature selection faster than wrappers and
create a more general subset of features. A well-known filter method is InformationGain based feature selection [14].
− Embedded methods include feature selection in the training process of the learning machine. Embedded methods are in general more efficient than wrappers since
they do not require the continuous retraining of the learning machine and splitting of the data (in validation and training set) [19]. An example of a learning
algorithm that embeds feature selection is CART [31].
A well-known and widely used feature selection (filter) method is Mutual Information (MI). This method attempts to compute the mutual information of a term t and
a class C. This mutual information is a measure of how much information the presence
or absence of a term contributes to making a correct decision whether or not a message
belongs to c [37]. In short, the algorithm selects k terms per class with the highest
mutual information.
3.2.3
Classification Techniques
As described above, machine learning techniques can be divided in three different categories:
− Supervised learning (also known as inductive learning) is the process of learning
a set of rules from labeled training samples [35].
− In unsupervised learning the algorithm is given a set of unlabeled samples and
is supposed to group these into classes, without any prior knowledge of labeled
output or the specific classes [35].
− Reinforcement learning is a technique where an algorithm depends on external
feedback to base its decision on. The learning algorithm is not told which actions
should be executed, but must determine the effectiveness of certain actions based
on the given reward. Without this feedback, the learning algorithm will have no
ground to decide which actions it should attempt [35].
We present two classification techniques, both supervised techniques.
30
Automated Detection of Offensive Language Behavior on Social Networking Sites
3.2.3.1
Naive Bayes
Naive Bayes (NB) is a simple, yet effective classifier that is based on the so-called Naive
Bayes assumption. This assumption states that all attributes of the examples are independent of each other givenQthe context of the class [54]. In a mathematical way this
is described as, P (X|C) = ni=1 P (Xi |C) where X = (X1 , . . . , Xn ) is a feature vector
and C represents a class. Calculating the document probability comes down to multiplying the probabilities of all individual words in that document [54]. The class with
the highest probability is then the final classification of the document. However, in [49]
multiple improvements are proposed to eliminate several well-known issues that torment
the regular naive bayes method. For example, the different features are given a score per
class based on the normalized tf-idf metric (cf. chapter 4). Disregarded the often
solid performance of Naive Bayes, it is often used to perform as a baseline opposed to
other more advanced classification methods [54].
3.2.3.2
Support Vector Machine
A Support Vector Machine (SVM) is often regarded as the classifier with most success, and yielding the highest accuracy results for many classification problems [66].
Much simplified, an SVM represents samples as points in a space, mapped to a highdimensional space, that constructs a hyperplane with a maximal Euclidean distance
between the different labeled samples. When classifying a new sample, it is mapped to
the same space and it depends on which side of the hyperplane it is located to predict
its class [63]. It is important to notice that this SVM hyperplane is determined by only
a small subset of the training instances, which are called support vectors (cf. Figure 3.2)
[63].
A classification problem mapped to a vector space can be linear seperable or nonlinear seperable. Two sets of points in an n-dimensional space are linearly seperable if
they can be competely seperated by a hyperplane. This hyperplane is a decision surface
of the form w · x + b = 0, where w is an adjustable weight vector, x is an input vector,
and b is a bias term [15]. SVMs will learn a linear threshold function, designed to solve
linear seperable data sets.
31
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 3.2: SVM hyperplane construction in the idealized case of linearly separable data. Obtained
from [63].
However, by using an appropriate kernel function, an SVM can be generalized, allowing it to learn nonlinear functions (such as polynomial or radial basic function) [22].
Nonlinear separation surfaces can be approximated by linear surfaces by mapping them
to a higher-dimensional space [64]. How this exactly happens lies outside the scope
of this thesis, however in figure 3.3 a visualisation of this proces is shown, where ϕ
represents the kernel function.
Figure 3.3: Feature vectors are mapped from a two-dimensional space to a three-dimensional
embedding space. Obtained from [64].
32
Automated Detection of Offensive Language Behavior on Social Networking Sites
Often data sets are not perfectly seperable which leads to either wrongly classified
training samples or overfitting. A required and important parameter when dealing with
Support Vector Machines is the cost parameter, C. This parameter controls the trade-off
between allowing training errors and forcing unadaptable margins. Increasing the value
of C will increase the cost of misclassified samples and will force the SVM to create a
more accurace model. However, this model may possibly not be able to generalize well
and thus also won’t correctly estimate new samples (overfitting). It is therefore essential
to find a good balanced value for the cost parameter C.
3.3
Lexicon Based Text Classification
Lexicon based text classification algorithms rely on manually constructed word lists.
Every word in the list belongs to a certain category. In the early 1990s there were
no standardized labeled data sets available [44]. As a result the most researchers only
considered proposals for systems or designed prototypes. Most systems also had no
learning capability and focused on simpler classification tasks [44]. Despite the fact that
these methods are rather static and outdated, they can still be useful in some cases.
In [61], fuzzy logic is combined with natural-language processing to analyze the affect
content in free text.
3.4
Evaluation
The evaluation of classification methods is based on the number of correctly classified
messages as opposed to falsely classified messages. We mention four different situations
when a new message is classified:
− True Positive (TP) : The classifier correctly indicates the message contains
offensive language. In other words, the message is rightly classified as sexual or
racist.
− True Negative (TN): The classifier correctly indicates the message doesn’t contain offensive language. In other words, the message is rightly classified as irrelevant.
− False Positive (FP): The classifier wrongly labels the message as a message
containing abusive language.
− False Negative (FN): The classifier wrongly indicates the message is irrelevant.
The most simple manner to evaluate the performance of a classification system is
to analyze its accuracy. Accuracy shows the general correctness of a classifier and is
calculated as follows:
TP + TN
Accuracy =
TP + TN + FP + FN
33
Automated Detection of Offensive Language Behavior on Social Networking Sites
However, a classification system that automatically labels samples as irrelevant (negative) would yield very high accuracy results when dealing with corpora that contain
only a very small amount of positive samples. Accuracy is thus no useful metric when
working in an environment where one category dominates the others. Therefore, it is
important to again make use of the metrics precision and recall, which are interpreted
the same as in information retrieval (cf. section 3.1.2).
These metrics are defined as in [42]:
Precision =
Recall =
TP
(TP + FP)
TP
(TP + FN)
By analyzing precision and recall we can constitute a better understanding of the performance of the detection of offensive messages.
This study also uses the F1 Measure, which is an evenly weighted combination of both
precision and recall:
precision × recall
F1 Measure = 2 ×
precision + recall
Apart from testing our data on a seperate test set, it also possible to extract a small part
from our training set to use as validation set. This method will be applied by dividing
the training set in k parts. The classifier is then subsequently trained on k − 1 parts and
tested on the remaining part. This technique is called cross validation and provides
us with an accurate view over the performance of a classifier.
34
Chapter 4
Design and Implementation
This chapter discusses the system’s most important design decisions and solutions. It
starts by describing an information retrieval system, extended with a query expansion
technique. Subsequently, the implementation of our offensive language detection system
is explained.
4.1
Information Retrieval
As already mentioned we intend to use an IR system to efficiently retrieve relevant
documents to build a training set. Section 3.1.7 discusses several open source software
packages that provide IR solutions. We decide to base our information retrieval system
on Apache Lucene. Lucene provides an API to index and search large amounts
of data on an efficient and performant manner. We choose Lucene above the other
solutions, because it is an advanced platform that can easily be extended and integrated
into an other system. Lastly, Lucene is well-known and elaborated support is provided
by a large community.
4.1.1
Query Expansion
The aim of query expansion, together with several techniques describing how to apply
it, are already extensively described (cf. section 3.1.6). However, we need to determine which query expansion method best suits our purposes. Avoiding query drift is
the biggest challenge when applying query expansion. Another issue that is faced when
implementing query expansion, this study in particular endures, is chat language. Chat
language implies that multiple relevant features (eg. ’eigenlijk’ ) occur in many different
forms (eg. ’eigelijk’, ’eiglijk’, ’eigelyk’, etc) in a text. Both global as automatic local
methods will have difficulties effectively expanding a query, since it is much harder than
usual to decide which features or documents are relevant. Also using lexical and semantic relations to perform more efficient query expansion seems not designated to use in
this study. The Dutch language does not have a Wordnet alternative and again we predict that faulty and incomplete chat messages won’t contain much structure or grammar.
35
Automated Detection of Offensive Language Behavior on Social Networking Sites
Therefore, the manual and local query expansion technique Rocchio is chosen (cf.
section 3.1.6.1). Rocchio classification performs query expansion by indicating which
documents are relevant (and which are not). In this way, we have more control and can
more easily avoid query drifts. Since the main goal of query expansion is to aid the
construction of a training set, it is not an issue that user input is required.
We implement Rocchio classification by making use of an existing extension on Lucene,
i.e. LucQE. LucQE (Lucene Query Expansion) provides several modules to be able to
perform query expansion [57]. One of these modules is Rocchio query expansion. This
method is slightly adopted to fit into our general system.
Figure 4.1 gives a graphical representation of the approach towards the first stage of
this study. The Netlog corpus is indexed using Lucene, which is also used to search
through the data. The workflow consists of following sequence:
− We insert a query into our Lucene system.
− Lucene provides all messages, ranked according to relevancy, matching the query.
− We select all messages that contain abusive language and specify whether they
belong to the ’sexual’ or ’racist’ category.
− In the next step Rocchio classification is applied by expanding the base query
with new words, originating from the relevant messages. Every message that was
labeled is now also saved, so it can be reproduced to build our training set.
− In the following step the expanded query is inserted into the IR system and matching messages are displayed.
− These new terms can also be saved to construct a thesaurus. This thesaurus then
contains informative terms grouped per category, which can be used to build a
descriptive word list with.
36
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 4.1: Graphical representation of this study’s approach.
4.2
Offensive Language Detection
To be able to satisfy our final goal two supervised classifiers are created. At first, a
Naive Bayes classifier is implemented and afterwards this study proceeds by designing
a Support Vector Machine. Naive Bayes classification is a rather simple method, but
has proven to perform solid in many cases. On the other hand we can also use the
performance of our NB classifier as a baseline to assess the overall performance of our
Support Vector Machine.
4.2.1
Naive Bayes
We describe the algorithm proposed by [49] and used in our system in more detail.
Assume we have a total of N features and we have a fixed set of m classes. The
following is calculated:
− Normalized Frequency for a term (Tf) in a document is calculated by dividing
the term frequency by the root mean square of terms frequencies in that document.
37
Automated Detection of Offensive Language Behavior on Social Networking Sites
− Weight Normalized Tf for a given feature in a given class equals the sum of
normalized frequencies of the feature across all the documents in the class.
− Weight Normalized Tf-Idf for a given feature in a class is the Tf-idf calculated
using standard idf multiplied by the ’Weight Normalized Tf’.
− The sum of the weight normalized Tf-Idf (W-N-Tf-Idf) for all the features in a
label is named Sigmak .
For every term in a new message a weight can be calculated depending on its weight
normalized Tf-Idf in that class:
W-N-Tf-Idf + 1
W eight = Log
Sigmak + N
The probability of a message belonging to a certain class is then calculated by running
over every term in the message and summing their weight.
4.2.2
Support Vector Machine
In order to implement our SVM algorithm an available, open source software package
is used. Well-known packages are LibSVM, LibLinear and SVM-Light. We chose
to work with LibLinear, which inherits many features of the popular SVM library libSVM [13]. Liblinear is built to optimally perform for linear high dimension classification
problems, thus typically for text classification.
To be able to work with LibLinear we need to transform our data in the data format
used by the LibLinear package. LibLinear, just as LibSVM and SVM-Light, expects to
receive an array of feature nodes per sample. This array of nodes represents a vector
in the n-dimensional vector space. In order to convert a textual message to an array of
feature nodes we extract all features out of this message and calculate for every feature
its ’Tf-Idf’ score.
This study also provides a feature selection method based on mutual inclusive information (cf. section 3.2.2). Mutual inclusive information is a filter method that is applied
by removing or ignoring all non informative terms. Since feature selection is a technique
to reduce the amount of noise, we analyze if our system benefits from using it.
4.2.3
Semantic Methods
Except from our two supervised classification methods, we also implement a classifier
based on prefabricated word lists. The main reason to also design a semantic classification method is to be able to detect ’outrage’ in a reaction to support or improve the
performance of our supervised classifiers. In this way we create two seperate methods
that do not, by default, share the same flaws.
38
Automated Detection of Offensive Language Behavior on Social Networking Sites
Sexual
Racist
Outrage
kuste
hoeren
kont
geile meiden
homo
doden
neger
eigen land
racistisch
vuil
vies
niet kunnen
Table 4.1: Random entries from our word lists.
Word lists
These word lists are constructed by using the query expansion module on the training
set and are refined in a manual way. In other words, the top terms are extracted from
our labeled ’racist’ messages to construct our ’racist’ word list. The same procedure is
followed in order to create a ’sexual’ word list. Finally, a list to detect ’outrage’ is constructed by searching and applying query expansion on reactions that express contempt.
These lists are then manually further adjusted and extended.
An entity in our word list contains the following properties:
<word> <score> <vital> <intensity> <centrality> <pos>
The score of a word is a measure of the importance of that word regarding to a specific
class. The score property is derived from the rocchio classification term ranking. The
property vital is a boolean that indicates whether the word is important or not. These
first two properties are only used in our initial algorithms, and are not taken into account
in the final implementation.
We eventually use the third attempt as our final algorithm (cf. chapter 5), which ignores
the properties score and vital. The main properties present in the word list are intensity,
centrality and pos. Intensity expresses the strength of a word, while centrality indicates
the degree of relatedness to a certain class. For example, the word ’neger’ has a greater
intensity level compared to the word ’zwarte’. Centrality is a measure that suggests
how much a term is linked to a class. Many words relate in some context to a class,
but in itself are no offensive words (eg. ’vuil’ ). Both centrality and intensity represent
decimals from zero to one. Finally, every word also has a part-of-speech tag, which is
not extensively used in this study, but allows future enhancements to create a more
advanced algorithm. The values of the intensity, centrality and pos tag are manually
crafted and adjusted, based on the authors experience and assessment skills.
Lastly, the possibility to add bigrams to the word lists is added. Bigrams can greatly improve performance since they often combine two regular words into one offensive concept
(eg. ’white power’ ).
39
Automated Detection of Offensive Language Behavior on Social Networking Sites
Algorithms
We implemented three different algorithms starting from a straightforward, simple algorithm to a more advanced algorithm, which uses the intensity and centrality concepts.
Simple Algorithm runs over the different features of a message and sums the score
of all features that have a match in the word list. The class with the highest score is the
eventual result of the classification. If no hits are found the message is logically classified
as irrelevant.
Vital Algorithm runs over every word of the message and checks if the word can be
found in a certain word list. If the word is indeed found in a word list, its score is added
to the total (per class) and we check whether it’s a vital word or not. Messages with
too few or no vital words in it are automatically classified as irrelevant. We make use
of the vital property to lower the amount of false positives. As we’ve already explained,
many words in our lists contribute to abusive language, but on itself are not offensive.
Since spelling correction is incorporated, words that at first are not found in a word list,
can still be matched to one closely related (based on the levenshtein distance). However,
this is a rather naive approach since very often words are not property corrected (cf.
chapter 5).
Fuzzy Algorithm runs per class over every word of the message and checks for possible hits. If a hit is found the intensity and centrality of the word are analyzed. The
score of a message per class is solely based on the intensity of the matched words. This
score is however reduced to zero whenever the unique hits in a message do not have a
total centrality of at least 1 and one centrality score at least 0.7. The Fuzzy algorithm,
in other words, fine-tunes the vitality concept used in the algorithm above. A certain
word can belong more or less to a certain class based on its centrality.
4.2.4
Combination of Methods
Our final classification method both makes use of our Support Vector Machine classifier and our Fuzzy classifier, based on word lists. It is worthy to mention that at
first it was not our intention to combine several classifiers into one whole. However, our
initial Support Vector Machine did not yield the expected results (cf. chapter 5). Thus,
we tried to improve our classification system by incorporating the information reactions
to a message provide. We did this, as described above, by implementing a semantic
classifier that can detect outrage in a reaction. On top this semantic classifier can also
detect regular abusive language.
This gives us the following sources of knowledge to design a final algorithm with:
− Primary classifier: SVM trained on our default training set (possible outcome
’sexual’, ’racist’, ’irrelevant’)
40
Automated Detection of Offensive Language Behavior on Social Networking Sites
− Secondary classifier: semantic classifier based on a weighted word list (possible
outcome ’sexual’, ’racist’, ’irrelevant’)
− Reaction classifier: semantic classifier also based on word list but now classifying only the reactions of a certain message (possible outcome ’outrage’, ’sexual’,
’racist’, ’irrelevant’ )
The solution whether a message is relevant or not is predicted by one of our classifiers in more than 90% of the cases. Thus, it is vital to use the correct classifier in
every situation. This algorithm takes into account the characteristics from our classifiers,
derived from multiple tests. We distinguish among four different cases:
− When both primary and secondary classifier give a message the same label, we
logically assume this label is the correct one.
− In the case that a message contains less features than a certain threshold, the
secondary classifier has the advantage over the primary.
− When primary and secondary dissent we analyze if any outrage or abusive language
in reactions can be found to support the decision of one of our classifiers.
− If on the contrary reactions do not contain any information we base our decision
on the normalized scores both classifiers assess to their decisions.
On the basis of the above cases we choose which classification result eventually will
be selected.
41
Chapter 5
Results
This chapter presents the results that are obtained from experiments applied on our
different methods. These results are based on the proposed metrics (cf. section 3.1.2
and 3.4) and are gradually improved by a continuous development process. We start
with stage one of our study, which is devoted to query expansion. We discuss the
functionality of our chosen query expansion method and compare this to its overall
performance. Further on, we extensively describe and discuss the performance of our
text classification system. We start with an elaborated view on the functioning of our
Naive Bayes classifier and end with the algorithm that combines the functionality of
different classifiers.
5.1
Query Expansion
To point out the utility of query expansion we compare the (average) precision and recall (cf. section 3.1.2) of different queries, both their simple and expanded forms. It is
important to note that queries in this experiments are focused on the retrieval of racist
messages. Queries that are build to retrieve sexual messages show similar results. We
focus on racist messages because they only occur rarely in the corpus, as opposed to
sexual messages that can be found more easily (cf. section 2.4.2).
To be able to measure the performance of our information retrieval system, we construct different (expandend) queries. By comparing the recall and precision of a simple
query with its expanded form, we should notice significant differences. We presume that
both precision and certainly recall will yield better results when the expanded query is
applied.
43
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.1.1
Rocchio Relevance Feedback
We briefly recap the main formula that should shift a new (expanded) query Qi+1 closer
to the ideal query Q (cf. section 3.1.6.1).
Qi+1 = α × Qi +
X
X
β
γ
di −
dj
×
×
|Du |
|Dv |
di ∈Du
5.1.1.1
dj ∈Dv
Initial Setup
The Rocchio algorithm depends on three main parameters (cf. section 3.1.6.1) α, β and
γ. The importance of the original query is denoted by α, while β and γ respectively
denote the importance of the selected relevant documents and the irrelevant documents.
Although, nowadays the influence of irrelevant messages is questioned and thus γ is often
reduced to zero.
In our implementation, we assign the following values to these parameters:
α = 0.75
β = 0.25
γ=0
5.1.1.2
Query 1
We start with a query containing the following words ’neger ’ and ’zwart’. This query is
expanded, which results in the query: “zwart neger marokkan wit werk virus turk clan
turk the ras jood ”. We now compare both recall and precision of these two queries.
44
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 5.1: Recall Comparison
Figure 5.1 and 5.2 obviously show that both precision and recall increase when the
expanded query is used. Recall of the expanded query outperforms the base query by
far. The simple query retrieves around 33 relevant documents, while the expanded query
retrieves 51. The same applies to precision, albeit less pronounced (cf. Figure 5.2). We
can also compare precision in relation to the retrieved number of relevant documents.
Again, we see a much improved result when using the expanded query (cf. Figure 5.3).
Lastly, Figure 5.4(a) displays the difference in average precision between both queries.
45
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 5.2: Precision Comparison
Figure 5.3: Precision-Recall Comparison
46
Automated Detection of Offensive Language Behavior on Social Networking Sites
(a) Average Precision Comparison of Query 1
(b) Average Precision Comparison of Query 2
Figure 5.4: Average Precision Comparison.
5.1.1.3
Query 2
Our second base query starts from “sieg heil ” and is eventually expanded to “white race
sieg parasiet heil vijand national fight pride trots 88 14 hitler nazi suprematie”. We only
compare the average precision results, as our results are similar to the numbers above (cf.
section 5.1.1.2). Query 2 also shows us the improved average precision of its expanded
form (cf. Figure 5.4(b)).
5.1.1.4
Combined Query
We combine different queries into one large query to see the eventual results. All the
relevant terms of the query can be found in Table 5.1. We emphasize relevant, since
an expanded query usually contains multiple irrelevant terms, which we filtered out to
avoid query drifting.
jood
neger
zwart
parasiet
fight
14
neger
virus
vuil
moslim
white
heil
pride
hitler
marokkaan
turk
vraatzucht
dik
race
vijand
trots
nazi
wit
clan
ras
kk
sieg
national
88
supremati
werk
turk
Table 5.1: Expanded query
Figure 5.5 clearly shows that our combined query outperforms all other queries in
terms of recall.
47
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 5.5: Number of relevant documents when retrieved 1000 documents
Figure 5.6: Average Precision Comparison
We notice that, in terms of average precision, the combined query does not perform
better than the seperately expanded queries (cf. Figure 5.6). This is because the average
48
Automated Detection of Offensive Language Behavior on Social Networking Sites
precision metric depends on the ranking of our retrieved relevant documents. In other
words, we depend on the ranking capability of Lucene. On top our combined query
contains 36 different terms which will automatically increase the amount of retrieved
documents. This, in turn, will make it more difficult for Lucene to properly rank these
documents and will result in more scattered relevant documents.
5.1.2
Conclusion
We notice a great improvement in both recall and precision when expanding a query.
The number of retrieved documents more than doubles when applying our combined
query as opposed to its base form. However, precision does not significantly improve
since the retrieval of documents also depends on the way results are ranked. Thus, it is
not always better to use very large queries, although this will generally increase recall,
precision will not. The main goal of this study was to use query expansion as a manner
to efficiently build a descriptive training set. Albeit, query expansion greatly improved
the finding of relevant messages, human interference is essential to guide the process.
5.2
Offensive Language Detection
This section presents the results of our flame detection system. We begin with an
introduction to our Naive Bayes classifier, proceed to our support vector machine and
semantic classifiers, to finally end with a combination of methods.
5.2.1
Naive Bayes
Our Naive Bayes is a nonregular implementation and takes into account a few improvements (cf. section 4.2.1). However, we do not use any form of feature selection. Our
Naive Bayes classifier is tested by applying cross validation (cf. section 3.4) as well as
testing it on our standard validation set (cf. section 2.4.2).
5.2.1.1
Evaluation
Table 5.2 displays the performances of our Naive Bayes classifier. For both tests the
accuracy achieves solid results of respectively 95% and 87%. In the remainder of this
chapter accuracy results are not analysed in-depth, because in all cases accuracy achieves
values from 90% to 99%. This is due to the large impact of irrelevant messages, which
account for more than 90% of the correctly classified messages.
We notice a big difference in precision when applying cross validation as opposed to
tests on our validation set. We attach more value to the results on our validation set,
which better reflects a realistic situation. Naive Bayes has a tendency to classify documents as flames too rapidly, which in the second case results in a low precision. The
reason why the NB classifier performs exceptionally well when applying cross validation
is twofold. At first, our training set is larger and contains far more relevant messages
49
Automated Detection of Offensive Language Behavior on Social Networking Sites
than our validation set. Secondly, much more than our validation set, our training set is
a coherent set of messages, which ensures that it will be much easier to classify messages
from the same set.
The simplicity of the Naive Bayes method does not allow it to properly unravel the
structure of a message, which leads to the negative results on the validation set.
Evaluation method
Precision (%)
Recall (%)
F1 Measure (%)
Cross Validation
Validation set
71.28
5.24
92.86
70.00
80.65
9.75
Table 5.2: Performance of Naive Bayes.
5.2.1.2
Conclusion
On a realistic validation set our Naive Bayes performs very pooly. One of the reasons
is off course the simplicity of the NB method, which does not permit to fully take
into account the structure of a message. Another flaw is the fact our validation set
contains too few relevant documents, which makes it hard to produce good results, as
falsely classifying one positive document (or the other way round) has a huge impact on
precision and recall numbers.
5.2.2
Support Vector Machine
The complexity of a Support Vector Machine is very high compared to a Naive Bayes
classifier. This section gives a complete overview, starting from our initial setup and
ending with our final version. We extensively examined multiple factors that possibly
determine or influence the performance of our SVM classifier. This study starts with
showing results that are not based on our final training set, but give a good indication
where and how we improved. By showing the complete process this study tries to give
a comprehensive picture of how we achieve our final results.
5.2.2.1
Initial Setup
As we have already discussed, our support vector machine is implemented based on the
LIBLINEAR library. An L2-regularized logistic regression function is used, our cost
parameter is set to 1.2, epsilon has a value of 0.1 and no bias is used.
5.2.2.2
Evaluation
We evaluate our classifier on the standard validation set. We notice our Support Vector
Machine classifies 45 messages falsely as ’positive’. This results in a precision of only
6%. Both precision and recall numbers are extremely low, resulting in a F1 Measure of
only 8.7%.
50
Automated Detection of Offensive Language Behavior on Social Networking Sites
TP
FP
TN
FN
3
45
1932
18
Precision (%)
Recall (%)
F1 Measure (%)
6.25
14.29
8.70
Table 5.3: Results of classifier after running on validation set. Precision drops to 6% due to the
rate of true positive against false positive.
In the next sections we will try to determine crucial factors that are responsible for
this bad performance.
5.2.2.3
Data Analysis
The first and most obvious action one should attempt to enhance the overall functionality of a supervised classifier is enhancing the training data. To try to limit the amount
of false positives, we extend the irrelevant messages in our training set. However, since
SVMs base their decision on only a small subset of samples (ie. support vectors), we
do not randomly add irrelevant messages. Instead we classify 50, 000 random messages,
originating from our data set and manually analyse all messages that are detected as
offensive. Each false positive is then added to our training set as an irrelevant message.
In this way, we change the decision making process and make sure the hyperplane that
seperates the different categories is modified.
This can be viewed as a form of reinforcement learning (cf. section 3.2.3). The messages
that are correctly classified as offensive are also added to our training set. Table 5.4
displays the results of this classification. Thus, approximately 900 irrelevant messages
are added to the training set together with more or less 150 offensive, predominantly
’sexual’ messages. Not every message is added to our training set since several messages
are similar in terms of content and structure.
Total
Sexual
Racist
Irrelevant
Precision
True positive
False positive
50000
774
438
48788
17.07%
205
996
Table 5.4: Results of classifier after running on randomly selected set.
To estimate the effect of our improved training set, we again test our classifier on
our validation set. Table 5.5 shows the results. We clearly see an improvement, though
the overall performance is still very poor.
51
Automated Detection of Offensive Language Behavior on Social Networking Sites
TP
FP
TN
FN
6
14
1963
15
Precision (%)
Recall (%)
F1 Measure (%)
30.00
28.57
29.27
Table 5.5: Results of classifier after improving the training set.
By repeating this process a number of times, we further enhanced our training set
and eventually attained the set described in section 2.4.3. One other way of brightening
our results is by increasing the amount of relevant messages in our validation set. Since
our validation set contains only a very low amount of relevant messages, one wrongly
classified message has a great impact on the eventual performance measures. Although,
our validation set is then no longer a realistic representation of our data set, we do get
a slightly better image of the functionality of our Support Vector Machine. We, thus,
added a number of relevant, though unknown messages to our validation set, which
increased the total of relevant messages to 29. Table 5.6 shows the final results of the
enhancement in both training as validation set.
TP
FP
TN
FN
15
22
1947
14
Precision (%)
Recall (%)
F1 Measure (%)
40.54
51.72
45.45
Table 5.6: Results of classifier after modifying validation set.
Figure 5.7 gives an overview of the evolution of the performance of our Support
Vector Machine.
Figure 5.7: Evolution of SVM performance on validation set.
Notwithstanding, we managed, whether or not in an artificial way, to increase the
performance of our SVM, the results are still far from satisfying. Therefore, we further
52
Automated Detection of Offensive Language Behavior on Social Networking Sites
analyse different aspects in the following sections.
5.2.2.4
Message Length Analysis
A thorough analysis of messages that have been falsely classified as positive, indicates
that 71.3% contains less than ten features. To get a clear view on the performance
on specifically small messages, our test set is split in two parts based on the size of the
messages. Our test set is split into one part with only messages smaller than a certain
threshold and one part containing only larger messages. SVM clearly performs on a
different level depending on which set it is executed. Figure 5.8 shows the difference in
precision between the two sets.
Figure 5.8: Difference in performance when splitting validation set in a set with small messages
and one with the larger messages.
In Figure 5.9 we see the results regarding precision when splitting the validation set
in two parts based on the number of features in a message. We can clearly see that a
message containing more features has a greater chance of being correctly classified.
53
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 5.9: Snapshot of precision results that shows difference between small validation set (messages that contain less than 20 features) and large validation set (solely messages that at least
contain 20 features).
Figure 5.10: Comparison of SVM performance when removing messages under a certain threshold
in both training and validation set.
Based on the above data we can conclude to eliminate all features under a certain
threshold. Instead of only eliminating these messages in our validation set we also remove
them from our training set. In this way, we avoid these small messages to negatively
affect the classifier when it attempts to build an accurate model.
Figure 5.10 shows the results of our classifier when completely removing messages that
54
Automated Detection of Offensive Language Behavior on Social Networking Sites
do not contain a certain minimum amount of features. We notice a great performance
improvement in both precision and recall. However, as soon as we remove messages that
contain more than ten features the performance drops again. This is due to the fact that
we are throwing away too much information, which results in a less accurate training
model.
We conclude that messages that do not contain at least five features are not suitable to
be handled and are thus ignored completely by our Support Vector Machine.
5.2.2.5
Selection of Features Analysis
One last attempt to improve the performance of our SVM is to apply feature selection.
As described in section 3.2.2, feature selection will try to eliminate noise by filtering
out all useless features and preserve only those that contain the most information. We
applied a method that is named Mutual Information (cf. section 4.2.2). However,
the results were not satisfying as performance did not improve. We decided not to
perform an in-depth analysis to what and why feature selection did not produce any
improvements. As feature selection is an art in itself this would cost too much time and
lead us too far away from the main subject.
5.2.2.6
Conclusion
We conclude our Support Vector Machine, when intensively analyzed and tuned, performs slightly above average. Figure 5.11 shows the overall progress of our SVM.
Figure 5.11: Precision and recall comparison of different stages of our Support Vector Machine
55
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.2.3
Semantic Classifier
In this section we present and compare the results of our semantic classifiers. The
algorithms we designed (cf. Section 4.2.3) were originally meant to detect outrage in
reactions. In this way it was possible to support or alter certain decisions taken by our
primary classifier. However, by also constructing word lists for the other classes the
semantic classifiers are regarded equivalent to our SVM or NB classifier.
5.2.3.1
Initial Setup
We created three classifiers, beginning with a rather simple implementation and ended
with a more advanced one. Each implementation has its own characteristics (cf. section
4.2.3). The simplest classifier detects abusive language too quickly as it attaches too
much belief to abusive words, which have multiple meanings depending on the context.
We hypothesize that this classifier will yield high recall numbers, but a rather low precision. The second classifier, named vital classifier, on the contrary focuses much more
on yielding high precision results, at the cost of recall. The last classifier, named fuzzy
classifier, tries to combine the best parts of the two previous ones.
5.2.3.2
Evaluation
Figure 5.12 shows the results of our different semantic classifiers. As expected our simple
classifier especially performs well regarding to recall, as opposed to our vital classifier
that reaches a much higher precision than the first one. Fuzzy classifier reaches same
precision level as vital classifier, but combines this with a recall of approximately 93%.
Figure 5.12: Comparison of performance of different semantic classifiers on validation set.
56
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.2.3.3
Reactions Classification
This section is devoted to determining the efficiency of classifying messages solely based
on its reactions. In chapter 2, an extensive analysis of the corpus is provided. We
concluded that merely 72% of all messages do not contain any reaction. However, a
message that does contain reactions has on average six (cf. section 2.2). Logically, we
can only take into account messages that do contain reactions for this experiment. Our
validation set is then reduced to 584 messages. However, since semantic methods are not
supervised methods, our training set can be used as a ’validation set’. Since our training
set includes 1416 messages that contain reactions, we decide to temporarily execute our
experiments using this set.
We apply three different methods to classify a message based on its reactions. The first
case focuses on detecting outrage in the reactions, while the second method searches for
reactions that contain abusive language. Thus, we assume that messages containing abusive language will elicit reactions that also contain abusive language. Our third method
combines both methods and attempts to find abusive reactions as well as reactions that
express outrage. Figure 5.13 shows a comparison of these different methods, executed
by using our fuzzy classifier.
Figure 5.13: Classification of messages solely based on the reactions to the message.
Our first as well as our second method achieves a rather high precision, but performs
poorly concerning recall. The combination of both methods achieves reasonable results,
but is not comparable to methods directly classifying the message itself. This seems
normal as reactions to a message does not necessarily behave the same way as the
message itself and are much less predictable.
57
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.2.3.4
Conclusion
We decide to select our fuzzy classifier to function as our final semantic classifier. On
top we conclude that using reactions to classify a message can be useful, but rather in
an assisting function than as a standalone solution.
5.2.4
Comparison of Single Methods
In the above sections we thoroughly analyzed the results of three single methods. For
the sake of completeness Table 5.7 displays the results of Naive Bayes on the modified
validation set.
TP
FP
TN
FN
25
248
1721
4
Precision (%)
Recall (%)
F1 Measure (%)
9.16
86.21
16.56
Table 5.7: Results of Naive Bayes classifier on the modified validation set.
Figure 5.14 combines the best results for each classifier seperately. Naive Bayes
achieves a very high recall, but combines it with the lowest precision. We focus on
our Support Vector Machine classifier and our Semantic Classifier which produce solid
results.
Figure 5.14: Comparison between different single classifiers.
5.2.4.1
Length of Message Analysis
Section 5.2.2.4 handles the fact that SVM has difficulties classifying small messages.
In this section we compare the functionality of both SVM and Fuzzy when applied to
58
Automated Detection of Offensive Language Behavior on Social Networking Sites
small messages. We filter all messages that contain more than eleven features out our
validation set; this eventually leaves 479 messages left in our validation set of which two
relevant.
Table 5.8 displays the results of the experiment. Figure 5.15 shows that fuzzy classifier outperforms SVM by far in terms of precision. These findings are used in the final
algorithm that favors fuzzy classifier when messages contain only few features.
Classifier
SVM
Fuzzy
Accuracy (%)
Precision (%)
Recall (%)
F1 Measure (%)
14.3
100.0
100.0
100.0
25.0
100.0
97.3
100.0
Table 5.8: Results of classifiers running on ’small’ validation set.
Figure 5.15: Comparison of precision of both classifiers running on ’small’ validation set.
5.2.4.2
Conclusion
Since in section 5.2.5, a combination of the SVM and fuzzy classifier is proposed, it is
important to effecively determine the different weaknesses and strengths of both classifiers. We conclude that smaller messages should not be classified by SVM and also notice
that fuzzy classifier yields a very high recall. Thus, when fuzzy classifier indicates the
message does not contain abusive language, there is a high chance the message indeed
is irrelevant.
59
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.2.5
Combined Algorithm
We can state that, albeit SVM and fuzzy classifier achieve ’reasonable’ results, the
overall performance is not satisfactory. Therefore, a combination of seperate classifiers
is designed to improve the functionality of the system. As described in section 4.2.4, we
use both our SVM and our fuzzy classifier to create a more reliable system.
5.2.5.1
Initial Setup
First, we solely focused on the belief each classifier attached to its classification. Thus,
we chose the classifier that got the highest score linked to a certain class. However, since
it is hard to compare the score of two completely different classification systems and a
lot of extra information can be derived from the context we designed a more complex
system. This algorithm is described in section 4.2.4 and depends on four general cases
to base its decision on.
5.2.5.2
Evaluation
Our advanced method achieves following results on our validation set.
TP
FP
TN
FN
23
0
1969
6
Precision (%)
Recall (%)
F1 Measure (%)
100.0
79.3
88.46
Table 5.9: Results of combination of different classifiers on validation set.
Table 5.9 displays the accuracy results that are achieved when classifying our validation set. Both precision and recall have improved significantly.
5.2.5.3
Comparison of Methods
The combination of different seperate classifiers clearly outperforms our single methods
(cf. Figure 5.16). Especially precision benefits from combining our classifiers, and even
reaches to perfection. Off course this is just a result obtained from our validation set. In
a realistic environment precision will not be perfect, but will still yield very high results.
Recall yields a less spectacular result, although outperforms almost all other attempts.
60
Automated Detection of Offensive Language Behavior on Social Networking Sites
Figure 5.16: Comparison of all implemented classification methods.
5.2.6
Corpus Classification
This sections presents the results concerning the classification of our whole data set.
Classifying the whole data set allowes us to estimate the basic functionality of our
classification system.
5.2.6.1
Detected Offensive Messages
Table 5.10 displays the distribution of messages among the different categories after the
classification.
Class
Amount
Percentage (%)
Irrelevant
Sexual
Racist
6,862,870
37,152
5,072
99.39
0.54
0.07
Table 5.10: Distribution of messages among the three classes: irrelevant, sexual and racist.
We notice that our systems detects a total of 42, 224 flame messages. This is equivalent with 0.60% of the whole corpus, while in chapter 2, we stated that 0.85% of the
messages, derived from the amount of offensive messages in our validation set, contains
abusive language. In total 30, 271 different users have posted at least one flame message.
61
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.2.6.2
Influence of Different Classifiers
Table 5.11 denotes the impact of the seperate methods on our final classification. Both
the overall share as the impact solely on the positively classified messages is shown. SVM
as well as fuzzy classifier both return the same results for the most part, which makes
the decision straightforward in most cases. Reactions only alter of modify the decisions
about 0.1% of the time, or 16% when solely focused on positive messages. Apart from
the situation where both classifiers give equal results, SVM is prefered in 53% of the
cases, while 47% is based on fuzzy classifier.
Situation
Same classification result
Reactions modify decision
Small message modifies decision
Other
Global (%)
Positive (%)
96.72
0.10
0.96
1.32
51.45
16.27
4.01
28.27
Table 5.11: Impact of different situations
5.2.7
Conclusion
The performance of our Naive Bayes method, as expected, is greatly outperformed by
both our SVM as our semantic method. This study notes that SVM, when optimally
tuned, achieves a precision of 69% and a recall of 62%. The semantic method, which
is based on word lists, especially achieves a very high recall of 93% and combines this
with a precision of 46%. By combining the information we attain from the semantic
method and the SVM classifier a considerable performance boost is achieved. Primarily,
the main reason of building a method based on word lists was to be able to extract the
information out of reactions. However, since the valued performance of this semantic
method on our validation set, it is considered an adequate classifier. Our final algorithm
achieves a precision of 100% and a recall of approximately 79%. All these tests are
performed on our modified validation set, which is our initial validation set with an
increased number of relevant documents.
62
Chapter 6
Discussion
This next to last chapter briefly summarizes the results obtained in our experiments.
It also tries to further describe and explain the different reasons why we obtain such
results. Finally we suggest possible enhancements and future improvements.
6.1
Main Findings
This section will give an overview of the main accomplishments and findings in this
study.
6.1.1
Corpus
The corpus, extracted from the social networking site Netlog, contained approximately
seven million messages. By extracting 2000 random messages out of the corpus, we
intend to
(a) create a standard validation set to perform our experiments on, and
(b) get a global idea of the number of relevant messages in the corpus.
The validation set contains two ’racist’ messages and fifteen ’sexual’ messages. A set
containing only a very low amount of relevant messages has the effect that it is hard
to properly estimate the performance and functionality of a classifier. Therefore, the
number of relevant messages is artificially increased in the validation set. Anew, this is
done by randomly selecting approximately 2000 messages, although now solely extracting
the relevant messages. Randomly chosen irrelevant messages in our standard validation
set are then swapped with the retrieved relevant messages.
6.1.2
Development
In the first stage, an information retrieval platform is set up in order to effectively retrieve
relevant documents. The system is expanded by implementing Rocchio query expansion,
63
Automated Detection of Offensive Language Behavior on Social Networking Sites
which enhances the IR process. By using query expansion the most informative words
are extracted from the relevant documents and added to the query. This method should
not only produce more relevant documents, but also lists the most important words that
characterise the relevant documents. A possibility to save these words and construct a
thesaurus is also implemented.
The second stage contained of implementing a supervised classifier to be able to detect offensive language. Two supervised classifiers, Naive Bayes and Support Vector
Machine, were created. Subsequently a ’semantic’ classifier based on word lists was constructed, increasing reliability at the cost of flexibility.
Finally, a combination of our SVM and semantic classifier were designed. Analysis
of the outcome of these different classfiers on the validation set points out that over
90% of the samples is correctly classified by one out of two or both classifiers. Based
on the context and the strengths and weaknesses of both classifiers, a final classification
method is developed.
6.1.3
Experimental Study
This section gives an overview of the main experimental results.
Query Expansion
We can state that rocchio query expansion, or query expansion in general, is very prone
to query drifts. Even with manual annotation of relevant documents, filtering irrelevant
terms out of the expanded query is required. However, we did prove that using an
expanded query as opposed to its base query, increases recall as well as precision. On
average, recall increased round 20% when expanding both query one and two. Especially
when creating one large query composed from different single queries, we noticed an
improvement in terms of recall. The average precision also increased when expanding
queries, but depends on the way these new relevant messages are ranked. Due to this
fact, precision increases when expanding a query that provides highly ranked relevant
messages. When relevant messages are more scattered precision will likewise be lower.
Offensive Language Detection
Our flame detection mechanism starts with developing different independant classifiers.
We start with a simple, but widely used classifier Naive Bayes. This classifier however
performs rather poorly on our realistic validation set. Albeit, an impressive recall is
achieved, only 9% of messages positively classified are in fact offensive messages. This
off course completely nullifies the high recall numbers.
The second classifier that is implemented is a Support Vector Machine, based on the
LibLinear library. We, at first do not produce any more significant results, as opposed
64
Automated Detection of Offensive Language Behavior on Social Networking Sites
to our Naive Bayes classifier. However, after extending and enhancing our training set
in combination with our modified validation set (cf. section 6.1.1), we manage to obain
reasonable results. The SVM classifier higly depends on the number of features a message contains. This can be clearly inferred from an analysis on the falsely detected flame
messages, of which 70% contains less than ten features. SVM precision is negatively
influenced by messages that do not contain a minimum amount of features. Another
method to enhance our SVM is by filtering features in such a way that only meaningful
features are kept. However, the implemented filter, based on mutual inclusive information, does not produce the expected results. Therefore, we do not apply any feature
selection method. Eventually our SVM reaches, in a best case scenario, a precision of
almost 70% and a recall of 62%. This result is obtained when ignoring messages that
do not contain more than five features. Since very small messages can contain abusive
language as much as large messages, this is considered a flaw.
Because of the limited performance of the SVM, we created another classifier, named semantic classifier, which is based on word lists. These methods are less dynamic and have
much less learning capabilities than supervised methods, though are more trustworthy.
We mainly designed such method to incorporate information provided by reactions in
our classification system. However, we notice our semantic method performs solid on
our validation set, with an especially high recall of 93%.
Finally, different classifiers are combined to create one efficient classification system.
This combined system achieves a precision of 100% on our validation set. Albeit, we
may conclude that this is an excellent result, at the same time should be noticed that
this validation set is not a perfect representation of the whole dataset.
Offensive Language Behavior
The final goal of this study is not only to detect offensive language, but to globally be
able to detect offensive language behavior. This means we intend to identify users that
deliberately post offensive messages. Due to timing constraints we were unable to fully
develop a detailed method to decide whether or not a user should get a label of offensive
language behavior. However, a classification of the complete data set was performed in
order to get an idea of the number of offensive messages and the distribution among the
different users. We note that approximately 30% less messages have been classified as
flames than initially expected when analyzing the number of relevant messages in the
validation set.
6.2
Discussion
This sections handles multiple concerns related to the retrieved results supported with
some examples.
65
Automated Detection of Offensive Language Behavior on Social Networking Sites
6.2.1
Domain-dependency
As described in section 1.5, domain-dependency is one of the biggest issues in the text
classification domain. This issue has become even more relevant since our combined
method partly supports on static word lists. Although, we attempted to add flexiblity
to these word lists, by automatically proposing new words, this has not been extensively
tested. Also, it is necessary to estimate a proper intensity and centrality value for
each word. These elements combined make it hard to estimate the performance of our
classification system on a completely different corpus.
6.2.2
Implicit Language
When messages contain implicit language, it is very difficult to properly decide if a
message should be defined as offensive. One example message that is falsely classified as
sexual is displayed underneath.
Megan Fox: “Ik voel me net een hoer”.
“Als actrice voel ik me een hoer.” De uitspraak komt van de verrukkelijke Megan Fox, onlangs nog verkozen tot meest sexy vrouw. Fox
vindt eigenlijk alle acteurs en actrices een soort van prositituees. ”Het
is vies“ Als je erover na denkt zijn we ...
To detect such messages more advanced techniques should be used (eg. taking quotes
under consideration, etc). Another example is shown below that brings up another issue
concerning our system. Since the main focus is on detection of offensive chat language,
informative and formal messages often are classified wrong when they appear to proclaim
offensive language, but in fact are just communicating about latest news facts, etc.
Poolse nachtclub gebruikt foto van Hitler om volk te lokken.
Een Poolse nachtclub is op de vingers getikt omdat ze een foto van
Adolf Hitler gebruikte om volk te lokken. Op de prent poseert de
nazileider met zonnebril, terwijl onder hem een adelaar van het Derde
Rijk afgebeeld staat.
6.2.3
Conclusion
In general can be stated that detecting offensive language is a very subjective and challenging domain. It is clear that such a system is a good helping tool, but is not suited
as a standalone application.
6.3
Future Improvements
We intended to build a dynamic and flexible system that can easily incorporate new categories. In order to add a new class, two key additions need to be constructed. First, a
66
Automated Detection of Offensive Language Behavior on Social Networking Sites
representative document set that is descriptive for a specific category should be gathered
and added to the general training set. Secondly, a word list needs to be created. It is
possible to partly extract the most informative words from the labeled set by applying
query expansion.
However, our system does lack the ability to improve and evolve together with the
changing behavior of offensive language, in particular with chat language. A potential
improvement would be to automatically suggest new words to our word list, concomitant
with automatically adjusting the values of already existing words. Notwithstanding we
implemented the ability to extend our word lists with new words that often arise in flame
messages, this module has not been extensively tested.
To gradually be able to adjust the behavior of the classifier, reinforcement learning
could be a helpful improvement. With feedback on certain messages, the training set
or word lists could be altered accordingly and thus could be adjusted to comply to new
trends and practices.
67
Chapter 7
Conclusion
7.1
Main Conclusions
The outset of this thesis states that we intend to detect offensive language behavior in an
automated manner. This study outlines two main techniques being query expansion and
text categorisation. The first technique is used in order to effectively retrieve relevant
documents out of the corpus, while the second technique is utilized to seperate irrelevant
messages from messages containing racist or sexual abusive language.
This corpus contains just over seven million messages, but relevant messages only occur
rarely. Analysis reveal that only 0.85% contains offensive language. Our results show
that by using query expansion we are able to increase recall and, when not using too
large queries, also precision.
Based on the constructed training set we design two supervised learning methods. The
first method, being Naive Bayes, performs rather poorly and achieves a precision of
9% and a high recall of 86%. This high recall is negated since precision is very low.
The second supervised classifier is a Support Vector Machine and is more extensively
studied as opposed to Naive Bayes. SVM achieves a reasonable 69% in terms of precision and 62% recall. However, messages that do not contain more than five features
are ignored, since SVM does not produce good results when small messages are classified.
To enhance overall results we intended to incorporate reactions to a message in our
classification process. In this way, we can take into account the possible information
reactions contain to support a possible decision on the message itself. Our study attempts to find reactions that express outrage or contain abusive language on their own.
In order to achieve this a classifier, based on prefabricated word lists, was created. This
’semantic’ classifier achieves a very high recall of 93% and a reasonable precision of 46%
on our standard validation set. Further analysis reveals that reactions only impact 0.1%
of the total decisions.
69
Automated Detection of Offensive Language Behavior on Social Networking Sites
Lastly, to improve the overall results a combination of methods is designed. This
combined method greatly improves performance of the classification system, since it
is capable of using the strengths of both SVM as semantic classifier and avoiding their
weaknesses. By using this method we managed to combine a perfect precision of 100%
with a solid recall of 79%. Although, these results are decent, they are not representative
for the overall corpus and certainly not for other corpora.
Offensive language detection is a groundbreaking stage that can be of great aid to automate certain aspects of language monitoring. The fact remains that up untill today
these machines are not able to truly understand the real meaning and emotions language
expresses.
70
Bibliography
[1] T. Saracevic B. Jansen, A. Spink. Real life, real users, and real needs: A study
and analysis of user queries on the web. Information Processing and Management,
36(2):207–227, 1996.
[2] Janyce Wiebe Theresa Wilson Rebecca Bruce Matthew Bell and Melanie Martin.
Learning subjective language. Computational Linguistics, 30:3:277–308, 2004.
[3] John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, bollywood, boomboxes and blenders: Domain adaptation for sentiment classification. 2007.
[4] Danah M. Boyd and Nicole B. Ellison. Social network sites: Definition, history, and
scholarship. Journal of Computer-Mediated Communication, 13(1):210–230, 2007.
[5] John Broglio, James P. Callan, and W. Bruce Croft. Inquery system overview. In
Proceedings of a workshop on held at Fredericksburg, Virginia: September 19-23,
1993, TIPSTER ’93, pages 47–67. Association for Computational Linguistics, 1993.
[6] Kevyn Collins-Thompson and Jamie Callan. Query expansion using random walk
models. In Proceedings of the 14th ACM international conference on Information
and knowledge management, CIKM ’05, pages 704–711. ACM, 2005.
[7] Steve Cronen-townsend, Yun Zhou, and W. Bruce Croft. A framework for selective query expansion. In In Proceedings of Thirteenth International Conference on
Information and Knowledge Management, pages 236–237. Press, 2004.
[8] Silviu Cucerzan and Eric Brill. Spelling correction as an iterative process that
exploits the collective knowledge of web users. In Proceedings of EMNLP, pages
293–300, 2004.
[9] Hang Cui, Ji-Rong Wen, Jian-Yun Nie, and Wei-Ying Ma. Probabilistic query
expansion using query logs. In Proceedings of the 11th international conference on
World Wide Web, WWW ’02, pages 325–332. ACM, 2002.
[10] H. Turtle D. Metzler, T. Strohman and W. B. Croft. Indri at trec 2004: terabyte
track. In Text Retrieval Conference, 2004.
71
Automated Detection of Offensive Language Behavior on Social Networking Sites
[11] Kushal Dave, Steve Lawrence, and David M. Pennock. Mining the peanut gallery:
opinion extraction and semantic classification of product reviews. In Proceedings of
the 12th international conference on World Wide Web, WWW ’03, pages 519–528,
New York, NY, USA, 2003. ACM.
[12] S. T. Dumais. Latent semantic indexing: Trec-3. In Proceedings of the Text REtrieval Conference (TREC-3), page 219. ACM, 1995.
[13] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. Liblinear: A
library for large linear classification. Journal of Machine Learning Research, 9:1871–
1874, 2008.
[14] George Forman. An extensive empirical study of feature selection metrics for
textclassification. J. Mach. Learn. Res., 3:1289–1305, March 2003.
[15] D Garrett, DA Peterson, CW Anderson, and MH Thaut. Comparison of linear,
nonlinear, and feature selection methods for eeg signal classification. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11:141 – 144, 2003.
[16] Andrew R. Golding and Dan Roth. A winnow-based approach to context-sensitive
spelling correction. Machine Learning, 34:107–130, 1999.
[17] Andrew R. Golding and Yves Schabes. Combining trigram-based and feature-based
methods for context-sensitive spelling correction. In Proceedings of the 34th annual meeting on Association for Computational Linguistics, ACL ’96, pages 71–78,
Stroudsburg, PA, USA, 1996. Association for Computational Linguistics.
[18] Andrew Gordon, Abe Kazemzadeh, Anish Nair, and Milena Petrova. Recognizing
expressions of commonsense psychology in english text. In Proceedings of the 41st
Annual Meeting on Association for Computational Linguistics - Volume 1, ACL
’03, pages 208–215, Stroudsburg, PA, USA, 2003. Association for Computational
Linguistics.
[19] Isabelle Guyon and André Elisseeff. An introduction to variable and feature selection. J. Mach. Learn. Res., 3:1157–1182, March 2003.
[20] Masoud Rahgozar Hadi Amiri, Abolfazl Ale Ahmad and Farhad Oroumchian. Query
expansion using wikipedia concept graph. 2008.
[21] Marti A. Hearst and Jan O. Pedersen. Reexamining the cluster hypothesis: Scatter/gather on retrieval results. pages 76–84, 1996.
[22] Thorsten Joachims. Text categorization with support vector machines: Learning
with many relevant features. In Claire Ndellec and Cline Rouveirol, editors, Machine
Learning: ECML-98, volume 1398 of Lecture Notes in Computer Science, pages
137–142. Springer Berlin / Heidelberg, 1998. 10.1007/BFb0026683.
72
Automated Detection of Offensive Language Behavior on Social Networking Sites
[23] Lee Rainie Keith N. Hampton, Lauren Sessions Goulet and KristenPurcell. Social
networking sites and our lives. 2010.
[24] S. Kotsiantis, I. Zaharakis, and P. Pintelas. Machine learning: a review of classification and combining techniques. Artificial Intelligence Review, 26:159–190, 2006.
10.1007/s10462-007-9052-3.
[25] Karen Kukich. Techniques for automatically correcting words in text. ACM Comput. Surv., 24(4):377–439, December 1992.
[26] T. K Landauer and S. Dumais. Latent semantic analysis. Scholarpedia, 3(8):4356,
2008.
[27] Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth
International Conference on Machine Learning, pages 331–339, 1995.
[28] Victor Lavrenko and W. Bruce Croft. Relevance based language models. In Proceedings of the 24th annual international ACM SIGIR conference on Research and
development in information retrieval, SIGIR ’01, pages 120–127. ACM, 2001.
[29] Y. H. Li and A. K. Jain. Classification of text documents. The Computer Journal,
41(8):537–546, 1998.
[30] Yinghao Li, Wing Pong Robert Luk, and Fu Lai Korris Ho, Kei Shiu Edward andChung. Improving weak ad-hoc queries using wikipedia asexternal corpus. In Proceedings of the 30th annual international ACM SIGIR conference on Research and
development in information retrieval, SIGIR ’07, pages 797–798. ACM, 2007.
[31] Wei-Yin Loh. Classification and regression trees. Wiley Interdisciplinary Reviews:
Data Mining and Knowledge Discovery, 1(1):14–23, 2011.
[32] Robert M. Losee. How part-of-speech tags affect text retrieval and filtering performance. CoRR, cmp-lg/9602001, 1996.
[33] Julie B Lovins. Development of a stemming algorithm. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE ELECTRONIC SYSTEMS LAB,
1968.
[34] Kazi Zubair Ahmed M. K. Altaf Mahmud. Detecting flames and insults in text.
In In Proceedings of 6th International Conference on Natural Language Processing,
2008.
[35] I. Maglogiannis, K. Karpouzis, B.A. Wallace, and J. Soldatos, editors. Emerging Artificial Intelligence Applications in Computer Engineering - Real Word AI
Systems with Applications in eHealth, HCI, Information Retrieval and Pervasive
Technologies. IOS Press, 2007.
[36] Christopher D. Manning and Hinrich Schtze. An Introduction to Information Retrieval, pages 177–185. Cambridge University Press, 2009.
73
Automated Detection of Offensive Language Behavior on Social Networking Sites
[37] Christopher D. Manning and Hinrich Schtze. An Introduction to Information Retrieval, pages 272–273. Cambridge University Press, 2009.
[38] Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a
large annotated corpus of english: the penn treebank. Comput. Linguist., 19(2):313–
330, June 1993.
[39] Andrew McCallum and Kamal Nigam. A comparison of event models for naive
bayes text classification. Dimension Contemporary German Arts And Letters,
752(1):4148, 1998.
[40] George A. Miller. WordNet: A Lexical Database for English., volume 38. ACM,
1995.
[41] Ruslan Mitkov, editor. The Oxford Handbook Of Computational Linguistics. Oxford
University Press, 2005.
[42] David L. Olson and Dursun Delen. Advanced Data Mining Techniques, page 138.
Springer Publishing Company, Incorporated, 1st edition, 2008.
[43] Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and
Douglas Johnson. Terrier information retrieval platform. In David Losada and
Juan Fernndez-Luna, editors, Advances in Information Retrieval, volume 3408 of
Lecture Notes in Computer Science, pages 517–519. Springer Berlin / Heidelberg,
2005.
[44] Bo Pang and Lillian Lee. Opinion mining and sentiment analysis. Found. Trends
Inf. Retr., 2(1-2):1–135, January 2008.
[45] Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up?: sentiment
classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing - Volume 10, EMNLP
’02, pages 79–86, Stroudsburg, PA, USA, 2002. Association for Computational Linguistics.
[46] Fuchun Peng, Nawaaz Ahmed, Xin Li, and Yumao Lu. Context sensitive stemming for web search. In Proceedings of the 30th annual international ACM SIGIR
conference on Research and development in information retrieval, SIGIR ’07, pages
639–646, New York, NY, USA, 2007. ACM.
[47] Yonggang Qiu and Hans-Peter Frei. Concept based query expansion. In Proceedings
of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’93, pages 160–169. ACM, 1993.
[48] Amir Razavi, Diana Inkpen, Sasha Uritsky, and Stan Matwin. Offensive language
detection using multi-level classification. In Advances in Artificial Intelligence, volume 6085 of Lecture Notes in Computer Science, pages 16–27. Springer Berlin /
Heidelberg, 2010.
74
Automated Detection of Offensive Language Behavior on Social Networking Sites
[49] Jason D. M. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger. Tackling the poor assumptions of naive bayes text classifiers. In In Proceedings of the
Twentieth International Conference on Machine Learning, pages 616–623, 2003.
[50] Philip Resnik. Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language. CoRR,
abs/1105.5444, 2011.
[51] Luke Richards. Social media in asia: Understanding the numbers., May 2012.
[52] Ellen Riloff and Janyce Wiebe. Learning extraction patterns for subjective expressions. In Proceedings of the 2003 conference on Empirical methods in natural
language processing, EMNLP ’03, pages 105–112, Stroudsburg, PA, USA, 2003.
Association for Computational Linguistics.
[53] Ellen Riloff, Janyce Wiebe, and Theresa Wilson. Learning subjective nouns using
extraction pattern bootstrapping. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4, CONLL ’03, pages 25–32,
Stroudsburg, PA, USA, 2003. Association for Computational Linguistics.
[54] I Rish. An empirical study of the naive bayes classifier. IJCAI 2001 Workshop on
Empirical Methods in Artificial Intelligence, 3(22):4146, 2001.
[55] J. J. Rocchio. The SMART Retrieval System: Experiments in Automatic Document
Processing, pages 313–323. Prentice Hall, 1971.
[56] C. Romm-Livermore and K. Setzekorn. Social Networking Communities and EDating Services: Concepts and Implications. IGI Global, 2008.
[57] Neil Rubens. The application of fuzzy logic to the construction of the ranking function of information retrieval systems. Computer Modelling and New Technologies,
10(1):20–27, 2006.
[58] Sarah Schrauwen. Machine learning approaches to sentiment analysis using the
dutch netlog corpus. Master’s thesis, University of Antwerp, 2010.
[59] Frdrique Segond, Anne Schiller, Gregory Grefenstette, and Jean pierre Chanod. An
experiment in semantic tagging using hidden markov model tagging. In ACL/EACL
Workshop on Automatic Information Extraction and Building of Lexical Semantic
Resources for NLP Applications, pages 78–81, 1997.
[60] Ellen Spertus. Smokey: Automatic recognition of hostile messages. In In Proc.
IAAI, pages 1058–1065, 1997.
[61] Pero Subasic and Alison Huettner. Affect analysis of text using fuzzy semantic
typing. In The Ninth IEEE International Conference on Fuzzy Systems. FUZZ
IEEE 2000., volume 2, pages 647– 652, 2000.
75
Automated Detection of Offensive Language Behavior on Social Networking Sites
[62] Luis Talavera. An evaluation of filter and wrapper methods for feature selection
in categorical clustering. In A. Famili, Joost Kok, Jos Pea, Arno Siebes, and
Ad Feelders, editors, Advances in Intelligent Data Analysis VI, volume 3646 of
Lecture Notes in Computer Science, pages 742–742. Springer Berlin / Heidelberg,
2005.
[63] Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipes
3rd Edition: The Art of Scientific Computing, chapter 16, pages 883–885. William
H. Press, 2007.
[64] Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. Numerical Recipes
3rd Edition: The Art of Scientific Computing, chapter 16, page 890. William H.
Press, 2007.
[65] Mike Thelwall. Fk yea i swear: cursing and gender in myspace. Corpora, 3(1):83–
107, 2008.
[66] Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res., 2:45–66, March 2002.
[67] Ellen M. Voorhees. Query expansion using lexical-semantic relations. In Proceedings
of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR ’94, pages 61–69. Springer-Verlag New York,
Inc., 1994.
[68] Jinxi Xu and W. Bruce Croft. Query expansion using local and global document
analysis. In Proceedings of the 19th annual international ACM SIGIR conference on
Research and development in information retrieval, SIGIR ’96, pages 4–11. ACM,
1996.
[69] Jinxi Xu and W. Bruce Croft. Corpus-based stemming using cooccurrence of word
variants. ACM Trans. Inf. Syst., 16(1):61–81, January 1998.
[70] Jinxi Xu and W. Bruce Croft. Improving the effectiveness of information retrieval
with local context analysis. ACM Trans. Inf. Syst., 18:79–112, January 2000.
[71] Zhi Xu and Sencun Zhu. Filtering offensive language in online communities using
grammatical relations. In Seventh annual Collaboration, Electronic messaging, AntiAbuse and Spam Conference, 2010.
[72] Jeonghee Yi, Tetsuya Nasukawa, Razvan Bunescu, and Wayne Niblack. Sentiment
analyzer: Extracting sentiments about a given topic using natural language processing techniques. In In IEEE Intl. Conf. on Data Mining (ICDM, pages 427–434,
2003.
[73] Hong Yu and Vasileios Hatzivassiloglou. Towards answering opinion questions: separating facts from opinions and identifying the polarity of opinion sentences. In
76
Automated Detection of Offensive Language Behavior on Social Networking Sites
Proceedings of the 2003 conference on Empirical methods in natural language processing, EMNLP ’03, pages 129–136, Stroudsburg, PA, USA, 2003. Association for
Computational Linguistics.
[74] M. Zhu. Recall, precision and average precision. Technical report, University of
Waterloo, 2004.
[75] Liron Zighelnic and Oren Kurland. Query-drift prevention for robust query expansion. In Proceedings of the 31st annual international ACM SIGIR conference on
Research and development in information retrieval, SIGIR ’08, pages 825–826, New
York, NY, USA, 2008. ACM.
77
List of Figures
2.1
2.2
Distribution of messages containing offensive language. . . . . . . . . . . . 15
Distribution of different classes in training set. . . . . . . . . . . . . . . . 16
3.1
3.2
Example of Rocchio classification. Obtained from [36]. . . . . . . . . . . . 22
SVM hyperplane construction in the idealized case of linearly separable
data. Obtained from [63]. . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Feature vectors are mapped from a two-dimensional space to a threedimensional embedding space. Obtained from [64]. . . . . . . . . . . . . . 32
3.3
4.1
Graphical representation of this study’s approach. . . . . . . . . . . . . . 37
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
Recall Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Precision Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Precision-Recall Comparison . . . . . . . . . . . . . . . . . . . . . . . .
Average Precision Comparison. . . . . . . . . . . . . . . . . . . . . . . .
Number of relevant documents when retrieved 1000 documents . . . . .
Average Precision Comparison . . . . . . . . . . . . . . . . . . . . . . .
Evolution of SVM performance on validation set. . . . . . . . . . . . . .
Difference in performance when splitting validation set in a set with small
messages and one with the larger messages. . . . . . . . . . . . . . . . .
Snapshot of precision results that shows difference between small validation set (messages that contain less than 20 features) and large validation
set (solely messages that at least contain 20 features). . . . . . . . . . .
Comparison of SVM performance when removing messages under a certain
threshold in both training and validation set. . . . . . . . . . . . . . . .
Precision and recall comparison of different stages of our Support Vector
Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Comparison of performance of different semantic classifiers on validation
set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Classification of messages solely based on the reactions to the message. .
Comparison between different single classifiers. . . . . . . . . . . . . . .
Comparison of precision of both classifiers running on ’small’ validation
set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.9
5.10
5.11
5.12
5.13
5.14
5.15
79
.
.
.
.
.
.
.
45
46
46
47
48
48
52
. 53
. 54
. 54
. 55
. 56
. 57
. 58
. 59
Automated Detection of Offensive Language Behavior on Social Networking Sites
5.16 Comparison of all implemented classification methods. . . . . . . . . . . . 61
80
List of Tables
2.1
2.2
Distribution of validation set. . . . . . . . . . . . . . . . . . . . . . . . . . 14
Distribution of different classes in training set. . . . . . . . . . . . . . . . 15
4.1
Random entries from our word lists. . . . . . . . . . . . . . . . . . . . . . 39
5.1
5.2
5.3
. 47
. 50
Expanded query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Performance of Naive Bayes. . . . . . . . . . . . . . . . . . . . . . . . .
Results of classifier after running on validation set. Precision drops to 6%
due to the rate of true positive against false positive. . . . . . . . . . . .
5.4 Results of classifier after running on randomly selected set. . . . . . . .
5.5 Results of classifier after improving the training set. . . . . . . . . . . .
5.6 Results of classifier after modifying validation set. . . . . . . . . . . . .
5.7 Results of Naive Bayes classifier on the modified validation set. . . . . .
5.8 Results of classifiers running on ’small’ validation set. . . . . . . . . . .
5.9 Results of combination of different classifiers on validation set. . . . . .
5.10 Distribution of messages among the three classes: irrelevant, sexual and
racist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.11 Impact of different situations . . . . . . . . . . . . . . . . . . . . . . . .
81
.
.
.
.
.
.
.
51
51
52
52
58
59
60
. 61
. 62