Scaling multi-class Support
Vector Machines using interclass confusion
Author:Shantanu
Sunita Sarawagi
Soumen Chakrabarti
Advisor:Dr Hsu
Graduate:ching-wen Hong
1
Content







1.Motivation
2.Objective
3.Introduction: (1).SVM (2).Using SVM to
solve multi-class problems. (3).Present a
method in this paper.
4.OUR APPROACH (1).Hierarchical
Approach (2).The GraphSVM algorithm
5.Experimental evaluation
6.Conclusion
7.Personal opinion
2
Motivation

Solve multi-class problems.
3
Objective




SVM excel at two-class discrinative
learning problems. The accuracy of SVM is
high.
SVM is difficult to solve multi-class
problems. Because training time is long.
The naïve Bayes(NB) classifier is much
faster than SVM in training time.
We propose a new technique for multi-way
classification which exploits the accuracy
of SVM and the speed of NB classifiers.
4
Introduction






1.SVM:
Input: a training set S=｛（x1,y1）,…,
（xN,yN）｝,xi is a vector, yi=1,-1
Output: a classifier f（x）=W．X+b
For example: Medical diagnosis
Xi =（age,sex,blood,…,genome,…）
Yi indicates the risk of cancer.
5
1.Linear SVM
6
Linear SVM
7
Linear SVM
8
Linear SVM
9
Linear SVM
10
2.Using SVM to solve multi-class
problems.



1. “one-vs-others” approach
For each of the N classes, We
construct a one-others (yes/no) SVM
for that class alone.
The winning SVM is the one which
says yes, and whose margin is
largest among all SVMs.
11
Using SVM to solve multi-class
problems



2.Accumulated votes approach
To construct SVMs between all
possible pairs of classes.
The winning class has the largest
number of accumulated votes.
12
3.Present a method in this paper.



1.Using scalability of NB classifiers
w.r.t. number of classes and
accuracy of SVMs.
The first stage :Using multi-class NB
classifier to a confusion matrix.
The second stage :Using SVM with
the “one-vs-others” approach.
13
OUR APPROACH

Confusion matrix: using NB and held-out
validation dataset.
14
OUR APPROACH
15
Hierarchical Approach


Top-level( L1) classifier(NB or SVM)
to discriminate amongst the top-level
clusters of labels.
Second-level(L2) we build multi-class
SVMs within each cluster of classes.
16
Evaluation of the hierarchical
approach





We compare four methods:
MCNB(one-vs-others)
MCSVM(one-vs-others)
Hier-NB (L1:NB,L2:NB),
Hier-SVM (L1:NB,L2:SVM)
17
Evaluation of the hierarchical
approach
18
Evaluation of the hierarchical
approach
19
Evaluation of the hierarchical
approach




NB-L2( 89.01%),combining with the NBL1 (93.56%),Hier-NB (83.28%),MCNB
(85.27%)
SVM-L2 with NB-L1(92.04%), HierSVM(86.12%),MCSVM(89.66%)
The main reason for the low accuracy of
the hierarchical approaches is the
compounding of errors at the two levels.
This led us to design a new algorithm
GraphSVM.
20
The GraphSVM algorithm




1.The confusion matrix obtained by a fast
multi-class NB classifier M1,
For each class i,F(i)={mis-classified as
class i },a threshold t% mis-classified.
In Figure1 ,
I=alt.atheism,t=3%,F(alt.atheism)={talk.
religion.misc,soc.religion.christian}.
2.Train a multi-class classifier M2(i) to
distinguish among the class{i}U F{i}.
21
.Experimental evaluation



1.Datasets
20-newsgroups:18828 news wire
articles from 20 Usenet group.We
randomly chose 70% of the
documents for training and 30% for
testing.
Reuter-21578:135 classes,8819
training documents and 1887 test
documents.
22
Overall comparison
23
Scalability with number of
classes
24
Scalability with number of
classes
25
Scalability with training set size
26
Effect of the threshold
parameter
27
Conclusion



GraphSVM is accurate and efficient in
multi-classes problem.
GraphSVM outerforms SVMs w.r.t. training
time and memory requirements.
GraphSVM is very simple to understand
and requires negligible coding,but it is
useful to deal with very large
classifiers(ten of thousands of classses
and millions of instances).
28
Personal opinion


GraphSVM may be worse is high
positive value of the threshold t.
It is nice that the accurate of
GraphSVM can not affected by the
threshold t.
29