Joint Models of Disagreement
and Stance in Online Debate
Dhanya Sridhar,1 James Foulds,1 Bert Huang,2
1
1
Lise Getoor, Marilyn Walker
University of California, Santa Cruz
2
Virginia Institute of Technology
1
1
Online Debate Forums
• Social media sites for debating issues
• Valuable resources for:
–
–
–
–
Argumentation
Dialogue
Sentiment
Opinion mining
2
CreateDebate.org
3
CreateDebate.org
Debate topic
4
CreateDebate.org
Debate topic
Posts
5
CreateDebate.org
Debate topic
Posts
Replies
6
CreateDebate.org
Debate topic
Posts
Replies
Reply polarity
7
4Forums.com
8
4Forums.com
Quotation
9
Online Debate Forums
Graph of posts:
tree structure
Graph of users:
= reply
link
loopy
structure
10
Online Debate Forums
Graph of posts:
tree structure
Graph of users:
loopy structure
11
Zoom in on an Example
12
Zoom in on an Example
13
Zoom in on an Example
I believe Obama
is likely the worst!
He’s been
infinitely more
effective than
Bush!
14
Task: Stance Classification
I believe Obama
is likely the worst!
He’s been
infinitely more
effective than
Bush!
Anti Obama
Pro Obama
Useful for advocacy and get-out-the-vote campaigns
15
Task: Stance Classification
I believe Obama
is likely the worst!
Posts express stance
He’s been
infinitely more
effective than
Bush!
Anti Obama
Pro Obama
Study argumentation and dialogue
16
Task: Disagreement Classification
I believe Obama
is likely the worst!
Disagree on stance
He’s been
infinitely more
effective than
Bush!
Anti Obama
Pro Obama
Study argumentation and dialogue
17
Task: Disagreement Classification
Posts express disagreement
I believe Obama
is likely the worst!
Disagree on stance
He’s been
infinitely more
effective than
Bush!
Anti Obama
Pro Obama
Study argumentation and dialogue
18
Classification Targets
• Stance
• Author-level
• Post-level
Stance
Stance
• Disagreement
• Author-level
• Post-level
• Textual
Stance
Stance
19
Classification Targets
Disagrees
• Stance
• Author-level
• Post-level
Disagrees
Stance
Stance
• Disagreement
• Author-level
• Post-level
• Textual
Disagrees
Stance
Disagrees
Stance
20
Classification Targets
Disagrees
• Stance
• Author-level
• Post-level
Disagrees
Stance
Stance
• Disagreement
• Author-level
• Post-level
• Textual
Disagrees
Stance
Disagrees
Stance
21
Related Work
• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACL
Collective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLP
Collective stance sequence labeling with CRF; benefits of user
stance consistency constraint
• Abbott et. al (2012), ACL
Local disagreement classifier using linguistic features
• Burfoot et. al (2011), ACL
Joint classification of stance and reply polarity in Congressional
debates
22
Related Work
• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACL
Collective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLP
Collective stance sequence labeling with CRF; benefits of user
stance consistency constraint
• Abbott et. al (2012), ACL
Local disagreement classifier using linguistic features
• Burfoot et. al (2011), ACL
Joint classification of stance and reply polarity in Congressional
debates
23
Related Work
• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACL
Collective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLP
Collective stance sequence labeling with CRF; benefits of user
stance consistency constraint
• Abbott et. al (2012), ACL
Local disagreement classifier using linguistic features
• Burfoot et. al (2011), ACL
Joint classification of stance and reply polarity in Congressional
debates
24
Related Work
• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACL
Collective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLP
Collective stance sequence labeling with CRF; benefits of user
stance consistency constraint
• Abbott et. al (2012), ACL
Local disagreement classifier using linguistic features
• Burfoot et. al (2011), ACL
Joint classification of stance and reply polarity in Congressional
debates
25
Related Work
• Walker et. al (2012), Decision Support Sciences
State-of-the-art local stance classifier using linguistic features
• Walker et. al (2012), NAACL
Collective stance classification using MaxCut over rebuttal links
• Hasan and Ng. (2013), EMNLP
Collective stance sequence labeling with CRF; benefits of user
stance consistency constraint
• Abbott et. al (2012), ACL
Local disagreement classifier using linguistic features
• Burfoot et. al (2011), ACL
Joint classification of stance and reply polarity in Congressional
debates
26
Stance Classification:
“Teach the Controversy”
• Previous work employs many modeling strategies
• How best to model stance in online debate?
• Answers may be different to Congressional debates
– Links have different semantics
– Posts much shorter than speeches. Many posts per author
– Dialogue is informal
27
Stance Classification:
“Teach the Controversy”
• Previous work employs many modeling strategies
• How best to model stance in online debate?
• Answers may be different to Congressional debates
– Links have different semantics
– Posts much shorter than speeches. Many posts per author
– Dialogue is informal
28
Modeling Question 1)
Modeling at author-level or post-level?
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
[Hasan and Ng 2013]
[Other Related Work]
Stance
29
Modeling Question 1)
Modeling at author-level or post-level?
Stance
Stance
Stance
Stance
Stance
Stance
Stance
Stance
[Hasan and Ng 2013]
[Other Related Work]
Stance
30
Modeling Question 2)
Collective classification vs. local classification?
Stance
Stance
Stance
Stance
Stance
Stance
Stance
[Walker et al. 2012, Hasan and Ng 2013]
Stance
31
Modeling Question 2)
Collective classification vs. local classification?
Stance
Stance
Stance
Stance
Stance
Stance
Stance
[Walker et al. 2012, Hasan and Ng 2013 ]
Stance
[Walker et al. 2012]
32
Modeling Question 3)
Jointly model disagreement together with stance?
Disagrees
Disagrees
Stance
Stance
Disagrees
Stance
Disagrees
Stance
33
[Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates]
Modeling Question 3)
Jointly model disagreement together with stance?
Disagrees
Disagrees
Disagrees
Disagrees
Stance
Stance
Stance
Stance
Disagrees
Stance
Disagrees
Stance
Stance
Disagrees
Disagrees
Stance
Stance
[Abbott et. al 2012 - Linguistic Features], [Burfoot et. al 2011 for Congressional Debates]
34
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference
– Large post-level graphs
– Loopy author-level graphs
• Systematic study of modeling options
35
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference
– Large post-level graphs
– Loopy author-level graphs
• Systematic study of modeling options
36
Our Contributions
• A unified framework to explore multiple models
• Fast, highly scalable inference
– Large post-level graphs
– Loopy author-level graphs
• Systematic study of modeling options
– Modeling recommendations
37
All Combinations of Models
Author
Local
Local
Author Coll.
Author
Author Joint
Collective
Post Local
Post
Post Coll.
Joint
Modeling
Granularity
Post Joint
Statistical Models
38
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical
model called Hinge-loss Markov Random Fields
5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2)
Rule Weight
Bach et. al (2015), ArXiV
Open source software: https://psl.umiacs.umd.edu
39
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical
model called Hinge-loss Markov Random Fields
5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2)
Rule Weight
Predicates
are
continuous
Random
Variables!
40
Probabilistic Soft Logic (PSL)
• Templating language for highly scalable graphical
model called Hinge-loss Markov Random Fields
Relaxations of Logical Operators
5.0: Disagrees(A1, A2) ^ Pro(A1)  ~Pro(A2)
Rule Weight
Predicates
are
continuous
Random
Variables!
41
Probabilistic Soft Logic (PSL)
• Rules instantiated with variables from real network
5.0: Disagrees(A1, A2) ^ Pro(A1) 
~Pro(A2)
Disagrees
Disagrees
Stance
Stance
Disagrees
Stance
Disagrees
Stance
42
Probabilistic Soft Logic (PSL)
• Rules instantiated with variables from real network
,
) ^ Pro(
5.0: Disagrees(
,
) ^ Pro(
)  ~Pro(
)
)  ~Pro(
)
…
5.0: Disagrees(
Continuous Random Variables!
43
Hinge-loss MRFs Over Continuous Variables
Conditional
random field
over
continuous
RVs in [0,1]
Bach et al. (2015), ArXiV
Bach et al. NIPS 12, Bach et al. UAI 13
44
Hinge-loss MRFs Over Continuous Variables
Conditional
random field
over
continuous
RVs in [0,1]
5.0: Disagrees(
Feature function for
each instantiated
rule
,
) ^ Pro(
Bach et al. NIPS 12, Bach et al. UAI 13
)  ~Pro(
)
45
Hinge-loss MRFs Over Continuous Variables
Conditional
random field
over
continuous
RVs in [0,1]
Bach et al. NIPS 12, Bach et al. UAI 13
Feature functions
are hinge-loss
functions
46
Hinge-loss MRFs Over Continuous Variables
Linear function
Encodes distance to satisfaction of
each instantiated rule
Bach et al. NIPS 12, Bach et al. UAI 13
47
Fast Inference in Hinge-loss MRFs
Convex, continuous inference
objective…
Convex optimization!
Solved using efficient, parallelizable algorithm:
Alternating Direction Method of Multipliers (ADMM)
Bach et al. NIPS 12, Bach et al. UAI 13
48
Constructing Local Predictors
Bush
Obama
Bag-of-words
believe
Unigrams, Bigrams, Lengths,
Initial n-grams, Repeated
Punctuation
49
Constructing Local Predictors
Bush
Obama
Bag-of-words
believe
Training Labels
Pro
Not Pro
Unigrams, Bigrams, Lengths,
Initial n-grams, Repeated
Punctuation
Logistic Regression
50
Constructing Local Predictors
Bush
Obama
Bag-of-words
believe
Pro
Training Labels
Not Pro
Unigrams, Bigrams, Lengths,
Initial n-grams, Repeated
Punctuation
Logistic Regression
LocalPro: 0.8
Observed Prediction Probabilities
LocalPro: 0.1
51
PSL Rules Shared by All Models
LocalPro: 0.8
Stance
LocalPro: 0.1
Stance
52
PSL Rules for Simple Collective Models
LocalPro: 0.8
Stance
Disagrees: 1.0
LocalPro: 0.1
Stance
53
Joint Disagreement Models
LocalPro: 0.8
Stance
LocalDis: 0.5
Disagrees
LocalPro: 0.1
Stance
54
Joint Disagreement Models
LocalPro: 0.8
Stance
LocalDis: 0.5
Disagrees
LocalPro: 0.1
Stance
55
Evaluation - Datasets
Details
Topics
4Forums.com
CreateDebate.org
Abortion, Gay Marriage,
Evolution, Gun Control
Abortion, Gay Rights,
Obama, Marijuana
Avg.
Users/Topic
336
311
Avg.
Posts/User
19
4
56
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
57
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
• Ground Truth for CreateDebate.org:
Majority
Stance
Stance
Stance
Stance
58
Evaluation Settings
• Prediction Tasks: Author Stance, Post Stance
• Ground Truth for CreateDebate.org:
Majority
Stance
Stance
Stance
Stance
• Ground Truth for 4Forums:
Stance
Stance
59
Author Stance Prediction – CreateDebate.org
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
60
Author Stance Prediction – CreateDebate.org
Post < Author
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
61
Author Stance Prediction – CreateDebate.org
Author-Joint
Model is best
Post < Author
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
62
Post Stance Prediction – CreateDebate.org
Author-Joint
Model still best!
Post < Author
(still!)
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
63
Author Stance Prediction – CreateDebate.org
Local < Collective < Joint
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
64
Author Stance Prediction – 4Forums.com
Naïve collective
harmful at author level!
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
65
Author Stance Prediction – 4Forums.com
Naïve collective
harmful at author level!
Accuracy
Post
Local
Post
Coll.
Post
Joint
Author Author Author
Joint
Local Coll.
66
Explanation for Naïve Collective’s Performance
Details
4Forums.com
CreateDebate.org
% Opposite
Stance Posts
71.6
73.9
% Opposite
Stance Authors
52.0
68.9
67
Explanation for Naïve Collective’s Performance
Naïve collective assumption
mostly true for posts
Details
4Forums.com
CreateDebate.org
% Opposite
Stance Posts
71.6
73.9
% Opposite
Stance Authors
52.0
68.9
68
Explanation for Naïve Collective’s Performance
Naïve collective assumption
mostly true for posts
Details
% Opposite
Stance Posts
% Opposite
Stance Authors
4Forums.com
CreateDebate.org
71.6
73.9
52.0
68.9
Assumption doesn’t hold
at author level!
69
Benefit of Disagreement Prediction
I agree with everything
except the last part. Safe
gun storage is very
important…
Agree
Anti Gun Control
I can agree with this. And
in case it seemed otherwise,
I know full well how to store
guns safely…
My point was that I don’t like
the idea of such a law…
Anti Gun Control
70
Benefit of Disagreement Prediction
I agree with everything
except the last part. Safe
gun storage is very
important…
Agree
Anti Gun Control
I can agree with this. And
in case it seemed otherwise,
I know full well how to store
guns safely…
My point was that I don’t like
the idea of such a law…
Anti Gun Control
71
Summary
• Unified modeling framework for efficiently,
systematically exploring all modeling choices
• Author-level joint disagreement and stance
model best, even for post-level prediction
• Disagreement model can be vital
when modeling at author level
72
Summary
• Unified modeling framework for efficiently,
systematically exploring all modeling choices
• Author-level joint disagreement and stance
model best, even for post-level prediction
• Disagreement model can be vital
when modeling at author level
73
Summary
• Unified modeling framework for efficiently,
systematically exploring all modeling choices
• Author-level joint disagreement and stance
model best, even for post-level prediction
• Disagreement model can be vital
when modeling at author level
74
Summary
• Unified modeling framework for efficiently,
systematically exploring all modeling choices
• Author-level joint disagreement and stance
model best, even for post-level prediction
• Disagreement model can be vital
when modeling at author level
Thank you for your attention!
75