Learning Conference Reviewer
Assignments
Adith Swaminathan
Guide :
Prof. Soumen Chakrabarti
Department of Computer Science and
Engineering,
Indian Institute of Technology, Bombay
Future Work (from BTP1)
Given WWW2010’s assignments, learn
Affinity_Param, Topic_Param and
Irritation
 Citations as edge features
 Load-Constrained Partial Assignments
 Better estimation of Assignment Quality

Background
 Conference
Reviewer-Paper
Assignment as a Many-Manymatching
[1]
 Minimum
(MCF)
Cost Network Flow
Conference Reviewer Assignment

Set of Reviewers, R, max #papers = L_i
Set of Papers, P, min #reviews = K

Assumption : Only require #reviews, not quality

Suppose we have cost function A_ij(y) for
<R_i, P_j>

ILP -> Assumption -> MCF
Two problems

Integer Linear Programs are NP-Hard!
–
–

Relax?
More assumptions?
How to determine A_ij?
–
–
M * N ~ 10000
Multimodal clues
ILP

Enforce structure on A_ij
–
–

-> Assumption -> MCF
Better model multimodality
Fewer parameters to fix
“Learn” A_ij using Structured Learning
Techniques
A_ij
=
T
w
Φ(R_i, P_j, y_ij)
Ramifications of Structured Costs

Costs decompose over <R_i, P_j> pairs
–
–

Decomposable Preference Auction
Polynomial Algorithms for DPAs [2]
Restricted notion of optimality
–
–
Per-reviewer/Per-paper constraint could be
combinatorial
Stability?
ILP -> Assumption -> MCF
Minimum Cost Network Flow
Directed graph G=(V,E), capacities
u(E)>= 0, costs c(E)
 Nodes have numbers b(V) : Sum(b(V))
=0
 Task : Find a function f: E->R+ which
satisfies the b-flow at minimum cost
 Successive Shortest Path Algorithm

Node features and Edge features
Profile
Affinity
Contents
Reviewer
Bid
Paper
Topics
Topic Overlap
Topics
The Loss Function



L_ij = w_1 * exp(-Affinity_ij) + w_2 * [[1 –
Topic_Overlap_ij]] + w_3 * Bid_Cost
Bid_Cost = Potential(R_i, P_j, y_ij)
Irritation (I) and Disappointment (D) needs to
be set
Assignment Quality Measures
Number of Bids Violated?
– Not a reliable measure.
 +ve Bids Violated
 –ve Bids Violated
 Assignments satisfying Topic Match
 Confidence?

Confidence == Quality?
 Very
sparse
Fewer than 5% observed
– Extrapolated Confidence?
–
 Reliable
Bids as a precursor of Confidence [3]
– Confidence-Augmented Loss?
–
Learning w’s

Transductive Ordinal Regression
–
–
–
–

Assume : Assignments are independent (Naïve)
Heuristic : Augment observed dataset
Extrapolate observed Confidence [4]
Learn w over extrapolated dataset
Support Vector Machine for Structured Outputs
–
–
–
Cast as soft-margin SVM formulation [5]
Upper-bound objective with a convex fn (Optimality?)
Minimize, using Cutting Plane (Approximate)
Transductive Ordinal Regression [6]
SVM Struct. [7]
Loss Augmented Inference ~ Most Violated Constraint
Loss is decomposable -> Modified MCF
PARA : Paper Assignment to
Reviewers Apparatus
Results
Bimodal Behaviour
•Reviewer either gets few or L_i papers
•Load Penalties [8]
•Introduce more parameters
•Infer using modified MCF
•Learning parameters?
•Load Rebalancing
•Tradeoff between MCF optimum and
old assignment
Penalise Reviewer Loads
Load Constrained Assignments
Avenues for Future Work
•Document Modelling for Affinity
Scores
•Objective Assignment Evaluation
•Transitive Citation Scores
•Load Penalty Parameter Estimation
References
1.
2.
3.
4.
The Conference Paper Assignment Problem, J.
Goldsmith, R.H. Sloan, 2007
MultiAgent Systems: Algorithmic, Game-Theoretic,
and Logical Foundations, Y. Shoham, K. LeytonBrown, 2009
Automating the Assignment of Submitted
Manuscripts to Reviewers, S.T. Dumais, J. Nielson,
1992
Semisupervised Regression with cotraining
algorithms, Z. Zhou, M. Li, 2007
References – contd.
5. Learning structured prediction models : A Large
Margin Approach, B. Taskar, et al, 2005
6. Ologit : Ordinal Logistic Regression for Zelig, G.
King, et al, 2007
7. SVM Learning for Interdependant and Structured
Output Spaces, I. Tsochantaridis, et al, 2004
8. Word Alignment via Quadratic Assignment, S.
Lacoste-Julien, et al, 2006