RelevanceandRankinginOnline
DatingSystems
Fernando Diaz1 Donald Metzler2
Sihem Amer-Yahia1
2 USC
1 Yahoo!
Labs
InformationSciencesInstitute
July20, 2010
1 / 48
ProblemDefinition
U
du
qu
Ruv
setofusers
userdescription
userquery
matchrelevance
Task: Foreachuser, u, rankallotherusers, v ∈ U − {u},
suchthatrelevantmatchesoccurabovenon-relevant
candidates.
2 / 48
DatingSystem
U
3 / 48
DatingSystem
setofcandidatesgiven u
3 / 48
DatingSystem
setofcandidates u isinterestedin
3 / 48
DatingSystem
setofcandidatesalsointerestedin u
3 / 48
DatingSystem
setofcandidatesalsointerestedin u
3 / 48
DatingSystem
setofcandidatesalsointerestedin u
3 / 48
Match-MakingSystems
• datingsites
4 / 48
Match-MakingSystems
• datingsites
• employmentsites
4 / 48
Match-MakingSystems
• datingsites
• employmentsites
• communityquestion
answering
4 / 48
Match-MakingSystems
• datingsites
• employmentsites
• communityquestion
answering
• paperreviewing
4 / 48
Match-MakingSystems
• datingsites
• employmentsites
• communityquestion
answering
• paperreviewing
• consumer-to-consumer
4 / 48
Outline
RelatedWork
RankingforMatch-MakingSystems
Relevance
RankingFeatures
RankingFunction
MethodsandMaterials
Results
Conclusion
5 / 48
RelatedWork
• Quantitativeanalysisofdating
• Demographicfactorsinfluencingpreferencesinonline
datingsystems[Hitsch etal. 2005].
• Linguisticanalysisofspeeddating[Ranganath etal.
2009].
• Rankingfordatabases
• Relevanceranking[Chaudhuri etal. 2004, Lavrenko et
al. 2007]
• Workload-basedfeedback[Wu etal. 2000]
• Personsearch
• Expertsearch[Balog2008]
• Searchinasocialnetwork[Kleinberg2000]
• Reviewermatching[Karimzadehgan etal. 2008]
6 / 48
StableMarriageProblem
[GaleandShapley1962]
• input: eachuserranksall
otherusers
• output: 1-1matchingof
users
• assumessingle
recommendation, nota
ranking
7 / 48
RankingforMatch-Making
Systems
8 / 48
RankingforMatch-MakingSystems
ProbabilityRankingPrinciple
Task: Foreach querier, u, rankeach candidate,
v ∈ U − {u}, by P (R|u, v).
9 / 48
RankingforMatch-MakingSystems
• Relevance: measuringmatchrelevance.
• Rankingfeatures: signalsusedtodeterminematch
relevance.
• Rankingfunction: combiningrankingfeaturesto
modelmatchrelevance.
10 / 48
RankingforMatch-Making
Systems
Relevance
11 / 48
TraditionalRelevance
user
document
One-sidedrelevance ininformationretrievalmeasuresthe
satisfactionoftheuserwithrespecttoadocument.
12 / 48
RelevanceforMatch-MakingSystems
user
user
Two-sidedrelevance measuresthemutualsatisfactionof
twousersinamatch.
13 / 48
IssueswithOne-SidedRelevanceina
Two-SidedDomain
UnreciprocatedContact
user
ranking
one-sided
two-sided
P@5
0.60
0.20
14 / 48
IssueswithOne-SidedRelevanceina
Two-SidedDomain
UndesiredAttention
A usermaybecontactedbyirrelevantindividuals.
15 / 48
Two-SidedRelevanceIsVerySubjective
• Human
• Relevanceisverysubjectivegivenonly
pre-introduction information(i.e. queriesandprofiles)
• Possiblewith post-introduction forms, questionnaires
[Ranganath etal. 2009].
• Automatic
• Possiblewithlongitudinalstudies[Gottman etal.
2002].
16 / 48
editoriallylabelingmatchrelevanceisvery, veryhard.
(relevanceisintheeyeofthebeholder.)
17 / 48
canweuseimplicitfeedbacksignalsintheproduction
systemtoinferrelevance?
(relevanceisintheeyeofthebeholder.)
18 / 48
Post-PresentationSignals
One-WayRelevance
P (R|q, d) ∝ f (clicks)
Intraditionalsearch, one-wayuserengagements(e.g. clicks)canbeusedtopredict
relevance[Joachims2002, …].
19 / 48
Post-PresentationSignals
Two-WayRelevance
P (R|u, v) ∝ f (numberofmessages)
Inmatch-makingsearch, two-wayuserengagementscanbeusedtopredictrelevance.
•
•
•
profileviews(e.g. ‘numberoftimesusersviewedeachother’sprofile’)
messageexchanges(e.g. ‘numberoftimesusersexchangedmessages’)
durationofinteractions(e.g. ‘periodoftimeoverwhichusersinteracted’)
20 / 48
ModelingRelevancewithPost-Presentation
Signals
1. Pseudo-labeledrelevance: pickhighprecisionsignals
toinferrelevanceforasubsetofmatches.
• ‘exchanged100messages’ → relevant
• ‘noreplytomessage’ → non-relevant
2. Predictedrelevance: modelpseudo-labeledrelevance
asafunctionofunlabeledsignals.
21 / 48
ModelingRelevancewithPost-Presentation
Signals
pseudo-labeled
P (R|u, v)
{0, 1}
coverage
low
22 / 48
ModelingRelevancewithPost-Presentation
Signals
pseudo-labeled
predicted
P (R|u, v)
{0, 1}
f (xpost ; Θpost )
coverage
low
medium
22 / 48
ModelingRelevancewithPost-Presentation
Signals
pseudo-labeled
predicted
predicted
P (R|u, v)
{0, 1}
f (xpost ; Θpost )
f (xpre ; Θpre )
coverage
low
medium
high
22 / 48
RankingforMatch-Making
Systems
RankingFeatures
23 / 48
Pre-PresentationSignals
One-WayRelevance
P (R|q, d) ∝ f (querytermmatches, popularity)
Intraditionalsearch, avarietyofstaticandquery-dependentfeaturescanbeusedto
predictrelevance.
24 / 48
Pre-PresentationSignals
Two-WayRelevance
P (R|q, d) ∝ f (querymatches, income)
Inmatch-makingsearch, asimilarvarietyofstaticandquery-dependentfeaturescanbe
usedtopredictrelevance.
25 / 48
QueryandDescriptionAttributes
DatingDataset
scalar
age
height
income
numchildren
numphotos
categorical
bodytype
city
country
desiresmorechildren
drinking
education
employment
ethnicity
eyecolor
featuredprofile
gender
haircolor
humorstyle
interests
languages
livingsituation
maritalstatus
newuser
occupation
personalitytype
politicalbent
religion
religiousactivity
romanticstyle
sexuality
smoking
socialstyle
starsign
state
subscriptionstatus
televisionviewer
zipcode
text
description
*allexcept‘description’arequeriable.
26 / 48
QueryingAttributes
Profile
querier
candidate
query
profile
1. desiredattributevalue
2. importance(‘mustmatch’, ‘nicetomatch’, ‘anymatch’)
onefeatureforeachattribute
27 / 48
RankingFeatures
Profile
querier
candidate
query
q
q
profile
d
d
←
−
−
→
• d : featuresofthequerier(candidateindependent)
• d : featuresofthecandidate(queryindependent)
onefeatureforeachattribute
28 / 48
RankingFeatures
Match
querier
candidate
query
q
q
profile
d
•
•
d
−
→
q : candidateattributesmatchingquery
←
−
q : querierattributesmatchingcandidate’squery
onefeatureforeachattribute
29 / 48
RankingFeatures
Similarity
querier
candidate
query
q
q
δ
profile
d
−
→
←
−
δi = | d i − d i |
−
→
←
−
δi = d i ⊕ d i
δtext = ⟨tu , tv ⟩
d
scalarfeatures
binaryfeatures
textfeatures
30 / 48
RankingFeatures
Summary
• Profile: match-independentattributesofthequerier
andcandidate.
• Match: match-dependentattributesofthequeryand
profile.
• Similarity: match-dependentsimilaritybetween
profiles.
31 / 48
RankingforMatch-Making
Systems
RankingFunction
32 / 48
Pre-PresentationSignals
Two-WayRelevance
P (R|q, d) ∝ f (profile, match, similarity)
Thefunctionalformof f willbeaboosteddecisiontree.
33 / 48
ModelingRelevance
GradientBoostedDecisionTrees
+...+
+
f (xpre ; Θpre )
=
T (x; Θ0 )
+
T (x; Θ1 )
+...+
T (x; Θm )
34 / 48
MethodsandMaterials
35 / 48
Data
train
validation
test
users
7,716
3,697
4,836
matches
17,538
7,543
10,636
gatheredinFall2009; onlyuserswithatleastonematchwithnon-zero P (R)
36 / 48
Evaluation
APu = ∑
1
∑
P (R|u, v) v
∑
P (R|u, v)
Prec(v)P (R|u, v)
v
P@k =
NDCGk =
1
k v∈R
k
k
∑
2P (R|u,vi ) − 1
i=1
log(i + 1)
∑ P (R|u, vi ) i−1
∏ (
)
ERR =
1 − P (R|u, vj )
r
i
j=1
• evaluationperformedforpseudo-andpredictedlabels
withprobabilisticretrievalmetrics.
• statisticalsignificanceusingpaired, one-tailed
bootstrap(usersamplingproportionaltoprobabilityof
strongestmatch)
37 / 48
Training
• Gradient-boosteddecisiontreeregressionagainst
P (R|u, v).
• Instance-weightingaccordingtoinverseclass
frequency(toaddressclassimbalance).
• Freeparameterstunedonvalidationset.
38 / 48
Runs
←
−
d
−
→
q
δ
←
→
q
query-independentprofileattributes
one-wayprofile-querymatch
profilesimilarity
two-wayprofile-querymatch
←
→→
→
Combinations δ ←
q and d ←
q .
39 / 48
Results
MatchFeatures
AP
←
−
d
0.485
δ
0.484
−
→
q
0.428
←
→
q
0.454
→
δ←
q
0.497
←
→←
d →
q
0.494
NDCG1
NDCG5
NDCG10
0.346
0.576
0.649
0.366
0.556
0.643
0.287
0.501
0.598
0.317
0.527
0.619
0.380
0.575
0.659
0.367
0.580
0.656
P@1
P@5
P@10
0.360
0.326
0.226
0.380
0.311
0.223
0.304
0.298
0.221
0.334
0.303
0.219
0.395
0.318
0.227
0.381
0.326
0.226
ERR
0.582
0.577
0.517
0.552
0.595
0.589
40 / 48
PredictedRelevance
MatchFeaturesAloneIneffective
AP
←
−
d
0.485
δ
0.484
−
→
q
0.428
←
→
q
0.454
→
δ←
q
0.497
←
→←
d →
q
0.494
NDCG1
NDCG5
NDCG10
0.346
0.576
0.649
0.366
0.556
0.643
0.287
0.501
0.598
0.317
0.527
0.619
0.380
0.575
0.659
0.367
0.580
0.656
P@1
P@5
P@10
0.360
0.326
0.226
0.380
0.311
0.223
0.304
0.298
0.221
0.334
0.303
0.219
0.395
0.318
0.227
0.381
0.326
0.226
ERR
0.582
0.577
0.517
0.552
0.595
0.589
41 / 48
PredictedRelevance
NoStatisticalDifferenceBetweenStrongestRuns
AP
←
−
d
0.485
δ
0.484
−
→
q
0.428
←
→
q
0.454
→
δ←
q
0.497
←
→←
d →
q
0.494
NDCG1
NDCG5
NDCG10
0.346
0.576
0.649
0.366
0.556
0.643
0.287
0.501
0.598
0.317
0.527
0.619
0.380
0.575
0.659
0.367
0.580
0.656
P@1
P@5
P@10
0.360
0.326
0.226
0.380
0.311
0.223
0.304
0.298
0.221
0.334
0.303
0.219
0.395
0.318
0.227
0.381
0.326
0.226
ERR
0.582
0.577
0.517
0.552
0.595
0.589
42 / 48
Arequeriesuseless?
• Yes, wearebetteroffstaticallyrankinguserssubjectto
afewdistanceandgenderconstraints.
43 / 48
Arequeriesuseless?
• Yes, wearebetteroffstaticallyrankinguserssubjectto
afewdistanceandgenderconstraints.
• No, thematchrepresentationisbad.
• Querylanguageorattributesmaybepoorly
represented.
• Usersmaynotbecomfortableenoughwiththequery
interface.
43 / 48
Conclusions
• Relevanceiscomplicatedandtask-dependent
• two-way
• highlysubjective
• IR ̸= websearch
• Preliminary evidencesuggestsqueriesmaynotbe
helpful.
44 / 48
FutureWork
• Detectingimplicitrelevancesignalsfordifferent
retrievaldomains.
• Designinginterfacestoencourageimplicitrelevance
feedback.
• Richerfeatures.
• Moreeffectivequerylanguagesandinterfaces.
45 / 48
Acknowledgments
SergiyMatusevych
SeenaCherangara
RameshGollapudi
RamanaLokanathan
46 / 48
SupplementalSlides
47 / 48
ImportantFeatures
←
−
d
featured
age
height
livingarrangement
subscriptionstatus
religiousactivity
employment
numphotos
religion
interests
newuser
smoking
activity
morekids
occupation
→
δ←
q
text(δ )
−
distance(←
q)
age(δ )
→
distance(−
q)
subscriptionstatus(δ )
height(δ )
featured(δ )
newuser(δ )
gender(δ )
numphotos(δ )
eyecolor(δ )
→
maxage(−
q)
religiousactivity(δ )
−
maxage(←
q)
featured(δ )
48 / 48