Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Inferring Local Synonyms for Improving Keyword
Suggestion in an On-line Advertisement System
L. Sarmento1
P. Trezentos2
J. P. Gonçalves3
E. Oliveira4
June 28, 2009
1
LIACC/FEUP, Portugal
ISCTE/ADETTI/Caixa Magica, Portugal
3
Portugal Telecom/SAPO, Portugal
4
LIACC/FEUP, Portugal
2
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Introduction
Method Description
Evaluation Methodology
Experimental Set-up
Results and Analysis
Conclusions and Future Work
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
The Problem
I
Content-Targeted Advertisement Systems focus on placing
ads on content-rich sites
I
Advertisers are interested in describing their ads using more
and better keywords so they can help ad placement
I
However, advertisers frequently associate only few keywords
(1 to 5) to their ads, decreasing the chances of matching
them with the most appropriate content
I
When no better options exist, ad brokers place generic ads,
but these have less chances of being clicked by web users
I
Under a pay-per-click scenario, this means less revenue for the
broker.
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
In this work...
I
We propose a keyword suggestion mechanism:
I
I
I
I
that mines a database of previously submitted ads to infer
“synonymy” relations
generates keyword suggestions based on synonymy
ranks keywords using a function that provides implicit
sense-disambiguation
We perform on-line experiments and compare the results of
our method with an alternative legacy method.
I
We propose several novel evaluation measures.
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Basic Concepts
I
©
ª
Let A be the set of ads: A = a1 , a2 , ...a|A| .
I
Each ad ai = (ti , di , Ki ):
I
I
I
I
ti is a title
di is a short text description
Ki is the
provided by the advertiser,
ª
© set of |Ki | keywords
Ki = ki1 , ki2 , ...ki|Ki | .
©
ª
Given a set of seed keywords, K 0 = k10 , k20 , ... , a keyword
suggestion function Fs that generates a ranked list of keyword
suggestions k1s , k2s , .... kns .
Fs (K 0 ) = {k1s , k2s , ....kns }
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
(1)
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Basic Concepts: Synonymy
I
I
Relevance is related to keyword “inter-changeability”
s
keyword
a relevant suggestion for seed set
© k0 i is0 considered
ª
0
K = k1 , k2 , ...
I
I
if it is inter-changeable with one or more elements from K 0
We will use the term synonym:
I
synonyms can be inter-changed without adulterating
“meaning” of the keyword set.
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Computing Synonymy (I)
I
Two keywords are synonyms in A if they systematically
co-occur with the same set of previously known keywords
I
Let [C (ki )] be the vector of keyword co-occurrences for
keyword ki :
[C (ki )] = [(k1 , fi1 ), (k2 , fi2 ), ...(ki , 0)...(kn , fin )]
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
(2)
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Computing Synonymy (II)
I
Let W be a feature weighting function (e.g. tf-idf).
I
The degree of synonymy sij between two keywords, ki and kj
can be computed by applying a vector similarity metric S (e.g
the cosine) to:
sij = S ([CW (ki )], [CW (kj )])
I
(3)
A keyword synonymy graph, Gs , can be obtained by computing
pairwise similarity between all co-occurrence vectors
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Keyword Suggestion and Ranking (I)
I
Given a seed keyword, k10 , relevant keyword suggestions may
be found amongst the nodes of Gs closest to the k10 node.
I
Best keyword suggestions will be located closer to the seed
node.
I
Fs (k10 ) = {kAs , kBs , kCs , kDs }, by this order
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Keyword Suggestion and Ranking (II)
I
Extra seeds help resolving ambiguities
I
E.g.: k10 = “orange”
I
I
kAs = “apple”, kBs = “banana”, kCs = “yellow” and kDs =
“red”.
Ambiguity will be solved by k20 = “blue”
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Evaluation of Keyword Suggestion Systems
I
We will perform on-line evaluation of our method, and
compare it against a legacy keyword suggestion system
I
I
http://anuncios.sapo.pt since March 2004
The legacy suggestion function FL combines 3 methods:
1. finds related words using the OpenOffice thesaurus;
2. selects from the query logs of a commercial web search engine
the most frequent search queries that lexically include
keywords input by advertiser;
3. selects keywords from the ads already in the database that
lexically include keywords input by advertiser.
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Evaluation?
I
But: what are we trying to evaluate and compare?
1. Method Usefulness
2. Impact on Revenue
I
Evaluation will be based on statistics about:
1. advertisers’ behaviour
2. ad performance
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Evaluation Methodology: FN (New) vs. FL (Legacy)
I
I
We can randomly choose FN (New) vs. FL (Legacy)
suggestion function with 50% probability to assist the
advertiser.
For each keyword submitted by advertisers, we can log:
I
I
I
I
I
source
rank
iterations until selection
#imp (i, j)
#clk (i, j)
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Evaluation Methodology: Measuring Method Usefulness
I
For each ad ai having keywords suggested by one of the two
competing systems we compute:
I
Suggestion ratio:
Sr (i) =
I
I
I
I
∫auto (i)
∫user (i) + ∫auto (i)
(4)
average suggestion rank, R(i)
T@1 (i)
T@10 (i)
Global performance figures for each suggestion function can be
obtained by averaging these statistics over corresponding ads
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Evaluation Methodology: Measuring Impact on Revenue
I
Again, for each ad ai having keywords suggested by one of the
two competing systems we compute:
I
I
I
I
I
I
I
average keyword printability, Pk
average keyword clickability, Ck
keyword printabilty efficiency, ²P
k
keyword clickability efficiency, ²Ck
average ad printability, Pa
average ad clickability, Ca
average click through rate, CTR
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Experimental Set-up (I)
I
For building the link graph, Gs , we used a set of 84,180 ads:
I
I
I
I
compiled over a period of about 5 years
ads have 14.14 keywords in average
63% only have one keyword associated
almost 70% have 5 or less keywords associated
I
Ads with more that 75 keywords (2022) were ignored to avoid
catch-all ads
I
Co-occurrence information for a set of 122,099 keywords
I
Vectors were weighted by Mutual Information
I
Vector comparison using the cosine metric
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Experimental Set-up (II)
I
Two systems, FN and FL , running in parallel
I
50% chances of being selected
I
15 weeks of experimentation
Unfortunately, only a small subset of ads (192 / approx. 5%)
include automatically suggested keywords:
I
I
I
69 ads by FN
123 ads by FL
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Two Scenarios
I
We studied two different Scenarios:
I
I
Scenario 1 - considers all ads: FNall and FLall
Scenario 2 - ads with no more than 75 keywords:FN75 and FL75 .
Advertisers tend to be more selective
#ads
#kwrd
#sug
kwrd
Scenario 1
FLall FNall
123
69
51.3 93.2
19.1 32.8
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Scenario 2
FL75 FN75
103
51
27.7 27.8
7.5 11.3
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Method Usefulness
Sr
R
T@1
T @10
I
Scenario 1
FLall
FNall
0.37
0.35
154.9 129.1
0.09
0.14
0.17
0.20
1.04
1.32
Scenario 2
FL75 FN75
0.27 0.40
65.1 27.6
0.06 0.32
0.22 0.49
1.14 1.34
I
Except for one case, FN scores better in all indicators.
I
Advertisers pick suggestions ranked higher when using FN
I
S r similar in Scenario 1 but higher in Scenario 2 for FN
I
Advertisers request suggestions more often when using FN .
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Impact on Revenue (1)
Pk
Ck
²P
k
²Ck
I
I
I
Scenario 2
FL75
FN75
1065.4 2679.0
0.80
2.35
0.11
0.22
0.048
0.10
large difference in Pk and Ck between both scenarios:
I
I
Scenario 1
FLall
FNall
6865.4 13896
10.0
9.7
0.17
0.18
0.071
0.08
we manually verified that there are a few ads in Scenario 1
that generate an enormous amount of prints and clicks,
Pk for FN is much higher
Ck values are similar in Scenario 1, but in Scenario 2 FN
scores 3x more
C
in all cases FN scores higher in ²P
k and ²k .
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Impact on Revenue (2)
Pa
Ca
CTR (×10−3 )
I
I
Scenario 1
FNall
131,279 450,713
191.5
312.1
1.45
0.69
FLall
Scenario 2
FL75
FN75
7,995 30,309
5.99
26.59
0.74
0.87
Pa and Ca , are higher for ads with suggestions from FN
However, in Scenario 1 FN produces higher CTR
I
We decided to investigate why...
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Impact on Revenue (3)
I
I
CTR values are higher for ads with keywords from FN when
the threshold is lower than 100 keywords (except for one case,
threshold=50)
CTR peaks for FL when threshold is set 200, as a result of a
few outliers found in that range.
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Results: Global Impact on Revenue (4)
I
Considering ads with ≤ 75 words, during the test period:
I
I
I
if all 154 ads had keywords suggested by FN :
I
I
2684 ads with no suggested keywords (CTR 0.15 × 10−3 )
only 154 (5.42%) ads have automatically suggested keywords
2,298,301 additional prints and 2,120 more clicks
Globally, this represents...
I
I
0.06% additional prints
0.38% additional clicks
...generated from only 154 of ads
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Conclusions (I)
I
We presented a keyword suggestion mechanism that:
I
I
I
I
mines information from ads database to infer local synonymy
information between keywords
uses synonymy information to perform relevant (and
non-obvious) keyword suggestions
performs implicit sense-disambiguation
We performed evaluation against a legacy keyword suggestion
system:
I
we proposed a set of novel performance measures
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Conclusions (II)
I
We showed that:
I
I
I
keywords suggested by the system we propose outperform
keywords suggested by the legacy system in several parameters
related to printability and clickability
ads with keywords suggested by the system we propose tend to
have higher CTR values
potentially, improvements can be more significant if more
advertisers use keyword suggestion
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Future Work
I
Two key points to improve:
1. inability to suggest keywords as a response to “unknown”
keywords input by the advertisers
I
We believe that this has severely reduced the number of times
the system was effectively used.
2. the inability of our system to generate suggestions out of the
set of known words.
I
Possible Solutions:
I
I
mine keyword co-occurrence information from other media that
we have available, namely search query logs and blog content
use such information to complement the information mined
from ad logs, or simply as a backup mechanism
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in
Outline Introduction Method Description Evaluation Methodology Experimental Set-up Results and Analysis Conclusions a
Thank you
I
Questions & comments?
L. Sarmento, P. Trezentos, J. P. Gonçalves, E. Oliveira
Inferring Local Synonyms for Improving Keyword Suggestion in