Distributional Lexical Semantics II:
Getting into the details
Marco Baroni
UPF Computational Semantics Course
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Corpus pre-processing
I
Minimally, corpus must be tokenized
I
POS tagging, lemmatization, dependency parsing. . .
Trade-off between deeper linguistic analysis and
I
I
I
I
need for language-specific resources
possible errors introduced at each stage of the analysis
more parameters to tune
Contexts and dimensions
I
One fundamental difference:
I
I
Non-lexical aspects of contexts as filters
Non-lexical aspects of contexts as links
Window-based
The dog barks in the alley.
I
Context as filter (e.g., Rapp 2003):
dog bark; bark dog; bark alley; alley bark
I
Context as link (e.g., HAL):
dog bark-r; bark dog-l; bark alley-r; alley bark-l
Dependency-based
The dog barks in the alley.
I
Context as filter (e.g., Padó & Lapata):
dog bark; bark dog
I
Context as link (e.g., Grefenstette 1994, Lin 1998, Curran
& Moens 2002):
dog bark-subj−1 ; bark dog-subj
Filters vs. links
I
I
With filters, data less sparse (man kills and kills man both
map to a kill dimension of the man vector)
With links
I
I
more sensitivity to semantic distinctions (kill-subj−1 and
kill-obj−1 are rather different things!)
links provide a form of “typing” of dimensions (the “subject”
dimensions, the “for” dimensions, etc.)
I
we will see importance of this when we discuss relational
similarity
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Dimension weighting
The basic intuition
word1
dog
dog
word2
small
domesticated
freq 1 2
855
29
freq 1
33,338
33,338
freq 2
490,580
918
Mutual Information
Church & Hanks (1990)
MI(w1 , w2 ) = log2
MI(w1 , w2 ) = log2
Pcorpus (w1 , w2 )
Pind (w1 , w2 )
Pcorpus (w1 , w2 )
Pcorpus (w1 )Pcorpus (w2 )
P(w1 , w2 ) =
P(w) =
fq(w1 , w2 )
N
fq(w)
N
Mutual Information
I
MI estimation (ignoring the logarithm):
P(w1 , w2 )
=
P(w1 )P(w2 )
fq(w1 ,w2 )
N
fq(w1 ) fq(w2 )
N
N
=
fq(w1 , w2 )
N2
fq(w1 , w2 )N
×
=
N
fq(w1 )fq(w2 )
fq(w1 )fq(w2 )
The core of Mutual Information
I
Logarithm does not change rank, N is constant:
fq(w1 , w2 )
fq(w1 )fq(w2 )
I
If we are looking at different dimensions w2 of the same w1
(e.g., small and domesticated as dimensions of dog),
fq(w1 ) is also constant:
fq(w1 , w2 )
fq(w2 )
Mutual Information core
word1
dog
dog
word2
small
domesticated
freq 1 2
855
29
freq 2
490,580
918
MI core
0.00174
0.03159
Other weighting methods
MI is sometimes criticized (e.g., Manning & Schütze 1999)
because it only takes relative frequency into account, and thus
overestimates the weight of rare events/dimensions:
word1
dog
dog
word2
domesticated
sgjkj
freq 1 2
29
1
freq 2
918
1
MI core
0.03159
1
Other weighting methods
I
A popular alternative is the Log-Likelihood Ratio (Dunning
1993)
I
“Core” of main term of log-likelihood ratio:
fq(w1 , w2 ) × MI(w1 , w2 )
I
(This term alone is also called Local Mutual Information,
see Evert 2008)
I
(To be precise, here MI is calculated using natural, not
base-2 logarithm)
word1
dog
dog
dog
word2
small
domesticated
sgjkj
freq 1 2
855
29
1
MI
3.96
6.85
10.31
LLR core
3382.87
198.76
10.31
Weighting methods
I
Many, many alternative weighting methods (Evert 2005,
2008)
I
In my experience (see also Schone & Jurafsky 2001), big
divide is between MI-like measures, that do not have an
absolute co-occurrence frequency term, and LLR-like
measures, that do
I
Unfortunately, attempts to determine the best weighting
method are inconclusive
Many many parameters
I
How many dimensions, and which?
I
Stop words?
I
Window-based weighting?
I
How do you transform raw frequencies?
I
Do you perform dimensionality reduction?
I
Etc.
I
See Bullinaria & Levy 2007, Bullinaria 2008 for a
systematic exploration of some of these parameters
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Contexts as vectors
dog
cat
car
runs
1
1
4
legs
4
5
0
6
Semantic space
5
cat (1,5)
3
2
1
car (4,0)
0
legs
4
dog (1,4)
0
1
2
3
runs
4
5
6
6
Semantic similarity as angle between vectors
5
cat (1,5)
3
2
1
car (4,0)
0
legs
4
dog (1,4)
0
1
2
3
runs
4
5
6
Measuring angles by computing cosines
I
I
Cosine is most common similarity measure in distributional
semantics, and the most sensible one from a geometrical
point of view
Ranges from 1 for parallel vectors (perfectly correlated
words) to 0 for orthogonal (perpendicular) words/vectors
I
I
It goes to -1 for parallel vectors pointing in opposite
directions (perfectly inversely correlated words), as long as
weighted co-occurrence matrix has negative values
(Angle is obtained from cosine by applying the arc-cosine
function, but it is rarely used in computational linguistics)
Trigonometry review
I
Build a right triangle by connecting the two vectors
I
Cosine is ratio of length of side adjacent to measured
angle to length of hypotenuse side
I
If we build triangle so that hypotenuse has length 1, cosine
will equal length of adjacent side (because we divide by 1)
1.0
0.8
0.6
y
0.4
0.2
0.2
0.4
y
0.6
0.8
1.0
Cosine
0.0
0.0
0.0
θ
0.2
0.4
0.6
x
0.8
1.0
θ
0.0
0.2
0.4
0.6
x
0.8
1.0
Computing the cosine
I
Given a and b, two vectors (segments from the origin) of
length 1 and with n dimensions, cosine is given by:
i=n
X
i=1
ai × bi
Computing the cosine
For simplicity, consider a two-dimensional space (the
classic “two-coordinates” space), where a has 0-value on
the second dimension (i.e., it is parallel to the x-axis)
I
Then, the a coordinates are 1 and 0; the b coordinates are
cos(θ) and sin(θ):
1.0
I
le
ng
th
0.4
=1
y
0.6
0.8
x= cos(θ)
y= sin(θ)
0.2
x=1
y=0
0.0
θ
0.0
0.2
0.4
0.6
x
0.8
1.0
1.0
Computing the cosine
le
0.4
ng
th
=1
y
0.6
0.8
x= cos(θ)
y= sin(θ)
0.2
x=1
y=0
0.0
θ
0.0
0.2
0.4
0.6
0.8
1.0
x
I
If we apply the formula:
i=n
X
ai × bi
i=1
we get:
(1 × cos(θ)) + (0 × sin(θ)) = cos(θ)
i.e., the cosine between the two vectors
1.0
Computing the cosine
le
0.4
ng
th
=1
y
0.6
0.8
x= cos(θ)
y= sin(θ)
0.2
x=1
y=0
0.0
θ
0.0
0.2
0.4
0.6
0.8
1.0
x
I
With a bit of trigonometry, this generalizes to any length-1
vector pair
I
Geometric intuition: we can always rotate the vectors by
same amount so that one is parallel to x-axis, without
changing the angle size
1.0
Computing the cosine
le
0.4
ng
th
=1
y
0.6
0.8
x= cos(θ)
y= sin(θ)
0.2
x=1
y=0
0.0
θ
0.0
0.2
0.4
0.6
0.8
1.0
x
I
If two vectors are not of length 1, we divide the dimensions
of each vector by its length (computed by Pythagoras
theorem), to obtain vectors in the same direction, but of
length 1, so that we can apply the same formula
1.0
Computing the cosine
le
0.4
ng
th
=1
y
0.6
0.8
x= cos(θ)
y= sin(θ)
0.2
x=1
y=0
0.0
θ
0.0
0.2
0.4
0.6
0.8
1.0
x
I
Putting the two steps together (normalization to length 1
and cosine computation for vectors of length 1) we obtain
the general formula to compute the cosine:
Pi=n
a ×b
qP i=1 i qPi
i=n 2
i=n 2
i=1 a ×
i=1 b
I
This generalizes to any n!
Computing the cosine
Example
Pi=n
ai × bi
qP
i=n 2
i=n 2
a
×
i=1
i=1 b
i=1
qP
dog
cat
car
runs
1
1
4
legs
4
5
0
√
cosine(dog,cat) = √ (1×1)+(4×5)
2
2
2
1 +4 ×
1 +52
= 0.9988681
arc-cosine(0.9988681) = 2.72 degrees
√
cosine(dog,car) = √ (1×4)+(4×0)
2
2
2
1 +4 ×
4 +02
= 0.2425356
arc-cosine(0.2425356) = 75.85 degrees
Computing the cosine
6
Example
5
cat (1,5)
4
dog (1,4)
3
1
2
75.85 degrees
car (4,0)
0
legs
2.72 degrees
0
1
2
3
runs
4
5
6
Cosine intuition
I
When computing the cosine, the values that two vectors
have for the same dimensions (coordinates) are multiplied
I
Two vectors/words will have a high cosine if they tend to
have high values for the same dimensions/contexts
I
If we center the vectors so that their mean value is 0, the
cosine of the centered vectors is the same as the Pearson
correlation coefficient
If, as it is often the case in computational linguistics, we
have only positive scores, and we do not center the
vectors, then the cosine can only take positive values, and
there is no “canceling out” effect
I
I
As a consequence, cosines tend to be very high, much
higher than the corresponding correlation coefficients
Other measures and the kernel trick
I
Cosines are well-defined, well understood way to measure
similarity in a vector space
I
Other measures based on other, often non-geometric
principles (Lin’s information theoretic measure,
Kullback/Leibler divergence. . . ) bring us outside the scope
of vector spaces, and their application to semantic vectors
can be iffy and ad-hoc
Stefan Evert’s advice:
I
I
I
I
Tweak your space, don’t tweak the similarity measure!
Map data to higher dimensionality space, and compute
cosine there
Some functions applied to two vectors give a result that is
equivalent to computing the cosine in a higher
dimensionality space, without requiring explicit computation
(the “kernel trick”)
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Dimensionality reduction
I
Distributional semantics recap:
I
I
I
I
Dimensionality reduction:
I
I
Collect co-occurrence counts for target words and corpus
contexts
(Optionally transform the raw counts into some other score)
Build the target-word-by-context matrix, with each word
represented by the vector of values it takes on each
contextual dimension
Reduce the target-word-by-context matrix to a lower
dimensionality matrix (a matrix with less – linearly
independent – columns/dimensions)
Two main reasons:
I
I
Smoothing: capture “latent dimensions” that generalize
over sparser surface dimensions (SVD)
Efficiency/space: sometimes the matrix is so large that you
don’t even want to construct it explicitly (Random Indexing)
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Singular Value Decomposition
I
I
General technique from Linear Algebra (essentially, the
same as Principal Component Analysis, PCA)
Given a matrix (e.g., a word-by-context matrix) of m × n
dimensionality, construct a m × k matrix, where k << n
(and k < m)
I
I
I
E.g., from a 20,000 words by 10,000 contexts matrix to a
20,000 words by 300 “latent dimensions” matrix
k is typically an arbitrary choice
From linear algebra, we know that and how we can find the
reduced m × k matrix with orthogonal dimensions/columns
that preserves most of the variance in the original matrix
Preserving variance
●
●
2
●
●
●
●
●
●
●
●
1
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ● ● ●
●
0
●
●
●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
● ●
●●
● ●
● ●●
● ● ●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
−2
dimension 2
●
variance = 1.26
−2
−1
0
dimension 1
1
2
Preserving variance
●
●
2
●
●
●
●
●
●
●
●
1
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ● ● ●
●
0
●
●
●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
● ●
●●
● ●
● ●●
● ● ●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
−2
dimension 2
●
−2
−1
0
dimension 1
1
2
Preserving variance
●
●
2
●
●
●
●
●
●
●
●
●
1
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●●●●
●
●
●
●
●
● ●
●● ● ●
●
●●
● ●
●
●
●
●●
●
●●● ●
● ●●
●
●
●
●
●
●
●
●●● ● ●
●
●
●
●
●
●
●
●●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
● ●●
●
●
●
●●
●
●
●
●
●●
●
●
●●
●●
●
●
●●
●●
●
●●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●
●
●
0
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●
−2
dimension 2
●
variance = 0.36
−2
−1
0
dimension 1
1
2
Preserving variance
●
●
2
●
●
●
●
●
●
●
●
1
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ● ● ●
●
0
●
●
●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
● ●
●●
● ●
● ●●
● ● ●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
−2
dimension 2
●
−2
−1
0
dimension 1
1
2
Preserving variance
●
●
2
●
●
●
●
●
●
●
1
●
●
●
●
●
●
●
●
0
●
●
●
●
●
●
● ●●
● ●
● ●● ●
●
● ●●
●
●
●●
●●
●
● ●
●
●●● ●●
●●
●●
●
●
●
●
●●●
●●
●●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●●
●●●
●
●
●
●●●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
−1
●
●●
●
●
●
●
●
●
●●● ● ● ●
●●
●
●●
●
●●●
●
●
●
●
●
−2
dimension 2
●
variance = 0.72
−2
−1
0
dimension 1
1
2
Preserving variance
●
●
2
●
●
●
●
●
●
●
●
1
●
●
●
●
●
●
●
●
●
●
●
●
●
●●● ● ● ●
●
0
●
●
●
●
●
●
●
●
●
●
●
●
●
−1
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
● ●
●●
● ●
● ●●
● ● ●●
●
●●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
−2
dimension 2
●
−2
−1
0
dimension 1
1
2
Preserving variance
●●
●
●
●
●
2
●
●
●
●
●
●
●●
●
●
●
●
●
●
1
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●●
● ●
●
●
●● ●●
● ●
● ●●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
● ●
● ●●
●
●
●●
●●
● ●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●● ● ● ●
0
●
−1
●
●
●
●
●
●
−2
dimension 2
●
variance = 0.9
−2
−1
0
dimension 1
1
2
The Singular Value Decomposition
I
Given original matrix A, compute covariance matrix
C = AAT
I
Find top k eigenvalues λ21 , λ22 , ..., λ2k of C and
corresponding eigenvectors (of length 1) ~u1 , ~u2 , ..., ~uk
The matrix U with columns λ1~u1 , λ2~u2 , ..., λ~uk is the best
m × k approximation to A
Pi=k 2
i=1 λi is the amount of variance preserved after
dimensionality reduction
I
I
60
●
●
40
●
20
●
0
context 2
80
100
Dimensionality reduction as generalization
0
20
40
60
context 1
80
100
Dimensionality reduction as generalization
wine
beer
car
cocaine
buy
31.2
15.4
40.5
3.2
sell
27.3
16.2
39.3
22.3
dim1
41.3
22.3
56.4
18.3
Interpreting the latent dimensions
I
Not too transparent, they seem to capture broad
semantic/topical areas
I
Words with highest positive and negative values on 2
randomly picked dimensions of a BNC-trained
window-based semantic space (Baroni & Lenci in press):
Dim
5
15
Top words
political, rhetoric, ideology,
thinking, religious
juice, colouring, dish
cream, salad
Bottom words
around, average, approximately,
compare, increase
police, policeman, road
drive, stop
SVD: Empirical assessment
I
In general, SVD really boosts performance (see, e.g., the
classic LSA work, Schütze 1997, Rapp 2003, Turney
2006. . . )
I
Results from our experiments with ukWaC-trained
dependency-filtered model on next slide (from
Herdaǧdelen et al 2009)
I
Picked top 300 latent dimensions
Classic semantic similarity tasks
with and without SVD reduction
task
with
SVD
RG
0.798
AP Cat
0.701
Hodgson
synonym 10.015
coord
11.157
antonym
7.724
conass
9.299
supersub 10.422
phrasacc
3.532
without
SVD
0.689
0.704
6.623
7.593
5.455
6.950
7.901
3.023
SVD and data sparseness
I
The smoothing provided by SVD is sometimes (e.g., by
Manning & Schütze) argued to help attenuating the
problem of data sparseness
I
I
As corpus size decreases, words are less likely to occur in
the very same contexts, but if they occur in globally similar
contexts, SVD might capture their similarity
In Herdaǧdelen et al (2009), we investigated robustness of
SVD against data sparseness by re-training the models on
down-sampled versions of ukWaC source corpus: 0.01%
(about 2 million words), 1% (about 20 million words), 10%
(about 200 million words)
Performance after down-sampling
Percentage proportion of full set performance
RG
AP Cat
synonym
antonym
conass
coord
phrasacc
supersub
0.1%
svd no svd
0.219
0.244
0.379
0.339
0.369
0.464
0.449
0.493
0.187
0.260
0.282
0.362
0.268
0.132
0.313
0.353
svd
0.676
0.723
0.493
0.768
0.451
0.527
0.849
0.645
1%
no svd
0.700
0.622
0.590
0.585
0.498
0.570
0.610
0.601
10%
svd no svd
0.911
0.829
0.923
0.886
0.857
0.770
1.044
0.849
0.857
0.704
0.927
0.810
0.920
0.868
0.936
0.752
SVD: Pros and cons
I
Pros:
I
I
I
Good performance
At least some indication of robustness against data
sparseness
Smoothing as generalization
I
I
Hare (2006): use SVD to smooth across textual and visual
features, generalizing from captioned images to unannotated
ones
Cons:
I
I
I
Non-incremental (SVD done once and for all on the full
co-occurrence matrix)
Latent dimensions are difficult to interpret
Does not scale up well
I
In my experience, problems SVD-ing matrices with over
20K × 20K dimensions
Outline
Building the model
Weighting dimensions
Cosine similarity
Dimensionality reduction
Singular Value Decomposition
Random Indexing
Random Indexing: The low-cost alternative
Sahlgren 2005
I
A pure dimensionality reduction technique (no “latent
semantics” effect)
I
No need to build the full co-occurrence matrix
I
Given m target words and the intended number of
dimensions k , start and end with an m × k matrix, for a
pre-determined k , independently of number n of
considered contexts
Random Indexing
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
–> dog 0 -1 0 0
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
–> dog 0 -1 0 0
owner of the dog
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
–> dog 0 -1 0 0
owner of the dog –> dog 1 -1 0 0
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
–> dog 0 -1 0 0
owner of the dog –> dog 1 -1 0 0
dog on the leash
Random Indexing
I
I
I
Represent each context element with a (low-dimensional)
index of randomly assigned 1, -1 and (mostly) 0
pet
0 -1 0 0
owner 1 0 0 0
leash -1 0 -1 0
(In this example unrealistically low k = 4)
As you go through corpus, add random index
corresponding to each context to target word contextual
vector:
dog 0 0 0 0
dog is a pet
–> dog 0 -1 0 0
owner of the dog –> dog 1 -1 0 0
dog on the leash –> dog 0 -1 -1 0
Random Indexing
I
With a reasonably large k (a few thousands dimensions),
good low dimensionality approximation of full
co-occurrence matrix
I
Cosine similarity (or other similarity measure) computed on
resulting contextual vectors
Pros and cons
I
Pros:
I
Very efficient: low dimensionality from the beginning to the
end, independently of number of contexts taken into
account
I
I
I
I
This will become more and more important as we work with
larger and larger corpora, richer contexts, etc.
Implementation trivial (assign random values to vector, sum
vectors)
Incremental: at any stage, target vectors constitute
low-dimensional semantic space
Cons:
I
No latent semantic space effect: contexts are “squashed”
randomly
I
I
Multiple-pass Random Indexing? Perform SVD on RI output?
Lower accuracy, at least on some tasks (Gorman and
Curran 2006)