Query processing:
optimizations
Paolo Ferragina
Dipartimento di Informatica
Università di Pisa
Reading 2.3
Augment postings with skip
pointers (at indexing time)
128
41
2
4
11
1
2


8
3
41
48
8
31
11
64
17
How do we deploy them ?
Where do we place them ?
128
21
31
Sec. 2.3
Sec. 2.3
Using skips
128
41
2
4
11
1
2
8
3
41
48
8
31
11
64
17
128
21
31
Suppose we’ve stepped through the lists until we process 8 on
each list. We match it and advance.
We then have 41 and 11 on the lower. 11 is smaller.
But the skip successor of 11 on the lower list is 31, so
we can skip ahead past the intervening postings.
Sec. 2.3
Placing skips

Tradeoff:


More skips  shorter spans  more likely to
skip. But lots of comparisons to skip
pointers.
Fewer skips  longer spans  few successful
skips. Less pointer comparisons.
Sec. 2.3
Placing skips

Simple heuristic for postings of length L




use L evenly-spaced skip pointers.
This ignores the distribution of query terms.
Easy if the index is relatively static.
This definitely useful for in-memory index

The I/O cost of loading a bigger list can
outweigh the gains!
Sec. 2.3
Placing skips, contd

What if it is known a distribution of access pk to
the k-th element of the inverted list?






You can solve it by
Shortest Path
w(i,j) = sumk=i..j pk
L^0(i,j) = average cost of accessing an item in the
sublist from i to j = sumk=i..j pk * (k-i+1)
L^1(1,n) = 1 (first skip cmp) + (cost to access the two lists)
minu>1 w(1,u-1) * L^0(1,u-1) + w(u,n) * L^1(u,n)
L^0(i,j) can be tabulated in O(n^2) time
Computing L^1(i,n) takes O(n), given L^1(j,n), for j>i
Computing the total L^1(1,n) takes O(n^2) time
Sec. 2.3
Placing skips, contd

What if it is also fixed the number of p skippointers that can be allocated?




Same as before but we add the parameter p
L^1_p(1,n) = 1 + min_{u>1} w(1,u-1) * L^0(1,u-1) +
w(u,n) * L^1_{p-1}(u,n)
L^1_0(i,j) = L^0(i,j), i.e. no pointers left, so scan
L^i(j,n) takes O(n) time [min calculation] if are
available the values for L^{i-1}(h,n) with h > j

So L^p(1,n) takes O(pn^2) time
Faster query = caching?
Two opposite approaches:
I.
Cache the query results (exploits query
locality)
II.
Cache pages of posting lists (exploits
term locality)
Query processing:
phrase queries and positional
indexes
Paolo Ferragina
Dipartimento di Informatica
Università di Pisa
Reading 2.4
Sec. 2.4
Phrase queries


Want to be able to answer queries such as
“stanford university” – as a phrase
Thus the sentence “I went at Stanford my
university” is not a match.
Sec. 2.4.1
Solution #1: Biword indexes

For example the text “Friends, Romans,
Countrymen” would generate the biwords




friends romans
romans countrymen
Each of these biwords is now an entry in the
dictionary
Two-word phrase query-processing is
immediate.
Sec. 2.4.1
Longer phrase queries

Longer phrases are processed by reducing
them to bi-word queries in AND
stanford university palo alto can be broken
into the Boolean query on biwords, such as
stanford university AND university palo AND
palo alto
Can have false positives!
Index blows up
Need the docs to verify
+
They are combined with other solutions
Sec. 2.4.2
Solution #2: Positional indexes

In the postings, store for each term and
document the position(s) in which that term
occurs:
<term, number of docs containing term;
doc1: position1, position2 … ;
doc2: position1, position2 … ;
etc.>
Sec. 2.4.2
Processing a phrase query

“to be or not to be”.

to:


be:


2:1,17,74,222,551; 4:8,16,190,429,433;
7:13,23,191; ...
1:17,19; 4:17,191,291,430,434;
5:14,19,101; ...
Same general method for proximity searches
Sec. 7.2.2
Query term proximity



Free text queries: just a set of terms typed
into the query box – common on the web
Users prefer docs in which query terms
occur within close proximity of each other
Would like scoring function to take this into
account – how?
Sec. 2.4.2
Positional index size



You can compress position values/offsets
Nevertheless, a positional index expands
postings storage by a factor 2-4 in English
Nevertheless, a positional index is now
commonly used because of the power and
usefulness of phrase and proximity queries
… whether used explicitly or implicitly in a
ranking retrieval system.
Sec. 2.4.3
Combination schemes

BiWord + Positional index is a profitable
combination


Biword is particularly useful for particular
phrases (“Michael Jackson”, “Britney
Spears”)
More complicated mixing strategies do
exist!
Sec. 7.2.3
Soft-AND

E.g. query rising interest rates

Run the query as a phrase query

If <K docs contain the phrase rising interest
rates, run the two phrase queries rising
interest and interest rates

If we still have <K docs, run the “vector space
query” rising interest rates (…see next…)

“Rank” the matching docs (…see next…)