Introduction to Cover Tree
Javen Qinfeng Shi
SML of NICTA
RSISE of ANU
2006 Oct. 15
Outline
™ Goal
™ Related
works
™ Cover tree definition
™ Optimized Implement in C++
™ Complexity
™ Further work
Outline
™ Goal
™ Related
works
™ Cover tree definition
™ Efficiently Implement in C++
™ Advantages and disadvantages
™ Further work
Goal
™
Speed up Nearest-neighbour search [John Langford etc. 2006]
Z Preprocess
a dataset S of n points in
some metric space X so that given a query
point p ∈ X , a point q ∈ S which
minimises d ( p, q) can be efficiently found
™
Also Kmeans
Outline
™ Goal
™ Related
works
™ Cover tree definition
™ Example
™ Optimized Implement in C++
™ Advantages and disadvantages
™ Further work
Relevant works
™ Kd
tree [Friedman, Bentley & Finkel 1977]
™ Ball tree family:
ZBall tree [Omohundro,1991]
ZMetric tree [Uhlmann, 1991]
ZAnchors metric tree [Moore 2000]
Z(Improved) Anchors metric tree [Charles2003]
™ Navigating
net [R.Krauthgamer, J.Lee 2004] ?
Lots of discussions and proofs on its computational bound
Kd tree
™ Univariate
axis-aligned splits
™ Split on widest dimension
™ O(N log N) to build,
O(N) space
Kd tree
Kd tree
y
x (axis)
Ball tree/metric tree
A set of points in R2
[Uhlmann 1991], [Omohundro 1991]
A ball-tree: level 1
[Uhlmann 1991], [Omohundro 1991]
A ball-tree: level 2
A ball-tree: level 3
A ball-tree: level 4
A ball-tree: level 5
Outline
Goal
™ Related works
™ Cover tree definition
™
Z Basic
definition
Z How to build a cover tree
Z Query a point/points
Optimized Implement in C
™ Advantages and disadvantages
™ Further work
™
Basic definition
™
™
™
™
A cover tree T on a dataset S is a leveled tree where each
level is indexed by an integer scale i which decreases as
the tree is descended
Ci denotes the set of nodes at level i
d(p,q) denotes the distance between poitns p and q
A valid tree satisfies the following properties
Z Nesting: Ci ⊂ Ci−1
Z Covering tree: For every node p ∈Ci −1 , there exists a node q∈Ci
satisfying d ( p, q) ≤ 2i and exactly one such q is a parent of p
Z Separation: For all nodes p, q ∈Ci , d ( p, q) > 2i
Nesting
™ Ci
⊂ Ci −1
Z Each
node in set
Ci has a selfchild
Z All nodes in set
Ci are also
nodes in sets Cj
where j<i
Z Set C-∞ contains
all the nodes in
dataset S
Cover tree
™
For every
node p ∈ Ci−1 ,
there exists a
node q ∈ Ci
satisfying
d ( p, q) ≤ 2i
and exactly
one such q is a
parent of p
Separation
i
d
(
p
,
q
)
>
2
™ For all nodes p, q ∈Ci ,
Cover tree struture
C∞
/ Ca
Root
C∞ −1
C∞ − 2
>2
Ci
< 2i
Ci −1
C−∞
i
Cover tree struture
9
11
8
3
5
10
7
6
1
y
5
4
7
3
2
4
9
2
6
8
1
0
0
2
4
6
x
8
10
How to build a Cover tree
A set of points in R2
How to build a Cover tree
A set of points in R2
How to build a Cover tree
Ca
d_max
Root
a=?
A set of points in R2
How to build a Cover tree
Ca
d_max
Root
a= ceil (log 2 (d _ max))
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
A set of points in R2
How to build a Cover tree
Ca
2^a
Root
Ca −1
2^(a-1)
Continue till every circle
contain only one point
Query for a single point
Ci +1
set Q∞ = C∞
Ci
for i from ∞ down to - ∞
consider the set of children of Qi :
set = {Children(q) : q ∈ Qi }
Ci −1
form next cover set :
Qi −1 = {q ∈ set : d ( p, q) ≤ d ( p, set ) + b}
return arg min q∈Q−∞ d ( p, q )
b=?
Query for a single point
i −1
b = ∑2 = 2
k
−∞
i
Query Examples
9
11
8
3
5
10
7
6
1
y
5
4
7
3
2
4
9
2
6
8
1
0
0
2
4
6
x
8
10
Query for Node 1
9
11
8
3
5
10
7
6
1
y
5
4
7
3
2
4
9
2
6
8
1
0
0
2
4
6
x
8
10
Query for Node 4
9
11
8
3
5
10
7
6
1
y
5
4
7
3
2
4
9
2
6
8
1
0
0
2
4
6
x
8
10
Query for Node 11
9
11
8
3
5
10
7
6
1
y
5
4
7
3
2
4
9
2
6
8
1
0
0
2
4
6
x
8
10
Query for many points
2^(j+1)
Optimized implement in C++
™ Alignment
memory (faster)
™ exploit the four-stage fully pipelined floatingpoint adder (faster)
™ Explicit cover tree (less space)
™ Avoid declaring new variable (less space)
™ Single-precision float constants (less space)
™ Function pointers (instead of switch)
™ Some differences from the paper
Alignment memory
™ pad
data structure to make their sizes a
multiple of a word, doubleword or quadword.
™ Cpu can fetch the data (with multiple size of
a word) from memory only one time,
otherwise have to fetch twice.
Alignment memory
Alignment memory
™ Cover
tree code:
™ Result:
exploit the four-stage fully pipelined
floating-point adder
exploit the four-stage fully pipelined
floating-point adder
™ Cover
tree code:
Explicit cover tree
™
Theory is based on an implicit implementation, but tree is
built with a condensed explicit implementation to
preserve O(n) space bound
Avoid declaring new variable
™ Use
the same variable
Zpoint_set here is the same one
Some differences from the paper
™ Paper:
™ Code:
Z2^i
Z1.3^i
ZScale i reduces (from
ZScale i
top to bottom)
increases(0~100)
n.scale = top_scale
- cur_scale;
Complexity
™ Expasion
constant C
Advantages and disadvantages
™
Advantage:
Z Query
many points at one time are faster (instead of query
each by each)
Z Less
™
required space
Disadvantage:
Z Building cover tree is still time consuming.
Z Expansion constant C(which is related to bound) is
hard to be
accurately determined
Z Recursion might be slower than iteration, and may
overflow the recursion stack in both C++ or python.
Further work
™ Provide
python interface (swig, pyrex)
™ Apply it into other classification methods
(k-means, crf,…)to speed them up.
™Thanks
for coming!