Learning with Parallel Vector Field
Xiaofei He, Zhejiang University
1
The Problem
信息
(训练集)
𝑓
𝑓: 𝑋 → 𝑌
2
Bird
我们考虑的𝑋和𝑌往往
是欧氏空间
Learning with Manifold: the Challenge
Manifold is unknown!

We have only data samples.
3
Isometry
Local isometry



Preserve metric (locally) of the manifold
Hessian locally linear embedding (HLLE)
Global isometry: Our ultimate goal



Preserve pairwise geodesics of the manifold
Isomap
Global isometry is difficult to
achieve directly !
4
Local Isometry
A differentiable mapping 𝜙 is called local isometry at
𝑝 ∈ ℳ if it is a diffeomorphism and preserve metric.
∀𝑋1 , 𝑋2 ∈ 𝑇𝑝 ℳ
𝑔 𝑋1 , 𝑋2 = 𝜙 ∗ ℎ 𝑋1 , 𝑋2 = ℎ 𝑑𝜙 𝑋1 , 𝑑𝜙 𝑋2
where 𝑔, ℎ are metric tensors on ℳ, 𝒩 respectively



Local isometry preserves geodesic distance in a neighborhood
Inner product in the tangent space is preserved
𝑑𝜙
𝑝
𝑇𝑝 ℳ
(ℳ, 𝑔)
5
𝜙(𝑝)
𝑇𝜙(𝑝) 𝒩
𝜙
(𝒩, ℎ)
Global Isometry
𝑑ℳ 𝑝, 𝑞 = 𝑑𝒩 𝜙 𝑝 , 𝜙(𝑞) , ∀𝑝, 𝑞 ∈ ℳ


Pairwise geodesic distance is preserved
A global isometry is a bijective distance preserving
map

𝜙
𝑝
𝜙(𝑝)
𝑑𝒩 (𝜙(𝑝), 𝜙(𝑞))
𝑑ℳ (𝑝, 𝑞)
𝑞
ℳ
𝜙(𝑞)
𝒩
6
Local and Global
Local isometry + bijection = Global isometry
 The smooth map 𝜙: ℳ → 𝒩 is a bijection
⟺ There is a homeomorphism between ℳ and 𝒩
⟺ ℳ and 𝒩 have the same topology
 Local isometry + topology = Global isometry
 An example that a local isometry is not global
isometry, since the topology changes!

𝜙 𝑡 = (cos 𝑡 , sin 𝑡 )
(𝑜, 2𝜋 + 𝜖)
7
Overlapped!
Topology
changes!
Limitations of Isomap (Based on Global Isometry)
Computation cost


Dense pairwise relationship
Incapable of handling non-convex data


Distortion of Isomap
A non-convex subset of the 2D
Euclidean space
8
Embedding obtained by Isomap
when convexity assumption breaks
Local Isometry: the Perspective of Vector Fields
Consider 𝜙: ℳ → 𝒩 = ℝ𝑑 , 𝜙 = 𝜙1 , … , 𝜙𝑑 , the
following three statements are equivalent
 1. 𝜙 is a local isometry
They are all defined locally!
 2. 𝒅𝝓 is orthonormal
 3. 𝑑𝜙𝑗 = 1, 𝑗 = 1, … , 𝑑 : normalization
𝑑𝜙𝑗 ⊥ 𝑑𝜙𝑘 , 𝑗, 𝑘 = 1, … , 𝑑; 𝑗 ≠ 𝑘 : orthogonality
𝒅𝝓𝟐
𝒅𝝓𝟏
9
Local Isometry: the Perspective of Vector Fields
Consider 𝜙: ℳ → 𝒩 = ℝ𝑑 , 𝜙 = 𝜙1 , … , 𝜙𝑑 , the
following three statements are equivalent
 1. 𝜙 is a local isometry
They are all defined locally!
 2. 𝒅𝝓 is orthonormal
 3. 𝑑𝜙𝑗 = 1, 𝑗 = 1, … , 𝑑 : normalization
𝑑𝜙𝑗 ⊥ 𝑑𝜙𝑘 , 𝑗, 𝑘 = 1, … , 𝑑; 𝑗 ≠ 𝑘 : orthogonality
𝒅𝝓𝟐
Problem: how to find
vector fields that they are
orthonormal globally ?
10
𝒅𝝓𝟏
Vector Fields We Need
Considering our problem in Euclidean space
 We can show that the vector fields has to be constant.
 For constant vector fields 𝑑𝜙𝑖 :


𝜙𝑖 preserve distance along the direction of 𝑑𝜙𝑖
𝑑𝜙𝑖 satisfies global orthonormality
𝒅𝝓𝟐
𝒅𝝓𝟏

11
Naturally, we can extend to the case of general manifold.
Basic Idea
Let 𝑉𝑖 , 𝑖 = 1, … , 𝑑 be the vector fields on manifold, then
 1. Each 𝑉𝑖 is as ‘constant’ as possible
min 𝐸 𝑉 =
𝛻𝑉
ℳ


2. They are normalized
𝑉(𝑥) = 1, ∀𝑥 ∈ ℳ
3. We can find a gradient field which can best fit 𝑉𝑖
min Φ 𝑓 =
𝛻𝑓 − 𝑉
ℳ

2
𝐹 𝑑𝑥
2 𝑑𝑥
4. They are orthogonal
𝑉𝑗 ⊥ 𝑉𝑘 , 𝑗, 𝑘 = 1, … , 𝑑; 𝑗 ≠ 𝑘
12
An Example
The unit-norm tangent vector field on a circle.
This is a constant vector field, but not a gradient
field!
13
Parallel Vector Field Embedding
Main Theorem
Let ℳ be a 𝑑-dimensional Riemannian manifold
embedded in ℝ𝑚 with induced metric on it. Assume
there is an isometry 𝜙: ℳ → 𝐷 ⊂ ℝ𝑑 , where 𝐷 is an
open connected subset of ℝ𝑑 . Then for a basis 𝑉𝑖 𝑑𝑖=1
of the null space of 𝐸(𝑉), ∃ 𝑓𝑖 : ℳ → ℝ whose gradient
fields satisfy 𝛻𝑓𝑖 = 𝑉𝑖 , 𝑖 = 1 … , 𝑑. And 𝑓 is an isometry.
𝐸 𝑉 =
𝛻𝑉
ℳ
14
2
𝐹 𝑑𝑥,
𝑠. 𝑡. 𝑉(𝑥) = 1, ∀𝑥 ∈ ℳ
PFE Results: Isometric embedding



Geometrical condition: the manifold can be
isometrically embedded into Euclidean space
Topology assumption is satisfied
Example: Swiss Roll
15
PFE Results: Isometric embedding



Geometrical condition: the manifold can be
isometrically embedded into Euclidean space
Topology assumption is satisfied
Example: Swiss Roll with Hole
16
PFE Results: As Isometric As Possible

Geometrical condition: the manifold cannot be
embedded in ℝ𝑑 isometrically


PFE tries to embed it as isometric as possible
Example: Gaussian
17
Implementation: Overview
Estimate the tangent space
1.

The geometric property of the manifold
Find the parallel vector field
2.

The property of mapping
Calculate the embedding
3.

18
The result
Implementation: Tangent Space

Local PCA



At each local neighborhood, SVD decomposition is
performed on the centered samples
The components corresponding to the leading 𝑑 singular
values are selected as a basis for the tangent space
Other more sophisticated algorithms can be used if
needed
19
Implementation: Parallel Vector Field
𝛻𝑉
𝑀
2
𝐹 𝑑𝑥
discrete
𝑛
𝑛
𝛻𝑉 𝑥𝑖
𝑖=1
2
𝐹
=
𝑃𝑖 𝑉 𝑥𝑗 − 𝑉 𝑥𝑖
2
𝑤𝑖𝑗
𝑖=1 𝑗~𝑖
The covariant derivative along
𝑢𝑖𝑗 = 𝑧𝑖𝑗 / 𝑧𝑖𝑗 is approximated
by 𝛻𝑢𝑖𝑗 𝑉 =
𝒫𝑖 𝑉 𝑥𝑗 −𝑉(𝑥𝑖 )
𝑑ℳ (𝑥𝑖 ,𝑥𝑗 )
, where
𝑥𝑗 = exp𝑥𝑖 (𝑧𝑖𝑗 ) and exp is the
exponential map.
20
Implementation: Embedding

Let 𝑦𝑖 = 𝑓 𝑥𝑖 , discretize the objective function
Φ 𝐲 =
𝛻𝑓 − 𝑉
ℳ

2
=
𝛻𝑓 𝑥𝑖 − 𝑉 𝑥𝑖
2
𝑖
By 1st order Taylor expansion in the local
𝜕Φ(𝐲)
neighborhood of 𝑥𝑖 , and setting
= 0, we can get
𝜕𝐲
the following equations:
𝑇
𝒫𝑖 𝑥𝑗 − 𝑥𝑖 𝑉 𝑥𝑖 = 𝑓 𝑥𝑗 − 𝑓(𝑥𝑖 )

Embedding is obtained from a linear system 𝐸𝐲 = 𝑏
21
Implementation: Summary
22
Experiment Results (Swiss Roll with Hole)
23
Experiment Results (Noisy Swiss Roll)
Embedding performance (average R-score of 10 random repetitions)
on samples from Swiss Roll with different scales (𝜎 2 ) of Gaussian
noises.
24
Experiment Results (Noisy Swiss Roll)
25
Vector Field of PFE (Local View)
26
Vector Field of Isomap (Local View)
27
Manifold Ranking


nearby points are likely to have the same ranking
scores.
points on the same structure (cluster or manifold)
are likely to have the same ranking scores..
(a) Two moons ranking problem
28
(b) Ranking by Euclidean distance
(c) Ideal ranking
PFRank

We aim to design a ranking function that has the
highest value at the query point ‘+’, and then
decreases linearly along the geodesics of the
manifold.
29
Objective function of PFRank



Brief review
2
𝑅0 𝑥𝑞 , 𝑦𝑞 , 𝑓 = 𝑓𝑞 − 𝑦𝑞
𝑅1 𝑓, 𝑉 = ℳ 𝛻𝑓 − 𝑉 2 𝑑𝑥
2
𝐹 𝑑𝑥
parallel vector field term

𝑅2 𝑉 =

But these terms are not enough for producing a good
ranking function. In the above result using parallel vector
field embedding directly, the center point is not given the
highest value ( shown by the size of the circle ), although
the three terms are already 0.
30
ℳ
𝛻𝑉
query term
linear function term
Add a new term

We need to add a term to change the direction of
the parallel field towards the maximum point ( the
query point) .
2

𝑅3 𝑉 = Σ𝑗~𝑞 𝑉𝑥𝑗 − 𝑃𝑗 𝑥𝑞 − 𝑥𝑗

Ensuring 𝑉𝑥 𝑗 and 𝑃𝑗 (𝑥𝑞 − 𝑥𝑗 ) to be similar forces the
vector field at 𝑥𝑗 to point from 𝑥𝑗 to 𝑥𝑞
31
Objective function of PFRank

𝐽 𝑓, 𝑉 = 𝑅0 + 𝜆1 𝑅1 + 𝜆2 𝑅2 + 𝜆3 𝑅3
min 𝐽(𝑓, 𝑉)

Set the derivatives to be 0

𝑓,𝑉
𝜕𝐽 𝑓,𝑉

𝜕𝑓

=
𝜕𝐽 𝑓,𝑉
0,
𝜕𝑉
=0
This finally leads to a linear equation system which
can be solved efficiently.
32
Experimental results

PFRank performs as expected.
33
Conclusion


A new perspective for manifold learning: vector fields.
Two new algorithms: Parallel Vector Field Embedding
and Ranking
34
Next Challenge: Topology!
Hairy Ball Theorem

There does not exist an everywhere nonzero vector
field on the 2-sphere 𝑆 2 . This implies that somewhere
on the surface of the Earth, there is a point with zero
horizontal wind velocity.
35
36