Jure Leskovec
Carnegie Mellon University
Joint work with Christos Faloutsos, Jon Kleinberg and Deepay Chakrabarti
a) World wide web
d) Communication
b) Internet (AS)
e) Citations
Jure Leskovec: Kronecker Graphs
c) Social networks
f) Protein interactions
2
10
Large networks share many structural properties
ƒ Scale‐free (power‐law degree distributions)
ƒ 6‐degrees of separation
ƒ Transitivity
ƒ
And we have models to think about them:
ƒ Preferential attachment
ƒ Small world
ƒ Copying model
Jure Leskovec: Kronecker Graphs
log pk
ƒ
10
10
10
10
5
Power‐law:
4
3
2
1
log‐log
10
0
10
0
10
1
10
2
log k
10
3
10
4
MSN Messenger network
(240M nodes, 1.3B edges)
Avg. path length=6.6
3
[KDD ’05]
ƒ
ƒ
ƒ
Internet
E(t)
What is the relation between the number of nodes and the edges over time?
a=1.2
Prior work assumes: constant average degree over time
Networks are denser over time Densification Power Law:
a … densification exponent (1 ≤ a ≤ 2)
Jure Leskovec: Kronecker Graphs
N(t)
Citations
E(t)
ƒ
a=1.6
N(t)
4
[KDD ’05]
Internet
diameter
Prior models and intuition say that the network diameter slowly grows (like log N, log log N)
ƒ
size of the graph
ƒ
Diameter shrinks over time
ƒ as the network grows the distances between the nodes slowly decrease
Jure Leskovec: Kronecker Graphs
diameter
Citations
time
5
None of the existing models generates graphs with these properties
ƒ We need a new model
ƒ
ƒ
Idea: Generate graphs recursively
Recursion (fractals)
Jure Leskovec: Kronecker Graphs
Skewed distributions (power‐laws)
6
ƒ
There are many obvious (but wrong) ways:
Initial graph
Recursive expansion
ƒ Does not densify, has increasing diameter
ƒ
Kronecker Product is a way of generating self‐similar matrices
Jure Leskovec: Kronecker Graphs
7
[PKDD ’05]
Intermediate stage
(3x3)
Adjacency matrix
(9x9)
Jure Leskovec: Kronecker Graphs
Adjacency matrix
8
[PKDD ’05]
ƒ
Kronecker product of matrices A and B is given by
NxM
KxL
N*K x M*L
ƒ
We define [PKDD ‘05] a Kronecker product of two graphs as a Kronecker product of their adjacency matrices
Jure Leskovec: Kronecker Graphs
9
[PKDD ’05]
ƒ
We propose a growing sequence of graphs by iterating the Kronecker product
ƒ
Each Kronecker multiplication exponentially increases the size of the graph
Gk has N1k nodes and E1k edges, so we get densification
ƒ
Jure Leskovec: Kronecker Graphs
10
[PKDD ’05]
ƒ
Continuing multypling with G1 we obtain G4 and so on …
G4 adjacency matrix
Jure Leskovec: Kronecker Graphs
11
[PKDD ’05]
ƒ
ƒ
ƒ
Create N1×N1 probability matrix P1
Compute the kth Kronecker power Pk
For each entry puv of Pk include an edge (u,v) with probability puv
0.5 0.2
0.1 0.3
P1
Kronecker
multiplication
0.25 0.10 0.10 0.04
0.05 0.15 0.02 0.06
0.05 0.02 0.15 0.06
0.01 0.03 0.03 0.09
P2=P1⊗P1
Jure Leskovec: Kronecker Graphs
Probability of edge pij
Instance
matrix K2
flip biased coins
12
[PKDD ’05]
ƒ
Intuition
ƒ Recursive growth of graph communities
ƒ Nodes get expanded to micro communities
ƒ Nodes in sub‐community link among themselves and to nodes from different communities
Jure Leskovec: Kronecker Graphs
13
ƒ
Node attribute representation
ƒ Nodes are described by attributes
▪ u=[1,0], v=[1, 1]
ƒ Parameter matrix gives linking probability:
v
p(u,v) = 0.1 * 0.5 = 0.15
1
1
0
11
0
0.5 0.2
0.1 0.3
Kronecker
multiplication
Jure Leskovec: Kronecker Graphs
u
10
01
00
11
0.25 0.10 0.10 0.04
10
0.05 0.15 0.02 0.06
01
0.05 0.02 0.15 0.06
00
0.01 0.03 0.03 0.09
14
ƒ
Using multiple initiators:
Attribute 1:
….
0.9
0.4
0.2
0.3
0.8
0.2
0.3
0.1
0.3
….
….
pij= 0.4 * 0.5 Attribute 2:
0.8
0.5
0.1
0.2
0.9
0.4
0.4
0.2
0.3
Jure Leskovec: Kronecker Graphs
15
[PKDD ’05]
ƒ
We prove [PKDD05] that Kronecker graphs have the following structural properties:
ƒ Properties of static networks
9Power Law Degree Distribution
9Power Law eigenvalue and eigenvector distribution
9Small Diameter
ƒ Properties of dynamic networks
9Densification Power Law
9Shrinking/Stabilizing Diameter
Jure Leskovec: Kronecker Graphs
16
[PKDD ’05]
ƒ
Theorem: Kronecker Graphs have multinomial
in‐ and out‐degree distribution
(which can be made to behave like a Power Law)
ƒ
Proof:
ƒ Let G1 have degrees d1, d2, …, dN
ƒ Kronecker multiplication with a node of degree d
gives degrees d∙d1, d∙d2, …, d∙dN
ƒ After Kronecker powering Gk has multinomial degree distribution
Jure Leskovec: Kronecker Graphs
17
[PKDD ’05]
ƒ
Theorem: The Kronecker Graph has multinomial distribution of its eigenvalues
ƒ
Theorem: The components of each eigenvector in Kronecker Graph follow a multinomial distribution
ƒ
Proof: Trivial by properties of Kronecker multiplication
Jure Leskovec: Kronecker Graphs
18
[PKDD ’05]
ƒ
Theorem: Kronecker graphs follow a Densification Power Law with densification exponent
ƒ
Proof:
ƒ If G1 has N1 nodes and E1 edges then Gk has Nk =
N1k nodes and Ek = E1k edges
ƒ And then Ek = Nka
ƒ Which is a Densification Power Law Jure Leskovec: Kronecker Graphs
19
[PKDD ’05]
ƒ
Theorem: Constant diameter: If G1 has diameter d then graph Gk also has diameter d
ƒ
Observation: Edges in Kronecker graphs:
where X are appropriate nodes
ƒ
Example:
Jure Leskovec: Kronecker Graphs
20
ƒ
Why models and realistic synthetic graphs:
ƒ Anomaly detection – abnormal behavior, evolution
ƒ Predictions – predicting future from the past
ƒ Simulations of new algorithms where real graphs are hard/impossible to collect
ƒ Graph sampling – many real world graphs are too large to deal with
ƒ “What if” scenarios
Jure Leskovec: Kronecker Graphs
21
[ICML ’07]
Want to generate realistic networks:
ƒ
Given a real network
ƒ
ƒ
ƒ
ƒ
Generate a synthetic network
Compare graphs properties, e.g., degree distribution
Good news: Kronecker graphs have the expressive power
But: How do we choose the parameters to match all of these at once?
Q: Which network properties do we care about?
A: Don’t commit, let’s match adjacency matrices
Jure Leskovec: Kronecker Graphs
22
[ICML ’07]
ƒ
Maximum likelihood estimation:
arg max
Θ
ƒ
P(
| Kronecker
Naïve estimation takes O(N!N2):
Θ) Θ=
a b
c d
ƒ N! for different node labelings:
ƒ N2 for traversing graph adjacency matrix
ƒ
Do stochastic gradient descent
We (KronFit) estimate the model in O(E)
Jure Leskovec: Kronecker Graphs
23
[ICML ’07]
G
ƒ
Given a graph G and Kronecker matrix Θ we calculate probability that Θ generated G P(G|Θ)
0.5 0.2
0.1 0.3
Θ
0.25
0.05
0.05
0.01
0.10
0.15
0.02
0.03
0.10
0.02
0.15
0.03
0.04
0.06
0.06
0.09
Θk
P(G|Θ)
1
0
1
1
0
1
0
1
1
0
1
1
1
1
1
1
G
P(G | Θ) = Π Θ k [u , v] Π (1 − Θ k [u , v])
( u ,v )∈G
( u ,v )∉G
Jure Leskovec: Kronecker Graphs
24
[ICML ’07]
Θk
Θ
0.25
0.10
0.10
0.04
0.5
0.2
0.05
0.15
0.02
0.06
0.1
0.3
0.05
0.02
0.15
0.06
0.01
0.03
0.03
0.09
1
3
2
4
2
G”
4
1
3
ƒ
ƒ
σ
G’
ƒ
1
0
1
0
0
1
1
1
1
1
1
1
0
0
1
1
1
0
1
1
0
1
0
1
1
0
1
1
1
1
1
1
P(G’|Θ) = P(G”|Θ)
Nodes are unlabeled
Graphs G’ and G” should have the same probability
P(G’|Θ) = P(G”|Θ)
One needs to consider all node correspondences σ
P(G | Θ) = ∑σ P(G | Θ, σ )P(σ )
ƒ
All correspondences are a priori equally likely
ƒ
There are O(N!)
correspondences
Jure Leskovec: Kronecker Graphs
25
[ICML ’07]
ƒ
ƒ
Assume we solved the correspondence problem
Calculating
P(G | Θ) = Π Θ k [σ u , σ v ] Π (1 − Θ k [σ u , σ v ])
( u , v )∈G
ƒ
ƒ
( u ,v )∉G
Takes O(N2) time
Infeasible for large graphs (N ~ 105)
σ… node labeling
0.25
0.10
0.10
0.04
1
0
1
1
0.05
0.15
0.02
0.06
0
1
0
1
0.05
0.02
0.15
0.06
1
0
1
1
0.01
0.03
0.03
0.09
0
0
1
1
Θkc
Jure Leskovec: Kronecker Graphs
σ
P(G|Θ, σ)
G
Part 1‐26
[ICML ’07]
ƒ
Log‐likelihood
ƒ
Gradient of log‐likelihood
ƒ
Sample the permutations from P(σ|G,Θ) and average the gradients
Jure Leskovec: Kronecker Graphs
27
[ICML ’07]
ƒ
Metropolis sampling
ƒ Start with a random permutation σ
ƒ Do local moves on the permutation
ƒ Accept the new permutation σ’
1
▪ If new permutation is better (gives higher likelihood)
▪ else accept with prob. proportional to the ratio of likelihoods (no need to calculate the normalizing constant!)
4
Swap node labels 1 and 4
3
2
3
2
4
1
2
3
4
1
0
1
0
0
1
1
1
1
1
1
1
0
1
1
1
1
1
2
3
4
1
1
1
0
1
1
1
0
1
1
1
1
0
0
1
1
Jure Leskovec: Kronecker Graphs
Can compute efficiently
Only need to account for changes in 2 rows / columns
28
[ICML ’07]
ƒ
ƒ
Calculating naively P(G|Θ,σ) takes O(N2)
Idea:
ƒ First calculate likelihood of empty graph, a graph with 0 edges
ƒ Correct the likelihood for edges that we observe in the graph
ƒ
By exploiting the structure of Kronecker product we obtain closed form for likelihood of an empty graph
Jure Leskovec: Kronecker Graphs
29
[ICML ’07]
ƒ
We approximate the likelihood:
Empty graph
ƒ
ƒ
ƒ
No‐edge likelihood
Edge likelihood
The sum goes only over the edges Evaluating P(G|Θ,σ) takes O(E) time
Real graphs are sparse, E << N2
We estimate the model in O(E)
Jure Leskovec: Kronecker Graphs
30
[ICML ’07]
ƒ
Experimental setup
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Given real graph
Stochastic gradient descent from random initial point
Obtain estimated parameters
Generate synthetic graphs Compare properties of both graphs
We do not fit the properties themselves We fit the likelihood and then compare the graph properties
Jure Leskovec: Kronecker Graphs
31
[ICML ’07]
We search the space of ~101,000,000 permutations
ƒ Fitting takes 2 hours
ƒ Real and Kronecker are very close
ƒ
Path lengths
node degree
Θ=
0.99 0.54
0.49 0.13
“Network” values
network value
probability
# reachable pairs
Degree distribution
∧
number of hops
Jure Leskovec: Kronecker Graphs
rank
32
[ICML ’07]
ƒ
Fitting scales linearly with the number of edges
Jure Leskovec: Kronecker Graphs
33
ƒ
Kronecker generalizes Erdos‐Renyi (Gnp):
ƒ Use 1x1 initiator (all cells have the same prob.)
ƒ
Kronecker generalizes RMat
ƒ Use 2x2 initiator
ƒ
Relation between Kronecker and RMat
ƒ like Gnp (n nodes, p edge prob), Gnm (n nodes, m edges):
ƒ Kronecker also encodes the number of edges ƒ (in RMat that is a separate parameter)
Jure Leskovec: Kronecker Graphs
34
ƒ
In practice we generate Kronecker graphs using the RMat like recursive deepening
ƒ
ƒ
ƒ
ƒ
Given prob. Kronecker initiator P
Expected number of edges: E = sum(P)k
Normalize P’ = 1/sum(P) * P
Perform recursive deepening using P’
P' =
a b
c d
a
c
a
c
b
b
d
d
a+b+c+d=1
Jure Leskovec: Kronecker Graphs
35
ƒ
Kronecker Graph model has ƒ provable properties
ƒ small number of parameters
ƒ
ƒ
ƒ
ƒ
Scalable algorithms for fitting Kronecker Graphs
Efficiently search large space (~101,000,000) of permutations
Kronecker graphs fit well real networks using few parameters
Kronecker graphs match graph properties without a priori deciding on which ones to fit
Jure Leskovec: Kronecker Graphs
36
ƒ
Graphs over Time: Densification Laws, Shrinking Diameters and Possible Explanations, by Jure Leskovec, Jon Kleinberg, Christos Faloutsos, ACM KDD 2005
ƒ
Realistic, Mathematically Tractable Graph Generation and Evolution, Using Kronecker Multiplication, by Jure Leskovec, Deepay Chakrabarti, Jon Kleinberg and Christos Faloutsos, PKDD 2005
ƒ
Scalable Modeling of Real Graphs using Kronecker Multiplication, by Jure Leskovec and Christos Faloutsos, ICML 2007
ƒ
Graph Evolution: Densification and Shrinking Diameters, by Jure Leskovec, Jon Kleinberg and Christos Faloutsos, ACM TKDD 2007
Jure Leskovec: Kronecker Graphs
37