Fast Sparse
Matrix Multiplication
Raphael Yuster
Haifa University (Oranim)
Uri Zwick
Tel Aviv University
ESA 2004
1
Matrix multiplication
j
i
A  ( ai j )

B  (bi j )
=
C  ( ci j )
n
cij   aik bkj
k 1
2
Matrix multiplication
Complexity
Authors
3
n
-
2.81
n
Strassen (1969)
n2.38
Coppersmith, Winograd (1990)
3
Sparse Matrix Multiplication

=
n - number of rows and columns
m - number of non-zero elements
The distribution of the non-zero
elements in the matrices is arbitrary!
4
Sparse Matrix Multiplication
j
k
k
=

Each element of B is multiplied
by at most n elements from A.
Complexity:
mn
5
Matrix multiplication
Complexity
Authors
2.38
n
Coppersmith, Winograd
(1990)
mn
-
0.7
1.2
2+o(1)
m n +n
here
Can we combine the two?
6
Comparison
mn
m0.7n1.2+n2

n2.38
Complexity =
n
r (m=nr)
7
A closer look at the naïve algorithm
=

a11
a 21
a 31
a11b11 a11b12
a21b11 a21b12
b11 b12 b13

=
8
Complexity of the naïve algorithm
a1 a2 a p
an
b1
b2
bp
bn
 a b   a  n  mn
Complexity =
i i
i
where
i ai , i bi  m
i
i
0  ai , bi  n
9
Best case for naïve algorithm
Regular case:
m
ai  bi 
n
 
m
i aibi  n  n
2
2
m

n
10
Worst case for naïve algorithm
m
ai  bi  n , 1  i 
n
m
ai  bi  0 ,
in
n
m 2
i aibi  n  n  mn
11
Worst case for naïve algorithm
m
ai  bi  n , 1  i 
n
ai  bi  0 ,
0
m
n

m
in
n
0
=
12
Rectangular Matrix multiplication
p
n
n
n
 p
= n
Coppersmith (1997):
Complexity ≤ n1.85p0.54+n2+o(1)
For p ≤ n0.29, complexity = n2+o(1) !!!
13
The combined algorithm
a1 a2 a p
an
b1
b2
A1
B1
bp
A2
B2
bn
Naïve sparse
Fast rectangular
Assume: a1b1 ≥ a2b2 ≥ … ≥ anbn
matrix multiplication
matrix multiplication
Choose:
0≤p≤n
Compute:
Complexity:
AB = A1B1+ A2B2
1.85
n
p
0.54
n
2+o(1)
  ak bk
k p
14
Analysis of combined algorithm
Theorem: There exists a p for which
n1.85 p 0.54  n 2+o(1)   ak bk
 m0.7 n1.2  n 2  o (1)
k p
2
Lemma:
m
ak bk 

p
k p
15
Multiplying three sparse matrices
A

Complexity of new
algorithm:
B

C
m0.64n1.46+n2+o(1)
n - number of rows and columns
m - number of non-zero elements
16
Applications
• Computing the square of a sparse graph
• Finding short cycles (YZ’04)
• Other applications?
17
Open problems
• A faster, more sophisticated, algorithm
for sparse matrix multiplication?
• A faster algorithm for multiplying
three or more sparse matrices?
• An O(m1-n1+) transitive closure algorithm?
18