Section 2.2 - Matrix-Matrix Multiplication:
Definition
Maggie Myers
Robert A. van de Geijn
The University of Texas at Austin
Practical Linear Algebra – Fall 2009
http://z.cs.utexas.edu/wiki/pla.wiki/
1
Theorem
Let LA : Rk → Rm and LB : Rn → Rk both be linear
transformations and, for all x ∈ Rn , define the function
LC : Rn → Rm by LC (x) = LA (LB (x)). Then LC (x) is a linear
transformations.
Proof
Let x, y ∈ Rn and α ∈ R. Then
LC (αx) = LA (LB (αx)) = LA (αLB (x)) = αLA (LB (x)) =
αLC (x).
LC (x + y) = LA (LB (x + y)) = LA (LB (x) + LB (y)) =
LA (LB (x)) + LA (LB (y)) = LC (x) + LC (y).
http://z.cs.utexas.edu/wiki/pla.wiki/
2
Towards a definition of matrix-matrix multiplication
Let linear transformations LA , LB , and LC be represented by
matrices A ∈ Rm×k , B ∈ Rk×n , and C ∈ Rm×n , respectively.
Then Cx = LC (x) = LA (LB (x)) = A(Bx).
The matrix-matrix multiplication (product) is defined so that
C = A × B = AB.
C is the composition of A and B just like LC is the
composition of LA and LB .
Dimensions must match
If
A is mA × nA matrix,
B is mB × nB matrix, and
C is mC × nC matrix.
then for C = AB to hold it must be the case that
mC = mA ,
n = n , and
http://z.cs.utexas.edu/wiki/pla.wiki/
3
How to compute the product of two matrices
Let


γ0,0
γ0,1 · · ·
γ0,n−1

γ1,0
γ1,1 · · ·
γ1,n−1 


C = 
..
..
..
.. 

.
.
.
. 
γm−1,0 γm−1,1 · · · γm−1,n−1


α0,0
α0,1 · · ·
α0,k−1

α1,0
α1,1 · · ·
α1,k−1 


A = 
..
..
..
.. 

.
.
.
. 



B = 

αm−1,0 αm−1,1 · · · αm−1,k−1

β0,0
β0,1 · · ·
β0,n−1
β1,0
β1,1 · · ·
β1,n−1 

..
..
..
..  .
.
.
.
. 
βk−1,0 βk−1,1 · · · βk−1,n−1
http://z.cs.utexas.edu/wiki/pla.wiki/
4
Recall
γi,j = eTi (Cej ): Cej “picks out” the jth column of C and
eTi (Cej ) then picks out the ith element of the jth column.
http://z.cs.utexas.edu/wiki/pla.wiki/
5
C = AB is the matrix such that Cx = A(Bx) for all x. Then
γi,j = eTi (Cej ) =
eTi (A (Bej ) ) .
| {z }
bj
{z
}
|
ith element of Abj
The ith element of vector Abj is given by the dot product of
the ith row of A with the vector bj .
The ith row of A and jth column of B are given by
0
`
αi,0
αi,1
···
αi,k−1
´
and
B
B
B
@
β0,j
β1,j
..
.
βk−1,j
1
C
C
C.
A
γi,j = eTi (A(Bx))ej =
Pk−1
αi,0 β0,j + αi,1 β1,j + · · · + αi,k−1 βk−1,j = p=0
αi,p βp,j .
http://z.cs.utexas.edu/wiki/pla.wiki/
6
Definition of Matrix-matrix multiplication
Let A ∈ Rm×k , B ∈ Rk×n , and C ∈ Rm×n . Then the
matrix-matrix multiplication (product)
C = AB
is computed by setting
γi,j =
k−1
X
αi,p βp,j = αi,0 β0,j + αi,1 β1,j + · · · + αi,k−1 βk−1,j .
p=0
Note
Cx = A(Bx) = (AB)x.
We can drop the parentheses, unless they are useful for clarity:
Cx = (AB)x = A(Bx) = ABx.
http://z.cs.utexas.edu/wiki/pla.wiki/
7
Example: Predicting the Weather
Recall that


0.4 0.3 0.1
P =  0.4 0.3 0.6 
0.2 0.4 0.3
was the transition matrix so that if x was the prediction for
today’s weather, then P x was the prediction for tomorrow’s
weather.
The transition matrix Q so that z = Qx is the prediction for
the day after tomorrow is given by Q = P × P , since
z = P (P x) = Qx:
0
Q=P ×P
=
=
http://z.cs.utexas.edu/wiki/pla.wiki/
10
0.4 0.3 0.1
0.4
@ 0.4 0.3 0.6 A @ 0.4
0.2 0.4 0.3
0.2
0
1
0.30 0.25 0.25
@ 0.40 0.45 0.40 A .
0.30 0.30 0.35
8
0.3
0.3
0.4
1
0.1
0.6 A
0.3
Example: Predicting the Weather
The transition matrix W so that z = W x is the prediction for
a week from today is given by
W
P
P × P × P × P}
| × P × P × {z
seven P s
= P (P (P (P (P (P (P )))))))
=
= (P × P ) × (P × P ) × (P × P ) × P
= (P × P × P ) × (P × P × P ) × P
P ×P ×P
0
0.4
= @ 0.4
0.2
0
0.4
= @ 0.4
0.2
0.3
0.3
0.4
0.3
0.3
0.4
1 00
10
0.1
0.4 0.3 0.1
0.4 0.3
0.6 A @@ 0.4 0.3 0.6 A @ 0.4 0.3
0.3
0.2 0.4 0.3
0.2 0.4
10
1
0.1
0.30 0.25 0.25
0.6 A @ 0.40 0.45 0.40 A = · · ·
0.3
0.30 0.30 0.35
http://z.cs.utexas.edu/wiki/pla.wiki/
9
11
0.1
0.6 AA
0.3
Example: Composing Two Rotations
cos(ρ + σ) − sin(ρ + σ)
C =
sin(ρ + σ)
cos(ρ + σ)
cos ρ − sin ρ
cos σ − sin σ
=
sin ρ
cos ρ
sin σ
cos σ
cos ρ cos σ − sin ρ sin σ − sin ρ cos σ − cos ρ sin σ
=
.
sin ρ cos σ + cos ρ sin σ
cos ρ cos σ − sin ρ sin σ
http://z.cs.utexas.edu/wiki/pla.wiki/
10
Conformal matrices
For matrix-matrix multiplication to be a legal operations, the
row and column dimensions of the matrices must obey the
mentioned constraints.
We say dimensions are conformal if the operation being
performed with the matrices and/or vectors are legal.
http://z.cs.utexas.edu/wiki/pla.wiki/
11
Computing The Matrix-Matrix Product
The following triple-nested loops compute C := AB + C:
for i = 0, . . . , m − 1
for j = 0, . . . , n − 1
for p = 0, . . . , k − 1
γi,j := αi,p βp,j + γi,j
endfor
endfor
endfor
If originally C = 0, then the above algorithm computes C := AB.
http://z.cs.utexas.edu/wiki/pla.wiki/
12
Simple triple-nested loops
for j = 0, . . . , n − 1
for i = 0, . . . , m − 1

for p = 0, . . . , k − 1 
k−1
X
γi,j := αi,p βp,j + γi,j
αi,p βp,j+γi,j
 γi,j :=
endfor
p=0
endfor
endfor
for j=1:n
for i=1:m
for p=1:k
C(i,j) += A(i,p) * B(p,j);
end
end
end
http://z.cs.utexas.edu/wiki/pla.wiki/
13
Double Nested Loop, calling dot
for j = 0, . . . , n − 1
for i = 0, . . . , m − 1

for p = 0, . . . , k − 1 
γi,j := αi,p βp,j + γi,j γi,j := ǎTi bj + γi,j

endfor
endfor
endfor
for j=1:n
for i=1:m
C(i,j) += SLAP_Dot( A(i,:), B(:,j) );
end
end
http://z.cs.utexas.edu/wiki/pla.wiki/
14
Single loop with call to gemv
for j = 0, . . . , n − 1

for i = 0, . . . , m − 1




for p = 0, . . . , k − 1

γi,j := αi,p βp,j + γi,j
cj := Abj + cj


endfor



endfor
endfor
for j=1:n
C(:,j) += ...
SLAP_Gemv( SLAP_NO_TRANSPOSE, ...
1, A, B(:,j), 1, C(:,j) );
end
http://z.cs.utexas.edu/wiki/pla.wiki/
15
Ordering the loops
When computing C = AB + C the three loops can be nested
in 3! = 6 different ways:
ijp, ipj, jip, jpi, pij, pji.
We will examine this more, later.
for = 0, . . . ,
for = 0, . . . ,
for = 0, . . . ,
γi,j := αi,p βp,j + γi,j
endfor
endfor
endfor
http://z.cs.utexas.edu/wiki/pla.wiki/
16