Matrix-Chain Multiplication Problem
•
Dynamic Programming
Examples
Given a chain <A1, A2, …, An> of n matrices
where Ai is of size Pi-1 × Pi
• How can we multiply them so that min # of
scalar multiplications is performed
Recursive solution
m[i, j] = 0
i=j
min {m[i, k] + m[k+1, j] + pi-1pk pj} , i < j
i≤k<j
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
4) for l ← 2 to n
; l is length of chain product
5) do for i ← 1 to n − l + 1
6)
do j ← i + l −1
7)
m[i, j] ← inf
8)
for k ← i to j − 1
9)
do q ← m[i, k] + m[k+1, j] + pi-1pk pj
10)
if q < m[i, j]
11)
then m[i, j] ← q
12)
s[i, j] ← k
13)return m and s
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
4) for l ← 2 to n
; l is length of chain product
5) do for i ← 1 to n − l + 1
6)
do j ← i + l −1
7)
m[i, j] ← inf
8)
for k ← i to j − 1
9)
do q ← m[i, k] + m[k+1, j] + pi-1pk pj
10)
if q < m[i, j]
11)
then m[i, j] ← q
12)
s[i, j] ← k
13)return m and s
m[i, j] =
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
i≤k<j
1
Pseudocode
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3)
do m[i, i] ← 0
4) for l ← 2 to n ; l is length of chain product
5)
do for i ← 1 to n − l + 1
i 1 2 3 ...... (n – l + 1)
6)
do j ← i + l −1
j l l+1 l+2 ...
n
7)
m[i, j] ← inf
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
l
2
3
4
......
4) for l ← 2 to n
5) do for i ← 1 to n − l + 1
j
i+1 i+2 i+3 ... ...
6)
do j ← i + l −1
7) 1
m[i, j] ← inf
each [i, j] has length l
1
2
3 ..... l l+1 l+2.. (n–l+1) ....
[i, j] [i, i+1] [i, i+2]
2
n
[i,
j 3
n
i+ n–1
[i, i+n – 1]
i]
4
i
l
j
5
6
possible range values of i (see line 5)
i
1
2
Pseudocode
3
4
5
6
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
l
2
3
4
......
4) for l ← 2 to n
5) do for i ← 1 to n − l + 1
j
i+1 i+2 i+3 ... ...
6)
do j ← i + l −1
7) 1
m[i, j] ← inf
[i, j] [i, i+1] [i, i+2]
2
n
i+ n–1
[i, i+n – 1]
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
l
2
3
4
......
4) for l ← 2 to n
5) do for i ← 1 to n − l + 1
j
i+1 i+2 i+3 ... ...
6)
do j ← i + l −1
7) 1
m[i, j] ← inf
[i, j] [i, i+1] [i, i+2]
n
i+ n–1
[i, i+n – 1]
2
[i, i
+
j 3
4
1] ⇒
j 3
l=
5
[i,
4
2
5
6
i+2
]
6
i
1
2
3
4
5
6
i
1
2
3
4
5
6
2
Pseudocode
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
l
2
3
4
......
4) for l ← 2 to n
5) do for i ← 1 to n − l + 1
j
i+1 i+2 i+3 ... ...
6)
do j ← i + l −1
7) 1
m[i, j] ← inf
[i, j] [i, i+1] [i, i+2]
n
i+ n–1
[i, i+n – 1]
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
l
2
3
4
......
4) for l ← 2 to n
5) do for i ← 1 to n − l + 1
j
i+1 i+2 i+3 ... ...
6)
do j ← i + l −1
7) 1
m[i, j] ← inf
[i, j] [i, i+1] [i, i+2]
2
2
j 3
j 3
4
[i,
5
5
6
1
i+ n–1
[i, i+n – 1]
4
i+3
]
6
i
n
2
3
4
5
6
[i,
i+n
-1]
i
1
2
Pseudocode
Matrix-chain-order (p) ; p = <po, p1, …, pn>
1) n ← length[p] − 1
2) for i ← 1 to n
3) do m[i, i] ← 0
4) for l ← 2 to n
; l is length of chain product
5) do for i ← 1 to n − l + 1
6)
do j ← i + l −1
7)
m[i, j] ← inf
8)
for k ← i to j − 1
9)
do q ← m[i, k] + m[k+1, j] + pi-1pk pj
10)
if q < m[i, j]
11)
then m[i, j] ← q
12)
s[i, j] ← k
13)return m and s
m[i, j] =
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
i≤k<j
3
4
5
6
Example: Table m
m[i, j] =
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
i≤k<j
0
1
400
2
800
j 3
p = <po, p1, …, p6>
p = <5, 10, 8, 10, 2, 6, 9>
0
0
160
4
0
120
5
6
i
1
2
3
4
0
108
0
5
6
m[i, i+1]
m[1, 2] = m[1, 1] + m[2, 2] + p 0p1 p2 = 5×10×8 = 400
m[2, 3] = m[2, 2] + m[3, 3] + p 1p2 p3 = 10×8 ×10 = 800
s[i, j] ← k
3
Example: Table S
Example: Table m
m[i, j] =
1
2
0
1
2
3
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
i≤k<j
j 4
5
2
400
0
j 3
800
800
320
4
6
5
i
1
2
3
4
5
0
160
0
286
120
0
288
108
0
4
5
6
6
i
1
2
p = <po, p1, …, p6>
p = <5, 10, 8, 10, 2, 6, 9>
3
m[i, i+2] e.g., m[1, 3]
min of k= 1 ... m[1, 1] + m[2, 3] + p 0p1 p3 = 0 + 800 + 5×10×10 = 1300
k= 2 ... m[1, 2] + m[3, 3] + p0p2 p3 = 400 + 0 + 5×8×10 = 800
Example: Table S
Example: Table m
m[i, j] =
1
2
2
3
j
3
4
5
6
2
3
4
2
400
0
j 3
800
800
320
440
4
5
1
0
1
2
4
i
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
i≤k<j
5
5
0
160
0
286
120
0
288
108
0
4
5
6
6
i
1
2
p = <po, p1, …, p6>
p = <5, 10, 8, 10, 2, 6, 9>
3
m[2, 5] find min when
k= 2 ... m[2, 2] + m[3, 5] + p 1p 2 p 5 = 0 + 286 + 10×8×6 = 766
k= 3 ... m[2, 3] + m[4, 5] + p1p3 p5 = 800 + 120 + 10×10×6 = 1520
k= 4 ... m[2, 4] + m[5, 5] + p1p4 p5 = 320 + 0 + 10×2×6 = 440
4
m[i, j] =
0
,i=j
min {m[i, k] + m[k+1, j] + pi-1 pk pj} , i < j
2
400
0
j 3
800
800
0
4
420
480
618
320
440
608
160
0
286
412
120
0
288
108
0
1
2
3
4
5
6
1
2
1
4
4
2
2
4
4
1
2
i≤k<j
0
1
2
400
0
j 3
800
800
320
440
4
5
1
2
6
i
160
0
286
120
0
288
108
0
4
5
6
6
i
5
p = <po, p1, …, p6>
p = <5, 10, 8, 10, 2, 6, 9>
0
0
1
Example: Table m
3
2
3
j 4
5
How many times m[2, 4] is used?
Three times: for m[2, 5], m[2, 6], m[1,4]
6
i
m
S
3
4
4
3
Constructing an optimal solution
Matrix-chain-multiply (A, S, i, j)
1) if j > i
2) then X ← matrix-chain-multiply (A, S, i, S [i, j])
3)
Y ← matrix-chain-multiply (A, S, S [i, j] + 1, j)
4)
return matrix-multiply (X, Y)
5) else return Ai
((A1, (A 2, A3), ((A 4, A 5), A6))
p = < 5, 10, 8, 10, 2, 6, 9 >
A1
10•8
A2
8•10
A3
10•2
A4
2•6
A5
8•10•2 = 160
6•9
A6
2•6•9 = 108
5•10
10•8
8•10
10•2
2•6
6•9
A1
A2
A3
A4
A5
A6
10•8•10 = 800
10•2•6 = 120
(A5, A6 )
X←MCM(1, S[1,6]=4)
Y ←MCM(5, 6)
(A2, (A 3, A4))
A1
5
Verifying
5•10
((A1, (A 2, (A3, A 4))), (A 5, A6 ))
X←MCM(1, S[1,4]=1) Y ←MCM(2, 4)
5
4
((A1, (A 2, (A3, A 4))), (A 5, A6 ))
MCM(1, 6)
(A1, (A 2, (A3, A 4)))
4
4
X←MCM(5, S[5,6]=5) Y←MCM(6, 6)
(A3, A4 )
A5
X←MCM(2, S[2,4]=2) Y ←MCM(3, 4)
10•8•2 = 160
m[2,4] = 160+160
5•10•10 = 500
10•6•9 = 540
A6
5•10•2 =100
A2
5•10•9 = 450
X←MCM(3, S[3,4]=3) Y ←MCM(4, 4)
A3
A4
Total = 800 + 120 + 500 + 540 + 450 > 618
5•2•9 = 90
Total = 160 + 108 + 160 + 100 + 90 = 618 = m[1,6]
5
Elements of dynamic programming
• Optimal substructure
– How do we determine reasonable subprogram space?
Analysis of
E.g., matrix-chain problem: input <A1, .. A n>
(
subspace1: <Ai, .. Aj>
subspace1
is preferred
subspace2: <Ai1, .. Ain> €
• Overlapping subproblems
Matrix-chain-order
– Want total # of distinct sub-problems to be polynomial
E.g., matrix-chain-order has θ(n2) distinct sub-problems
(there are n (case i=j) + Combination(n, 2) (case i<j) sub-problems)
– Take advantage by sharing(reusing) solutions of sub-problems
•
Find optimal solution from recurrences
– Bottom up strategy, e.g., matrix-chain-order
– Topdown strategy, e.g., recursive-matrix-chain
Recursive topdown
Recursive topdown: Analysis
Recursive-matrix-chain (p, i, j) ; p = <po, p1, …, pn>
1)
2)
3)
4)
5)
6)
7)
8)
if i=j
m[i, j] = 0
,i=j
then return 0
min {m[i, k] + m[k+1, j] + pi-1pk pj } , i < j
m[i, j] ← inf
i≤k<j
for k ← i to j − 1
do q ← recursive-matrix-chain (p,i, k) +
recursive-matrix-chain (p, k+1, j) + pi-1pk p j
if q < m[i, j]
then m[i, j] ← q
return m[i, j]
Example: Call recursive-matrix-chain(p, 1, 4)
Recursion can be traced by DFS
k=1
1..1
2..2 3..4
2..4
1..4
1..2
k=2
2..3 4..4 1..1 2..2
Recursive-matrix-chain (p, i, j) ; p = <po, p1, …, pn>
1) if i=j
m[i, j] = 0
,i=j
2) then return 0
min {m[i, k] + m[k+1, j] + pi-1pk pj } , i < j
3) m[i, j] ← inf
i≤k<j
4) for k ← i to j − 1
5)
do q ← recursive-matrix-chain (p,i, k) +
recursive-matrix-chain (p, k+1, j) + pi-1pk pj
6) if q < m[i, j]
7)
then m[i, j] ← q
1.. k size k
call 1.. n
8) return m[i, j]
3..4
k+1 .. n size n-k
size n
k=3
1..3
4..4
3..3 4..4 1..1 2..3
n−1
1..2 3..3
T(n) ≥ θ(1) +
∑ [T(k) + T(n − k) + θ (1)]
k=1
3..3 4..4
2..2 3..3
2..2 3..3
1)-2), 6)-8)
1..1 2..2
4)
5)
€
6
n−1
T(n) ≥ θ(1) +
• not always efficient
= θ (1) + 2 ∑ T(i) + θ (n − 1)
i=1
€ T(k) ≥ 2k-1 for all k < n
Assume
n−1
n−1
i=1
i= 0
T(n) ≥ 2 ∑ 2i−1 + θ (n) = ∑ 2i − 1+ θ (n) ≥ 2n − 2 + cn ≥ 2n > 2n−1
€
Topdown recursive
∑ [T(k) + T(n − k) + θ (1)]
k=1
n−1
Therefore, T(n) =
for some c > 0
Ω(2n)
€
– repeatedly resolves each sub-problem each time
it reappears in the recursive tree
– for independent sub-problems
--> Divide & conquer
– when sub-problems overlap with small total
number of different sub-problems
-->Dynamic Programming (maybe)
Memoization
Idea: use topdown recursive alg to maintain table with subproblem solutions
Memoization-matrix-chain (p)
1)
n ← length[p] − 1
2)
for i ← 1 to n
3)
do for j ← i to n
4)
do m[i, j] ← inf
5)
return lookup-chain(p, 1, n)
Lookup-chain (p, i, j)
1) if m[i, j] < inf
2)
then return m[i, j]
3) if i = j
4)
then m[i, j] ← 0
5)
else for k ← i to j-1
Recursive-matrix-chain (p, i, j)
1) if i=j
2) then return 0
3) m[i, j] ← inf
4) for k ← i to j − 1
5)
do q ← recursive-matrix-chain (p,i, k) +
recursive-matrix-chain (p, k+1, j) + pi-1 pk pj
6) if q < m[i, j]
7)
then m[i, j] ← q
8) return m[i, j]
Memoization
Example:
Using memoization will save computation as shown in
enclosed areas
k=1
1..1
2..2 3..4
3..3 4..4
2..4
1..4
1..2
k=2
2..3 4..4 1..1 2..2
2..2 3..3
k=3
3..4
1..3
4..4
3..3 4..4 1..1 2..3
2..2 3..3
1..2 3..3
1..1 2..2
7
Memoization: Analysis
Summary
Idea: there are θ(n2) calls to lookup-chain. Each call takes O(n) time
T(n) = O(n3)
Memoization-matrix-chain (p)
1)
2)
θ(n2)3)
4)
5)
n ← length[p] − 1
for i ← 1 to n
do for j ← i to n
do m[i, j] ← inf
return lookup-chain(p, 1, n)
Lookup-chain (p, i, j)
1) if m[i, j] < inf
2)
then return m[i, j]
3) if i = j
4)
then m[i, j] ← 0
5)
else for k ← i to j-1
Recursive-matrix-chain (p, i, j)
1)
if i=j
2)
then return 0
3)
m[i, j] ← inf
4)
for k ← i to j − 1
5)
do q ← recursive-matrix-chain (p,i, k) +
recursive-matrix-chain (p, k+1, j) + pi-1pk p j
6)
if q < m[i, j]
7)
then m[i, j] ← q
8)
return m[i, j]
O(1)
O(n)
Given an optimation problem
• check optimal subtructure
• find recurive definition of value of an opt. soln.
• compute opt. value and construct opt. solution
• exhaustive enumeration - Ω(4n/n3/2)
• repeatedly solve subprob - topdown recursive Ω (2n), O(2n-1n)
• reuse opt sol of subprob
- memoization - O(n3)
- bottom up DP - O(n3) - better than memoization by a const
factor
8