Kronecker Products, Tensor
Decompositions
and 3D Imaging Applications
Misha E. Kilmer
Tufts University
James Nagy
Emory University
Lisa Perrone
Hawaii Pacific University
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 1/35
Outline
I
Preconditioning for discrete ill-posed problems
I
The matrix approximation problem
The role of tensors
I Theoretical Results
I Numerical results
I Conclusions and future work
I
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 2/35
Problem Description
Model:
Kf = ĝ + e = g
I K
is ill-conditioned, no gap in SV spectrum
I K
is triply Toeplitz or triply T+H
K T K may have similar structure in reconstruction
I
For an n × n × n image, K has n3 columns!
I
Noise, e, is white and unknown.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 3/35
SVD Analysis
Assume we can compute
K = [U1 U2 ]
Σ1 0
0 Σ2
V1T
V2T
,
where Σ1 is k × k and corresponds to components such that
uTi g ≈ uTi ĝ.
Noise contaminated exact solution:
dominant
z
}|
{
−1
T
T
g)
+
(U
f = V1 Σ−1
V
Σ
g)
(U
2 2
1
2
1
| {z }
≈U2T e
T g)
Truncated SVD solution: freg = V1 Σ−1
(U
1
1
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 4/35
Iterative Regularization
We cannot compute the SVD of K ! But K is structured, so
we can compute matrix-vector products using FFT’s quickly
O(n3 log n) flops if image is n × n × n.
This means we should use an iterative regularization
scheme (eg. CGLS, MRNSD). We stop iterating before the
solution converges to the exact solution of the system.
Cost is O(N j) where a matvec costs O(N ) and j is the
number of iterations until semi-convergence.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 5/35
Ideal Preconditioning
Major difficulty is that j can be large! Consider the left
preconditioned system
M −1 Kf = M −1 g.
If we could compute the SVD of K , the ideal preconditioner
would be
T Σ1 0
V1
M = [U1 U2 ]
.
T
0 I
V2
Then, semi-convergence in 1 iteration since:
I
The singular values of M −1 K are 1 and Σ2 .
I M −1 g
looks similar to the TSVD solution so no noise
introduced.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 6/35
Goal
Since we cannot compute the SVD of K , the main goal of
our work is to approximate K by a matrix for which we can
compute (efficiently) and store (implicitly) the SVD, and use
the SVD of the approximation to construct M .
Key : Using the structure of K , the matrix approximation
problem can be reduced to a computationally tractable
problem involving 3-way arrays (tensors).
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 7/35
Kronecker Products
B ⊗ C is the block matrix

b11 C b12 C
 b21 C b22 C

 ..
..
 .
.

· · · b1n C
· · · b2n C 

..  .
..
. 
.
bm1 C bm2 C · · · bmn C
A ⊗ B ⊗ C is the double-block matrix:

a11 B ⊗ C a12 B ⊗ C · · · a1n B ⊗ C
 a21 B ⊗ C a22 B ⊗ C · · · a2n B ⊗ C


..
..
..
..

.
.
.
.
am1 B ⊗ C am2 B ⊗ C · · · amn B ⊗ C



.

Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 8/35
SVDs of Kronecker Products
If A = Ua Σa VaT , B = Ub Σb VbT , C = Uc Σc VcT , then
A ⊗ B ⊗ C = (Ua ⊗ Ub ⊗ Uc )(Σa ⊗ Σb ⊗ Σc )(VaT ⊗ VbT ⊗ VcT ),
which is an SVD (under appropriate ordering).
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 9/35
Matrix Approximation Problem
Find Ai , Bi , Ci with the appropriate structure such that
kK −
s
X
Ai ⊗ Bi ⊗ Ci kF
i=1
is minimized. (reason for s > 1 will be discussed later)
Related 2D work:
I
Kamm & Nagy, ‘00
I
Nagy, & Ng, Perrone, ‘04
Perrone, ‘05
I K. & Nagy, ‘06
I
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 10/35
Banded Toeplitz
Complete information about this banded Toeplitz matrix is
captured in a central vector of length n:


1 2 3 0 0
 6 1 2 3 0 




 7 6 1 2 3 .


 0 7 6 1 2 
0 0 7 6 1
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 11/35
Doubly Toeplitz
This doubly Toeplitz matrix can be represented by its
central column, which we reshape:















a
r
0
p
t
0
0
0
0
b
a
r
q
p
t
0
0
0
0
b
a
0
q
p
0
0
0
x
l
0
a
r
0
p
t
0
h
x
l
b
a
r
q
p
t
0
h
x
0
b
a
0
q
p
0
0
0
x
l
0
a
r
0
0
0
0
h
x
l
b
a
r
0
0
0
0
h
x
0
b
a








,








h b q
P = x a p 
l r t
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 12/35
Triply Toeplitz
Same structure as doubly Toeplitz, except now each Ti is a
doubly Toeplitz matrix.
Similarly, if banded structure, the matrix is completely
represented by its central column , which reshape into a
3rd order tensor P .
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 13/35
Structure
If we assume that the blurring in the 3D image is spatially
invariant and that the image satisfies 0 boundary conditions,
K will be triply Toeplitz with this banded structure.
If we assume reflexive boundary conditions, K will be triply
Toeplitz+Hankel with special banding.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 14/35
Matrix Approximation Problem Revisited
Note that if A, B, C are banded Toeplitz (or T+H), they are
uniquely specified by their respective central columns,
vectors a, b, c.
Also, A ⊗ B ⊗ C is triply Toeplitz (T+H) with special banding
structure.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 15/35
Theorem 1
Let K be triply Toeplitz (banded) and P be the 3D tensor
defining the central column of K . Let ai , bi , ci be the central
columns of banded Toeplitz matrices Ai , Bi , Ci , respectively.
Then
kK −
s
X
i=1
Ai ⊗ Bi ⊗ Ci kF = kP̄ −
s
X
c¯i ◦ b¯i ◦ āi kF ,
i=1
where the bar notation implies a (diagonal) weighting on the
faces of P and the vectors ai , bi , ci , and ◦ implies 3way outer
product.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 16/35
Theorem 2
Similar result when K is triply Toeplitz + Hankel (banded),
except the weighting matrix is a very special upper
triangular matrix.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 17/35
Tensor approximation
kK −
s
X
Ai ⊗ Bi ⊗ Ci kF = kP̄ −
i=1
s
X
c¯i ◦ b¯i ◦ āi kF ,
i=1
Computing optimal approximation to K requires computing
the optimal rank-s approximation to the tensor P̄ .
I
For s = 1, unique solution.
I
For s > 1, is there a solution? (uniqueness requires
mild constaints). Can we compute it?
I
A good choice for small s may not be known a priori.
I
There are no “orthogonality” constraints on the
decomposition, and none needed for our application.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 18/35
Tensor Decompositions
Choices:
I PARAFAC model (N-way Toolbox by R. Bro)
I
HOSVD [ de Lathauwer, de Moor and Vandewalle, ‘00]
P̄ =
r3
r2 X
r1 X
X
i=1 j=1 k=1
I
δijk ui ◦ vj ◦ wk ≈
s
X
δ̃im uim ◦ vim ◦ wim .
m=1
Other, possibilities with little degradation? Work with
Perrone, Martin
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 19/35
Approximation
kK −
s
X
i=1
Ai ⊗ Bi ⊗ Ci kF = kP̄ −
s
X
c̄i ◦ b̄i ◦ āi kF
i=1
I
Optimal tensor solution can be calculated if s = 1, used
to construct Ai , Bi , Ci .
I
Settle for HOSVD approximation for s > 1.
To determine s, observe |δ̃i |. Gaps/decay for small
values of s imply K is well approximated by small
number of terms in the Kronecker sum.
I The HOSVD for the tensor should not be confused with
the SVD of K .
I
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 20/35
Preconditioner
I
Case s = 1, compute SVD from Kronecker product
approximation directly and use it to construct M .
I
Case s > 1, one more level of approximation required
(skip details), but M still ends up specified in
Kronecker form.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 21/35
Compute/Storage Summary
For an n × n × n image (an SVD of K would cost O(n9 )
flops, storage)
I
Compute approximation/preconditioner: <= O(n4 )
flops
I
Store a few columns of SVD for 3, n × n matrices.
(minimal)
I
Preconditioner application O(n3 ) for small k
I
Matvec O(n3 log(n))
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 22/35
Example 1
Blurring effects caused by optical limits in 3D microscopy,
on a stack of 20, 128 × 103 slices of a dendrite. Yields triply
Toeplitz blurring operator.
Both examples:
Matlab 7
I RestoreTools [Nagy, Palmer, Perrone, ‘04]
I
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 23/35
True Image Slices (1,7,13,19)
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 24/35
Blurred Image Slices
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 25/35
Largest 50 |δ̃i|
1
10
0
Magnitude of δ values computed by HOSVD
10
−1
10
−2
10
−3
10
−4
10
−5
10
−6
10
0
5
10
15
20
25
30
35
Index of sorted δ values computed by HOSVD
40
45
50
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 26/35
Relative Errors
0.65
MRNSD
PMRNSD, THOSVD(1)
PMRNSD, THOSVD(3)
0.6
Relative error
0.55
0.5
0.45
0.4
0.35
0.3
0
5
10
15
20
25
Iteration
30
35
40
45
50
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 27/35
Reconstructed Images
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 28/35
Example 2
Blurring effects caused by partial volume averaging in spiral
CT
κ(x, y, z) = (κ1 (x, y, z) + κ2 (x, y, z) + κ3 (x, y, z))/3 ,
where
κi (x, y, z) = p
1
(2π)3 σi3
e
−(x2 +y 2 +z 2 )/2σi2
,
and σi were chosen randomly with 1 ≤ σi ≤ 2.
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 29/35
True Image Slices 1,8,15,22
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 30/35
Blurred, Noisy Slices 1,8,15,22
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 31/35
Largest |δ̃i |
1
10
0
Magnitude of δ values computed by HOSVD
10
−1
10
−2
10
−3
10
−4
10
−5
10
−6
10
0
2
4
6
8
10
12
14
16
Index of sorted δ values computed by HOSVD
18
20
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 32/35
Convergence History
0.32
CGLS
PCGLS, THOSVD(1)
PCGLS, THOSVD(4)
0.3
Relative error
0.28
0.26
0.24
0.22
0.2
0.18
0
5
10
15
20
25
Iteration
30
35
40
45
50
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 33/35
Restored Images
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 34/35
Conclusions and Future Work
I
Derived Kronecker product approximations to
operators in image processing applications, used them
to compute preconditioners making 3D deblurring
computationally tractable.
I
Optimal approximation if one term is used.
I
Raises interesting questions about tensor
decompositions, which are “best”.
I
Extension to dense structured matrices (image
processing, PDE’s).
I
Kronecker approximations do provide useful basis for
edge preserving regularization (work with Per Christian
Hansen).
Kronecker Products, Tensor Decompositionsand 3D Imaging Applications – p. 35/35