Compression of CNNs
Mooyeol Baek
Xiangyu Zhang, Jianhua Zou, Xiang Ming, Kaiming He, Jian Sun:
Efficient and Accurate Approximations of Nonlinear Convolutional Networks.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, Dongjun Shin:
Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications.
Motivation
• It’s practically important to accelerate the
test-time computation of CNNs.
• CNN filters can be approximately
decomposed into a series of smaller filters
by row-rank approximation.
Approaches
• Zhang et al.
k
m
c
n
k
1
n
c
d’
1
d’
n
d
n
m
• Kim et al.
k
m
m
1
c
1
m
c
k
m
c’
n
c’
1
n
d’
1
d’
n
d
n
Efficient and Accurate
Approximations of
Nonlinear Convolutional
Networks.
Xiangyu Zhang, Jianhua Zou, Xiang Ming,
Kaiming He, Jian Sun
Contribution
• Low-rank approximation minimizing the
reconstruction error of nonlinear responses.
• Asymmetric reconstruction to reduce the
accumulated error of multiple approximated
layers.
• Empirical observation of PCA energy to select
proper rank.
Low-rank Approximation
k
m
c
n
k
c
m
1
n
d’
1
d’
n
d
n
Low-rank Approximation
Relaxation
𝐳 fixed
𝐌, 𝐛 fixed
Asymmetric Reconstruction
• Uses non-approximate responses to reduce
the accumulated error of multiple approximated
layers.
Original
Approximated
Rank Selection
Experiments [1]
• Linear vs. Nonlinear
Experiments [2]
• Symmetric vs. Asymmetric
Experiments [3]
• Rank selection
Compression of
Deep Convolutional
Neural Networks
for Fast and Low Power
Mobile Applications.
Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo,
Taelim Choi, Lu Yang, Dongjun Shin
Contribution
• One-shot whole network compression scheme
which consists of simple three steps:
1.
2.
3.
Rank selection
(Variational Bayesian matrix factorization)
Low-rank tensor decomposition
(Tucker decomposition)
Fine-tuning.
Tensor Decomposition
• Tucker decomposition
Tensor Decomposition
• Zhang et al.
k
m
c
n
k
1
n
c
d’
1
d’
n
d
n
m
• Kim et al.
k
m
m
1
c
1
m
c
k
m
c’
n
c’
1
n
d’
1
d’
n
d
n
Fine-tuning
Experiments [1]
Experiments [2]