Quoc-Khanh Mai
ABSTRACT
Convolutional Neural Networks have demonstrated excellent ability in image recognition, yet they frequently have significant computational and memory requirements. It is advantageous to create a tensor factorization framework that can effectively compress networks in order to solve this problem. With its ability to express high-dimensional tensors using a smaller set of core tensors and fewer parameters, the Tensor-Train (TT) structure is a viable choice for compressing neural networks. The choice of appropriate TT-ranks, however, is now without theoretical guarantees. In our study, we introduce a method that employs the widely recognized heuristic network slimming structure through batch normalization techniques. This method serves as metrics to gauge the requisite parameter count, encompassing the intricacies of the training data complexity derived from the original CNNs. Our focus remains on the TT-rank, which denotes the scale parameter choice facilitated by the estimated size. This approach provides a comprehensive strategy to ascertain network compression requirements and resource optimization. Our results reveal that our calculated TT-ranks lead to substantial reduction in computational complexity and memory usage, while maintaining competitive accuracy compared to baseline CNNs.
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition (2009), 248–255.Google Scholar
Cross Ref
- Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. 2016. Ultimate tensorization: compressing convolutional and FC layers alike. arXiv preprint arXiv:1611.03214 (2016).Google Scholar
- Patrick Gelß. 2017. The Tensor-Train Format and Its Applications: Modeling and Analysis of Chemical Reaction Networks, Catalytic Processes, Fluid Flows, and Brownian Dynamics. Ph. D. Dissertation.Google Scholar
- Patrick Gelß. 2023. The tensor-train format and its applications. Accessed: January 2, 2023.Google Scholar
- Song Han, Huizi Mao, and William J. Dally. 2016. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization, and Huffman Coding. arXiv preprint arXiv:1510.00149 (2016). arxiv:1510.00149 [cs.CV]Google Scholar
- Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv preprint arXiv:1503.02531 (2015). arxiv:1503.02531 [stat.ML]Google Scholar
- S. Houben, J. Stallkamp, J. Salmen, and C. Igel. 2013. GTSRB: German Traffic Sign Recognition Benchmark. http://benchmark.ini.rub.de/?section=gtsrb&subsection=news.Google Scholar
- A. Krizhevsky and G. Hinton. 2009. Learning multiple layers of features from tiny images. Technical Report. Tech Report.Google Scholar
- Y. Li, X. Li, Y. Li, X. Sun, Y. Zhang, and Y. Zhang. 2021. HLRTF: Hierarchical Low-Rank Tensor Factorization for Inverse Problems in Multi-Dimensional Imaging. IEEE Transactions on Medical Imaging 40, 4 (2021), 1094–1105. https://doi.org/10.1109/TMI.2020.3030494Google ScholarCross Ref
- Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. 2017. Learning Efficient Convolutional Networks through Network Slimming. In 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy, 2755–2763. https://doi.org/10.1109/ICCV.2017.298Google ScholarCross Ref
- I. V. Oseledets. 2011. Tensor-Train decomposition. SIAM J. Scientific Computing 33, 5 (2011), 2295–2317.Google ScholarDigital Library
- Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556 (2015).Google Scholar
- D. Tian, S. Yamagiwa, and K. Wada. 2022. Heuristic Method for Minimizing Model Size of CNN by Combining Multiple Pruning Techniques. Sensors (Basel) 22, 15 (2022), 5874. https://doi.org/10.3390/s22155874Google ScholarCross Ref
- H. Zhou, J. M. Alvarez, and F. Porikli. 2016. Less is more: Towards compact cnns. In ECCV.Google Scholar
Index Terms
- Tensor Analysis Of Convolutional Neural Network For Reducing Network Parameters
Recommendations
Reliable identification of redundant kernels for convolutional neural network compression
Highlights- A CNN pruning criterion based on the norm of feature map.
- A novel layer-wise Ln-...
AbstractTo compress deep convolutional neural networks (CNNs) with large memory footprint and long inference time, this paper proposes a novel pruning criterion based on layer-wise L n -norms of feature maps to ...
Quantized Tensor Neural Network
Tensor network as an effective computing framework for efficient processing and analysis of high-dimensional data has been successfully applied in many fields. However, the performance of traditional tensor networks still cannot match the strong fitting ...
Fake Faces Identification via Convolutional Neural Network
IH&MMSec '18: Proceedings of the 6th ACM Workshop on Information Hiding and Multimedia SecurityGenerative Adversarial Network (GAN) is a prominent generative model that are widely used in various applications. Recent studies have indicated that it is possible to obtain fake face images with a high visual quality based on this novel model. If ...
Comments