Abstract
The more complex the deep neural networks (DNNs) are, the more diverse the learning tasks they can be applied to. However, for complex DNNs, it is difficult to deploy them on to the edge devices, which have limited computation and storage resources. In this paper, we propose an automatic neurons clustering (ANC) approach for deep architecture compression, it can reduce the computation and storage consumption without degrading the model performance. Specifically, an automatic clustering algorithm is used to discover similar neurons in each layer of the deep architecture, then the similar neurons and the corresponding connections are merged based on the results of automatic clustering. After fine-tuning, a more compact and less storage space occupied neural network is obtained, with no performance degradation compared to the original deep architecture. This compression method is fully applicable to fully connected layer and convolutional layer, both of which are common modules of popular DNNs. The analysis of neuron redundancy in DNNs is performed on a deep belief network (DBN), and it is verified that there is great redundancy among neurons in DNNs. To verify the effectiveness of the proposed ANC, we conducted experiments on DBN and VGGNet-16 using MNIST, CIFAR-10 and CIFAR-100 datasets. The experimental results demonstrate that our method can effectively perform deep architecture compression without losing network performance. After fine-tuning, it can even obtain higher accuracy than the original network. In addition, the superiority of ANC is further demonstrated by comparing it with related network compression methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, H., et al.: Data-free learning of student networks. In: 17 International Conference on Computer Vision, Seoul, pp. 3513–3521. IEEE Computer Society (2019)
Ding, X., Ding, G., Zhou, X., Guo, Y., Han, J., Liu, J.: Global sparse momentum SGD for pruning very deep neural networks. In: 32nd Annual Conference on Neural Information Processing Systems, Vancouver, pp. 6379–6391 (2019)
Ding, X., Zhang, X., Han, J., Ding, G.: Repmlp: re-parameterizing convolutions into fully-connected layers for image recognition. CoRR abs/2105.01883 (2021)
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: Repvgg: making vgg-style convnets great again. CoRR abs/2101.03697 (2021)
of Electronic Engineering, D., Computer Science, U.o.M.: senmentation and clustering. [EB/OL] (2012). https://www.eecs.umich.edu/vision/teaching/EECS442_2012/lectures/seg_cluster.pdf
Fukunaga, K., Hostetler, L.D.: The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Trans. Inf. Theory 21(1), 32–40 (1975)
Goodfellow, I.J., et al.: Generative adversarial networks. CoRR abs/1406.2661 (2014)
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In: 4th International Conference on Learning Representations (ICLR), San Juan (2016)
Hassibi, B., Stork, D.G.: Second order derivatives for network pruning: Optimal brain surgeon. In: 5th Advances in Neural Information Processing Systems, Denver, Colorado, pp. 164–171. Morgan Kaufmann, Denver (1992)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 4th IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, pp. 770–778. IEEE Computer Society (2016)
Hinton, G.E., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. CoRR abs/1503.02531 (2015)
Howard, A., et al.: Searching for mobilenetv3. In: 17th International Conference on Computer Vision, Seoul, pp. 1314–1324. IEEE Computer Society (2019)
Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR abs/1704.04861 (2017)
Huang, Y., et al.: Efficient training of giant neural networks using pipeline parallelism. In: 33rd Annual Conference on Neural Information Processing Systems (NIPS), Vancouver, pp. 103–112 (2019)
Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and<1mb model size. CoRR abs/1602.07360 (2016)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: 26th Advances in Neural Information Processing Systems, Lake Tahoe. pp. 1106–1114 (2012)
Lebedev, V., Lempitsky, V.S.: Fast convnets using group-wise brain damage. In: 4th IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, pp. 2554–2564. IEEE Computer Society (2016)
Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. In: 5th International Conference on Learning Representations (ICLR), Toulon (2017)
Liu, X., Wang, L., Liu, W., Zhong, G.: Incremental layers resection: a novel method to compress neural networks. IEEE Access 7, 172167–172177 (2019)
Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., Zhang, C.: Learning efficient convolutional networks through network slimming. In: 16th International Conference on Computer Vision, Venice, pp. 2755–2763. IEEE Computer Society (2017)
Ma, N., Zhang, X., Zheng, H.-T., Sun, J.: ShuffleNet V2: practical guidelines for efficient CNN architecture design. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 122–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_8
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: hints for thin deep nets. In: 3rd International Conference on Learning Representations (ICLR), San Diego (2015)
Sandler, M., Howard, A.G., Zhu, M., Zhmoginov, A., Chen, L.: Mobilenetv 2: Inverted residuals and linear bottlenecks. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, pp. 4510–4520. IEEE Computer Society (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: 3rd International Conference on Learning Representations (ICLR), San Diego (2015)
Tolstikhin, I.O., et al.: Mlp-mixer: An all-mlp architecture for vision. CoRR abs/2105.01601 (2021)
Vaswani, A., et al.: Attention is all you need. In: 31st Annual Conference on Neural Information Processing Systems, Long Beach, pp. 5998–6008 (2017)
Wu, B., et al.: Visual transformers: token-based image representation and processing for computer vision. CoRR abs/2006.03677 (2020)
Zagoruyko, S., Komodakis, N.: Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In: 5th International Conference on Learning Representations, Toulon. OpenReview.net (2017)
Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: 6th IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, pp. 6848–6856. IEEE Computer Society (2018)
Zhong, G., Liu, W., Yao, H., Li, T., Sun, J., Liu, X.: Merging similar neurons for deep networks compression. Cogn. Comput. 12(3), 577–588 (2020)
Acknowledgement
This work was supported by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, the Joint Fund of the Equipments Pre-Research and Ministry of Education of China under Grant No. 6141A0 20337, the Science and Technology Program of Qingdao under Grant No. 21-1-4-ny-19-nsh, the Natural Science Foundation of Shandong Province under Grant No. ZR2020MF131, and the Open Fund of Engineering Research Center for Medical Data Mining and Application of Fujian Province under Grant No. MDM2018007. Thanks to Zhaoxu Ding for his assistance in writing this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, X., Liu, W., Wang, LN., Zhong, G. (2021). Deep Architecture Compression with Automatic Clustering of Similar Neurons. In: Ma, H., et al. Pattern Recognition and Computer Vision. PRCV 2021. Lecture Notes in Computer Science(), vol 13022. Springer, Cham. https://doi.org/10.1007/978-3-030-88013-2_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-88013-2_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88012-5
Online ISBN: 978-3-030-88013-2
eBook Packages: Computer ScienceComputer Science (R0)