Abstract
Recent large CNNs have delivered impressive performance but their storage requirement and computational cost limit a wide range of their applications in mobile devices and large-scale Internet industry. Works focusing on storage compression have led a great success. Recently how to reduce computational cost draws more attention. In this paper, we propose an algorithm to reduce computational cost, which is often solved by sparsification and matrix decomposition methods. Since the computation is dominated by the convolutional operations, we focus on the compression of convolutional layers. Unlike sparsification and matrix decomposition methods which usually derive from mathematics, we receive inspiration from transfer learning and biological neural networks. We transfer the knowledge in state-of-the-art large networks to compressed small ones, via layer-wise training. We replace the complex convolutional layers in large networks with more efficient modules and keep their outputs in each-layer consistent. Modules in the compressed small networks are more efficient, and their design draws on biological neural networks. For AlexNet model, we achieve 3.62× speedup, with 0.11% top-5 error rate increase. For VGG model, we achieve 5.67× speedup, with 0.43% top-5 error rate increase.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1106–1114 (2012)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint: arXiv:1409.1556
Denton, E., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. In: Advances in Neural Information Processing Systems, pp. 1269–1277 (2014)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015). arXiv preprint: arXiv:1503.02531
Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2016)
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (2015). arXiv preprint: arXiv:1510.00149
Figurnov, M., Ibraimova, A., Vetrov, D.P., Kohli, P.: PerforatedCNNs: acceleration through elimination of redundant convolutions. In: Advances in Neural Information Processing Systems, pp. 947–955 (2016)
Kim, Y.D., Park, E., Yoo, S., Choi, T., Yang, L., Shin, D.: Compression of deep convolutional neural networks for fast and low power mobile applications (2015). arXiv preprint: arXiv:1511.06530
Nicholls, J.G., Martin, A.R., Wallace, B.G., Fuchs, P.A.: From Neuron to Brain. Sinauer Associates, Sunderland (2001)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678 (2014)
Chelazzi, L., Miller, E.K., Duncanf, J.: A neural basis for visual search in inferior temporal lobe. Nature 363(6427), 345–347 (1993)
Roe, A.W., Chelazzi, L., Connor, C.E., Conway, B.R., Fujita, I., et al.: Toward a unified theory of visual area V4. Neuron 74(1), 12–29 (2012)
Acknowledgements
This work is supported by the 973 project 2015CB351803, NSFC No. 61572451 and No. 61390514, Youth Innovation Promotion Association CAS CX2100060016, and Fok Ying Tung Education Foundation WF2100060004.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Zeng, L., Tian, X. (2017). Layer-Wise Training to Create Efficient Convolutional Neural Networks. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10635. Springer, Cham. https://doi.org/10.1007/978-3-319-70096-0_65
Download citation
DOI: https://doi.org/10.1007/978-3-319-70096-0_65
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70095-3
Online ISBN: 978-3-319-70096-0
eBook Packages: Computer ScienceComputer Science (R0)