Skip to main content
SpringerLink
Log in

High-parameter-efficiency convolutional neural networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Currently, in order to deploy the convolutional neural networks (CNNs) on the mobile devices and address the over-fitting problem caused by the less abundant datasets, reducing the redundancy of parameters is the main target to construct the mobile CNNs. Based on this target, this paper proposes two novel convolutional kernels, multiple group reused convolutions (MGRCs) and decomposed point-wise convolutions (DPCs), to improve the efficiency of parameters by removing the parameter’s redundancy. The summation of MGRC and DPC is called high-parameter-efficiency convolutions (HPEC) in this paper, and the relevant CNNs can be called HPE-CNNs. Experimental results showed that, compared with the traditional convolutional kernels, HPEC can greatly decrease the model size without affecting the performance. Additionally, since the HPE-CNNs reduce the redundancy of parameters more thoroughly than the other mobile CNNs, they can address the over-fitting problem more effectively on the challenging datasets with less abundant training information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Chollet F (2016) Xception: deep learning with depthwise separable convolutions. CoRR arXiv:abs/1610.02357

  2. Darlow LN, Crowley EJ, Antoniou A, Storkey AJ (2018) CINIC-10 is not ImageNet or CIFAR-10. CoRR arXiv:abs/1810.03505

  3. De Lathauwer L (2008) Decompositions of a higher-order tensor in block terms part II: definitions and uniqueness. SIAM J Matrix Anal Appl 30(3):1033–1066

    Article  MathSciNet  Google Scholar 

  4. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition, 2009 (CVPR 2009). IEEE, pp 248–255

  5. Gastaldi X (2017) Shake-shake regularization. CoRR arXiv:abs/1705.07485

  6. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  7. He Y, Zhang X, Sun J (2017) Channel pruning for accelerating very deep neural networks. In: 2017 IEEE international conference on computer vision (ICCV), pp 1398–1406. https://doi.org/10.1109/ICCV.2017.155

  8. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR arXiv:abs/1704.04861

  9. Hu Y, Sun S, Li J, Wang X, Gu Q (2018) A novel channel pruning method for deep neural network compression. CoRR arXiv:abs/1805.11394

  10. Huang G, Li Y, Pleiss G, Liu Z, Hopcroft JE, Weinberger KQ (2017) Snapshot ensembles: Train 1, get M for free. CoRR arXiv:abs/1704.00109

  11. Huang G, Liu S, van der Maaten L, Weinberger KQ (2017) Condensenet: an efficient densenet using learned group convolutions. CoRR arXiv:abs/1711.09224

  12. Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. In: IEEE conference on computer vision and pattern recognition, pp 2261–2269

  13. Huang G, Sun Y, Liu Z, Sedra D, Weinberger K (2016) Deep networks with stochastic depth. In: European conference on computer vision. Springer, New York, pp 646–661

  14. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Tech. rep, Citeseer

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25(2):2012

    Google Scholar 

  16. Larsson G, Maire M, Shakhnarovich G (2016) Fractalnet: ultra-deep neural networks without residuals. CoRR arXiv:abs/1605.07648

  17. Lu Y, Lu G, Zhang B, Xu Y, Li J (2019) Super sparse convolutional neural networks. In: Proceedings of the thirty-third AAAI conference on artificial intelligence, vol 33, 4440–4447. https://doi.org/10.1609/aaai.v33i01.33014440. https://aaai.org/ojs/index.php/AAAI/article/view/4356

  18. Luo JH, Wu J, Lin W (2017) Thinet: a filter level pruning method for deep neural network compression. In: 2017 IEEE international conference on computer vision (ICCV), pp 5068–5076. https://doi.org/10.1109/ICCV.2017.541

  19. Ma N, Zhang X, Zheng H, Sun J (2018) Shufflenet V2: practical guidelines for efficient CNN architecture design. CoRR arXiv:abs/1807.11164

  20. Patryk C, Ilya L, Frank H (2017) A downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR arXiv:abs/1707.08819

  21. Romero A, Ballas N, Kahou SE, Chassang A, Gatta C, Bengio Y (2014) Fitnets: hints for thin deep nets. CoRR arXiv:abs/1412.6550

  22. Sandler M, Howard AG, Zhu M, Zhmoginov A, Chen L (2018) Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. CoRR arXiv:abs/1801.04381

  23. Singh S, Hoiem D, Forsyth D (2016) Swapout: learning an ensemble of deep architectures. In: Advances in neural information processing systems, pp 28–36

  24. Sun K, Li M, Liu D, Wang J (2018) Igcv3: interleaved low-rank group convolutions for efficient deep neural networks. CoRR arXiv:abs/1806.00178

  25. Sutskever I, Martens J, Dahl G, Hinton G (2013) On the importance of initialization and momentum in deep learning. In: International conference on machine learning, pp 1139–1147

  26. Wang J, Wei Z, Zhang T, Zeng W (2016) Deeply-fused nets. CoRR arXiv:abs/1605.07716

  27. Xie G, Wang J, Zhang T, Lai J, Hong R, Qi G (2018) Interleaved structured sparse convolutional neural networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 8847–8856. https://doi.org/10.1109/CVPR.2018.00922

  28. Xie S, Girshick R, Dollar P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), vol 00, pp 5987–5995. https://doi.org/10.1109/CVPR.2017.634

  29. Zagoruyko S, Komodakis N (2016) Wide residual networks. In: Richard ERH, Wilson C, Smith WAP (eds) Proceedings of the British machine vision conference (BMVC). BMVA Press, pp 87.1–87.12. https://doi.org/10.5244/C.30.87

  30. Zhang T, Qi G, Xiao B, Wang J (2017) Interleaved group convolutions. In: 2017 IEEE international conference on computer vision (ICCV), pp 4383–4392. https://doi.org/10.1109/ICCV.2017.469

  31. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 6848–6856. https://doi.org/10.1109/CVPR.2018.00716

  32. Zhao L, Wang J, Li X, Tu Z, Zeng W (2016) On the connection of deep fusion to ensembling. CoRR arXiv:abs/1611.07718

Download references

Acknowledgements

This work is supported by NSFC Fund (61332011, 61906162), Shenzhen Fundamental Research Fund (JCYJ20170811155442454, JCVJ20180306172023949), Shenzhen Medical Biometrics Perception and Analysis Engineering Laboratory, Postdoctoral Science Foundation (2019TQ0316), Shenzhen Research Institute of Big Data, Shenzhen Institute of Artificial Intelligence and Robotics for Society, and Shenzhen Fundamental Research Fund from Shenzhen Science, Technology and Innovation Commission.

Author information

Authors and Affiliations

  1. Harbin Institute of Technology (Shenzhen), Shenzhen, China

    Yao Lu, Guangming Lu & Yuanrong Xu

  2. The Chinese University of Hong Kong (Shenzhen), Shenzhen, China

    Jinxing Li & David Zhang

  3. University of Science and Technology of China, Hefei, China

    Jinxing Li

Authors
  1. Yao Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinxing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanrong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangming Lu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, Y., Lu, G., Li, J. et al. High-parameter-efficiency convolutional neural networks. Neural Comput & Applic 32, 10633–10644 (2020). https://doi.org/10.1007/s00521-019-04596-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04596-w

Keywords

Search

Navigation