Skip to main content
SpringerLink
Log in

SRCNN-PIL: Side Road Convolution Neural Network Based on Pseudoinverse Learning Algorithm

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Deep neural networks offer advanced procedures for many learning tasks because of the ability to extract preferable features at every network layer. The evolved efficiency of extra layers inside a deep network will come at the expense of appended latency and power consumption in feedforward inference. As networks continue to grow and deepen, these outcomes become exceedingly prohibitive for energy-sensitive and real-time software. To overcome this problem, we propose the Side Road Network (SRN), an innovative deep network structure that is enhanced with further side road (SR) classifiers. The SR classifiers are trained by Pseudoinverse learning algorithm (PIL). The PIL algorithm does not integrate crucial user-dependent parameters such as momentum constant or learning rate. The SRN structure allows the prediction of results for a major portion of test samples to exit the network earlier via these SR classifiers since samples can be inferred with certainty. We analyze SRN structure using different models such as VGG, ResNet, WRN, and MobileNet. We evaluate the performance of SRN on three image datasets—CIFAR10, CIFAR100, and Tiny ImageNet—and show that it can improve the model prediction at earlier layers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Buciluǎ C, Caruana R, Niculescu-Mizil A (2006) Model compression. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp 535–541. ACM

  2. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE

  3. Dubey A, Chatterjee M, Ahuja N (2018) Coreset-based neural network compression. In: 15th European conference, Munich, Germany, September 8–14, 2018, proceedings, part vii. Springer, Cham

  4. Golub GH et al (1996) Cf vanloan, matrix computations. The Johns Hopkins

  5. Guo P (2018) A vest of the pseudoinverse learning algorithm. arXiv preprint arXiv:1805.07828

  6. Guo P, Lyu MR (2004) A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data. Neurocomputing 56:101–121

    Article  Google Scholar 

  7. Guo P, Wang K, Zhou X (2018) Pilae: a non-gradient descent learning scheme for deep feedforward neural networks. arXiv preprint arXiv:1811.01545

  8. Guo P, Zhao D, Han M, Feng S (2019) Pseudoinverse learners: new trend and applications to big data. In: INNS big data and deep learning conference. Springer, pp 158–168

  9. Han S, Mao H, Dally WJ (2016) Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. In: ICLR: International conference on learning representations

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  12. Hu Y, Huber A, Anumula J, Liu SC (2018) Overcoming the vanishing gradient problem in plain recurrent networks. arXiv preprint arXiv:1801.06105

  13. Huang G, Sun Y, Liu Z, Sedra D, Weinberger KQ (2016) Deep networks with stochastic depth. In: European conference on computer vision. Springer, pp 646–661

  14. Jin T, Hong S (2019) Split-cnn: splitting window-based operations in convolutional neural networks for memory system optimization. In: Proceedings of the twenty-fourth international conference on architectural support for programming languages and operating systems. ACM, pp 835–847

  15. Juefei-Xu F, Boddeti VN, Savvides M (2018) Perturbative neural networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, pp 3310–3318

  16. Kaya Y, Hong S, Dumitras T (2019) Shallow-deep networks: understanding and mitigating network overthinking. In: International conference on machine learning, pp 3301–3310

  17. Kim YD, Park E, Yoo S, Choi T, Yang L, Shin D (2016) Compression of deep convolutional neural networks for fast and low power mobile applications. In: ICLR: international conference on learning representations

  18. Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. Technical report, Citeseer

  19. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  20. Lavin A, Gray S (2016) Fast algorithms for convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4013–4021

  21. Lee CY, Xie S, Gallagher P, Zhang Z, Tu Z (2015) Deeply-Supervised nets. In: Proceedings of the eighteenth international conference on artificial intelligence and statistics, vol 38. PMLR, pp 562–570

  22. Lin R, Ko CY, He Z, Chen C, Wong N (2020) Hotcake: Higher order tucker articulated kernels for deeper CNN compression

  23. Mathieu M, Henaff M, LeCun Y (2014) Fast training of convolutional networks through FFTs. In: ICLR: international conference on learning representations (ICLR)

  24. Real E, Aggarwal A, Huang Y, Le QV (2019) Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 4780–4789

  25. Ren M, Pokrovsky A, Yang B, Urtasun R (2018) Sbnet: Sparse blocks network for fast inference. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8711–8720

  26. Salehinejad H, Valaee S (2020) Edropout: energy-based dropout and pruning of deep neural networks. arXiv preprint arXiv:2006.04270

  27. Sankaranarayanan S, Jain A, Lim SN (2017) Guided perturbations: self-corrective behavior in convolutional neural networks. In: 2017 IEEE international conference on computer vision (ICCV), pp 3582–3590

  28. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR: international conference on learning representations

  29. Son S, Nah S, Mu Lee K (2018) Clustering convolutional kernels to compress deep neural networks. In: Proceedings of the European conference on computer vision (ECCV), pp 216–232

  30. Swaminathan S, Garg D, Kannan R, Andres F (2020) Sparse low rank factorization for deep neural network compression. Neurocomputing 398:185–196

    Article  Google Scholar 

  31. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9

  32. Teerapittayanon S, McDanel B, Kung HT (2016) Branchynet: Fast inference via early exiting from deep neural networks. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE, pp 2464–2469

  33. Vanhoucke V, Senior A, Mao MZ (2011) Improving the speed of neural networks on CPUS

  34. Wang J, Guo P, Xin X (2018) Review of pseudoinverse learning algorithm for multilayer neural networks and applications. In: International symposium on neural networks. Springer, pp 99–106

  35. Wang K, Guo P (2021) A robust automated machine learning system with pseudoinverse learning. Cogn Comput 1–12

  36. Wang S, Zhou T, Bilmes J (2019) Jumpout: improved dropout for deep neural networks with relus. In: International conference on machine learning, pp 6668–6676

  37. Xu Y, Wang Y, Zhou A, Lin W, Xiong H (2018) Deep neural network compression with single and multiple level quantization. In: Thirty-second AAAI conference on artificial intelligence

  38. Zagoruyko S, Komodakis N (2016) Wide residual networks. In: British machine vision conference

  39. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China

    Mohammed A. B. Mahmoud, Ping Guo, Ahmed Fathy & Kan Li

  2. School of Systems Science, Beijing Normal University, Beijing, China

    Ping Guo

Authors
  1. Mohammed A. B. Mahmoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ahmed Fathy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammed A. B. Mahmoud.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is fully supported by the grants from the National Natural Science Foundation of China (61375045), and the Joint Research Fund in Astronomy (U1531242) under cooperative agreement between the National Natural Science Foundation of China (NSFC) and Chinese Academy of Sciences (CAS)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahmoud, M.A.B., Guo, P., Fathy, A. et al. SRCNN-PIL: Side Road Convolution Neural Network Based on Pseudoinverse Learning Algorithm. Neural Process Lett 53, 4225–4237 (2021). https://doi.org/10.1007/s11063-021-10595-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-021-10595-7

Keywords

Search

Navigation