Skip to main content

Advertisement

Springer Nature Link
Log in

Effects of the Flatness Network Parameter Threshold on the Performance of the Rectified Linear Unit Memristor-Like Activation Function in Deep Learning

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

In this contribution, we improve of the performance of the Rectified Linear Unit Memristor Like Activation Function with the implication to help training process of CNN without a lot of epochs by computing the best value of the flatness network parameter (p). In this regards the flatness network parameter threshold (p) has been investigated and a good performance of the activation function has been discovered at p = 4.5. We firstly used the MNIST and the CIFAR-10 datasets to trained and test the Alex-Net architecture model of convolutional neural network (CNN) and we showed better performances of Rectified Linear Unit Memristor-like Activation Function compared to those of the literature. We noticed that the performance of Alex-Net also improved and the better performance was recorded when p = 4.5 with 99.50%, 99.25%, 98.81% for training, validation and testing accuracy respectively when using the MNIST. These results open the outcome to reduce the training time of neural networks when this activation function is used.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Cerquitelli T, Meo M, Curado M, Skorin-Kapov L, Tsiropoulou EE. Machine learning empowered computer networks. Elsevier; 2023. p. 109807.

  2. Chandrasekar A, Rakkiyappan R. Impulsive controller design for exponential synchronization of delayed stochastic memristor-based recurrent neural networks. Neurocomputing. 2016;173:1348–55. https://doi.org/10.1016/j.neucom.2015.08.088.

    Article  Google Scholar 

  3. Chen H-C, Widodo AM, Wisnujati A, Rahaman M, Lin JC-W, Chen L, Weng C-E. AlexNet convolutional neural network for disease detection and classification of tomato leaf. Electronics. 2022;11(6):951. https://doi.org/10.3390/electronics11060951.

    Article  Google Scholar 

  4. Clevert D-A, Unterthiner T, Hochreiter S. Fast and accurate deep network learning by exponential linear units (elus). 2015. arXiv:1511.07289. https://doi.org/10.48550/arXiv.1511.07289.

  5. Duan F, Chapeau-Blondeau F, Abbott D. Optimized injection of noise in activation functions to improve generalization of neural networks. Chaos Solitons Fractals. 2024;178: 114363. https://doi.org/10.1016/j.chaos.2023.114363.

    Article  Google Scholar 

  6. Elakkiya MK. Novel deep learning models with novel integrated activation functions for autism screening: AutiNet and MinAutiNet. Expert Syst Appl. 2024;238: 122102. https://doi.org/10.1016/j.eswa.2023.122102.

    Article  Google Scholar 

  7. Emanuel RH, Docherty PD, Lunt H, Möller K. The effect of activation functions on accuracy, convergence speed, and misclassification confidence in CNN text classification: a comprehensive exploration. J Supercomput. 2024;80(1):292–312. https://doi.org/10.1007/s11227-023-05441-7.

    Article  Google Scholar 

  8. Fernandez A, Mali A. Stable and robust deep learning by hyperbolic tangent exponential linear unit (TeLU). 2024. arXiv:2402.02790.

  9. Gaur L, Bhatia U, Jhanjhi N, Muhammad G, Masud M. Medical image-based detection of COVID-19 using deep convolution neural networks. Multim Syst. 2023;29(3):1729–38. https://doi.org/10.1007/s00530-021-00794-6.

    Article  Google Scholar 

  10. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. 2015.

  11. Karlik B, Olgac AV. Performance analysis of various activation functions in generalized MLP architectures of neural networks. Int J Artif Intell Expert Syst. 2011;1(4):111–22.

    Google Scholar 

  12. Ketwongsa W, Boonlue S, Kokaew U. A new deep learning model for the classification of poisonous and edible mushrooms based on improved AlexNet convolutional neural network. Appl Sci. 2022;12(7):3409. https://doi.org/10.3390/app12073409.

    Article  Google Scholar 

  13. Klambauer G, Unterthiner T, Mayr A, Hochreiter S. Self-normalizing neural networks. Adv Neural Inf Process Syst. 2017;30.

  14. Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images. 2009.

  15. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM. 2017;60(6):84–90. https://doi.org/10.1145/3065386.

    Article  Google Scholar 

  16. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324. https://doi.org/10.1109/5.726791.

    Article  Google Scholar 

  17. Maas AL, Hannun AY, Ng AY. Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of icml. 2013.

  18. Menaka R, Karthik R, Saranya S, Niranjan M, Kabilan S. An improved AlexNet model and cepstral coefficient-based classification of autism using EEG. Clin EEG Neurosci. 2024;55(1):43–51. https://doi.org/10.1177/15500594231178274.

    Article  Google Scholar 

  19. Nair V, Hinton GE. Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10). 2010.

  20. Priya A, Vasudevan V. Brain tumor classification and detection via hybrid alexnet-gru based on deep learning. Biomed Signal Process Control. 2024;89: 105716. https://doi.org/10.1016/j.bspc.2023.105716.

    Article  Google Scholar 

  21. Prodromakis T, Peh BP, Papavassiliou C, Toumazou C. A versatile memristor model with nonlinear dopant kinetics. IEEE Trans Electron Dev. 2011;58(9):3099–105. https://doi.org/10.1109/TED.2011.2158004.

    Article  Google Scholar 

  22. Rajanand A, Singh P. ErfReLU: adaptive activation function for deep neural network. Pattern Anal Appl. 2024;27(2):68.

    Article  Google Scholar 

  23. Rakkiyappan R, Chandrasekar A, Cao J. Passivity and passification of memristor-based recurrent neural networks with additive time-varying delays. IEEE Trans Neural Netw Learn Syst. 2014;26(9):2043–57. https://doi.org/10.1109/TNNLS.2014.2365059.

    Article  MathSciNet  Google Scholar 

  24. Ramachandran P, Zoph B, Le QV. Searching for activation functions. 2017. arXiv:1710.05941.

  25. Shen S-L, Zhang N, Zhou A, Yin Z-Y. Enhancement of neural networks with an alternative activation function tanhLU. Expert Syst Appl. 2022;199: 117181.

    Article  Google Scholar 

  26. Strukov DB, Williams RS. Exponential ionic drift: fast switching and low volatility of thin-film memristors. Appl Phys A. 2009;94(3):515–9. https://doi.org/10.1007/s00339-008-4975-3.

    Article  Google Scholar 

  27. Subramanian B, Jeyaraj R, Ugli RAA, Kim J. APALU: a trainable, adaptive activation function for deep learning networks. 2024. arXiv:2402.08244. https://doi.org/10.48550/arXiv.2402.08244.

  28. Sun J, Cai X, Sun F, Zhang J. Scene image classification method based on Alex-Net model. In: 2016 3rd international conference on informative and cybernetics for computational social systems (ICCSS). 2016.

  29. Xu B, Wang N, Chen T, Li M. Empirical evaluation of rectified activations in convolutional network. 2015. arXiv:1505.00853. https://doi.org/10.48550/arXiv.1505.00853.

  30. Yu Y, Adu K, Tashi N, Anokye P, Wang X, Ayidzoe MA. Rmaf: Relu-memristor-like activation function for deep learning. IEEE Access. 2020;8:72727–41. https://doi.org/10.1109/ACCESS.2020.2987829.

    Article  Google Scholar 

  31. Zha J, Huang H, Liu Y. A novel window function for memristor model with application in programming analog circuits. IEEE Trans Circuits Syst II Express Briefs. 2015;63(5):423–7. https://doi.org/10.1109/TCSII.2015.2505959.

    Article  Google Scholar 

  32. Zhang K, Wu Q, Liu A, Meng X. Can deep learning identify tomato leaf disease? Adv Multim. 2018;2018(1):6710865. https://doi.org/10.1155/2018/6710865.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Unit of Condensed Matter, Electronics and Signal Processing (RU-MACETS), Department of Physics, Faculty of Science, University of Dschang, P. O. Box 67, Dschang, Cameroon

    Marcelle Tchepgoua Mbakop, Justin Roger Mboupda Pone & Romanic Kengne

  2. Research Unit of Automation and Applied Computer (RU-AIA), Electrical Engineering Department of IUT-FV, University of Dschang, P.O. Box 134, Bandjoun, Cameroon

    Marcelle Tchepgoua Mbakop & Justin Roger Mboupda Pone

  3. Research Unit of Fundamental Computer Sciences and Artificial Intelligence (RU-FCSAI), Department of Computer Sciences, Faculty of Science, University of Dschang, P. O. Box 67, Dschang, Cameroon

    Priva Chassem Kamdem

Authors
  1. Marcelle Tchepgoua Mbakop
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Justin Roger Mboupda Pone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Priva Chassem Kamdem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Romanic Kengne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Justin Roger Mboupda Pone.

Ethics declarations

Conflict of interest

Authors, declare they have no conflict of interest regarding the publication of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tchepgoua Mbakop, M., Mboupda Pone, J.R., Chassem Kamdem, P. et al. Effects of the Flatness Network Parameter Threshold on the Performance of the Rectified Linear Unit Memristor-Like Activation Function in Deep Learning. SN COMPUT. SCI. 5, 1156 (2024). https://doi.org/10.1007/s42979-024-03507-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-024-03507-8

Keywords

Search

Navigation