Skip to main content
SpringerLink
Log in

Convergence analysis on the deterministic mini-batch learning algorithm for noise resilient radial basis function networks

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

This paper gives a formal convergence analysis on the mini-batch training algorithm for noise resilient radial basis function (RBF) networks. Unlike the conventional analysis which assumes that the mini-batch process is operated in a stochastic manner, we consider that the mini-batch training process is operated in a deterministic manner. The deterministic process divides the training samples into a number of fixed mini-batches, and the mini-batches are presented in a fixed order. This paper first states the noise resilient objective function for weight noise and weight fault. We then derive the mini-batch training algorithm for this noise resilient objective function. Our main contribution is the convergence analysis on the mini-batch training algorithm. We show that under the deterministic setting, the mini-batch training algorithm converges. The converged weight vector is asymptotically close to the optimal batch mode solution. Also, we derive the sufficient conditions (the learning rate range) for convergence. Our theoretical results can be applied to not only the noise resilient objective function but also a large class of objective functions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. In this case, the numbers \(\kappa _i\)’s of samples in mini-batches should be greater than the number M of RBF nodes.

  2. Note that to the best of knowledge, there is not other mini-batch algorithms for the noise resilient issue.

References

  1. Movassagh AA, Alzubi JA, Gheisari M, Rahimi M, Mohan S, Abbasi AA, Nabipour N (2021) Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model, J Ambient Intell Human Comput 1–9

  2. Soni B, Mathur P, Bora A (2021) In depth analysis, applications and future issues of artificial neural network. In: Enabling AI applications in data science, Springer, pp 149–183

  3. Mhara MAOA (2021) Complexity neural networks for estimating flood process in internet-of-things empowered smart city, Available at SSRN 3775433

  4. Gheisari M, Najafabadi HE, Alzubi JA, Gao J, Wang G, Abbasi AA, Castiglione A (2021) Obpp: An ontology-based framework for privacy-preserving in iot-based smart city. Fut Generation Comput Syst 123:1–13

    Article  Google Scholar 

  5. Chandrasekaran K, Selvaraj J, Amaladoss CR, Veerapan L (2021) Hybrid renewable energy based smart grid system for reactive power management and voltage profile enhancement using artificial neural network, Energy Sources, Part A: Recovery, Utilization, and Environmental Effects, 1–24

  6. Alzubi OA, Nazir J, Hamdoun H (2015) Cyber attack challenges and resilience for smart grids, Euro J Sci Res

  7. Abukharis S, Alzubi JA, Alzubi OA, Alamri S (2014) Packet error rate performance of ieee802. 11g under bluetooth interface. Res J Appl Sci Eng Technol 8(12):1419–1423

    Article  Google Scholar 

  8. Chan Y-C, Wong EW, Leung CS (2021) Evaluating non-hierarchical overflow loss systems using teletraffic theory and neural networks. IEEE Commun Lett 25(5):1486–1490

    Article  Google Scholar 

  9. Alzubi JA (2015) Optimal classifier ensemble design based on cooperative game theory. Res J Appl Sci Eng Technol 11(12):1336–1343

    Article  Google Scholar 

  10. Rzepecki Ł, Jaśkowski P (2021) Application of game theory against nature in supporting bid pricing in construction. Symmetry 13(1):132

    Article  Google Scholar 

  11. Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61:32–48

    Article  MATH  Google Scholar 

  12. Zhang J, Li Y, Xiao W, Zhang Z (2020) Non-iterative and fast deep learning: multilayer extreme learning machines. J Frankl Inst 357(13):8925–8955

    Article  MathSciNet  MATH  Google Scholar 

  13. Zhang J, Xiao W, Li Y, Zhang S (2018) Residual compensation extreme learning machine for regression. Neurocomputing 311:126–136

    Article  Google Scholar 

  14. Leung C-S, Wan WY, Feng R (2016) A regularizer approach for RBF networks under the concurrent weight failure situation. IEEE Trans Neural Netw Learn Syst 28(6):1360–1372

    Article  Google Scholar 

  15. Haykin S (1999) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall, Upper Saddle River, NJ, USA

    MATH  Google Scholar 

  16. Karamichailidou D, Kaloutsa V, Alexandridis A (2021) Wind turbine power curve modeling using radial basis function neural networks and tabu search. Renewable Energy 163:2137–2152

    Article  Google Scholar 

  17. Wu H, Han Y, Geng Z, Fan J, Xu W (2022) Production capacity assessment and carbon reduction of industrial processes based on novel radial basis function integrating multi-dimensional scaling. Sustain Energy Technol Assess 49

  18. Fei J, Wang T (2019) Adaptive fuzzy-neural-network based on rbfnn control for active power filter. Int J Mach Learn Cybernet 10(5):1139–1150

    Article  Google Scholar 

  19. Masters D, Luschi C (2018) Revisiting small batch training for deep neural networks, arXiv preprint arXiv:1804.07612

  20. Jin X, Sun W, Jin Z (2020) A discriminative deep association learning for facial expression recognition. Int J Mach Learn Cybernet 11(4):779–793

    Article  Google Scholar 

  21. Cheng E-J, Chou K-P, Rajora S, Jin B-H, Tanveer M, Lin C-T, Young K-Y, Lin W-C, Prasad M (2019) Deep sparse representation classifier for facial recognition and detection system. Pattern Recognit Lett 125:71–77

    Article  Google Scholar 

  22. Ghosh S, Pal A, Jaiswal S, Santosh K, Das N, Nasipuri M (2019) Segfast-v2: Semantic image segmentation with less parameters in deep learning for autonomous driving. Int J Mach Learn Cybernet 10(11):3145–3154

    Article  Google Scholar 

  23. Fujiyoshi H, Hirakawa T, Yamashita T (2019) Deep learning-based image recognition for autonomous driving. IATSS Res 43(4):244–252

    Article  Google Scholar 

  24. Wang Z, Zhou X, Wang W, Liang C (2020) Emotion recognition using multimodal deep learning in multiple psychophysiological signals and video. Int J Mach Learn Cybernet 11(4):923–934

    Article  Google Scholar 

  25. Si C, Tao Y, Qiu J, Lai S, Zhao J (2021) Deep reinforcement learning based home energy management system with devices operational dependencies. Int J Mach Learn Cybernet 12(6):1687–1703

    Article  Google Scholar 

  26. Zhou Y, Wang J, Liu Y, Yan R, Ma Y (2021) Incorporating deep learning of load predictions to enhance the optimal active energy management of combined cooling, heating and power system, Energy, 121134

  27. Wang X, Zhao Y, Pourpanah F (2020) Recent advances in deep learning

  28. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: a brief review, Comput Intell Neurosci 2018

  29. Torres JM, Comesaña CI, Garcia-Nieto PJ (2019) Machine learning techniques applied to cybersecurity. Int J Mach Learn Cybernet 10(10):2823–2836

    Article  Google Scholar 

  30. Ni D, Xiao Z, Lim MK (2019) A systematic review of the research trends of machine learning in supply chain management, Int J Mach Learn Cybernet, 1–20

  31. Sum J, Leung C-S, Ho K (2012) Convergence analyses on on-line weight noise injection-based training algorithms for MLPs. IEEE Trans Neural Netw Learn Syst 23(11):1827–1840

    Article  Google Scholar 

  32. Zhang H, Wu W, Liu F, Yao M (2009) Boundedness and convergence of online gradient method with penalty for feedforward neural networks. IEEE Trans Neural Netw 20(6):1050–1054

    Article  Google Scholar 

  33. White H (1989) Some asymptotic results for learning in single hidden-layer feedforward network models. J Am Stat Assoc 84(408):1003–1013

    Article  MathSciNet  MATH  Google Scholar 

  34. Liu B, Kaneko T (1969) Error analysis of digital filters realized with floating-point arithmetic. Proc IEEE 57(10):1735–1747

    Article  Google Scholar 

  35. Jeannerod C-P, Rump SM (2013) Improved error bounds for inner products in floating-point arithmetic. SIAM J Matrix Anal Appl 34(2):338–344

    Article  MathSciNet  MATH  Google Scholar 

  36. Diniz PS (2020) The least-mean-square (lms) algorithm, In: Adaptive Filtering, Springer, pp 61–102

  37. Burr JB (1991) Digital neural network implementations. Neural Netw Concepts Appl Implementations 3:237–285

    Google Scholar 

  38. Bolt G, Austin J, Morgan G (1992) Fault tolerant multi-layer perceptron networks, Citeseer

  39. Martolia R, Jain A, Singla L (2015) Analysis & survey on fault tolerance in radial basis function networks, In: International Conference on Computing, Communication & Automation, IEEE, pp 469–473

  40. Murakami M, Honda N (2007) Fault tolerance comparison of ids models with multilayer perceptron and radial basis function networks, In: 2007 International Joint Conference on Neural Networks, IEEE, pp 1079–1084

  41. Liu S-M, Tang L, Huang N-C, Tsai D-Y, Yang M-X, Wu K-C (2020) Fault-tolerance mechanism analysis on nvdla-based design using open neural network compiler and quantization calibrator, In: 2020 International symposium on VLSI design, automation and test (VLSI-DAT), IEEE, pp 1–3

  42. Yamazaki K, Tsutsumi T, Takahashi H, Higami Y, Aikyo T, Takamatsu Y, Yotsuyanagi H, Hashizume M (2009) A novel approach for improving the quality of open fault diagnosis, In: 2009 22nd International Conference on VLSI Design, IEEE, pp 85–90

  43. Leung CS, Wang H-J, Sum J (2010) On the selection of weight decay parameter for faulty networks. IEEE Trans Neural Netw 21(8):1232–1244

    Article  Google Scholar 

  44. Leung C-S, Sum JP-F (2012) RBF networks under the concurrent fault situation. IEEE Trans Neural Netw Learn Syst 23(7):1148–1155

    Article  Google Scholar 

  45. Feng R-B, Han Z-F, Wan WY, Leung C-S (2017) Properties and learning algorithms for faulty rbf networks with coexistence of weight and node failures. Neurocomputing 224:166–176

    Article  Google Scholar 

  46. Konečnỳ J, Liu J, Richtárik P, Takáč M (2015) Mini-batch semi-stochastic gradient descent in the proximal setting. IEEE J Selected Top Signal Process 10(2):242–255

    Article  Google Scholar 

  47. Qian Q, Jin R, Yi J, Zhang L, Zhu S (2015) Efficient distance metric learning by adaptive sampling and mini-batch stochastic gradient descent (sgd). Mach Learn 99(3):353–372

    Article  MathSciNet  MATH  Google Scholar 

  48. Amari S-I (1993) Backpropagation and stochastic gradient descent method. Neurocomputing 5(4–5):185–196

    Article  MATH  Google Scholar 

  49. Li X, Orabona F (2019) On the convergence of stochastic gradient descent with adaptive stepsizes, In: The 22nd international conference on artificial intelligence and statistics, PMLR, pp 983–992

  50. Bottou L et al (1991) Stochastic gradient learning in neural networks. Proc Neuro-Nımes 91(8):12

    Google Scholar 

  51. Cao Y, Gu Q (2019) Generalization bounds of stochastic gradient descent for wide and deep neural networks. Adv Neural Inform Process Syst 32:10836–10846

    Google Scholar 

  52. Cha E, Leung C-S, Wong E (2020) Convergence of mini-batch learning for fault aware rbf networks. In: Yang H, Pasupa K, Leung AC-S, Kwok JT, Chan JH, King I (eds) Neural Information Processing. Springer International Publishing, Cham, pp 545–553

    Chapter  Google Scholar 

  53. Chen S (2006) Local regularization assisted orthogonal least squares regression. Neurocomputing 69(4–6):559–585

    Article  Google Scholar 

  54. Asuncion A, Newman D (2007) Uci machine learning repository

  55. Alcalá-Fdez J, Fernández A, Luengo J, Derrac J, García S, Sánchez L, Herrera F (2011) Keel data-mining software tool: data set repository, integration of algorithms and experimental analysis framework., Journal of Multiple-Valued Logic & Soft Computing 17

  56. Chang C-C, Lin C-J (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):1–27

    Article  Google Scholar 

Download references

Acknowledgements

This work is partially supported by Key Project of Science and Technology Innovation 2030 supported by the Ministry of Science and Technology of China (Grant No. 2018AAA0101301), GRF-RGC General Research Fund CityU 11203820 (9042958) and CityU 11209819 (9042816).

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China

    Hiu Tung Wong & Chi-Sing Leung

  2. Department of Computer Science, City University of Hong Kong, Hong Kong, China

    Sam Kwong

Authors
  1. Hiu Tung Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chi-Sing Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sam Kwong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sam Kwong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wong, H.T., Leung, CS. & Kwong, S. Convergence analysis on the deterministic mini-batch learning algorithm for noise resilient radial basis function networks. Int. J. Mach. Learn. & Cyber. 13, 2677–2690 (2022). https://doi.org/10.1007/s13042-022-01550-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01550-6

Keywords

Search

Navigation