Skip to main content
Springer Nature Link
Log in

A New Computational Approach to the Levenberg-Marquardt Learning Algorithm

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13588))

Included in the following conference series:

  • 597 Accesses

Abstract

A new parallel computational approach to the Levenberg-Marquardt learning algorithm is presented. The proposed solution is based on the AVX instructions to effectively reduce the high computational load of this algorithm. Detailed parallel neural network computations are explicitly discussed. Additionally obtained acceleration is shown based on a few test problems.

This work has been supported by the Polish National Science Center under Grant 2017/27/B/ST6/02852.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bartczuk, Ł, Przybył, A., Cpałka, K.: A new approach to nonlinear modelling of dynamic systems based on fuzzy rules. Int. J. Appl. Math. Comput. Sci. (AMCS) 26(3), 603–621 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. Bilski, J.: The UD RLS algorithm for training the feedforward neural networks. Int. J. Appl. Math. Comput. Sci. 15(1), 101–109 (2005)

    MATH  Google Scholar 

  3. Bilski, J., Litwiński, S., Smola̧g, J.: Parallel realisation of QR algorithm for neural networks learning. In: Rutkowski, L., Siekmann, J.H., Tadeusiewicz, R., Zadeh, L.A. (eds.) ICAISC 2004. LNCS (LNAI), vol. 3070, pp. 158–165. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24844-6_19

    Chapter  Google Scholar 

  4. Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent RTRN neural network learning. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 11–16. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69731-2_2

    Chapter  Google Scholar 

  5. Bilski, J., Smola̧g, J.: Parallel realisation of the recurrent elman neural network learning. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010. LNCS (LNAI), vol. 6114, pp. 19–25. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13232-2_3

    Chapter  Google Scholar 

  6. Bilski, J., Smoląg, J.: Parallel realisation of the recurrent multi layer perceptron learning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012. LNCS (LNAI), vol. 7267, pp. 12–20. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29347-4_2

    Chapter  Google Scholar 

  7. Bilski, J., Smoląg, J.: Parallel approach to learning of the recurrent Jordan neural network. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7894, pp. 32–40. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38658-9_3

    Chapter  Google Scholar 

  8. Bilski, J.: Parallel structures for feedforward and dynamical neural networks (in Polish). AOW EXIT (2013)

    Google Scholar 

  9. Bilski, J., Smoląg, J., Galushkin, A.I.: The parallel approach to the conjugate gradient learning algorithm for the feedforward neural networks. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2014. LNCS (LNAI), vol. 8467, pp. 12–21. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07173-2_2

    Chapter  Google Scholar 

  10. Bilski, J., Smola̧g, J.: Parallel architectures for learning the RTRN and elman dynamic neural networks. IEEE Trans. Parallel Distrib. Syst. PP(99), (2014). https://doi.org/10.1109/TPDS.2014.2357019

  11. Bilski, J., Kowalczyk, B., Marchlewska A., Żurada J.M.: Local levenberg-marquardt algorithm for learning feedforwad neural networks. J. Artif. Intell. Soft Comput. Res. 10(4), 299–316 (2020). https://doi.org/10.2478/jaiscr-2020-0020

  12. Bilski, J., Kowalczyk, B., Marjański, A., Gandor, M., Żurada, J.M.: A novel fast feedforward neural networks training algorithm. J. Artif. Intell. Soft Comput. Res. 11(4), 287–306 (2021). https://doi.org/10.2478/jaiscr-2021-0017

    Article  Google Scholar 

  13. Bilski J., Rutkowski L., Smola̧g J., Tao D., A novel method for speed training acceleration of recurrent neural networks. Inf. Sci. 553, 266–279 (2021). https://doi.org/10.1016/j.ins.2020.10.025

  14. Chu, J.L., Krzyzak, A.: The recognition of partially occluded objects with support vector machines, convolutional neural networks and deep belief networks. J. Artif. Intell. Soft Comput. Res. 4(1), 5–19 (2014)

    Article  Google Scholar 

  15. Cpałka, K., Rutkowski, L.: Flexible takagi-sugeno fuzzy systems. In: Proceedings of the International Joint Conference on Neural Networks, Montreal, pp. 1764–1769 (2005)

    Google Scholar 

  16. Cpałka, K., Łapa, K., Przybył, A., Zalasiński, M.: A new method for designing neuro-fuzzy systems for nonlinear modelling with interpretability aspects. Neurocomputing 135, 203–217 (2014)

    Article  Google Scholar 

  17. Cpalka, K., Rebrova, O., Nowicki, R. et al.: On design of flexible neuro-fuzzy systems for nonlinear modelling. Int. J. Gener. Syst. 42(6), Special Issue: SI, 706–720 (2013)

    Google Scholar 

  18. Fahlman, S.: Faster learning variations on backpropagation: an empirical study. In: Proceedings of Connectionist Models Summer School, Los Atos (1988)

    Google Scholar 

  19. Gabryel, M., Przybyszewski, K.: Methods of searching for similar device fingerprints using changes in unstable parameters. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2020. LNCS (LNAI), vol. 12416, pp. 325–335. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61534-5_29

    Chapter  Google Scholar 

  20. Gabryel, M., Scherer, M.M., Sułkowski, Ł, Damaševičius, R.: Decision making support system for managing advertisers by ad fraud detection. J. Artif. Intell. Soft Comput. Res. 11, 331–339 (2021)

    Article  Google Scholar 

  21. Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neuralnetworks 5(6), 989–993 (1994)

    Google Scholar 

  22. Korytkowski, M., Rutkowski, L., Scherer, R.: From ensemble of fuzzy classifiers to single fuzzy rule base classifier. In: Rutkowski, L., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2008. LNCS (LNAI), vol. 5097, pp. 265–272. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69731-2_26

    Chapter  Google Scholar 

  23. Korytkowski, M., Scherer, R.: Negative correlation learning of neuro-fuzzy system. LNAI 6113, 114–119 (2010)

    Google Scholar 

  24. Łapa, K., Przybył, A., Cpałka, K.: A new approach to designing interpretable models of dynamic systems. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 523–534. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_48

    Chapter  Google Scholar 

  25. Łapa, K., Zalasiński, M., Cpałka, K.: A new method for designing and complexity reduction of neuro-fuzzy systems for nonlinear modelling. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7894, pp. 329–344. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38658-9_30

    Chapter  Google Scholar 

  26. Marqardt, D.: An algorithm for last-sqares estimation of nonlinear paeameters. J. Soc. Ind. Appl. Math. 431–441 (1963)

    Google Scholar 

  27. Niksa-Rynkiewicz, T., Szewczuk-Krypa, N., Witkowska, A., Cpałka, K., Zalasiński, M., Cader, A.: Monitoring regenerative heat exchanger in steam power plant by making use of the recurrent neural network. J. Artif. Intell. Soft Comput. Res. 11(2), 143–155 (2021). https://doi.org/10.2478/jaiscr-2021-0009

    Article  Google Scholar 

  28. Patan, K., Patan, M.: Optimal training strategies for locally recurrent neural networks. J. Artif. Intell. Soft Comput. Res. 1(2), 103–114 (2011)

    MATH  Google Scholar 

  29. Riedmiller, M., Braun, H.: A direct method for faster backpropagation learning: the RPROP Algorithm. In: IEEE International Conference on Neural Networks, San Francisco (1993)

    Google Scholar 

  30. Romaszewski, M., Gawron, P., Opozda, S.: Dimensionality reduction of dynamic msh animations using HO-SVD. J. Artif. Intell. Soft Comput. Res. 3(3), 277–289 (2013)

    Article  Google Scholar 

  31. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. Parallel Distributed Processing, vol. 1, ch. 8, Rumelhart, D.E., McCelland, J. (red.). The MIT Press, Cambridge, Massachusetts (1986)

    Google Scholar 

  32. Rutkowski, L.: Multiple Fourier series procedures for extraction of nonlinear regressions from noisy data. IEEE Trans. Sig. Process. 41(10), 3062–3065 (1993)

    Article  MATH  Google Scholar 

  33. Rutkowski, L.: Identification of MISO nonlinear regressions in the presence of a wide class of disturbances. IEEE Trans. Inf. Theor. 37(1), 214–216 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  34. Rutkowski, L., Jaworski, M., Pietruczuk, L., Duda, P.: Decision trees for mining data streams based on the gaussian approximation. IEEE Trans. Knowl. Data Eng. 26(1), 108–119 (2014)

    Article  MATH  Google Scholar 

  35. Rutkowski, L., Przybył, A., Cpałka, K., Er, M.J.: Online speed profile generation for industrial machine tool based on neuro-fuzzy approach. In: Rutkowski, L., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2010. LNCS (LNAI), vol. 6114, pp. 645–650. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13232-2_79

    Chapter  Google Scholar 

  36. Rutkowski, L., Rafajlowicz, E.: On optimal global rate of convergence of some nonparametric identification procedures. IEEE Trans. Autom. Control 34(10), 1089–1091 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  37. Rutkowski, T., Łapa, K., Jaworski, M., Nielek, R., Rutkowska, D.: On explainable flexible fuzzy recommender and its performance evaluation using the akaike information criterion. In: Gedeon, T., Wong, K.W., Lee, M. (eds.) ICONIP 2019. CCIS, vol. 1142, pp. 717–724. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-36808-1_78

    Chapter  Google Scholar 

  38. Smola̧g, J., Bilski, J.: A systolic array for fast learning of neural networks. In: Proceedings of V Conference Neural Networks and Soft Computing, Zakopane, pp. 754–758 (2000)

    Google Scholar 

  39. Smola̧g, J., Rutkowski, L., Bilski, J.: Systolic array for neural networks. In: Proceedings of IV Conference Neural Networks and Their Applications, Zakopane, pp. 487–497 (1999)

    Google Scholar 

  40. Starczewski, A.: A clustering method based on the modified RS validity index. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 242–250. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_23

    Chapter  Google Scholar 

  41. Starczewski J.T. Advanced Concepts in Fuzzy Logic and Systems with Membership Uncertainty, volume 284 of Studies in Fuzziness and Soft Computing. Springer, Berlin (2013). https://doi.org/10.1007/978-3-642-29520-1

  42. Starczewski, J.T., Goetzen, P., Napoli, Ch.: Triangular fuzzy-rough set based fuzzification of fuzzy rule-based systems. J. Artif. Intell. Soft Comput. Res. 10, 271–285 (2020)

    Article  Google Scholar 

  43. Tadeusiewicz, R.: Neural Networks (in Polish). AOW RM (1993)

    Google Scholar 

  44. Werbos, J.: Backpropagation through time: what it does and how to do it. In: Proceedings of the IEEE, vol. 78, no. 10 (1990)

    Google Scholar 

  45. Wilamowski, B.M., Yo, H.: Neural network learning without backpropagation. IEEE Trans. Neural Network. 21(11), 1793–1803 (2010)

    Article  Google Scholar 

  46. Zalasiński, M., Cpałka, K.: New approach for the on-line signature verification based on method of horizontal partitioning. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2013. LNCS (LNAI), vol. 7895, pp. 342–350. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38610-7_32

    Chapter  Google Scholar 

  47. Zalasiński, M., Łapa, K., Cpałka, K.: Prediction of values of the dynamic signature features. Expert Syst. Appl. 104, 86–96 (2018)

    Article  Google Scholar 

  48. El Zini J., Rizk Y., Awad M.: An optimized parallel implementation of non-iteratively trained recurrent neural networks. Journal of Artif. Intell. Soft Comput. Res. 11(1), 33–50 (2021). https://doi.org/10.2478/jaiscr-2021-0003

Download references

Author information

Authors and Affiliations

  1. Institute of Computational Intelligence, Czȩstochowa University of Technology, Czȩstochowa, Poland

    Jarosław Bilski, Barosz Kowalczyk & Jacek Smola̧g

Authors
  1. Jarosław Bilski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barosz Kowalczyk
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jacek Smola̧g
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jarosław Bilski .

Editor information

Editors and Affiliations

  1. Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Science and Technology, Kraków, Poland

    Ryszard Tadeusiewicz

  6. University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bilski, J., Kowalczyk, B., Smola̧g, J. (2023). A New Computational Approach to the Levenberg-Marquardt Learning Algorithm. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2022. Lecture Notes in Computer Science(), vol 13588. Springer, Cham. https://doi.org/10.1007/978-3-031-23492-7_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23492-7_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23491-0

  • Online ISBN: 978-3-031-23492-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics