Abstract
A new algorithm for feedforward neural networks training is presented. Its core is based on the Givens rotations and QR decomposition (GQR) with an application of a line search method. Similar algorithms based on the QR decomposition utilize a runtime fixed size of the training step. In some situations that might result in inaccurate weight corrections in a given step. The proposed algorithm solves this issue by finding the exact spot of the optimal solution. The performance of the proposed algorithm has been tested on several benchmarks and various networks.
This work has been supported by the Polish National Science Center under Grant 2017/27/B/ST6/02852 and the program of the Polish Minister of Science and Higher Education under the name “Regional Initiative of Excellence” in the years 2019–2022 project number 020/RID/2018/19, the amount of financing PLN 12,000,000.00.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gabryel, M., Damaševičius, R., Przybyszewski, K.: Application of the bag-of-words algorithm in classification the quality of sales leads. In: Rutkowski, L., et al. (eds.) ICAISC 2018. LNCS (LNAI), vol. 10841, pp. 615–622. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91253-0_57
Gabryel, M., Damaševičius, R.: The image classification with different types of image features. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 497–506. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59063-9_44
Starczewski, A., Cader, A.: Determining the EPS parameter of the DBSCAN algorithm. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11509, pp. 420–430. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20915-5_38
Koren, O., Hallin, C.A., Perel, N., Bendet, D.: Decision-making enhancement in a big data environment: application of the k-means algorithm to mixed data. J. Artif. Intell. Soft Comput. Res. 9(4), 293–302 (2019)
Albawi, S., Mohammed, T.A., Al-Zawi, S.: Understanding of a convolutional neural network. In: 2017 International Conference on Engineering and Technology (ICET), pp. 1–6 (2017)
Taqi, A.M., Awad, A., Al-Azzo, F., Milanova, M.: The impact of multi-optimizers and data augmentation on tensorFlow convolutional neural network performance. In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 140–145, April 2018
Rutkowski, T., Romanowski, J., Woldan, P., Staszewski, P., Nielek, R., Rutkowski, L.: A content-based recommendation system using neuro-fuzzy approach. In: 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–8 (2018)
Rutkowski, T., Łapa, K., Nowicki, R., Nielek, R., Grzanek, K.: On explainable recommender systems based on fuzzy rule generation techniques. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2019. LNCS (LNAI), vol. 11508, pp. 358–372. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20912-4_34
Nobukawa, S., Nishimura, H., Yamanishi, T.: Pattern classification by spiking neural networks combining self-organized and reward-related spike-timing-dependent plasticity. J. Artif. Intell. Soft Comput. Res. 9(4), 283–291 (2019)
Wang, X., Guo, Y., Wang, Y., Jinhua, Yu.: Automatic breast tumor detection in ABVS images based on convolutional neural network and superpixel patterns. Neural Comput. Appl. 31(4), 1069–1081 (2019)
Mohamed Shakeel, P., Tobely, T.E.E.E., Al-Feel, H., Manogaran, G., Baskar, S.: Neural network based brain tumor detection using wireless infrared imaging sensor. IEEE Access 7, 5577–5588 (2019)
Cai, X., Qian, Y., Bai, Q., Liu, W.: Exploration on the financing risks of enterprise supply chain using back propagation neural network. J. Comput. Appl. Math. 367, 112457 (2020)
Moghaddam, A.H., Moghaddam, M.H., Esfandyari, M.: Stock market index prediction using artificial neural network. J. Econ. Financ. Adm. Sci. 21(41), 89–93 (2016)
Qi, S., Jin, K., Li, B., Qian, Y.: The exploration of internet finance by using neural network. J. Comput. Appl. Math. 369, 112630 (2020)
de Souza, G.B., da Silva Santos, D.F., Pires, R.G., Marananil, A.N., Papa, J.P.: Deep features extraction for robust fingerprint spoofing attack detection. J. Artif. Intell. Soft Comput. Res. 9(1), 41–49 (2019)
Yurii, K., Liudmila, G.: Application of artificial neural networks in vehicles’ design self-diagnostic systems for safety reasons. In: Transportation Research Procedia, 20:283–287: 12th International Conference “Organization and Traffic Safety Management in large cities” SPbOTSIC-2016, 28–30 September 2016. St, Petersburg, Russia (2017)
Lam, M.W.Y.: One-match-ahead forecasting in two-team sports with stacked Bayesian regressions. J. Artif. Intell. Soft Comput. Res. 8(3), 159–171 (2018)
Mou, Y., Kun, X.: The media inequality: comparing the initial human-human and human-AI social interactions. Comput. Hum. Behav. 72, 432–440 (2017)
Shewalkar, A., Nyavanandi, D., Ludwig, S.A.: Performance evaluation of deep neural networks applied to speech recognition: RNN, LSTM and GRU. J. Artif. Intell. Soft Comput. Res. 9(4), 235–245 (2019)
Costa, M., Oliveira, D., Pinto, S., Tavares, A.: Detecting driver’s fatigue, distraction and activity using a non-intrusive AI-based monitoring system. J. Artif. Intell. Soft Comput. Res. 9(4), 247–266 (2019)
Abbas, M., Javaid, M., Liu, J.-B., Teh, W.C., Cao, J.: Topological properties of four-layered neural networks. J. Artif. Intell. Soft Comput. Res. 9(2), 111–122 (2019)
Kamimura, R.: Supposed maximum mutual information for improving generalization and interpretation of multi-layered neural networks. J. Artif. Intell. Soft Comput. Res. 9(2), 123–147 (2019)
Liu, J.-B., Zhao, J., Wang, S., Javaid, M., Cao, J.: On the topological properties of the certain neural networks. J. Artif. Intell. Soft Comput. Res. 8(4), 257–268 (2018)
Fahlman, S.E.: An empirical study of learning speed in back-propagation networks, Technical report (1988)
Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: IEEE International Conference on Neural Networks, vol. 1, pp. 586–591, March 1993
Sutskever, I., Martens, J., Dahl, G., Hinton, G.: On the importance of initialization and momentum in deep learning. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, ICML 2013, vol. 28, pp. III-1139–III-1147. JMLR.org (2013)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Trans. Neural Netw. 5, 989–993 (1994)
Werbos, J.: Beyond regression: new tools for prediction and analysis in the behavioral sciences. Harvard University (1974)
Bilski, J., Kowalczyk, B., Żurada, J.M.: Application of the givens rotations in the neural network learning algorithm. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9692, pp. 46–56. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39378-0_5
Charalambous, C.: Conjugate gradient algorithm for efficient training of artificial neural networks. IEE Proc. G Circuits Devices Syst. 139(3), 301–310 (1992)
Bilski, J., Kowalczyk, B., Żurada, J.M.: Application of the givens rotations in the neural network learning algorithm. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9692, pp. 46–56. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39378-0_5
Bilski, J., Kowalczyk, B., Żurada, J.M.: Parallel implementation of the givens rotations in the neural network learning algorithm. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2017. LNCS (LNAI), vol. 10245, pp. 14–24. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59063-9_2
Bilski, J., Kowalczyk, B., Grzanek, K.: The parallel modification to the Levenberg-Marquardt algorithm. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds.) ICAISC 2018. LNCS (LNAI), vol. 10841, pp. 15–24. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91253-0_2
Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in optimal control of linear dynamic processes. Int. J. Control 91(7), 1522–1540 (2018)
Rafajłowicz, E., Rafajłowicz, W.: Iterative learning in repetitive optimal control of linear dynamic processes. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2016. LNCS (LNAI), vol. 9692, pp. 705–717. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39378-0_60
Jurewicz, P., Rafajłowicz, W., Reiner, J., Rafajłowicz, E.: Simulations for tuning a laser power control system of the cladding process. In: Saeed, K., Homenda, W. (eds.) CISIM 2016. LNCS, vol. 9842, pp. 218–229. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-45378-1_20
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Bilski, J., Kowalczyk, B., Żurada, J.M. (2020). A New Algorithm with a Line Search for Feedforward Neural Networks Training. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2020. Lecture Notes in Computer Science(), vol 12415. Springer, Cham. https://doi.org/10.1007/978-3-030-61401-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-61401-0_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61400-3
Online ISBN: 978-3-030-61401-0
eBook Packages: Computer ScienceComputer Science (R0)