Abstract
An efficient optimization of network weights has been the primary goal of the artificial neural network (ANN) research community since decades. The aim of every optimization problem is to minimize the network cost which is some form of error function between the desired and the actual network outputs, during the training phase. The conventional gradient-based optimization algorithms like backpropagation are likely to get trapped in local minima and are sensitive to choices of initial weights. The evolutionary algorithms have proved their usefulness in introducing randomness into the optimization procedure, since they work on a global search strategy and induce a globally minimum solution for the network weights. In this paper, we particularly focus on ANN trained by Particle Swarm Optimization (ANN-PSO), in which the local-best and global-best particle positions represent possible solutions to the set of network weights. The global-best position of the swarm, which corresponds to the minimum cost function over time, is determined in our work by minimizing a new non-extensive cross-entropy error cost function. The non-extensive cross-entropy is derived from the non-extensive entropy with Gaussian gain that has proven to give minimum values for regular textures containing periodic information represented by uneven probability distributions. The new cross-entropy is defined, and its utility for optimizing the network weights to a globally minimum solution is analyzed in this paper. Extensive experimentation on two different versions: the baseline ANN-PSO and one of its most recent variants IOPSO-BPA, on benchmark datasets from the UCI repository, with comparisons to the state of the art, validates the efficacy of our method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anand A, Suganthi L (2017) Forecasting of electricity demand by hybrid ANN-PSO models. Int J Energy Optim Eng 6(4):66–83
Brescia M, Cavuoti S, D'Angelo G, D'Abrusco R, Deniskina N, Garofalo M, Skordovski B (2008) The VO-Neural project: recent developments and some applications. arXiv preprint. arXiv:0806.1006
Chen D, Cao X, Wen F, Sun J (2013) Blessing of dimensionality: high-dimensional feature and its efficient compression for face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3025–3032
Chen Z, Zhan Z, Shi W, Chen W, Zhang J (2016) When neural network computation meets evolutionary computation: a survey. In: International symposium on neural networks, Springer, Cham, pp 603–612
Chen XL, Fu JP, Yao JL, Gan JF (2018) Prediction of shear strength for squat RC walls using a hybrid ANN–PSO model. Eng Comput 34(2):367–383
Dorigo M, Birattari M, Stutzle T (2006) Ant colony optimization. IEEE Comput Intell Mag 1(4):28–39
Dunne RA, Campbell NA (1997) On the pairing of the softmax activation and cross-entropy penalty functions and the derivation of the softmax activation function. In Proceedings of the 8th Australian conference on the neural networks, Melbourne, vol 181, p 185
Frank A, Asuncion A (2010) UCI machine learning repository. Irvine, CA: University of California, School of Information and Computer Science. https://archive.ics.uci.edu/ml. Accessed 24 May 2019
Gharghan SK, Nordin R, Ismail M, Ali JA (2016) Accurate wireless sensor localization technique based on hybrid PSO-ANN algorithm for indoor and outdoor track cycling. IEEE Sens J 16(2):529–541
Goldberg DE, Holland JH (1988) Genetic algorithms and machine learning. Mach Learn 3(2):95–99
Hajihassani M, Kalatehjari R, Marto A, Mohamad H, Khosrotash M (2020) 3D prediction of tunneling-induced ground movements based on a hybrid ANN and empirical methods. Eng Comput 36(1):251–269
Horst R (1984) On the global minimization of concave functions. Oper Res Spektrum 6(4):195–205
Kennedy J, Eberhart R (1995) Particle swarm optimization (PSO). In Proceedings of IEEE international conference on neural networks, Perth, Australia, pp 1942–1948
Khan JA, Raja MAZ, Rashidi MM, Syam MI, Wazwaz AM (2015) Nature-inspired computing approach for solving non-linear singular Emden-Fowler problem arising in electromagnetic theory. Connect Sci 27(4):377–396
Kiranyaz S, Ince T, Yildirim A, Gabbouj M (2009) Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw 22(10):1448–1462
Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86
Liu W, Wang Z, Liu X, Zeng N, Liu Y, Alsaadi FE (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26
McCaffrey JA (2013) Neural network training using particle swarm optimization. https://visualstudiomagazine.com/articles/2013/12/01/neural-network-training-using-particle-swarm-optimization.aspx. Accessed 24 May 2019
Mendes R, Cortez P, Rocha M, Neves J (2002) Particle swarms for feedforward neural network training. In: Proceedings of the 2002 international joint conference on neural networks. IJCNN'02 (Cat. No. 02CH37290), vol 2, pp 1895–1899
Montana DJ, Davis L (1989) Training feedforward neural networks using genetic algorithms. IJCAI 89:762–767
Nielsen MA (2015) Neural networks and deep learning, vol 25. Determination press, San Francisco
Raja MAZ, Shah FH, Alaidarous ES, Syam MI (2017) Design of bio-inspired heuristic technique integrated with interior-point algorithm to analyze the dynamics of heartbeat model. Appl Soft Comput 52:605–629
Raja MAZ, Shah AA, Mehmood A, Chaudhary NI, Aslam MS (2018) Bio-inspired computational heuristics for parameter estimation of nonlinear Hammerstein controlled autoregressive system. Neural Comput Appl 29(12):1455–1474
Raza S, Mokhlis H, Arof H, Naidu K, Laghari JA, Khairuddin ASM (2016) Minimum-features-based ANN-PSO approach for islanding detection in distribution system. IET Renew Power Gener 10(9):1255–1263
Rumelhart DE, Hinton GE, Williams RJ (1985) Learning internal representations by error propagation (No. ICS-8506). California University, San Diego
Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379–423
Shore J, Johnson R (1981) Properties of cross-entropy minimization. IEEE Trans Inf Theory 27(4):472–482
Socha K, Blum C (2007) An ant colony optimization algorithm for continuous optimization: application to feed-forward neural network training. Neural Comput Appl 16(3):235–247
Susan S, Dwivedi M (2014) Dynamic growth of hidden-layer neurons using the non-extensive entropy. In: 2014 Fourth international conference on communication systems and network technologies IEEE, pp 491–495
Susan S, Hanmandlu M (2013) A non-extensive entropy feature and its application to texture classification. Neurocomputing 120:214–225
Susan S, Hanmandlu M (2015) Unsupervised detection of nonlinearity in motion using weighted average of non-extensive entropies. SIViP 9(3):511–525
Susan S, Hanmandlu M (2018) Color texture recognition by color information fusion using the non-extensive entropy. Multidimension Syst Signal Process 29(4):1269–1284
Susan S, Sharma M (2017) Automatic texture defect detection using Gaussian mixture entropy modeling. Neurocomputing 239:232–237
Susan S, Singh V (2011) On the discriminative power of different feature subsets for handwritten numeral recognition using the box-partitioning method. In: 2011 annual IEEE India conference IEEE, pp 1–5
Susan S, Sharawat P, Singh S, Meena R, Verma A, Kumar M (2015) Fuzzy C-means with non-extensive entropy regularization. In: 2015 IEEE international conference on signal processing, informatics, communication and energy systems (SPICES) IEEE, pp 1–5
Susan S, Ranjan R, Taluja U, Rai S, Agarwal P (2019) Neural net optimization by weight-entropy monitoring. Computational intelligence: theories, applications and future directions-volume II. Springer, Singapore, pp 201–213
Tizhoosh HR (2005) Opposition-based learning: a new scheme for machine intelligence. In: Computational intelligence for modelling, control and automation, 2005 and international conference on intelligent agents, web technologies and internet commerce, international conference on IEEE, vol 1, pp 695–701
Yaghini M, Khoshraftar MM, Fallahi M (2011) HIOPGA: a new hybrid metaheuristic algorithm to train feedforward neural networks for Prediction. In: Proceedings of the international conference on data mining, pp 18–21
Yaghini M, Khoshraftar MM, Fallahi M (2013) A hybrid algorithm for artificial neural network training. Eng Appl Artif Intell 26(1):293–301
Zameer A, Arshad J, Khan A, Raja MAZ (2017) Intelligent and robust prediction of short term wind power using genetic programming based ensemble of neural networks. Energy Convers Manag 134:361–372
Zhang GP (2000) Neural networks for classification: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 30(4):451–462
Zhang L, Suganthan PN (2016) A survey of randomized algorithms for training neural networks. Inf Sci 364:146–155
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
All five authors are associated with the Department of Information Technology, Delhi Technological University, Delhi.
Ethical approval
This article does not contain any studies with animals performed by any of the authors. This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Susan, S., Ranjan, R., Taluja, U. et al. Global-best optimization of ANN trained by PSO using the non-extensive cross-entropy with Gaussian gain. Soft Comput 24, 18219–18231 (2020). https://doi.org/10.1007/s00500-020-05080-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-020-05080-7