Abstract
Training Artificial Neural Networks (ANNs) is a non-trivial task. In the last years, there has been a growing interest in the academic community in understanding how those structures work and what strategies can be adopted to improve the efficiency of the trained models. Thus, the novel approach proposed in this paper is the inclusion of the entropy metric to analyse the training process. Herein, indeed, an investigation on the accuracy computation process in relation to the entropy of the intra-layers’ weights of multilayer perceptron (MLP) networks is proposed. From the analysis conducted on two well-known datasets with several configurations of the ANNs, we discovered that there is a connection between those two metrics (i.e., accuracy and entropy). These promising results can be helpful in defining, in the future, new criteria to evaluate the training process goodness in real-time by optimising it and allow faster detection of its trend.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., Alsaadi, F.E.: A survey of deep neural network architectures and their applications. Neurocomputing 234, 11–26 (2017)
Jumper, J., et al.: Highly accurate protein structure prediction with AlphaFold. Nature 96(7873), 583–589 (2021)
Grassia, M., De Domenico, M., Mangioni, G.: Machine learning dismantling and early-warning signals of disintegration in complex systems. Nature Commun. 12, 5190 (2021)
Young, S.R., Rose, D.C., Karnowski, T.P., Lim, S.-H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, pp. 1–5 (2015)
Assunção, F., Lourenço, N., Machado, P., Ribeiro, B.: Evolving the topology of large scale deep neural networks. In: Castelli, M., Sekanina, L., Zhang, M., Cagnoni, S., García-Sánchez, P. (eds.) EuroGP 2018. LNCS, vol. 10781, pp. 19–34. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77553-1_2
Mattioli, F., Caetano, D., Cardoso, A., Naves, E., Lamounier, E.: An experiment on the use of genetic algorithms for topology selection in deep learning. J. Electr. Comput. Eng. 2019, 1–12 (2019)
Mocanu, D.C., Mocanu, E., Stone, P., Nguyen, P.H., Gibescu, M., Liotta, A.: Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Commun. 9(1), 1–12 (2018)
Cavallaro, L., Bagdasar, O., De Meo, P., Fiumara, G., Liotta, A.: Artificial neural networks training acceleration through network science strategies. Soft Comput. 24(23), 17787–17795 (2020)
Liu, S., Ni’mah, I., Menkovski, V., Mocanu, D.C., Pechenizkiy, M.: Efficient and effective training of sparse recurrent neural networks. Neural Comput. Appl. 33, 1–12 (2021)
Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)
Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: Going beyond Euclidean data. IEEE Signal Process. Mag. 34, 18–42 (2017)
Gómez-Gardeñes, J., Latora, V.: Entropy rate of diffusion processes on complex networks. Phys. Rev. E 78, 065102 (2008)
Bianconi, G., Coolen, A.C.C., Perez Vicente, C.J.: Entropies of complex networks with hierarchically constrained topologies. Phys. Rev. E 78, 016114 (2008)
Bianconi, G.: Entropy of network ensembles. Phys. Rev. E 79, 036114 (2009)
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)
Cover, T., Thomas, J.: Elements of Information Theory, pp 33–36. Wiley, New York (1991)
Goodfellow, I.J., Bengio, Y., Courville, A., Deep Learning. MIT Press, Cambridge (2016). http://www.deeplearningbook.org
Gori, M.: Chapter 5 - deep architectures. In: Gori, M., (ed.) Machine Learning, pp. 236–338. Morgan Kaufmann (2018)
Li, Y., Cai, W., Li, Y., Du, X.: Key node ranking in complex networks: a novel entropy and mutual information-based approach. Entropy 22(1), 52 (2020)
Alves, L.G.A., Mangioni, G., Rodrigues, F.A., Panzarasa, P., Moreno, Y.: Unfolding the complexity of the global value chain: strength and entropy in the single-layer, multiplex, and multi-layer international trade networks. Entropy 20(12), 909 (2018)
Omar, Y.M., Plapper, P.: A survey of information entropy metrics for complex networks. Entropy 22(12), 1417 (2020)
Bhattacharjee, A., et al.: Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proc. Natl. Acad. Sci. 98(24), 13790–13795 (2001)
Cai, D., He, X., Han, J., Huang, T.S.: Graph regularized nonnegative matrix factorization for data representation. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1548–1560 (2010)
Cai, D., He, X., Han, J.: Speed up kernel discriminant analysis. VLDB J. 20(1), 21–33 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cavallaro, L. et al. (2022). Relations Between Entropy and Accuracy Trends in Complex Artificial Neural Networks. In: Benito, R.M., Cherifi, C., Cherifi, H., Moro, E., Rocha, L.M., Sales-Pardo, M. (eds) Complex Networks & Their Applications X. COMPLEX NETWORKS 2021. Studies in Computational Intelligence, vol 1073. Springer, Cham. https://doi.org/10.1007/978-3-030-93413-2_38
Download citation
DOI: https://doi.org/10.1007/978-3-030-93413-2_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-93412-5
Online ISBN: 978-3-030-93413-2
eBook Packages: EngineeringEngineering (R0)