Skip to main content

Advertisement

SpringerLink
Log in

Multilayered neural architectures evolution for computing sequences of orthogonal polynomials

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

This article presents an evolutionary algorithm to autonomously construct full-connected multilayered feedforward neural architectures. This algorithm employs grammar-guided genetic programming with a context-free grammar that has been specifically designed to satisfy three important restrictions. First, the sentences that belong to the language produced by the grammar only encode all valid neural architectures. Second, full-connected feedforward neural architectures of any size can be generated. Third, smaller-sized neural architectures are favored to avoid overfitting. The proposed evolutionary neural architectures construction system is applied to compute the terms of the two sequences that define the three-term recurrence relation associated with a sequence of orthogonal polynomials. This application imposes an important constraint: training datasets are always very small. Therefore, an adequate sized neural architecture has to be evolved to achieve satisfactory results, which are presented in terms of accuracy and size of the evolved neural architectures, and convergence speed of the evolutionary process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abdul-Rahman, O., Munetomo, M., Akama, K.: An improved binary-real coded genetic algorithm for real parameter optimization. In: Proceedings of 3rd World Congress on Nature and Biologically Inspired Computing (2011)

  2. Alonso, F., Martínez, L., Pérez, A., Santamaría, A., Valente, J.: Modeling medical time series using grammar-guided genetic programming. In: Proceedings of the 8th Industrial Conference on Advances in Data Mining: Medical Applications, E-Commerce, Marketing, and Theoretical Aspects (2008)

  3. Augasta, M., Kathirvalavakumar, T.: Pruning algorithms of neural networks - a comparative study. Cent. Eur. J. Comput. Sci. 3(3), 105–115 (2013)

    Google Scholar 

  4. Barrios Rolanía, D., López Lagomasino, G., Torrano, E.: Location of zeros and asymptotics of polynomials satisfying three-terms recurrence relations with complex coefficients. Russ. Acad. Sci. Sb. Math. 80(2), 309–333 (1995)

    Google Scholar 

  5. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  6. Bernardos, P., Vosniakos, G.: Optimizing feedforward artificial neural network architecture. Eng. Appl. Artif. Intel. 20, 365–382 (2007)

    Article  Google Scholar 

  7. Cecchi, M.M., Zaglia, M.R.: Computing the coefficients of a recurrence formula for numerical integration by moments and modified moments. J. Comput. Appl. Math. 49, 207–216 (1993)

    Article  MathSciNet  Google Scholar 

  8. Chebyshev, P.L.: Sur les fractions continues. J. Math. Pures Appl. 2(3), 289–270 (1858)

    Google Scholar 

  9. Cheney, E.: Introduction to Approximation Theory. AMS Chelsea Pub, Providence (2000)

    Google Scholar 

  10. Chihara, T.: An Introduction to Orthogonal Polynomials. Gordon and Breach, New York (1978)

    MATH  Google Scholar 

  11. Couchet, J., Manrique, D., Porras, J.: Grammar-guided neural architecture evolution. Lect. Notes Comput. Sci 4527, 437–446 (2007)

    Article  Google Scholar 

  12. De Jong, K.: Evolutionary Computation: a Unified Approach. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  13. Dorado, J.: Cooperative strategies to select automatically training patterns and neural architectures with genetic algorithms. University of La Corua. Spain, Ph.D. Thesis (1999)

  14. Fernando, C., Banarse, D., Besse, F., Jaderberg, M., Pfau, D., Reynolds, M., Lactot, M., Wierstra, D.: Convolution by evolution: Differentiable pattern producing networks. In: Proceedings of the Genetic and Evolutionary Computation Conference. GECCO 2016. Denver (2016)

  15. Font, J., Manrique, D., Ramos Criado, P., del Río, D.: Partition based real-valued encoding scheme for evolutionary algorithms. Nat. Comput. 15(3), 477–492 (2016)

    Article  MathSciNet  Google Scholar 

  16. Font, J., Manrique, D., Ríos, J.: Redes de neuronas artificiales y computación evolutiva. Madrid, Spain. Fundación General de la Universidad Politécnica de Madrid (2009)

  17. Garro, B., Sossa, H., Vázquez, R.: Artificial neural network synthesis by means of artificial bee colony (abc) algorithm. In: Proceedings of IEEE Congress on Evolutionary Computation (2011)

  18. Gautschi, W.: Computational aspects of three-term recurrence relations. SIAM Rev. 9(1), 24–82 (1967)

    Article  MathSciNet  Google Scholar 

  19. Gautschi, W.: Orthogonal Polynomials: Computation and Approximation. Numerical Mathematics and Scientific Computation. Oxford University Press, New York (2004)

    MATH  Google Scholar 

  20. Gautschi, W.: Orthogonal Polynomials in Matlab. Exercises and Solutions. Software, Environments and Tools. SIAM, Philadelphia (2016)

    MATH  Google Scholar 

  21. Golub, G.H., Welsch, J.H.: Calculation of gauss quadrature rules. Math. Comp. 23(106), 221–230 (1969)

    Article  MathSciNet  Google Scholar 

  22. Han, H., Qiao, J.: A structure optimisation algorithm for feedforward neural network construction. Neurocomputing 99, 347–357 (2013)

    Article  Google Scholar 

  23. Haykin, S.: Neural Networks and Learning Machines. Pearson, Upper Saddle River (2010)

    Google Scholar 

  24. Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv:1207.0580v1 (2012)

  25. Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991)

    Article  Google Scholar 

  26. Huang, G., Chen, L., Siew, C.: Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neural Netw. 17(4), 879–892 (2006)

    Article  Google Scholar 

  27. Kari, L., Rozenberg, G.: The many facets of natural computing. Commun. ACM 51(10), 72–83 (2008)

    Article  Google Scholar 

  28. Koza, J.: Genetic programming: on the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)

    MATH  Google Scholar 

  29. Loiacono, D., Cardamone, L., Lanzi, P.: Automatic track generation for high-end racing games using evolutionary computation. IEEE Trans. Comput. Intell. AI Games 3(3), 245–259 (2011)

    Article  Google Scholar 

  30. Loshchilov, I., Hutter, F.: CMA-ES For hyperparameter optimization of deep neural networks.arXiv:1604.07269v1 (2016)

  31. Maas, A., Hannun, A., Ng, A.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of ICML Workshop on Deep Learning for Audio, Speech and Language Processing (2013)

  32. Manrique, D., Ríos, J., Rodr(́i)guez-Patón, A.: Artificial Neural Networks in Real Life Applications, Ch. Self-Adapting Neutral Intelligent Systems Using Evolutionary Techniques. Idea Group, Barcelona (2006)

    Google Scholar 

  33. Mathai, A., Haubold, H.: Erdélyi-kober fractional integral operators from a statistical perspective II. Cogent Math. 4(1). https://doi.org/10.1080/23311835.2017.1309769 (2017)

  34. McKay, R., Hoai, N., Whigham, P., Shan, Y., O’Neill, M.: Grammar-based genetic programming: a survey. GGenet. Program Evolvable Mach. 11(3), 365–396 (1988)

    Google Scholar 

  35. Miikkulainen, R.: Encyclopedia of Machine Learning, Chap. Neuroevolution. Springer, New York (2010)

    Google Scholar 

  36. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. arXiv:1703.00548v2 (2017)

  37. Nielsen, M.: Neural Networks and Deep Learning. Determination Press (2015)

  38. Poli, R., Langdon, W., McPhee, N., Koza, J.: A Field Guide to Genetic Programming. www.Lulu.com, UK (2008)

  39. Principe, J.C., Euliano, N.R., Lefebvre, W.C.: Neural and Adaptive Systems, Fundamentals Through Simulations. Wiley, New York (2000)

    Google Scholar 

  40. Pulina, L., Tacchella, A.: NeVer: a tool for artificial neural networks verification. Ann. Math. Artif. Intell. 62(3), 403–425 (2011)

    Article  Google Scholar 

  41. Ramos Criado, P.: New techniques for grammar guided genetic programming: dealing with large derivation trees and high cardinality terminal symbol sets. Ph.D. Thesis, Universidad Politécnica de Madrid. https://doi.org/10.20868/UPM.thesis.48795 (2017)

  42. Samarasinghe, S.: Neural Networks for Applied Sciences and Engineering: From Fundamentals to Complex Pattern Recognition. CRC Press, Boca Raton (2017)

    MATH  Google Scholar 

  43. Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  44. Siddiqi, A., Lucas, S.: A comparison of matrix rewriting versus direct encoding method for evolving neural networks. In: Proceedings of 1998 IEEE International Conference on Evolutionary Computation (1998)

  45. Simon, D.: Evolutionary Optimization Algorithms. Wiley, New York (2013)

    Google Scholar 

  46. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  47. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10, 99–127 (2002)

    Article  Google Scholar 

  48. Stanley, K.O., Miikkulainen, R.: Competitive coevolution through evolutionary complexification. J. Artif. Intell. Res. 21, 63–100 (2004)

    Article  Google Scholar 

  49. Sullivan, T.J.: Orthogonal Polynomials and Applications, pp 133–164. Springer, New York (2015)

    Google Scholar 

  50. Tapas, S., Simanta, H., Jana, N.: Artificial neural network training using differential evolutionary algorithm for classification. Adv. Intel. Soft Comp. 132, 769–778 (2012)

    Article  Google Scholar 

  51. Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning, pp 1058–1066. PMLR, Atlanta (2013)

  52. Wang, J., Wang, H., Chen, Y., Liu, C.: A constructive algorithm for unsupervised learning with incremental neural network. J. Appl. Res. Technol. 13(2), 188–196 (2015)

    Article  Google Scholar 

  53. Whigham, P.: Grammatically-based genetic programming. In: Proceedings of the Workshop on Genetic Programming: From Theory to Real-World Applications (1995)

  54. Yu, X., Gen, M.: Introduction to Evolutionary Algorithms. Springer, London (2010)

    Book  Google Scholar 

  55. Zayernouri, M., Zhang, Z.: New trends in applications of orthogonal polynomials and special functions. Math 3 (special issue) (2016)

  56. Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv:1611.01578v2 (2016)

Download references

Acknowledgements

Thanks to the referees and editors for their valuable comments and suggestions that helped us to improve the quality and contents of the article.

Author information

Authors and Affiliations

  1. ETS Ingeniería Civil, Universidad Politécnica de Madrid, Madrid, Spain

    Dolores Barrios Rolanía

  2. Departamento de Inteligencia Artificial, ETSI Informáticos, Universidad Politécnica de Madrid, Madrid, Spain

    Guillermo Delgado Martínez & Daniel Manrique

Authors
  1. Dolores Barrios Rolanía
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guillermo Delgado Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Manrique
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Manrique.

Additional information

This work was partially supported by research grant MTM2014-54053-P of Ministerio de Economía y Competitividad, Spain.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barrios Rolanía, D., Delgado Martínez, G. & Manrique, D. Multilayered neural architectures evolution for computing sequences of orthogonal polynomials. Ann Math Artif Intell 84, 161–184 (2018). https://doi.org/10.1007/s10472-018-9601-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-018-9601-2

Keywords

Mathematics Subject Classification (2010)

Search

Navigation