Skip to main content
SpringerLink
Log in

Leveraging More of Biology in Evolutionary Reinforcement Learning

  • Conference paper
  • First Online:
Applications of Evolutionary Computation (EvoApplications 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14635))

  • 127 Accesses

Abstract

In this paper, we survey the use of additional biologically inspired mechanisms, principles, and concepts in the area of evolutionary reinforcement learning (ERL). While recent years have witnessed the emergence of a swath of metaphor-laden approaches, many merely echo old algorithms through novel metaphors. Simultaneously, numerous promising ideas from evolutionary biology and related areas, ripe for exploitation within evolutionary machine learning, remain in relative obscurity. To address this gap, we provide a comprehensive analysis of innovative, often unorthodox approaches in ERL that leverage additional bio-inspired elements. Furthermore, we pinpoint research directions in the field with the largest potential to yield impactful outcomes and discuss classes of problems that could benefit the most from such research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    It should be noted that ERL also includes approaches in which different state-action pairs are directly explored, as well as meta-RL methods.

References

  1. Aranha, C., et al.: Metaphor-based metaheuristics, a call for action: the elephant in the room. Swarm Intell. 16(1), 1–6 (2022)

    Article  Google Scholar 

  2. Sörensen, K.: Metaheuristics-the metaphor exposed. Int. Trans. Oper. Res. 22(1), 3–18 (2015)

    Article  MathSciNet  Google Scholar 

  3. Kutschera, U., Niklas, K.J.: The modern theory of biological evolution: an expanded synthesis. Naturwissenschaften 91, 255–276 (2004)

    Article  Google Scholar 

  4. Barton, N.H.: The “new synthesis’’. Proc. Nat. Acad. Sci. 119(30), e2122147119 (2022)

    Article  Google Scholar 

  5. Yuen, S., Ezard, T.H.G., Sobey, A.J.: Epigenetic opportunities for evolutionary computation. R. Soc. Open Sci. 10(5), 221256 (2023)

    Article  Google Scholar 

  6. Grudniewski, P.A., Sobey, A.J.: cMLSGA: a co-evolutionary multi-level selection genetic algorithm for multi-objective optimization. arXiv preprint arXiv:2104.11072 (2021)

  7. Barton, N., Paixão, T.: Can quantitative and population genetics help us understand evolutionary computation? In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 1573–1580 (2013)

    Google Scholar 

  8. Pontius, J.U., et al.: Initial sequence and comparative analysis of the cat genome. Genome Res. 17(11), 1675–1689 (2007)

    Article  Google Scholar 

  9. Vassiliades, V., Mouret, J.-B.: Discovering the elite hypervolume by leveraging interspecies correlation. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 149–156 (2018)

    Google Scholar 

  10. Khadka, S., Tumer, K.: Evolutionary reinforcement learning. arXiv preprint arXiv:1805.07917 (2018)

  11. Vie, A., Kleinnijenhuis, A.M., Farmer, D.J.: Qualities, challenges and future of genetic algorithms: a literature review. arXiv preprint arXiv:2011.05277 (2020)

  12. Dagdia, Z.C., Avdeyev, P., Bayzid, M.S.: Biological computation and computational biology: survey, challenges, and discussion. Artif. Intell. Rev. 54, 4169–4235 (2021)

    Article  Google Scholar 

  13. Miikkulainen, R., Forrest, S.: A biological perspective on evolutionary computation. Nat. Mach. Intell. 3(1), 9–15 (2021)

    Article  Google Scholar 

  14. Silver, D., et al.: Mastering the game of go without human knowledge. nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

  15. Nguyen, H., La, H.: Review of deep reinforcement learning for robot manipulation. In: 2019 Third IEEE International Conference on Robotic Computing (IRC), pp. 590–595. IEEE (2019)

    Google Scholar 

  16. Zhou, S.K., Le, H.N., Luu, K., Nguyen, H.V., Ayache, N.: Deep reinforcement learning in medical imaging: a literature review. Med. Image Anal. 73, 102193 (2021)

    Article  Google Scholar 

  17. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  18. Qian, H., Yang, Yu.: Derivative-free reinforcement learning: a review. Front. Comp. Sci. 15(6), 156336 (2021)

    Article  Google Scholar 

  19. Such, F.P., Madhavan, V., Conti, E., Lehman, J., Stanley, K.O., Clune, J.: Deep neuroevolution: genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning. arXiv preprint arXiv:1712.06567 (2017)

  20. Salimans, T., Ho, J., Chen, X., Sidor, S., Sutskever, I.: Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864 (2017)

  21. Yang, S., Ong, Y.-S., Jin, Y.; Evolutionary Computation in Dynamic and Uncertain Environments, vol. 51. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-49774-5

  22. Sun, H., Zhang, W., Runxiang, Yu., Zhang, Y.: Motion planning for mobile robots-focusing on deep reinforcement learning: a systematic review. IEEE Access 9, 69061–69081 (2021)

    Article  Google Scholar 

  23. Jin, Y., Branke, J.: Evolutionary optimization in uncertain environments - a survey. IEEE Trans. Evol. Comput. 9(3), 303–317 (2005)

    Article  Google Scholar 

  24. Jiang, M., Huang, Z., Qiu, L., Huang, W., Yen, G.G.: Transfer learning-based dynamic multiobjective optimization algorithms. IEEE Trans. Evol. Comput. 22(4), 501–514 (2017)

    Article  Google Scholar 

  25. Stanley, K.O., Lehman, J., Soros, L.: Open-endedness: the last grand challenge you’ve never heard of (2017)

    Google Scholar 

  26. Mora, C., Tittensor, D.P., Adl, S., Simpson, A.G.B., Worm, B.: How many species are there on earth and in the ocean? PLoS Biol. 9(8), e1001127 (2011)

    Article  Google Scholar 

  27. Rasmussen, S., Sibani, P.: Two modes of evolution: optimization and expansion. Artif. Life 25(1), 9–21 (2019)

    Article  Google Scholar 

  28. Packard, N., et al.: An overview of open-ended evolution: editorial introduction to the open-ended evolution ii special issue. Artif. Life 25(2), 93–103 (2019)

    Article  Google Scholar 

  29. Lehman, J., Stanley, K.O.: Novelty search and the problem with objectives. In: Riolo, R., Vladislavleva, E., Moore, J. (eds.) Genetic Programming Theory and Practice IX. Genetic and Evolutionary Computation. Springer, New York (2011). https://doi.org/10.1007/978-1-4614-1770-5_3

  30. Pugh, J.K., Soros, L.B., Stanley, K.O.: Quality diversity: a new frontier for evolutionary computation. Front. Robot. AI 3, 40 (2016)

    Article  Google Scholar 

  31. Pugh, J.K., Soros, L.B., Szerlip, P.A., Stanley, K.O.: Confronting the challenge of quality diversity. In: Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, pp. 967–974 (2015)

    Google Scholar 

  32. Earle, S., Snider, J., Fontaine, M.C., Nikolaidis, S., Togelius, J.: Illuminating diverse neural cellular automata for level generation. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 68–76 (2022)

    Google Scholar 

  33. Chand, S., Howard, D.: Path towards multilevel evolution of robots. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 1381–1382 (2020)

    Google Scholar 

  34. Stanley, K.O., Lehman, J.: Why Greatness Cannot Be Planned. The Myth of the Objective. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15524-1

    Book  Google Scholar 

  35. Riederer, J.M., Tiso, S., van Eldijk, T.J.B., Weissing, F.J.: Capturing the facets of evolvability in a mechanistic framework. Trends Ecol. Evol. 37(5), 430–439 (2022)

    Article  Google Scholar 

  36. Dawkins, R.: The evolution of evolvability. In: Artificial Life, pp. 201–220. Routledge (2019)

    Google Scholar 

  37. Watson, R.A., Szathmáry, E.: How can evolution learn? Trends Ecol. Evol. 31(2), 147–157 (2016)

    Article  Google Scholar 

  38. Lehman, J., Stanley, K.O.: Evolvability is inevitable: increasing evolvability without the pressure to adapt. PLoS ONE 8(4), e62186 (2013)

    Article  Google Scholar 

  39. Mengistu, H., Lehman, J., Clune, J.: Evolvability search: directly selecting for evolvability in order to study and produce it. In: 2016 Proceedings of the Genetic and Evolutionary Computation Conference, pp. 141–148 (2016)

    Google Scholar 

  40. Gajewski, A., Clune, J., Stanley, K.O., Lehman, J.: Evolvability ES: scalable and direct optimization of evolvability. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 107–115 (2019)

    Google Scholar 

  41. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135. PMLR (2017)

    Google Scholar 

  42. Katona, A., Franks, D.W., Walker, J.A.: Quality evolvability ES: evolving individuals with a distribution of well performing and diverse offspring. In: The 2022 Conference on Artificial Life, ALIFE 2022. MIT Press (2021)

    Google Scholar 

  43. Gašperov, B., Đurasević, M.: On evolvability and behavior landscapes in neuroevolutionary divergent search. arXiv preprint arXiv:2306.09849 (2023)

  44. Doncieux, S., Paolo, G., Laflaquière, A., Coninx, A.: Novelty search makes evolvability inevitable. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 85–93 (2020)

    Google Scholar 

  45. Shorten, D., Nitschke, G.: How evolvable is novelty search? In: 2014 IEEE International Conference on Evolvable Systems, pp. 125–132. IEEE (2014)

    Google Scholar 

  46. Medvet, E., Daolio, F., Tagliapietra, D.: Evolvability in grammatical evolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 977–984 (2017)

    Google Scholar 

  47. Liu, D., Virgolin, M., Alderliesten, T., Bosman, P.A.N.: Evolvability degeneration in multi-objective genetic programming for symbolic regression. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 973–981 (2022)

    Google Scholar 

  48. Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Computat. 11(1), 1–18 (2003)

    Article  Google Scholar 

  49. Shala, G., Biedenkapp, A., Awad, N., Adriaensen, S., Lindauer, M., Hutter, F.: Learning step-size adaptation in CMA-ES. In: Bäck, T., et al. (eds.) PPSN 2020, Part I. LNCS, vol. 12269, pp. 691–706. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58112-1_48

    Chapter  Google Scholar 

  50. Krause, O., Arbonès, D.R., Igel, C.: CMA-ES with optimal covariance update and storage complexity. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  51. Heidrich-Meisner, V., Igel, C.: Uncertainty handling CMA-ES for reinforcement learning. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, pp. 1211–1218 (2009)

    Google Scholar 

  52. Branke, J., Mattfeld, D.C.: Anticipation and flexibility in dynamic scheduling. Int. J. Prod. Res. 43(15), 3103–3129 (2005)

    Article  Google Scholar 

  53. Pinto, L., Davidson, J., Sukthankar, R., Gupta, A.: Robust adversarial reinforcement learning. In: International Conference on Machine Learning, pp. 2817–2826. PMLR (2017)

    Google Scholar 

  54. Masel, J., Trotter, M.V.: Robustness and evolvability. Trends Genet. 26(9), 406–414 (2010)

    Article  Google Scholar 

  55. Wagner, A.: Robustness and evolvability: a paradox resolved. Proc. R. Soc. B Biol. Sci. 275(1630), 91–100 (2008)

    Article  Google Scholar 

  56. Spencer, C.C.A., et al.: The influence of recombination on human genetic diversity. PLoS Genet. 2(9), e148 (2006)

    Article  Google Scholar 

  57. Zainuddin, F.A., Samad, Md.F.A., Tunggal, D.: A review of crossover methods and problem representation of genetic algorithm in recent engineering applications. Int. J. Adv. Sci. Technol. 29(6s), 759–769 (2020)

    Google Scholar 

  58. Paixão, T., Barton, N.: A variance decomposition approach to the analysis of genetic algorithms. In: Proceedings of the 15th Annual Conference on Genetic and Evolutionary Computation, pp. 845–852 (2013)

    Google Scholar 

  59. Rochet, S.: Epistasis in genetic algorithms revisited. Inf. Sci. 102(1–4), 133–155 (1997)

    Article  MathSciNet  Google Scholar 

  60. Mitchell, M., Holland, J.H., Forrest, S.: The royal road for genetic algorithms: fitness landscapes and GA performance. Technical report, Los Alamos National Lab., NM (United States) (1991)

    Google Scholar 

  61. Polani, D., Miikkulainen, R.: Fast reinforcement learning through eugenic neuro-evolution, pp. 99–277. The University of Texas at Austin, AI (1999)

    Google Scholar 

  62. Polani, D., Miikkulainen, R.: Eugenic neuro-evolution for reinforcement learning. In: Proceedings of the 2nd Annual Conference on Genetic and Evolutionary Computation, pp. 1041–1046 (2000)

    Google Scholar 

  63. Ventresca, M., Ombuki-Berman, B.: Epistasis in multi-objective evolutionary recurrent neuro-controllers. In: 2007 IEEE Symposium on Artificial Life, pp. 77–84. IEEE (2007)

    Google Scholar 

  64. Flageat, M., Cully, A.: Uncertain quality-diversity: evaluation methodology and new methods for quality-diversity in uncertain domains. IEEE Trans. Evol. Comput. (2023). https://doi.org/10.1109/TEVC.2023.3273560

  65. Huizinga, J., Stanley, K.O., Clune, J.: The emergence of canalization and evolvability in an open-ended, interactive evolutionary system. Artif. Life 24(3), 157–181 (2018)

    Article  Google Scholar 

  66. Katona, A., Lourenço, N., Machado, P., Franks, D.W., Walker, J.A.: Utilizing the untapped potential of indirect encoding for neural networks with meta learning. In: Castillo, P.A., Jiménez Laredo, J.L. (eds.) EvoApplications 2021. LNCS, vol. 12694, pp. 537–551. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72699-7_34

    Chapter  Google Scholar 

  67. Wang, R., et al.: Enhanced poet: open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In: International Conference on Machine Learning, pp. 9940–9951. PMLR (2020)

    Google Scholar 

  68. Karafotias, G., Hoogendoorn, M., Eiben, Á.E.: Parameter control in evolutionary algorithms: trends and challenges. IEEE Trans. Evol. Comput. 19(2), 167–187 (2014)

    Article  Google Scholar 

  69. Rand, W.: Genetic Algorithms in Dynamic and Coevolving Environments. Ph.D. thesis. Citeseer

    Google Scholar 

  70. Bedau, M.A., Packard, N.H.: Evolution of evolvability via adaptation of mutation rates. Biosystems 69(2–3), 143–162 (2003)

    Article  Google Scholar 

  71. Aleti, A.: An adaptive approach to controlling parameters of evolutionary algorithms. Swinburne University of Technology (2012)

    Google Scholar 

  72. Xu, K., Ma, Y., Li, W.: Dynamics-aware novelty search with behavior repulsion. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1112–1120 (2022)

    Google Scholar 

  73. Weber, M., Schübeler, D.: Genomic patterns of DNA methylation: targets and function of an epigenetic mark. Curr. Opin. Cell Biol. 19(3), 273–280 (2007)

    Article  Google Scholar 

  74. Turner, B.M.: Histone acetylation and an epigenetic code. BioEssays 22(9), 836–845 (2000)

    Article  Google Scholar 

  75. Hu, T.: Evolvability and rate of evolution in evolutionary computation. Ph.D. thesis, Memorial University of Newfoundland (2010)

    Google Scholar 

  76. Wang, Y., Liu, H., Sun, Z.: Lamarck rises from his grave: parental environment-induced epigenetic inheritance in model organisms and humans. Biol. Rev. 92(4), 2084–2111 (2017)

    Article  Google Scholar 

  77. Mukhlish, F., Page, J., Bain, M.: Reward-based epigenetic learning algorithm for a decentralised multi-agent system. Int. J. Intell. Unmanned Syst. 8(3), 201–224 (2020)

    Article  Google Scholar 

  78. Mukhlish, F., Page, J., Bain, M.: From reward to histone: combining temporal-difference learning and epigenetic inheritance for swarm’s coevolving decision making. In: 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp. 1–6. IEEE (2020)

    Google Scholar 

  79. Page, J., Armstrong, R., Mukhlish, F.: Simulating search and rescue operations using swarm technology to determine how many searchers are needed to locate missing persons/objects in the shortest time. In: Naweed, A., Bowditch, L., Sprick, C. (eds.) ASC 2019. CCIS, vol. 1067, pp. 106–112. Springer, Singapore (2019). https://doi.org/10.1007/978-981-32-9582-7_8

    Chapter  Google Scholar 

  80. Sousa, J.A.B., Costa, E.: Designing an epigenetic approach in artificial life: the EpiAL model. In: Filipe, J., Fred, A., Sharp, B. (eds.) ICAART 2010. CCIS, vol. 129, pp. 78–90. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19890-8_6

    Chapter  Google Scholar 

  81. Boyko, A., Kovalchuk, I.: Epigenetic control of plant stress response. Environ. Mol. Mutagen. 49(1), 61–72 (2008)

    Article  Google Scholar 

  82. Khetarpal, K., Riemer, M., Rish, I., Precup, D.: Towards continual reinforcement learning: a review and perspectives. J. Artif. Intell. Res. 75, 1401–1476 (2022)

    Article  MathSciNet  Google Scholar 

  83. Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: zeros, signs, and the supermask. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  84. Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., Rastegari, M.: What’s hidden in a randomly weighted neural network? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11893–11902 (2020)

    Google Scholar 

  85. Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)

  86. Kimura, M.: The Neutral Theory of Molecular Evolution. Cambridge University Press, Cambridge (1983)

    Book  Google Scholar 

  87. Galván, E.: Neuroevolution in deep learning: the role of neutrality. arXiv preprint arXiv:2102.08475 (2021)

  88. Dal Piccol Sotto, L.F., Mayer, S., Garcke, J.: The pole balancing problem from the viewpoint of system flexibility. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 427–430 (2022)

    Google Scholar 

  89. Galván-López, E., Poli, R., Kattan, A., O’Neill, M., Brabazon, A.: Neutrality in evolutionary algorithms\(\ldots \) what do we know? Evol. Syst. 2, 145–163 (2011)

    Article  Google Scholar 

  90. Odling-Smee, F.J., Laland, K.N., Feldman, M.W.: Niche Construction: The Neglected Process in Evolution (MPB-37). Princeton University Press (2013)

    Google Scholar 

  91. Flynn, E.G., Laland, K.N., Kendal, R.L., Kendal, J.R.: Target article with commentaries: developmental niche construction. Dev. Sci. 16(2), 296–313 (2013)

    Article  Google Scholar 

  92. Dawkins, R.: The Extended Phenotype: The Long Reach of the Gene. Oxford University Press (2016)

    Google Scholar 

  93. Millhouse, T., Moses, M., Mitchell, M.: Frontiers in evolutionary computation: a workshop report. arXiv preprint arXiv:2110.10320 (2021)

  94. Perolat, J., Leibo, J.Z., Zambaldi, V., Beattie, C., Tuyls, K., Graepel, T.: A multi-agent reinforcement learning model of common-pool resource appropriation. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  95. Baker, B., et al.: Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 (2019)

  96. Hamon, G., Nisioti, E., Moulin-Frier, C.: Eco-evolutionary dynamics of non-episodic neuroevolution in large multi-agent environments. In: Proceedings of the Companion Conference on Genetic and Evolutionary Computation, pp. 143–146 (2023)

    Google Scholar 

  97. Berseth, G., et al.: SMiRL: surprise minimizing reinforcement learning in unstable environments. arXiv preprint arXiv:1912.05510 (2019)

  98. Friston, K.: The free-energy principle: a rough guide to the brain? Trends Cogn. Sci. 13(7), 293–301 (2009)

    Article  Google Scholar 

  99. Lipson, H., et al.: Principles of modularity, regularity, and hierarchy for scalable systems. J. Biol. Phys. Chem. 7(4), 125 (2007)

    Article  Google Scholar 

  100. Mengistu, H., Huizinga, J., Mouret, J.-B., Clune, J.: The evolutionary origins of hierarchy. PLoS Comput. Biol. 12(6), e1004829 (2016)

    Article  Google Scholar 

  101. Clune, J., Mouret, J.-B., Lipson, H.: The evolutionary origins of modularity. Proc. R. Soc. B Biol. Sci. 280(1755), 20122863 (2013)

    Article  Google Scholar 

  102. Hutsebaut-Buysse, M., Mets, K., Latré, S.: Hierarchical reinforcement learning: a survey and open research challenges. Mach. Learn. Knowl. Extr. 4(1), 172–221 (2022)

    Article  Google Scholar 

  103. Abramowitz, S., Nitschke, G.: Scalable evolutionary hierarchical reinforcement learning. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 272–275 (2022)

    Google Scholar 

  104. Hansen, T.F.: The evolution of genetic architecture. Annu. Rev. Ecol. Evol. Syst. 37, 123–157 (2006)

    Article  Google Scholar 

  105. Wright, A.H., Laue, C.L.: Evolving complexity is hard. In: Trujillo, L., Winkler, S.M., Silva, S., Banzhaf, W. (eds.) Genetic Programming Theory and Practice XIX. Genetic and Evolutionary Computation. Springer, Singapore (2023). https://doi.org/10.1007/978-981-19-8460-0_10

  106. Smith, S.D., Pennell, M.W., Dunn, C.W., Edwards, S.V.: Phylogenetics is the new genetics (for most of biodiversity). Trends Ecol. Evol. 35(5), 415–425 (2020)

    Article  Google Scholar 

  107. Shonkwiler, R.W., Herod, J.: Phylogenetics. In: Mathematical Biology. UTM, pp. 497–537. Springer, New York (2009). https://doi.org/10.1007/978-0-387-70984-0_15

    Chapter  Google Scholar 

  108. Cussat-Blanc, S., Harrington, K., Pollack, J.: Gene regulatory network evolution through augmenting topologies. IEEE Trans. Evol. Comput. 19(6), 823–837 (2015)

    Article  Google Scholar 

  109. Dolson, E., Ofria, C.: Ecological theory provides insights about evolutionary computation. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 105–106 (2018)

    Google Scholar 

  110. Moreno, M.A., Dolson, E., Rodriguez-Papa, S.: Toward phylogenetic inference of evolutionary dynamics at scale. In: Artificial Life Conference Proceedings 35, vol. 2023, p. 79 (2023)

    Google Scholar 

  111. Lalejini, A., Moreno, M.A., Hernandez, J.G., Dolson, E.: Phylogeny-informed fitness estimation. arXiv preprint arXiv:2306.03970 (2023)

  112. Salehi, A., Coninx, A., Doncieux, S.: Few-shot quality-diversity optimization. IEEE Robot. Autom. Lett. 7(2), 4424–4431 (2022)

    Article  Google Scholar 

  113. Rainford, P.F., Porter, B.: Using phylogenetic analysis to enhance genetic improvement. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 849–857 (2022)

    Google Scholar 

  114. Knapp, J.S., Peterson, G.L.: Natural evolution speciation for NEAT. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 1487–1493. IEEE (2019)

    Google Scholar 

  115. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)

    Article  Google Scholar 

  116. Dixit, G.: Learning to coordinate in sparse asymmetric multiagent systems (2023)

    Google Scholar 

  117. Hannun, A.: The role of evolution in machine intelligence. arXiv preprint arXiv:2106.11151 (2021)

  118. Turney, P., Whitley, D., Anderson, R.W.: Evolution, learning, and instinct: 100 years of the Baldwin effect. Evol. Comput. 4(3), iv–viii (1996)

    Google Scholar 

  119. Abrantes, J.P., Abrantes, A.J., Oliehoek, F.A.: Mimicking evolution with reinforcement learning. arXiv preprint arXiv:2004.00048 (2020)

  120. Stanton, C., Clune, J.: Curiosity search: producing generalists by encouraging individuals to continually explore and acquire skills throughout their lifetime. PLoS ONE 11(9), e0162235 (2016)

    Article  Google Scholar 

  121. Salimans, T., Chen, R.: Learning Montezuma’s revenge from a single demonstration. arXiv preprint arXiv:1812.03381 (2018)

  122. Schmidgall, S.: Adaptive reinforcement learning through evolving self-modifying neural networks. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion, pp. 89–90 (2020)

    Google Scholar 

  123. Yaman, A., Iacca, G., Mocanu, D.C., Coler, M., Fletcher, G., Pechenizkiy, M.: Evolving plasticity for autonomous learning under changing environmental conditions. Evol. Comput. 29(3), 391–414 (2021)

    Article  Google Scholar 

  124. Davies, A.: On the interaction of function, constraint and complexity in evolutionary systems. Ph.D. thesis, University of Southampton (2014)

    Google Scholar 

  125. Macallum, A.B.: The paleochemistry of the body fluids and tissues. Physiol. Rev. 6(2), 316–357 (1926)

    Article  Google Scholar 

  126. Pfeiffer, J., Ruder, S., Vulić, I., Ponti, E.M.: Modular deep learning. arXiv preprint arXiv:2302.11529 (2023)

  127. Stickland, A.C., Murray, I.: BERT and PALs: projected attention layers for efficient adaptation in multi-task learning. In: International Conference on Machine Learning, pp. 5986–5995. PMLR (2019)

    Google Scholar 

  128. Sunagawa, J., Yamaguchi, R., Nakaoka, S.: Evolving neural networks through bio-inspired parent selection in dynamic environments. Biosystems 218, 104686 (2022)

    Article  Google Scholar 

  129. Tang, Y., Nguyen, D., Ha, D.: Neuroevolution of self-interpretable agents. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 414–424 (2020)

    Google Scholar 

  130. Gaier, A., Ha, D.: Weight agnostic neural networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  131. Freeman, D., Ha, D., Metz, L.: Learning to predict without looking ahead: world models without forward prediction. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  132. Fisher, R.A.: XV.-the correlation between relatives on the supposition of mendelian inheritance. Earth Environ. Sci. Trans. R. Soc. Edinburgh 52(2), 399–433 (1919)

    Google Scholar 

  133. Smith, D., Tokarchuk, L., Wiggins, G.: Exploring conflicting objectives with MADNS: multiple assessment directed novelty search. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 23–24 (2016)

    Google Scholar 

  134. Smith, D., Tokarchuk, L., Wiggins, G.: Harnessing phenotypic diversity towards multiple independent objectives. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 961–968 (2016)

    Google Scholar 

  135. Uiterwaal, S.F., Lagerstrom, I.T., Luhring, T.M., Salsbery, M.E., DeLong, J.P.: Trade-offs between morphology and thermal niches mediate adaptation in response to competing selective pressures. Ecol. Evol. 10(3), 1368–1377 (2020)

    Article  Google Scholar 

  136. Walsh, B.: Crops can be strong and sensitive. Nat. Plants 3(9), 694–695 (2017)

    Article  Google Scholar 

  137. Ofria, C., Adami, C., Collier, T.C.: Selective pressures on genomes in molecular evolution. J. Theoret. Biol. 222(4), 477–483 (2003)

    Article  MathSciNet  Google Scholar 

  138. Back, T.: Selective pressure in evolutionary algorithms: a characterization of selection mechanisms. In: Proceedings of the First IEEE Conference on Evolutionary Computation. IEEE World Congress on Computational Intelligence, pp. 57–62. IEEE (1994)

    Google Scholar 

  139. Tari, S., Basseur, M., Goëffon, A.: An extended neighborhood vision for hill-climbing move strategy design. In: Amodeo, L., Talbi, E.-G., Yalaoui, F. (eds.) Recent Developments in Metaheuristics. ORSIS, vol. 62, pp. 109–124. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-58253-5_7

    Chapter  Google Scholar 

  140. Gottlieb, J., Oudeyer, P.-Y.: Towards a neuroscience of active sampling and curiosity. Nat. Rev. Neurosci. 19(12), 758–770 (2018)

    Article  Google Scholar 

  141. Baldassarre, G.: Intrinsic motivations and open-ended learning. arXiv preprint arXiv:1912.13263 (2019)

  142. Santucci, V.G., Oudeyer, P.-Y., Barto, A., Baldassarre, G.: Intrinsically motivated open-ended learning in autonomous robots. Front. Neurorobot. 3, 115 (2020)

    Article  Google Scholar 

  143. Colas, C., Karch, T., Sigaud, O., Oudeyer, P.-Y.: Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: a short survey. J. Artif. Intell. Res. 74, 1159–1199 (2022)

    Article  MathSciNet  Google Scholar 

  144. Georgeon, O.L., Marshall, J.B., Gay, S.: Interactional motivation in artificial systems: between extrinsic and intrinsic motivation. In: 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL), pp. 1–2. IEEE (2012)

    Google Scholar 

  145. Reinitz, J., Vakulenko, S., Grigoriev, D., Weber, A.: Adaptation, fitness landscape learning and fast evolution. F1000Research 8, 358 (2019)

    Google Scholar 

  146. Kouvaris, K.: How evolution learns to evolve: principles of induction in the evolution of adaptive potential. Ph.D. thesis, University of Southampton (2018)

    Google Scholar 

  147. Stanley, K.O., D’Ambrosio, D.B., Gauci, J.: A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)

    Article  Google Scholar 

  148. Bai, H., Cheng, R., Jin, Y.: Evolutionary reinforcement learning: a survey. Intell. Comput. 2, 0025 (2023)

    Article  Google Scholar 

  149. Gomez, F.J., Togelius, J., Schmidhuber, J.: Measuring and optimizing behavioral complexity for evolutionary reinforcement learning. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds.) ICANN 2009. LNCS, vol. 5769, pp. 765–774. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04277-5_77

    Chapter  Google Scholar 

  150. Draghi, J., Wagner, G.P.: Evolution of evolvability in a developmental model. Evolution 62(2), 301–315 (2008)

    Article  Google Scholar 

  151. Van Valen, L.: Two modes of evolution. Nature 252(5481), 298–300 (1974)

    Article  Google Scholar 

  152. Lehman, J., Stanley, K.O.: Evolving a diversity of virtual creatures through novelty search and local competition. In: Proceedings of the 13th Annual Conference on Genetic and Evolutionary Computation, pp. 211–218 (2011)

    Google Scholar 

  153. Lavin, A., et al.: Simulation intelligence: towards a new generation of scientific methods. arXiv preprint arXiv:2112.03235 (2021)

  154. Banzhaf, W., et al.: Defining and simulating open-ended novelty: requirements, guidelines, and challenges. Theor. Biosci. 135, 131–161 (2016)

    Article  Google Scholar 

  155. Dawkins, R.: The Selfish Gene. Oxford University Press (2016)

    Google Scholar 

  156. Song, X., Gao, W., Yang, Y., Choromanski, K., Pacchiano, A., Tang, Y.: ES-MAML: simple hessian-free meta learning. arXiv preprint arXiv:1910.01215 (2019)

  157. Cesa-Bianchi, N., Lugosi, G.: Prediction, Learning, and Games. Cambridge University Press (2006)

    Google Scholar 

  158. Finn, C., Rajeswaran, A., Kakade, S., Levine, S.: Online meta-learning. In: International Conference on Machine Learning, pp. 1920–1930. PMLR (2019)

    Google Scholar 

  159. Yao, H., Zhou, Y., Mahdavi, M., Li, Z.J., Socher, R., Xiong, C.: Online structured meta-learning. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6779–6790 (2020)

    Google Scholar 

  160. Rajasegaran, J., Finn, C., Levine, S.: Fully online meta-learning without task boundaries. arXiv preprint arXiv:2202.00263 (2022)

  161. Cully, A.: Multi-emitter map-elites: improving quality, diversity and data efficiency with heterogeneous sets of emitters. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 84–92 (2021)

    Google Scholar 

  162. Mercado, R., Munoz-Jimenez, V., Ramos, M., Ramos, F.: Generation of virtual creatures under multidisciplinary biological premises. Artif. Life Robot. 27(3), 495–505 (2022)

    Article  Google Scholar 

  163. Stock, M., Gorochowski, T.: Open-endedness in synthetic biology: a route to continual innovation for biological design. Sci. Adv. 10, eadi3621 (2023)

    Google Scholar 

  164. Borg, J.M., Buskell, A., Kapitany, R., Powers, S.T., Reindl, E., Tennie, C.: Evolved open-endedness in cultural evolution: a new dimension in open-ended evolution research. Arti. Life, 1–22 (2023)

    Google Scholar 

  165. Samvelyan, M., et al.: Minihack the planet: a sandbox for open-ended reinforcement learning research. arXiv preprint arXiv:2109.13202 (2021)

  166. Menashe, J., Stone, P.: Escape room: a configurable testbed for hierarchical reinforcement learning. arXiv preprint arXiv:1812.09521 (2018)

  167. Kaznatcheev, A.: Algorithmic biology of evolution and ecology. Ph.D. thesis, University of Oxford (2020)

    Google Scholar 

  168. Beslon, G., Liard, V., Parsons, D.P., Rouzaud-Cornabas, J.: Of evolution, systems and complexity. In: Crombach, A. (ed.) Evolutionary Systems Biology, pp. 1–18. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-71737-7_1

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia

    Bruno Gašperov, Marko Đurasević & Domagoj Jakobovic

Authors
  1. Bruno Gašperov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marko Đurasević
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Domagoj Jakobovic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno Gašperov .

Editor information

Editors and Affiliations

  1. University of York, York, UK

    Stephen Smith

  2. University of Coimbra, Coimbra, Portugal

    João Correia

  3. University of Málaga, Málaga, Spain

    Christian Cintrano

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gašperov, B., Đurasević, M., Jakobovic, D. (2024). Leveraging More of Biology in Evolutionary Reinforcement Learning. In: Smith, S., Correia, J., Cintrano, C. (eds) Applications of Evolutionary Computation. EvoApplications 2024. Lecture Notes in Computer Science, vol 14635. Springer, Cham. https://doi.org/10.1007/978-3-031-56855-8_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56855-8_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56854-1

  • Online ISBN: 978-3-031-56855-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation