Jae-Yoon Jung
ABSTRACT
In this paper, we investigate the use of nested evolution in which each step of one evolutionary process involves running a second evolutionary process. We apply this approach to build an evolutionary system for reinforcement learning (RL) problems. Genetic programming based on a descriptive encoding is used to evolve the neural architecture, while an evolution strategy is used to evolve the connection weights. We test this method on a non-Markovian RL problem involving an autonomous foraging agent, finding that the evolved networks significantly outperform a rule-based agent serving as a control. We also demonstrate that nested evolution, partitioning into subpopulations, and crossover operations all act synergistically in improving performance in this context.
- H. G. Beyer. The Theory of Evolution Strategies. Springer, Berlin, 2001. Google Scholar
Digital Library
- H. G. Beyer and H. P. Schwefel. Evolution strategies - a comprehensive introduction. Natural Computing, 1:3--52, 2002. Google ScholarDigital Library
- W. H. Hsu and S. M. Gustafson. Genetic programming and multi-agent layered learning by reinforcements. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 764--771, 2002. Google ScholarDigital Library
- C. Igel. Neuroevolution for reinforcement learning using evolution strategies. Congress on Evolutionary Computation (CEC), volume 4, pages 2588--2595, IEEE Press, 2003.Google ScholarCross Ref
- T. Jansen and I. Wegener. On the local performance of simulated annealing and the (1+1) evolutionary algorithm. In Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pages 469--476, 2006. Google ScholarDigital Library
- J. Y. Jung and J. A. Reggia. Nested evolution of an autonomous agent using descriptive encoding. In Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, pages 285--286, 2008. Google ScholarDigital Library
- J. Y. Jung and J. A. Reggia. Evolutionary design of neural network architectures using a descriptive encoding language. IEEE Transactions on Evolutionary Computation, 10(6):676--688, Dec. 2006. Google ScholarDigital Library
- Y. Kassahun and G. Sommer. Efficient reinforcement learning through evolutionary acquisition of neural topologies. In European Symposium on Artificial Neural Networks (ESANN), pages 259--266, d-side, 2005.Google Scholar
- W. N. Martin, J. Lienig, and J. P. Cohoon. Island (migration) models: evolutionary algorithms based on punctuated equilibria. In T. Bäck et al., editors, Handbook of Evolutionary Computation, pages 101--124, Institute of Physics Publishing and Oxford University Press, 1997.Google Scholar
- J. H. Metzen, M. Edgington, Y. Kassahun, and F. Kirchner. Analysis of an evolutionary reinforcement learning method in a multiagent domain. In Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems, pages 291--298, 2008. Google ScholarDigital Library
- F. Saibene and A. E. Minetti. Biomechanical and physiological aspects of legged locomotion in humans. European Journal of Applied Physiology, 88(4):297--316, 2003.Google ScholarCross Ref
- J. A. Reggia, S. Goodall, Y. Shkuro, and M. Glezer. The callosal dilemma: Explaining diaschisis in the context of hemispheric rivalry via a neural network model. Neurological Research, 23:465--471, 2001.Google ScholarCross Ref
- J. A. Reggia, R. Schulz, G. Wilkinson, and J. Uriagereka. Conditions enabling the evolution of inter-agent signaling in an artificial world. Artificial Life, 7(1):3--32, 2001. Google ScholarDigital Library
- E. Ruppin. Evolutionary autonomous agents: A neuroscience perspective. Nature Reviews Neuroscience, 3(2):132--141, 2002.Google ScholarCross Ref
- N. Siebel and G. Sommer. Evolutionary reinforcement learning of artificial neural networks. International Journal of Hybrid Intelligent Systems, 4(3):171--183, 2007. Google ScholarDigital Library
- K. O. Stanley and R. Miikkulainen. Evolving neural network through augmenting topologies. Evolutionary Computation, 10(2):99--127, 2002. Google ScholarDigital Library
- P. Stone, R. S. Sutton, and G. Kuhlmann. Reinforcement Learning for RoboCup-Soccer Keepaway. Adaptive Behavior, 13(3):165--188, 2005.Google ScholarCross Ref
- R. S. Sutton and A. G. Barto. Reinforcement Learning An Introduction. MIT Press, 1998. Google ScholarDigital Library
- I. Szita and A. Lörincz. Learning tetris using the noisy cross-entropy method. Neural Computation 18(12): 2936--2941, 2006. Google ScholarDigital Library
- M. E. Taylor, S. Whiteson, and P. Stone. Comparing evolutionary and temporal difference methods in a reinforcement learning domain. In Proceedings of the 8th Annual Conference on Genetic and Evolutionary Computation, pages 1321--1328, 2006. Google ScholarDigital Library
- S. Whiteson, M. E. Taylor, and P. Stone. Empirical studies in action selection for reinforcement learning. Adaptive Behavior, 15(1):33--50, 2007. Google ScholarDigital Library
- X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9):1423--1447, Sept. 1999.Google ScholarCross Ref
- X. Yao and Y. Liu. Evolutionary artificial neural networks that learn and generalize well. In Proceedings of the 1996 IEEE International Conference on Neural Networks, pages 159--164, 1996.Google Scholar
Index Terms
- Evolving an autonomous agent for non-Markovian reinforcement learning
Recommendations
Nested evolution of an autonomous agent using descriptive encoding
GECCO '08: Proceedings of the 10th annual conference on Genetic and evolutionary computationIn this paper, we investigate the use of nested evolution in which each step of one evolutionary process involves running a second evolutionary process. We apply this approach to build a neuroevolution system for reinforcement learning (RL) problems. ...
Evolving hierarchical memory-prediction machines in multi-task reinforcement learning
AbstractA fundamental aspect of intelligent agent behaviour is the ability to encode salient features of experience in memory and use these memories, in combination with current sensory information, to predict the best action for each situation such that ...
Neuroevolution strategies for episodic reinforcement learning
Because of their convincing performance, there is a growing interest in using evolutionary algorithms for reinforcement learning. We propose learning of neural network policies by the covariance matrix adaptation evolution strategy (CMA-ES), a ...
Comments