skip to main content
10.1145/1388969.1389080acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
tutorial

Evolving neural networks

Published: 12 July 2008 Publication History

Abstract

Neuroevolution, i.e. evolution of artificial neural networks, has recently emerged as a powerful technique for solving challenging reinforcement learning problems. Compared to traditional (e.g. value-function based) methods, neuroevolution is especially strong in domains where the state of the world is not fully known: the state can be disambiguated through recurrency, and novel situations handled through pattern matching. In this tutorial, we will review (1) neuroevolution methods that evolve fixed-topology networks, network topologies, and network construction processes, (2) ways of combining traditional neural network learning algorithms with evolutionary methods, and (3) applications of neuroevolution to game playing, robot control, resource optimization, and cognitive science.

References

[1]
A. Agogino, K. Tumer, and R. Miikkulainen, Efficient credit assignment through evaluation function decomposition http://nn.cs.utexas.edu/keyword?agogino:gecco05, in: Proceedings of the Genetic and Evolutionary Computation Conference (2005).]]
[2]
R. Aharonov-Barki, T. Beker, and E. Ruppin, Emergence of memory-Driven command neurons in evolved artificial agents, Neural Computation, 13(3):691--716 (2001).]]
[3]
P. J. Angeline, G. M. Saunders, and J. B. Pollack, An evolutionary algorithm that constructs recurrent neural networks, IEEE Transactions on Neural Networks, 5:54--65 (1994).]]
[4]
J. M. Baldwin, A new factor in evolution, The American Naturalist, 30:441--451, 536--553 (1896).]]
[5]
R. K. Belew, Evolution, learning and culture: Computational metaphors for adaptive algorithms, Complex Systems, 4:11--49 (1990).]]
[6]
B. D. Bryant and R. Miikkulainen, Neuroevolution for adaptive teams http://nn.cs.utexas.edu/keyword?bryant:cec03, in: Proceedings of the 2003 Congress on Evolutionary Computation (CEC 2003), volume 3, 2194--2201, IEEE, Piscataway, NJ (2003).]]
[7]
B. D. Bryant and R. Miikkulainen, Acquiring visibly intelligent behavior with example-guided neuroevolution, in: Proceedings of the Twenty-Second National Conference on Artificial Intelligence, AAAI Press, Menlo Park, CA (2007).]]
[8]
D. J. Chalmers, The evolution of learning: An experiment in genetic connectionism, in: Connectionist Models: Proceedings of the 1990 Summer School, D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, eds., 81--90, San Francisco: Kaufmann (1990).]]
[9]
K. Chellapilla and D. B. Fogel, Evolution, neural networks, games, and intelligence, Proceedings of the IEEE, 87:1471--1496 (1999).]]
[10]
C.-C. Chen and R. Miikkulainen, Creating melodies with evolving recurrent neural networks http://nn.cs.utexas.edu/keyword?chen:ijcnn01, in: Proceedings of the INNS-IEEE International Joint Conference on Neural Networks, 2241--2246, IEEE, Piscataway, NJ (2001).]]
[11]
D. Cliff, I. Harvey, and P. Husbands, Explorations in evolutionary robotics, Adaptive Behavior, 2:73--110 (1993).]]
[12]
R. Cornelius, K. O. Stanley, and R. Miikkulainen, Constructing adaptive AI using knowledge-based neuroevolution http://nn.cs.utexas.edu/keyword?cornelius:wisdom06, in: AI Game Programming Wisdom 3, S. Rabin, ed., 693--708, Charles River Media, Revere, MA (2006).]]
[13]
D. B. D'Ambrosio and K. O. Stanley, A novel generative encoding for exploiting neural network sensor and output geometry, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2007).]]
[14]
D. B. D'Ambrosio and K. O. Stanley, Generative encoding for multiagent learning, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2008).]]
[15]
N. S. Desai and R. Miikkulainen, Neuro-evolution and natural deduction http://nn.cs.utexas.edu/keyword?desai:ecnn00, in: Proceedings of The First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, 64--69, IEEE, Piscataway, NJ (2000).]]
[16]
J. Fan, R. Lau, and R. Miikkulainen, Utilizing domain knowledge in neuroevolution http://nn.cs.utexas.edu/keyword?fan:icml03, in: Machine Learning: Proceedings of the 20th Annual Conference (2003).]]
[17]
D. Floreano and F. Mondada, Evolutionary neurocontrollers for autonomous mobile robots, Neural Networks, 11:1461--1478 (1998).]]
[18]
D. B. Fogel, Blondie24: Playing at the Edge of AI, Kaufmann, San Francisco (2001).]]
[19]
D. B. Fogel, T. J. Hays, S. L. Hahn, and J. Quon, Further evolution of a self-learning chess program, in: Proceedings of the IEEE Symposium on Computational Intelligence and Games, IEEE, Piscataway, NJ (2005).]]
[20]
B. Fullmer and R. Miikkulainen, Using marker-based genetic encoding of neural networks to evolve finite-state behaviour http://nn.cs.utexas.edu/keyword?fullmer:evolving, in: Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life, F. J. Varela and P. Bourgine, eds., 255--262, MIT Press, Cambridge, MA (1992).]]
[21]
J. J. Gauci and K. O. Stanley, A case study on the critical role of geometric regularity in machine learning, in: Proceedings of the Twenty-Third National Conference on Artificial Intelligence, AAAI Press, Menlo Park, CA (2008).]]
[22]
F. Gomez, Robust Non-Linear Control Through Neuroevolution, Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin (2003).]]
[23]
F. Gomez, D. Burger, and R. Miikkulainen, A neuroevolution method for dynamic resource allocation on a chip multiprocessor in: Proceedings of the INNS-IEEE International Joint Conference on Neural Networks, 2355--2361, IEEE, Piscataway, NJ (2001).]]
[24]
F. Gomez and R. Miikkulainen, Incremental evolution of complex general behavior Adaptive Behavior, 5:317--342 (1997).]]
[25]
F. Gomez and R. Miikkulainen, Active guidance for a finless rocket using neuroevolution in: Proceedings of the Genetic and Evolutionary Computation Conference, 2084--2095, Kaufmann, San Francisco (2003).]]
[26]
F. Gomez, J. Schmidhuber, and R. Miikkulainen, Accelerated neural evolution through cooperatively coevolved synapses Journal of Machine Learning Research (in press).]]
[27]
B. Greer, H. Hakonen, R. Lahdelma, and R. Miikkulainen, Numerical optimization with neuroevolution in: Proceedings of the 2002 Congress on Evolutionary Computation, 361--401, IEEE, Piscataway, NJ (2002).]]
[28]
F. Gruau and D. Whitley, Adding learning to the cellular development of neural networks: Evolution and the Baldwin effect, Evolutionary Computation, 1:213--233 (1993).]]
[29]
G. E. Hinton and S. J. Nowlan, How learning can guide evolution, Complex Systems, 1:495--502 (1987).]]
[30]
A. K. Hoover, M. P. Rosario, and K. O. Stanley, Scaffolding for interactively evolving novel drum tracks for existing songs, in: Proceedings of the Sixth European Workshop on Evolutionary and Biologically Inspired Music, Sound, Art and Design, Springer, Berlin (2008).]]
[31]
G. S. Hornby, S. Takamura, J. Yokono, O. Hanagata, M. Fujita, and J. Pollack, Evolution of controllers from a high-level simulator to a high DOF robot, in: Evolvable Systems: From Biology to Hardware; Proceedings of the Third International Conference, 80--89, Springer, Berlin (2000).]]
[32]
C. Igel, Neuroevolution for reinforcement learning using evolution strategies in: Proceedings of the 2003 Congress on Evolutionary Computation, R. Sarker, R. Reynolds, H. Abbass, K. C. Tan, B. McKay, D. Essam, and T. Gedeon, eds., 2588--2595, IEEE Press, Piscataway, NJ (2003).]]
[33]
A. Keinan, B. Sandbank, C. C. Hilgetag, I. Meilijson, and E. Ruppin, Axiomatic scalable neurocontroller analysis via the shapley value, Artificial Life, 12:333--352 (2006).]]
[34]
N. Kohl and R. Miikkulainen, Evolving neural networks for fractured domains, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2008).]]
[35]
N. Kohl, K. O. Stanley, R. Miikkulainen, M. Samples, and R. Sherony, Evolving a real-world vehicle warning system in: Proceeedings of the Genetic and Evolutionary Computation Conference (2006).]]
[36]
Y. Liu, X. Yao, and T. Higuchi, Evolutionary ensembles with negative correlation learning, IEEE Transactions on Evolutionary Computation, 4:380--387 (2000).]]
[37]
A. J. Lockett, C. L. Chen, and R. Miikkulainen, Evolving explicit opponent models in game playing in: Proceeedings of the Genetic and Evolutionary Computation Conference (2007).]]
[38]
R. Maclin and J. W. Shavlik, Creating advice-taking reinforcement learners, Machine Learning, 22(1-3):251--281 (1996).]]
[39]
J. R. McDonnell and D. Waagen, Evolving recurrent perceptrons for time-series modeling, IEEE Transactions on Evolutionary Computation, 5:24--38 (1994).]]
[40]
P. McQuesten, Cultural Enhancement of Neuroevolution Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX (2002). Technical Report AI-02-295.]]
[41]
R. Miikkulainen, B. D. Bryant, R. Cornelius, I. V. Karpov, K. O. Stanley, and C. H. Yong, Computational intelligence in games in: Computational Intelligence: Principles and Practice, G. Y. Yen and D. B. Fogel, eds., IEEE Computational Intelligence Society, Piscataway, NJ (2006).]]
[42]
E. Mjolsness, D. H. Sharp, and B. K. Alpert, Scaling, machine learning, and genetic neural nets, Advances in Applied Mathematics, 10:137--163 (1989).]]
[43]
G. Monroy, K. O. Stanley, and R. Miikkulainen, Coevolution of neural networks using a layered pareto archive in: Proceeedings of the Genetic and Evolutionary Computation Conference (2006).]]
[44]
D. J. Montana and L. Davis, Training feedforward neural networks using genetic algorithms, in: Proceedings of the 11th International Joint Conference on Artificial Intelligence, 762--767, San Francisco: Kaufmann (1989).]]
[45]
D. E. Moriarty, Symbiotic Evolution of Neural Networks in Sequential Decision Tasks Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin (1997). Technical Report UT-AI97-257.]]
[46]
D. E. Moriarty and R. Miikkulainen, Evolving obstacle avoidance behavior in a robot arm in: From Animals to Animats 4: Proceedings of the Fourth International Conference on Simulation of Adaptive Behavior, P. Maes, M. J. Mataric, J.-A. Meyer, J. Pollack, and S. W. Wilson, eds., 468--475, Cambridge, MA: MIT Press (1996).]]
[47]
D. E. Moriarty and R. Miikkulainen, Forming neural networks through efficient and adaptive co-evolution Evolutionary Computation, 5:373--399 (1997).]]
[48]
D. E. Moriarty, A. C. Schultz, and J. J. Grefenstette, Evolutionary algorithms for reinforcement learning, Journal of Artificial Intelligence Research, 11:199--229 (1999).]]
[49]
S. Nolfi, J. L. Elman, and D. Parisi, Learning and evolution in neural networks, Adaptive Behavior, 2:5--28 (1994).]]
[50]
S. Nolfi and D. Floreano, Evolutionary Robotics, MIT Press, Cambridge (2000).]]
[51]
S. Nolfi and D. Parisi, Good teaching inputs do not correspond to desired responses in ecological neural networks Neural Processing Letters, 1(2):1--4 (1994).]]
[52]
D. Pardoe, M. Ryoo, and R. Miikkulainen, Evolving neural network ensembles for control problems in: Proceedings of the Genetic and Evolutionary Computation Conference (2005).]]
[53]
M. A. Potter and K. A. D. Jong, Cooperative coevolution: An architecture for evolving coadapted subcomponents, Evolutionary Computation, 8:1--29 (2000).]]
[54]
J. Reisinger and R. Miikkulainen, Acquiring evolvability through adaptive representations, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2007).]]
[55]
J. Reisinger, K. O. Stanley, and R. Miikkulainen, Evolving reusable neural modules in: Proceedings of the Genetic and Evolutionary Computation Conference (2004).]]
[56]
C. D. Rosin and R. K. Belew, New methods for competitive evolution, Evolutionary Computation, 5 (1997).]]
[57]
T. P. Runarsson and M. T. Jonsson, Evolution and design of distributed learning rules, in: Proceedings of The First IEEE Symposium on Combinations of Evolutionary Computation and Neural Networks, 59--63, IEEE, Piscataway, NJ (2000).]]
[58]
E. Ruppin, Evolutionary autonomous agents: A neuroscience perspective, Nature Reviews Neuroscience (2002).]]
[59]
J. D. Schaffer, D. Whitley, and L. J. Eshelman, Combinations of genetic algorithms and neural networks: A survey of the state of the art, in: Proceedings of the International Workshop on Combinations of Genetic Algorithms and Neural Networks, D. Whitley and J. Schaffer, eds., 1--37, IEEE Computer Society Press, Los Alamitos, CA (1992).]]
[60]
J. Secretan, N. Beato, D. B. D'Ambrosio, A. Rodriguez, A. Campbell, and K. O. Stanley, Picbreeder: Evolving pictures collaboratively online, in: Proceedings of Computer Human Interaction Conference, ACM, New York (2008).]]
[61]
C. W. Seys and R. D. Beer, Evolving walking: The anatomy of an evolutionary search, in: From Animals to Animats 8: Proceedings of the Eight International Conference on Simulation of Adaptive Behavior, S. Schaal, A. Ijspeert, A. Billard, S. Vijayakumar, J. Hallam, and J.-A. Meyer, eds., 357--363, MIT Press, Cambridge, MA (2004).]]
[62]
A. A. Siddiqi and S. M. Lucas, A comparison of matrix rewriting versus direct encoding for evolving neural networks, in: Proceedings of IEEE International Conference on Evolutionary Computation, 392--397, IEEE, Piscataway, NJ (1998).]]
[63]
Y. F. Sit and R. Miikkulainen, Learning basic navigation for personal satellite assistant using neuroevolution in: Proceedings of the Genetic and Evolutionary Computation Conference (2005).]]
[64]
K. O. Stanley, Efficient Evolution of Neural Networks Through Complexification Ph.D. thesis, Department of Computer Sciences, The University of Texas at Austin, Austin, TX (2003).]]
[65]
K. O. Stanley, B. D. Bryant, and R. Miikkulainen, Real-time neuroevolution in the NERO video game, IEEE Transactions on Evolutionary Computation, 9(6):653--668 (2005).]]
[66]
K. O. Stanley and R. Miikkulainen, Evolving neural networks through augmenting topologies, Evolutionary Computation, 10:99--127 (2002).]]
[67]
K. O. Stanley and R. Miikkulainen, A taxonomy for artificial embryogeny, Artificial Life, 9(2):93--130 (2003).]]
[68]
K. O. Stanley and R. Miikkulainen, Competitive coevolution through evolutionary complexification, Journal of Artificial Intelligence Research, 21:63--100 (2004).]]
[69]
K. O. Stanley and R. Miikkulainen, Evolving a roving eye for Go, in: Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004), Springer Verlag, Berlin (2004).]]
[70]
D. G. Stork, S. Walker, M. Burns, and B. Jackson, Preadaptation in neural circuits, in: International Joint Conference on Neural Networks (Washington, DC), 202--205, IEEE, Piscataway, NJ (1990).]]
[71]
M. Taylor, S. Whiteson, and P. Stone, Comparing evolutionary and temporal difference methods in a reinforcement learning domain, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2006).]]
[72]
J. Togelius and S. M. Lucas, Evolving robust and specialized car racing skills, in: IEEE Congress on Evolutionary Computation, 1187--1194, IEEE, Piscataway, NJ (2006).]]
[73]
J. Urzelai, D. Floreano, M. Dorigo, and M. Colombetti, Incremental robot shaping, Connection Science, 10:341--360 (1998).]]
[74]
V. Valsalam and R. Miikkulainen, Modular neuroevolution for multilegged locomotion, in: Proceeedings of the Genetic and Evolutionary Computation Conference (2008).]]
[75]
A. van Eck Conradie, R. Miikkulainen, and C. Aldrich, Adaptive control utilising neural swarming, in: Proceedings of the Genetic and Evolutionary Computation Conference, W. B. Langdon, E. Cantú-Paz, K. E. Mathias, R. Roy, D. Davis, R. Poli, K. Balakrishnan, V. Honavar, G. Rudolph, J. Wegener, L. Bull, M. A. Potter, A. C. Schultz, J. F. Miller, E. K. Burke, and N. Jonoska, eds., San Francisco: Kaufmann (2002).]]
[76]
A. van Eck Conradie, R. Miikkulainen, and C. Aldrich, Intelligent process control utilizing symbiotic memetic neuro-evolution, in: Proceedings of the 2002 Congress on Evolutionary Computation (2002).]]
[77]
G. M. Werner and M. G. Dyer, Evolution of communication in artificial organisms, in: Proceedings of the Workshop on Artificial Life (ALIFE '90), C. G. Langton, C. Taylor, J. D. Farmer, and S. Rasmussen, eds., 659--687, Reading, MA: Addison-Wesley (1991).]]
[78]
G. M. Werner and M. G. Dyer, Evolution of herding behavior in artificial animals, in: Proceedings of the Second International Conference on Simulation of Adaptive Behavior, J.-A. Meyer, H. L. Roitblat, and S. W. Wilson, eds., Cambridge, MA: MIT Press (1992).]]
[79]
S. Whiteson, N. Kohl, R. Miikkulainen, and P. Stone, Evolving keepaway soccer players through task decomposition, Machine Learning, 59:5--30 (2005).]]
[80]
S. Whiteson and P. Stone, Evolutionary function approximation for reinforcement learning, Journal of Machine Learning Research, 7:877--917 (2006).]]
[81]
S. Whiteson, P. Stone, K. O. Stanley, R. Miikkulainen, and N. Kohl, Automatic feature selection in neuroevolution, in: Proceedings of the Genetic and Evolutionary Computation Conference (2005).]]
[82]
S. Whiteson and D. Whiteson, Stochastic optimization for collision selection in high energy ysics, in: Proceedings of the Nineteenth Annual Innovative Applications of Artificial Intelligence Conference (2007).]]
[83]
D. Whitley, S. Dominic, R. Das, and C. W. Anderson, Genetic reinforcement learning for neurocontrol problems, Machine Learning, 13:259--284 (1993).]]
[84]
A. P. Wieland, Evolving controls for unstable systems, in: Connectionist Models: Proceedings of the 1990 Summer School, D. S. Touretzky, J. L. Elman, T. J. Sejnowski, and G. E. Hinton, eds., 91--102, San Francisco: Kaufmann (1990).]]
[85]
X. Yao, Evolving artificial neural networks, Proceedings of the IEEE, 87(9):1423--1447 (1999).]]
[86]
C. H. Yong and R. Miikkulainen, Cooperative coevolution of multi-agent systems, Technical Report AI01-287, Department of Computer Sciences, The University of Texas at Austin (2001).]]
[87]
C. H. Yong and R. Miikkulainen, Cooperative coevolution of multi-agent systems, Technical Report AI07-338, Department of Computer Sciences, The University of Texas at Austin (2007).]]
[88]
C. H. Yong, K. O. Stanley, R. Miikkulainen, and I. Karpov, Incorporating advice into evolution of neural networks, in: Proceedings of the Second Artificial Intelligence and Interactive Digital Entertainment Conference, AAAI Press, Menlo Park, CA (2006).]]

Cited By

View all
  • (2011)Applying evolutionary computation to mitigate uncertainty in dynamically-adaptive, high-assurance middlewareJournal of Internet Services and Applications10.1007/s13174-011-0049-43:1(51-58)Online publication date: 3-Dec-2011
  • (2009)NEAT in increasingly non-linear control situationsProceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers10.1145/1570256.1570282(2091-2096)Online publication date: 8-Jul-2009

Index Terms

  1. Evolving neural networks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    GECCO '08: Proceedings of the 10th annual conference companion on Genetic and evolutionary computation
    July 2008
    1182 pages
    ISBN:9781605581316
    DOI:10.1145/1388969
    • Conference Chair:
    • Conor Ryan,
    • Editor:
    • Maarten Keijzer
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 July 2008

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. control
    2. evolutionary computation
    3. games
    4. neural networks
    5. neuroevolution
    6. robotics

    Qualifiers

    • Tutorial

    Conference

    GECCO08
    Sponsor:

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 18 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2011)Applying evolutionary computation to mitigate uncertainty in dynamically-adaptive, high-assurance middlewareJournal of Internet Services and Applications10.1007/s13174-011-0049-43:1(51-58)Online publication date: 3-Dec-2011
    • (2009)NEAT in increasingly non-linear control situationsProceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers10.1145/1570256.1570282(2091-2096)Online publication date: 8-Jul-2009

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media