skip to main content
10.1145/2463372.2463513acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Shaping fitness function for evolutionary learning of game strategies

Published: 06 July 2013 Publication History

Abstract

In evolutionary learning of game-playing strategies, fitness evaluation is based on playing games with certain opponents. In this paper we investigate how the performance of these opponents and the way they are chosen influence the efficiency of learning. For this purpose we introduce a simple method for shaping the fitness function by sampling the opponents from a biased performance distribution. We compare the shaped function with existing fitness evaluation approaches that sample the opponents from an unbiased performance distribution or from a coevolving population. In an extensive computational experiment we employ these methods to learn Othello strategies and assess both the absolute and relative performance of the elaborated players. The results demonstrate the superiority of the shaping approach, and can be explained by means of performance profiles, an analytical tool that evaluate the evolved strategies using a range of variably skilled opponents.

References

[1]
K. J. Binkley, K. Seehart, and M. Hagiwara. A Study of Artificial Neural Network Architectures for Othello Evaluation Functions. Transactions of the Japanese Society for Artificial Intelligence, 22(5):461--471, 2007.
[2]
A. Bucci, J. B. Pollack, and E. de Jong. Automated extraction of problem structure. In K. D. et al., editor, Genetic and Evolutionary Computation -- GECCO-2004, Part I, volume 3102 of Lecture Notes in Computer Science, pages 501--512, Seattle, WA, USA, 26--30 June 2004. Springer-Verlag.
[3]
E. K. Burke, M. Hyde, G. Kendall, G. Ochoa, E. Ozcan, and J. R. Woodward. A classification of hyper-heuristics approaches. In M. Gendreau and J.-Y. Potvin, editors, Handbook of Metaheuristics, volume 57 of International Series in Operations Research & Management Science, chapter 15, pages 449--468. Springer, 2nd edition, 2010.
[4]
S. Y. Chong, M. K. Tan, and J. D. White. Observing the evolution of neural networks learning to play the game of othello. Evolutionary Computation, IEEE Transactions on, 9(3):240--251, 2005.
[5]
S. Y. Chong, P. Tino, D. C. Ku, and Y. Xin. Improving Generalization Performance in Co-Evolutionary Learning. IEEE Transactions on Evolutionary Computation, 16(1):70--85, 2012.
[6]
E. D. de Jong. The maxsolve algorithm for coevolution. In H.-G. B. et al., editor, GECCO 2005: Proceedings of the 2005 conference on Genetic and evolutionary computation, volume 1, pages 483--489, Washington DC, USA, 25--29 June 2005. ACM Press.
[7]
E. D. de Jong and J. B. Pollack. Ideal Evaluation from Coevolution. Evolutionary Computation, 12(2):159--192, Summer 2004.
[8]
A. Eiben and S. Smit. Parameter tuning for configuring and analyzing evolutionary algorithms. Swarm and Evolutionary Computation, 1(1):19--31, 2011.
[9]
T. Erez and W. Smart. What does shaping mean for computational reinforcement learning? In Development and Learning, 2008. ICDL 2008. 7th IEEE International Conference on, pages 215 --219, aug. 2008.
[10]
S. G. Ficici and J. B. Pollack. Pareto optimality in coevolutionary learning. In J. Kelemen and P. Sosík, editors, Advances in Artificial Life, 6th European Conference, ECAL 2001, volume 2159 of Lecture Notes in Computer Science, pages 316--325, Prague, Czech Republic, 2001. Springer.
[11]
W. Jaskowski and K. Krawiec. Formal analysis, hardness and algorithms for extracting internal structure of test-based problems. Evolutionary Computation, 19(4):639--671, 2011.
[12]
W. Jaskowski, K. Krawiec, and B. Wieloch. Evolving strategy for a probabilistic game of imperfect information using genetic programming. Genetic Programming and Evolvable Machines, 9(4):281--294, 2008.
[13]
W. Jaskowski, P. Liskowski, M. Szubert, and K. Krawiec. Improving coevolution by random sampling. In GECCO'13: Proceedings of the 15th annual conference on Genetic and Evolutionary Computation, Amsterdam, The Netherlands, July 2013. ACM.
[14]
H. Juillé and J. B. Pollack. Coevolving the "ideal" trainer: Application to the discovery of cellular automata rules. In University of Wisconsin, pages 519--527. Morgan Kaufmann, 1998.
[15]
K. Krawiec, W. Ja\'skowski, and M. Szubert. Evolving small-board go players using coevolutionary temporal difference learning with archive. International Journal of Applied Mathematics and Computer Science, 21(4):717--731, 2011.
[16]
K. Krawiec and M. Szubert. Learning n-tuple networks for othello by coevolutionary gradient search. In N. K. et al, editor, GECCO 2011 Proceedings, pages 355--362. ACM, ACM, 2011.
[17]
S. M. Lucas. Learning to play Othello with N-tuple systems. Australian Journal of Intelligent Information Processing Systems, Special Issue on Game Technology, 9(4):1--20, 2007.
[18]
S. M. Lucas and T. P. Runarsson. Temporal difference learning versus co-evolution for acquiring othello position evaluation. In CIG, pages 52--59, 2006.
[19]
S. Luke and R. P. Wiegand. When coevolutionary algorithms exhibit evolutionary dynamics. In A. M. Barry, editor, GECCO 2002: Proceedings of the Bird of a Feather Workshops, Genetic and Evolutionary Computation Conference, pages 236--241, New York, 2002.
[20]
E. Popovici, A. Bucci, R. P. Wiegand, and E. D. de Jong. Handbook of Natural Computing, chapter Coevolutionary Principles. Springer-Verlag, 2011.
[21]
J. Randløv and P. Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 463--471. Morgan Kaufmann, San Francisco, CA, 1998.
[22]
C. D. Rosin and R. K. Belew. New methods for competitive coevolution. Evolutionary Computation, 5(1):1--29, 1997.
[23]
S. Samothrakis, S. Lucas, T. Runarsson, and D. Robles. Coevolving Game-Playing Agents: Measuring Performance and Intransitivities. IEEE Transactions on Evolutionary Computation, (99):1--15, 2012.
[24]
B. Skinner. The behavior of organisms: An experimental analysis. Appleton-Century, 1938.
[25]
R. A. Watson and J. B. Pollack. Coevolutionary dynamics in a minimal substrate. In L. S. et al., editor, Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pages 702--709, San Francisco, California, USA, 7--11 July 2001. Morgan Kaufmann.

Cited By

View all
  • (2021)Solving the Rubik's cube with stepwise deep learningExpert Systems10.1111/exsy.1266538:3Online publication date: 24-Jan-2021
  • (2021)A Systematic Review of Coevolution in Real-Time Strategy GamesIEEE Access10.1109/ACCESS.2021.31157689(136647-136665)Online publication date: 2021
  • (2020)Solving complex problems with coevolutionary algorithmsProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3389874(832-858)Online publication date: 8-Jul-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '13: Proceedings of the 15th annual conference on Genetic and evolutionary computation
July 2013
1672 pages
ISBN:9781450319638
DOI:10.1145/2463372
  • Editor:
  • Christian Blum,
  • General Chair:
  • Enrique Alba
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2013

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. coevolution
  2. fitness evaluation
  3. othello
  4. shaping

Qualifiers

  • Research-article

Conference

GECCO '13
Sponsor:
GECCO '13: Genetic and Evolutionary Computation Conference
July 6 - 10, 2013
Amsterdam, The Netherlands

Acceptance Rates

GECCO '13 Paper Acceptance Rate 204 of 570 submissions, 36%;
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2021)Solving the Rubik's cube with stepwise deep learningExpert Systems10.1111/exsy.1266538:3Online publication date: 24-Jan-2021
  • (2021)A Systematic Review of Coevolution in Real-Time Strategy GamesIEEE Access10.1109/ACCESS.2021.31157689(136647-136665)Online publication date: 2021
  • (2020)Solving complex problems with coevolutionary algorithmsProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3389874(832-858)Online publication date: 8-Jul-2020
  • (2019)Solving complex problems with coevolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3319619.3323384(975-1001)Online publication date: 13-Jul-2019
  • (2019)Stepwise Evolutionary Learning Using Deep Learned Guidance FunctionsArtificial Intelligence XXXVI10.1007/978-3-030-34885-4_4(50-62)Online publication date: 19-Nov-2019
  • (2018)Solving complex problems with coevolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3205651.3207888(880-906)Online publication date: 6-Jul-2018
  • (2017)Accelerating coevolution with adaptive matrix factorizationProceedings of the Genetic and Evolutionary Computation Conference10.1145/3071178.3071320(457-464)Online publication date: 1-Jul-2017
  • (2017)Solving complex problems with coevolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3067695.3067705(782-806)Online publication date: 15-Jul-2017
  • (2016)Solving Complex Problems with Coevolutionary AlgorithmsProceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion10.1145/2908961.2926989(687-713)Online publication date: 20-Jul-2016
  • (2016)Coevolutionary CMA-ES for Knowledge-Free Learning of Game Position EvaluationIEEE Transactions on Computational Intelligence and AI in Games10.1109/TCIAIG.2015.24647118:4(389-401)Online publication date: Dec-2016
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media