skip to main content
10.1145/2001576.2001626acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
research-article

Learning n-tuple networks for othello by coevolutionary gradient search

Published: 12 July 2011 Publication History

Abstract

We propose Coevolutionary Gradient Search, a blueprint for a family of iterative learning algorithms that combine elements of local search and population-based search. The approach is applied to learning Othello strategies represented as n-tuple networks, using different search operators and modes of learning. We focus on the interplay between the continuous, directed, gradient-based search in the space of weights, and fitness-driven, combinatorial, coevolutionary search in the space of entire n-tuple networks. In an extensive experiment, we assess both the objective and relative performance of algorithms, concluding that the hybridization of search techniques improves the convergence. The best algorithms not only learn faster than constituent methods alone, but also produce top ranked strategies in the online Othello League.

References

[1]
P. J. Angeline and J. B. Pollack. Competitive Environments Evolve Better Solutions for Complex Tasks. In S. Forrest, editor, Proceedings of the 5th International Conference on Genetic Algorithms, pages 264--270, 1993.
[2]
W. W. Bledsoe and I. Browning. Pattern Recognition and Reading by Machine. In Papers presented at the December 1--3, 1959, eastern joint IRE-AIEE-ACM computer conference, IRE-AIEE-ACM '59 (Eastern), pages 225--232, New York, NY, USA, 1959. ACM.
[3]
M. Buro. Logistello: A Strong Learning Othello Program. In 19th Annual Conference Gesellschaft für Klassifikation e.V., 1995.
[4]
S. Y. Chong, M. K. Tan, and J. D. White. Observing the Evolution of Neural Networks Learning to Play the Game of Othello. IEEE Trans. Evolutionary Computation, 9(3):240--251, 2005.
[5]
D. Denaro and D. Parisi. Cultural Evolution in a Population of Neural Networks. In Proceedings of the 8th italian workshop on neural nets, 1997.
[6]
S. G. Ficici. Solution Concepts in Coevolutionary Algorithms. PhD thesis, Waltham, MA, USA, 2004. Adviser-Jordan B. Pollack.
[7]
D. B. Fogel. Blondie24: Playing at the Edge of AI. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2002.
[8]
K.-J. Kim, H. Choi, and S.-B. Cho. Hybrid of Evolution and Reinforcement Learning for Othello Players. Computational Intelligence and Games, 2007. CIG 2007. IEEE Symposium on, pages 203--209, 2007.
[9]
A. Kolcz and N. M. Allinson. N-tuple Regression Network. Neural Netw., 9:855--869, July 1996.
[10]
K. Krawiec and M. Szubert. Coevolutionary Temporal Difference Learning for Small-Board Go. In IEEE Congress on Evolutionary Computation (CEC 2010), pages 1513--1520, 2010.
[11]
S. Lucas. Learning to Play Othello with N-tuple Systems. Australian Journal of Intelligent Information Processing Systems, Special Issue on Game Technology, 9(4):01--20, 2007.
[12]
S. Lucas and T. P. Runarsson. Othello Competition; http://algoval.essex.ac.uk:8080/othello/League.jsp.
[13]
S. M. Lucas and T. P. Runarsson. Temporal Difference Learning versus Co-Evolution for Acquiring Othello Position Evaluation. In CIG, pages 52--59, 2006.
[14]
S. Luke. ECJ 20 -- A Java-based Evolutionary Computation Research System. http://cs.gmu.edu/ eclab/projects/ecj/, 2010.
[15]
E. P. Manning. Using Resource-Limited Nash Memory to Improve an Othello Evaluation Function. IEEE Transactions on Computational Intelligence and AI in Games, 2(1):40--53, 2010.
[16]
J. B. Pollack and A. D. Blair. Co-Evolution in the Successful Learning of Backgammon Strategy. Machine Learning, 32(3):225--240, 1998.
[17]
R. Rohwer and M. Morciniec. A Theoretical and Experimental Account of N-tuple Classifier performance. Neural Comput., 8:629--642, April 1996.
[18]
C. D. Rosin and R. K. Belew. New Methods for Competitive Coevolution. Evolutionary Computation, 5(1):1--29, 1997.
[19]
T. P. Runarsson and S. Lucas. Co-Evolution versus Self-Play Temporal Difference Learning for Acquiring Position Evaluation in Small-Board Go. IEEE Transactions on Evolutionary Computation, 9, 2005.
[20]
K. O. Stanley. Efficient Evolution of Neural Networks Through Complexification. PhD thesis, Department of Computer Sciences, The University of Texas at Austin, 2004.
[21]
R. S. Sutton. Learning to Predict by the Methods of Temporal Differences. Machine Learning, 3:9--44, 1988.
[22]
M. Szubert. cECJ -- Coevolutionary Computation in Java. http://www.cs.put.poznan.pl/mszubert/projects/cecj.html, 2010.
[23]
M. Szubert, W. Jaskowski, and K. Krawiec. Coevolutionary Temporal Difference Learning for Othello. In IEEE Symposium on Computational Intelligence and Games, 2009.
[24]
G. Tesauro. Temporal Difference Learning and TD-Gammon. Commun. ACM, 38(3):58--68, 1995.
[25]
R. A. Watson and J. B. Pollack. Coevolutionary Dynamics in a Minimal Substrate. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2001), pages 702--709, 2001.

Cited By

View all
  • (2024)Online Attentive Kernel-Based Off-Policy Temporal Difference LearningApplied Sciences10.3390/app14231111414:23(11114)Online publication date: 28-Nov-2024
  • (2023)Online attentive kernel-based temporal difference learningKnowledge-Based Systems10.1016/j.knosys.2023.110902278(110902)Online publication date: Oct-2023
  • (2022)Hybrid Training Strategies: Improving Performance of Temporal Difference Learning in Board GamesApplied Sciences10.3390/app1206285412:6(2854)Online publication date: 10-Mar-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '11: Proceedings of the 13th annual conference on Genetic and evolutionary computation
July 2011
2140 pages
ISBN:9781450305570
DOI:10.1145/2001576
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 July 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. coevolution
  2. n-tuple networks
  3. othello
  4. temporal difference learning

Qualifiers

  • Research-article

Conference

GECCO '11
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)9
  • Downloads (Last 6 weeks)3
Reflects downloads up to 19 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Online Attentive Kernel-Based Off-Policy Temporal Difference LearningApplied Sciences10.3390/app14231111414:23(11114)Online publication date: 28-Nov-2024
  • (2023)Online attentive kernel-based temporal difference learningKnowledge-Based Systems10.1016/j.knosys.2023.110902278(110902)Online publication date: Oct-2023
  • (2022)Hybrid Training Strategies: Improving Performance of Temporal Difference Learning in Board GamesApplied Sciences10.3390/app1206285412:6(2854)Online publication date: 10-Mar-2022
  • (2021)Negotiation in Hidden Identity: Designing Protocol for Werewolf GameRecent Advances in Agent-based Negotiation10.1007/978-981-16-0471-3_6(87-102)Online publication date: 11-May-2021
  • (2020)Solving complex problems with coevolutionary algorithmsProceedings of the 2020 Genetic and Evolutionary Computation Conference Companion10.1145/3377929.3389874(832-858)Online publication date: 8-Jul-2020
  • (2020)Adversarial genetic programming for cyber security: a rising application domain where GP mattersGenetic Programming and Evolvable Machines10.1007/s10710-020-09389-yOnline publication date: 2-Apr-2020
  • (2019)Solving complex problems with coevolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3319619.3323384(975-1001)Online publication date: 13-Jul-2019
  • (2019)An Efficient Training Strategy for a Temporal Difference Learning Based Tic-Tac-Toe Automatic PlayerInventive Computation Technologies10.1007/978-3-030-33846-6_47(423-430)Online publication date: 3-Nov-2019
  • (2018)I Know You Better Than You Know YourselfProceedings of the 6th International Conference on Human-Agent Interaction10.1145/3284432.3284453(144-152)Online publication date: 4-Dec-2018
  • (2018)Solving complex problems with coevolutionary algorithmsProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3205651.3207888(880-906)Online publication date: 6-Jul-2018
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media