Skip to main content

Advertisement

SpringerLink
Log in

Methods for approximating value functions for the Dominion card game

  • Special Issue
  • Published:
Evolutionary Intelligence Aims and scope Submit manuscript

Abstract

Artificial neural networks have been successfully used to approximate value functions for tasks involving decision making. In domains where decisions require a shift in judgment as the overall state changes, it is hypothesized here that methods utilizing multiple artificial neural networks are likely to provide a benefit as an approximation of a value function over those that employ a single network. The card game Dominion was chosen as the domain to examine this. This paper compares artificial neural networks generated by multiple machine learning methods successfully applied to other games (such as in TD-Gammon) to a genetic algorithm method for generating two neural networks for different phases of the game along with evolving the transition point. The results demonstrate a greater success ratio with the genetic algorithm applied to two neural networks. This suggests that future work examining more complex neural network configurations and richer evolutionary exploration could apply to Dominion as well as other domains necessitating shifts in strategy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Haykin S (1998) Neural networks: a comprehensive foundation. Prentice Hall, Upper Saddle River, NJ

  2. Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366. doi:10.1016/0893-6080(89)90020-8

    Article  Google Scholar 

  3. Paliwal M, Kumar U (2009) Neural networks and statistical techniques: a review of applications. Expert Syst Appl 36(1):2–17. doi:10.1016/j.eswa.2007.10.005

    Article  Google Scholar 

  4. Coulom R (2004) High-accuracy value-function approximation with neural networks applied to the Acrobot. In: ESAAN’2004 Proceedings of 12th European symposium on artificial neural networks, Bruges, Belgium, pp 7–12. doi:10.1.1.144.1025

  5. Goldberg D, Holland J (1988) Genetic algorithms and machine learning. Mach Learn 3(2–3):95–99. doi:10.1023/A:1022602019183

    Article  Google Scholar 

  6. Leng K, Chen X (2012) A genetic algorithm approach for TOC-based supply chain coordination. Appl Math Inf Sci 6(3):767–774

    Google Scholar 

  7. Wong S, Feng S (2012) A study on genetic algorithm and neural network for mini-games. J Inf Sci Eng 28:145–159. doi:10.1109/TAAI.2010.35

    Google Scholar 

  8. Winder R (2013) Generating artificial neural networks for value function approximation in a domain requiring a shifting strategy. In: Applications of evolutionary computation. Lecturer notes in computer science, vol 7835. Springer, Berlin, Heidelberg, pp 294–303. doi:10.1007/978-3-642-37192-9_30

  9. Tesauro G (1992) Practical issues in temporal difference learning. Mach Learn 8:257–277. doi:10.1007/978-1-4615-3618-5_3

    MATH  Google Scholar 

  10. Tesauro G (1995) Temporal difference learning and TD-Gammon. Commun ACM 38(3):58–68. doi:10.1145/203330.203343

    Article  Google Scholar 

  11. Sutton R, Barto A (1998) Reinforcement learning. MIT Press, Cambridge

    Google Scholar 

  12. Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536. doi:10.1038/323533a0

    Article  Google Scholar 

  13. Ghory I (2004) Reinforcement learning in board games. Technical report CSTR-04-004, Department of Computer Science, University of Bristol, UK

  14. Pfeiffer M (2004) Reinforcement learning of strategies for Settlers of Catan. In: Proceedings of the international conference on computer games: artificial intelligence, design and education, reading

  15. Konen W, Bartz-Beielstein T (2009) Reinforcement learning for games: failures and successes. In: GECCO’09 Proceedings of the 11th annual conference companion on genetic evolutionary computation conference: late breaking papers. ACM, New York, pp 2641–2648. doi:10.1145/1570256.1570375

  16. Galway L, Charles D, Black M (2009) Improving temporal difference game agent control using a dynamic exploration rate during control learning. In: Proceedings of IEEE Symposium on Computational Intelligence and Games, Milano, pp 38–45. doi:10.1109/CIG.2009.5286497

  17. Pollack J, Blair A (1998) Co-evolution in the successful learning of backgammon strategy. Mach Learn 32(3):225–240. doi:10.1023/A:1007417214905

    Article  MATH  Google Scholar 

  18. Oon W, Henz M (2007) M2ICAL analyses HC-Gammon. In: AAAI’07 Proceedings of the 22nd AAAI conference on artificial intelligence. AAAI Press, Vancouver, British Columbia, Canada, pp 621–626

  19. Azaria Y, Sipper M (2005) GP-Gammon: using genetic programming to evolve backgammon players. Genet Program 3447:132–142. doi:10.1007/978-3-540-31989-4_12

    Article  Google Scholar 

  20. Koza J (1992) Genetic programming on the programming of computers by means of natural selection (complex adaptive systems). The MIT Press, Cambridge

    Google Scholar 

  21. Willis M, Hiden H, Marenbach P, McKay B, Montague G (1997) Genetic programming: an introduction and survey of applications. In: Proceedings of second international conference on GALESIA’97, Glasgow, pp 314–319. doi:10.1049/cp:19971199

  22. Bensabbat A, Sipper M (2012) Evolving board-game players with genetic programming. In: GECCO’11 Proceedings of the 13th annual conference companion on genetic and evolutionary computation. ACM, New York, pp 739–742. doi:10.1145/2001868.2002080

  23. Fogel D, Hays T, Hahn S, Quon J (2004) A self-learning evolutionary chess program. Proc IEEE 92(12):1947–1954. doi:10.1109/JPROC.2004.837633

    Article  Google Scholar 

  24. Hauptman A, Sipper M (2005) GP-EndChess: using genetic programming to evolve chess endgame players. Genet Program 3447:120–131. doi:10.1007/978-3-540-31989-4_11

    Article  Google Scholar 

  25. Chellapilla K, Fogel D (1999) Evolving neural networks to play checkers without relying on expert knowledge. IEEE Trans Neural Netw 10(6):1382–1391. doi:10.1109/72.809083

    Article  Google Scholar 

  26. Cai X, Wensch D (2007) Computer Go: a grand challenge to AI. Stud Comput Intell 63:443–465. doi:10.1007/978-3-540-71984-7_16

    Article  Google Scholar 

  27. Richards N, Moriarty D, Mikkulainen R (1998) Evolving neural networks to play Go. Appl Intell 8(1):85–96. doi:10.1023/A:1008224732364

    Article  Google Scholar 

  28. Lin C Ting C (2011) Emergent tactical formation using genetic algorithm in real-time strategy games. In: International conference on technologies and applications of artificial intelligence, Chung-Li, pp 325–330. doi:10.1109/TAAI.2011.63

  29. Othman N, Decraene J, Cai W, Hu N, Low M, Gouaillard A (2012) Simulation-based optimization of StarCraft tactical AI through evolutionary computation. In: IEEE conference on computational intelligence and games, Granada, pp 394–401. doi:10.1109/CIG.2012.6374182

  30. Tong C, On C, Teo J, Kiring A (2011) Evolving neural controllers using GA for Warcraft 3-real time strategy game. In: Sixth international conference on bio-inspired computing: theories and applications, Penang, pp 15–20. doi:10.1109/BIC-TA.2011.70

  31. Mahlmann T, Togelius J, Yannakakis G (2012) Evolving card sets toward balancing Dominion. In: IEEE congress on evolutionary computation (CEC), Brisbane, pp 1–8. doi:10.1109/CEC.2012.6256441

  32. Vaccarino DX (2008) Dominion game rules. http://www.riograndegames.com/uploads/Game/Game_278_gameRules.pdf. Accessed Apr 2013

  33. Vaccarino DX (2009) Dominion: intrigue game rules. http://www.riograndegames.com/uploads/Game/Game_306_gameRules.pdf. Accessed Apr 2013

  34. Vaccarino DX (2009) Dominion: seaside game rules. http://www.riograndegames.com/uploads/Game/Game_326_gameRules.pdf. Accessed Apr 2013

  35. Vaccarino DX (2010) Dominion: prosperity game rules. http://www.riograndegames.com/uploads/Game/Game_361_gameRules.pdf. Accessed Apr 2013

  36. Riedmiller M, Braun H (1993) A direct adaptive method for faster backpropagation learning: the RPROP algorithm. In: IEEE conference on neural networks, San Francisco, CA, pp 586–591. doi:10.1109/ICNN.1993.298623

Download references

Author information

Authors and Affiliations

  1. The MITRE Corporation, McLean, VA, USA

    Ransom K. Winder

Authors
  1. Ransom K. Winder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ransom K. Winder.

Appendix: Algorithm for reinforcement learning methods (R/TD) using RPROP

Appendix: Algorithm for reinforcement learning methods (R/TD) using RPROP

figure afigure a

Rights and permissions

Reprints and permissions

About this article

Cite this article

Winder, R.K. Methods for approximating value functions for the Dominion card game. Evol. Intel. 6, 195–204 (2014). https://doi.org/10.1007/s12065-013-0096-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12065-013-0096-9

Keywords

Search

Navigation