Skip to main content
SpringerLink
Log in

Adaptive Agents in Minecraft: A Hybrid Paradigm for Combining Domain Knowledge with Reinforcement Learning

  • Conference paper
  • First Online:
Autonomous Agents and Multiagent Systems (AAMAS 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10643))

Included in the following conference series:

Abstract

We present a pilot study focused on creating flexible Hierarchical Task Networks that can leverage Reinforcement Learning to repair and adapt incomplete plans in the simulated rich domain of Minecraft. This paper presents an early evaluation of our algorithm using simulation for adaptive agents planning in a dynamic world. Our algorithm uses an hierarchical planner and can theoretically be used for any type of “bot”. The main aim of our study is to create flexible knowledge-based planners for robots, which can leverage exploration and guide learning more efficiently by imparting structure using domain knowledge. Results from simulations indicate that a combined approach using both HTN and RL is more flexible than HTN alone and more efficient than RL alone.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/Microsoft/malmo.

  2. 2.

    https://bitbucket.org/dananau/pyhop.

References

  1. Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009). https://doi.org/10.1016/j.robot.2008.10.024

    Article  Google Scholar 

  2. Becker, W.C.: Parents are Teachers: A Child Management Program (1971)

    Google Scholar 

  3. Belker, T., Hammel, M., Hertzberg, J.: Learning to optimize mobile robot navigation based on HTN plans. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA 2003, vol. 3, pp. 4136–4141. IEEE (2003)

    Google Scholar 

  4. Berk, L.E., Winsler, A.: Scaffolding Children’s Learning: Vygotsky and Early Childhood Education. NAEYC Research into Practice Series, vol. 7. ERIC (1995)

    Google Scholar 

  5. Breazeal, C., Buchsbaum, D., Gray, J., Gatenby, D., Blumberg, B.: Learning from and about others: towards using imitation to bootstrap the social understanding of others by robots. Learning 11(1-2) (2006)

    Google Scholar 

  6. Breazeal, C., Scassellati, B.: Robots that imitate humans. Trends Cogn. Sci. 6(11), 481–487 (2002)

    Article  Google Scholar 

  7. Cox, M.T., Raja, A.: Metareasoning: Thinking about Thinking. MIT Press, Cambridge (2011)

    Book  Google Scholar 

  8. Erol, K., Hendler, J.A., Nau, D.S.: UMCP: a sound and complete procedure for hierarchical task-network planning. In: AIPS, vol. 94, pp. 249–254 (1994)

    Google Scholar 

  9. Erol, K., Hendler, J., Nau, D.S.: HTN planning: complexity and expressivity. In: AAAI, vol. 94, pp. 1123–1128 (1994)

    Google Scholar 

  10. Fitzgerald, T., Thomaz, A., Goel, A.: Human-Robot Co-Creativity: Task Transfer of a Spectrum of Similarity. In: Proceedings of Seventh International Conference on Computational Creativity, Atlanta, June 2017 (2017)

    Google Scholar 

  11. Hayashi, H., Tokura, S., Hasegawa, T., Ozaki, F.: Dynagent: an incremental forward-chaining HTN planning agent in dynamic domains. In: Baldoni, M., Endriss, U., Omicini, A., Torroni, P. (eds.) DALT 2005. LNCS (LNAI), vol. 3904, pp. 171–187. Springer, Heidelberg (2006). https://doi.org/10.1007/11691792_11

    Chapter  Google Scholar 

  12. Hogg, C., Kuter, U., Muñoz-Avila, H.: Learning hierarchical task networks for nondeterministic planning domains. In: IJCAI International Joint Conference on Artificial Intelligence, pp. 1708–1714 (2009)

    Google Scholar 

  13. Jones, J.K., Goel, A.K.: Perceptually grounded self-diagnosis and self-repair of domain knowledge. Knowl. Based Syst. 27, 281–301 (2012)

    Article  Google Scholar 

  14. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1278 (2013). http://repository.cmu.edu/robotics

    Article  Google Scholar 

  15. Magnenat, S., Chappelier, J.-C., Mondada, F.: Integration of online learning into htn planning for robotic tasks. In: AAAI Spring Symposium: Designing Intelligent Robots (2012)

    Google Scholar 

  16. Magnenat, S., Voelkle, M., Mondada, F.: Planner9, a HTN planner distributed on groups of miniature mobile robots. In: Xie, M., Xiong, Y., Xiong, C., Liu, H., Hu, Z. (eds.) ICIRA 2009. LNCS (LNAI), vol. 5928, pp. 1013–1022. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10817-4_99

    Chapter  Google Scholar 

  17. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013). ArXiv preprint: arXiv:1312.5602

  18. Mohseni-Kabir, A., Rich, C., Chernova, S., Sidner, C.L., Miller, D.: Interactive hierarchical task learning from a single demonstration. In: Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction, pp. 205–212. ACM (2015)

    Google Scholar 

  19. Morimoto, J., Doya, K.: Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. Robot. Auton. Syst. 36, 37–51 (2001)

    Article  MATH  Google Scholar 

  20. Murdock, J.W., Goel, A.K.: Meta-case-based reasoning: self-improvement through self-understanding. J. Exp. Theor. Artif. Intell. 20(1), 1–36 (2008)

    Article  Google Scholar 

  21. Nau, D., Au, T.-C., Ilghami, O., Kuter, U., Murdock, J.W., Wu, D., Yaman, F.: SHOP2: an HTN planning system. J. Artif. Intell. Res. 20, 379–404 (2003)

    MATH  Google Scholar 

  22. Nau, D., Cao, Y., Lotem, A., Munoz-Avila, H.: SHOP: simple hierarchical ordered planner (1999)

    Google Scholar 

  23. Nejati, N., Langley, P., Konik, T.: Learning hierarchical task networks by observation. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 665–672. ACM (2006)

    Google Scholar 

  24. Sidowski, J.B., Wycko, L.B., Tabory, L.: The influence of reinforcement and punishment in a minimal social situation. J. Abnorm. Soc. Psychol. 52(1), 115 (1956)

    Article  Google Scholar 

  25. Sun, R., Zhang, X.: Top-down versus bottom-up learning in cognitive skill acquisition. Cogn. Syst. Res. 5(1), 63–89 (2004)

    Article  Google Scholar 

  26. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1. MIT Press, Cambridge (1998)

    Google Scholar 

  27. Tessler, C., Givony, S., Zahavy, T., Mankowitz, D.J., Mannor, S.: A deep hierarchical approach to lifelong learning in minecraft, CoRR, volume abs/1604.07255 (2016). http://arxiv.org/abs/1604.07255

  28. Ulam, P., Goel, A., Jones, J., Murdock, W.: Using model-based reflection to guide reinforcement learning. In: Reasoning, Representation, and Learning in Computer Games, p. 107 (2005)

    Google Scholar 

  29. Ulam, P., Jones, J., Goel, A.K.: Combining model-based meta-reasoning and reinforcement learning for adapting game-playing agents. In: Artificial Intelligence and Interactive Digital Entertainment Conference, pp. 132–137 (2008)

    Google Scholar 

  30. Williamson, M., Decker, K., Sycara, K.: Unified information and control flow in hierarchical task networks. In: Proceedings of the AAAI 1996 Workshop on Theories of Planning, Action, and Control (1996)

    Google Scholar 

  31. Wolfe, J., Marthi, B., Russell, S.: Combined task and motion planning for mobile manipulation. In: International Conference on Automated Planning and Scheduling, pp. 254–257 (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Contextual Robotics Institute, University of California, San Diego, La Jolla, CA, 92043, USA

    Priyam Parashar

  2. American Family Insurance, Chicago, IL, USA

    Bradley Sheneman

  3. School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, 30332-0250, USA

    Ashok K. Goel

Authors
  1. Priyam Parashar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bradley Sheneman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ashok K. Goel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Priyam Parashar .

Editor information

Editors and Affiliations

  1. University of Central Florida, Orlando, Florida, USA

    Gita Sukthankar

  2. IIIA-CSIC, Bellaterra, Spain

    Juan A. Rodriguez-Aguilar

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Parashar, P., Sheneman, B., Goel, A.K. (2017). Adaptive Agents in Minecraft: A Hybrid Paradigm for Combining Domain Knowledge with Reinforcement Learning. In: Sukthankar, G., Rodriguez-Aguilar, J. (eds) Autonomous Agents and Multiagent Systems. AAMAS 2017. Lecture Notes in Computer Science(), vol 10643. Springer, Cham. https://doi.org/10.1007/978-3-319-71679-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-71679-4_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-71678-7

  • Online ISBN: 978-3-319-71679-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation