Skip to main content
Springer Nature Link
Log in

Finding Hidden Hierarchy in Reinforcement Learning

  • Conference paper
Knowledge-Based Intelligent Information and Engineering Systems (KES 2005)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3683))

  • 1081 Accesses

Abstract

HEXQ is a reinforcement learning algorithm that decomposes a problem into subtasks and constructs a hierarchy using state variables. The maximum number of levels is constrained by the number of variables representing a state. In HEXQ, values learned for a subtask can be reused in different contexts if the subtasks are identical. If not, values for non-identical subtasks need to be trained separately. This paper introduces a method that tackles these two restrictions. Experimental results show that this method can save the training time dramatically.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  2. Dayan, P., Hinton, G.E.: Feudal reinforcement learning. In: Hanson, S.J., et al. (eds.) Advances in Neural Information Processing Systems, vol. 5, pp. 271–278. Morgan Kaufmann, San Mateo (1993)

    Google Scholar 

  3. Singh, S.P.: Reinforcement learning with a hierarchy of abstract models. In: Proceedings of the Tenth National Conference on Artificial Intelligence, San Jose, CA, USA (1992)

    Google Scholar 

  4. Hengst, B.: Discovering Hierarchy in Reinforcement Learning with HEXQ. In: Maching Learning: Proceedings of the Nineteenth International Conference on Machine Learning 2002 (2003)

    Google Scholar 

  5. Kernighan, B.W., Lin, C.: An Efficient Heuristic Procedure for Partitioning Graphs. Bell Systems Technology J. 49(2), 292–370 (1970)

    Google Scholar 

  6. Hochbaum, D.S., Pathria, A.: The bottleneck graph partition problem. Networks 28(4), 221–225 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  7. Dutt, S.: New Faster Kernighan-Lin-Type Graph-Partitioning Algorithms. In: ICCAD 1993: Proceedings of the 1993 IEEE/ACM international conference on Computer-aided design, Santa Clara, California, United States (1993)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Autonomous Systems, Information and Communication Technology Centre, CSIRO, PO Box 76, Epping, NSW 1710, Australia

    Geoff Poulton & Ying Guo

  2. University of NSW, Australia

    Wen Lu

Authors
  1. Geoff Poulton
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wen Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Business, La Trobe University, 3086, Melbourne, Victoria, Australia

    Rajiv Khosla

  2. Centre for SMART systems Engineering Research Centre, University of Brighton, BN2 4GJ, Moulsecoomb, Brighton, UK

    Robert J. Howlett

  3. School of Electrical and Information Engineering, Knowledge Based Intelligent Engineering Systems Centre, University of South Australia, 5095, Mawson Lakes, SA, Australia

    Lakhmi C. Jain

Rights and permissions

Reprints and permissions

Copyright information

© 2005 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Poulton, G., Guo, Y., Lu, W. (2005). Finding Hidden Hierarchy in Reinforcement Learning. In: Khosla, R., Howlett, R.J., Jain, L.C. (eds) Knowledge-Based Intelligent Information and Engineering Systems. KES 2005. Lecture Notes in Computer Science(), vol 3683. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11553939_79

Download citation

  • DOI: https://doi.org/10.1007/11553939_79

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-28896-1

  • Online ISBN: 978-3-540-31990-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics