Skip to main content
SpringerLink
Log in

Uncertainty-Aware Hierarchical Reinforcement Learning Robust to Noisy Observations

  • Conference paper
  • First Online:
Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1 (FTC 2022 2022)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 559))

Included in the following conference series:

Abstract

This work proposes UA-HRL, an uncertainty-aware hierarchical reinforcement learning framework for mitigating the problems caused by noisy sensor data. The system is composed of an ensemble of predictive models that learns the environment’s underlying dynamics and estimates the uncertainty through their prediction variances and a two-level Hierarchical Reinforcement Learning agent that integrates the uncertainty estimates into the decision-making process. It is also shown how frame-stacking can be combined with the uncertainty estimation for the agent to make better decisions despite the aleatoric noise present in the observations. In the end, results obtained in a simulation environment are presented and discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abdar, M., et al.: A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243–297 (2021)

    Article  Google Scholar 

  2. Bacon, P.-L., Harb, J., Precup, D.: The option-critic architecture. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  3. Badre, D., Hoffman, J., Cooney, J.W., D’esposito, M.: Hierarchical cognitive control deficits following damage to the human frontal lobe. Nat. Neurosci. 12(4), 515–522 (2009)

    Article  Google Scholar 

  4. Botvinick, M., Ritter, S., Wang, J.X., Kurth-Nelson, Z., Blundell, C., Hassabis, D.: Reinforcement learning, fast and slow. Trends Cogn. Sci. 23(5), 408–422 (2019)

    Article  Google Scholar 

  5. Botvinick, M.M., Niv, Y., Barto, A.G.: Hierarchically organized behavior and its neural foundations: a reinforcement learning perspective. Cognition 113(3), 262–280 (2009)

    Article  Google Scholar 

  6. Botvinick, M.M.: Hierarchical reinforcement learning and decision making. Curr. Opinion Neurobiol. 22(6), 956–962 (2012)

    Article  Google Scholar 

  7. Fort, S., Hu, H., Lakshminarayanan, B.: Deep ensembles: a loss landscape perspective. arXiv preprint arXiv:1912.02757 (2019)

  8. Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059. PMLR (2016)

    Google Scholar 

  9. Haider, T., Roza, F.S., Eilers, D., Roscher, K., Günnemann, S.: Domain shifts in reinforcement learning: identifying disturbances in environments (2021)

    Google Scholar 

  10. Henne, M., Schwaiger, A., Roscher, K., Weiss, G.: Benchmarking uncertainty estimation methods for deep learning with safety-related metrics. In: SafeAI@ AAAI, pp. 83–90 (2020)

    Google Scholar 

  11. Henne, M., Schwaiger, A., Weiss, G.: Managing uncertainty of AI-based perception for autonomous systems. In: AISafety@ IJCAI (2019)

    Google Scholar 

  12. Hoel, C.-J., Wolff, K., Laine, L.: Tactical decision-making in autonomous driving by reinforcement learning with uncertainty estimation. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1563–1569. IEEE (2020)

    Google Scholar 

  13. Jong, N.K., Hester, T., Stone, P.: The utility of temporal abstraction in reinforcement learning. In: AAMAS (1), pp. 299–306. Citeseer (2008)

    Google Scholar 

  14. Kahn, G., Villaflor, A., Pong, V., Abbeel, P., Levine, S.: Uncertainty-aware reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01182 (2017)

  15. Kulkarni, T.D., Narasimhan, K., Saeedi, A., Tenenbaum, J.: Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  16. Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J.E., Stoica, I.: Tune: a research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 (2018)

  17. Lütjens, B., Everett, M., How, J.P.: Safe reinforcement learning with model uncertainty estimates. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8662–8668. IEEE (2019)

    Google Scholar 

  18. Mnih, V., et al.: Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013)

  19. Nachum, O., Gu, S.S., Lee, H., Levine, S.: Data-efficient hierarchical reinforcement learning. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  20. Pertsch, K., Lee, Y., Lim, J.J.: Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010.11944 (2020)

  21. Ribas-Fernandes, J.J.F., et al.: A neural signature of hierarchical reinforcement learningd. Neuron 71(2), 370–379 (2011)

    Article  Google Scholar 

  22. Schrittwieser, J., et al.: Mastering Atari, go, chess and shogi by planning with a learned model. Nature 588(7839), 604–609 (2020)

    Article  Google Scholar 

  23. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

  24. Schwaiger, A., Sinhamahapatra, P., Gansloser, J., Roscher, K.: Is uncertainty quantification in deep learning sufficient for out-of-distribution detection? In: AISafety@ IJCAI (2020)

    Google Scholar 

  25. Schwaiger, F., et al.: From black-box to white-box: examining confidence calibration under different conditions. arXiv preprint arXiv:2101.02971 (2021)

  26. Sedlmeier, A., Gabor, T., Phan, T., Belzner, L., Linnhoff-Popien, C.: Uncertainty-based out-of-distribution detection in deep reinforcement learning. arXiv preprint arXiv:1901.02219 (2019)

  27. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  Google Scholar 

  28. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

  29. Sutton, R.S., Precup, D., Singh, S.: Between MDPS and semi-MDPS: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112(1–2), 181–211 (1999)

    Article  MathSciNet  Google Scholar 

  30. Van Amersfoort, J., Smith, L., Teh, Y.W., Gal, Y.: Uncertainty estimation using a single deep deterministic neural network. In: International Conference on Machine Learning, pp. 9690–9700. PMLR (2020)

    Google Scholar 

  31. Vezhnevets, A.S., et al.: Feudal networks for hierarchical reinforcement learning. In: International Conference on Machine Learning, pp. 3540–3549. PMLR (2017)

    Google Scholar 

  32. Yang, Z., Merrick, K., Jin, L., Abbass, H.A.: Hierarchical deep reinforcement learning for continuous action control. IEEE Trans. Neural Netw. Learn. Syst. 29(11), 5174–5184 (2018)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

This work was funded by the Bavarian Ministry for Economic Affairs, Regional Development and Energy as part of a project to support the thematic development of the Institute for Cognitive Systems.

Author information

Authors and Affiliations

  1. Fraunhofer IKS, Munich, Germany

    Felippe Schmoeller Roza

Authors
  1. Felippe Schmoeller Roza
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felippe Schmoeller Roza .

Editor information

Editors and Affiliations

  1. Faculty of Science and Engineering, Saga University, Saga, Japan

    Kohei Arai

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Roza, F.S. (2023). Uncertainty-Aware Hierarchical Reinforcement Learning Robust to Noisy Observations. In: Arai, K. (eds) Proceedings of the Future Technologies Conference (FTC) 2022, Volume 1. FTC 2022 2022. Lecture Notes in Networks and Systems, vol 559. Springer, Cham. https://doi.org/10.1007/978-3-031-18461-1_35

Download citation

Publish with us

Policies and ethics

Search

Navigation