Abstract
Planning under partial obervability is essential for autonomous robots. A principled way to address such planning problems is the Partially Observable Markov Decision Process (POMDP). Although solving POMDPs is computationally intractable, substantial advancements have been achieved in developing approximate POMDP solvers in the past two decades. However, computing robust solutions for systems with complex dynamics remains challenging. Most on-line solvers rely on a large number of forward-simulations and standard Monte-Carlo methods to compute the expected outcomes of actions the robot can perform. For systems with complex dynamics, e.g., those with non-linear dynamics that admit no closed form solution, even a single forward simulation can be prohibitively expensive. Of course, this issue exacerbates for problems with long planning horizons. This paper aims to alleviate the above difficulty. To this end, we propose a new on-line POMDP solver, called Multilevel POMDP Planner (MLPP), that combines the commonly known Monte-Carlo-Tree-Search with the concept of Multilevel Monte-Carlo to speed-up our capability in generating approximately optimal solutions for POMDPs with complex dynamics. Experiments on four different problems of POMDP-based torque control, navigation and grasping indicate that MLPP substantially outperforms state-of-the-art POMDP solvers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agha-Mohammadi, A.A., Chakravorty, S., Amato, N.M.: FIRM: feedback controller-based information-state roadmap-a framework for motion planning under uncertainty. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4284–4291. IEEE (2011)
Anderson, D.F., Higham, D.J.: Multilevel Monte Carlo for continuous time Markov chains, with applications in biochemical kinetics. Multiscale Model. Simul. 10(1), 146–179 (2012)
Arulampalam, M.S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking. IEEE Trans. Signal Process. 50(2), 174–188 (2002)
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2–3), 235–256 (2002)
Bai, H., Hsu, D.: Unmanned aircraft collision avoidance using continuous-state POMDPs. In: Robotics: Science and Systems VII, vol. 1, pp. 1–8 (2012)
Bai, H., Hsu, D., Lee, W.S.: Integrated perception and planning in the continuous space: a POMDP approach. Int. J. Robot. Res. 33(9), 1288–1302 (2014)
Bierig, C., Chernov, A.: Approximation of probability density functions by the multilevel Monte Carlo maximum entropy method. J. Comput. Phys. 314, 661–681 (2016)
Giles, M.B.: Multilevel Monte Carlo path simulation. Oper. Res. 56(3), 607–617 (2008)
Giles, M.B.: Multilevel Monte Carlo methods. Acta Numer. 24, 259–328 (2015)
He, R., Brunskill, E., Roy, N.: PUMA: planning under uncertainty with macro-actions. In: Proceedings of the National Conference on Artificial Intelligence, vol. 2 (2010)
Heinrich, S.: Multilevel Monte Carlo methods. In: Margenov, S., Waśniewski, J., Yalamov, P. (eds.) Large-Scale Scientific Computing. LNCS, pp. 58–67. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45346-6_5
Hoerger, M., Kurniawati, H., Elfes, A.: A software framework for planning under partial observability. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–9. IEEE (2018)
Hoey, J., Poupart, P.: Solving POMDPs with continuous or large discrete observation spaces. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI 2005, pp. 1332–1338. Morgan Kaufmann Publishers Inc., San Francisco (2005)
Horowitz, M., Burdick, J.: Interactive non-prehensile manipulation for grasping via POMDPs. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 3257–3264. IEEE (2013)
Hsiao, K., Kaelbling, L.P., Lozano-Perez, T.: Grasping POMDPs. In: 2007 IEEE International Conference on Robotics and Automation, pp. 4685–4692. IEEE (2007)
Klimenko, D., Song, J., Kurniawati, H.: TAPIR: a software toolkit for approximating and adapting POMDP solutions online. In: Proceedings of the Australasian Conference on Robotics and Automation (2014)
Kocsis, L., Szepesvári, C.: Bandit based Monte-Carlo planning. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS, vol. 4212, pp. 282–293. Springer, Heidelberg (2006). https://doi.org/10.1007/11871842_29
Kurniawati, H., Du, Y., Hsu, D., Lee, W.S.: Motion planning under uncertainty for robotic tasks with long time horizons. Int. J. Robot. Res. 30(3), 308–323 (2011)
Kurniawati, H., Hsu, D., Lee, W.S.: SARSOP: efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: Proceedings of the Robotics: Science and Systems (2008)
Kurniawati, H., Yadav, V.: An online POMDP solver for uncertainty planning in dynamic environment. In: Proceedings of the International Symposium on Robotics Research (2013)
Luo, Y., Bai, H., Hsu, D., Lee, W.S.: Importance sampling for online planning under uncertainty. Int. J. Robot. Res. 38, 162–181 (2018). p. 0278364918780322
Owen, A.B.: Monte Carlo Theory, Methods and Examples (2013)
Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of Markov decision processes. Math. Oper. Res. 12(3), 441–450 (1987)
Pineau, J., Gordon, G., Thrun, S.: Point-based value iteration: an anytime algorithm for POMDPs (2003)
Rhee, C., Glynn, P.W.: A new approach to unbiased estimation for SDE’s. In: Proceedings of the Winter Simulation Conference, p. 17. Winter Simulation Conference (2012)
Seiler, K.M., Kurniawati, H., Singh, S.P.: An online and approximate solver for POMDPs with continuous action space. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2290–2297. IEEE (2015)
Silver, D., Veness, J.: Monte-Carlo planning in large POMDPs. In: Advances in Neural Information Processing Systems, pp. 2164–2172 (2010)
Smith, R.: Open dynamics engine. http://www.ode.org/
Smith, T., Simmons, R.: Point-based POMDP algorithms: improved analysis and implementation (2005)
Somani, A., Ye, N., Hsu, D., Lee, W.S.: DESPOT: Online POMDP planning with regularization. In: Advances In Neural Information Processing Systems, pp. 1772–1780 (2013)
Sondik, E.J.: The optimal control of partially observable Markov decision processes. Ph.D. thesis, Stanford, California (1971)
Spong, M.W., Hutchinson, S., Vidyasagar, M.: Robot Modeling and Control, vol. 3. Wiley, New York (2006)
Sunberg, Z.N., Kochenderfer, M.J.: Online algorithms for POMDPs with continuous state, action, and observation spaces. In: Twenty-Eighth International Conference on Automated Planning and Scheduling (2018)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2012)
Wang, E., Kurniawati, H., Kroese, D.P.: An on-line planner for POMDPs with large discrete action space: a quantile-based approach. In: ICAPS, pp. 273–277. AAAI Press (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hoerger, M., Kurniawati, H., Elfes, A. (2022). Multilevel Monte-Carlo for Solving POMDPs Online. In: Asfour, T., Yoshida, E., Park, J., Christensen, H., Khatib, O. (eds) Robotics Research. ISRR 2019. Springer Proceedings in Advanced Robotics, vol 20. Springer, Cham. https://doi.org/10.1007/978-3-030-95459-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-95459-8_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95458-1
Online ISBN: 978-3-030-95459-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)