Abstract
Partially Observable Markov Decision Process (POMDP) provides a probabilistic model for decision making under uncertainty. Point-based value iteration algorithms are effective approximate algorithms to solve POMDP problems. Belief selection is a key step of point-based algorithm. In this paper we provide a belief selection method based on the uncertainty of belief point. The algorithm first computes the uncertainties of the belief points that could be reached, and then selects the belief points that have lower uncertainties and whose distances to the current belief set are larger than a threshold. The experimental results indicate that this method is effective to gain an approximate long-term discounted reward using fewer belief states than the other point-based algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Sondik, E.J.: The optimal control of partially observable Markov processes over the infinite horizon: Discounted costs. Operations Research 26(2), 282–304 (1978)
Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101, 99–134 (1998)
Hu, Q.Y., Liu, J.Y.: An Introduction to Markov Decision Processes. Xi Dian University Press, Xi’an (2000) (in Chinese)
Zhang, N.L., Zhang, W.: Speeding up the Convergence of Value Iteration in Partially Observable Markov Decision Processes. Journal of Artificial Intelligence Research (JAIR) 14, 29–51 (2001)
Pineau, J., Gordon, G., Thrun, S.: Point-based value iteration: An anytime algorithm for POMDPs. In: Int. Joint Conf. on Artificial Intelligence (IJCAI), Acapulco, Mexico, pp. 1025–1030 (2003)
Smith, T., Simmons, R.: Heuristic search value iteration for POMDPs. In: Proc. of Uncertainty in Artificial Intelligence (UAI) (2004)
Spaan, M.T.J., Vlassis, N.: Perseus: Randomized point-based value iteration for POMDPs. Journal of Artificial Intelligence Research (JAIR) 24, 195–220 (2005)
Izadi, M.T., Precup, D., Azar, D.: Belief selection in point-based planning algorithms for POMDPs. In: Proceedings of Canadian Conference on Artificial Intelligence (AI), Quebec City, Canada, pp. 383–394 (2006)
Izadi, M.T., Precup, D.: Exploration in POMDP belief space and its impact on value iteration approximation. In: European Conference on Artificial Intelligence (ECAI), Riva del Garda, Italy (2006)
Shani, G., Brafman, R.I., Shimony, S.E.: Forward search value iteration for POMDPs. In: Proc. Int. Joint Conf. on Artificial Intelligence (IJCAI), pp. 2619–2624 (2007)
Pineau, J., Gordon, G., Thrun, S.: Point-based approximations for fast POMDP solving. Technical Report, SOCS-TR-2005.4, School of Computer Science, McGill University (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Feng, Q., Zhou, X., Huang, H., Zhang, X. (2009). An Uncertainty-Based Belief Selection Method for POMDP Value Iteration. In: Sossai, C., Chemello, G. (eds) Symbolic and Quantitative Approaches to Reasoning with Uncertainty. ECSQARU 2009. Lecture Notes in Computer Science(), vol 5590. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02906-6_72
Download citation
DOI: https://doi.org/10.1007/978-3-642-02906-6_72
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02905-9
Online ISBN: 978-3-642-02906-6
eBook Packages: Computer ScienceComputer Science (R0)