Abstract
Recently, the optimization-by-inference approach has been proposed as a new means for solving high-dimensional optimization problems quickly. Approximate Inference COntrol (AICO) is one of the most successful and promising methods that implement the optimization-by-inference approach. AICO is able to solve stochastic optimal control problems and has already been successfully used in many applications. However, it is known that the iterative inference of AICO sometimes fails to converge to the optimal solution. To make the optimization more robust, in this paper, we propose to take model uncertainty into account. In AICO, the cost function to be minimized is accurate around a particular state of a given stochastic system, but the accuracy is uncertain in regions far from that state. Because using such an uncertain function is harmful to the convergence, we modify AICO, so that it does not use the function in uncertain regions. Our method is easy to implement and does not add much computational time to the original AICO. Experiments using two different scenarios show that our method substantially improves AICO in terms of the rate at which the algorithm produces convergent results.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Attias H (2003) Planning by probabilistic inference. In: Proceedings of the ninth international workshop on artificial intelligence and statistics
Verma D, Rao RP (2005) Goal-based imitation as probabilistic inference over graphical models. In: Advances in neural information processing systems, pp 1393–1400
Toussaint M, Storkey A (2006) Probabilistic inference for solving discrete and continuous state Markov decision processes. In: Proceedings of the 23rd international conference on machine learning, pp 945–952
Toussaint M (2009) Robot trajectory optimization using approximate inference. In: Proceedings of the 26th international conference on machine learning, pp 1049–1056
Kappen HJ, Gómez V, Opper M (2012) Optimal control as a graphical model inference problem. Mach Learn 87(2):159–182
Kumar A, Zilberstein S, Toussaint M (2015) Probabilistic inference techniques for scalable multiagent decision making. J Artif Intell Res 53(1):223–270
Minka TP (2001), Expectation propagation for approximate Bayesian inference. In: Proceedings of the 17th conference on uncertainty in artificial intelligence, pp 362–369
Rawlik K, Toussaint M, Vijayakumar S (2010), An approximate inference approach to temporal optimization in optimal control. In: Advances in neural information processing systems, pp 2011–2019
Jetchev N, Toussaint M (2013) Fast motion planning from experience: trajectory prediction for speeding up movement generation. Auton Robots 34(1–2):111–127
Ivan V, Zarubin D, Toussaint M, Komura T, Vijayakumar S (2013) Topology-based representations for motion planning and generalization in dynamic environments with interactions. Int J Robot Res 32(9–10):1151–1163
Zarubin D, Pokorny FT, Song D, Toussaint M, Kragic D (2013) Topological synergies for grasp transfer. In: Hand synergies-how to tame the complexity of grapsing, workshop, IEEE international conference on robotics and automation
Kadoya T, Itoh H, Fukumoto H, Wakuya H, Furukawa T (2014) Movement imitation in a humanoid robot with approximate inference control. In: Proceedings of the 19th international symposium on artificial life and robotics, pp 260–263
Watter M, Springenberg J, Boedecker J, Riedmiller M (2015) Embed to control: a locally linear latent dynamics model for control from raw images. In: Advances in neural information processing systems, pp 2728–2736
Toussaint M (2009) Pros and cons of truncated Gaussian EP in the context of approximate inference control. NIPS workshop on probabilistic approaches for robotics and control
Rüeckert E, Mindt M, Peters J, Neumann G (2014) Robust policy updates for stochastic optimal control. In: Proceedings of the 14th IEEE-RAS international conference on humanoid robots (humanoids), pp 388–393
Zarubin D, Ivan V, Toussaint M, Komura T, Vijayakumar S (2012) Hierarchical motion planning in topological representations. In: International conference on robotics science and systems
Rawlik K, Toussaint M, Vijayakumar S (2012) On stochastic optimal control and reinforcement learning by approximate inference. In: International conference on robotics science and systems
Acknowledgements
We would like to thank the reviewers for their valuable comments. This study was partially supported by the Ministry of Education, Culture, Sports, Science and Technology in Japan, Grant-in-Aid for Scientific Research (C) 15K00341.
Author information
Authors and Affiliations
Corresponding author
Additional information
This work was presented in part at the 19th International Symposium on Artificial Life and Robotics, Beppu, Oita, January 22–24, 2014.
About this article
Cite this article
Itoh, H., Sakai, Y., Kadoya, T. et al. Using model uncertainty for robust optimization in approximate inference control. Artif Life Robotics 22, 327–335 (2017). https://doi.org/10.1007/s10015-017-0361-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10015-017-0361-6