Abstract
Value iteration network (VIN) improves the generalization of a policy-based neural network by embedding a planning module. However, this module performs value iteration on the entire state space of a Markov decision process and all states in the space are updated by sweeping the state space systematically, regardless of their significance. This paper introduces an improved version of VIN with a novel planning module, called asynchronous value iteration network (AVIN), performing value updates on some states more frequently than other states asynchronously, depending on their significance/urgency to improve a policy. The new planning module utilizes the urgency of the states to prioritize updates at important states. We measure the urgency in a way of enhancing the global awareness, leading to an improvement of the generalization ability of policies. AVIN with the new module makes the value updates more efficient and effective, thus significantly demonstrating better generalization on unknown environments.
Z. Pan, Z. Zhang and Z. Chen—Contributed equally to this work.
This work is in part supported by the National Natural Science Foundation of China under Grant Nos. 61876119 and 61502323.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barto, A.G., Bradtke, S.J., Singh, S.P.: Learning to act using real-time dynamic programming. Artif. Intell. 72(1–2), 81–138 (1995)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn. Athena Scientific, Belmont (2005)
Browne, C., et al.: A survey of Monte Carlo tree search methods. IEEE Trans. Comput. Intell. AI Games 4(1), 1–43 (2012)
Dai, J., et al.: Deformable convolutional networks. CoRR abs/1703.06211 (2017). http://arxiv.org/abs/1703.06211
Karkus, P., Hsu, D., Lee, W.S.: QMDP-Net: deep learning for planning under partial observability. In: NIPS, pp. 4697–4707 (2017)
Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR, pp. 3431–3440 (2015)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Moore, A.W., Atkeson, C.G.: Prioritized sweeping: reinforcement learning with less data and less time. Mach. Learn. 13, 103–130 (1993)
Niu, S., Chen, S., Guo, H., Targonski, C., Smith, M.C., Kovacevic, J.: Generalized value iteration networks: life beyond lattices. In: AAAI, pp. 6246–6253 (2018)
Oh, J., Singh, S., Lee, H.: Value prediction network. In: NIPS, pp. 6120–6130 (2017)
Ross, S., Gordon, G.J., Bagnell, D.: A reduction of imitation learning and structured prediction to no-regret online learning. In: AISTATS, pp. 627–635 (2011)
Schaal, S.: Is imitation learning the route to humanoid robots? Trends Cogn. Sci. 3(6), 233–242 (1999)
Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized experience replay. CoRR abs/1511.05952 (2016). http://arxiv.org/abs/1511.05952
Sutton, R.S.: Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In: ICML, pp. 216–224 (1990)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning. MIT Press, Cambridge (1998)
Sutton, R.S., Precup, D., Singh, S.P.: Between MDPs and Semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112(1–2), 181–211 (1999)
Tamar, A., Levine, S., Abbeel, P., Wu, Y., Thomas, G.: Value iteration networks. In: NIPS, pp. 2146–2154 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Pan, Z., Zhang, Z., Chen, Z. (2018). Asynchronous Value Iteration Network. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11302. Springer, Cham. https://doi.org/10.1007/978-3-030-04179-3_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-04179-3_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04178-6
Online ISBN: 978-3-030-04179-3
eBook Packages: Computer ScienceComputer Science (R0)