Abstract
The MineRL competition is designed for the development of reinforcement learning and imitation learning algorithms that can efficiently leverage human demonstrations to drastically reduce the number of environment interactions needed to solve the complex ObtainDiamond task with sparse rewards. To address the challenge, in this paper, we present SEIHAI, a Sample-efficient Hierarchical AI, that fully takes advantage of the human demonstrations and the task structure. Specifically, we split the task into several sequentially dependent subtasks, and train a suitable agent for each subtask using reinforcement learning and imitation learning. We further design a scheduler to select different agents for different subtasks automatically. SEIHAI takes the first place in the preliminary and final of the NeurIPS-2020 MineRL competition.
H. Mao, C. Wang, X. Hao, Y. Mao, Y. Lu, C. Wu—These authors contribute equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The competition requires that the submitted methods should not include any meta-actions or rule-based heuristics.
- 2.
Specifically, over 60 million frames of human demonstrations and 8 million online interactions with the environment.
- 3.
Most human demonstrations are scattered among several small environments, like Navigation, Treechop.
- 4.
For example, we test deep ResNet [10], but the performance has not improved much, while the training time has increased a lot.
- 5.
We also test the Generative Adversarial Imitation Learning (GAIL) algorithm [12], but it is not as stable as SQIL.
- 6.
This is different from the Q-value in reinforcement learning, since here Q(s, a) does not estimate the expected cumulative rewards.
- 7.
Note that the minimum number is learned from the demonstrations, and this is allowed by the competition organizers.
- 8.
References
Amiranashvili, A., Dorka, N., Burgard, W., Koltun, V., Brox, T.: Scaling imitation learning in minecraft. arXiv preprint arXiv:2007.02701 (2020)
Blei, D.M., Griffiths, T.L., Jordan, M.I.: The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J. ACM (JACM) 57(2), 1–30 (2010)
Fujimoto, S., Meger, D., Precup, D.: Off-policy deep reinforcement learning without exploration. In: International Conference on Machine Learning, pp. 2052–2062. PMLR (2019)
Gu, S., Holly, E., Lillicrap, T., Levine, S.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3389–3396. IEEE (2017)
Guss, W.H., et al.: The MineRL 2020 competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:2101.11071 (2021)
Guss, W.H., et al.: NeurIPS 2019 competition: the MineRL competition on sample efficient reinforcement learning using human priors. arXiv preprint arXiv:1904.10079 (2019)
Guss, W.H., et al.: The MineRL competition on sample efficient reinforcement learning using human priors. arXiv e-prints (2019)
Guss, W.H., et al.: MineRL: a large-scale dataset of minecraft demonstrations. arXiv preprint arXiv:1907.13440 (2019)
Guss, W.H., et al.: Towards robust and domain agnostic reinforcement learning competitions: MineRL 2020. In: NeurIPS 2020 Competition and Demonstration Track, pp. 233–252. PMLR (2021)
He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_38
Hester, T., et al.: Deep Q-learning from demonstrations. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Ho, J., Ermon, S.: Generative adversarial imitation learning. Adv. Neural Inf. Process. Syst. 29, 4565–4573 (2016)
Kang, B., Jie, Z., Feng, J.: Policy optimization with demonstrations. In: International Conference on Machine Learning, pp. 2469–2478. PMLR (2018)
Kumar, A., Fu, J., Tucker, G., Levine, S.: Stabilizing off-policy Q-learning via bootstrapping error reduction. arXiv preprint arXiv:1906.00949 (2019)
Kumar, A., Zhou, A., Tucker, G., Levine, S.: Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779 (2020)
Likas, A., Vlassis, N., Verbeek, J.J.: The global k-means clustering algorithm. Pattern Recogn. 36(2), 451–461 (2003)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. In: ICLR (2016)
Lloyd, S.: Least squares quantization in PCM. IEEE Trans. Inf. Theor. 28(2), 129–137 (1982)
MacQueen, J., et al.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, pp. 281–297. Oakland, CA, USA (1967)
Mao, H., Gong, Z., Ni, Y., Xiao, Z.: ACCNet: actor-coordinator-critic net for “learning-to-communicate” with deep multi-agent reinforcement learning. arXiv preprint arXiv:1706.03235 (2017)
Mao, H., et al.: Neighborhood cognition consistent multi-agent reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7219–7226 (2020)
Mao, H., Zhang, Z., Xiao, Z., Gong, Z.: Modelling the dynamic joint policy of teammates with attention multi-agent DDPG. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems (2019)
Mao, H., Zhang, Z., Xiao, Z., Gong, Z., Ni, Y.: Learning agent communication under limited bandwidth by message pruning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5142–5149 (2020)
Mao, H., Zhang, Z., Xiao, Z., Gong, Z., Ni, Y.: Learning multi-agent communication with double attentional deep reinforcement learning. Auton. Agents Multi Agent Syst. 34(1), 1–34 (2020). https://doi.org/10.1007/s10458-020-09455-w
Milani, S., et al.: Retrospective analysis of the 2019 MineRL competition on sample efficient reinforcement learning. In: NeurIPS 2019 Competition and Demonstration Track, pp. 203–214. PMLR (2020)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J.A., Abbeel, P., Peters, J., et al.: An algorithmic perspective on imitation learning. Found. Trends Rob. 7(1–2), 1–179 (2018)
Reddy, S., Dragan, A.D., Levine, S.: SQIL: imitation learning via reinforcement learning with sparse rewards. In: ICLR (2019)
Scheller, C., Schraner, Y., Vogel, M.: Sample efficient reinforcement learning through learning from demonstrations in minecraft. In: NeurIPS 2019 Competition and Demonstration Track, pp. 67–76. PMLR (2020)
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., Riedmiller, M.: Deterministic policy gradient algorithms. In: ICML, pp. 387–395. PMLR (2014)
Skrynnik, A., Staroverov, A., Aitygulov, E., Aksenov, K., Davydov, V., Panov, A.I.: Hierarchical deep q-network from imperfect demonstrations in minecraft. Cogn. Syst. Res. 65, 74–78 (2021)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Vecerik, M., et al.: Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08817 (2017)
Acknowledgement
The authors would like to thank Mengchen Zhao, Weixun Wang, Rundong Wang, Shixun Wu, Zhanbo Feng and the anonymous reviewers for their comments.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Mao, H. et al. (2022). SEIHAI: A Sample-Efficient Hierarchical AI for the MineRL Competition. In: Chen, J., Lang, J., Amato, C., Zhao, D. (eds) Distributed Artificial Intelligence. DAI 2021. Lecture Notes in Computer Science(), vol 13170. Springer, Cham. https://doi.org/10.1007/978-3-030-94662-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-94662-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94661-6
Online ISBN: 978-3-030-94662-3
eBook Packages: Computer ScienceComputer Science (R0)