Abstract
Push recovery of a humanoid robot is a challenging task because of many different levels of control and behaviour, from walking gait to dynamic balancing. This research focuses on the active balancing and push recovery problems that allow inexpensive humanoid robots to balance while standing and walking, and to compensate for external forces. In this research, we have proposed a push recovery mechanism that employs two machine learning techniques, Reinforcement Learning and Deep Reinforcement Learning, to learn recovery step trajectories during push recovery using a closed-loop feedback control. We have implemented a 3D model using the Robot Operating System and Gazebo. To reduce wear and tear on the real robot, we used this model for learning the recovery steps for different impact strengths and directions. We evaluated our approach in both in the real world and in simulation. All the real world experiments are performed by Polaris, a teen-sized humanoid robot.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Buja, A., Stuetzle, W., Shen, Y.: Loss functions for binary class probability estimation and classification: Structure and applications. Working draft. Accessed 3 Nov 2005
Feng, S., Whitman, E., Xinjilefu, X., Atkeson, C.G.: Optimization based full body control for the atlas robot. In: 2014 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 120–127. IEEE (2014)
Gil, C.R., Calvo, H., Sossa, H.: Learning an efficient gait cycle of a biped robot based on reinforcement learning and artificial neural networks. Appl. Sci. 9(3), 502 (2019)
Google. Tensorflow (2017). URL https://www.tensorflow.org
Gu, S., Holly, E., Lillicrap, T., Levine, S.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3389–3396. IEEE (2017)
Haarnoja, T., Zhou, A., Ha, S., Tan, J., Tucker, G., Levine, S.: Learning to walk via deep reinforcement learning (2018). arXiv preprint arXiv:1812.11103
Han, J., Moraga, C.: The influence of the sigmoid function parameters on the speed of backpropagation learning. In: Mira, J., Sandoval, F. (eds.) IWANN 1995. LNCS, vol. 930, pp. 195–201. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-59497-3_175
Hosseinmemar, A.: Push recovery and active balancing for inexpensive humanoid robots. PhD thesis, Department of Computer Science, University of Manitoba, Winnipeg, Canada (2019)
Hosseinmemar, A., Baltes, J., Anderson, J., Lau, M.C., Lun, C.F., Wang, Z.: Closed-loop push recovery for an inexpensive humanoid robot. In: Mouhoub, M., Sadaoui, S., Ait Mohamed, O., Ali, M. (eds.) IEA/AIE 2018. LNCS (LNAI), vol. 10868, pp. 233–244. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92058-0_22. ISBN 978-3-319-92058-0
Hosseinmemar, A., Baltes, J., Anderson, J., Lau, M.C., Lun, C.F., Wang, Z.: Closed-loop push recovery for inexpensive humanoid robots. Appl. Intell. 49(11), 3801–3814 (2019). https://doi.org/10.1007/s10489-019-01446-z
Kim, D., Lee, J., Sentis, L.: Robust dynamic locomotion via reinforcement learning and novel whole body controller (2017). arXiv preprint arXiv:1708.02205
Kuindersma, S., et al.: Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Auton. Rob. 40(3), 429–455 (2015). https://doi.org/10.1007/s10514-015-9479-3
Kumar, A., Paul, N., Omkar, S.: Bipedal walking robot using deep deterministic policy gradient (2018). arXiv preprint arXiv:1807.05924
Morimoto, J., Cheng, G., Atkeson, C.G., Zeglin, G.: A simple reinforcement learning algorithm for biped walking. In: IEEE International Conference on Robotics and Automation, Proceedings, ICRA 2004, vol. 3, pp. 3030–3035. IEEE (2004)
Peng, X.B., Abbeel, P., Levine, S., van de Panne, M.: Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. (TOG) 37(4), 143 (2018)
Robocup. Robocup technical challenge (2018). http://www.robocuphumanoid.org/wp-content/uploads/RCHL-2018-Rules-Proposal_final.pdf
Sadeghnejad, J.S., et al.: Autman humanoid kid size team description paper (2016)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1. MIT Press, Cambridge (1998)
Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Hosseinmemar, A., Anderson, J., Baltes, J., Lau, M.C., Wang, Z. (2020). Push Recovery and Active Balancing for Inexpensive Humanoid Robots Using RL and DRL. In: Fujita, H., Fournier-Viger, P., Ali, M., Sasaki, J. (eds) Trends in Artificial Intelligence Theory and Applications. Artificial Intelligence Practices. IEA/AIE 2020. Lecture Notes in Computer Science(), vol 12144. Springer, Cham. https://doi.org/10.1007/978-3-030-55789-8_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-55789-8_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55788-1
Online ISBN: 978-3-030-55789-8
eBook Packages: Computer ScienceComputer Science (R0)