Abstract
Reinforcement learning recently shows great progress on legged robots, while bipedal robots in high dimensions but narrow solution space are still challenging to learn. The typical methods introduce the reference joints motion to guide the learning process; however, obtaining a high-quality reference trajectory is nontrivial, and imitation suffers from the local minimum. For general reference-free scenarios, the bipedal robot is discouraged by the early termination and biased sample collection. Inspired by the assistive learning commonly shown in biped animals, we introduce the assistive force to aid the learning process without the requirement of reference trajectories. The learned assistant could be a curricula to lead motor skills learning and is eliminated in the end to shape the learned motion to be plausible. We analyze the assistive system and verify its effectiveness in multiple challenging bipedal skills (Videos and supplementary materials: https://fanshi14.github.io/me/isrr22.).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Boston dynamics atlas (2022). https://www.bostondynamics.com/atlas. Accessed 01 May 2022
Kondo kxr robot (2022). https://kondo-robot.com/product-category/robot/kxrseries. Accessed 01 May 2022
Asano, Y., Okada, K., Inaba, M.: Design principles of a human mimetic humanoid: humanoid platform to study human intelligence and internal body system. Sci. Robot. 2(13), eaaq0899 (2017)
Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41–48 (2009)
Chapman, D., Daoust, T., Ormos, A., Lewis, J.: Weightshift: Accelerating animation at framestore with physics (2020)
Chen, S., Zhang, B., Mueller, M.W., Rai, A., Sreenath, K.: Learning torque control for quadrupedal locomotion. arXiv preprint arXiv:2203.05194 (2022)
Hirai, K., Hirose, M., Haikawa, Y., Takenaka, T.: The development of Honda humanoid robot. In: Proceedings. 1998 IEEE International Conference on Robotics and Automation (Cat. No. 98CH36146), vol. 2, pp. 1321–1326. IEEE (1998)
Hwangbo, J., et al.: Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4(26) (2019)
Hwangbo, J., Lee, J., Hutter, M.: Per-contact iteration method for solving contact dynamics. IEEE Robot. Autom. Lett. 3(2), 895–902 (2018)
Kim, N.H., Ling, H.Y., Xie, Z., van de Panne, M.: Flexible motion optimization with modulated assistive forces. Proc. ACM Comput. Graph. Interact. Tech. 4(3), 1–25 (2021)
Kojima, K., et al.: Development of life-sized high-power humanoid robot Jaxon for real-world use. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 838–843. IEEE (2015)
Kojio, Y., et al.: Unified balance control for biped robots including modification of footsteps with angular momentum and falling detection based on capturability. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 497–504. IEEE (2019)
Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M.: Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5(47) (2020). https://robotics.sciencemag.org/content/5/47/eabc5986
Li, Z., et al.: Reinforcement learning for robust parameterized locomotion control of bipedal robots. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2811–2817. IEEE (2021)
Miki, T., Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M.: Learning robust perceptive locomotion for quadrupedal robots in the wild. Sci. Robot. 7(62), eabk2822 (2022)
Mordatch, I., Todorov, E., Popović, Z.: Discovery of complex behaviors through contact-invariant optimization. ACM Trans. Graph. (TOG) 31(4), 1–8 (2012)
Natale, L., Bartolozzi, C., Pucci, D., Wykowska, A., Metta, G.: iCub: the not-yet-finished story of building a robot child. Sci. Robot. 2(13), eaaq1026 (2017)
Panne, M.v.d., Lamouret, A.: Guided optimization for balanced locomotion. In: Terzopoulos, D., Thalmann, D. (eds) Computer Animation and Simulation ’95, pp. 165–177. Springer, Cham (1995).https://doi.org/10.1007/978-3-7091-9435-5_13
Peng, X.B., Abbeel, P., Levine, S., van de Panne, M.: Deepmimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans. Graph. (TOG) 37(4), 1–14 (2018)
Pinto, L., Davidson, J., Sukthankar, R., Gupta, A.: Robust adversarial reinforcement learning. In: International Conference on Machine Learning, pp. 2817–2826. PMLR (2017)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)
Shi, F., Anzai, T., Kojio, Y., Okada, K., Inaba, M.: Learning agile hybrid whole-body motor skills for thruster-aided humanoid robots. In: 2021 IEEE International Conference on Intelligent Robots and Systems (IROS) (2022)
Shi, F., et al.: Circus anymal: a quadruped learning dexterous manipulation with its limbs. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2316–2323 (2021)
Siekmann, J., Godse, Y., Fern, A., Hurst, J.: Sim-to-real learning of all common bipedal gaits via periodic reward composition. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 7309–7315 (2021)
Siekmann, J., Green, K., Warila, J., Fern, A., Hurst, J.: Blind bipedal stair traversal via sim-to-real reinforcement learning. Robot. Sci. Syst. (RSS) (2021)
Siekmann, J., et al.: Learning memory-based control for human-scale bipedal locomotion. Robot. Sci. Syst. (RSS) (2020)
Starke, S., Zhao, Y., Komura, T., Zaman, K.: Local motion phases for learning multi-contact character movements. ACM Trans. Graph. (TOG) 39(4), 1–54 (2020)
Thananjeyan, B., et al.: Recovery RL: safe reinforcement learning with learned recovery zones. IEEE Robot. Autom. Lett. 6(3), 4915–4922 (2021)
Tidd, B., Hudson, N., Cosgun, A.: Guided curriculum learning for walking over complex terrain. arXiv preprint arXiv:2010.03848 (2020)
Wrotek, P., Jenkins, O.C., McGuire, M.: Dynamo: dynamic, data-driven character control with adjustable balance. In: Proceedings of the 2006 ACM SIGGRAPH symposium on Videogames, pp. 61–70 (2006)
Xie, Z., Clary, P., Dao, J., Morais, P., Hurst, J., Panne, M.: Learning locomotion skills for cassie: iterative design and sim-to-real. In: Conference on Robot Learning, pp. 317–329. PMLR (2020)
Xu, K., Verma, S., Finn, C., Levine, S.: Continual learning of control primitives: skill discovery via reset-games. Adv. Neural. Inf. Process. Syst. 33, 4999–5010 (2020)
Yang, T.Y., Zhang, T., Luu, L., Ha, S., Tan, J., Yu, W.: Safe reinforcement learning for legged locomotion. arXiv preprint arXiv:2203.02638 (2022)
Yu, W., Turk, G., Liu, C.K.: Learning symmetric and low-energy locomotion. ACM Trans. Graph. (TOG) 37(4), 1–12 (2018)
Yuan, Y., Kitani, K.: Residual force control for agile human behavior imitation and extended motion synthesis. Adv. Neural. Inf. Process. Syst. 33, 21763–21774 (2020)
Zhang, K., Hu, B., Basar, T.: On the stability and convergence of robust adversarial reinforcement learning: a case study on linear quadratic systems. Adv. Neural. Inf. Process. Syst. 33, 22056–22068 (2020)
Acknowledgment
We would like to thank Prof. Marco Hutter and Dr. Kaiqing Zhang for their insightful discussion.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shi, F. et al. (2023). Reference-Free Learning Bipedal Motor Skills via Assistive Force Curricula. In: Billard, A., Asfour, T., Khatib, O. (eds) Robotics Research. ISRR 2022. Springer Proceedings in Advanced Robotics, vol 27. Springer, Cham. https://doi.org/10.1007/978-3-031-25555-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-25555-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25554-0
Online ISBN: 978-3-031-25555-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)