Abstract
Locomotion control of legged robots is a challenging problem. Recently, reinforcement learning has been applied to legged locomotion and made a great success. However, the reward signal design remains a challenging problem to produce a humanlike motion such as walking and running. Although imitation learning provides a way to mimic the behavior of humans or animals, the obtained motion may be restricted due to the over-constrained property of this method. Here we propose a novel and simple way to generate humanlike behavior by using feedforward enhanced reinforcement learning (FERL). In FERL, the control action is composed of a feedforward part and a feedback part, where the feedforward part is a periodic time-dependent signal generated by a state machine and the feedback part is a state-dependent signal obtained by a neural network. By using FERL with a simple feedforward of two feet stepping up and down alternately, we achieve humanlike walking and running for a simulated biped robot, Ranger Max. Comparison results show that the feedforward is key to generating humanlike behavior, while the policy trained with no feedforward only results in some strange gaits. FERL may also be extended to other legged robots to generate various locomotion styles, which provides a competitive alternative for imitation learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Heess, N., et al.: Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017)
Hwangbo, J., et al.: Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4(26), eaau5872 (2019)
Xie, Z., Berseth, G., Clary, P., Hurst, J., van de Panne, M.: Feedback control for cassie with deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1241–1246. IEEE (2018)
Li, Z., et al.: Reinforcement learning for robust parameterized locomotion control of bipedal robots. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2811–2817. IEEE (2021)
Peng, X.B., Coumans, E., Zhang, T., Lee, T.W., Tan, J., Levine, S.: Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784 (2020)
Shao, Y., Jin, Y., Liu, X., He, W., Wang, H., Yang, W.: Learning free gait transition for quadruped robots via phase-guided controller. IEEE Robot. Autom. Let. 7(2), 1230–1237 (2021)
Wu, Q., Zhang, C., Liu, Y.: Custom sine waves are enough for imitation learning of bipedal gaits with different styles. In: 2022 IEEE International Conference on Mechatronics and Automation (ICMA), pp. 499–505. IEEE (2022)
Siekmann, J., Godse, Y., Fern, A., Hurst, J.: Sim-to-real learning of all common bipedal gaits via periodic reward composition. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 7309–7315. IEEE (2021)
Siekmann, J., Green, K., Warila, J., Fern, A., Hurst, J.: Blind bipedal stair traversal via sim-to-real reinforcement learning. arXiv preprint arXiv:2105.08328 (2021)
Bellegarda, G., Chen, Y., Liu, Z., Nguyen, Q.: Robust high-speed running for quadruped robots via deep reinforcement learning. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10364–10370. IEEE (2022)
Vollenweider, E., Bjelonic, M., Klemm, V., Rudin, N., Lee, J., Hutter, M.: Advanced skills through multiple adversarial motion priors in reinforcement learning. arXiv preprint arXiv:2203.14912 (2022)
Escontrela, A., et al.: Adversarial motion priors make good substitutes for complex reward functions. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 25–32. IEEE (2022)
Li, T., Geyer, H., Atkeson, C.G., Rai, A.: Using deep reinforcement learning to learn high-level policies on the atrias biped. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 263–269. IEEE (2019)
Tan, W., et al.: A hierarchical framework for quadruped locomotion based on reinforcement learning. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8462–8468. IEEE (2021)
Castillo, G. A., Weng, B., Zhang, W., Hereid, A.: Robust feedback motion policy design using reinforcement learning on a 3d digit bipedal robot. In: 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5136–5143. IEEE (2021)
Wang, Z., Wei, W., Xie, A., Zhang, Y., Wu, J., Zhu, Q.: Hybrid bipedal locomotion based on reinforcement learning and heuristics. Micromachines 13(10), 1688 (2022)
Bhounsule, P.A., et al.: Low-bandwidth reflex-based control for lower power walking: 65 km on a single battery charge. Int. J. Robot. Res. 33(10), 1305–1321 (2014)
Acknowledgment
This work was supported by the National Natural Science Foundation of China No. 62003188 and No.92248304.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Ye, L., Wang, X., Liang, B. (2023). Realizing Human-like Walking and Running with Feedforward Enhanced Reinforcement Learning. In: Yang, H., et al. Intelligent Robotics and Applications. ICIRA 2023. Lecture Notes in Computer Science(), vol 14270. Springer, Singapore. https://doi.org/10.1007/978-981-99-6492-5_38
Download citation
DOI: https://doi.org/10.1007/978-981-99-6492-5_38
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-6491-8
Online ISBN: 978-981-99-6492-5
eBook Packages: Computer ScienceComputer Science (R0)