Skip to main content
Springer Nature Link
Log in

GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model

  • Conference paper
  • First Online:
Algorithmic Foundations of Robotics XV (WAFR 2022)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 25))

Included in the following conference series:

Abstract

Model-free reinforcement learning (RL) for legged locomotion commonly relies on a physics simulator that can accurately predict the behaviors of every degree of freedom of the robot. In contrast, approximate reduced-order models are commonly used for many model predictive control strategies. In this work we abandon the conventional use of high-fidelity dynamics models in RL and we instead seek to understand what can be achieved when using RL with a much simpler centroidal model when applied to quadrupedal locomotion. We show that RL-based control of the accelerations of a centroidal model is surprisingly effective, when combined with a quadratic program to realize the commanded actions via ground contact forces. It allows for a simple reward structure, reduced computational costs, and robust sim-to-real transfer. We show the generality of the method by demonstrating flat-terrain gaits, stepping-stone locomotion, two-legged in-place balance, balance beam locomotion, and direct sim-to-real transfer.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Laikago and A1 are quadrupedal robots made by Unitree Robotics.

References

  1. Agrawal, A., Chen, S., Rai, A., Sreenath, K.: Vision-aided dynamic quadrupedal locomotion on discrete terrain using motion libraries. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 4708–4714. IEEE (2022)

    Google Scholar 

  2. Allshire, A., Martín-Martín, R., Lin, C., Manuel, S., Savarese, S., Garg, A.: Laser: learning a latent action space for efficient reinforcement learning. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2021)

    Google Scholar 

  3. Berseth, G., Xie, C., Cernek, P., Van de Panne, M.: Progressive reinforcement learning with distillation for multi-skilled motion control (2018). arXiv:1802.04765

  4. Bharadhwaj, H., Kumar, A., Rhinehart, N., Levine, S., Shkurti, F., Garg, A.: Conservative safety critics for exploration (2020). arXiv:2010.14497

  5. Bledt, G., Kim, S.: Extracting legged locomotion heuristics with regularized predictive control. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 406–412. IEEE (2020)

    Google Scholar 

  6. Bledt, G., Powell, M.J., Katz, B., Di Carlo, J., Wensing, P.M., Kim, S.: Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

    Google Scholar 

  7. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press (2004)

    Google Scholar 

  8. Carius, J., Farshidian, F., Hutter, M.: Mpc-net: a first principles guided policy search. IEEE Robot. Autom. Lett. 5(2), 2897–2904 (2020)

    Article  Google Scholar 

  9. Da, X., Grizzle, J.: Combining trajectory optimization, supervised machine learning, and model structure for mitigating the curse of dimensionality in the control of bipedal robots. Int. J. Robot. Res. 38(9) (2019)

    Google Scholar 

  10. Da, X., Xie, Z., Hoeller, D., Boots, B., Anandkumar, A., Zhu, Y., Babich, B., Garg, A.: Learning a contact-adaptive controller for robust, efficient legged locomotion (2020). arXiv:2009.10019

  11. Di Carlo, J., Wensing, P.M., Katz, B., Bledt, G., Kim, S.: Dynamic locomotion in the mit cheetah 3 through convex model-predictive control. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE (2018)

    Google Scholar 

  12. Ding, Y., Pandala, A., Park, H.W.: Real-time model predictive control for versatile dynamic motions in quadrupedal robots. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 8484–8490. IEEE (2019)

    Google Scholar 

  13. Duan, H., Dao, J., Green, K., Apgar, T., Fern, A., Hurst, J.: Learning task space actions for bipedal locomotion. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1276–1282. IEEE (2021)

    Google Scholar 

  14. Feng, S., Xinjilefu, X., Atkeson, C.G., Kim, J.: Optimization based controller design and implementation for the atlas robot in the darpa robotics challenge finals. In: 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 1028–1035. IEEE (2015)

    Google Scholar 

  15. Gangapurwala, S., Geisert, M., Orsolino, R., Fallon, M., Havoutis, I.: Rloc: terrain-aware legged locomotion using reinforcement learning and optimal control. IEEE Trans. Robot. (2022)

    Google Scholar 

  16. Gangapurwala, S., Mitchell, A., Havoutis, I.: Guided constrained policy optimization for dynamic quadrupedal robot locomotion. IEEE Robot. Autom. Lett. 5(2), 3642–3649 (2020)

    Article  Google Scholar 

  17. García, G., Griffin, R., Pratt, J.: Mpc-based locomotion control of bipedal robots with line-feet contact using centroidal dynamics. In: 2020 IEEE-RAS 20th International Conference on Humanoid Robots (Humanoids), pp. 276–282. IEEE (2021)

    Google Scholar 

  18. García, G., Griffin, R., Pratt, J.: Time-varying model predictive control for highly dynamic motions of quadrupedal robots. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 7344–7349. IEEE (2021)

    Google Scholar 

  19. Gehring, C., Coros, S., Hutter, M., Bloesch, M., Hoepflinger, M.A., Siegwart, R.: Control of dynamic gaits for a quadrupedal robot. In: 2013 IEEE international conference on Robotics and automation, pp. 3287–3292. IEEE (2013)

    Google Scholar 

  20. Gong, Y., Grizzle, J.: Angular momentum about the contact point for control of bipedal locomotion: validation in a lip-based controller (2020). arXiv:2008.10763

  21. Gonzalez, C., Barasuol, V., Frigerio, M., Featherstone, R., Caldwell, D.G., Semini, C.: Line walking and balancing for legged robots with point feet. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3649–3656. IEEE (2020)

    Google Scholar 

  22. Grandia, R., Farshidian, F., Dosovitskiy, A., Ranftl, R., Hutter, M.: Frequency-aware model predictive control. IEEE Robot. Autom. Lett. 4(2) (2019)

    Google Scholar 

  23. Grandia, R., Taylor, A.J., Ames, A.D., Hutter, M.: Multi-layered safety for legged robots via control barrier functions and model predictive control. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2021)

    Google Scholar 

  24. Green, K., Godse, Y., Dao, J., Hatton, R.L., Fern, A., Hurst, J.: Learning spring mass locomotion: guiding policies with a reduced-order model. IEEE Robot. Autom. Lett. 6(2), 3926–3932 (2021)

    Article  Google Scholar 

  25. Heiden, E., Millard, D., Coumans, E., Sheng, Y., Sukhatme, G.S.: Neuralsim: augmenting differentiable simulators with neural networks. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2021)

    Google Scholar 

  26. Hwangbo, J., Lee, J., Dosovitskiy, A., Bellicoso, D., Tsounis, V., Koltun, V., Hutter, M.: Learning agile and dynamic motor skills for legged robots. Sci. Robot. 4(26), eaau5872 (2019)

    Google Scholar 

  27. Jain, D., Iscen, A., Caluwaerts, K.: From pixels to legs: hierarchical learning of quadruped locomotion (2020). arXiv:2011.11722

  28. Kajita, S., Tani, K.: Study of dynamic biped locomotion on rugged terrain-derivation and application of the linear inverted pendulum mode. In: Proceedings of the 1991 IEEE International Conference on Robotics and Automation, pp. 1405–1406. IEEE Computer Society (1991)

    Google Scholar 

  29. Koolen, T., Bertrand, S., Thomas, G., De Boer, T., Wu, T., Smith, J., Englsberger, J., Pratt, J.: Design of a momentum-based control framework and application to the humanoid robot atlas. Int. J. Hum. Robot. 13(01), 1650007 (2016)

    Article  Google Scholar 

  30. Koolen, T., De Boer, T., Rebula, J., Goswami, A., Pratt, J.: Capturability-based analysis and control of legged locomotion, part 1: theory and application to three simple gait models. Int. J. Robot. Res. 31(9) (2012)

    Google Scholar 

  31. Kuindersma, S., Deits, R., Fallon, M., Valenzuela, A., Dai, H., Permenter, F., Koolen, T., Marion, P., Tedrake, R.: Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Auton. Robot. 40(3) (2016)

    Google Scholar 

  32. Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., Hutter, M.: Learning quadrupedal locomotion over challenging terrain. Sci. Robot. 5(47) (2020)

    Google Scholar 

  33. Li, T., Calandra, R., Pathak, D., Tian, Y., Meier, F., Rai, A.: Planning in learned latent action spaces for generalizable legged locomotion. IEEE Robot. Autom. Lett. 6(2), 2682–2689 (2021)

    Article  Google Scholar 

  34. Li, T., Geyer, H., Atkeson, C.G., Rai, A.: Using deep reinforcement learning to learn high-level policies on the atrias biped. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 263–269. IEEE (2019)

    Google Scholar 

  35. Li, T., Srinivasan, K., Meng, M.Q.H., Yuan, W., Bohg, J.: Learning hierarchical control for robust in-hand manipulation. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 8855–8862. IEEE (2020)

    Google Scholar 

  36. Lin, Y.C., Ponton, B., Righetti, L., Berenson, D.: Efficient humanoid contact planning using learned centroidal dynamics prediction. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 5280–5286. IEEE (2019)

    Google Scholar 

  37. Makoviychuk, V., et al.: Isaac gym: high performance gpu-based physics simulation for robot learning (2021). arXiv:2108.10470

  38. Martín-Martín, R., Lee, M.A., Gardner, R., Savarese, S., Bohg, J., Garg, A.: Variable impedance control in end-effector space: an action space for reinforcement learning in contact-rich tasks. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1010–1017. IEEE (2019)

    Google Scholar 

  39. Mastalli, C., Havoutis, I., Focchi, M., Caldwell, D.G., Semini, C.: Motion planning for quadrupedal locomotion: coupled planning, terrain mapping, and whole-body control. IEEE Trans. Robot. 36(6), 1635–1648 (2020)

    Article  Google Scholar 

  40. Meduri, A., Khadiv, M., Righetti, L.: Deepq stepper: a framework for reactive dynamic walking on uneven terrain. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2099–2105. IEEE (2021)

    Google Scholar 

  41. Paigwar, K., Krishna, L., Tirumala, S., Khetan, N., Sagi, A., Joglekar, A., Bhatnagar, S., Ghosal, A., Amrutur, B., Kolathaya, S.: Robust quadrupedal locomotion on sloped terrains: a linear policy approach (2020). arXiv:2010.16342

  42. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32 (2019)

    Google Scholar 

  43. Peng, X.B., Coumans, E., Zhang, T., Lee, T.W., Tan, J., Levine, S.: Learning agile robotic locomotion skills by imitating animals (2020). arXiv:2004.00784

  44. Raibert, M.H.: Legged Robots that Balance. MIT Press (1986)

    Google Scholar 

  45. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms (2017). arXiv:1707.06347

  46. Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., Vanhoucke, V.: Sim-to-real: Learning agile locomotion for quadruped robots (2018). arXiv:1804.10332

  47. Tsounis, V., Alge, M., Lee, J., Farshidian, F., Hutter, M.: Deepgait: planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robot. Autom. Lett. 5(2), 3699–3706 (2020)

    Article  Google Scholar 

  48. Viereck, J., Righetti, L.: Learning a centroidal motion planner for legged locomotion. In: 2021 IEEE International Conference on Robotics and Automation (ICRA) (2021)

    Google Scholar 

  49. Xie, Z., Da, X., van de Panne, M., Babich, B., Garg, A.: Dynamics randomization revisited: a case study for quadrupedal locomotion. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4955–4961. IEEE (2021)

    Google Scholar 

  50. Xiong, X., Ames, A.: 3-d underactuated bipedal walking via h-lip based gait synthesis and stepping stabilization. IEEE Trans. Robot. (2022)

    Google Scholar 

  51. Yang, C., Yuan, K., Zhu, Q., Yu, W., Li, Z.: Multi-expert learning of adaptive legged locomotion. Sci. Robot. 5(49), eabb2174 (2020)

    Google Scholar 

  52. Yang, Y., Caluwaerts, K., Iscen, A., Zhang, T., Tan, J., Sindhwani, V.: Data efficient reinforcement learning for legged robots. In: Conference on Robot Learning, pp. 1–10. PMLR (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of British Columbia, Kelowna, Canada

    Zhaoming Xie & Michiel van de Panne

  2. NVIDIA, Santa Clara, USA

    Zhaoming Xie, Xingye Da, Buck Babich & Animesh Garg

  3. University of Toronto, Vector Institute, Toronto, Canada

    Animesh Garg

Authors
  1. Zhaoming Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingye Da
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Buck Babich
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Animesh Garg
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michiel van de Panne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhaoming Xie .

Editor information

Editors and Affiliations

  1. Center for Ubiquitous Computing, University of Oulu, Oulu, Finland

    Steven M. LaValle

  2. Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA

    Jason M. O’Kane

  3. Department of Aerospace Engineering, University of Maryland, College Park, MD, USA

    Michael Otte

  4. Gates Computer Science, Stanford University, Stanford, CA, USA

    Dorsa Sadigh

  5. Department of Computer Science, University of Maryland, College Park, MD, USA

    Pratap Tokekar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xie, Z., Da, X., Babich, B., Garg, A., de Panne, M.v. (2023). GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model. In: LaValle, S.M., O’Kane, J.M., Otte, M., Sadigh, D., Tokekar, P. (eds) Algorithmic Foundations of Robotics XV. WAFR 2022. Springer Proceedings in Advanced Robotics, vol 25. Springer, Cham. https://doi.org/10.1007/978-3-031-21090-7_31

Download citation

Publish with us

Policies and ethics