Skip to main content
Springer Nature Link
Log in

Sim-to-real transfer of co-optimized soft robot crawlers

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This work provides a complete framework for the simulation, co-optimization, and sim-to-real transfer of the design and control of soft legged robots. Soft robots have “mechanical intelligence”: the ability to passively exhibit behaviors that would otherwise be difficult to program. Exploiting this capacity requires consideration of the coupling between design and control. Co-optimization provides a way to reason over this coupling. Yet, it is difficult to achieve simulations that are both sufficiently accurate to allow for sim-to-real transfer and fast enough for contemporary co-optimization algorithms. We describe a modularized model order reduction algorithm that improves simulation efficiency, while preserving the accuracy required to learn effective soft robot design and control. We propose a reinforcement learning-based co-optimization framework that identifies several soft crawling robots that outperform an expert baseline with zero-shot sim-to-real transfer. We study generalization of the framework to new terrains, and the efficacy of domain randomization as a means to improve sim-to-real transfer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. A publicly available implementation of our joint optimization framework is available at https://github.com/cbschaff/evolving-soft-robots.

References

  • Bern, J. M., Banzet, P., Poranne, R., et al. (2019). Trajectory optimization for cable-driven soft robot locomotion. In Proceedings of Robotics: Science and Systems (RSS).

  • Bern, J. M., Schnider, Y., Banzet, P., et al. (2020). Soft robot control with a learned differentiable model. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft).

  • Bhatia, J., Jackson, H., Tian, Y., et al. (2021). Evolution gym: A large-scale benchmark for evolving soft robots. In Advances in Neural Information Processing Systems (NeurIPS).

  • Bongard, J. (2011). Morphological change in machines accelerates the evolution of robust behavior. Proceedings of the National Academy of Sciences, 108(4), 1234–1239.

    Article  Google Scholar 

  • Bravo-Palacios, G., Del Prete, A., & Wensing, P. M. (2020). One robot for many tasks: Versatile co-design through stochastic programming. IEEE Robotics and Automation Letters, 5(2), 1680–1687.

    Article  Google Scholar 

  • Bruder, D., Gillespie, B., Remy, C. D., et al. (2019). Modeling and control of soft robots using the Koopman operator and model predictive control. In Proceedings of Robotics: Science and Systems (RSS).

  • Centurelli, A., Arleo, L., Rizzo, A., et al. (2022). Closed-loop dynamic control of a soft manipulator using deep reinforcement learning. IEEE Robotics and Automation Letters, 7(2), 4741–4748.

    Article  Google Scholar 

  • Chen, F., & Wang, M.Y. (2020). Design optimization of soft robots: A review of the state of the art. IEEE Robotics & Automation Magazine.

  • Chen, T., He, Z., & Ciocarlie, M. (2020). Hardware as policy: Mechanical and computational co-optimization using deep reinforcement learning. arXiv:2008.04460.

  • Cheney, N., MacCurdy, R., Clune, J., et al. (2013). Unshackling evolution: Evolving soft robots with multiple materials and a powerful generative encoding. In Proceedings of the Annual Conference on Genetic and Evolutionary Computation (GECCO).

  • Coevoet, E., Morales-Bieze, T., Largilliere, F., et al. (2017). Software toolkit for modeling, simulation, and control of soft robots. Advanced Robotics 31.

  • Collins, J., Chand, S., Vanderkop, A., et al. (2021). A review of physics simulators for robotic applications. IEEE Access.

  • Culha, U., Demir, S. O., Trimpe, S., et al. (2020). Learning of sub-optimal gait controllers for magnetic walking soft millirobots. In Proceedings of Robotics: Science and Systems (RSS).

  • Deimel, R., Irmisch, P., Wall, V., et al. (2017). Automated co-design of soft hand morphology and control strategy for grasping. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

  • Digumarti, K. M., Gehring, C., Coros, S., et al. (2014). Concurrent optimization of mechanical design and locomotion control of a legged robot. In Mobile Service Robotics. p 315–323.

  • Drotman, D., Jadhav, S., Sharp, D., et al. (2021). Electronics-free pneumatic circuits for controlling soft-legged robots. Science Robotics 6(51).

  • Dubied, M., Michelis, M. Y., Spielberg, A., et al. (2022). Sim-to-real for soft robots using differentiable fem: Recipes for meshing, damping, and actuation. IEEE Robotics and Automation Letters, 7(2), 5015–5022.

    Article  Google Scholar 

  • Farhat, C., Avery, P., Chapman, T., et al. (2014). Dimensional reduction of nonlinear finite element dynamic models with finite rotations and energy-based mesh sampling and weighting for computational efficiency. International Journal for Numerical Methods in Engineering, 98(9), 625–662.

    Article  MathSciNet  MATH  Google Scholar 

  • Faure, F., Duriez, C., Delingette, H., et al. (2012). SOFA: A multi-model framework for interactive physical simulation. In Soft Tissue Biomechanical Modeling for Computer Assisted Surgery. Studies in Mechanobiology, Tissue Engineering and Biomaterials, p 283–321.

  • Gamus, B., Salem, L., Gat, A. D., et al. (2020). Understanding inchworm crawling for soft-robotics. IEEE Robotics and Automation Letters, 5(2), 1397–1404.

    Article  Google Scholar 

  • Geilinger, M., Poranne, R., Desai, R., et al. (2018). Skaterbots: Optimization-based design and motion synthesis for robotic creatures with legs and wheels. ACM Transactions on Graphics (TOG) 37(4).

  • Goury, O., & Duriez, C. (2018). Fast, generic, and reliable control and simulation of soft robots using model order reduction. Transactions on Robotics, 34(6), 1565–1576.

    Article  Google Scholar 

  • Ha, D. (2019). Reinforcement learning for improving agent design. Artificial Life, 25(4), 352–365.

    Article  Google Scholar 

  • Ha, S., Coros, S., Alspach, A., et al. (2017). Joint optimization of robot design and motion parameters using the implicit function theorem. In Proceedings of Robotics: Science and Systems (RSS).

  • Haarnoja, T., Zhou, A., Abbeel, P., et al. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the International Conference on Machine Learning (ICML).

  • Hiller, J., & Lipson, H. (2014). Dynamic simulation of soft multimaterial 3D-printed objects. Soft Robotics, 1, 88–101.

    Article  Google Scholar 

  • Howison, T., Hauser, S., Hughes, J., et al. (2020). Reality-assisted evolution of soft robots through large-scale physical experimentation: A review. arXiv:2009.13960.

  • Hu, Y., Liu, J., Spielberg, A., et al. (2019). ChainQueen: A real-time differentiable physical simulator for soft robotics. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).

  • Huang, W., Mordatch, I., & Pathak, D. (2020). One policy to control them all: Shared modular policies for agent-agnostic control. arXiv:2007.04976.

  • Ju, H., Juan, R., Gomez, R., et al. (2022). Transferring policy of deep reinforcement learning from simulation to reality for robotics. Nature Machine Intelligence pp 1–11.

  • Kim, D., Kim, S. H., Kim, T., et al. (2021). Review of machine learning methods in soft robotics. PLOS ONE 16(2).

  • Kleeberger, K., Bormann, R., Kraus, W., et al. (2020). A survey on learning-based robotic grasping. Current Robotics Reports, 1, 239–249.

    Article  Google Scholar 

  • Kriegman, S., Walker, S., Shah, D., et al. (2019). Automated shapeshifting for function recovery in damaged robots. In Proceedings of Robotics: Science and Systems (RSS).

  • Kriegman, S., Nasab, A. M., Shah, D., et al. (2020). Scalable sim-to-real transfer of soft robot designs. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft)

  • Kurin, V., Igl, M., Rocktäschel, T., et al. (2021). My body is a cage: The role of morphology in graph-based incompatible control. arXiv:2010.01856.

  • Lee, K., Kim, S., Lim, S., et al. (2020). Generalized Tsallis entropy reinforcement learning and its application to soft mobile robots. In Proceedings of Robotics: Science and Systems (RSS).

  • Li, Y., Wang, X., & Kwok, K. W. (2022). Towards adaptive continuous control of soft robotic manipulator using reinforcement learning. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 7074–7081.

  • Lipson, H., & Pollack, J. B. (2000). Automatic design and manufacture of robotic lifeforms. Nature, 406, 974–978.

    Article  Google Scholar 

  • Lipton, J. I., MacCurdy, R., Manchester, Z., et al. (2018). Handedness in shearing auxetics creates rigid and compliant structures. Science, 360(6389), 632–635.

    Article  Google Scholar 

  • Luo, J., & Hauser, K. (2015). Robust trajectory optimization under frictional contact with iterative learning. In Proceedings of Robotics: Science and Systems (RSS).

  • Ma, P., Du, T., Zhang, J. Z., et al. (2021). DiffAqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation. arXiv preprint arXiv:2104.00837.

  • Majidi, C., Shepherd, R. F., Kramer, R. K., et al. (2013). Influence of surface traction on soft robot undulation. The International Journal of Robotics Research, 32(13), 1577–1584.

    Article  Google Scholar 

  • Morimoto, R., Nishikawa, S., Niiyama, R., et al. (2021). Model-free reinforcement learning with ensemble for a soft continuum robot arm. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft).

  • Morzadec, T., Marcha, D., & Duriez, C. (2019). Toward shape optimization of soft robots. In Proceedings of the IEEE International Conference on Soft Robotics (RoboSoft).

  • Mosadegh, B., Polygerinos, P., Keplinger, C., et al. (2014). Pneumatic networks for soft robotics that actuate rapidly. Advanced Functional Materials, 24(15), 2163–2170.

    Article  Google Scholar 

  • Murata, S., & Kurokawa, H. (2007). Self-reconfigurable robots. IEEE Robotics & Automation Magazine, 14(1), 71–78.

    Article  Google Scholar 

  • Park, J. H., & Asada, H. (1994). Concurrent design optimization of mechanical structure and control for high speed robots. Journal of Dynamic Systems, Measurement, and Control, 116(3), 344–356.

    Article  MATH  Google Scholar 

  • Pathak, D., Lu, C., Darrell, T., et al. (2019). Learning to control self-assembling morphologies: A study of generalization via modularity. arXiv:1902.05546.

  • Paul, C., & Bongard, J. C. (2001). The road less travelled: Morphology in the optimization of biped robot locomotion. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

  • Paul, C., Valero-Cuevas, F. J., & Lipson, H. (2006). Design and control of tensegrity robots for locomotion. Transactions on Robotics 22(5).

  • Pinskier, J., & Howard, D. (2021). From bioinspiration to computer generation: Developments in autonomous soft robot design. Advanced Intelligent Systems.

  • Qi, H., Joyce, K., & Boyce, M. (2003). Durometer hardness and the stress-strain behavior of elastomeric materials. Rubber Chemistry and Technology, 76(2), 419–435.

    Article  Google Scholar 

  • Rafsanjani, A., Zhang, Y., Liu, B., et al. (2018). Kirigami skins make a simple soft actuator crawl. Science Robotics 3(15).

  • Rus, D., & Tolley, M. T. (2015). Design, fabrication and control of soft robots. Nature, 521, 467–475.

    Article  Google Scholar 

  • Schaff, C., Yunis, D., Chakrabarti, A., et al. (2019). Jointly learning to construct and control agents using deep reinforcement learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).

  • Schaff, C., Sedal, A., & Walter, M. R. (2022). Soft robots learn to crawl: Jointly optimizing design and control with sim-to-real transfer. In Proceedings of Robotics: Science and Systems (RSS).

  • Seo, J., Paik, J., & Yim, M. (2019). Modular reconfigurable robotics. Annual Review of Control, Robotics, and Autonomous Systems, 2, 63–88.

    Article  Google Scholar 

  • Shepherd, R. F., Ilievski, F., Choi, W., et al. (2011). Multigait soft robot. Proceedings of the National Academy of Sciences, 108(51), 20,400-20,403.

    Article  Google Scholar 

  • Sims, K. (1994). Evolving virtual creatures. In Proceeding of the International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH).

  • Spielberg, A., Araki, B., Sung, C., et al. (2017). Functional co-optimization of articulated robots. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).

  • Spielberg, A., Amini, A., Chin, L., et al. (2021). Co-learning of task and sensor placement for soft robotics. IEEE Robotics and Automation Letters, 6(2), 1208–1215.

    Article  Google Scholar 

  • Talamini, J., Medvet, E., Bartoli, A., et al. (2019). Evolutionary synthesis of sensing controllers for voxel-based soft robots. In Proceedings of the Artificial Life Conference (ALIFE).

  • Tan, J., Zhang, T., Coumans, E., et al. (2018). Sim-to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems (RSS).

  • Taylor, O., & Rodriguez, A. (2019). Optimal shape and motion planning for dynamic planar manipulation. Autonomous Robots, 43(2), 327–344.

    Article  MathSciNet  Google Scholar 

  • Tiboni, G., Protopapa, A., Tommasi, T., et al. (2023). Domain randomization for robust, affordable and effective closed-loop control of soft robots. arXiv preprint arXiv:2303.04136.

  • Tobin, J., Fong, R., Ray, A., et al. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 23–30

  • Varghese, N. V., & Mahmoud, Q. (2020). A survey of multi-task deep reinforcement learning. Electronics.

  • Vikas, V., Cohen, E., Grassi, R., et al. (2016). Design and locomotion control of a soft robot using friction manipulation and motor-tendon actuation. IEEE Transactions on Robotics, 32(4), 949–959.

    Article  Google Scholar 

  • Villarreal-Cervantes, M. G., Cruz-Villar, C. A., Alvarez-Gallegos, J., et al. (2013). Robust structure-control design approach for mechatronic systems. Transactions on Mechatronics, 18(5), 1592–1601.

    Article  Google Scholar 

  • Vitanov, I., Rizqi, A., & Althoefer, K. (2020). Shape reconstruction of soft-body manipulator: A learning-based approach. In Proceedings of the Annual Conference Towards Autonomous Robotic Systems (TAROS)

  • Wang, J., & Chortos, A. (2022). Control strategies for soft robot systems. Advanced Intelligent Systems, 4(5), 2100,165.

    Article  Google Scholar 

  • Weiss, J. (2019). A tutorial on the proper orthogonal decomposition. In AIAA Aviation 2019 Forum, p 3333.

  • Whitman, J., Travers, M., & Choset, H. (2021). Learning modular robot control policies. arXiv:2105.10049.

  • Zhang, J. Z., Zhang, Y., Ma, P., et al. (2022). Sim2real for soft robotic fish via differentiable simulation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp 12,598–12,605.

  • Zhao, A., Xu, J., Konaković-Luković, M., et al. (2020). Robogrammar: Graph grammar for terrain-optimized robot design. ACM Transactions on Graphics (TOG) 39(6).

  • Zhu, Y., Rossiter, J., & Hauser, H. (2019). Learning in growing robots: Knowledge transfer from tadpole to frog robot. In Proceedings of the International Conference on Biomimetic and Biohybrid Systems.

Download references

Acknowledgements

We thank Olivier Goury and Hugo Talbot for assistance with the SOFA implementation, and Arthur MacKeith.

Funding

Partially funded by the National Sciences and Engineering Research Council of Canada (NSERC), Discovery Grants program (RGPIN 1512).

Author information

Author notes

  1. C. Schaff and A. Sedal have contributed equally to this work.

Authors and Affiliations

  1. Department of Mechanical Engineering, McGill University, 817 Sherbrooke St W, Montreal, QC, H3A 0C3, Canada

    Audrey Sedal & Shiyao Ni

  2. TTI-Chicago, 6045 S Kenwood Ave, Chicago, IL, 60637, USA

    Charles Schaff & Matthew R. Walter

Authors
  1. Charles Schaff
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Audrey Sedal
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shiyao Ni
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Matthew R. Walter
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

AS and CS contributed equally to this manuscript. AS, CS and MW wrote the text. AS, MW and CS developed the framework and performed physical experiments. CS and SN performed computations. SN generated the video. All authors reviewed the manuscript.

Corresponding author

Correspondence to Audrey Sedal.

Ethics declarations

Competing interests

Not applicable.

Ethical approval

Not applicable.

Availability of data and materials

Code is available at the framework’s Github Page: https://github.com/macrobotics-lab/evolving-soft-robots/tree/main. Data is available at https://github.com/macrobotics-lab/AuRo_evolving_sobot_data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 860 KB)

Supplementary file 2 (mp4 561629 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schaff, C., Sedal, A., Ni, S. et al. Sim-to-real transfer of co-optimized soft robot crawlers. Auton Robot 47, 1195–1211 (2023). https://doi.org/10.1007/s10514-023-10130-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-023-10130-8

Keywords