skip to main content
research-article

Learning a family of motor skills from a single motion clip

Published: 19 July 2021 Publication History

Abstract

We present a new algorithm that learns a parameterized family of motor skills from a single motion clip. The motor skills are represented by a deep policy network, which produces a stream of motions in physics simulation in response to user input and environment interaction by navigating continuous action space. Three novel technical components play an important role in the success of our algorithm. First, it explicitly constructs motion parameterization that maps action parameters to their corresponding motions. Simultaneous learning of motion parameterization and motor skills significantly improves the performance and visual quality of learned motor skills. Second, continuous-time reinforcement learning is adopted to explore temporal variations as well as spatial variations in motion parameterization. Lastly, we present a new automatic curriculum generation method that explores continuous action space more efficiently. We demonstrate the flexibility and versatility of our algorithm with highly dynamic motor skills that can be parameterized by task goals, body proportions, physical measurements, and environmental conditions.

Supplementary Material

VTT File (3450626.3459774.vtt)
MP4 File (a93-lee.mp4)
MP4 File (3450626.3459774.mp4)
Presentation.

References

[1]
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). 265--283.
[2]
Yeuhi Abe, C Karen Liu, and Zoran Popović. 2004. Momentum-based parameterization of dynamic character motion. In Proceedings of the 2004 ACM SIGGRAPH/Eurographics symposium on Computer animation. 173--182.
[3]
Kfir Aberman, Peizhuo Li, Dani Lischinski, Olga Sorkine-Hornung, Daniel Cohen-Or, and Baoquan Chen. 2020a. Skeleton-Aware Networks for Deep Motion Retargeting. ACM Transactions on Graphics 39, 4 (2020), 62.
[4]
Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020b. Unpaired Motion Style Transfer from Video to Animation. ACM Transactions on Graphics 39, 4 (2020), 64.
[5]
Adobe Systems Incs. 2018. Mixamo. https://www.mixamo.com
[6]
Shailen Agrawal, Shuo Shen, and Michiel van de Panne. 2014. Diverse motions and character shapes for simulated skills. IEEE transactions on visualization and computer graphics 20, 10 (2014).
[7]
Shailen Agrawal and Michiel van de Panne. 2016. Task-based Locomotion. ACM Transactions on Graphics 35, 4 (2016).
[8]
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. 2017. Hindsight Experience Replay. In Advances in Neural Information Processing Systems. 5048--5058.
[9]
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. 41--48.
[10]
Kevin Bergamin, Simon Claver, Daniel Holden, and James Richard Forbes. 2019. DReCon: Data-Driven Responsive Control of Physics-Based Characters. ACM Transactions on Graphics 38, 6, Article 1 (2019).
[11]
Gaurav Bharaj, Stelian Coros, Bernhard Thomaszewski, James Tompkin, Bernd Bickel, and Hanspeter Pfister. 2015. Computational Design of Walking Automata. In Proceedings of the 14th ACM SIGGRAPH / Eurographics Symposium on Computer Animation. 93--100.
[12]
Alexander Clegg, Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. 2018. Learning to Dress: Synthesizing Human Dressing Motion via Deep Reinforcement Learning. ACM Transactions on Graphics 37, 6, Article 179 (2018).
[13]
CMU. 2002. CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.edu
[14]
Michael Gleicher. 1998. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 33--42.
[15]
Keith Grochow, Steven L. Martin, Aaron Hertzmann, and Zoran Popović. 2004. Style-Based Inverse Kinematics. ACM Transactions on Graphics 23, 3 (2004), 522--531.
[16]
Perttu Hämäläinen, Perttu, Joose Rajamäki, and C. Karen Liu. 2015. Online Control of Simulated Humanoids Using Particle Belief Propagation. ACM Transactions on Graphics 34, 4, Article 81 (2015).
[17]
Chris Hecker, Bernd Raabe, Ryan W Enslow, John DeWeese, Jordan Maynard, and Kees van Prooijen. 2008. Real-time motion retargeting to highly varied user-created morphologies. ACM Transactions on Graphics 27, 3 (2008), 1--11.
[18]
Nicolas Heess, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. 2017. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv:1707.02286 (2017).
[19]
Edmond S. L. Ho, Taku Komura, and Chiew-Lan Tai. 2010. Spatial Relationship Preserving Character Motion Adaptation. ACM Transactions on Graphics 29, 4, Article 33 (2010).
[20]
Daniel Holden, Oussama Kanoun, Maksym Perepichka, and Tiberiu Popa. 2020. Learned Motion Matching. ACM Transactions on Graphics 39, 4, Article 53 (2020).
[21]
Seokpyo Hong, Daseong Han, Kyungmin Cho, Joseph S. Shin, and Junyong Noh. 2019. Physics-Based Full-Body Soccer Motion Control for Dribbling and Shooting. ACM Transactions on Graphics 38, 4, Article 74 (2019).
[22]
Yifeng Jiang, Tom Van Wouwe, Friedl De Groote, and C. Karen Liu. 2019. Synthesis of Biologically Realistic Human Motion Using Joint Torque Actuation. ACM Transactions on Graphics 38, 4, Article 72 (2019).
[23]
Manmyung Kim, Kyunglyul Hyun, Jongmin Kim, and Jehee Lee. 2009. Synchronized multi-character motion editing. ACM transactions on graphics 28, 3, Article 79 (2009).
[24]
Jeongseok Lee, Michael X Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S Srinivasa, Mike Stilman, and C Karen Liu. 2018a. DART: Dynamic animation and robotics toolkit. The Journal of Open Source Software 3, 22 (2018).
[25]
Jehee Lee and Sung Yong Shin. 1999. A hierarchical approach to interactive motion editing for human-like figures. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 39--48.
[26]
Kyungho Lee, Seyoung Lee, and Jehee Lee. 2018b. Interactive character animation by learning multi-objective control. ACM Transactions on Graphics 37, 6, Article 180 (2018).
[27]
Kang Hoon Lee, Myung Geol Choi, and Jehee Lee. 2006. Motion Patches: Building blocks for virtual environments annotated with motion data. ACM Transactions on Graphics 25, 3 (2006), 898--906.
[28]
Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019. Scalable Muscle-actuated Human Simulation and Control. ACM Transactions on Graphics 38, 4, Article 73 (2019).
[29]
Sergey Levine and Vladlen Koltun. 2013. Guided policy search. In International Conference on Machine Learning. 1--9.
[30]
Sergey Levine, Jack M. Wang, Alexis Haraux, Zoran Popović, and Vladlen Koltun. 2012. Continuous Character Control with Low-Dimensional Embeddings. ACM Transactions on Graphics 31, 4, Article 28 (2012).
[31]
C. Karen Liu, Aaron Hertzmann, and Zoran Popović. 2005. Learning Physics-Based Motion Style with Nonlinear Inverse Optimization. ACM Transactions on Graphics 24, 3 (2005), 1071--1081.
[32]
Libin Liu, KangKang Yin, and Baining Guo. 2015. Improving Sampling-Based Motion Control. Computer Graphics Forum 34, 2 (2015), 415--423.
[33]
Libin Liu, KangKang Yin, Michiel van de Panne, and Baining Guo. 2012. Terrain Runner: Control, Parameterization, Composition, and Planning for Highly Dynamic Motions. ACM Transactions on Graphics 31, 6, Article 154 (2012).
[34]
Libin Liu, KangKang Yin, Michiel van de Panne, Tianjia Shao, and Weiwei Xu. 2010. Sampling-Based Contact-Rich Motion Control. ACM Transactions on Graphics 29, 4, Article 128 (2010).
[35]
Ying-Sheng Luo, Jonathan Hans Soeseno, Trista Pei-Chun Chen, and Wei-Chao Chen. 2020. CARL: Controllable Agent with Reinforcement Learning for Quadruped Locomotion. ACM Transactions on Graphics 39, 4, Article 38 (2020).
[36]
Anna Majkowska and Petros Faloutsos. 2007. Flipping with physics: motion editing for acrobatics. In Proceedings of the 2007 ACM SIGGRAPH/Eurographics symposium on Computer animation. 35--44.
[37]
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. 2019. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems (2019).
[38]
James McCann, Nancy S Pollard, and Siddartha S Srinavisa. 2006. Physics-Based Motion Retiming. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 205--214.
[39]
Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2020. Catch amp; Carry: Reusable Neural Controllers for Vision-Guided Whole-Body Tasks. ACM Transactions on Graphics 39, 4, Article 39 (2020), 14 pages.
[40]
Sehee Min, Jungdam Won, Seunghwan Lee, Jungnam Park, and Jehee Lee. 2019. SoftCon: Simulation and Control of Soft-Bodied Animals with Biomimetic Actuators. ACM Transactions on Graphics 38, 6, Article 208 (2019).
[41]
Igor Mordatch and Emo Todorov. 2014. Combining the benefits of function approximation and trajectory optimization. In Robotics: Science and Systems.
[42]
Igor Mordatch, Emanuel Todorov, and Zoran Popović. 2012. Discovery of Complex Behaviors through Contact-Invariant Optimization. ACM Transactions on Graphics 31, 4, Article 43 (2012).
[43]
Tomohiko Mukai and Shigeru Kuriyama. 2005. Geostatistical Motion Interpolation. ACM Transactions on Graphics 24, 3 (2005), 1062--1070.
[44]
Soohwan Park, Hoseok Ryu, Seyoung Lee, Sunmin Lee, and J. Lee. 2019. Learning predict-and-simulate policies from unorganized human motion data. ACM Transactions on Graphics 38, 6, Article 205 (2019).
[45]
Sang Il Park, Hyun Joon Shin, and Sung Yong Shin. 2002. On-Line Locomotion Generation Based on Motion Blending. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 105--111.
[46]
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics 37, 4, Article 143 (2018).
[47]
Xue Bin Peng, Glen Berseth, Kangkang Yin, and Michiel Van De Panne. 2017. DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning. ACM Transactions on Graphics 36, 4, Article 41 (2017).
[48]
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. 2019. MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies. In Advances in Neural Information Processing Systems. 3686--3697.
[49]
Charles Rose, Michael F Cohen, and Bobby Bodenheimer. 1998. Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics and Applications 18, 5 (1998), 32--40.
[50]
Kwang Won Sok, Katsu Yamane, Jehee Lee, and Jessica Hodgins. 2010. Editing dynamic human motions via momentum and force. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer animation. 11--20.
[51]
Sebastian Starke, Yiwei Zhao, Taku Komura, and Kazi Zaman. 2020. Local Motion Phases for Learning Multi-Contact Character Movements. ACM Transactions on Graphics 39, 4, Article 54 (2020).
[52]
Jack M Wang, David J Fleet, and Aaron Hertzmann. 2007. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence 30, 2 (2007), 283--298.
[53]
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2020. A Scalable Approach to Control Diverse Behaviors for Physically Simulated Characters. ACM Transactions on Graphics 39, 4, Article 33 (2020).
[54]
Jungdam Won and Jehee Lee. 2019. Learning body shape variation in physics-based characters. ACM Transactions on Graphics 38, 6, Article 207 (2019).
[55]
Jungdam Won, Kyungho Lee, Carol O'Sullivan, Jessica K Hodgins, and Jehee Lee. 2014. Generating and ranking diverse multi-character interactions. ACM Transactions on Graphics 33, 6, Article 219 (2014).
[56]
Jungdam Won, Jongho Park, Kwanyu Kim, and Jehee Lee. 2017. How to Train Your Dragon: Example-Guided Control of Flapping Flight. ACM Transactions on Graphics 36, 6, Article 198 (2017).
[57]
Jungdam Won, Jungnam Park, and Jehee Lee. 2018. Aerobatics control of flying creatures via self-regulated learning. ACM Transactions on Graphics 37, 6, Article 181 (2018).
[58]
Zhaoming Xie, Hung Yu Ling, Nam Hee Kim, and Michiel van de Panne. 2020. ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills. In Proceedings of the 2006 ACM SIGGRAPH/Eurographics Symposium on Computer Animation.
[59]
Yuting Ye and C. Karen Liu. 2012. Synthesis of Detailed Hand Manipulations Using Contact Sampling. ACM Transactions on Graphics 31, 4, Article 41 (2012).
[60]
Zhiqi Yin and KangKang Yin. 2020. Linear Time Stable PD Controllers for Physics-Based Character Animation. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 191--200.
[61]
Ri Yu, Hwangpil Park, and Jehee Lee. 2019. Figure Skating Simulation from Video. Computer Graphics Forum 38, 7 (2019), 225--234.
[62]
Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Transactions on Graphics 37, 4, Article 144 (2018).
[63]
M Ersin Yumer and Niloy J Mitra. 2016. Spectral Style Transfer for Human Motion between Independent Actions. ACM Transactions on Graphics 35, 4 (2016).

Cited By

View all
  • (2024)CBIL: Collective Behavior Imitation Learning for Fish from Real VideosACM Transactions on Graphics10.1145/368790443:6(1-17)Online publication date: 19-Dec-2024
  • (2024)ReGAIL: Toward Agile Character Control From a Single Reference MotionProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696330(1-10)Online publication date: 21-Nov-2024
  • (2024)Interactive Design of Stylized Walking Gaits for Robotic CharactersACM Transactions on Graphics10.1145/365822743:4(1-16)Online publication date: 19-Jul-2024
  • Show More Cited By

Index Terms

  1. Learning a family of motor skills from a single motion clip

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 40, Issue 4
      August 2021
      2170 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3450626
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2021
      Published in TOG Volume 40, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. data-driven animation
      2. deep reinforcement learning
      3. learning motor skills
      4. physics-based simulation

      Qualifiers

      • Research-article

      Funding Sources

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)133
      • Downloads (Last 6 weeks)9
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)CBIL: Collective Behavior Imitation Learning for Fish from Real VideosACM Transactions on Graphics10.1145/368790443:6(1-17)Online publication date: 19-Dec-2024
      • (2024)ReGAIL: Toward Agile Character Control From a Single Reference MotionProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696330(1-10)Online publication date: 21-Nov-2024
      • (2024)Interactive Design of Stylized Walking Gaits for Robotic CharactersACM Transactions on Graphics10.1145/365822743:4(1-16)Online publication date: 19-Jul-2024
      • (2024)Physics-based Scene Layout Generation from Human MotionACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657517(1-10)Online publication date: 13-Jul-2024
      • (2024)SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised DistillationACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657492(1-11)Online publication date: 13-Jul-2024
      • (2024)Machine Learning Approaches for 3D Motion Synthesis and Musculoskeletal Dynamics Estimation: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.330875330:8(5810-5829)Online publication date: Aug-2024
      • (2024)A Two-Part Transformer Network for Controllable Motion SynthesisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328440230:8(5047-5062)Online publication date: Aug-2024
      • (2024)Walking representation and simulation based on multi-source image fusion and multi-agent reinforcement learning for gait rehabilitationArtificial Intelligence in Medicine10.1016/j.artmed.2024.102945156:COnline publication date: 1-Oct-2024
      • (2023)Neural Categorical Priors for Physics-Based Character ControlACM Transactions on Graphics10.1145/361839742:6(1-16)Online publication date: 5-Dec-2023
      • (2023)Adaptive Tracking of a Single-Rigid-Body Character in Various EnvironmentsSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618187(1-11)Online publication date: 10-Dec-2023
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media