Generalization of Real-Time Motion Control with DRL Using Conditional Rewards and Symmetry Constraints
Authors: 
Luis Ilderlandio da Silva Oliveira, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim Bento Cavalcante-NetoAuthors Info & Claims
Luis Ilderlandio da Silva Oliveira, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim Bento Cavalcante-NetoAuthors Info & ClaimsPages 103 - 112
Abstract
Deep Reinforcement Learning has been increasingly explored as a method for generating physics-based motions in articulated characters. However, effective control tools are still necessary to better guide the learning process and provide animators with greater control and reliability over the resulting animations. This paper proposes new control tools, including the generalization of real-time control, conditional rewards, symmetry constraints, and a user interface. Real-time control allows dynamic adjustment of chosen parameters, conditional rewards simplify the competition between rewards, symmetry constraints reduce uncoordinated movements, and the user interface facilitates training and animation parameter specification. The proposed control tools show promise in improving the quality and control of physics-based character animation.
Supplemental Material
MP4 File
Supplementary video
- Download
- 91.77 MB
References
[1]
Kevin Bergamin, Simon Clavet, Daniel Holden, and James Richard Forbes. 2019. DReCon: data-driven responsive control of physics-based characters. ACM Trans. Graph. 38, 6, Article 206 (2019), 11 pages. https://doi.org/10.1145/3355089.3356536
[2]
Nuttapong Chentanez, Matthias Müller, Miles Macklin, Viktor Makoviychuk, and Stefan Jeschke. 2018. Physics-based motion capture imitation with deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games(MIG ’18). ACM, Article 1, 10 pages.
[3]
Stelian Coros, Philippe Beaudoin, and Michiel Van de Panne. 2009. Robust task-based control policies for physics-based characters. In ACM SIGGRAPH Asia 2009 papers. ACM, Yokohama, Japan, 1–9.
[4]
Antonio Santos de Sousa, Rubens Fernandes Nunes, Creto Augusto Vidal, Joaquim Bento Cavalcante-Neto, and Danilo Borges da Silva. 2021. Physics-Based Motion Control Through DRL’s Reward Functions. In Symposium on Virtual and Augmented Reality. IEEE, 127–136.
[5]
Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. 2022. Adversarial motion priors make good substitutes for complex reward functions. IEEE. In International Conference on Intelligent Robots and Systems (IROS), Vol. 2.
[6]
Vihanga Gamage, Cathy Ennis, and Robert Ross. 2021. Data-Driven Reinforcement Learning for Virtual Character Animation Control. arxiv:2104.06358 [cs.LG]
[7]
Thomas Geijtenbeek and Nicolas Pronost. 2012. Interactive character animation using simulated physics: A state-of-the-art review. In Computer graphics forum, Vol. 31. Wiley Online Library, 2492–2515.
[8]
Thomas Geijtenbeek, Michiel van de Panne, and A. Frank van der Stappen. 2013. Flexible muscle-based locomotion for bipedal creatures. ACM Transactions on Graphics (TOG) 32, 6, Article 206 (2013), 11 pages.
[9]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 80). PMLR, 1861–1870.
[10]
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin Riedmiller, and David Silver. 2017. Emergence of Locomotion Behaviours in Rich Environments. arxiv:1707.02286
[11]
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. 2020. Unity: A General Platform for Intelligent Agents. arxiv:1809.02627 [cs.LG]
[12]
Ariel Kwiatkowski, Eduardo Alvarado, Vicky Kalogeiton, C Karen Liu, Julien Pettré, Michiel van de Panne, and Marie-Paule Cani. 2022. A survey on reinforcement learning methods in character animation. In Computer Graphics Forum, Vol. 41. Wiley Online Library, 613–639.
[13]
Sicen Li, Yiming Pang, Panju Bai, Zhaojin Liu, Jiawei Li, Shihao Hu, Liquan Wang, and Gang Wang. 2023. Learning Agility and Adaptive Legged Locomotion via Curricular Hindsight Reinforcement Learning. arxiv:2310.15583
[14]
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous Control with Deep Reinforcement Learning. arXiv (2015). arxiv:1509.02971
[15]
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG) 37, 4 (2018), 1–14.
[16]
Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018. SFV: Reinforcement Learning of Physical Skills from Videos. ACM Trans. Graph. 37, 6, Article 178 (Nov. 2018), 14 pages.
[17]
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. 2021. AMP: adversarial motion priors for stylized physics-based character control. ACM Trans. Graph. 40, 4, Article 144 (2021), 20 pages. https://doi.org/10.1145/3450626.3459670
[18]
Jiawei Ren, Cunjun Yu, Siwei Chen, Xiao Ma, Liang Pan, and Ziwei Liu. 2023. DiffMimic: Efficient Motion Mimicking with Differentiable Physics. arxiv:2304.03274 [cs.CV]
[19]
Yujin Tang, Jie Tan, and Tatsuya Harada. 2020. Learning Agile Locomotion via Adversarial Training. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 6098–6105.
[20]
Jungdam Won, Deepak Gopinath, and Jessica Hodgins. 2022. Physics-based character controllers using conditional VAEs. ACM Trans. Graph. 41, 4, Article 96 (2022), 12 pages. https://doi.org/10.1145/3528223.3530067
[21]
Pawel Wrotek, Odest Chadwicke Jenkins, and Morgan McGuire. 2006. Dynamo: dynamic, data-driven character control with adjustable balance. In Proceedings of the 2006 ACM SIGGRAPH Symposium on Videogames. ACM, 61–70.
[22]
Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, and Florian Shkurti. 2021. Physics-based human motion estimation and synthesis from videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11532–11541.
[23]
Zeshi Yang and Zhiqi Yin. 2021. Efficient Hyperparameter Optimization for Physics-based Character Animation. Proc. ACM Comput. Graph. Interact. Tech. 4, 1, Article 11 (apr 2021), 19 pages. https://doi.org/10.1145/3451254
[24]
Zhiqi Yin, Zeshi Yang, Michiel Van De Panne, and Kangkang Yin. 2021. Discovering diverse athletic jumping strategies. ACM Trans. Graph. 40, 4, Article 91 (2021), 17 pages. https://doi.org/10.1145/3450626.3459817
[25]
Wenhao Yu, Greg Turk, and C. Karen Liu. 2018. Learning Symmetric and Low-Energy Locomotion. ACM Trans. Graph. 37, 4, Article 144 (jul 2018), 12 pages. https://doi.org/10.1145/3197517.3201397
[26]
Victor B Zordan and Jessica K Hodgins. 2002. Motion capture-driven simulations that hit and react. In Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 89–96.
Index Terms
- Generalization of Real-Time Motion Control with DRL Using Conditional Rewards and Symmetry Constraints
Index terms have been assigned to the content through auto-classification.
Recommendations
MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representations
In this work, we present MoConVQ, a novel unified framework for physics-based motion control leveraging scalable discrete representations. Building upon vector quantized variational autoencoders (VQ-VAE) and model-based reinforcement learning, our ...
Physics-Based Motion Control Through DRL’s Reward Functions
SVR '21: Proceedings of the 23rd Symposium on Virtual and Augmented RealityProducing natural physically-based motions of articulated characters is a challenging problem. The animator needs to figure out high-dimensional parameters of a motion controller to get good visual quality, while still having to deal with the basic ...
DeepMimic: example-guided deep reinforcement learning of physics-based character skills
A longstanding goal in character animation is to combine data-driven specification of behavior with a system that can execute a similar behavior in a physical simulation, thus enabling realistic responses to perturbations and environmental variation. We ...
Comments
Information & Contributors
Information
Published In
September 2024
346 pages
ISBN:9798400709791
DOI:10.1145/3691573
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 September 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SVR 2024
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 18Total Downloads
- Downloads (Last 12 months)18
- Downloads (Last 6 weeks)4
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format