Abstract
Intelligent choreographic from music is a popular field of study currently. Many works use fragment splicing to generate new motions, which lacks motion diversity. When the input is only music, the frame-by-frame generation methods lead to similar motions generated by the same music. Some works improve this problem by adding motions as one of the inputs, but requires a high number of frames. In this paper, a new transformer-based neural network, TG-dance, is proposed for predicting high-quality 3D dance motions that follow the musical rhythms. We propose a new idea of multi-level expansion of motion sequences and design a new motion encoder, using a multi-level transformer-upsampling layer. The multi-head attention in the transformer allows better access to contextual information. The upsampling can greatly reduce motion frames input, and is memory friendly. We use generative adversarial network to effectively improve the quality of generated motions. We designed experiments on the publicly available large dataset AIST++. The experimental results show that TG-dance network outperforms the latest models in quantitative and qualitative. Our model inputs fewer frames of motion sequences and audio features to predict high-quality 3D dance motion sequences that follow the rhythm of the music.
This work was supported by the Shanghai Natural Science Foundation of China under Grant No.19ZR1419100 and the Shanghai talent development funding of China under Grant No.2021016.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Duan, Y., et al.: Semi-supervised learning for in-game expert-level music-to-dance translation. arXiv preprint arXiv:2009.12763 (2020)
Bowden, R.: Learning statistical models of human motion. In: IEEE Workshop on Human Modeling, Analysis and Synthesis (CVPR), vol. 2000 (2000)
Pullen, K., Bregler, C.: Animating by multi-level sampling. In: Proceedings Computer Animation 2000, pp. 36–42. IEEE (2000)
Chen, K., et al.: ChoreoMaster: choreography-oriented music-driven dance synthesis. ACM Trans. Graph. (TOG) 40(4), 1–13 (2021)
Ye, Z., et al.: ChoreoNet: towards music to dance synthesis with choreographic action unit. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 744–752 (2020)
Guo, X., Zhao, Y., Li, J.: DanceIt: music-inspired dancing video synthesis. IEEE Trans. Image Process. 30, 5559–5572 (2021)
Arikan, O., Forsyth, D.A.: Interactive motion generation from examples. ACM Trans. Graph. (TOG) 21(3), 483–490 (2002)
Kovar, L., Gleicher, M., Pighin, F.: Motion graphs. In: ACM SIGGRAPH 2008 classes, pp. 1–10 (2008)
Kim, T.H., Park, S.I., Shin, S.Y.: Rhythmic-motion synthesis based on motion-beat analysis. ACM Trans. Graph. (TOG) 22(3), 392–401 (2003)
Jain, A., Zamir, A. R., Savarese, S., Saxena, A.: Structural-RNN: deep learning on spatio-temporal graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5308–5317 (2016)
Ghosh, P., Song, J., Aksan, E., Hilliges, O.: Learning human motion models for long-term predictions. In: 2017 International Conference on 3D Vision (3DV), pp. 458–466. IEEE (2017)
Wallace, B., Martin, C. P., Torresen, J., Nymoen, K.: Towards movement generation with audio features. arXiv preprint arXiv:2011.13453 (2020)
Ahn, H., Kim, J., Kim, K., Oh, S.: Generative autoregressive networks for 3d dancing move synthesis from music. IEEE Robot. Autom. Lett. 5(2), 3501–3508 (2020)
Ferreira, J.P., et al.: Learning to dance: a graph convolutional adversarial network to generate realistic dance motions from audio. Comput. Graph. 94, 11–21 (2021)
Zhuang, W., Wang, C., Chai, J., Wang, Y., Shao, M., Xia, S.: Music2Dance: DanceNet for music-driven dance generation. ACM Trans. Multimed. Comput. Commun. Appl. (TOMM) 18(2), 1–21 (2022)
Lee, J., Kim, S., Lee, K.: Listen to dance: music-driven choreography generation using autoregressive encoder-decoder network. arXiv preprint arXiv:1811.00818 (2018)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., Ding, Z.: 3D human pose estimation with spatial and temporal transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11656–11665 (2021)
Huang, C. Z. A., et al.: Music transformer. arXiv preprint arXiv:1809.04281 (2018)
Bhattacharya, U., Rewkowski, N., Banerjee, A., Guhan, P., Bera, A., Manocha, D.: Text2Gestures: a transformer-based network for generating emotive body gestures for virtual agents. In: 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 1–10. IEEE (2021)
Gan, C., Huang, D., Chen, P., Tenenbaum, J.B., Torralba, A.: Foley music: learning to generate music from videos. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12356, pp. 758–775. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_44
Cai, Y., et al.: Learning progressive joint propagation for human motion prediction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12352, pp. 226–242. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58571-6_14
Aksan, E., Kaufmann, M., Cao, P., Hilliges, O.: A spatio-temporal transformer for 3D human motion prediction. In: 2021 International Conference on 3D Vision (3DV), pp. 565–574. IEEE (2021)
Huang, R., Hu, H., Wu, W., Sawada, K., Zhang, M., Jiang, D.: Dance revolution: long-term dance generation with music via curriculum learning. arXiv preprint arXiv:2006.06119 (2020)
Li, J., et al.: Learning to generate diverse dance motions with transformer. arXiv preprint arXiv:2008.08171 (2020)
Li, B., Zhao, Y., Zhelun, S., Sheng, L.: DanceFormer: music conditioned 3D dance generation with parametric motion transformer. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, no. 2, pp. 1272–1279 (2022)
Li, R., Yang, S., Ross, D. A., Kanazawa, A.: Ai choreographer: music conditioned 3D dance generation with AIST++. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13401–13412 (2021)
McFee, B., et al.: librosa: audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference. vol. 8, pp. 18–25 (2015)
Jiang, Y., Chang, S., Wang, Z.: TransGAN: two transformers can make one strong GAN. arXiv preprint arXiv:2102.07074 1(3) (2021)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein GANs. In: Advances in Neural Information Processing Systems. vol. 30 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, D., Zhang, Y., Li, Z., Liu, J. (2023). TG-Dance: TransGAN-Based Intelligent Dance Generation with Music. In: Dang-Nguyen, DT., et al. MultiMedia Modeling. MMM 2023. Lecture Notes in Computer Science, vol 13833. Springer, Cham. https://doi.org/10.1007/978-3-031-27077-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-27077-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-27076-5
Online ISBN: 978-3-031-27077-2
eBook Packages: Computer ScienceComputer Science (R0)