Skip to main content
SpringerLink
Log in

Human motion prediction with gated recurrent unit model of multi-dimensional input

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The issue of human motion prediction aimed to predict sequences of joint positions or joint rotations of human skeleton has recently grown in importance. The Recurrent Neural Network is widely applied on the sequence prediction problems which has been proved effective. However it is difficult to train the model with human skeleton data of multi-dimensional as input, which would do naive forecasting to produce motionless sequence. To address the problem, it is a consensus that additional information will help to improve the accuracy, thus the angular velocities are extracted from the joint rotations as the input to enhance the prediction. Further more, this work adopts proper strategies on the basis of a stacked Gated Recurrent Unit network and verify them on the human motion prediction task. The experimental results show that our network outperforms the state-of-art on the short-term prediction task, and generates plausible action sequences in a relatively long period of time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Barsoum E, Kender J, Liu Z (2018) HP-GAN: Probabilistic 3D human motion prediction via GAN. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 1418–1427

  2. Boulic R, Thalmann NM, Thalmann D (1990) A global human walking model with real-time kinematic personification. Vis Comput 6(6):344–358

    Article  Google Scholar 

  3. Brownlee J (2017) Long short-term memory networks with python: develop sequence prediction models with deep learning. Machine Learning Mastery, Jason Brownlee

  4. Butepage J, Black M.J., Kragic D., Kjellstrom H (2017) Deep representation learning for human motion prediction and classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6158–6166

  5. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078

  6. Dong M, Xu C (2019) On retrospecting human dynamics with attention. In: Proceedings of the 28th international joint conference on artificial intelligence. AAAI Press, pp 708–714

  7. Euler L (1758) Novi commentarii academiae scientiarum petropolitanae, American

  8. Fragkiadaki K, Levine S, Felsen P, Malik J (2015) Recurrent network models for human dynamics. In: Proceedings of the IEEE international conference on computer vision, pp 4346–4354

  9. Gopalakrishnan A, Mali A, Kifer D, Giles L, Ororbia AG (2019) A neural temporal model for human motion prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 12116–12125

  10. Gui LY, Wang YX, Liang X, Moura JM (2018) Adversarial geometry-aware human motion prediction. In: Proceedings of the european conference on computer vision (ECCV), pp 786– 803

  11. Gui LY, Zhang K, Wang YX, Liang X, Moura JM, Veloso M (2018) Teaching robots to predict human motion. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 562–567

  12. Gurbuz SZ, Amin MG (2019) Radar-based human-motion recognition with deep learning: Promising applications for indoor monitoring. IEEE Sig Process Mag 36(4):16–28

    Article  Google Scholar 

  13. Hazewinkel M, Gubareni N, Kirichenko VV (2004) Algebras rings and modules. USA: Springer, New York

    MATH  Google Scholar 

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  15. Ionescu C, Papava D, Olaru V, Sminchisescu C (2013) Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans Pattern Anal Mach Intell 36 (7):1325–1339

    Article  Google Scholar 

  16. Jain A, Zamir AR, Savarese S, Saxena A (2016) Structural-rnn: Deep learning on spatio-temporal graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5308–5317

  17. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:14126980

  18. Kundu JN, Gor M, Babu RV (2019) Bihmp-gan: Bidirectional 3d human motion prediction gan. In: Proceedings of the AAAI conference on artificial intelligence no. 01, vol 33, pp 8553– 8560

  19. Lamb AM, Goyal AGAP, Zhang Y, Zhang S, Courville AC, Bengio Y (2016) Professor forcing: A new algorithm for training recurrent networks. In: Advances in neural information processing systems, pp 4601–4609

  20. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  21. Li C, Zhang Z, Sun Lee W, Hee Lee G (2018) Convolutional sequence to sequence model for human dynamics. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5226–5234

  22. Li H, Shen Y, Zhu Y (2018) Stock price prediction using attention-based multiinput lstm. In: Asian conference on machine learning, pp 454–469

  23. Liu J, Shahroudy A, Wang G, Duan LY, Chichung AK (2019) Skeleton-based online action prediction using scale selection network. IEEE Trans Pattern Anal Mach Intell 42(6):1453–1467

    Article  Google Scholar 

  24. Liu Z, Wu S, Jin S, Liu Q, Lu S, Zimmermann R, Cheng L (2019) Towards natural and accurate future motion prediction of humans and animals. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 10004–10012

  25. Martinez J, Black MJ, Romero J (2017) On human motion prediction using recurrent neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2891–2900

  26. Pavllo D, Feichtenhofer C, Auli M, Grangier D (2019) Modeling human motion with quaternion-based neural networks. arXiv:190107677

  27. Plappert M, Mandery C, Asfour T (2018) Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robot Auton Syst 109:13–26

    Article  Google Scholar 

  28. Pourpanah F, Lim CP, Hao Q (2019) A reinforced fuzzy artmap model for data classification. Int J Mach Learn Cybern 10(7):1643–1655

    Article  Google Scholar 

  29. Qi Z, Shu X, Tang J (2018) Dotanet: Two-stream match-recurrent neural networks for predicting social game result. In: 2018 IEEE fourth international conference on multimedia big data (BigMM), pp 1–5

  30. Sang HF, Chen ZZ, He DK (2020) Human motion prediction based on attention mechanism. Multimed Tools Appl 79(9):5529–5544

    Article  Google Scholar 

  31. Song Y, Demirdjian D, Davis R (2012) Continuous body and hand gesture recognition for natural human-computer interaction. ACM Trans Interact Intell Syst (TiiS) 2(1):5

    Google Scholar 

  32. Strickland J (2008) What is a gimbal–and what does it have to do with nasa

  33. Sugiartawan P, Pulungan R, Sari AK (2017) Prediction by a hybrid of wavelet transform and long-short-term-memory neural network. Int J Adv Comput Sci Appl 8(2):326–332

    Google Scholar 

  34. Tang Y, Ma L, Liu W, Zheng W (2018) Long-term human motion prediction by modeling motion context and enhancing motion dynamic. arXiv:180502513

  35. Tanisaro P, Heidemann G (2018) An empirical study on bidirectional recurrent neural networks for human motion recognition. In: 25th international symposium on temporal representation and reasoning (TIME 2018) no. 21, vol 120, pp 1–19

  36. Tong L, Song Q, Ge Y, Liu M (2013) Hmm-based human fall detection and prediction method using tri-axial accelerometer. IEEE Sensors J 13(5):1849–1856

    Article  Google Scholar 

  37. Toshev A, Szegedy C (2014) Deeppose: Human pose estimation via deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1653–1660

  38. Xu YT, Li Y, Meger D (2019) Human motion prediction via pattern completion in latent representation space. In: 2019 16th conference on computer and robot vision (CRV), pp 57–64

  39. Yan S, Xiong Y, Lin D (2018) Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Thirty-second AAAI conference on artificial intelligence, pp 12026–12035

  40. Ding C, Liu K et al (2021) Spatio-temporal attention on manifold space for 3D human action recognition. In: Applied intelligence, pp 560–570

  41. Men Q, Ho ES, Shum HP, Leung H (2020) A quadruple diffusion convolutional recurrent network for human motion prediction. In: IEEE transactions on circuits and systems for video technology

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China (61807002).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Beijing Institute of Technology, No.5, South Zhongguancun Road, Haidian District, Beijing, 100081, Beijing, People’s Republic of China

    Yue Yu, Niehao Tian & XiangYu Hao

  2. State Key Laboratory of Smart Manufacturing for Special Vehicles and Transmission System, Inner Mongolia No. 2 Mailbox, 014030, Baotou City, People’s Republic of China

    Tao Ma & Chunguang Yang

Authors
  1. Yue Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Niehao Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. XiangYu Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tao Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chunguang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Yu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Y., Tian, N., Hao, X. et al. Human motion prediction with gated recurrent unit model of multi-dimensional input. Appl Intell 52, 6769–6781 (2022). https://doi.org/10.1007/s10489-021-02764-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02764-x

Keywords

Search

Navigation