Skip to main content

Advertisement

SpringerLink
Log in

Fusion learning-based recurrent neural network for human motion prediction

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Human motion prediction is an important research frontier, which is a key supporting technology in the fields of human–robot collaboration, automatic driving, etc. As is well known, long-term motion prediction is one most challenging direction. This paper mainly focuses on how to eliminate cumulative errors to overcome the fossilization of long-term motion sequences and aims to improve the reliability of prediction results. This paper proposed an algorithm named “fusion loss learning network,” which is based on gated recurrent unit, to solve the above-mentioned problem. A fusion training method was established by combining the sampling of each step of the GRU unit with true value and output value of each previous step, which helped recover from the errors in the long-term prediction sequences. This method achieved promising results on the Human 3.6 M dataset. The results show that the proposed method could significantly improve the performance of long-term human motion prediction, and the total prediction error is reduced by 7.25% on average.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Koppula HS, Saxena A (2016) Anticipating human activities using object affordances for reactive robotic response. IEEE Trans Pattern Anal Mach Intell 38(1):14–29

    Article  Google Scholar 

  2. Du X, Vasudevan R, Roberson MJ (2019) Bio-LSTM: a biomechanically inspired recurrent neural network for 3-D pedestrian pose and gait prediction. IEEE RA-L 4(2):1501–1508

    Google Scholar 

  3. Jain A, Zamir AR, Savarese S, Saxena A (2016) Structural-RNN: deep learning on spatio-temporal graphs. Presented at 2016 CVPR. [Online]. Available: https://128.84.21.199/pdf/1511.05298.pdf

  4. Martinez J, Black MJ, Romero J (2017) On human motion prediction using recurrent neural networks. Presented at 2017 CVPR. [Online]. Available: https://arxiv.org/pdf/1705.02445.pdf

  5. Safonova A, Hodgins JK (2009) Synthesizing human motion from intuitive constraints, in studies in computational intelligence. Springer, Berlin, Heidelberg, pp 15–39

    Google Scholar 

  6. Wang JM, Fleet DJ, Hertzmann A (2008) gaussian process dynamical models for human motion. IEEE Trans Pattern Anal Mach Intell 30(2):283–298. https://doi.org/10.1109/TPAMI.2007.1167

    Article  Google Scholar 

  7. Levine S, Wang JM, Haraux A, Popović Z, Koltun V (2012) Continuous character control with low-dimensional embeddings. ACM Trans Graph 31(4):1–10

    Article  Google Scholar 

  8. Pavlovi V, Rehg JM, Maccormick J (2000) Learning switching linear models of human motion. Presented at 2000 NIPS. [Online]. Available: http://dl.acm.org/citation.cfm?id=3008751.3008888

  9. Fragkiadaki K, Levine S, Felsen P, Malik J (2015) Recurrent network models for human dynamics. Presented at 2015 ICCV. [Online]. Available: https://ieeexplore.ieee.org/document/#

  10. Komura T, Habibie I, Holden D, Schwarz J, Yearsley J (2017) A recurrent variational autoencoder for human motion synthesis. Presented at 2017 BMVC. [Online]. Available: https://doi.org/10.5244/c.31.119

  11. Zhou Y, Li Z, Xiao S, He C, Huang Z, Li H (2018) Auto-conditioned recurrent networks for extended complex human motion synthesis. Presented at 2018 ICLR. [Online]. Available: https://arxiv.org/abs/1707.05363

  12. Tang Y, Ma L, Liu W, Zheng WS (2018) Long-term human motion prediction by modeling motion context and enhancing mtion dnamic. Presented at 2018 IJCAI. [Online]. Available: https://www.ijcai.org/proceedings/2018/0130.pdf

  13. Li C, Zhang Z, Lee WS, Lee GH (2018) Convolutional sequence to sequence model for human dynamics. Presented at 2018 CVPR. [Online]. Available: https://ieeexplore.ieee.org/document/#

  14. Guo X, Choi J (2019) Human motion prediction via learning local structure representations and temporal dependencies. Presented at 2019 AAAI. [Online]. Available: https://arxiv.org/pdf/1902.07367v2.pdf

  15. Holden D, Komura T, Saito J (2017) Phase-functioned neural networks for character control. ACM Trans Graph 36(4):1–13

    Article  Google Scholar 

  16. Barsoum E, Kender J, Liu Z (2018) HP-GAN: probabilistic 3D human motion prediction via GAN. Presented at 2018 CVPRW. [Online]. Available: https://ieeexplore.ieee.org/document/#

  17. Gui LY, Wang YX, Liang X, Moura JMF (2018) Adversarial geometry-aware human motion prediction. Presented at 2018 ECCV. [Online]. Available:https://link.springer.com/chapter/10.1007%2F978-3-030-01225-0_48

  18. Graves A (2012) Supervised sequence labelling with recurrent neural networks. Springer, Berlin, Heidelberg

    Book  Google Scholar 

  19. Cho K, Merriënboer BV, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. Presented at 2014 EMNLP. [Online]. Available: http://aclweb.org/anthology/D/D14/D14-1179.pdf

  20. Pavllo D, Grangier D, Auli M (2018) QuaterNet: a quaternion-based recurrent model for human motion. Presented at 2018 BMVC. [Online]. Available: http://bmvc2018.org/contents/papers/0675.pdf

  21. Bengio S, Vinyals O, Jaitly N, Shazeer N (2015) Scheduled sampling for sequence prediction with recurrent neural networks. Presented at 2015 NIPS. [Online]. Available: http://arxiv.org/abs/1506.03099

  22. Goyal A, Lamb A, Zhang Y, Zhang S, Courville A, Bengio Y (2016) Professor forcing: a new algorithm for training recurrent networks. Presented at 2016 NIPS. [Online]. Available: http://papers.nips.cc/paper/6098-professor-forcing-a-new-algorithm-for-training-recurrent-networks

  23. Williams RJ (1989) A learning algorithm for continually running fully recurrent neural networks. Neural Comput 1(2):270–280

    Article  Google Scholar 

  24. Ionescu C, Papava D, Olaru V, Sminchisescu C (2014) Human3.6m: large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Trans Pattern Anal Mach Intell 36(7):1325–1339

    Article  Google Scholar 

  25. Gopalakrishnan A, Mali A, Kifer D, Giles CL, Ororbia AG (2019) A neural temporal model for human motion prediction. Presented at 2019 CVPR. [Online]. Available: http://openaccess.thecvf.com/content_CVPR_2019/papers/Gopalakrishnan_A_Neural_Temporal_Model_for_Human_Motion_Prediction_CVPR_2019_paper.pdf

Download references

Acknowledgements

This work was supported in part by the Key Program of NSFC (Grant No. U1908214), Special Project of Central Government Guiding Local Science and Technology Development (Grant No. 2021JH6/10500140), Program for the Liaoning Distinguished Professor, Program for Innovative Research Team in University of Liaoning Province, Dalian and Dalian University, and the Science and Technology Innovation Fund of Dalian (Grant No. 2020JJ25CY001).

Author information

Authors and Affiliations

  1. Key Laboratory of Advanced Design and Intelligent Computing (Ministry of Education), School of Soft Engineering, Dalian University, Dalian, 116622, China

    Chongyang Guo, Rui Liu, Chao Che, Dongsheng Zhou & Qiang Zhang

  2. School of Mechanical Engineering, Dalian University of Technology, Dalian, 116024, China

    Dongsheng Zhou, Qiang Zhang & Xiaopeng Wei

Authors
  1. Chongyang Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chao Che
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dongsheng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaopeng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dongsheng Zhou.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, C., Liu, R., Che, C. et al. Fusion learning-based recurrent neural network for human motion prediction. Intel Serv Robotics 15, 245–257 (2022). https://doi.org/10.1007/s11370-021-00403-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-021-00403-5

Keywords

Search

Navigation