Skip to main content
Springer Nature Link
Log in

Partially Occluded Skeleton Action Recognition Based on Multi-stream Fusion Graph Convolutional Networks

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13002))

Included in the following conference series:

  • 2266 Accesses

Abstract

Skeleton-based action recognition methods have been widely developed in recent years. However, the occlusion problem is still a difficult problem at present. Existing skeleton action recognition methods are usually based on complete skeleton data, and their performance is greatly reduced in occluded skeleton action recognition tasks. In order to improve the recognition accuracy on occluded skeleton data, a multi-stream fusion graph convolutional network (MSFGCN) is proposed. The proposed multi-stream fusion network consists of multiple streams, and different streams can handle different occlusion cases. In addition, joint coordinates, relative coordinates, small-scale temporal differences and large-scale temporal differences are extracted simultaneously to construct more discriminative multimodal features. In particular, to the best of our knowledge, we are the first to propose the simultaneous extraction of temporal difference features at different scales, which can more effectively distinguish between actions with different motion amplitude. Experimental results show that the proposed MSFGCN obtains state-of-the-art performance on occluded skeleton datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ding, Z., Wang, P., Ogunbona, P.O., Li, W.: Investigation of different skeleton features for CNN-based 3D action recognition. In: 2017 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 617–622. IEEE (2017)

    Google Scholar 

  2. Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110–1118 (2015)

    Google Scholar 

  3. Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: Twenty-Third International Joint Conference on Artificial Intelligence (2013)

    Google Scholar 

  4. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., Tian, Q.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595–3603 (2019)

    Google Scholar 

  5. Li, Y., Xia, R., Liu, X., Huang, Q.: Learning shape-motion representations from geometric algebra spatio-temporal model for skeleton-based action recognition. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 1066–1071. IEEE (2019)

    Google Scholar 

  6. Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.Y., Kot, A.C.: NTU RGB+ D 120: a large-scale benchmark for 3d human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. 42(10), 2684–2701 (2019)

    Article  Google Scholar 

  7. Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W.: Disentangling and unifying graph convolutions for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 143–152 (2020)

    Google Scholar 

  8. Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+ D: a large scale dataset for 3D human activity analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010–1019 (2016)

    Google Scholar 

  9. Sharma, S., Kiros, R., Salakhutdinov, R.: Action recognition using visual attention. In: Neural Information Processing Systems (NIPS) Time Series Workshop (2015)

    Google Scholar 

  10. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12026–12035 (2019)

    Google Scholar 

  11. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans. Image Process. 29, 9532–9545 (2020)

    Article  Google Scholar 

  12. Si, C., Chen, W., Wang, W., Wang, L., Tan, T.: An attention enhanced graph convolutional LSTM network for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1227–1236 (2019)

    Google Scholar 

  13. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv:1406.2199 (2014)

  14. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  15. Song, Y.F., Zhang, Z., Shan, C., Wang, L.: Richly activated graph convolutional network for robust skeleton-based action recognition. IEEE Transactions on Circuits and Systems for Video Technology (2020)

    Google Scholar 

  16. Thakkar, K., Narayanan, P.: Part-based graph convolutional network for action recognition. arXiv preprint arXiv:1809.04983 (2018)

  17. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: Proceedings of the IEEE international Conference on Computer Vision, pp. 4489–4497 (2015)

    Google Scholar 

  18. Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D skeletons as points in a lie group. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 588–595 (2014)

    Google Scholar 

  19. Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290–1297. IEEE (2012)

    Google Scholar 

  20. Wang, L., Huynh, D.Q., Koniusz, P.: A comparative review of recent Kinect-based action recognition algorithms. IEEE Trans. Image Process. 29, 15–28 (2019)

    Article  MathSciNet  Google Scholar 

  21. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  22. Zanfir, M., Leordeanu, M., Sminchisescu, C.: The moving pose: an efficient 3D kinematics descriptor for low-latency action recognition and detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2752–2759 (2013)

    Google Scholar 

  23. Zhang, P., Lan, C., Xing, J., Zeng, W., Xue, J., Zheng, N.: View adaptive recurrent neural networks for high performance human action recognition from skeleton data. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2117–2126 (2017)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Science Foundation of China under Grant 62101346, in part by the Guangdong Basic and Applied Basic Research Foundation under Grant 2021A1515011702 and in part by the Stable Support Plan for Shenzhen Higher Education Institutions under Grant 20200812104316001.

Author information

Authors and Affiliations

  1. Shenzhen University, No. 3688, Nanhai Avenue, Nanshan District, Shenzhen, China

    Dan Li & Wuzhen Shi

Authors
  1. Dan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wuzhen Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wuzhen Shi .

Editor information

Editors and Affiliations

  1. University of Geneva, Carouge, Switzerland

    Nadia Magnenat-Thalmann

  2. University of Minnesota, Minneapolis, MN, USA

    Victoria Interrante

  3. EPFL, Lausanne, Switzerland

    Daniel Thalmann

  4. University of Crete, Heraklion, Crete, Greece

    George Papagiannakis

  5. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  6. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  7. University of Calgary, Calgary, AB, Canada

    Marina Gavrilova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, D., Shi, W. (2021). Partially Occluded Skeleton Action Recognition Based on Multi-stream Fusion Graph Convolutional Networks. In: Magnenat-Thalmann, N., et al. Advances in Computer Graphics. CGI 2021. Lecture Notes in Computer Science(), vol 13002. Springer, Cham. https://doi.org/10.1007/978-3-030-89029-2_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-89029-2_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-89028-5

  • Online ISBN: 978-3-030-89029-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics