Skip to main content
SpringerLink
Log in

Skeleton-Based Human Action Recognition via Multi-Knowledge Flow Embedding Hierarchically Decomposed Graph Convolutional Network

  • Conference paper
  • First Online:
Computer-Aided Design and Computer Graphics (CADGraphics 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14250))

  • 89 Accesses

Abstract

Skeleton-based action recognition has great potential and extensive application scenarios such as virtual reality and human-robot interaction due to its robustness under complex background and different viewing angles. Recent approaches converted skeleton sequences into spatial-temporal graphs and adopted graph convolutional networks to extract features. Multi-modality recognition and attention mechanisms have also been proposed to boost accuracy. However, the complex feature extraction modules and multi-stream ensemble have increased computational complexity significantly. Thus, most existing methods failed to meet lightweight industrial requirements and lightweight methods were unable to output sufficiently accurate results. To tackle the problem, we propose multi-knowledge flow embedding graph convolutional network, which can achieve high accuracy while maintaining lightweight. We first construct multiple knowledge flows by extracting diverse features from different hierarchically decomposed graphs. Each knowledge flow not only contains information on target class, but also stores profound information for non-target class. Inspired by knowledge distillation, we designed a novel multi-knowledge flow embedding module, which can effectively embed the knowledge into a student model without increasing model complexity. Moreover, student model can be enhanced dramatically by learning simultaneously from complementary knowledge flows. Extensive experiments on authoritative datasets demonstrate that our approach outperforms state-of-the-art with significantly lower computational complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Berg, L.P., Vance, J.M.: Industry use of virtual reality in product design and manufacturing: a survey. Virtual Reality 21, 1–17 (2017). https://doi.org/10.1007/s10055-016-0293-9

    Article  Google Scholar 

  2. Chen, P., Liu, S., Zhao, H., Jia, J.: Distilling knowledge via knowledge review. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5008–5017 (2021). https://doi.org/10.1109/CVPR46437.2021.00497

  3. Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., Hu, W.: Channel-wise topology refinement graph convolution for skeleton-based action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13359–13368 (2021). https://doi.org/10.1109/ICCV48922.2021.01311

  4. Chen, Z., Li, S., Yang, B., Li, Q., Liu, H.: Multi-scale spatial temporal graph convolutional network for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 1113–1122 (2021). https://doi.org/10.1609/aaai.v35i2.16197

  5. Cheng, K., Zhang, Y., Cao, C., Shi, L., Cheng, J., Lu, H.: Decoupling GCN with DropGraph module for skeleton-based action recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12369, pp. 536–553. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_32

    Chapter  Google Scholar 

  6. Cheng, K., Zhang, Y., He, X., Chen, W., Cheng, J., Lu, H.: Skeleton-based action recognition with shift graph convolutional network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 183–192 (2020). https://doi.org/10.1109/CVPR42600.2020.00026

  7. Chi, H.G., Ha, M.H., Chi, S., Lee, S.W., Huang, Q., Ramani, K.: InfoGCN: representation learning for human skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20186–20196 (2022). https://doi.org/10.1109/CVPR52688.2022.01955

  8. Cho, S., Maqbool, M., Liu, F., Foroosh, H.: Self-attention network for skeleton-based human action recognition. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 635–644 (2020). https://doi.org/10.1109/WACV45572.2020.9093639

  9. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015). https://doi.org/10.48550/arXiv.1503.02531

  10. Ji, X., Zhao, Q., Cheng, J., Ma, C.: Exploiting spatio-temporal representation for 3D human action recognition from depth map sequences. Knowl.-Based Syst. 227, 107040 (2021). https://doi.org/10.1016/j.knosys.2021.107040

  11. Lee, J., Lee, M., Lee, D., Lee, S.: Hierarchically decomposed graph convolutional networks for skeleton-based action recognition. arXiv preprint arXiv:2208.10741 (2022). https://doi.org/10.48550/arXiv.2208.10741

  12. Li, M., Chen, S., Chen, X., Zhang, Y., Wang, Y., Tian, Q.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595–3603 (2019). https://doi.org/10.1109/CVPR.2019.00371

  13. Liu, B., Cai, H., Ju, Z., Liu, H.: RGB-D sensing based human action and interaction analysis: a survey. Pattern Recognit. 94, 1–12 (2019). https://doi.org/10.1016/j.patcog.2019.05.020

    Article  Google Scholar 

  14. Liu, J., Shahroudy, A., Perez, M., Wang, G., Duan, L.Y., Kot, A.C.: NTU RGB+ D 120: a large-scale benchmark for 3D human activity understanding. IEEE Trans. Pattern Anal. Mach. Intell. 42(10), 2684–2701 (2019). https://doi.org/10.1109/TPAMI.2019.2916873

    Article  Google Scholar 

  15. Liu, Y., Zhang, H., Li, Y., He, K., Xu, D.: Skeleton-based human action recognition via large-kernel attention graph convolutional network. IEEE Trans. Visual Comput. Graphics 29(5), 2575–2585 (2023). https://doi.org/10.1109/TVCG.2023.3247075

    Article  Google Scholar 

  16. Liu, Y., Zhang, H., Xu, D., He, K.: Graph transformer network with temporal kernel attention for skeleton-based action recognition. Knowl.-Based Syst. 240, 108146 (2022). https://doi.org/10.1016/j.knosys.2022.108146

  17. Liu, Z., Zhang, H., Chen, Z., Wang, Z., Ouyang, W.: Disentangling and unifying graph convolutions for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 143–152 (2020). https://doi.org/10.1109/CVPR42600.2020.00022

  18. Plizzari, C., Cannici, M., Matteucci, M.: Skeleton-based action recognition via spatial and temporal transformer networks. Comput. Vis. Image Underst. 208, 103219 (2021). https://doi.org/10.1016/j.cviu.2021.103219

  19. Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014). https://doi.org/10.48550/arXiv.1412.6550

  20. Shahroudy, A., Liu, J., Ng, T.T., Wang, G.: NTU RGB+ D: a large scale dataset for 3D human activity analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010–1019 (2016). https://doi.org/10.1109/CVPR.2016.115

  21. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with directed graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7912–7921 (2019). https://doi.org/10.1109/CVPR.2019.00810

  22. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12026–12035 (2019). https://doi.org/10.1109/CVPR.2019.01230

  23. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Decoupled spatial-temporal attention network for skeleton-based action-gesture recognition. In: Ishikawa, H., Liu, C.-L., Pajdla, T., Shi, J. (eds.) ACCV 2020. LNCS, vol. 12626, pp. 38–53. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-69541-5_3

    Chapter  Google Scholar 

  24. Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019). https://doi.org/10.48550/arXiv.1910.10699

  25. Vishwakarma, S., Agrawal, A.: A survey on activity recognition and behavior understanding in video surveillance. Vis. Comput. 29, 983–1009 (2013). https://doi.org/10.1007/s00371-012-0752-6

    Article  Google Scholar 

  26. Wang, L., et al.: Temporal segment networks: towards good practices for deep action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 20–36. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_2

    Chapter  Google Scholar 

  27. Wang, Y., et al.: 3DV: 3D dynamic voxel for action recognition in depth video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 511–520 (2020). https://doi.org/10.1109/CVPR42600.2020.00059

  28. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (TOG) 38(5), 1–12 (2019). https://doi.org/10.1145/3326362

    Article  Google Scholar 

  29. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018). https://doi.org/10.1609/aaai.v32i1.12328

  30. Ye, F., Pu, S., Zhong, Q., Li, C., Xie, D., Tang, H.: Dynamic GCN: context-enriched topology learning for skeleton-based action recognition. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 55–63 (2020). https://doi.org/10.1145/3394171.3413941

  31. Yim, J., Joo, D., Bae, J., Kim, J.: A gift from knowledge distillation: fast optimization, network minimization and transfer learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4133–4141 (2017). https://doi.org/10.1109/CVPR.2017.754

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China [grant numbers 62162068, 62061049]; Yunnan Province Ten Thousand Talents Program and Yunling Scholars Special Project [grant number YNWRYLXZ2018-022]; Yunnan Provincial Science and Technology Department-Yunnan University “Double First Class” Construction Joint Fund Project [grant number 202301BF070001-025]; and the Science Research Fund Project of Yunnan Provincial Department of Education under [grant number 2021Y027].

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, China

    Yanqiu Li, Yanan Liu, Hao Zhang, Shouzheng Sun & Dan Xu

Authors
  1. Yanqiu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shouzheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dan Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dan Xu .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Shi-Min Hu

  2. Nanyang Technological University, Singapore, Singapore

    Yiyu Cai

  3. Cardiff University, Cardiff, UK

    Paul Rosin

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, Y., Liu, Y., Zhang, H., Sun, S., Xu, D. (2024). Skeleton-Based Human Action Recognition via Multi-Knowledge Flow Embedding Hierarchically Decomposed Graph Convolutional Network. In: Hu, SM., Cai, Y., Rosin, P. (eds) Computer-Aided Design and Computer Graphics. CADGraphics 2023. Lecture Notes in Computer Science, vol 14250. Springer, Singapore. https://doi.org/10.1007/978-981-99-9666-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-9666-7_13

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-9665-0

  • Online ISBN: 978-981-99-9666-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation