Skip to main content
SpringerLink
Log in

Short-term path signature for skeleton-based action recognition

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Skeleton-based action recognition (SBAR) is an important task in the field of computer vision. Learning effective action representations from skeleton sequences and improving the performance of action recognition models remain challenging problems. To capture effective features from skeleton sequences, a novel feature called a short-term path signature (STPS) is proposed in this work. Based on the STPS, a plug-and-play module is proposed to achieve improved SBAR. In this module, the STPS is applied as input, and a spatial-temporal graph convolutional network (ST-GCN) is used to learn action features. Finally, a multistream ST-GCN is built to achieve SBAR. The proposed method is verified on the NTU-RGB+D dataset. Several ablation experiments are conducted to verify the effectiveness of the proposed module. The experimental results show that the proposed STPS is beneficial for improving the accuracy of action recognition networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Availability of data and materials

All data generated or analyzed during this study are included in this published article.

References

  1. Caetano, C., Brémond, F., Schwartz, WR.: Skeleton image representation for 3d action recognition based on tree structure and reference joints. In: 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), IEEE, pp. 16–23 (2019a)

  2. Caetano, C., Sena, J., Brémond, F.: et al Skelemotion: A new representation of skeleton joint sequences based on motion information for 3d action recognition. In: 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), IEEE, pp. 1–8 (2019b)

  3. Cheng, K., Zhang, Y., He, X., et al.: Extremely lightweight skeleton-based action recognition with shiftgcn++. IEEE Trans. Image Process. 30, 7333–7348 (2021). https://doi.org/10.1109/TIP.2021.3104182

    Article  Google Scholar 

  4. Chéron, G., Laptev, I., Schmid, C.: P-cnn: Pose-based cnn features for action recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3218–3226 (2015)

  5. Chevyrev, I., Kormilitzin, A.: A primer on the signature method in machine learning. arXiv preprint arXiv:1603.03788 (2016)

  6. Du, Y., Fu, Y., Wang, L.: Skeleton based action recognition with convolutional neural network. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), IEEE, pp. 579–583 (2015)

  7. Gu, Y., Sheng, W., Ou, Y.: et al Human action recognition with contextual constraints using a rgb-d sensor. In: 2013 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 674–679 (2013)

  8. Hambly, B., Lyons, T.: Uniqueness for the signature of a path of bounded variation and the reduced path group. Annal. Math. 109–167 (2010)

  9. Hu, JF., Zheng, WS., Lai, J.: et al Jointly learning heterogeneous features for rgb-d activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5344–5352 (2015)

  10. Hussein, ME., Torki, M., Gowayyed, MA.: et al Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations. In: Twenty-third International Joint Conference on Artificial Intelligence (2013)

  11. Ke, Q., Bennamoun, M., An, S., et al.: Learning clip representations for skeleton-based 3d action recognition. IEEE Trans. Image Process. 27(6), 2842–2855 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  12. Lee, I., Kim, D., Kang, S.: et al Ensemble deep learning for skeleton-based action recognition using temporal sliding lstm networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1012–1020 (2017)

  13. Lev, G., Sadeh, G., Klein, B.: et al Rnn fisher vectors for action recognition and image annotation. In: European Conference on Computer Vision, Springer, pp. 833–850 (2016)

  14. Li, B., Dai, Y., Cheng, X.: et al Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep cnn. In: 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), IEEE, pp. 601–604 (2017)

  15. Li, M., Chen, S., Chen, X.: et al Actional-structural graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595–3603 (2019)

  16. Liu, M., Liu, H., Chen, C.: Enhanced skeleton visualization for view invariant human action recognition. Pattern. Recogn. 68, 346–362 (2017)

    Article  Google Scholar 

  17. Liu, Z., Zhang, H., Chen, Z.: et al Disentangling and unifying graph convolutions for skeleton-based action recognition. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 140–149, https://doi.org/10.1109/CVPR42600.2020.00022 (2020)

  18. Lyons, T.: Rough paths, signatures and the modelling of functions on streams. arXiv preprint arXiv:1405.4537 (2014)

  19. Lyons, T.J., Caruana, M., Lévy, T.: Differential Equations Driven by Rough Paths. Springer (2007)

    Book  MATH  Google Scholar 

  20. Mostafa, A., Peng, W., Zhao, G.: Hyperbolic spatial temporal graph convolutional networks. In: 2022 IEEE International Conference on Image Processing (ICIP), pp 3301–3305, https://doi.org/10.1109/ICIP46576.2022.9897522 (2022)

  21. Ni, H.: A multi-dimensional stream and its signature representation. arXiv preprint arXiv:1509.03346 (2015)

  22. Papadopoulos, K., Ghorbel, E., Aouada, D.: et al Vertex feature encoding and hierarchical temporal modeling in a spatial-temporal graph convolutional network for action recognition. arXiv preprint arXiv:1912.09745 (2019)

  23. Peng, W., Hong, X., Chen, H., et al.: Learning graph convolutional network for skeleton-based human action recognition by neural searching. Proceed. AAAI Conf. Artif. Intell. 34(03), 2669–2676 (2020). https://doi.org/10.1609/aaai.v34i03.5652

    Article  Google Scholar 

  24. Peng, W., Shi, J., Xia, Z.: et al Mix dimension in poincaré geometry for 3d skeleton-based action recognition. In: Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, MM ’20, p 1432-1440, https://doi.org/10.1145/3394171.3413910 (2020b)

  25. Peng, W., Hong, X., Zhao, G.: Tripool: Graph triplet pooling for 3d skeleton-based action recognition. Pattern Recognit. PP. 115:107921 (2021)

  26. Peng, W., Shi, J., Varanka, T., et al.: Rethinking the st-gcns for 3d skeleton-based human action recognition. Neurocomputing 454, 45–53 (2021). https://doi.org/10.1016/j.neucom.2021.05.004

    Article  Google Scholar 

  27. Peng, W., Varanka, T., Mostafa, A.: et al Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence pp. 1. https://doi.org/10.1109/TPAMI.2021.3136921 (2021c)

  28. Ree, R.: Lie elements and an algebra associated with shuffles. Annal. Math. 210–220 (1958)

  29. Ren, H., Zhang, HB., Shi, Q.: et al Diffnet: Discriminative feature fusion network of multisurface skeleton project images for action recognition. In: 2021 13th International Conference on Machine Learning and Computing, pp. 438–443 (2021)

  30. Ren, Z., Meng, J., Yuan, J.: et al Robust hand gesture recognition with kinect sensor. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 759–760 (2011)

  31. Shi, L., Zhang, Y., Cheng, J.: et al Skeleton-based action recognition with directed graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7912–7921 (2019a)

  32. Shi, L., Zhang, Y., Cheng, J.: et al Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12,026–12,035 (2019b)

  33. Shi, L., Zhang, Y., Cheng, J., et al.: Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Trans. Image Process. 29, 9532–9545 (2020). https://doi.org/10.1109/TIP.2020.3028207

    Article  MATH  Google Scholar 

  34. Song, YF., Zhang, Z., Wang, L.: Richly activated graph convolutional network for action recognition with incomplete skeletons. In: 2019 IEEE International Conference on Image Processing (ICIP), IEEE, pp. 1–5 (2019)

  35. Tang, Y., Tian, Y., Lu, J.: et al Deep progressive reinforcement learning for skeleton-based action recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5323–5332 (2018)

  36. Theodoridis, T., Hu, H.: Action classification of 3d human models using dynamic anns for mobile robot surveillance. In: 2007 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 371–376 (2007)

  37. Vemulapalli, R., Chellapa, R.: Rolling rotations for recognizing human actions from 3d skeletal data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4471–4479 (2016)

  38. Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3d skeletons as points in a lie group. In: Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 588–595 (2014)

  39. Wang, L., Huynh, D.Q., Koniusz, P.: A comparative review of recent kinect-based action recognition algorithms. IEEE Trans. Image Process. 29, 15–28 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  40. Wang, X., Gao, L., Wang, P., et al.: Two-stream 3-d convnet fusion for action recognition in videos with arbitrary size and length. IEEE Trans. Multimedia 20(3), 634–644 (2017)

    Article  Google Scholar 

  41. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Thirty-second AAAI Conference on Artificial Intelligence (2018)

  42. Yang, F., Wu, Y., Sakti, S.: et al Make skeleton-based action recognition model smaller, faster and better. In: Proceedings of the ACM Multimedia asia. pp. 1–6 (2019)

  43. Yang, W., Lyons, TJ., Ni, H.: et al Leveraging the path signature for skeleton-based human action recognition. CoRR abs/1707.03993. http://arxiv.org/abs/1707.03993, (2017)

  44. Zhang, HB., Zhang, YX., Zhong, B.: et al A comprehensive survey of vision-based human action recognition methods. Sensors 19(5). https://doi.org/10.3390/s19051005, https://www.mdpi.com/1424-8220/19/5/1005 (2019)

  45. Zhang, P., Lan, C., Zeng, W.: et al Semantics-guided neural networks for efficient skeleton-based human action recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1112–1121 (2020)

  46. Zhou, Q., Yu, S., Wu, X.: et al Hmms-based human action recognition for an intelligent household surveillance robot. In: 2009 IEEE International Conference on Robotics and Biomimetics (ROBIO), IEEE, pp. 2295–2300 (2009)

Download references

Funding

This work was supported by the Natural Science Foundation of China [No. 61871196, and 61602220]; the Promotion Program for Young and Middle-aged Teacher in Science and Technology Research of Huaqiao University [ZQN-YX601] and the Key Research and Development Program of Jiangxi Province [20202BBEL53019].

Author information

Author notes

  1. H-B. Zhang, H-T. Ren: These authors contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science and Technology, Huaqiao University, Xiamen, 361000, Fujian, China

    Hong-Bo Zhang & Hao-Tian Ren

  2. Key Laboratory of Computer Vision and Machine Learning (Huaqiao University), Fujian Province University, Xiamen 361000, Fujian, China

    Jia-Yu Liang

  3. College of Mathematics and Information Technology, Jiangsu Second Normal University, Nanjing, 210000, Jiangsu, China

    Zhi-Jun Song

  4. Institute of Energy Research, Jiangxi Academy of Sciences, Nanchang, 330096, Jiangxi, China

    Miao-Hui Zhang

Authors
  1. Hong-Bo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao-Tian Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jia-Yu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhi-Jun Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Miao-Hui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H-BZ contributed to conceptualization; H-TR contributed to methodology; J-YL contributed to writing—original draft preparation; H-BZ, Z-JS, and M-HZ contributed to writing—review and editing.

Corresponding author

Correspondence to Miao-Hui Zhang.

Ethics declarations

Conflict of interest

All of authors have no conflict of interest.

Ethics approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, HB., Ren, HT., Liang, JY. et al. Short-term path signature for skeleton-based action recognition. SIViP 17, 1925–1934 (2023). https://doi.org/10.1007/s11760-022-02404-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02404-y

Keywords

Search

Navigation