Skip to main content
SpringerLink
Log in

Spatial Temporal Attention Graph Convolutional Networks with Mechanics-Stream for Skeleton-Based Action Recognition

  • Conference paper
  • First Online:
Computer Vision – ACCV 2020 (ACCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12626))

Included in the following conference series:

Abstract

The static relationship between joints and the dynamic importance of joints leads to high accuracy in skeletal action recognition. Nevertheless, existing methods define the graph structure beforehand by skeletal patterns, so they cannot capture features considering the relationship between joints specific to actions. Moreover, the importance of joints is expected to be different for each action. We propose spatial-temporal attention graph convolutional networks (STA-GCN). It acquires an attention edge that represents a static relationship between joints for each action and an attention node that represents the dynamic importance of joints for each time. STA-GCN is the first method to consider joint importance and relationship at the same time. The proposed method consists of multiple networks, that reflect the difference of spatial (coordinates) and temporal (velocity and acceleration) characteristics as mechanics-stream. We aggregate these network predictions as final result. We show the potential that the attention edge and node can be easily applied to existing methods and improve the performance. Experimental results with NTU-RGB+D and NTU-RGB+D120 demonstrate that it is possible to obtain a attention edge and node specific to the action that can explain behavior and achieves state-of-the-art performances.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Soo Kim, T., Reiter, A.: Interpretable 3D human action analysis with temporal convolutional networks. In: Computer Vision and Pattern Recognition Workshop (CVPRW) (2017)

    Google Scholar 

  2. Mengyuan, L., Hong, L., Chen, C.: Enhanced skeleton visualization for view invariant human action recognition. Pattern Recogn. 68, 346–362 (2017)

    Article  Google Scholar 

  3. Ke, Q., Bennamoun, M., An, S., Sohel, F., Boussaid, F.: A new representation of skeleton sequences for 3D action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  4. Jun, L., Amir, S., Gang, W., Ling-Yu, D., Alex, K.: Skeleton-based online action prediction using scale selection network. Trans. Pattern Anal. Mach. Intell. (TPAMI) 42, 1453–1467 (2019)

    Google Scholar 

  5. Qiuhong, K., Mohammed, B., Senjian, A., Ferdous, S., Boussaid, F.: Learning clip representations for skeleton-based 3D action recognition. Trans. Image Process. (TIP) 27, 2842–2855 (2018)

    Article  MathSciNet  Google Scholar 

  6. Caetano, C., Sena, J., Brémond, F., dos Santos, J.A., Schwartz, W.R.: SkeleMotion: a new representation of skeleton joint sequences based on motion information for 3D action recognition. In: Advanced Video and Signal-based Surveillance (AVSS) (2019)

    Google Scholar 

  7. Yong, D., Wei, W., Liang, W.: Hierarchical recurrent neural network for skeleton based action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  8. Amir, S., Jun, L., Tian-Tsong, N., Gang, W.: NTU RGB+D: a large scale dataset for 3D human activity analysis. In: Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  9. Liu, J., Shahroudy, A., Xu, D., Wang, G.: Spatio-temporal LSTM with trust gates for 3D human action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 816–833. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_50

    Chapter  Google Scholar 

  10. Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In: Association for the Advancement of Artificial Intelligence (AAAI) (2017)

    Google Scholar 

  11. Jun, L., Amir, S., Dong, X., Alex, K., Gang, W.: Skeleton-based action recognition using spatio-temporal LSTM network with trust gates. Trans. Pattern Anal. Mach. Intell. (TPAMI) 40, 3007–3021 (2018)

    Article  Google Scholar 

  12. Liu, J., Wang, G., Hu, P., Duan, L.Y., Kot, A.C.: Global context-aware attention LSTM networks for 3D action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  13. Jun, L., Gang, W., Duan, L.-Y., Abdiyeva, K., Kot, A.C.: Skeleton-based human action recognition with global context-aware attention LSTM networks. Trans. Image Process. (TIP) 27, 1586–1599 (2018)

    Article  MathSciNet  Google Scholar 

  14. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Association for the Advancement of Artificial Intelligence (AAAI) (2018)

    Google Scholar 

  15. Si, C., Jing, Y., Wang, W., Wang, L., Tan, T.: Skeleton-based action recognition with spatial reasoning and temporal stack learning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 106–121. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_7

    Chapter  Google Scholar 

  16. Thakkar, K., Narayanan, P.J.: Part-based graph convolutional network for action recognition. In: The British Machine Vision Conference (BMVC) (2018)

    Google Scholar 

  17. Maosen, L., Siheng, C., Xu, C., Ya, Z., Yanfeng, W., Tian, Q.: Actional-structural graph convolutional networks for skeleton-based action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  18. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  19. Si, C., Chen, W., Wang, W., Wang, L., Tan, T.: An attention enhanced graph convolutional LSTM network for skeleton-based action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  20. Lei, S., Yifan, Z., Jian, C., Hanqing, L.: Skeleton-based action recognition with directed graph neural networks. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  21. Hussein, M.E., Torki, M., Gowayyed, M.A., El-Saban, M.: Human action recognition using a temporal hierarchy of covariance descriptors on 3D joint locations. In: International Joint Conference on Artificial Intelligence (IJCAI) (2013)

    Google Scholar 

  22. Raviteja, V., Felipe, A., Rama, C.: Human action recognition by representing 3D skeletons as points in a lie group. In: Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  23. Basura, F., Gavves, E., Oramas, M.J., Amir, G., Tinne, T.: Modeling video evolution for action recognition. In: Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  24. Joan, B., Wojciech, Z., Arthur, S., Yann, L.: Spectral networks and locally connected networks on graphs. In: International Conference on Learning Representations (ICLR) (2014)

    Google Scholar 

  25. Michaël, D., Xavier, B., Pierre, V.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in Neural Information Processing Systems (NIPS) (2016)

    Google Scholar 

  26. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (ICLR) (2017)

    Google Scholar 

  27. Duvenaud, D., et al.: Convolutional networks on graphs for learning molecular fingerprints. In: Advances in Neural Information Processing Systems (NIPS) (2015)

    Google Scholar 

  28. Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., Dahl, G.E.: Neural message passing for quantum chemistry. In: International Conference on Machine Learning (ICML) (2017)

    Google Scholar 

  29. Yao, L., Mao, C., Luo, Y.: Graph convolutional networks for text classification. In: Association for the Advancement of Artificial Intelligence (AAAI) (2019)

    Google Scholar 

  30. Xiang, Z., Junbo, Z., Yann, L.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems (NIPS) (2015)

    Google Scholar 

  31. Wang, X., Gupta, A.: Videos as space-time region graphs. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 413–431. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_25

    Chapter  Google Scholar 

  32. Qi, S., Wang, W., Jia, B., Shen, J., Zhu, S.-C.: Learning human-object interactions by graph parsing neural networks. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11213, pp. 407–423. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01240-3_25

    Chapter  Google Scholar 

  33. Yao, T., Pan, Y., Li, Y., Mei, T.: Exploring visual relationship for image captioning. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 711–727. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_42

    Chapter  Google Scholar 

  34. Yang, X., Tang, K., Zhang, H., Cai, J.: Auto-encoding scene graphs for image captioning. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  35. Hiroshi, F., Tsubasa, H., Takayoshi, Y., Hironobu, F.: Attention branch network: learning of attention mechanism for visual explanation. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  36. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  37. R., S.R., Michael, C., Abhishek, D., Ramakrishna, V., Devi, P., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  38. Chattopadhyay, A., Sarkar, A., Howlader, P., Balasubramanian, V.N.: Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks. In: Winter Conference on Applications of Computer Vision (WACV) (2017)

    Google Scholar 

  39. Jun, L., Amir, S., Mauricio, P., Gang, W., Duan, L.Y., Kot, A.C.: NTU RGB+D 120: a large-scale benchmark for 3D human activity understanding. Trans. Pattern Anal. Mach. Intell. (TPAMI) 42, 1–18 (2019)

    Google Scholar 

  40. Hu, J.F., Zheng, W.S., Ma, L., Wang, G., Lai, J., Zhang, J.: Early action prediction by soft regression. Trans. Pattern Anal. Mach. Intell. (TPAMI) 41, 2568–2583 (2019)

    Article  Google Scholar 

  41. Jianfang, H., Wei-Shi, Z., Jian-Huang, L., Jianguo, Z.: Jointly learning heterogeneous features for RGB-D activity recognition. In: Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  42. Liu, M., Yuan, J.: Recognizing human actions as the evolution of pose estimation maps. In: Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  43. Caetano, C., Brémond, F., Schwartz, W.R.: Skeleton image representation for 3D action recognition based on tree structure and reference joints. In: Conference on Graphics, Patterns and Images (SIBGRAPI) (2019)

    Google Scholar 

Download references

Acknowledgement

This paper is based on results obtained from a project, JPNP20006, commissioned by the New Energy and Industrial Technology Development Organization (NEDO).

Author information

Authors and Affiliations

  1. Chubu University, 1200 Matsumotocho, Kasugai-shi, Aichi, Japan

    Katsutoshi Shiraki, Tsubasa Hirakawa, Takayoshi Yamashita & Hironobu Fujiyoshi

Authors
  1. Katsutoshi Shiraki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tsubasa Hirakawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Takayoshi Yamashita
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hironobu Fujiyoshi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katsutoshi Shiraki .

Editor information

Editors and Affiliations

  1. Waseda University, Tokyo, Japan

    Hiroshi Ishikawa

  2. Institute of Automation of Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

  3. Czech Technical University in Prague, Prague, Czech Republic

    Tomas Pajdla

  4. University of Pennsylvania, Philadelphia, PA, USA

    Jianbo Shi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shiraki, K., Hirakawa, T., Yamashita, T., Fujiyoshi, H. (2021). Spatial Temporal Attention Graph Convolutional Networks with Mechanics-Stream for Skeleton-Based Action Recognition. In: Ishikawa, H., Liu, CL., Pajdla, T., Shi, J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science(), vol 12626. Springer, Cham. https://doi.org/10.1007/978-3-030-69541-5_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-69541-5_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-69540-8

  • Online ISBN: 978-3-030-69541-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation