Skip to main content
SpringerLink
Log in

Serial Spatial and Temporal Transformer for Point Cloud Sequences Recognition

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14495))

Included in the following conference series:

  • 207 Accesses

Abstract

Point cloud sequences are unordered and irregular, which means extracting spatial and temporal features from them is challenging. This paper presents a novel network named Serial Spatial and Temporal Transformer (SerialSTTR) for point cloud sequences recognition. Specifically, point-based self-attention is used to gather global information on each point at the spatial level, and frame-based self-attention is used to reconstruct the sequences with motion features at the temporal level. In addition, an orderly local module is proposed to supplement the local feature learning ability that spatial transformer lacks. And relative position encoding is adopted to complete the order information for temporal transformer. Extensive experiments demonstrate that the SerialSTTR achieves the state-of-the-art performance on 3D human action recognition with the challenging dataset MSR-Action3D. And to show its generalizability, experiments on gesture recognition with SHREC’17 dataset are performed, which also present competitive results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aouaidjia, K., Sheng, B., Li, P., Kim, J., Feng, D.D.: Efficient body motion quantification and similarity evaluation using 3-d joints skeleton coordinates. IEEE Trans. Syst. Man Cybern. Syst. 51(5), 2774–2788 (2019)

    Article  Google Scholar 

  2. Chen, L., Zhang, Q.: Ddgcn: graph convolution network based on direction and distance for point cloud learning. Vis. Comput. 39(3), 863–873 (2023)

    Article  Google Scholar 

  3. Chen, Y., Zhao, L., Peng, X., Yuan, J., Metaxas, D.: Construct dynamic graphs for hand gesture recognition via spatial-temporal attention. In: Proceedings of the British Machine Vision Conference (BMVC), pp. 48.1-48.13 (2019)

    Google Scholar 

  4. De Smedt, Q., Wannous, H., Vandeborre, J.P.: Skeleton-based dynamic hand gesture recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–9 (2016)

    Google Scholar 

  5. De Smedt, Q., Wannous, H., Vandeborre, J.P., Guerry, J., Le Saux, B., Filliat, D.: Shrec’17 track: 3d hand gesture recognition using a depth and skeletal dataset. In: 3DOR-10th Eurographics Workshop on 3D Object Retrieval, pp. 1–6 (2017)

    Google Scholar 

  6. Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

  7. Fan, H., Yang, Y.: Pointrnn: point recurrent neural network for moving point cloud processing. arXiv preprint arXiv:1910.08287 (2019)

  8. Fan, H., Yang, Y., Kankanhalli, M.: Point 4d transformer networks for spatio-temporal modeling in point cloud videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14204–14213 (2021)

    Google Scholar 

  9. Fan, H., Yu, X., Ding, Y., Yang, Y., Kankanhalli, M.: Pstnet: point spatio-temporal convolution on point cloud sequences. arXiv preprint arXiv:2205.13713 (2022)

  10. Guo, M.H., Cai, J.X., Liu, Z.N., Mu, T.J., Martin, R.R., Hu, S.M.: Pct: point cloud transformer. Comput. Visual Media 7, 187–199 (2021)

    Article  Google Scholar 

  11. He, P., Emami, P., Ranka, S., Rangarajan, A.: Learning scene dynamics from point cloud sequences. Int. J. Comput. Vision 130(3), 669–695 (2022)

    Article  Google Scholar 

  12. Himeur, C.E., Lejemble, T., Pellegrini, T., Paulin, M., Barthe, L., Mellado, N.: Pcednet: a lightweight neural network for fast and interactive edge detection in 3d point clouds. ACM Trans. Graph. (TOG) 41(1), 1–21 (2021)

    Article  Google Scholar 

  13. Hou, J., Wang, G., Chen, X., Xue, J.H., Zhu, R., Yang, H.: Spatial-temporal attention res-tcn for skeleton-based dynamic hand gesture recognition. In: Computer Vision - ECCV 2018 Workshops, pp. 273–286 (2019)

    Google Scholar 

  14. Li, W., Zhang, Z., Liu, Z.: Action recognition based on a bag of 3d points. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 9–14. IEEE (2010)

    Google Scholar 

  15. Li, X., Huang, Q., Wang, Z., Yang, T.: Virtualactionnet: a strong two-stream point cloud sequence network for human action recognition. J. Vis. Commun. Image Represent. 89, 103641 (2022)

    Article  Google Scholar 

  16. Lin, L., Huang, P., Fu, C.W., Xu, K., Zhang, H., Huang, H.: On learning the right attention point for feature enhancement. Sci. China Inf. Sci. 66(1), 1–13 (2023)

    Article  Google Scholar 

  17. Lin, X., Sun, S., Huang, W., Sheng, B., Li, P., Feng, D.D.: Eapt: efficient attention pyramid transformer for image processing. IEEE Trans. Multimed., 50–61 (2021)

    Google Scholar 

  18. Liu, J., Xu, D.: Geometrymotion-net: a strong two-stream baseline for 3d action recognition. IEEE Trans. Circuits Syst. Video Technol. 31(12), 4711–4721 (2021)

    Article  Google Scholar 

  19. Liu, X., Yan, M., Bohg, J.: Meteornet: deep learning on dynamic 3d point cloud sequences. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9246–9255 (2019)

    Google Scholar 

  20. Lu, H., Nie, J.: Coarse registration of point cloud base on deep local extremum detection and attentive description. Available at SSRN 4106324

    Google Scholar 

  21. Luo, W., Yang, B., Urtasun, R.: Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 3569–3577 (2018)

    Google Scholar 

  22. Min, Y., Zhang, Y., Chai, X., Chen, X.: An efficient pointlstm for point clouds based gesture recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5761–5770 (2020)

    Google Scholar 

  23. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

    Google Scholar 

  24. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. Adv. Neural. Inf. Process. Syst. 30, 5099–5108 (2017)

    Google Scholar 

  25. Riegler, G., Osman Ulusoy, A., Geiger, A.: Octnet: learning deep 3d representations at high resolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3577–3586 (2017)

    Google Scholar 

  26. Tu, Z., et al.: A survey of variational and cnn-based optical flow techniques. Sig. Process. Image Commun. 72, 9–24 (2019)

    Article  Google Scholar 

  27. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł, Polosukhin, I.: Attention is all you need. Adv. Neural. Inf. Process. Syst. 30, 6000–6010 (2017)

    Google Scholar 

  28. Vieira, A.W., Nascimento, E.R., Oliveira, G.L., Liu, Z., Campos, M.F.M.: STOP: space-time occupancy patterns for 3D action recognition from depth map sequences. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 252–259. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33275-3_31

    Chapter  Google Scholar 

  29. Wang, J., Liu, Z., Wu, Y., Yuan, J.: Mining actionlet ensemble for action recognition with depth cameras. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1290–1297. IEEE (2012)

    Google Scholar 

  30. Wang, Y., et al.: 3dv: 3D dynamic voxel for action recognition in depth video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 511–520 (2020)

    Google Scholar 

  31. Wei, Y., Liu, H., Xie, T., Ke, Q., Guo, Y.: Spatial-temporal transformer for 3d point cloud sequences. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1171–1180 (2022)

    Google Scholar 

  32. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, pp. 7444–7452 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering and Science, Shanghai University, Shanghai, 200444, China

    Shiqi Zou & Jingqiao Zhang

Authors
  1. Shiqi Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingqiao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jingqiao Zhang .

Editor information

Editors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  2. Shanghai Jiao Tong University, Shanghai, China

    Lei Bi

  3. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  4. MIRALab-CUI, University of Geneve, Carouge, Geneve, Switzerland

    Nadia Magnenat-Thalmann

  5. Swiss Federal Institute of Technology, Lausanne, Switzerland

    Daniel Thalmann

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zou, S., Zhang, J. (2024). Serial Spatial and Temporal Transformer for Point Cloud Sequences Recognition. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14495. Springer, Cham. https://doi.org/10.1007/978-3-031-50069-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50069-5_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50068-8

  • Online ISBN: 978-3-031-50069-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation