Skip to main content
SpringerLink
Log in

STAPointGNN: Spatial-Temporal Attention Graph Neural Network for Gesture Recognition Using Millimeter-Wave Radar

  • Conference paper
  • First Online:
Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2023)

  • 145 Accesses

Abstract

Gesture recognition plays a pivotal role in enabling natural and intuitive human-computer interaction (HCI), finding applications in diverse domains such as smart homes, robot control, and virtual reality. Thanks to advances in computer vision, the most popular method currently is to use the camera for gesture recognition. However, the camera struggles to function properly in poor lighting and inclement weather, and risks invading privacy. Due to the robust and non-invasive features of millimeter-wave radar, gesture recognition based on millimeter-wave radar has received extensive attention from researchers in recent years. In this paper, we propose a novel graph neural network named STAPointGNN for gesture recognition using millimeter-wave radar. In order to better extract features in the spatial and temporal dimensions of point clouds collected by millimeter-wave radar, we designed a spatial-temporal attention mechanism based on graph neural network. We also propose a novel point flow embedding method to capture the motion features of the point clouds in adjacent frames. To verify the superiority of our method, we conduct experiments on two public millimeter-wave radar gesture recognition datasets. The results show that our model outperforms existing mainstream algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/fengxudi/mmWave-gesture-dataset.

  2. 2.

    https://github.com/mmTransGes/mTransSee_Dataset.

References

  1. Adib, F., Kabelac, Z., Katabi, D., Miller, R.C.: 3D tracking via body radio reflections. In: 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), pp. 317–329 (2014)

    Google Scholar 

  2. Ali, A., et al.: End-to-end dynamic gesture recognition using mmWave radar. IEEE Access 10, 88692–88706 (2022)

    Article  Google Scholar 

  3. Van den Bergh, M., et al.: Real-time 3D hand gesture interaction with a robot for understanding directions from humans. In: 2011 Ro-Man, pp. 357–362. IEEE (2011)

    Google Scholar 

  4. Chen, L., Zhang, Y., Peng, L.: METIER: a deep multi-task learning based activity and user recognition model using wearable sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4(1), 1–18 (2020)

    Article  Google Scholar 

  5. Chen, Y., Lymberopoulos, D., Liu, J., Priyantha, B.: FM-based indoor localization. In: Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services, pp. 169–182 (2012)

    Google Scholar 

  6. Desai, S., Desai, A.: Human computer interaction through hand gestures for home automation using microsoft kinect. In: Modi, N., Verma, P., Trivedi, B. (eds.) Human computer interaction through hand gestures for home automation using microsoft kinect. AISC, vol. 508, pp. 19–29. Springer, Singapore (2017). https://doi.org/10.1007/978-981-10-2750-5_3

    Chapter  Google Scholar 

  7. Fang, B., Sun, F., Liu, H., Liu, C.: 3D human gesture capturing and recognition by the IMMU-based data glove. Neurocomputing 277, 198–207 (2018)

    Article  Google Scholar 

  8. Gong, P., Wang, C., Zhang, L.: MMPoint-GNN: graph neural network with dynamic edges for human activity recognition through a millimeter-wave radar. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2021)

    Google Scholar 

  9. Guo, M.H., Cai, J.X., Liu, Z.N., Mu, T.J., Martin, R.R., Hu, S.M.: PCT: Point cloud transformer. Comput. Vis. Media 7, 187–199 (2021)

    Article  Google Scholar 

  10. He, W., Wu, K., Zou, Y., Ming, Z.: WiG: WiFi-based gesture recognition system. In: 2015 24th International Conference on Computer Communication and Networks (ICCCN), pp. 1–7. IEEE (2015)

    Google Scholar 

  11. Hettiarachchi, N., Ju, Z., Liu, H.: A new wearable ultrasound muscle activity sensing system for dexterous prosthetic control. In: 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 1415–1420. IEEE (2015)

    Google Scholar 

  12. Huang, Y., Wang, Y., Shi, K., Gu, C., Fu, Y., Zhuo, C., Shi, Z.: HDNet: hierarchical dynamic network for gait recognition using millimeter-wave radar. In: ICASSP 2023–2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE (2023)

    Google Scholar 

  13. Indra, D., Madenda, S., Wibowo, E.P., et al.: Indonesian sign language recognition based on shape of hand gesture. Procedia Comput. Sci. 161, 74–81 (2019)

    Article  Google Scholar 

  14. Kakoty, N.M., Sharma, M.D.: Recognition of sign language alphabets and numbers based on hand kinematics using a data glove. Procedia Comput. Sci. 133, 55–62 (2018)

    Article  Google Scholar 

  15. Ketykó, I., Kovács, F., Varga, K.Z.: Domain adaptation for semg-based gesture recognition with recurrent neural networks. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–7. IEEE (2019)

    Google Scholar 

  16. Lin, J., Ding, Y.: A temporal hand gesture recognition system based on hog and motion trajectory. Optik 124(24), 6795–6798 (2013)

    Article  Google Scholar 

  17. Liu, H., et al.: mTranssee: enabling environment-independent mmWave sensing based gesture recognition via transfer learning. Proc. ACM Interact. Mobile, Wearable Ubiquitous Technol. 6(1), 1–28 (2022)

    Article  MathSciNet  Google Scholar 

  18. Liu, H., et al.: Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing. Proc. ACM Interact. Mobile, Wearable and Ubiquitous Technol. 4(4), 1–28 (2020)

    Article  Google Scholar 

  19. Liu, H., et al.: M-gesture: Person-independent real-time in-air gesture recognition using commodity millimeter wave radar. IEEE Internet Things J. 9(5), 3397–3415 (2021)

    Article  MathSciNet  Google Scholar 

  20. Liu, Yu., Wang, Y., Liu, H., Zhou, A., Liu, J., Yang, N.: Long-range gesture recognition using millimeter wave radar. In: Yu, Z., Becker, C., Xing, G. (eds.) GPC 2020. LNCS, vol. 12398, pp. 30–44. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-64243-3_3

    Chapter  Google Scholar 

  21. Liu, Z., et al.: Video swin transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3202–3211 (2022)

    Google Scholar 

  22. Lu, Y., Huang, B., Yu, C., Liu, G., Shi, Y.: Designing and evaluating hand-to-hand gestures with dual commodity wrist-worn devices. Proc. ACM Interact., Mobile, Wearable Ubiquitous Technol. 4(1), 1–27 (2020)

    Article  Google Scholar 

  23. Meng, Z., et al.: Gait recognition for co-existing multiple people using millimeter wave sensing. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 849–856 (2020)

    Google Scholar 

  24. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652–660 (2017)

    Google Scholar 

  25. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  26. Radu, V., Henne, M.: Vision2Sensor: knowledge transfer across sensing modalities for human activity recognition. Proc. ACM Interact., Mobile, Wearable Ubiquitous Technol. 3(3), 1–21 (2019)

    Article  Google Scholar 

  27. Sagayam, K.M., Hemanth, D.J.: Hand posture and gesture recognition techniques for virtual reality applications: a survey. Virtual Reality 21, 91–107 (2017)

    Article  Google Scholar 

  28. Sharp, T., et al.: Accurate, robust, and flexible real-time hand tracking. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3633–3642 (2015)

    Google Scholar 

  29. Shi, W., Rajkumar, R.: Point-GNN: graph neural network for 3D object detection in a point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1711–1719 (2020)

    Google Scholar 

  30. Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc. (2014)

    Google Scholar 

  31. Singh, A.D., Sandha, S.S., Garcia, L., Srivastava, M.: RadHAR: human activity recognition from point clouds generated through a millimeter-wave radar. In: Proceedings of the 3rd ACM Workshop on Millimeter-wave Networks and Sensing Systems, pp. 51–56 (2019)

    Google Scholar 

  32. Sun, J.H., Ji, T.T., Zhang, S.B., Yang, J.K., Ji, G.R.: Research on the hand gesture recognition based on deep learning. In: 2018 12th International Symposium on Antennas, Propagation and EM Theory (ISAPE), pp. 1–4. IEEE (2018)

    Google Scholar 

  33. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  34. Wang, C., Gong, P., Zhang, L.: Stpointgcn: spatial temporal graph convolutional network for multiple people recognition using millimeter-wave radar. In: ICASSP 2022–2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3433–3437. IEEE (2022)

    Google Scholar 

  35. Wang, S., Song, J., Lien, J., Poupyrev, I., Hilliges, O.: Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 851–860 (2016)

    Google Scholar 

  36. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (tog) 38(5), 1–12 (2019)

    Article  Google Scholar 

  37. Wang, Y., Wu, K., Ni, L.M.: WiFall: device-free fall detection by wireless networks. IEEE Trans. Mob. Comput. 16(2), 581–594 (2016)

    Article  Google Scholar 

  38. Yan, B., Wang, P., Du, L., Chen, X., Fang, Z., Wu, Y.: mmGesture: semi-supervised gesture recognition system using mmWave radar. Expert Syst. Appl. 213, 119042 (2023)

    Article  Google Scholar 

  39. Yassin, A., Nasser, Y., Al-Dubai, A.Y., Awad, M.: MOSAIC: simultaneous localization and environment mapping using mmWave without a-priori knowledge. IEEE Access 6, 68932–68947 (2018)

    Article  Google Scholar 

  40. Yu, N., Wang, W., Liu, A.X., Kong, L.: QGesture: quantifying gesture distance and direction with WiFi signals. Proc. ACM Interact. Mobile, Wearable Ubiquitous Technol. 2(1), 1–23 (2018)

    Article  Google Scholar 

  41. Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259–16268 (2021)

    Google Scholar 

  42. Zhao, P., et al.: Heart rate sensing with a robot mounted mmWave radar. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2812–2818. IEEE (2020)

    Google Scholar 

Download references

Acknowledgement

This work was supported by National Key R &D Program of China (2021ZD0113502) and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0103).

Author information

Authors and Affiliations

  1. Academy for Engineering and Technology, Fudan University, Shanghai, China

    Jun Zhang, Chunyu Wang, Shunli Wang & Lihua Zhang

  2. Jilin Provincial Key Laboratory of Intelligence Science and Engineering, Changchun, China

    Lihua Zhang

  3. Engineering Research Center of AI and Robotics, Ministry of Education, Shanghai, China

    Jun Zhang & Lihua Zhang

  4. Engineering Research Center of AI and Unmanned Vehicle Systems of Jilin Province, Changchun, China

    Lihua Zhang

Authors
  1. Jun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chunyu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shunli Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lihua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lihua Zhang .

Editor information

Editors and Affiliations

  1. Shanghai University, Shanghai, China

    Honghao Gao

  2. Xi’an Jiaotong-Liverpool, Suzhou, China

    Xinheng Wang

  3. University of Peloponnese, Patra, Greece

    Nikolaos Voros

Rights and permissions

Reprints and permissions

Copyright information

© 2024 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, J., Wang, C., Wang, S., Zhang, L. (2024). STAPointGNN: Spatial-Temporal Attention Graph Neural Network for Gesture Recognition Using Millimeter-Wave Radar. In: Gao, H., Wang, X., Voros, N. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2023. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 562. Springer, Cham. https://doi.org/10.1007/978-3-031-54528-3_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-54528-3_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-54527-6

  • Online ISBN: 978-3-031-54528-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation