Skip to main content
SpringerLink
Log in

Lightweight micro-motion gesture recognition based on MIMO millimeter wave radar using Bidirectional-GRU network

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Non-contact gesture recognition is a novel form of human–computer interaction. It has broad prospects in many applications, such as Augmented Reality/Virtual Reality, smart homes and intelligent medical systems. Therefore, it has become a research hotspot in recent years. This paper investigates a lightweight micro-motion gesture recognition method based on Multiple Input and Multiple Output millimeter wave radar. We employ TI’s MMWCAS radar, comprising four cascaded AWR1243 radar boards, to collect gesture data. During the data pre-processing stage, we extract the Range-time Map, Doppler-time Map, Azimuth-time Map and Elevation-time Map of the dynamic gestures to characterize the dynamic motion features. These maps are then simplified into a one-dimensional vector to reduce data volume. We propose an 8HBi-GRU model, which combines the Bidirectional Gate Recurrent Unit (Bi-GRU) with a multi-head self-attention mechanism, to identify twelve types of micro-motion gestures using feature vectors. The model achieves an accuracy of 98.24\(\%\), with precision and recall rates exceeding 0.97 and 0.98, respectively, for ten of the gesture types. Experimental results demonstrate that the proposed 8HBi-GRU model achieves lightweight gesture recognition rapidly and requires minimal storage space compared to image-based deep learning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability statement

The datasets generated during the current study are not publicly available because they are self-built by our lab but are available from the corresponding author on reasonable request. The datasets will be open-sourced in the future as the research work of the project is completed.

References

  1. Verdadero MS, Martinez-Ojeda CO, and Dela Cruz JC. Hand gesture recognition system as an alternative interface for remote controlled home appliances. In 2018 IEEE 10th international conference on humanoid, nanotechnology, information technology, communication and control, environment and management (HNICEM), pages 1–5, 2018

  2. Wisener WJ, Rodriguez JD, Ovando A, Woolford C, and Patel K (2023). A top-view hand gesture recognition system for iot applications. In 2023 5th international conference on smart systems and inventive technology (ICSSIT), pages 430–434,

  3. Kim KM, and Choi JI (2019). Passengers gesture recognition model in self-driving vehicles : gesture recognition model of the passengers obstruction of the vision of the driver. In 2019 IEEE 4th international conference on computer and communication systems (ICCCS), pages 239–242

  4. Qi W, Fan H, Xu Y, Su H, and Aliverti A (2022), A 3d-cldnn based multiple data fusion framework for finger gesture recognition in human-robot interaction. In 2022 4th international conference on control and robotics (ICCR), pages 383–387,

  5. Nooruddin N, Dembani R, and Maitlo N (2020) Hgr: hand-gesture-recognition based text input method for ar/vr wearable devices. In 2020 IEEE international conference on systems, man, and cybernetics (SMC), pages 744–751

  6. Breland DS, Dayal A, Jha A, Yalavarthy PK, Pandey OJ, Cenkeramaddi LR (2021) Robust hand gestures recognition using a deep cnn and thermal images. IEEE Sens J 21(23):26602–26614

    Article  Google Scholar 

  7. Zhang W, Wang J, Lan F (2021) Dynamic hand gesture recognition based on short-term sampling neural networks. IEEE/CAA J Automatica Sinica 8(1):110–120

    Article  Google Scholar 

  8. Leon DG, Groli J, Yeduri SR, Rossier D, Mosqueron R, Pandey OJ, Cenkeramaddi LR (2022) Video hand gestures recognition using depth camera and lightweight cnn. IEEE Sens J 22(14):14610–14619

    Article  Google Scholar 

  9. Salami D, Hasibi R, Palipana S, Popovski P, Michoel T, and Sigg S (2022) Tesla-rapture: a lightweight gesture recognition system from mmwave radar sparse point clouds. IEEE Transact Mobile Comput, 1–1,

  10. Ninos A, Hasch J, and Zwick T (2022). Multi-user macro gesture recognition using mmwave technology. In 2021 18th European Radar Conference (EuRAD), pages 37–40. IEEE

  11. Rong Y, Mishra KV, and Bliss DW (2022). Sparse processing for driver respiration monitoring using in-vehicle mmwave radar. In 2022 IEEE/MTT-S international microwave symposium-IMS 2022, pages 440–443. IEEE, 2022

  12. Chang HY, Lin CH, Lin YC, Chung WH, and Lee TS (2020) Dl-aided nomp: a deep learning-based vital sign estimating scheme using fmcw radar. In 2020 IEEE 91st vehicular technology conference (VTC2020-Spring), pages 1–7. IEEE

  13. Zhang J, Wu Y, Chen Y, and Chen T (2020) Health-radio: towards contactless myocardial infarction detection using radio signals. IEEE Transact Mobile Comput, PP(99): 1–1

  14. Zhang J, Yuan W, Chen Y, Wang J, Huang J, Zhang Q (2022) Ubi-fatigue: toward ubiquitous fatigue detection via contactless sensing. IEEE Internet Things J 9(15):14103–14115

    Article  Google Scholar 

  15. Hang S, Ovur SE, Zhou X, Qi W, Ferrigno G, De Momi E (2020) Depth vision guided hand gesture recognition using electromyographic signals. Adv Robot 34(15):985–997

    Article  Google Scholar 

  16. Qi W, Ovur SE, Li Z, Marzullo A, Song R (2021) Multi-sensor guided hand gesture recognition for a teleoperated robot using a recurrent neural network. IEEE Robot Autom Lett 6(3):6039–6045

    Article  Google Scholar 

  17. Tian C, Zhaoyang X, Wang L, Liu Y (2023) Arc fault detection using artificial intelligence: challenges and benefits. Math Biosci Eng 20(7):12404–12432

    Article  Google Scholar 

  18. Dekker B, Jacobs S, Kossen AS, Kruithof MC, Huizing AG, Geurts M (2017) Gesture recognition with a low power fmcw radar and a deep convolutional neural network. In 2017 European radar conference (EURAD), pages 163–166. IEEE

  19. Wang S, Li Z, Huang R, Wang R, Li J, and Xu Z (2020) Hand gesture recognition scheme based on millimeter-wave radar with convolutional neural network. In IET international radar conference (IET IRC 2020), volume , 488–492

  20. Yu JT, Yen L, and Tseng PH (2020) mmwave radar-based hand gesture recognition using range-angle image. In 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring), pages 1–5. IEEE

  21. Zhang G, Lan S, Zhang K, and Ye L (2020) Temporal-range-doppler features interpretation and recognition of hand gestures using mmw fmcw radar sensors. In 2020 14th European conference on antennas and propagation (EuCAP), pages 1–4. IEEE

  22. Liu Z, Liu H, Ma C (2022) A robust hand gesture sensing and recognition based on dual-flow fusion with fmcw radar. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  23. Shi Y, Li L, Yang J, Wang Y, Hao S (2023) Center-based transfer feature learning with classifier adaptation for surface defect recognition. Mech Syst Signal Process 188:110001

    Article  Google Scholar 

  24. Lien J, Nicholas Gillian M, Karagozler E, Amihood P, Schwesig C, Olson E, Raja H, Poupyrev I (2016) Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Transact Graphics (TOG) 35(4):1–19

    Article  Google Scholar 

  25. Wang S, Song J, Lien J, Poupyrev I, and Hilliges O (2016) Interacting with soli: exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th annual symposium on user interface software and technology, pages 851–860

  26. Liu Z, Yang D, Wang Y, Lu M, and Li R (2023) Egnn: graph structure learning based on evolutionary computation helps more in graph neural networks. Appl Soft Comput, page 110040

  27. Wang Y, Liu Z, Xu J, and Yan W (2022) Heterogeneous network representation learning approach for ethereum identity identification. IEEE Transact Comput Soc Syst

  28. Leem SK, Khan F, Cho SH (2020) Detecting mid-air gestures for digit writing with radio sensors and a cnn. IEEE Trans Instrum Meas 69(4):1066–1081

    Article  Google Scholar 

  29. Debajit KVS, Priyanka G, Bhuyan MK (2020) Deep network-based hand gesture recognition using optical flow guided trajectory images. In 2020 IEEE applied signal processing conference (ASPCON), pages 252–256. IEEE

  30. Bhavanasi G, Werthen-Brabants L, Dhaene T, and Couckuyt I (2022) Patient activity recognition using radar sensors and machine learning. Neural Comput Appl, pages 1–16

  31. Kim Y and Toomajian B (2017) Application of doppler radar for the recognition of hand gestures using optimized deep convolutional neural networks. In 2017 11th European conference on antennas and propagation (EUCAP), pages 1258–1260. IEEE

  32. Wu Q, Zhao D et al (2018), Dynamic hand gesture recognition using fmcw radar sensor for driving assistance. In 2018 10th international conference on wireless communications and signal processing (WCSP), pages 1–6. IEEE

  33. Skaria S, Al-Hourani A, Lech M, Evans RJ (2019) Hand-gesture recognition using two-antenna doppler radar with deep convolutional neural networks. IEEE Sens J 19(8):3041–3048

    Article  Google Scholar 

  34. Molchanov P, Gupta S, Kim K, and Pulli K (2015) Short-range fmcw monopulse radar for hand-gesture sensing. In 2015 IEEE radar conference (RadarCon), pages 1491–1496. IEEE

  35. TDang TL, Nguyen HT, Dao DM, Nguyen HV, Luong DL, Nguyen BT, Kim S, and Monet N (2022) Shape: a dataset for hand gesture recognition. Neural Comput Appl, pages 1–14

  36. Ahmed S, Kim W, Park J, Cho SH (2022) Radar-based air-writing gesture recognition using a novel multistream cnn approach. IEEE Internet Things J 9(23):23869–23880

    Article  Google Scholar 

  37. Park G, Chandrasegar VK, Park JG, and Koh J (2022) Increasing accuracy of hand gesture recognition using convolutional neural network. In 2022 International conference on artificial intelligence in information and communication (ICAIIC), pages 251–255. IEEE

  38. Shen X, Zheng H, Feng X, Jinsong H (2022) Ml-hgr-net: a meta-learning network for fmcw radar based hand gesture recognition. IEEE Sens J 22(11):10808–10817

    Article  Google Scholar 

  39. Choi J-W, Ryu S-J, Kim J-H (2019) Short-range radar based real-time hand gesture recognition using lstm encoder. IEEE Access 7:33610–33618

    Article  Google Scholar 

  40. Xia D, Yang N, Jian S, Hu Y, and Li H (2022) Sw-bilstm: a spark-based weighted bilstm model for traffic flow forecasting. Multimedia Tools Appl, pages 1–26

  41. Lv X, Dai C, Liu H, Tian Y, Chen L, Lang Y, Tang R, and He J (2022) Gesture recognition based on semg using multi-attention mechanism for remote control. Neural Comput Appl, pages 1–11

  42. Stove AG (1992) Linear fmcw radar techniques. IEEE Proc Part F 139:343–350

    Google Scholar 

  43. Winkler V (2007) Range doppler detection for automotive fmcw radars. In 2007 European Radar Conference, pages 166–169. IEEE

  44. Robey FC, Coutts S, Weikle D, McHarg JC, and Cuomo K (2004) Mimo radar theory and experimental results. In Conference record of the thirty-eighth asilomar conference on signals, systems and computers, volume 1, pages 300–304. IEEE

  45. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, and Polosukhin I (2017) Attention is all you need. Adv Neural Inform Process Syst, 30

  46. Cho K, Van Merrienboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, and Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. Comput Sci

  47. Alirezazad K and Maurer L (2022) Fmcw radar-based hand gesture recognition using dual-stream cnn-gru model. In 2022 24th international microwave and radar conference (MIKON), pages 1–5. IEEE

  48. Shrestha A, Li H, Le Kernec J, Fioranelli F (2020) Continuous human activity classification from fmcw radar with bi-lstm networks. IEEE Sens J 20(22):13607–13619

    Article  Google Scholar 

  49. Tong L, Ma H, Lin Q, He J, Peng L (2022) A novel deep learning bi-gru-i model for real-time human activity recognition using inertial sensors. IEEE Sens J 22(6):6164–6174

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China under Grant No. 61671185 and 62071153.

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, 150001, China

    Yaqin Zhao, Yuqing Song, Longwen Wu, Ruchen Lv & Hikmat Ullah

  2. 8511 Research Institute, China Aerospace Science and Industry Corporation, Nanjing, 211100, China

    Puqiu Liu

Authors
  1. Yaqin Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuqing Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Longwen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Puqiu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruchen Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hikmat Ullah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Longwen Wu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y., Song, Y., Wu, L. et al. Lightweight micro-motion gesture recognition based on MIMO millimeter wave radar using Bidirectional-GRU network. Neural Comput & Applic 35, 23537–23550 (2023). https://doi.org/10.1007/s00521-023-08978-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08978-z

Keywords

Search

Navigation