Skip to main content
SpringerLink
Log in

MDHandNet: a lightweight deep neural network for hand gesture/sign language recognition based on micro-doppler images

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Edge computing is agreed to provide services on the network edge closer to the terminal, including all technologies on the Internet of Things (IoT). Today, various terminals running on the edge of the network, such as mobile phones, notebook computers, handheld game consoles, wearable devices, and the resurgent virtual reality devices, have developed into the main platform for accessing and interacting information. Due to the limitations of size and application environment, these edge terminals need new interaction ways that adapt to them and meet human preferences. As an interactive way, gesture interaction is intelligent, convenient, and intuitive. It can get rid of the limitations of device size and application environment. It is one of the most expressive natural interactive ways of human beings and is most suitable for portable terminals. As a way of communication, gesture is also a non-verbal way of communication for people to express their thoughts and feelings. Sign language is considered as a structured gesture and used as an information communication system. Hand gesture recognition (HGR)/sign language recognition (SLR) technology can make machines understand people’s actions and their meanings. It is the basis of various IoT applications such as gesture interaction and sign language communication services at the edge of the network and has broad application prospects. In this paper, we develop an embedded measurement system for HG/SL actions based on an ultra-wideband radar and measure the radar echo signals of 15-type sign language actions, form a micro-Doppler (MD) image dataset. According to the characteristics of MD images, we propose a novel lightweight network MDHandNet for HGR/SLR, and the recognition accuracy is 97.1%. Compared with other competitive methods, the proposed MDHandNet not only has encouraging performance advantages, and but also has less parameters and lower computational complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Amin, M.G., Zeng, Z., Shan, T.: Hand gesture recognition based on radar micro-doppler signature envelopes. In: 2019 IEEE Radar Conference (RadarConf), pp. 1–6. IEEE (2019)

  2. Andronache, C., Negru, M., Neacsu, A., Cioroiu, G., Radoi, A., Burileanu, C.: Towards extending real-time emg-based gesture recognition system. In: 2020 43rd International Conference on Telecommunications and Signal Processing (TSP), pp. 301–304. IEEE (2020)

  3. Attygalle, N.T., Leiva, L.A., Kljun, M., Sandor, C., Plopski, A., Kato, H., Čopič Pucihar, K.: No interface, no problem: gesture recognition on physical objects using radar sensing. Sensors 21(17), 5771 (2021)

    Article  Google Scholar 

  4. Biao, J., Yu, P., Kuang, X., Zhang, Z.: Dynamic gesture recognition method based on millimeter-wave radar by one-dimensional series neural network. Journal of Electronics and Information 43(9), 2743–2750 (2021)

    Google Scholar 

  5. Chen, Z., Li, G., Fioranelli, F., Griffiths, H.: Dynamic hand gesture classification based on multistatic radar micro-doppler signatures using convolutional neural network. In: 2019 IEEE Radar Conference (RadarConf), pp. 1–5. IEEE (2019)

  6. Chen, V.C., Li, F., Ho, S.S., Wechsler, H.: Micro-doppler effect in radar: phenomenon, model, and simulation study. IEEE Transactions on Aerospace and electronic systems 42(1), 2–21 (2006)

    Article  Google Scholar 

  7. Cheok, M.J., Omar, Z., Jaward, M.H.: A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics 10(1), 131–153 (2019)

    Article  Google Scholar 

  8. Choi, J.W., Ryu, S.J., Kim, J.H.: Short-range radar based real-time hand gesture recognition using lstm encoder. IEEE Access 7, 33610–33618 (2019)

    Article  Google Scholar 

  9. Chollet, F.: Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1251–1258 (2017)

  10. Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., Wei, Y.: Deformable convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 764–773 (2017)

  11. Dekker, B., Jacobs, S., Kossen, A., Kruithof, M., Huizing, A., Geurts, M.: Gesture recognition with a low power fmcw radar and a deep convolutional neural network. In: 2017 European Radar Conference (EURAD), pp. 163–166. IEEE (2017)

  12. García-Bautista, G., Trujillo-Romero, F., Caballero-Morales, S.O.: Mexican sign language recognition using kinect and data time warping algorithm. In: 2017 International Conference on Electronics, Communications and Computers (CONIELECOMP), pp. 1–5. IEEE (2017)

  13. Guo, S., Liu, F., Yuan, X., Zou, C., Chen, L., Shen, T.: Hspog: An optimized target recognition method based on histogram of spatial pyramid oriented gradients. Tsinghua Science and Technology 26(4), 475–483 (2021)

    Article  Google Scholar 

  14. Gupta, S., Molchanov, P., Yang, X., Kim, K., Tyree, S., Kautz, J.: Towards selecting robust hand gestures for automotive interfaces. In: 2016 IEEE Intelligent Vehicles Symposium (IV), pp. 1350–1357. IEEE (2016)

  15. Hazra, S., Santra, A.: Robust gesture recognition using millimetric-wave radar system. IEEE Sensors Letters 2(4), 1–4 (2018)

    Article  Google Scholar 

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  17. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 37(9), 1904–1916 (2015)

    Article  Google Scholar 

  18. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv:1503.02531 (2015)

  19. Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324 (2019)

  20. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 (2017)

  21. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  22. Hu, Y.C., Patel, M., Sabella, D., Sprecher, N., Young, V.: Mobile edge computing–a key technology towards 5g. ETSI White Paper 11(11), 1–16 (2015)

    Google Scholar 

  23. Huu, P.N., Thu, H.N.T.: Proposal gesture recognition algorithm combining cnn for health monitoring. In: 2019 6th NAFOSTED Conference on Information and Computer Science (NICS), pp. 209–213. IEEE (2019)

  24. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(<\)0.5 mb model size. arXiv:1602.07360 (2016)

  25. Information Technology Cloud Computing Edge Computing Landscape. Standards

  26. John, V., Umetsu, M., Boyali, A., Mita, S., Imanishi, M., Sanma, N., Shibata, S.: Real-time hand posture and gesture-based touchless automotive user interface using deep learning. In: 2017 IEEE Intelligent Vehicles Symposium (IV), pp. 869–874. IEEE (2017)

  27. Karam, M.: A Framework for Research and Design of Gesture-Based Human-Computer Interactions. Ph.D. thesis, University of Southampton (2006)

  28. Kaya, F., Tuncer, A.F., Yildiz, Ş.K.: Detection of the turkish sign language alphabet with strain sensor based data glove. In: 2018 26th Signal Processing and Communications Applications Conference (SIU), pp. 1–4. IEEE (2018)

  29. Kim, Y., Toomajian, B.: Application of doppler radar for the recognition of hand gestures using optimized deep convolutional neural networks. In: 2017 11th European Conference on Antennas and Propagation (EUCAP), pp. 1258–1260. IEEE (2017)

  30. Kim, Y., Toomajian, B.: Hand gesture recognition using micro-doppler signatures with convolutional neural network. IEEE Access 4, 7125–7130 (2016)

    Article  Google Scholar 

  31. Köse, Ö.D., Saraçlar, M.: Joint distance and representation learning for sign language videos. In: 2020 28th Signal Processing and Communications Applications Conference (SIU), pp. 1–4. IEEE (2020)

  32. Kulhandjian, H., Sharma, P., Kulhandjian, M., D’Amours, C.: Sign language gesture recognition using doppler radar and deep learning. In: 2019 IEEE Globecom Workshops (GC Wkshps), pp. 1–6. IEEE (2019)

  33. Lee, H.R., Park, J., Suh, Y.J.: Improving classification accuracy of hand gesture recognition based on 60 ghz fmcw radar with deep learning domain adaptation. Electronics 9(12), 2140 (2020)

    Article  Google Scholar 

  34. Lei, W., Soong, A.C., Jianghua, L., Yong, W., Classon, B., Xiao, W., Mazzarese, D., Yang, Z., Saboorian, T.: 5g system architecture. In: 5G System Design, pp. 297–339. Springer (2021)

  35. Li, Y., Wang, X., Shi, B., Zhu, M.: Hand gesture recognition using ir-uwb radar with shufflenet v2. In: Proceedings of the 5th International Conference on Control Engineering and Artificial Intelligence, pp. 126–131 (2021)

  36. Li, G., Zhang, R., Ritchie, M., Griffiths, H.: Sparsity-driven micro-doppler feature extraction for dynamic hand gesture recognition. IEEE Transactions on Aerospace and Electronic Systems 54(2), 655–665 (2017)

    Article  Google Scholar 

  37. Li, B., Yang, J., Yang, Y., Li, C., Zhang, Y.: Sign language/gesture recognition based on cumulative distribution density features using uwb radar. IEEE Transactions on Instrumentation and Measurement 70, 1–13 (2021)

    Google Scholar 

  38. Liu, H., Zhou, A., Dong, Z., Sun, Y., Zhang, J., Liu, L., Ma, H., Liu, J., Yang, N.: M-gesture: person-independent real-time in-air gesture recognition using commodity millimeter wave radar[J]. IEEE Internet of Things Journal, pp. 1–1 (2021)

  39. Liu, W., Wu, G., Ren, F., Kang, X.: Dff-resnet: An insect pest recognition model based on residual networks. Big Data Mining and Analytics 3(4), 300–310 (2020)

    Article  Google Scholar 

  40. Liu, Q., Wu, X., Jiang, Y., Liu, X., Zhang, Y., Xu, X., Qi, L.: A fully connected deep learning approach to upper limb gesture recognition in a secure fes rehabilitation environment. International Journal of Intelligent Systems 36(5), 2387–2411 (2021)

    Article  Google Scholar 

  41. Lu, Y., Lang, Y.: Sign language recognition with cw radar and machine learning. In: 2020 21st International Radar Symposium (IRS), pp. 31–34. IEEE (2020)

  42. Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 116–131 (2018)

  43. Malysa, G., Wang, D., Netsch, L., Ali, M.: Hidden markov model-based gesture recognition with fmcw radar. In: 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1017–1021. IEEE (2016)

  44. Nigam, S., Shamoon, M., Dhasmana, S., Choudhury, T.: A complete study of methodology of hand gesture recognition system for smart homes. In: 2019 International Conference on Contemporary Computing and Informatics (IC3I), pp. 289–294. IEEE (2019)

  45. Park, J., Jang, J., Lee, G., Koh, H., Kim, C., Kim, T.W.: A time domain artificial intelligence radar system using 33-ghz direct sampling for hand gesture recognition. IEEE Journal of Solid-State Circuits 55(4), 879–888 (2020)

    Article  Google Scholar 

  46. Sakamoto, T., Gao, X., Yavari, E., Rahman, A., Boric-Lubecke, O., Lubecke, V.M.: Radar-based hand gesture recognition using iq echo plot and convolutional neural network. In: 2017 IEEE Conference on Antenna Measurements & Applications (CAMA), pp. 393–395. IEEE (2017)

  47. Sakamoto, T., Gao, X., Yavari, E., Rahman, A., Boric-Lubecke, O., Lubecke, V.M.: Hand gesture recognition using a radar echo i-q plot and a convolutional neural network. IEEE Sensors Letters 2(3), 1–4 (2018)

    Article  Google Scholar 

  48. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

  49. Sarhan, N., Frintrop, S.: Transfer learning for videos: from action recognition to sign language recognition. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 1811–1815. IEEE (2020)

  50. Sawasdee, S., Pumrin, S.: Elderly care notification system using hand posture recognition. In: 2014 Fourth International Conference on Digital Information and Communication Technology and its Applications (DICTAP), pp. 304–309. IEEE (2014)

  51. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  52. Skaria, S., Al-Hourani, A., Lech, M., Evans, R.J.: Hand-gesture recognition using two-antenna doppler radar with deep convolutional neural networks. IEEE Sensors Journal 19(8), 3041–3048 (2019)

    Article  Google Scholar 

  53. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  54. Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: platform-aware neural architecture search for mobile. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820–2828 (2019)

  55. Top, G.: Strategic Technology Trends for 2020. David Cearley, Brian Burke (2019)

  56. van Dorp, P., Groen, F.: Human walking estimation with radar. IEE Proceedings-Radar, Sonar and Navigation 150(5), 356–365 (2003)

    Article  Google Scholar 

  57. Victoria, A.H., Maragatham, G.: Gesture recognition of radar micro doppler signatures using separable convolutional neural networks. Materials Today: Proceedings (2021)

  58. Wang, K., Liu, Z., Lin, Y., Lin, J., Han, S.: Haq: Hardware-aware automated quantization with mixed precision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8612–8620 (2019)

  59. Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  60. Yang, L., Li, G.: Sparsity aware dynamic gesture classification using dual-band radar. In: 2018 19th International Radar Symposium (IRS), pp. 1–6. IEEE (2018)

  61. Zhang, J., Shi, Z.: Deformable deep convolutional generative adversarial network in microwave based hand gesture recognition system. In: 2017 9th International Conference on Wireless Communications and Signal Processing (WCSP), pp. 1–6. IEEE (2017)

  62. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6848–6856 (2018)

  63. Zhu, M., Gupta, S.: To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv:1710.01878 (2017)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 62101378 and 62171318.

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China

    Yang Yang, Junhan Li, Beichen Li & Yutong Zhang

Authors
  1. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junhan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Beichen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yutong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beichen Li.

Ethics declarations

Competing interests

The authors declair no conflict of interest.

Additional information

This article belongs to the Topical Collection: Special Issue on Resource Management at the Edge for Future Web, Mobile and IoT Applications

Guest Editors: Qiang He, Fang Dong, Chenshu Wu, and Yun Yang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Li, J., Li, B. et al. MDHandNet: a lightweight deep neural network for hand gesture/sign language recognition based on micro-doppler images. World Wide Web 25, 1951–1969 (2022). https://doi.org/10.1007/s11280-021-00985-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00985-1

Keywords

Search

Navigation