Skip to main content
SpringerLink
Log in

A novel sEMG-based dynamic hand gesture recognition approach via residual attention network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the emergence of more and more lightweight, convenient and cheap surface electromyography signal (sEMG) snsors, gesture recognition based on sEMG sensors has attracted much attention of researchers. In this study, combined with the sEMG sensor, a novel dynamic hand gesture recognition approach is proposed for effective and accurate dynamic gesture prediction. Here, a portable sEMG sensor (Myo wristband) is adopted to acquire the multi-channel sEMG signals of dynamic hand gestures and the continuous wavelet transformation (CWT) is proposed for data preprocessing to acquire the time-frequency maps. Due to the success of powerful contextual feature representation capability of deep convolutional neural networks (DCNNs), a deep residual attention network is proposed for accurate prediction of time-frequency maps. To effectively extract the key spatial and channel features from multi-channel sEMG signals, a residual attention network is proposed to act as the backbone network for effective feature representation. Besides, In the proposed recognition network, a multi-scale feature enhancement (MFE) module and an attention fusion block (AFB) are proposed, which respectively improve the multi-scale expression ability of the network and effectively realize multi-scale feature enhancement, respectively. Experimental results show that the proposed recognition network could achieve a superior detection ability compared with other state-of-the-art recognition models. The source code and dataset are available at https://github.com/lyangucas92/Ges_Net.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availibility

The data that support the findings of this study are available upon reasonable request.

Notes

  1. PaddlePaddle: https://www.paddlepaddle.org.cn/

References

  1. Allard UC, Nougarou F, Fall CL, Giguère P, Gosselin C, Laviolette F, Gosselin B (2016) A convolutional neural network for robotic arm guidance using semg based frequency-features. In Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp 2464–2470

  2. Al-Timemy AH, Khushaba RN, Bugmann G, Escudero J (2015) Improving the performance against force variation of emg controlled multifunctional upper-limb prostheses for transradial amputees. IEEE Trans Neural Syst Rehabil Eng 24(6):650–661

    Article  Google Scholar 

  3. Chen L, Fu J, Wu Y, Li H, Zheng B (2020) Hand gesture recognition using compact cnn via surface electromyography signals. Sensors 20(3):672

    Article  Google Scholar 

  4. Cheng H, Yang L, Liu Z (2015) Survey on 3d hand gesture recognition. IEEE Transactions on Circuits and Systems for Video Technology 26(9):1659–1673

    Article  Google Scholar 

  5. Côté-Allard U, Fall CL, Campeau-Lecours A, Gosselin C, Laviolette F, Gosselin B (2017) Transfer learning for semg hand gestures recognition using convolutional neural networks. In Proceedings of IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 1663–1668

  6. Dong Y, Liu J, Yan W (2021) Dynamic hand gesture recognition based on signals from specialized data glove and deep learning algorithms. IEEE Trans Instrum Meas 70:1–14

    Google Scholar 

  7. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

  8. Duan F, Ren X, Yang Y (2018) A gesture recognition system based on time domain features and linear discriminant analysis. IEEE Transactions on Cognitive and Developmental Systems 13(1):200–208

    Article  Google Scholar 

  9. Fatimah B, Singh P, Singhal A, Pachori RB (2021) Hand movement recognition from semg signals using fourier decomposition method. Biocybernetics Biomed Eng 41(2):690–703

    Article  Google Scholar 

  10. Feng Q, Yang C, Wu X, Li Z (2013) A smart tv interaction system based on hand gesture recognition by using rgb-d sensor. In Proceedings of International Conference on Mechatronic Sciences, Electric Engineering and Computer (MEC). IEEE, pp 1319–1322

  11. Gao Q, Chen Y, Ju Z, Liang Y (2021) Dynamic hand gesture recognition based on 3d hand pose estimation for human-robot interaction. IEEE Sensors J

  12. Ghotkar AS, Khatal R, Khupase S, Asati S, Hadap M (2012) Hand gesture recognition for indian sign language. In Proceedings of International Conference on Computer Communication and Informatics. IEEE, pp 1–4

  13. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 770–778

  14. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861

  15. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 4700–4708

  16. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 7132–7141

  17. Jiang S, Gao Q, Liu H, Shull PB (2020) A novel, co-located emgfmg-sensing wearable armband for hand gesture recognition. Sensors Actuators A Phys 301:111738

    Article  Google Scholar 

  18. Joseph K, Khan S, Khan FS, Balasubramanian VN (2021) Towards open world object detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 5830–5840

  19. Kang P, Li J, Fan B, Jiang S, Shull PB (2021) Wrist-worn hand gesture recognition while walking via transfer learning. IEEE J Biomed Health Inform

  20. Khodabandelou G, Jung P-G, Amirat Y, Mohammed S (2020) Attention-based gated recurrent unit for gesture recognition. IEEE Trans Autom Sci Eng 18(2):495–507

    Article  Google Scholar 

  21. Lee S, Hara S, Yamada Y (2009) A safety measure for control mode switching of skill-assist for effective automotive manufacturing. IEEE Trans Autom Sci Eng 7(4):817–825

    Article  Google Scholar 

  22. Liu K, Chen C, Jafari R, Kehtarnavaz N (2014) Fusion of inertial and depth sensor data for robust hand gesture recognition. IEEE Sensors J 14(6):1898–1903

    Article  Google Scholar 

  23. Liu M-K, Lin Y-T, Qiu Z-W, Kuo C-K, Wu C-K (2020) Hand gesture recognition by a mmg-based wearable device. IEEE Sensors J 20(24):14 703--14 712

    Article  Google Scholar 

  24. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp 10 012–10 022

  25. Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025

  26. Mehta S, Rastegari M (2021) Mobilevit: light-weight, generalpurpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178

  27. Nahid N, Rahman A, Ahad M (2020) Deep learning based surface emg hand gesture classification for low-cost myoelectric prosthetic hand. In Proceedings of Joint 9th International Conference on Informatics, Electronics & Vision (ICIEV) and 4th International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, pp 1–8

  28. Altilio R, Panella M (2018) A smartphone-based application using machine learning for gesture recognition: Using feature extraction and template matching via hu image moments to recognize gestures. IEEE Consum Electron Mag 8(1):25–29

  29. Phinyomark A, Phukpattaranont P, Limsakul C (2012) Feature reduction and selection for emg signal classification. Expert Syst Appl 39(8):7420–7431

    Article  Google Scholar 

  30. Phinyomark A, Scheme E (2018) An investigation of temporally inspired time domain features for electromyographic pattern recognition. In Proceedings of 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 5236–5240

  31. Plouffe G, Cretu A-M (2015) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316

    Article  Google Scholar 

  32. Qureshi SA, Raza SEA, Hussain L, Malibari AA, Nour MK, Rehman Au, Al-Wesabi FN, Hilal AM (2022) Intelligent ultralight deep learning model for multi-class brain tumor detection. Appl Sci 12(8):3715

    Article  Google Scholar 

  33. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimedia 15(5):1110–1120

    Article  Google Scholar 

  34. Rioul O, Duhamel P (1992) Fast algorithms for discrete and continuous wavelet transforms. IEEE Trans Inf Theory 38(2):569–586

    Article  MathSciNet  Google Scholar 

  35. Sharma S, Singh S (2021) Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Syst Appl 182:115657

    Article  Google Scholar 

  36. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  37. Song W, Han Q, Lin Z, Yan N, Luo D, Liao Y, Zhang M, Wang Z, Xie X, Wang A et al (2019) Design of a flexible wearable smart semg recorder integrated gradient boosting decision tree based hand gesture recognition. IEEE Trans Biomed Circ Syst 13(6):1563–1574

    Article  Google Scholar 

  38. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 1–9

  39. Tan M, Pang R, Le QV (2020) Efficientdet: Scalable and efficient object detection. In Proc IEEE/CVF Conf Comput Vis Pattern Recognit (CVPR), pp 10 781–10 790

  40. Wei W, Dai Q, Wong Y, Hu Y, Kankanhalli M, Geng W (2019) Surface-electromyography-based gesture recognition by multi-view deep learning. IEEE Trans Biomed Eng 66(10):2964–2973

    Article  Google Scholar 

  41. Wen R, Wang Q, Li Z (2021) Human hand movement recognition using infinite hidden markov model based semg classification. Biomed Signal Process Control 68:102592

    Article  Google Scholar 

  42. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp 3–19

  43. Xie S, Girshick R, Dollár P, Tu Z, He K (2017) Aggregated residual transformations for deep neural networks. In Proc IEEE Conf Comput Vis Pattern Recognit (CVPR), pp 1492–1500

  44. Yang Y, Duan F, Ren J, Xue J, Lv Y, Zhu C, Yokoi H (2020) Performance comparison of gesture recognition system based on different classifiers. IEEE Trans Cogn Dev Syst 13(1):141–150

    Article  Google Scholar 

  45. Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the kinect sensor. IEEE Transactions on Circuits and Systems for Video Technology 24(11):1935–1944

    Article  Google Scholar 

  46. Yuan G, Liu X, Yan Q, Qiao S, Wang Z, Yuan L (2020) Hand gesture recognition using deep feature fusion network based on wearable sensors. IEEE Sensors J 21(1):539–547

    Google Scholar 

  47. Yu F, Koltun V (2015) Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122

  48. Zhang X, Chen X, Li Y, Lantz V, Wang K, Yang J (2011) A framework for hand gesture recognition based on accelerometer and emg sensors. IEEE Trans Syst Man Cybern Syst Hum 41(6):1064–1076

    Article  Google Scholar 

  49. Zhang Y, Cao C, Cheng J, Lu H (2018) Egogesture: A new dataset and benchmark for egocentric hand gesture recognition. IEEE Trans Multimedia 20(5):1038–1050

    Article  Google Scholar 

Download references

Acknowledgements

All authors would like to thank the anonymous referees for their valuable suggestions and comments.

Funding

This work was supported by the National Key Research & Development Project of China (2020-YFB1313701), the National Natural Science Foundation of China (No. 62003309) and Outstanding Foreign Scientist Support Project in Henan Province of China (No. GZS2019008).

Author information

Authors and Affiliations

  1. School of Electrical and Information Engineering, Zhengzhou University, 450001, Henan Province, China

    Yanhong Liu, Xingyu Li, Hongnian Yu & Lei Yang

  2. Root Perception and Control Engineering Laboratory, 450001, Henan Province, China

    Yanhong Liu, Xingyu Li & Lei Yang

  3. Built Environment, Edinburgh Napier University, EH10 5DT, Edinburgh, UK

    Hongnian Yu

Authors
  1. Yanhong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hongnian Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lei Yang.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Li, X., Yu, H. et al. A novel sEMG-based dynamic hand gesture recognition approach via residual attention network. Multimed Tools Appl 83, 9329–9349 (2024). https://doi.org/10.1007/s11042-023-15748-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15748-5

Keywords

Search

Navigation