Skip to main content

Advertisement

SpringerLink
Log in

An approach to continuous hand movement recognition using SEMG based on features fusion

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

With the continuous development of EMG acquisition technology and artificial intelligence technology, EMG signal analysis has been extensively studied in human–computer interaction, rehabilitation training, prosthetic control and remote device control. As hand movements become more and more complex, hand movement recognition based on surface electromyography (sEMG) has become a hotspot. In this paper, by using multi-features fusion-based Long Short-Term Memory convolutional neural network (MFFCNN-LSTM), a continuous hand movement recognition method based on time-domain and time–frequency-spectrum features of forearm sEMG signal is proposed. Ten basic hand movements including rest action are identified. Firstly, the hand movement data is cut from NinaPro db8 dataset to extract effective sEMG signal fragments. Secondly, the empirical Fourier decomposition method is used to denoise the sEMG signals. Thirdly, the time-domain and time–frequency-spectrum features of sEMG signals from different channels are extracted, and sent to two parallel CNN networks to extract the high-dimension features, respectively. Fourthly, the high-dimension features are fused as the input of LSTM, a fully connected layer and a softmax layer to recognize the continuous hand movements. Finally, MFFCNN-LSTM is compared with the support vector machine, CNN and LSTM on the same computer. The experimental results show that the recognition accuracy, sensitivity and specificity of MFFCNN-LSTM on NinaPro db8 dataset are 98.5%, 95.25% and 95.5%, respectively. It has higher recognition accuracy on the five public datasets than other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Park, H., An, B., Baek, J., Lee, D., Seo, J.: Selection of grasping target and control system of robotic prosthetic hand using images and deep learning. J. Inst. Control 26(5), 312–317 (2020)

    Google Scholar 

  2. Rheem, H., Becker, D.V., Craig, S.D.: Assessing learning effort with hand motion tracking methods. Appl. Cogn. Psychol. 35, 606–620 (2021). https://doi.org/10.1002/acp.3784

    Article  Google Scholar 

  3. Vishwakarma, D.K., and Grover, V.: Hand gesture recognition in low-intensity environment using depth images, in Editor (Ed.)^(Eds.): Book Hand gesture recognition in low-intensity environment using depth images, 429–433(2017, edn.)

  4. Jain, R., Karsh, R.K., Barbhuiya, A.A.: Encoded motion image-based dynamic hand gesture recognition. Vis. Comput. (2021). https://doi.org/10.1007/s00371-021-02259-3

    Article  Google Scholar 

  5. Zhang W, Wang J, Lan F (2021) Dynamic hand gesture recognition based on short-term sampling neural networks, IEEE/CAA J. Autom. Sin., 8, (01), 114-124. DOI: https://doi.org/10.1109/JAS.2020.1003465

  6. Tan, Y.S., Lim, K.M., Tee, C., Lee, C.P., Low, C.Y.: Convolutional neural network with spatial pyramid pooling for hand gesture recognition. Neural Comput. Appl. 33, 5339–5351 (2021). https://doi.org/10.1007/s00521-020-05337-0

    Article  Google Scholar 

  7. Li, H., Wu, L., Wang, H., Han, C., Quan, W., Zhao, J.: Hand gesture recognition enhancement based on spatial fuzzy matching in leap motion. IEEE Trans. Ind. Inf. 16(3), 1885–1894 (2020). https://doi.org/10.1109/TII.2019.2931140

    Article  Google Scholar 

  8. Wang, B., Li, Y., Lang, H., Wang, Y.: Hand gesture recognition and motion estimation using the kinect sensor. Control. Intell. Syst. 48(1), 17–24 (2020). https://doi.org/10.2316/J.2020.201-0014

    Article  Google Scholar 

  9. Lee, C., Kim, J., Cho, S., Kim, J., and Kwon, S.: Development of Real-Time Hand Gesture Recognition for Tabletop Holographic Display Interaction Using Azure Kinect, Sensors, 20(16), 4566(2020) DOI: https://doi.org/10.3390/s20164566

  10. Bilal, S., Akmeliawati, R., Shafie, A.A., Salami, M.J.E.: Hidden Markov model for human to computer interaction: a study on human hand gesture recognition. Artif. Intell. Rev. 40(4), 495–516 (2013). https://doi.org/10.1007/s10462-011-9292-0

    Article  Google Scholar 

  11. Chen, G., and Ge, K.: A fusion recognition method based on multifeature hidden markov model for dynamic hand gesture, Comput. Intell. Neurosci., 2020 (12), 8871605(2020). https://doi.org/10.1155/2020/8871605

  12. Jaramillo-Yanez, A., Unapanta, L., and Benalcázar, M.E.: Short-Term Hand Gesture Recognition using Electromyography in the Transient State, Support Vector Machines, and Discrete Wavelet Transform, in Editor (Ed.)^(Eds.): Book Short-Term Hand Gesture Recognition using Electromyography in the Transient State, Support Vector Machines, and Discrete Wavelet Transform (2020, edn.)

  13. Winarno, H.A., Poernama, A.I., Soesanti, I., and Nugroho, H.A.: Evaluation on EMG electrode reduction in recognizing the pattern of hand gesture by using SVM Method, J. Phys. Conf. Ser., 1577 (1), 012044 (2019)

  14. Ahsan, M.R., Ibrahimy, M.I., and Khalifa, O.O.: Electromygraphy (EMG) signal based hand gesture recognition using artificial neural network (ANN), in Editor (Ed.)^(Eds.): Book Electromygraphy (EMG) signal based hand gesture recognition using artificial neural network (ANN) (2011, edn.)

  15. Saeed, B., Zia-ur-Rehman, M., Gilani, S.O., Amin, F., Waris, A., Jamil, M., Shafique, M.: Leveraging ANN and LDA Classifiers for characterizing different hand movements using EMG signals. Arab. J. Sci. Eng. 46, 1761–1769 (2021). https://doi.org/10.1007/s13369-020-05044-x

    Article  Google Scholar 

  16. Pradeep Kumar, B.P., Manjunatha, M.B.: Performance analysis of KNN, SVM and ANN techniques for gesture recognition system, Indian J. Sci. Technol., 9, (S1)(2016). DOI: https://doi.org/10.17485/ijst/2016/v9is1/111145

  17. Pinzón-Arenas, J.O., Jiménez-Moreno, R., and Herrera-Benavides, J.E.: Convolutional neural network for hand gesture recognition using 8 different EMG Signals, in Editor (Eds.): Book Convolutional neural network for hand gesture recognition using 8 different EMG Signals (2019, edn.)

  18. Velandia, N.S., Moreno, R.J., Rubiano, A.: CNN architectures for hand gesture recognition using EMG signals throw wavelet feature extraction. J. Eng. Appl. Sci. 14(11), 3528–3537 (2019). https://doi.org/10.36478/jeasci.2019.3528.3537

    Article  Google Scholar 

  19. Asif, A.R., Waris, A., Gilani, S.O., Jamil, M., Ashraf, H., Shafique, M., and Niazi, I.K.: Performance evaluation of convolutional neural network for hand gesture recognition using EMG, Sensors (Basel, Switzerland), 20(6)(2020). DOI: https://doi.org/10.3390/s20061642

  20. Kolla, B.P.: Accurate hand gesture recognition using CNN and RNN approaches. Int. J. Adv. Trends Computer Sci. Eng. 9(3), 3216–3222 (2020). https://doi.org/10.30534/ijatcse/2020/114932020

    Article  Google Scholar 

  21. Xing, Y., Caterina, G.D., Soraghan, J.: A new spiking convolutional recurrent neural network (SCRNN) with applications to event-based hand gesture recognition. Front. Neurosci. 14, 590164 (2020). https://doi.org/10.3389/fnins.2020.590164

    Article  Google Scholar 

  22. Jabbari, M., Khushaba, R.N., and Nazarpour, K.: EMG-based hand gesture classification with long short-term memory deep recurrent neural networks, 2020 42nd annual international conference of the IEEE engineering in medicine and biology society (EMBC), 3302–3305(2020)

  23. LeCun, Boser, Denker, Henderson, Howard, Hubbard, and Jackel: Backpropagation applied to handwritten Zip Code Recognition, Neural Comput. (1989). DOI: https://doi.org/10.1162/neco.1989.1.4.541

  24. Chen, H.F., Tong, R.Z., Chen, M.J., Fang, Y.F., Liu, H.H., and Ieee: A hybrid cnn-svm classifier for hand gesture recognition with surface emg signals: Proceedings of 2018 International Conference on Machine Learning and Cybernetics (Ieee, 2018), 619–624(2018).

  25. Too, J., Abdullah, A.R., Saad, N.M., Ali, N.M., Zawawi, T.N.S.T.: Deep convolutional neural network for featureless electromyogram pattern recognition using time-frequency distribution. Sens. Lett. 16(8), 92–99 (2018). https://doi.org/10.1166/sl.2018.3926

    Article  Google Scholar 

  26. Jordan, M.I. (1986) Attractor Dynamics and parallellism in a connectionist sequential machine, Proc.annu.conf.of the Cognitive Science Society Amherst Ma, 531–546.

  27. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzziness Knowledge Based Syst. 06(02), 107–116 (1998). https://doi.org/10.1142/S0218488598000094

    Article  MATH  Google Scholar 

  28. Krasoulis, A., Vijayakumar, S., Nazarpour, K.: Effect of user practice on prosthetic finger control with an intuitive myoelectric decoder. Front. Neurosci. 13, 891 (2010). https://doi.org/10.3389/fnins.2019.00891

    Article  Google Scholar 

  29. Astuti, W., Sediono, W., Aibinu, A.M., Akmeliawati, R., and Salami, M.J.E.: Adaptive Short Time Fourier Transform (STFT) Analysis of seismic electric signal (SES): A comparison of Hamming and rectangular window, 2012 IEEE Symposium on Industrial Electronics and Applications, 372–379 (2012)

  30. Sbrollini, A., Agostinelli, A., Di Nardo, F., Maranesi, E., Mengarelli, A., Fioretti, S., and Burattini, L.: Evaluation of the Low-Frequency Components in Surface Electromyography, in Patton, J., Barbieri, R., Ji, J., Jabbari, E., Dokos, S., Mukkamala, R., Guiraud, D., Jovanov, E., Dhaher, Y., Panescu, D., Vangils, M., Wheeler, B., and Dhawan, A.P. (Eds.): 2016 38th Annual International Conference of the Ieee Engineering in Medicine and Biology Society , 3622–3625 (2016)

  31. Singh, P., Joshi, S.D., Patney, R.K., Saha, K.: The Fourier decomposition method for nonlinear and non-stationary time series analysis Proc. R. Soc. A. 473, 20160871 (2017). https://doi.org/10.1098/rspa.2016.0871

    Article  MATH  Google Scholar 

  32. Zhou W , Feng Z, Xu Y F , et al. Empirical Fourier Decomposition: An Accurate Adaptive Signal Decomposition Method, arXiv: Signal Processing: (2020)

  33. Ahmed, Bamakhramah, Soliman, A., Mahmoud, Saeed, A., and Al-Tunaiji: Six Order Cascaded Power Line Notch Filter for ECG Detection Systems with Noise Shaping, Circuits Syst. Signal Process 33, 2385–2400 (2014). https://doi.org/10.1007/s00034-014-9761-1

  34. Huang, Norden E., Shen, Samuel S.P.: Hilbert-Huang transform and its applications. World Scientific. 16 (2014). https://doi.org/10.1142/5862

  35. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  36. Wang, C., Guo, W., Zhang, H., Guo, L., Lin, C.: sEMG-based continuous estimation of grasp movements by long-short term memory network. Biomed. Signal Process. Control 59, 101774 (2020). https://doi.org/10.1016/j.bspc.2019.101774

    Article  Google Scholar 

  37. W. Chen, Z. Zhang: Hand gesture recognition using sEMG signals based on support vector machine. 2019 IEEE 8th joint international information technology and artificial intelligence conference (ITAIC), 230–234(2019). DOI: https://doi.org/10.1109/ITAIC.2019.8785542

  38. Erzen, A.T.: A new CNN approach for hand gesture classification using sEMG data. J. Innovative Sci. Eng. (JISE) 4(1), 44–55 (2020). https://doi.org/10.38088/jise.730957

    Article  Google Scholar 

  39. Fatimah, B., Singh, P., Singhal, A., et al.: Hand movement recognition from sEMG signals using Fourier decomposition method. Biocybernetics Biomed. Eng. 2(41), 690–703 (2021). https://doi.org/10.1016/j.bbe.2021.03.004

    Article  Google Scholar 

  40. Phinyomark, A., Scheme, E.: EMG pattern recognition in the era of big data and deep learning. Big Data Cogn. Comput. 2, 21 (2018). https://doi.org/10.3390/bdcc2030021

    Article  Google Scholar 

  41. Lobov S., Krilova N., Kastalskiy I., Kazantsev V., Makarov V.A.: Latent factors limiting the performance of sEMG-interfaces. Sensors. 18(4), 1122. DOI: https://doi.org/10.3390/s18041122

  42. Saetta, G., Cognolato, M., Atzori, M., et al.: Gaze, behavioral, and clinical data for phantom limbs after hand amputation from 15 amputees and 29 controls. Sci. Data 7, 60 (2020). https://doi.org/10.1038/s41597-020-0402-1

    Article  Google Scholar 

  43. Du Y, Jin W, Wei W, Hu Y, Geng W.: Surface EMG-based inter-session gesture recognition enhanced by deep domain adaptation. Sensors (Basel). 17(3), 458(2017). DOI: https://doi.org/10.3390/s17030458

  44. Georgi, M., Amma, C., Schultz, T.: Recognizing hand and finger gestures with IMU based Motion and EMG based Muscle Activity Sensing. 99–108 (2015). https://doi.org/10.5220/0005276900990108

  45. Kaczmarek, P., Mańkowski, T., Tomczyński, J., putEMG-A Surface Electromyography Hand Gesture Recognition Dataset. Sensors (Basel, Switzerland), 19(16), 3548(2019). DOI: https://doi.org/10.3390/s19163548

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFB1401200; in part by the Key Project of Hebei Province Department of Education under Grant ZD2020146; in part by the Hebei Province Postdoctoral Scientific Research Project under Grant B2019005001; in part by the Program for Top 100 Innovative Talents in Colleges and Universities of Hebei Province under grant SLRC2017022; and in part by the Natural Science Foundation of China under grant 61703133 and 61673158.

Author information

Authors and Affiliations

  1. College of Electronic and Information Engineering, Yanshan University, Qinhuangdao, 066000, China

    Jun Li, Lixin Wei, Yintang Wen & Hongrui Wang

  2. College of Electronic and Information Engineering, Hebei University, Baoding, 071002, China

    Jun Li, Xiaoguang Liu & Hongrui Wang

Authors
  1. Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lixin Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yintang Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoguang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongrui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hongrui Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Wei, L., Wen, Y. et al. An approach to continuous hand movement recognition using SEMG based on features fusion. Vis Comput 39, 2065–2079 (2023). https://doi.org/10.1007/s00371-022-02465-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02465-7

Keywords

Search

Navigation