Skip to main content

Advertisement

Springer Nature Link
Log in

Fusion of kinematic and physiological sensors for hand gesture recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The uncertainty of hand gestures, the variability of gestures across subjects, and the high cost of collecting a large amount of annotated data lead to a great challenge to the robust recognition of gestures, and thus it remains quite crucial to capture the informative features of hand movements and to mitigate inter-subject variations. To this end, we propose a gesture recognition model that uses two different types of sensors and optimizes the feature space towards enhanced accuracy and better generalization. Specifically, we use an accelerometer and a surface electromyography sensor to capture kinematic and physiological signals of hand movements. We use a sliding window to divide the streaming sensor data and then extract time-domain and frequency-domain features from each segment to return feature vectors. Afterwards, the feature space is optimized with a feature selector and a gesture recognizer is optimized. To handle the case where no labeled training data are available for a new user, we apply the transfer learning technique to reuse the cross-subject knowledge. Finally, extensive comparative experiments concerning different classification models, different sensors, and different types of features are conducted. Results show that the joint use of kinematic and physiological sensors generally outperforms the use of single sensor, indicating the synthetic effect of different sensors, and that the use of transfer learning helps improve the cross-subject recognition accuracy. In addition, we quantitatively investigate the impact of null gesture on a gesture recognizer and results indicate that null gesture would lower its accuracy, enlightening related studies to consider it.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The data of this study are available from the corresponding author upon reasonable request.

References

  1. Jia L, Zhou X, Xue C (2022) Non-trajectory-based gesture recognition in human-computer interaction based on hand skeleton data. Multimed Tools Appl 81(15):20509–20539

    Article  Google Scholar 

  2. Gadekallu TR, Srivastava G, Liyanage M, Iyapparaja M, Chowdhary CL, Koppu S, Maddikunta PR (2022) Hand gesture recognition based on a Harris hawks optimized convolution neural network. Computers Electrical Engin 100:107836

    Article  Google Scholar 

  3. Bhushan S, Alshehri M, Keshta I, Chakraverti AK, Rajpurohit J, Abugabah A (2022) An experimental analysis of various machine learning algorithms for hand gesture recognition. Electronics 11(6):968

    Article  Google Scholar 

  4. Zhang Z, Tian Z, Zhou M (2018) Latern: Dynamic continuous hand gesture recognition using FMCW radar sensor. IEEE Sensors J 18(8):3278–3289

    Article  Google Scholar 

  5. Qi J, Jiang G, Li G, Sun Y, Tao B (2019) Intelligent human-computer interaction based on surface EMG gesture recognition. IEEE Access 7:61378–61387

    Article  Google Scholar 

  6. Guarino A, Malandrino D, Zaccagnino R, Capo C, Lettieri N (2023) Touchscreen gestures as images. A transfer learning approach for soft biometric traits recognition. Expert Syst Appl 219:119614

    Article  Google Scholar 

  7. Kim E, Helal S, Nugent C, Beattie M (2015) Analyzing activity recognition uncertainties in smart home environments. ACM Trans Intelli Syst Tech 6(4):1–28

    Google Scholar 

  8. Wang A, Chen G, Wu X, Liu X, An N, Chang CY (2018) Towards human activity recognition: A hierarchical feature selection framework. Sensors 18(11):3629

    Article  Google Scholar 

  9. Bai L, Yao L, Wang X, Kanhere SS, Guo B, Yu Z (2020), Adversarial multi-view networks for activity recognition. Proc ACM Interactive, Mob, Wearable Ubiquit Technol 4(2):1-22

  10. Lima WS, Braganca HL, Souto EJ (2021) NOHAR: Novelty discrete data stream for human activity recognition based on smartphones with inertial sensors. Expert Syst Appl 166:114093

    Article  Google Scholar 

  11. Jaramillo-Yánez A, Benalcázar M, Mena-Maldonado E (2020) Real-time hand gesture recognition using surface electromyography and machine learning: A systematic literature review. Sensors 20(9):2467

    Article  Google Scholar 

  12. Guo L, Lu Z, Yao L (2021) Human-machine interaction sensing technology based on hand gesture recognition: A review. IEEE Trans Human-Mach Syst 51(4):300–309

    Article  Google Scholar 

  13. Si Y, Chen S, Li M, Li S, Pei Y, Guo X (2022) Flexible strain sensors for wearable hand gesture recognition: From devices to systems. Adv Intell Syst 4(2):2100046

    Article  Google Scholar 

  14. Al Farid F, Hashim N, Abdullah J, Bhuiyan MR, Shahida Mohd Isa WN, Uddin J, Haque MA, Husen MN (2022) A structured and methodological review on vision-based hand gesture recognition system. J Imaging 8(6):153

    Article  Google Scholar 

  15. Bhaumik G, Verma M, Govil M, Vipparthi S (2023) Hyfinet: Hybrid feature attention network for hand gesture recognition. Multimed Tools Appl 82(4):4863–4882

    Article  Google Scholar 

  16. Dhiman C, Vishwakarma D (2019) A robust framework for abnormal human action recognition using R-transform and Zernike moments in depth videos. IEEE Sensors J 19(3):5195–5203

    Article  Google Scholar 

  17. Nyo MT, Mebarek-Oudina F, Hlaing SS, Khan NA (2022) Otsu’s thresholding technique for MRI image brain tumor segmentation. Multimed Tools Appl 81(30):43837–43849

    Article  Google Scholar 

  18. Pagan J, Fallahzadeh R, Pedram M, Jl R-M, Moya JM, Ayala JL, Ghasemzadeh H (2019) Toward ultra-low-power remote health monitoring: An optimal and adaptive compressed sensing framework for activity recognition. IEEE Trans Mobile Comput 18(3):658–673

    Article  Google Scholar 

  19. Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man, and Cybernetics, Part C (Applications and Reviews) 42(6):790–808

    Article  Google Scholar 

  20. Zhao H, Xu R, Shu M, Hu J (2016) Physiological-signal-based key negotiation protocols for body sensor networks: A survey. Simul Model Pract Theory 65:32–44

    Article  Google Scholar 

  21. Chen G, Wang W, Wang Z et al (2020) Two-dimensional discrete feature based spatial attention capsnet for sEMG signal recognition. Appl Intell 50:3503–3520

    Article  Google Scholar 

  22. Sayin FS, Ozen S, Baspinar U (2018) Hand gesture recognition by using sEMG signals for human machine interaction applications. In: 2018 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA). IEEE, pp 27–30

  23. Zou W, Zhang D, Lee DJ (2022) A new multi-feature fusion based convolutional neural network for facial expression recognition. Appl Intell 52(3):2918–2929

  24. Saeedi R, Sasani K, Norgaard S, Gebremedhin AH (2018) Personalized human activity recognition using wearables: a manifold learning-based knowledge transfer. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, pp 1193–1196

  25. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using Kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  26. Song T, Zhao H, Liu Z, Liu H, Sun D (2021) Intelligent human hand gesture recognition by local–global fusing quality-aware features. Future Gener Comp Syst 115(7043):298–303

    Article  Google Scholar 

  27. Chen FS, Fu CM, Huang CL (2003) Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vision Comput 21(8):745–758

    Article  Google Scholar 

  28. Wang A, Wu X, Zhao L, Chen H, Zhao S (2021) Physical activity recognition from accelerometer data using multi-view aggregation. J Appl Sci Engineer 24(4):611–620

    Google Scholar 

  29. Xia Q, Wada A, Korpela J, Maekawa T, Namioka Y (2019) Unsupervised factory activity recognition with wearable sensors using process instruction information. Proc ACM Interactive, Mob, Wearable Ubiquit Technol 3(2):1-23

  30. Jiang W, Ye X, Chen R, Su F, Lin M, Ma Y, Huang S (2021) Wearable on-device deep learning system for hand gesture recognition based on FPGA accelerator. Math Biosci Eng 18(1):132–153

    Article  Google Scholar 

  31. Wang A, Chen G, Yang J, Zhao S, Chang CY (2016) A comparative study on human activity recognition using inertial sensors in a smartphone. IEEE Sensors J 16(11):4566–4578

    Article  Google Scholar 

  32. Sapienza S, Ros PM, Guzman D, Rossi F, Terracciano R, Cordedda E, Demarchi D (2018) On-line event-driven hand gesture recognition based on surface electromyographic signals. In: 2018 IEEE international symposium on circuits and systems (ISCAS). IEEE, pp 1–5

  33. Yang Y, Duan F, Liu Z, Zhu C (2018) A time-domain hand gesture recognition system based on two-channel sEMG signals. In: Proceedings of the 12th International Convention on Rehabilitation Engineering and Assistive Technology. ICREAT, pp 31–36

  34. Zhang Y, Chen Y, Yu H, Yang X, Zeng B (2021) A feature adaptive learning method for high-density sEMG-based gesture recognition. Proc ACM Interactive, Mob, Wearable Ubiquit Technol 5(1):1–26

  35. Dardas NH, Georganas ND (2011) Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans Instrumen Meas 60(11):3592–3607

    Article  Google Scholar 

  36. Ceolini E, Frenkel C, Shrestha SB, Taverni G, Khacef L, Payvand M, Donati E (2020) Hand-gesture recognition based on EMG and eventbased camera sensor fusion: A benchmark in neuromorphic computing. Front Neurosci 14:637

    Article  Google Scholar 

  37. Lu Z, Chen X, Li Q, Zhang X, Zhou P (2014) A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices. IEEE Trans Human-Mach Syst 44(2):293–299

    Article  Google Scholar 

  38. Jiang S, Lv B, Guo W, Zhang C, Wang H, Sheng X, Shull PB (2018) Feasibility of wrist-worn, real-time hand, and surface gesture recognition via sEMG and IMU sensing. IEEE Trans Ind Inf 14(8):3376–3385

    Article  Google Scholar 

  39. Guo X, Xu W, Tang WQ, Wen C (2019) Research on optimization of static gesture recognition based on convolution neural network. In: 2019 4th International Conference on Mechanical, Control and Computer Engineering (ICMCCE). IEEE, pp 398–3982

  40. Sun JH, Ji TT, Zhang SB, Yang JK, Ji GR (2018) Research on the hand gesture recognition based on deep learning. In: 2018 12th International Symposium on Antennas, Propagation and EM Theory (ISAPE). IEEE, pp 1–4

  41. Lee HK, Kim JH (1999) An HMM-based threshold model approach for gesture recognition. IEEE Trans Pattern Analy Mache Intelli 21(10):961–973

    Article  Google Scholar 

  42. Bargellesi N, Carletti M, Cenedese A, Susto G, Terzi M (2019) A random forest-based approach for hand gesture recognition with wireless wearable motion capture sensors. IFAC-PapersOnLine 52(11):128–133

    Article  MathSciNet  Google Scholar 

  43. Pomboza-Junez G, Terriza JH (2016) Hand gesture recognition based on sEMG signals using support vector machines. In: 2016 IEEE 6th International Conference on Consumer Electronics-Berlin (ICCE-Berlin). IEEE, pp 174–178

  44. Yu Z, Zhao J, Wang Y, He L, Wang S (2021) Surface EMG-based instantaneous hand gesture recognition using convolutional neural network with the transfer learning method. Sensors 21(7):2540

    Article  Google Scholar 

  45. Wang J, Chen Y, Hu L, Peng X, Philip SY (2018) Stratified transfer learning for cross-domain activity recognition. In: 2018 IEEE international conference on pervasive computing and communications (PerCom). IEEE, pp 1–10

  46. Côté-Allard U, Fall CL, Campeau-Lecours A, Gosselin C, Laviolette F, Gosselin B (2017) Transfer learning for sEMG hand gestures recognition using convolutional neural networks. In: 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, pp 1663–1668

  47. Colacino FM, Emiliano R, Mace BR (2012) Subject-specific musculoskeletal parameters of wrist flexors and extensors estimated by an emg-driven musculoskeletal model. Med Eng Phys 34(5):531–540

    Article  Google Scholar 

  48. Banos O, Galvez JM, Damas M, Pomares H, Rojas I (2014) Window size impact in human activity recognition. Sensors 14(4):6474–6499

    Article  Google Scholar 

  49. Wang A, Meng Y, Liu J, Zhao S, Chen G (2021) Multi-domain feature extraction for human activity recognition using wearable sensors. In: 2021 International Conference on Networking and Network Applications (NaNA). IEEE, pp 254–259

  50. Pan SP, Tsang IW, Kwok JT, Qiang Y (2010) Domain adaptation via transfer component analysis. IEEE Trans Neural Networks 22(2):199–210

    Article  Google Scholar 

  51. Long M, Wang J, Ding G, Sun J, Yu PS (2013) Transfer feature learning with joint distribution adaptation. In: Proceedings of the IEEE international conference on computer vision. IEEE, pp 2200–2207

  52. Gong B, Shi Y, Sha F, Grauman K (2012) Geodesic flow kernel for unsupervised domain adaptation. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2066–2073

  53. Chen G, Wang A, Zhao S, Liu L, Chang CY (2018) Latent feature learning for activity recognition using simple sensors in smart homes. Multimed Tools Appl 77:15201–15219

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to express sincere gratitude to the reviewers.

Funding

This work was partially supported by the National Natural Science Foundation of China (Grant No. 62176082) and the Featured Innovation Project of the Department of Education of Guangdong Province (Grant No. 2021KTSCX117).

Author information

Authors and Affiliations

  1. School of Electronic Information Engineering, Foshan University, Foshan, 528225, China

    Aiguo Wang, Huancheng Liu, Chundi Zheng & Huihui Chen

  2. Department of Computer Science and Information Engineering, Tamkang University, Taipei, 251301, Taiwan

    Chih-Yung Chang

Authors
  1. Aiguo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huancheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chundi Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Huihui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chih-Yung Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Aiguo Wang, Huancheng Liu, Chundi Zheng, Huihui Chen; Methodology, Aiguo Wang, Huancheng Liu, Chih-Yung Chang; Formal analysis, Aiguo Wang, Chih-Yung Chang; Writing—original draft preparation, Huancheng Liu; Writing—review and editing, Aiguo Wang, Chih-Yung Chang. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Chih-Yung Chang.

Ethics declarations

Institutional review board statement

Not applicable.

Informed consent

Not applicable.

Conflicts of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, A., Liu, H., Zheng, C. et al. Fusion of kinematic and physiological sensors for hand gesture recognition. Multimed Tools Appl 83, 68013–68040 (2024). https://doi.org/10.1007/s11042-024-18283-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-024-18283-z

Keywords