Skip to main content
SpringerLink
Log in

Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, a pen-based 3D hand gesture dataset and recognition method using ultrasonic positioning and inertial data is proposed. First, considering that 3D hand gestures have six degrees of freedom, a 3D hand gesture dataset based on trajectory shape attributes, motion direction attributes and pen attitude attributes is proposed. Then, each attribute of the gesture is processed according to its priority, and the corresponding data channel and recognition method are selected to determine the 3D hand gesture label. Finally, experimental verification is conducted using a 3D multi-channel pen-like interactive device. For a 10-gesture set, the gesture recognition rates achieved ranged from 86.5–99.5%, depending on whether a single or multiple templates and thresholds are used. The results show that the 3D hand gesture recognition method proposed in this paper can recognize pen-based gestures effectively and solve the problem of traditional gesture recognition methods not being able to recognize 3D hand gestures containing multiple attributes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Abid MR, Petriu EM, Amjadian E (2015) Dynamic Sign Language Recognition for Smart Home Interactive Application Using Stochastic Linear Formal Grammar. IEEE Transactions on Instrum Meas 64:596–605

    Article  Google Scholar 

  2. Akan E, Tora H, Uslu B (2017) Hand gesture classification using inertial based sensors via a neural network. In: 2017 24th IEEE International Conference on Electronics, Circuits and Systems (ICECS). pp. 140–143

  3. Akl A, Valaee S (2010) Accelerometer-based gesture recognition via dynamic-time warping, affinity propagation, & compressive sensing. In: 2010 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 2270–2273

  4. Akl A, Feng C, Valaee S (2011) A Novel Accelerometer-Based Gesture Recognition System. IEEE Trans Sig Proc 59:6197–6205

    Article  MathSciNet  MATH  Google Scholar 

  5. Asano T, Honda S (2010) Visual interface system by character handwriting gestures in the air. In: 19th International Symposium in Robot and Human Interactive Communication. pp 56–61

  6. Baddeley A (1992) Working memory. Sci 255:556–559

    Article  Google Scholar 

  7. Che Y, Qi Y, Song Y (2019) Real-Time 3D Hand Gesture Based Mobile Interaction Interface. In: 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). pp 228–232

  8. Chen M, AlRegib G, Juang B-H (2016) Air-Writing Recognition—Part I: Modeling and Recognition of Characters, Words, and Connecting Motions. IEEE Trans Human-Mach Syst 46:403–413

    Article  Google Scholar 

  9. Chen J, Yu F, Yu J, Lin L (2020) A Three-Dimensional Ultrasonic Pen-Type Input Device With Millimeter-Level Accuracy for Human-Computer Interaction. IEEE Access 8:143837–143847

    Article  Google Scholar 

  10. Chen J, Yu F, Yu J, Lin L (2021) A Three-Dimensional Pen-Like Ultrasonic Positioning System Based on Quasi-Spherical PVDF Ultrasonic Transmitter. IEEE Sens J 21:1756–1763

    Article  Google Scholar 

  11. Cheng H, Yang L, Liu Z (2016) Survey on 3D Hand Gesture Recognition. IEEE Transactions on Circuits and Systems for Video Technology 26:1659–1673

    Article  Google Scholar 

  12. Djemal A, Hellara H, Barioul R, et al (2022) Real-Time Model for Dynamic Hand Gestures Classification based on Inertial Sensor. In: 2022 IEEE 9th International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA). pp. 1–6

  13. Dudak P, Sladek I, Dudak J, Sedivy S (2016) Application of inertial sensors for detecting movements of the human body. In: 2016 17th International Conference on Mechatronics - Mechatronika (ME). pp. 1–5

  14. Hsieh C-C, Liou D-H (2015) Novel Haar features for real-time hand gesture recognition using SVM. J Real-Time Image Proc 10:357–370

    Article  Google Scholar 

  15. Hsu Y-L, Chu C-L, Tsai Y-J, Wang J-S (2014) An inertial pen with dynamic time warping recognizer for handwriting and gesture recognition. IEEE Sens J 15:154–163

    Google Scholar 

  16. Ji Z, Li Z-Y, Li P, An M (2015) A new effective wearable hand gesture recognition algorithm with 3-axis accelerometer. In: 2015 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD). pp. 1243–1247

  17. Katsura S, Ohishi K (2007) Acquisition and Analysis of Finger Motions by Skill Preservation System. IEEE Transactions on Industrial Electronics 54:3353–3361

    Article  Google Scholar 

  18. Keskin C, Erkan AŞ, Akarun L (2003) REAL TIME HAND TRACKING AND 3D GESTURE RECOGNITION FOR INTERACTIVE INTERFACES USING HMM

  19. Long AC, Landay JA, Rowe LA (1997) PDA and Gesture Use in Practice: Insights for Designers of Pen-based User Interfaces

  20. Marasovic T, Papic V, Zanchi V (2015) LMNN metric learning and fuzzy nearest neighbour classifier for hand gesture recognition. J Multimod User Interfac 9:211–221

    Article  Google Scholar 

  21. Mingji H, Lin L (2016) Study on brush gesture feature recognition based on the 3D point cloud. In: 2016 IEEE 11th Conference on Industrial Electronics and Applications (ICIEA). pp 1634–1638

  22. Pan T-Y, Kuo C-H, Liu H-T, Hu M-C (2019) Handwriting Trajectory Reconstruction Using Low-Cost IMU. IEEE Trans Emerg Topics Comput Intel 3:261–270

    Google Scholar 

  23. Pezzuoli F, Corona D, Corradini ML (2020) Recognition and Classification of Dynamic Hand Gestures by a Wearable Data-Glove. SN Comput Sci 2:5

    Article  Google Scholar 

  24. Pomboza-Junez G, Terriza JH (2016) Hand gesture recognition based on sEMG signals using Support Vector Machines. In: 2016 IEEE 6th International Conference on Consumer Electronics - Berlin (ICCE-Berlin). pp 174–178

  25. Qi J, Jiang G, Li G et al (2019) Intelligent Human-Computer Interaction Based on Surface EMG Gesture Recognition. IEEE Access 7:61378–61387

    Article  Google Scholar 

  26. Sali Shajideen SM, Preetha VH (2018) Human-Computer Interaction System Using 2D and 3D Hand Gestures. In: 2018 International Conference on Emerging Trends and Innovations In Engineering And Technological Research (ICETIETR). pp 1–4

  27. Takahashi H, Kitazono Y (2016) Integration of Hand Gesture and Multi Touch Gesture with Glove Type Device. In: 2016 4th Intl Conf on Applied Computing and Information Technology/3rd Intl Conf on Computational Science/Intelligence and Applied Informatics/1st Intl Conf on Big Data, Cloud Computing, Data Science & Engineering (ACIT-CSII-BCD). pp 81–86

  28. Teachasrisaksakul K, Wu L, Yang G-Z, Lo B (2018) Hand Gesture Recognition with Inertial Sensors. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 3517–3520

  29. Wang J-S, Chuang F-C (2012) An Accelerometer-Based Digital Pen With a Trajectory Recognition Algorithm for Handwritten Digit and Gesture Recognition. IEEE Trans Indus Electron 59:2998–3007

    Article  Google Scholar 

  30. Xin Y, Bi X, Ren X (2011) Acquiring and Pointing: An Empirical Study of Pen-Tilt-Based Interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, pp 849–858

  31. Xu R, Zhou S, Li WJ (2012) MEMS Accelerometer Based Nonspecific-User Hand Gesture Recognition. IEEE Sensors Journal 12:1166–1173

    Article  Google Scholar 

  32. Zhu Y, Yuan B (2014) Real-time hand gesture recognition with Kinect for playing racing video games. In: 2014 International Joint Conference on Neural Networks (IJCNN). pp 3240–3246

  33. Zhu C, Yang J, Shao Z, Liu C (2021) Vision Based Hand Gesture Recognition Using 3D Shape Context. IEEE/CAA J Automatica Sinica 8:1600–1613

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China. (Grand 2016YFB1001301)

Author information

Authors and Affiliations

  1. College of Communication Engineering, Jilin University, Changchun, 130022, China

    Siyu Liu, Jian Chen, Cheng Wang & Lin Lin

Authors
  1. Siyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cheng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Chen.

Ethics declarations

Competing Interests

Human–computer interaction, 3D hand gesture recognition, 3D positioning, 3D pen-like interaction, Inertial sensor, Multichannel interaction.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Chen, J., Wang, C. et al. Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition. Multimed Tools Appl 82, 41841–41859 (2023). https://doi.org/10.1007/s11042-023-15252-w

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15252-w

Keywords

Search

Navigation