Skip to main content
SpringerLink
Log in

Hand gesture recognition using Leap Motion via deterministic learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the development of multimedia technology, traditional interactive tools, such as mouse and keyboard, cannot satisfy users’ requirements. Touchless interaction has received considerable attention in recent years with benefit of removing barriers of physical contact. Leap Motion is an interactive device which can be used to collect information of dynamic hand gestures, including coordinate, acceleration and direction of fingers. The aim of this study is to develop a new method for hand gesture recognition using jointly calibrated Leap Motion via deterministic learning. Hand gesture features representing hand motion dynamics, including spatial position and direction of fingers, are derived from Leap Motion. Hand motion dynamics underlying motion patterns of different gestures which represent Arabic numbers (0-9) and capital English alphabets (A-Z) are modeled by constant radial basis function (RBF) neural networks. Then, a bank of estimators is constructed by the constant RBF networks. By comparing the set of estimators with a test gesture pattern, a set of recognition errors are generated. The average L1 norms of the errors are taken as the recognition measure according to the smallest error principle. Finally, experiments are carried out to demonstrate the high recognition performance of the proposed method. By using the 2-fold, 10-fold and leave-one-person-out cross-validation styles, the correct recognition rates for the Arabic numbers are reported to be 94.2%, 95.1% and 90.2%, respectively, for the English alphabets are reported to be 89.2%, 92.9% and 86.4%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Beh J, Han D, Durasiwami R, Ko H (2014) Hidden Markov Model on a unit hypersphere space for gesture trajectory recognition. Pattern Recogn Lett 36:144–153

    Article  Google Scholar 

  2. Beh J, Han D, Ko H (2014) Rule-based trajectory segmentation for modeling hand motion trajectory. Pattern Recogn 47(4):1586–1601

    Article  Google Scholar 

  3. Chen M, AlRegib G, Juang BH (2013) Feature processing and modeling for 6D motion gesture recognition. IEEE Trans Multimed 15(3):561–571

    Article  Google Scholar 

  4. Cheng H, Yang L, Liu Z (2016) Survey on 3D hand gesture recognition. IEEE Tran Circ Syst Video Technol 26(9):1659–1673

    Article  Google Scholar 

  5. Ding G, Zhou J, Guo Y, Lin Z, Zhao S, Han J (2017) Large-scale image retrieval with sparse embedded hashing. Neurocomputing 257:24–36

    Article  Google Scholar 

  6. Ding G, Chen W, Zhao S, Han J, Liu Q (2018) Real-Time Scalable visual tracking via quadrangle kernelized correlation filters. IEEE Trans Intell Transp Syst 19 (1):140–150

    Article  Google Scholar 

  7. Du G, Zhang P, Liu X (2016) Markerless Human-manipulator interface using Leap Motion with interval Kalman filter and improved particle filter. IEEE Trans Indust Inform 12(2):694–704

    Article  Google Scholar 

  8. Elmezain M, Al-Hamadi A (2007) Gesture recognition for alphabets from hand motion trajectory using hidden markov models. In: IEEE international symposium on signal processing and information technology, pp 1192–1197

  9. Farrell J (1998) Stability and approximator convergence in nonparametric nonlinear adaptive control. IEEE Trans Neural Netw 9(5):1008–1020

    Article  Google Scholar 

  10. Gorinevsky D (1995) On the persistency of excitation in radial basis function network identification of nonlinear systems. IEEE Trans Neural Netw 6(5):1237–1244

    Article  Google Scholar 

  11. Hafiz AR, Al-Nuaimi AY, Amin MF, Murase K (2015) Classification of skeletal wireframe representation of hand gesture using complex-valued neural network. Neural Process Lett 42(3):649–664

    Article  Google Scholar 

  12. Haken H, Kelso JS, Bunz H (1985) A theoretical model of phase transitions in human hand movements. Biol Cybern 51(5):347–356

    Article  MathSciNet  Google Scholar 

  13. Hettig J, Saalfeld P, Luz M, Becker M, Skalej M, Hansen C (2017) Comparison of gesture and conventional interaction techniques for interventional neuroradiology. Int J CARS 12(9):1643–1653

    Article  Google Scholar 

  14. Herekar RR, Dhotre SR (2014) Handwritten character recognition based on zoning using Euler number for English alphabets and numerals. IOSR J Comput Eng 16(4):75–88

    Article  Google Scholar 

  15. Huang DY, Hu WC, Chang SH (2011) Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination. Expert Syst Appl 38(5):6031–6042

    Article  Google Scholar 

  16. Ju Z, Liu H (2014) Human hand motion analysis with multisensory information. IEEE/ASME Trans Mechatron 19(2):456–466

    Article  Google Scholar 

  17. Kane L, Khanna P (2016) A framework to plot and recognize hand motion trajectories towards development of non-tactile interfaces. Procedia Comput Sci 84:6–13

    Article  Google Scholar 

  18. Katahira R, Soga M (2015) Development and evaluation of a system for AR enabling realistic display of gripping motions using Leap Motion controller. Procedia Comput Sci 60:1595–1603

    Article  Google Scholar 

  19. Kumar P, Gauba H, Roy PP, Dogra DP (2017) Coupled HMM-based multi-sensor data fusion for sign language recognition. Pattern Recogn Lett 86:1–8

    Article  Google Scholar 

  20. Kumar P, Gauba H, Roy PP, Dogra DP (2017) A multimodal framework for sensor based sign language recognition. Neurocomputing. https://doi.org/10.1016/j.neucom.2016.08.132

    Article  Google Scholar 

  21. Kundu S, Chhabra HS, Ara SS, Mishra RP (2017) Optical character recognition using 26-point feature extraction and ANN. Int J Adv Res Comput Sci Softw Eng 7(5):156–162

    Article  Google Scholar 

  22. Lamberti L, Camastra F (2012) Handy: a real-time three color glove-based gesture recognizer with learning vector quantization. Expert Syst Appl 39(12):10489–10494

    Article  Google Scholar 

  23. Lin WY, Hsieh CY (2013) Kernel-based representation for 2D/3D motion trajectory retrieval and classification. Pattern Recogn 46(3):662–670

    Article  Google Scholar 

  24. Liu N, Lovell BC, Kootsookos PJ (2003) Evaluation of hmm training algorithms for letter hand gesture recognition. In: Proceedings of the 3rd IEEE international symposium on signal processing and information technology, pp 648–651

  25. Liu N, Lovell BC, Kootsookos PJ, Davis RI (2004) Model structure selection & training algorithms for an HMM gesture recognition system. In: IEEE 9th international workshop on frontiers in handwriting recognition, pp 100–105

  26. Lu W, Tong Z, Chu J (2016) Dynamic hand gesture recognition with Leap Motion controller. IEEE Signal Processing Letters 23(9):1188–1192

    Article  Google Scholar 

  27. Lu G, Zhou Y, Li X, Kudo M (2016) Efficient action recognition via local position offset of 3D skeletal body joints. Multimed Tools Appl 75(6):3479–3494

    Article  Google Scholar 

  28. Maqueda AI, del-Blanco CR, Jaureguizar F, Garcła N (2015) Human-computer interaction based on visual hand-gesture recognition using volumetric spatiograms of local binary patterns. Comp Vision Image Underst 141:126–137

    Article  Google Scholar 

  29. Marin G, Dominio F, Zanuttigh P (2016) Hand gesture recognition with jointly calibrated Leap Motion and depth sensor. Multimed Tools Appl 75(22):14991–15015

    Article  Google Scholar 

  30. Mohandes M, Aliyu S, Deriche M (2014) Arabic sign language recognition using the leap motion controller. In: IEEE 23rd international symposium on industrial electronics, pp 960–965

  31. Park J, Sandberg IW (1991) Universal approximation using radial-basis-function networks. Neural Comput 3(2):246–257

    Article  Google Scholar 

  32. Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: A review. Comp Vision Image Underst 141:152–165

    Article  Google Scholar 

  33. Ren Z, Yuan J, Meng J, Zhang Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Trans Multimed 15(5):1110–1120

    Article  Google Scholar 

  34. Sahoo MK, Nayak J, Mohapatra S, Nayak BK, Behera HS (2015) Character recognition using firefly based back propagation neural network. In: Computational intelligence in data mining-volume 2. Springer, New Delhi

  35. Sarafianos N, Boteanu B, Ionescu B, Kakadiaris IA (2016) 3D human pose estimation: a review of the literature and analysis of covariates. Comput Vis Image Underst 152:1–20

    Article  Google Scholar 

  36. Shen X, Hua G, Williams L, Wu Y (2012) Dynamic hand gesture recognition: an exemplar-based approach from motion divergence fields. Image Vis Comput 30 (3):227–235

    Article  Google Scholar 

  37. Shen J, Luo Y, Wu Z, Tian Y, Deng Q (2016) CUDA-Based real-time hand gesture interaction and visualization for CT volume dataset using leap motion. Vis Comput 32(3):359–370

    Article  Google Scholar 

  38. Sohn MK, Lee SH, Kim DJ, Kim B, Kim H (2012) A comparison of 3D hand gesture recognition using dynamic time warping. In: Proceedings of the 27th conference on image and vision computing, pp 418–422

  39. Tekin B, Katircioglu I, Salzmann M, Lepetit V, Fua P (2016) Structured prediction of 3d human pose with deep neural networks. arXiv:1605.05180

  40. Tompson J, Stein M, Lecun Y, Perlin K (2014) Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans Graph 33(5):169

    Article  Google Scholar 

  41. Tubaiz N, Shanableh T, Assaleh K (2015) Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Trans Human-Mach Syst 45 (4):526–533

    Article  Google Scholar 

  42. Vafadar M, Behrad A (2015) A vision based system for communicating in virtual reality environments by recognizing human hand gestures. Multimed Tools Appl 74 (18):7515–7535

    Article  Google Scholar 

  43. Wang M, Chen WY, Li XD (2016) Hand gesture recognition using valley circle feature and Hu’s moments technique for robot movement control. Measurement 94:734–744

    Article  Google Scholar 

  44. Wang C, Chen T, Chen G, Hill DJ (2009) Deterministic learning of nonlinear dynamical systems. Int J Bifurcation Chaos 19(4):1307–1328

    Article  MathSciNet  Google Scholar 

  45. Wang Q, Xu YR, Bai X, Xu D, Chen YL, Wu X (2014) Dynamic gesture recognition using 3D trajectory. In: IEEE 4th international conference on information science and technology, pp 598–601

  46. Wang C, Hill DJ (2006) Learning from neural control. IEEE Trans Neural Netw 17(1):130–146

    Article  Google Scholar 

  47. Wang C, Hill DJ (2007) Deterministic learning and rapid dynamical pattern recognition. IEEE Trans Neural Netw 18(3):617–630

    Article  Google Scholar 

  48. Wang C, Hill DJ (2009) Deterministic learning theory for identification, recognition and control. CRC Press, Boca Raton

    Google Scholar 

  49. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13(5):6380–6393

    Article  Google Scholar 

  50. Xu Y, Wang Q, Bai X, Chen YL, Wu X (2014) A novel feature extracting method for dynamic gesture recognition based on support vector machine. In: IEEE international conference on information and automation, pp 437–441

  51. Xue Y, Ju Z, Xiang K, Chen J, Liu H (2017) Multiple sensors based hand motion recognition using adaptive directed acyclic graph. Appl Sci 7(4):358

    Article  Google Scholar 

  52. Yang C, Han DK, Ko H (2017) Continuous hand gesture recognition based on trajectory shape information. Pattern Recogn Lett 99:39–47

    Article  Google Scholar 

  53. Yao Y, Fu Y (2014) Contour model-based hand-gesture recognition using the Kinect sensor. IEEE Trans Circ Syst Video Technol 24(11):1935–1944

    Article  Google Scholar 

  54. Zaiti IA, Pentiuc SG, Vatavu RD (2015) On free-hand TV control: experimental results on user-elicited gestures with Leap Motion. Pers Ubiquit Comput 19(5-6):821–838

    Article  Google Scholar 

  55. Zhang J, Li W, Ogunbona PO, Wang P, Tang C (2016) RGB-D-based action recognition datasets: A survey. Pattern Recogn 60:86–105

    Article  Google Scholar 

  56. Zhao S, Yao H, Gao Y, Ji R, Ding G (2017) Continuous probability distribution prediction of image emotions via multitask shared sparse regression. IEEE Trans Multimed 19(3):632–645

    Article  Google Scholar 

  57. Zhou Y, Jiang G, Lin Y (2016) A novel finger and hand pose estimation technique for real-time hand gesture recognition. Pattern Recogn 49:102–114

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant nos. 61773194, 61304084), by the Program for New Century Excellent Talents in Fujian Province University, and by the Science and Technology Project of Longyan City (Grant no. 2017LY69).

Author information

Authors and Affiliations

  1. School of Mechanical & Electrical Engineering, Longyan University, Longyan, 364012, People’s Republic of China

    Wei Zeng & Qinghui Wang

  2. School of Automation Science and Engineering, South China University of Technology, Guangzhou, 510640, People’s Republic of China

    Cong Wang

Authors
  1. Wei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qinghui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei Zeng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, W., Wang, C. & Wang, Q. Hand gesture recognition using Leap Motion via deterministic learning. Multimed Tools Appl 77, 28185–28206 (2018). https://doi.org/10.1007/s11042-018-5998-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5998-1

Keywords

Search

Navigation