Skip to main content
SpringerLink
Log in

3D word spotting using leap motion sensor

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Leap motion sensor provides a new way of interaction with computers or mobile devices. With this sensor, users can write in air by moving palm or finger, thus, avoiding traditional pen and paper for writing. The strokes of air-writing or 3D writing is different from conventional way of writing. In 3D writing, the words are connected by continuous lines instead of space between them. Also, the arbitrary size of characters and presence of frequent jitters in strokes make the recognition tasks of such words and sentences difficult. To understand the semantics of a word without recognizing each character of words, the alternative process called “word-spotting” is being used. Word-spotting is often useful than conventional recognition systems to understand complex handwriting. Hence, we propose a novel word spotting methodology for 3D text using Leap motion sensor data. Spotting/detection of a keyword in 3D sentences is carried out using Hidden Markov Model (HMM) framework. From experimental study, an average of 41.7 is recorded in terms of Mean-Average-Precision (MAP). The efficiency of the system is demonstrated by comparing traditional segmentation based system. The improved performance shows that the system could be used in developing novel applications in Human-Computer-Interaction (HCI) domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

References

  1. Agarwal C, Dogra DP, Saini R, Roy PP (2015) Segmentation and recognition of text written in 3d using leap motion interface. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), November, pp 539–543

  2. Amma C, Georgi M, Schultz T (2012) Airwriting: Hands-free mobile text input by spotting and continuous recognition of 3d-space handwriting with inertial sensors. In: 16Th international symposium on wearable computers, pp 52–59

  3. Baljekar P, Lehman JF, Singh R (2014) Online word-spotting in continuous speech with recurrent neural networks. In: Spoken language technology workshop, pp 536–541

  4. Bassily D, Georgoulas C, Guettler J, Linner T, Bock T (2014) Intuitive and adaptive robotic arm manipulation using the leap motion controller. In: ISR/Robotik 2014; 41st International Symposium on Robotics; Proceedings of, pp 1–7. VDE

  5. Bharath A, Madhvanath S (2012) Hmm-based lexicon-driven and lexicon-free word recognition for online handwritten indic scripts. IEEE PAMI 34 (4):670–682

    Article  Google Scholar 

  6. Behera S, Roy PP, Dogra DP (2018) Fast recognition and verification of 3D air signatures using convex hulls. Expert Syst Appl 100:106–119

    Article  Google Scholar 

  7. Chen M, AlRegib G, Juang BH (2016) Air-writing recognition part i: Modeling and recognition of characters, words, and connecting motions. IEEE Trans Hum Mach Syst 46(3):403–413

    Article  Google Scholar 

  8. Cho OH, Lee ST (2014) A study about honey bee dance serious game for kids using hand gesture. International Journal of Multimedia and Ubiquitous Engineering 9(6):397–404

    Article  Google Scholar 

  9. Chuan CH, Regina E, Guardino C (2014) American sign language recognition using leap motion sensor. In: Machine learning and applications (ICMLA), 2014 13th international conference on, pp 541–544. IEEE

  10. Das A, Bhunia AK, Roy PP, Pal U (2015) Handwritten word spotting in indic scripts using foreground and background information. In: 3Rd IAPR asian conference on pattern recognition, pp 426–430

  11. Espana-Boquera S, Castro-Bleda MJ, Gorbe-Moya J, Zamora-Martinez F (2011) Improving offline handwritten text recognition with hybrid hmm/ann models. IEEE PAMI 33(4):767–779

    Article  Google Scholar 

  12. Fischer A, Keller A, Frinken V, Bunke H (2010) Hmm-based word spotting in handwritten documents using subword models. In: 20Th international conference on pattern recognition, pp 3416–3419

  13. Fischer A, Keller A, Frinken V, Bunke H (2012) Lexicon-free handwritten word spotting using character hmms. Pattern Recogn Lett 33(7):934–942

    Article  Google Scholar 

  14. Frinken V, Fischer A, Manmatha R, Bunke H (2012) A novel word spotting method based on recurrent neural networks. IEEE PAMI 34(2):211–224

    Article  Google Scholar 

  15. Jaeger S, Manke S, Reichert J, Waibel A (2001) Online handwriting recognition: the npen++ recognizer. Int J Doc Anal Recognit 3(3):169–180

    Article  Google Scholar 

  16. Khademi M, Mousavi Hondori H, McKenzie A, Dodakian L, Lopes CV, Cramer SC (2014) Free-hand interaction with leap motion controller for stroke rehabilitation. In: Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems, pp 1663–1668. ACM

  17. Kovalchuk A, Wolf L, Dershowitz N (2014) A simple and fast word spotting method. In: 14Th international conference on frontiers in handwriting recognition, pp 3–8

  18. Kumar P, Saini R, Behera SK, Dogra DP, Roy PP (2017) Real-time recognition of sign language gestures and air-writing using leap motion. In: 2017 Fifteenth IAPR international conference on machine vision applications (MVA), pp 157–160. IEEE

  19. Kumar P, Saini R, Roy P, Dogra D (2017) Study of text segmentation and recognition using leap motion sensor. IEEE Sensors J 17(5):1293–1301

    Article  Google Scholar 

  20. Kumar P, Saini R, Roy PP (2016) Dogra, D.P.: 3d text segmentation and recognition using leap motion. Multimedia Tools and Applications 76:16491–16510

    Article  Google Scholar 

  21. Lai CS, Shi BE (2001) A one-pass strategy for keyword spotting and verification. In: International conference on acoustics, speech, and signal processing, vol. 1, pp 377–380

  22. Mittal A, Kumar P, Roy PP, Balasubramanian R, Chaudhuri BB (2019) A modified LSTM model for continuous sign language recognition using leap motion., 19(16), 7056–7063. IEEE Sensors J 19(16):7056–7063

    Article  Google Scholar 

  23. Markussen A, Jakobsen MR, Hornbæk K (2014) Vulture: a mid-air word-gesture keyboard. In: 32Nd conference on human factors in computing systems, pp 1073–1082

  24. Motion L (2015) Leap motion controller. https://www.leapmotion.com

  25. Mukherjee S, Ahmed SA, Dogra DP, Kar S, Roy PP (2019) Fingertip detection and tracking for recognition of air-writing in videos. Expert Syst Appl 136:217–229

    Article  Google Scholar 

  26. Nigam I, Vatsa M, Singh R (2014) Leap signature recognition using hoof and hot features. In: Image processing (ICIP), 2014 IEEE international conference on, pp 5012–5016. IEEE

  27. Potter LE, Araullo J, Carter L (2013) The leap motion controller: a view on sign language. In: Proceedings of the 25th Australian computer-human interaction conference: augmentation, application, innovation, collaboration, pp 175–178. ACM

  28. Rodríguez-Serrano JA, Perronnin F (2009) Handwritten word-spotting using hidden markov models and universal vocabularies. Pattern Recogn 42 (9):2106–2116

    Article  MATH  Google Scholar 

  29. Roy PP, Bhunia AK, Das A, Dhar P, Pal U (2017) Keyword spotting in doctor’s handwriting on medical prescriptions. Expert Syst Appl 76:113–128

    Article  Google Scholar 

  30. Wang K, Belongie S (2010) Word spotting in the wild. In: European conference on computer vision, pp 591–604

  31. Weichert F, Bachmann D, Rudak B, Fisseler D (2013) Analysis of the accuracy and robustness of the leap motion controller. Sensors 13 (5):6380–6393

    Article  Google Scholar 

  32. Xu N, Wang W, Qu X (2015) On-line sample generation for in-air written chinese character recognition based on leap motion controller. In: Pacific rim conference on multimedia, pp 171–180

  33. Xu N, Wang W, Qu X (2015) Recognition of in-air handwritten chinese character based on leap motion controller. In: International conference on image and graphics, pp 160–168

  34. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE multimedia 19(2):4–10

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, IIT Roorkee, Roorkee, India

    Partha Pratim Roy, Pradeep Kumar & Shweta Patidar

  2. EISLAB Machine Learning, Department of Computer Science, Electrical and Space Engineering, Luleå tekniska universitet, Luleå, Sweden

    Rajkumar Saini

Authors
  1. Partha Pratim Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pradeep Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shweta Patidar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rajkumar Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pradeep Kumar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roy, P.P., Kumar, P., Patidar, S. et al. 3D word spotting using leap motion sensor. Multimed Tools Appl 80, 11671–11689 (2021). https://doi.org/10.1007/s11042-020-10229-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10229-5

Keywords

Search

Navigation