Skip to main content

Advertisement

SpringerLink
Log in

Air Writing: Recognizing Multi-Digit Numeral String Traced in Air Using RNN-LSTM Architecture

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Air writing provides a more natural and immersive way of interacting with devices, with the potential of having significant application in fields like augmented reality and education. However, such systems often rely on expensive hardware, making them less accessible for general purposes. In this study, we propose a robust and inexpensive system for the recognition of multi-digit numerals traced in an air-writing environment which uses only a generic device camera for input. We employ a sliding window-based algorithm to isolate a small segment of the input for processing. A dual network configuration consisting of RNN-LSTM networks are used for noise elimination and digit recognition. We conduct our experiments on English numerals using the MNIST dataset as the baseline model to allow easy adaptability of our method. Our results are further improved by the use of Pendigits and ISI-Air online datasets. We observed a drop in accuracy with increase in the number of digits owing to the accumulation of transition noise. However, bi-directional scanning considerably reduces the impact of such noise on the recognition accuracy. Under standard conditions, our system produced an accuracy of 98.75% and 85.27% for single and multi-digit English numerals, respectively. Incorporation of selective frame skipping in the sliding window algorithm resulted in a 60% reduction in computational time, significantly improving the system performance. We provide a link to the source code of our system at the end of this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Availability of Data and Material

Our custom online air-written English numeral dataset ISI-Air is freely available for research purposes and non-commercial use at https://github.com/adildsw/isi-air/. The training sets, noise dataset and experimental results can be obtained from https://github.com/adildsw/air-writing-v2/.

Code Availability

The source code of our system can be found at https://github.com/adildsw/air-writing-v2/.

References

  1. Agarwal C, Dogra DP, Saini R, Roy PP. Segmentation and recognition of text written in 3d using leap motion interface. In: 2015 3rd IAPR Asian conference on pattern recognition (ACPR) (2015), IEEE, pp. 539–543.

  2. Alimoglu F, Alpaydin E. Methods of combining multiple classifiers based on different representations for pen-based handwritten digit recognition. In: Proceedings of the fifth Turkish artificial intelligence and artificial neural networks symposium (TAINN 96) (1996), Citeseer.

  3. Alkhamisi AO, Arabia S, Monowar MM, et al. Rise of augmented reality: current and future application areas. Int J Internet Distrib Syst. 2013;1(04):25.

    Article  Google Scholar 

  4. Amma C, Gehrig D, Schultz T. Airwriting recognition using wearable motion sensors. In: Proceedings of the 1st augmented human international conference. 2010, pp. 1–8.

  5. Amma C, Georgi M, Schultz T. Airwriting: a wearable handwriting recognition system. Person Ubiquitous Comput. 2014;18(1):191–203.

    Article  Google Scholar 

  6. Cao Z, Hidalgo G, Simon T, Wei S-E, Sheikh Y. Openpose: realtime multi-person 2d pose estimation using part affinity fields. 2018. arXiv:1812.08008.

  7. Chaplin DJ. Chroma key method and apparatus, Sept. 28 1993. US Patent 5,249,039.

  8. Chen M, AlRegib G, Juang B-H. Air-writing recognition—Part I: modeling and recognition of characters, words, and connecting motions. IEEE Trans Hum Mach Syst. 2015;46(3):403–13.

    Article  Google Scholar 

  9. Chen M, AlRegib G, Juang B-H. Air-writing recognition—Part II: detection and recognition of writing activity in continuous stream of motion data. IEEE Trans Hum Mach Syst. 2015;46(3):436–44.

    Article  Google Scholar 

  10. Huang Y, Liu X, Jin L, Zhang X. Deepfinger: a cascade convolutional neuron network approach to finger key point detection in egocentric vision with mobile camera. In: 2015 IEEE international conference on systems, man, and cybernetics (2015), IEEE, pp. 2944–2949.

  11. Junker H, Amft O, Lukowicz P, Tröster G. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognit. 2008;41(6):2010–24.

    Article  Google Scholar 

  12. Kim DH, Choi HI, Kim JH. 3D space handwriting recognition with ligature model. In: International symposium on ubiquitious computing systems. Springer; 2006. pp. 41–56.

  13. Kingma DP, Ba J. Adam: a method for stochastic optimization. 2014. arXiv:1412.6980.

  14. LeCun Y. The mnist database of handwritten digits. 1998. http://yann.lecun.com/exdb/mnist/. Accessed 3 June 2019.

  15. Lee H-K, Kim JH. An hmm-based threshold model approach for gesture recognition. IEEE Trans Pattern Anal Mach Intell. 1999;10:961–73.

    Google Scholar 

  16. Levenshtein VI. Binary codes capable of correcting deletions. Insertions and reversals. Soviet Physics Doklady. 1966;10:707.

    MathSciNet  Google Scholar 

  17. Lun R, Zhao W. A survey of applications and human motion recognition with microsoft kinect. Int J Pattern Recognit Artif Intell. 2015;29(05):1555008.

    Article  Google Scholar 

  18. Mekni M, Lemieux A. Augmented reality: applications, challenges and future trends. In: 13th international conference on applied computer and applied computational science. 2014. p. 205–14.

  19. Mitra S, Acharya T. Gesture recognition: a survey. IEEE Trans Syst Man Cybern Part C (Appl Rev). 2007;37(3):311–24.

    Article  Google Scholar 

  20. Oh JK, Cho S-J, Bang W-C, Chang W, Choi E, Yang J, Cho J, Kim DY. Inertial sensor based recognition of 3-D character gestures with an ensemble classifiers. In: Ninth international workshop on frontiers in handwriting recognition (2004), IEEE, pp. 112–117.

  21. Qiu-yu Z, Jun-chi L, Mo-Yi Z, Hong-xiang D, Lu L. Hand gesture segmentation method based on YCBCR color space and k-means clustering. Int J Signal Process Image Process Pattern Recognit. 2015;8(5):105–16.

    Google Scholar 

  22. Rahman A, Roy P, Pal U. Continuous motion numeral recognition using RNN architecture in air-writing environment. In: Asian conference on pattern recognition. Springer; 2019, pp. 76–90.

  23. Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. 2015, pp. 91–99.

  24. Roy P, Ghosh S, Pal U. A CNN based framework for unistroke numeral recognition in air-writing. In: 2018 16th international conference on frontiers in handwriting recognition (ICFHR) (2018), IEEE, pp. 404–409.

  25. Wang P, Li W, Ogunbona P, Wan J, Escalera S. RGB-D-based human motion recognition with deep learning: a survey. Comput Vis Image Underst. 2018;171:118–39.

    Article  Google Scholar 

  26. Wang Y, Zhang B, Peng C. SRHandNet: real-time 2D hand pose estimation with simultaneous region localization. In: IEEE transactions on image processing, vol 29, pp 2977–2986.

  27. Weichert F, Bachmann D, Rudak B, Fisseler D. Analysis of the accuracy and robustness of the leap motion controller. Sensors. 2013;13(5):6380–93.

    Article  Google Scholar 

  28. Ye Z, Zhang X, Jin L, Feng Z, Xu S. Finger-writing-in-the-air system using kinect sensor. In: 2013 IEEE international conference on multimedia and expo workshops (ICMEW) (2013), IEEE, pp. 1–4.

  29. Zhang X, Ye Z, Jin L, Feng Z, Xu S. A new writing experience: finger writing in the air using a kinect sensor. IEEE Multimed. 2013;20(4):85–93.

    Article  Google Scholar 

  30. Zhang Z. Microsoft kinect sensor and its effect. IEEE Multimed. 2012;19(2):4–10.

    Article  Google Scholar 

Download references

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

Author information

Authors and Affiliations

  1. Department of Information Technology, Heritage Institute of Technology, Kolkata, India

    Adil Rahman

  2. Computer Vision and Pattern Recognition Unit, Indian Statistical Institute, Kolkata, India

    Prasun Roy & Umapada Pal

Authors
  1. Adil Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prasun Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umapada Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adil Rahman.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Machine Learning in Pattern Analysis” guest edited by Reinhard Klette, Brendan McCane, Gabriella Sanniti di Baja, Palaiahnakote Shivakumara and Liang Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, A., Roy, P. & Pal, U. Air Writing: Recognizing Multi-Digit Numeral String Traced in Air Using RNN-LSTM Architecture. SN COMPUT. SCI. 2, 20 (2021). https://doi.org/10.1007/s42979-020-00384-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-020-00384-9

Keywords

Search

Navigation