Skip to main content
SpringerLink
Log in

A CNN Based Air-Writing Recognition Framework for Multilinguistic Characters and Digits

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Air writing is a practice of writing the linguistic characters in free space utilizing the six degrees of freedom of hand motion. Researchers have proposed various methods to approach air-writing based on one or more dedicated external hardware, increasing production cost, and reducing hardware redundancy. To solve this problem, we propose a system that uses a generic webcam to detect and recognize the virtually written characters by a user as per their will, removing the dependence on external hardware and increasing the hardware redundancy. This system performs detection using HSV color space for creating the mask of the tracker or the tracking object and morphological operations for refinement of the mask. This system gives the user the freedom to select a writing object of any color, shape, or material for tracking purposes. The trajectory of the contour of the object’s mask is tracked and rendered on a virtual window. The air-written character is recognized using the convolutional neural network (CNN). The CNN is trained and tested on four different datasets, which are English handwritten characters of 26 different classes (A–Z), MNIST dataset with 10 different classes (0–9), Devanagari handwritten character dataset consisting of 36 different classes (ka-gya), and Devanagari handwritten digits consisting of 10 classes (0–9). The CNN is also tested on a custom dataset of air-written digits (0–9) consisting of 3 samples of each class by 20 different individuals. The accuracy achieved by the proposed system for isolated characters on respective datasets is 99.75%, 99.73%, 99.13%, 99.97%, and 99.81%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

References

  1. Microsoft Corporation. Azure Depth Platform. Time-of-flight camera. https://devblo-gs.microsoft.com/azure-depth-platform/time-of-flight-camera-system-overview.

  2. Leap Motion Inc. Leap Motion. https://www.ultraleap.com.

  3. Microsoft Corporation. Azure Kinect Developer Kit. 2019. https://docs.microsoft.com/en-us/azure/kinect-dk.

  4. Itaguchi Y, Yamada C, Fukuzawa K. Writing in the air: facilitative effects of finger writing in older adults. PLoS ONE. 2019;14(12): e0226832. https://doi.org/10.1371/journal.pone.0226832.

    Article  Google Scholar 

  5. Rispens J, Berckelaer IA. Hyperlexia: definition and criterion. Writ Lang Disord. 1991. https://doi.org/10.1007/978-94-011-3732-4_8.

    Article  Google Scholar 

  6. Alam MS, Kwon K-C, Alam MA, Abbass MY, Imtiaz SM, Kim N. Trajectory-based air-writing recognition using deep neural network and depth sensor. Sensors. 2020;20(2):376. https://doi.org/10.3390/s20020376.

    Article  Google Scholar 

  7. Chen M, AlRegib G, Juang BH. Air-writing recognition–part I: modeling and recognition of characters, words, and connecting motions. IEEE Trans Hum Mach Syst. 2016;46(3):403–13. https://doi.org/10.1109/THMS.2015.2492598.

    Article  Google Scholar 

  8. Chen M, Alregib G, Juang BH. Air-writing recognition–part II: detection and recognition of writing activity in continuous stream of motion data. IEEE Trans Hum Mach Syst. 2016;46(3):436–44. https://doi.org/10.1109/THMS.2015.2492599.

    Article  Google Scholar 

  9. Mohammadi S, Maleki R. Air-writing recognition system for Persian numbers with a novel classifier. Vis Comput. 2020;36:1001–15. https://doi.org/10.1007/s00371-019-01717-3.

    Article  Google Scholar 

  10. Dash A, Sahu A, Shringi R, et al. AirScript–creating documents in air. In: 14th international conference on document analysis and recognition. 2017. p. 908–13.

  11. Yanay T, Shmueli E. Air-writing recognition using smart-bands. Pervasive Mobile Comput. 2020;66: 101183. https://doi.org/10.1016/j.pmcj.2020.101183.

    Article  Google Scholar 

  12. Kumar P, Saini R, Behera SK, Dogra DP, Roy PP. Real-time recognition of sign language gestures and air-writing using leap motion. In: 2017 fifteenth IAPR international conference on machine vision applications (MVA). 2017. p. 157–60.https://doi.org/10.23919/MVA.2017.7986825.

  13. Yin Y, Xie L, Gu T, Lu Y, Lu S. AirContour: building contour-based model for in-air writing gesture recognition. ACM Trans Sensor Netw. 2019;15(4):25. https://doi.org/10.1145/3343855.

    Article  Google Scholar 

  14. Arsalan M, Santra A. Character recognition in air-writing based on network of radars for human-machine interface. IEEE Sens J. 2019;19(19):8855–64. https://doi.org/10.1109/JSEN.2019.2922395.

    Article  Google Scholar 

  15. Islam R, Mahmud H, Hasan MK, Rubaiyeat H. Alphabet recognition in air writing using depth information. 2016.

  16. Luo Y, Liu J, Shimamoto S. wearable air-writing recognition system employing dynamic time warping. In: 2021 IEEE 18th annual consumer communications & networking conference (CCNC). 2021. p. 1–6. https://doi.org/10.1109/CCNC49032.2021.9369458.

  17. Arsalan M, Santra A, Issakov V. Radar trajectory-based air-writing recognition using temporal convolutional network. In: 2020 19th IEEE international conference on machine learning and applications (ICMLA). 2020. p. 1454–59. https://doi.org/10.1109/ICMLA51294.2020.00225.

  18. Chen H, Ballal T, Muqaibel AH, Zhang X, Al-Naffouri TY. Air writing via receiver array-based ultrasonic source localization. IEEE Trans Instrum Meas. 2020;69(10):8088–101. https://doi.org/10.1109/TIM.2020.2991573.

    Article  Google Scholar 

  19. Choi J-W, Ryu S-J, Kim J-H. Short-range radar based real-time hand gesture recognition using LSTM encoder. IEEE Access. 2019. https://doi.org/10.1109/ACCESS.2019.2903586.

    Article  Google Scholar 

  20. Roy PP, Kumar P, Patidar S, et al. 3D word spotting using leap motion sensor. Multimed Tools Appl. 2021;80:11671–89. https://doi.org/10.1007/s11042-020-10229-5.

    Article  Google Scholar 

  21. Tolentino RE, Roderos CJ, Ballesteros NJ, Papag MR, Tengco AV, Sunglao AP. Recognition of air-drawn alphanumeric characters by applying slope orientation sequence matching algorithm using leap motion controller. In: 2019 IEEE 11th international conference on humanoid, nanotechnology, information technology, communication and control, environment, and management (HNICEM). 2019. p. 1–5. https://doi.org/10.1109/HNICEM48295.2019.9072879.

  22. Bastas G, Kritsis K, Katsouros V. Air-writing recognition using deep convolutional and recurrent neural network architectures. In: 17th international conference on frontiers in handwriting recognition (ICFHR). 2020. p. 7–12. https://doi.org/10.1109/ICFHR2020.2020.00013.

  23. Saini R, Kumar P, Patidar S, Roy P, Liwicki M. Trilingual 3D script identification and recognition using leap motion sensor. In: 2019 international conference on document analysis and recognition workshops (ICDARW). 2019. p. 24–8. https://doi.org/10.1109/ICDARW.2019.40076.

  24. Alam MS, Kwon K-C, Imtiaz SM, Hossain M, Shinde R, Kim J, Kim N. Air-writing recognition using a fusion CNN-LSTM neural network. 2021.

  25. Uysal C, Filik T. RF-Wri: an efficient framework for RF-based device-free air-writing recognition. IEEE Sens J. 2021;21(16):17906–16. https://doi.org/10.1109/JSEN.2021.3082514.

    Article  Google Scholar 

  26. Ahlawat S, Choudhary A, Nayyar A, Singh S, Yoon B. Improved handwritten digit recognition using convolutional neural networks (CNN). Sensors. 2020;20(12):3344.

    Article  Google Scholar 

  27. OpenCV. Contours: getting started. https://docs.opencv.org/3.4/d4/d73/tutorial_py_contours_begin.html.

  28. OpenCV. Image Filtering. https://docs.opencv.org/4.x/d4/d86/group__imgproc__filter.html.

  29. OpenCV. Contours Hierarchy. https://docs.opencv.org/4.x/d9/d8b/tutorial_py_contours_hierarchy.html.

  30. Alzubi J, Nayyar A, Kumar A. Machine learning from theory to algorithms: an overview. J Phys Conf Ser. 2018;1142(1): 012012.

    Article  Google Scholar 

  31. Sachin P. A-Z handwritten Data: Sourced from NIST and NMIST large datasets for handwritten letters. 2016. https://www.kaggle.com/sachinpatel21/az-handwritten-alphabets-in-csv-format.

  32. Deng L. The mnist database of handwritten digit images for machine learning research. IEEE Signal Process Mag. 2012;29(6):141–2.

    Article  Google Scholar 

  33. Acharya S, Pant AK, Gyawali PK. Deep learning based large scale handwritten Devanagari character recognition. In: Proceedings of the 9th international conference on software, knowledge, information management and applications (SKIMA). 2015. p. 121–26.

Download references

Author information

Author notes

  1. Abhishek Chaudhary and Abhishek Sharma have contributed equally to this work.

Authors and Affiliations

  1. Department of Data Science and Artificial Intelligence, International Institute of Information Technology, Naya Raipur, Chhattisgarh, India

    Prabhat Kumar & Abhishek Chaudhary

  2. Department of Electronics and Communication Engineering, International Institute of Information Technology, Naya Raipur, Chhattisgarh, India

    Abhishek Sharma

Authors
  1. Prabhat Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abhishek Chaudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abhishek Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Prabhat Kumar.

Ethics declarations

Conflict of interest

Prabhat Kumar declares that he has no conflict of interest. Abhishek Chaudhary declares that he has no conflict of interest. Abhishek Sharma declares that he has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Soft Computing for Real Time Engineering Applications” guest edited by Kanubhai K. Patel and Pritpal Singh.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, P., Chaudhary, A. & Sharma, A. A CNN Based Air-Writing Recognition Framework for Multilinguistic Characters and Digits. SN COMPUT. SCI. 3, 453 (2022). https://doi.org/10.1007/s42979-022-01362-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01362-z

Keywords

Search

Navigation