Skip to main content
SpringerLink
Log in

Airwriting: a wearable handwriting recognition system

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

We present a wearable input system which enables interaction through 3D handwriting recognition. Users can write text in the air as if they were using an imaginary blackboard. The handwriting gestures are captured wirelessly by motion sensors applying accelerometers and gyroscopes which are attached to the back of the hand. We propose a two-stage approach for spotting and recognition of handwriting gestures. The spotting stage uses a support vector machine to identify those data segments which contain handwriting. The recognition stage uses hidden Markov models (HMMs) to generate a text representation from the motion sensor data. Individual characters are modeled by HMMs and concatenated to word models. Our system can continuously recognize arbitrary sentences, based on a freely definable vocabulary. A statistical language model is used to enhance recognition performance and to restrict the search space. We show that continuous gesture recognition with inertial sensors is feasible for gesture vocabularies that are several orders of magnitude larger than traditional vocabularies for known systems. In a first experiment, we evaluate the spotting algorithm on a realistic data set including everyday activities. In a second experiment, we report the results from a nine-user experiment on handwritten sentence recognition. Finally, we evaluate the end-to-end system on a small but realistic data set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. wortschatz.uni-leipzig.de/html/wliste.

  2. wortschatz.uni-leizpig.de/html/wliste.

References

  1. Amft O, Amstutz R, Smailagic A, Siewiorek D, Tröster G (2009) Gesture-controlled user input to complete questionnaires on wrist-worn watches. In: Human–computer interaction. Novel interaction methods and techniques. Lecture Notes in Computer Science, vol 5611. Springer, Heidelberg, pp 131–140

  2. Amma C, Gehrig D, Schultz T (2010) Airwriting recognition using wearable motion sensors. In: Proceedings of the 1st augmented human international conference (AH’10). doi:10.1145/1785455.1785465

  3. Amma C, Georgi M, Schultz T (2012) Airwriting: hands-free mobile text input by spotting and continuous recognition of 3D-space handwriting with inertial sensors. In: IEEE 16th international symposium on wearable computers (ISWC), pp 52–59

  4. Amma C, Schultz T (2012) Airwriting: demonstrating mobile text input by 3D-space handwriting. In: Proceedings of the ACM international conference on intelligent user interfaces (IUI’12)

  5. Bang WC, Chang W, Kang KH, Choi ES, Potanin A, Kim DY (2003) Self-contained spatial input device for wearable computers. In: Proceedings of the IEEE international symposium on wearable computers (ISWC’03)

  6. Chen M, AlRegib G, Juang B (2012) 6D motion gesture recognition using spatio-temporal features. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 2341–2344

  7. Ephraim Y, Merhav N (2002) Hidden markov processes. IEEE Trans Inf Theory 48(6):1518–1569. doi:10.1109/TIT.2002.1003838

    Article  MATH  MathSciNet  Google Scholar 

  8. Gustafson S, Bierwirth D, Baudisch P (2010) Imaginary interfaces: spatial interaction with empty hands and without visual feedback. In: Proceedings of the 23nd annual ACM symposium on user interface software and technology (UIST’10)

  9. Hein A, Hoffmeyer A, Kirste T (2009) Utilizing an accelerometric bracelet for ubiquitous gesture-based interaction. In: Universal access in human–computer interaction. Intelligent and ubiquitous interaction environments. Lecture Notes in Computer Science, vol 5615. Springer, Heidelberg, pp 519–527

  10. Huang X, Acero A, Hon H (2001) Spoken language processing. Prentice Hall, NJ

    Google Scholar 

  11. Junker H, Amft O, Lukowicz P, Tröster G (2008) Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognit 41(6):2010–2024

    Article  MATH  Google Scholar 

  12. Kallio S, Kela J, Mantyjarvi J (2003) Online gesture recognition system for mobile interaction. In: Proceedings of the IEEE international conference on systems, man and cybernetics (ICSMC’03)

  13. Kim D, Choi H, Kim J (2006) 3D space handwriting recognition with ligature model. In: Ubiquitous computing systems. Lecture Notes in Computer Science, vol 4239. Springer, Heidelberg, pp 41–56

  14. Lee HK, Kim J (1999) An hmm-based threshold model approach for gesture recognition. IEEE Trans Pattern Anal Mach Intell 21(10):961–973. doi:10.1109/34.799904

    Article  Google Scholar 

  15. Lyons K, Starner T, Plaisted D, Fusia J, Lyons A, Drew A, Looney EW (2004) Twiddler typing: one-handed chording text entry for mobile phones. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’04)

  16. MacKenzie IS, Soukoreff RW, Helga J (2011) 1 thumb, 4 buttons, 20 words per minute: design and evaluation of h4-writer. In: Proceedings of the 24th annual ACM symposium on user interface software and technology (UIST’11)

  17. McGuire R, Hernandez-Rebollar J, Starner T, Henderson V, Brashear H, Ross D (2004) Towards a one-way american sign language translator. In: Proceedings of the sixth IEEE international conference on automatic face and gesture recognition (FGR’04)

  18. Mistry P, Maes P, Chang L (2009) Wuw-wear ur world: a wearable gestural interface. In: Proceedings of the 27th international conference extended abstracts on human factors in computing systems (CHI EA ’09)

  19. Mitra S, Acharya T (2007) Gesture recognition: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 37:311–324

    Article  Google Scholar 

  20. Odell J, Valtchev V, Woodland P, Young S (1994) A one pass decoder design for large vocabulary recognition. In: Proceedings of the workshop on human language technology. Association for Computational Linguistics, pp 405–410

  21. Plamondon R, Srihari S (2000) Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans Pattern Anal Mach Intell 22(1):63–84

    Article  Google Scholar 

  22. Rabiner L. (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Google Scholar 

  23. Raffa G, Lee J, Nachman L, Song J (2010) Don’t slow me down: bringing energy efficiency to continuous gesture recognition. In: Proceedings of the international symposium on wearable computers (ISWC’10)

  24. Schultz T (2002) GlobalPhone: a multilingual speech and text database developed at Karlsruhe University. In: Proceedings of the international conference on spoken language processing (ICSLP’02)

  25. Soltau H, Metze F, Fügen C, Waibel A (2001) A one-pass decoder based on polymorphic linguistic context assignment. In: IEEE workshop on automatic speech recognition and understanding (ASRU ’01)

  26. Stiefmeier T, Roggen D, Tröster G, Ogris G, Lukowicz P (2008) Wearable activity tracking in car manufacturing. IEEE Pervasive Comput 7(2):42

    Article  Google Scholar 

  27. Stolcke A (2002) Srilm—an extensible language modeling toolkit. In: International conference on spoken language processing

  28. Tamaki E, Miyaki T, Rekimoto J (2009) Brainy hand: an ear-worn hand gesture interaction device. In: Proceedings of the 27th international conference extended abstracts on human factors in computing systems (CHI EA ’09)

  29. Woodman OJ (2007) An introduction to inertial navigation. Technical report. University of Cambridge

Download references

Author information

Authors and Affiliations

  1. Cognitive Systems Lab, Institute for Anthropomatics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

    Christoph Amma, Marcus Georgi & Tanja Schultz

Authors
  1. Christoph Amma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Georgi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tanja Schultz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoph Amma.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Amma, C., Georgi, M. & Schultz, T. Airwriting: a wearable handwriting recognition system. Pers Ubiquit Comput 18, 191–203 (2014). https://doi.org/10.1007/s00779-013-0637-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-013-0637-3

Keywords

Search

Navigation