Skip to main content
SpringerLink
Log in

Wearable Computers for Sign Language Recognition

  • Chapter
  • First Online:
Handbook of Large-Scale Distributed Computing in Smart Healthcare

Part of the book series: Scalable Computing and Communications ((SCC))

Abstract

A Sign Language Recognition (SLR) system translates signs performed by deaf individuals into text/speech in real time. Low cost sensor modalities, inertial measurement unit (IMU) and surface electromyography (sEMG), are both useful to detect hand/arm gestures. They are capable of capturing signs and are complementary to each other for recognizing signs. In this book chapter, we propose a wearable system for recognizing American Sign Language (ASL) in real-time, fusing information from an inertial sensor and sEMG sensors. The best subset of features from a wide range of well-studied features is selected using an information gain based feature selection approach. Four popular classification algorithms are evaluated for 80 commonly used ASL signs on four subjects. With the selected feature subset and a support vector machine classifier, our system achieves 96.16 and 85.24% average accuracies for intra-subject and intra-subject cross session evaluation respectively. The significance of adding sEMG for American Sign Language recognition is explored and the best channel of sEMG is highlighted.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. W. C. Stokoe, “Sign language structure: An outline of the visual communication systems of the American deaf,” Journal of deaf studies and deaf education, vol. 10, no. 1, pp. 3–37, 2005.

    Google Scholar 

  2. D. Barberis, N. Garazzino, P. Prinetto, G. Tiotto, A. Savino, U. Shoaib, and N. Ahmad, “Language resources for computer assisted translation from italian to italian sign language of deaf people,” in Proceedings of Accessibility Reaching Everywhere AEGIS Workshop and International Conference, Brussels, Belgium (November 2011), 2011.

    Google Scholar 

  3. A. B. Grieve-Smith, “Signsynth: A sign language synthesis application using web3d and PERL,” in Gesture and Sign Language in Human-Computer Interaction, pp. 134–145, Springer, 2002.

    Google Scholar 

  4. B. Vicars, “Basic ASL: First 100 signs.”

    Google Scholar 

  5. E. Costello, American sign language dictionary. Random House Reference &, 2008.

    Google Scholar 

  6. T. Starner, J. Weaver, and A. Pentland, “Real-time american sign language recognition using desk and wearable computer based video,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 20, no. 12, pp. 1371–1375, 1998.

    Google Scholar 

  7. C. Vogler and D. Metaxas, “A framework for recognizing the simultaneous aspects of American sign language,” Computer Vision and Image Understanding, vol. 81, no. 3, pp. 358–384, 2001.

    Google Scholar 

  8. T. E. Starner, “Visual recognition of American sign language using Hidden Markov models.,” tech. rep., DTIC Document, 1995.

    Google Scholar 

  9. C. Oz and M. C. Leu, “American sign language word recognition with a sensory glove using artificial neural networks,” Engineering Applications of Artificial Intelligence, vol. 24, no. 7, pp. 1204–1213, 2011.

    Google Scholar 

  10. E. Malaia, J. Borneman, and R. B. Wilbur, “Analysis of ASL motion capture data towards identification of verb type,” in Proceedings of the 2008 Conference on Semantics in Text Processing, pp. 155–164, Association for Computational Linguistics, 2008.

    Google Scholar 

  11. A. Y. Benbasat and J. A. Paradiso, “An inertial measurement framework for gesture recognition and applications,” in Gesture and Sign Language in Human-Computer Interaction, pp. 9–20, Springer, 2002.

    Google Scholar 

  12. O. Amft, H. Junker, and G. Troster, “Detection of eating and drinking arm gestures using inertial body-worn sensors,” in Wearable Computers, 2005. Proceedings. Ninth IEEE International Symposium on, pp. 160–163, IEEE, 2005.

    Google Scholar 

  13. A. B. Ajiboye and R. F. Weir, “A heuristic fuzzy logic approach to EMG pattern recognition for multifunctional prosthesis control,” Neural Systems and Rehabilitation Engineering, IEEE Transactions on, vol. 13, no. 3, pp. 280–291, 2005.

    Google Scholar 

  14. J.-U. Chu, I. Moon, and M.-S. Mun, “A real-time EMG pattern recognition based on linear-nonlinear feature projection for multifunction myoelectric hand,” in Rehabilitation Robotics, 2005. ICORR 2005. 9th International Conference on, pp. 295–298, IEEE, 2005.

    Google Scholar 

  15. Y. Li, X. Chen, X. Zhang, K. Wang, and J. Yang, “Interpreting sign components from accelerometer and sEMG data for automatic sign language recognition,” in Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pp. 3358–3361, IEEE, 2011.

    Google Scholar 

  16. Y. Li, X. Chen, X. Zhang, K. Wang, and Z. J. Wang, “A sign-component-based framework for chinese sign language recognition using accelerometer and sEMG data,” Biomedical Engineering, IEEE Transactions on, vol. 59, no. 10, pp. 2695–2704, 2012.

    Google Scholar 

  17. L.-G. Zhang, Y. Chen, G. Fang, X. Chen, and W. Gao, “A vision-based sign language recognition system using tied-mixture density hmm,” in Proceedings of the 6th international conference on Multimodal interfaces, pp. 198–204, ACM, 2004.

    Google Scholar 

  18. M. M. Zaki and S. I. Shaheen, “Sign language recognition using a combination of new vision based features,” Pattern Recognition Letters, vol. 32, no. 4, pp. 572–577, 2011.

    Google Scholar 

  19. T.-Y. Pan, L.-Y. Lo, C.-W. Yeh, J.-W. Li, H.-T. Liu, and M.-C. Hu, “Real-time sign language recognition in complex background scene based on a hierarchical clustering classification method,” in Multimedia Big Data (BigMM), 2016 IEEE Second International Conference on, pp. 64–67, IEEE, 2016.

    Google Scholar 

  20. M. W. Kadous et al., “Machine recognition of AUSLAN signs using powergloves: Towards large-Lexicon recognition of sign language,” in Proceedings of the Workshop on the Integration of Gesture in Language and Speech, pp. 165–174, Citeseer, 1996.

    Google Scholar 

  21. M. G. Kumar, M. K. Gurjar, and M. S. B. Singh, “American sign language translating glove using flex sensor,” Imperial Journal of Interdisciplinary Research, vol. 2, no. 6, 2016.

    Google Scholar 

  22. D. Sherrill, P. Bonato, and C. De Luca, “A neural network approach to monitor motor activities,” in Engineering in Medicine and Biology, 2002. 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES Conference, 2002. Proceedings of the Second Joint, vol. 1, pp. 52–53, IEEE, 2002.

    Google Scholar 

  23. X. Chen, X. Zhang, Z.-Y. Zhao, J.-H. Yang, V. Lantz, and K.-Q. Wang, “Hand gesture recognition research based on surface EMG sensors and 2D-accelerometers,” in Wearable Computers, 2007 11th IEEE International Symposium on, pp. 11–14, IEEE, 2007.

    Google Scholar 

  24. V. E. Kosmidou and L. J. Hadjileontiadis, “Sign language recognition using intrinsic-mode sample entropy on sEMG and accelerometer data,” Biomedical Engineering, IEEE Transactions on, vol. 56, no. 12, pp. 2879–2890, 2009.

    Google Scholar 

  25. S. Wei, X. Chen, X. Yang, S. Cao, and X. Zhang, “A component-based vocabulary-extensible sign language gesture recognition framework,” Sensors, vol. 16, no. 4, p. 556, 2016.

    Google Scholar 

  26. J. Kim, J. Wagner, M. Rehm, and E. André, “Bi-channel sensor fusion for automatic sign language recognition,” in Automatic Face & Gesture Recognition, 2008. FG’08. 8th IEEE International Conference on, pp. 1–6, IEEE, 2008.

    Google Scholar 

  27. J.-S. Wang and F.-C. Chuang, “An accelerometer-based digital pen with a trajectory recognition algorithm for handwritten digit and gesture recognition,” Industrial Electronics, IEEE Transactions on, vol. 59, no. 7, pp. 2998–3007, 2012.

    Google Scholar 

  28. V. Nathan, J. Wu, C. Zong, Y. Zou, O. Dehzangi, M. Reagor, and R. Jafari, “A 16-channel bluetooth enabled wearable EEG platform with dry-contact electrodes for brain computer interface,” in Proceedings of the 4th Conference on Wireless Health, p. 17, ACM, 2013.

    Google Scholar 

  29. C. J. De Luca, L. Donald Gilmore, M. Kuznetsov, and S. H. Roy, “Filtering the surface emg signal: Movement artifact and baseline noise contamination,” Journal of biomechanics, vol. 43, no. 8, pp. 1573–1579, 2010.

    Google Scholar 

  30. I. Mesa, A. Rubio, I. Tubia, J. De No, and J. Diaz, “Channel and feature selection for a surface electromyographic pattern recognition task,” Expert Systems with Applications, vol. 41, no. 11, pp. 5190–5200, 2014.

    Google Scholar 

  31. R. Merletti and P. Di Torino, “Standards for reporting EMG data,” J Electromyogr Kinesiol, vol. 9, no. 1, pp. 3–4, 1999.

    Google Scholar 

  32. A. Phinyomark, C. Limsakul, and P. Phukpattaranont, “A novel feature extraction for robust EMG pattern recognition,” arXiv preprint arXiv:0912.3973, 2009.

  33. M. Zhang and A. A. Sawchuk, “Human daily activity recognition with sparse representation using wearable sensors,” Biomedical and Health Informatics, IEEE Journal of, vol. 17, no. 3, pp. 553–560, 2013.

    Google Scholar 

  34. S. H. Khan and M. Sohail, “Activity monitoring of workers using single wearable inertial sensor.”

    Google Scholar 

  35. O. Paiss and G. F. Inbar, “Autoregressive modeling of surface EMG and its spectrum with application to fatigue,” Biomedical Engineering, IEEE Transactions on, no. 10, pp. 761–770, 1987.

    Google Scholar 

  36. A. M. Khan, Y.-K. Lee, S. Y. Lee, and T.-S. Kim, “A triaxial accelerometer-based physical-activity recognition via augmented-signal features and a hierarchical recognizer,” Information Technology in Biomedicine, IEEE Transactions on, vol. 14, no. 5, pp. 1166–1172, 2010.

    Google Scholar 

  37. I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,” The Journal of Machine Learning Research, vol. 3, pp. 1157–1182, 2003.

    Google Scholar 

  38. J. R. Quinlan, C4. 5: programs for machine learning. Elsevier, 2014.

    Google Scholar 

  39. C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Transactions on Intelligent Systems and Technology, vol. 2, pp. 27:1–27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

  40. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The weka data mining software: an update,” ACM SIGKDD explorations newsletter, vol. 11, no. 1, pp. 10–18, 2009.

    Google Scholar 

  41. M. Z. Jamal, “Signal acquisition using surface EMG and circuit design considerations for robotic prosthesis,” 2012.

    Google Scholar 

  42. J. Wu, Z. Tian, L. Sun, L. Estevez, and R. Jafari, “Real-time american sign language recognition using wrist-worn motion and surface EMG sensors,” in Wearable and Implantable Body Sensor Networks (BSN), 2015 IEEE 12th International Conference on, pp. 1–6, IEEE, 2015.

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Science Foundation, under grants CNS-1150079 and ECCS-1509063. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding organizations.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Texas A&M University, College Station, USA

    Jian Wu

  2. Departments of Biomedical Engineering, Computer Science and Engineering, and Electrical and Computer Engineering, Center of Remote Health Technologies and Systems, Texas A&M University, College Station, USA

    Roozbeh Jafari

Authors
  1. Jian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roozbeh Jafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Wu .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer Engineering, North Dakota State University , Fargo, North Dakota, USA

    Samee U. Khan

  2. School of Information Technologies, University of Sydney, Sydney, New South Wales, Australia

    Albert Y. Zomaya

  3. Department of Computer Science, COMSATS Institute of Information Technology, Islamabad, Pakistan

    Assad Abbas

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Wu, J., Jafari, R. (2017). Wearable Computers for Sign Language Recognition. In: Khan, S., Zomaya, A., Abbas, A. (eds) Handbook of Large-Scale Distributed Computing in Smart Healthcare. Scalable Computing and Communications. Springer, Cham. https://doi.org/10.1007/978-3-319-58280-1_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-58280-1_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-58279-5

  • Online ISBN: 978-3-319-58280-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation