Skip to main content
SpringerLink
Log in

Arabic glove-talk (AGT): A communication aid for vocally impaired

  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

This paper presents a practical real time system for mapping dynamic glove-based hand gestures into Arabic speech. Arabic Glove-Talk (AGT) is a prototype for an intelligent system implemented to solve the problem of communication between the vocally impaired and other people. Various reasons increase the difficulty of dynamic gesture recognition. Neuro-fuzzy approaches are described to overcome this difficulty. The difficult task of gesture spotting is solved using a distance-based measure. We use the 5th Glove device to capture hand gestures. The system learns to recognise a basic vocabulary of 32 gestures. The basic vocabulary is extended to 128 gestures is tested on a test set, including 640 gestures using different types of classifiers to assign an unknown gesture to the corresponding spoken Arabic word. The minimum distance classifier, the neuro-fuzzy perceptron and the 1D-self-organising feature map based classifier result in 96.25%, 97.82% and 100% correct spoken words, respectively. After training, talkers successfully produced Arabic speech at nearly 75–90 words per minute.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cipolla R, Okaoto Y, Kuno Y. Robust structure from motion using motion parallax. Proceedings of the 4th International Conference on Computer Vision, IEEE Press 1993: 374–382

  2. Davis J, Shah M. Recognizing hand gestures. Proceedings of the European Conference on Computer Vision, 1, Springer-Verlag 1994; 331–340

  3. Starner T, Pentland A. Visual recognition of American Sign Language using Hidden Markov Models. Proceedings of the International Workshop on Face and Gesture Recognition, Zurich 1995: 184–194

  4. Maggioni M. Gesture computer — new ways of operating a computer. Proceedings of the International Workshop on Face and Gesture Recognition, Zurich 1995: 166–171

  5. Heap T, Hogg D. Toward 3D hand tracking using a deformable model. International Conference of Face and Gesture Recognition, Killington, VT 1996: 140–145

  6. Moghaddam B, Pentland A. Probabilistic visual learning for object recognition. Proceedings of the 5th International Conference on Computer Vision, IEEE Press 1995: 768–793

  7. Kunch J, Huang T. Vision based hand modeling and tracking for virtual teleconferencing and telecollaboration. Proceedings of the International Conference on Computer Vision, IEEE Press 1995: 666–671

  8. Ahmad T, Taylor C, Lanitis A, Cootes T. Tracking and recognizing hand gestures using statistical shape models. Image & Vision Computing 1997; 15: 345–352

    Google Scholar 

  9. Freeman W, Weissman C. Television control by hand gestures. Proceedings of the first IWAFGR 1995: 179–183

  10. Pandya A, Mcy R. Pattern Recognition with Neural Networks in C++. CRC Press, 1996

  11. Fels S. Glove-TalkII: Mapping Hand Gestures to Speech Using Neural Networks — An Approach to Building Adaptive Interfaces. PhD thesis, Computer Science Department, University of Toronto, 1994

  12. Fels S, Hinton G. GloveTalk: A neural network inteface between a DataGlove and a speech synthesizer. IEEE Transactions on Neural Networks 1993; 4: 2–8

    Google Scholar 

  13. Wexelblat D. A feature-based approach to continuous gesture analysis. Master's thesis, MIT, 1993

  14. Takahashi T, Kishino F. Gesture coding based in experiments with a hand gesture interface device. SIGCHI Bulletin 1991; 23(2): 67–73

    Google Scholar 

  15. Murakami K, Taguchi H. Gesture recognition using recurrent neural networks. Computer-Human Interactin '91 Conference Proceedings 1991: 237–242

  16. Kramer J, Leifer L. The Talking Glove: A speaking aid for nonvocal deaf and deaf-blind individuals. Proceedings of RESNA 12th Annual Conference 1989: 471–472

  17. Kramer J, Leifer LA. Talking Glove for nonverbal deaf individuals. Technical Report CDR TR 1990 0312, Centre For Design Research, Stanford University, 1990

  18. Kramer J. Communication system for deaf, deaf-blind and nonvocal individuals using instrumented gloves. Patent 5,047,952, Virtual Technologies, 1991

  19. Hand C, Sexton I, Mullan M. A linguistic approach to the recognition of hand gestures. In: Designing Future Interaction Conference. Ergonomics Society/IEE, April 1994

  20. Becker DA. Sensei: A Real-Time Recognition, feedback and Training system for T'ai Chi Gestures. MSc Thesis, Massachuetts Institute of Technology, May 1997

  21. Hallahan WI. DECtalk software: Text-to-speech technology and implementation. Digital Technical Journal 11 April 1996

  22. Nauck D. A fuzzy perceptron as a generic model for neuro-fuzzy approaches. Proceedings Fuzzy-Systeme'94, Munich, October 1994

  23. Tolba, AS, Abu-Rezeq AN. A self organizing feature map for automated visual inspection of textile products. Computers in Industry 1997; 32(3): 319–333

    Google Scholar 

  24. Lee C, Yangsheng Xu. Online interactive learning of gestures for human/robor interface. IEEE International Conference on Robotics and Automation, Minneapolis 1996; 4: 2982–2987

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Physics Department, Kuwait University, P.O. Box 5969, 13060, Safat, Kuwait

    A. S. Tolba & A. N. Abu-Rezq

Authors
  1. A. S. Tolba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. N. Abu-Rezq
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. S. Tolba.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Tolba, A.S., Abu-Rezq, A.N. Arabic glove-talk (AGT): A communication aid for vocally impaired. Pattern Analysis & Applic 1, 218–230 (1998). https://doi.org/10.1007/BF01234769

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF01234769

Keywords

Search

Navigation