Skip to main content
SpringerLink
Log in

ICANDO: Low cost multimodal interface for hand disabled people

  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

The article presents the multimodal user interface ICANDO (Intellectual Computer AssistaNt for Disabled Operators) that was awarded with the first prize of the Loco Mummy Software Contest in 2006. The interface is intended mainly for assistance to the persons without hands or with disabilities of their hands or arms but could be useful for ordinary users at hands-free contactless human-computer interaction too. It combines the module for automatic recognition of voice commands in English, French and Russian as well as the head tracking module in one multimodal interface. ICANDO interface was applied for hands-free work with Graphical User Interface of a personal computer in such tasks as Internet communication and work with graphical and text documents. The article describes the aim and the architecture of the interface, the methods for speech recognition and head tracking, information fusion and synchronization of the multimodal streams. The presented results of testing and exploitation of ICANDO user interface have confirmed high accuracy and robustness of the interface for contactless operation with a computer. The comparison of multimodal and standard ways of interaction has discovered that the first one is slower by a factor of 1.9 that is quite well for hands-free interaction between a computer and an impaired person.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. S. Argyropoulos, K. Moustakas, A. Karpov, O. Aran, D. Tzovaras, T. Tsakiris, G. Varni, and B. Kwon, A multimodal framework for the communication of the disabled”, inProceedings of the eNTERFACE 2007 Summer Workshop on Multimodal Interfaces, (Istanbul, Turkey), 2007. 21

  2. O. Aran, I. Ari, A. Benoit, A. Carrillo, F. Fanard, P. Campr, L. Akarun, A. Caplier, M. Rombaut, and B. Sankur, “Sign-Tutor: An Interactive Sign Language Tutoring Tool”, inProceedings of the eNTERFACE 2006 Summer Workshop on Multimodal Interfaces, (Dubrovnik, Croatia), 2006. 21

  3. A. Ronzhin, A. Karpov, M. Zelezny, and R. Mesheryakov, “Smart Multimodal Assistant for Disabled”, inProceedings of the 12 th International Conference on Human-Computer Interaction HCI International 2007, (Beijing, PR China), 2007. 21

  4. P. Cisar, J. Zelinka, M. Zelezny, A. Karpov, and A. Ronzhin, “Audio-Visual Speech Recognition for Slavonic Languages (Czech and Russian)”, inProceedings of the 11-th International Conference on Speech and Computer SPECOM 2006, (St. Petersburg, Russia), pp. 493–498, 2006. 21

  5. M. Zelezny, Z. Krnoul, P. Cisar, and J. Matousek, “Design, implementation and evaluation of the Czech realistic audio-visual speech synthesis”,Signal Processing, vol. 86, no. 12, pp. 3657–3673, 2006. 21

    Article  Google Scholar 

  6. V. Bondarenko, L. Balatskaya, and E. Choynzonov, “Application of biological feedback in the system for rehabilitation after full glottal resection”,Siberian Oncology Journal, vol. 4, pp. 17–20, 2004. 21

    Google Scholar 

  7. F. Corno, L. Farinetti, and I. Signorile, “An eye-gaze input device for people with severe motor disabilities”, inProceedings of the SSGRR-2002: International Conference on Advances in Infrastructure for e-Business, e-Education, e-Science, and e-Medicine on the Internet, (L’Aquila, Italy). 2002. 22

  8. “LC Technologies, Inc. Eyegaze Systems”. http://www.eyegaze.com. 22

  9. F. T. Garcia-Moreno, “Eye Gaze Tracking System Visual Mouse Application Development”, tech. rep., Ecole Nationale Superiere de Physique de Strasbourg (ENSPS) and School of Computer Science Queen’s University Belfast, 2001. 22

  10. K. Grauman, M. Betke, J. Lombardi, J. Gips, and G. Bradski, “Communication via Eye Blinks and Eyebrow Raises: Video-Based Human-Computer Interfaces”,Universal Access in the Information Society, vol. 4, pp. 359–373, 2003. 22

    Article  Google Scholar 

  11. R. Bates and H. Istance, “Why are eye mice unpopular? A detailed comparison of head and eye controlled assistive technology pointing devices”, inProceedings of the 1-st Cambridge Workshop on Universal Access and Assistive Technology, (Cambridge, UK), 2002. 22

  12. S. Oviatt,Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, ch. Multimodal interfaces, pp. 286–304. Lawrence Erlbaum Assoc, Mahwah, NJ, 2003. 22

    Google Scholar 

  13. A. Karpov, A. Ronzhin, and A. Cadiou, “A multimodal system ICANDO: Intellectual Computer AssistaNt for Disabled Operators”, inProceedings of the 9-th International Conference on Spoken Language Processing INTERSPEECH-ICSLP 2006, (Pittsburgh, PA, USA), pp. 1998–2001, 2006. 22

  14. J. Richter,Programming Applications for Microsoft Windows. Microsoft Press, 1999. 22

  15. A. Ronzhin and A. Karpov, “Russian Voice Interface”,Pattern Recognition and Image Analysis, Vol. 17, no. 2, pp. 321–336, 2007. 23

    Article  Google Scholar 

  16. A. Ronzhin, A. Karpov, A. Timofeev, and M. Litvinov, “Multimodal human-computer interface for assisting neurosurgical system”, inProceedings of the 11-th International Conference on Human-Computer Interaction HCII 2005, (Las Vegas, USA), 2005. 23

  17. S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland,The HTK Book (for HTK Version 3.4). Cambridge University Engineering Department, 2006. 23

  18. D. Ward, A. Blackwell, and D. MacKay, “Dasher: A data entry interface using continuous gestures and language models”, inProceedings of the ACM Symposium on User Interface Software and Technology UIST’2000, (New York, USA), pp. 129–137, New York: ACM, 2000. 24

    Google Scholar 

  19. “NaturalPoint SmartNAV trackers”. http://www.naturalpoint.com/smartnav. 24

  20. “InterSence trackers”. http://www.isense.com. 24

  21. J.-Y. Bouguet, “Pyramidal implementation of the lucas kanade feature tracker”, tech. rep., Intel Corporation, Microprocessor Research Labs, 2000. 24

  22. A. Benoit and A. Caplier, “Biological approach for head motion detection and analysis”, inProceedings of the 13-th European Signal Processing Conference EUSIPCO-2005, pp. Antalya, Turkey, 2005. 24

  23. A. Karpov, A. Ronzhin, A. Nechaev, and S. Chernakova, “Multimodal system for hands-free PC control”, inProceedings of the 13-th European Signal Processing Conference EUSIPCO-2005, (Antalya, Turkey), 2005. 24

  24. “Intel OpenCV library”. http://www.intel.com/technology/computing/opencv. 24

  25. R. Lienhart and J. Maydt, “An Extended Set of Haar-like Features for Rapid Object Detection”. inProceedings of the IEEE International Conference on Image Processing ICIP2002, (Rochester, New York, USA), pp. 900–903, 2002. 24

  26. D. Gorodnichy and G. Roth, “Nouse “Use your nose as a mouse” perceptual vision technology for hands-free games and interfaces”,Image and Vision Computing, vol. 22, no. 12, pp. 931–942, 2004. 24

    Article  Google Scholar 

  27. “Loco Mummy Software Contest”. http://www.locomummy.org. 28

  28. “SIMILAR, European Network of Excellence”. http://www.similar.cc. 28

  29. “First Channel of the Russian TV”. http://www.ltv.ru/owa/win/ort6_main.main? p_news_title_id=82825&p_news_razdel_id=4. 28

  30. “SPIIRAS Speech Informatics Group web-site”. http://www.spiiras.nw.ru/speech/demo/assistive.html. 28

  31. “OpenInterface European STREP project”. http://www.openinterface.org. 28

Download references

Author information

Authors and Affiliations

  1. Speech Informatics Group, St. Petersburg Institute for Informatics and Automation of the Russian Academy of Sciences, SPIIRAS, Saint-Petersburg, Russia

    Alexey Karpov & Andrey Ronzhin

Authors
  1. Alexey Karpov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrey Ronzhin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexey Karpov.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Karpov, A., Ronzhin, A. ICANDO: Low cost multimodal interface for hand disabled people. J Multimodal User Interfaces 1, 21–29 (2007). https://doi.org/10.1007/BF02910056

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02910056

Keywords

Search

Navigation