Abstract
In this paper, a novel system is proposed to provide alternative tools and interfaces to blind and deaf-and-mute people and enable their communication and interaction with the computer. Several modules are developed to transform signals into other perceivable forms so that the transmitted message is conveyed despite one’s disabilities. The proposed application integrates haptics, audio and visual output, computer vision, sign language analysis and synthesis, speech recognition and synthesis to provide an interactive environment where the blind and deaf-and-mute users can collaborate. All the involved technologies are integrated into a treasure hunting game application that is jointly played by the blind and deaf-and-mute user. The integration of the multimodal interfaces into a game application serves both as an entertainment and a pleasant education tool to the users.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1–2):116–134
Richter K, Hellenschmidt M (2004) Interacting with the ambience: multimodal interaction and ambient intelligence. In: Proceedings of the W3C workshop on multi-modal interaction, vol 19, Sophia Antipolis, France, July 2004
Marsic I, Medl A, Flanagan J (2000) Natural communication with information systems. Proc IEEE 88:1354–1366
Lumsden J, Brewster SA (2003) A paradigm shift: alternative interaction techniques for use with mobile and wearable devices. In: Proceedings of the 13th annual IBM centers for advanced studies conference (CASCON 2003), Toronto, Canada, pp 97–100
Tangelder JWH, Schouten BAM (2006) Transparent face recognition in an unconstrained environment using a sparse representation from multiple still images. In: ASCI 2006 conference, Lommel, Belgium, June 2006
Raman TV (2003) Multimodal interaction design principles for multimodal interaction. In: Proceedings of computer human interaction (CHI 2003), Fort Lauderdale, USA, pp 5–10
Luciano C, Banerjee P, Florea L, Dawe G (2005) Design of the ImmersiveTouch™: a high-performance haptic augmented virtual reality system. In: Proceedings of the 11th international conference on human-computer interaction, Las Vegas, Nevada, July 2005
Sjostrom C (1999) Touch access for people with disabilities. Licentiate thesis, CERTEC Lund University, Sweden, 1999
Nelson B, Ketelhut D, Clarke J, Bowman C, Dede C (2005) Design-based research strategies for developing a scientific inquiry curriculum in a multi-user virtual environment. Educ Technol 45(1):21–28
Lim CP, Nonis D, Hedberg J (2006) Gaming in a 3D multiuser virtual environment: engaging students in Science lessons. Br J Educ Technol 37(2):211–231
Scoy V, Kawai I, Darrah S, Rash F (2000) Haptic display of mathematical functions for teaching mathematics to students with vision disabilities. In: Haptic human-computer interaction workshop
Moustakas K, Nikolakis G, Tzovaras D, Deville B, Marras I, Pavlek J (2000) Multimodal tools and interfaces for the intercommunication between visually impaired and deaf-and-mute people. In: Proceedings of eNTERFACE 2006, Dubrovnik, Croatia, July 2006
Tamura S, Iwano K, Furui S (2005) A stream-weight optimization method for multi-stream HMMS based on likelihood value normalization. In: Proceedings of the IEEE international conference on acoustics, speech, and signal processing (ICASSP’05), vol 1
Erzin E, Yemez Y, Tekalp A (2005) Multimodal speaker identification using an adaptive classifier cascade based on modality reliability. IEEE Trans Multimedia 7(5):840–852
Yound S et al (2006) The HTK book, HTK Version 3.4. Cambridge University Engineering Department
Rabiner L, Juang B (1993) Fundamentals of speech recognition. Englewood Cliffs, Prentice-Hall
Nefian A, Liang L, Pi X, Liu X, Murphy K (2002) Dynamic Bayesian networks for audio-visual speech recognition. EURASIP J Appl Signal Process 2002(11):1274–1288
Jayaram S, Schmugge S, Shin MC, Tsap LV (2004) Effect of colorspace transformation, the illuminance component, and color modeling on skin detection. In: Proceedings of IEEE conference on computer vision and pattern recognition (CVPR)
Aran O, Akarun L (2006) Recognizing two handed gestures with generative, discriminative and ensemble methods via Fisher kernels. In: International workshop on multimedia content representation, classification and security (MRCS ’06), Istanbul, Turkey, September 2006
Papadogiorgaki M, Grammalidis N, Tzovaras D, Strintzis MG (2005) Text-to-sign language synthesis tool. In: 13th European signal processing conference (EUSIPCO2005), Antalya, Turkey, September 2005
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Argyropoulos, S., Moustakas, K., Karpov, A.A. et al. Multimodal user interface for the communication of the disabled. J Multimodal User Interfaces 2, 105–116 (2008). https://doi.org/10.1007/s12193-008-0012-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-008-0012-2