Abstract
In this paper we propose a visual-based speech learning framework to assist deaf persons by comparing the lip movements between a student and an E-tutor in an intelligent tutoring system. The framework utilizes lip reading technologies to determine if a student learns the correct pronunciation. Different from conventional speech recognition systems, which usually recognize a speaker’s utterance, our speech learning framework focuses on recognizing whether a student pronounces are correct according to an instructor’s utterance by using visual information. We propose a method by extracting dynamic shape difference features (DSDF) based on lip shapes to recognize the pronunciation difference. The preliminary experimental results demonstrate the robustness and effectiveness of our approach on a database we collected, which contains multiple persons speaking a small number of selected words.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Awad, S.: The Application of Digital Speech Processing to Stuttering Therapy. IEEE Instrumentation and Measurement (1997)
Chen, S., Tian, Y., Liu, Q., Metaxas, D.: Segment and Recognize Expression Phase by Fusion of Motion Area and Neutral Divergence Features. In: IEEE Int’l Conf. on Automatic Face and Gesture Recognition, AFGR (2011)
Chen, S., Tian, Y., Liu, Q., Metaxas, D.: Recognizing Expressions from Face and Body Gesture by Temporal Normalized Motion and Appearance Features. In: IEEE Int’l Conf. Computer Vision and Pattern Recognition Workshop for Human Communicative Behavior Analysis, CVPR4HB (2011)
Cootes, T., Taylor, C., Cooper, D., Graham, J.: Active Shape Models – Their Training and Application. Computer Vision and Image Understanding (1995)
Hailpern, J., Karahalios, K., DeThorne, L., Halle, J.: Encouraging Speech and Vocalization in Children with Autistic Spectrum Disorder. In: Workshop on Technology in Mental Health, CHI 2008 (2008)
Lavagetto, F.: Converting speech into lip movements: a multimedia telephone for hard of hearing people. IEEE Transactions on Rehabilitation Engineering 3(1), 90–102 (1995)
Marschark, M., Sapere, P., Convertino, C., Mayer, C., Wauters, L., Sarchet, T.: Are deaf students’ reading challenges really about reading? American Annals of the Deaf 154(4), 357-176 (2009)
Matthews, I., Cootes, T., Bangham, J., Cox, S., Harvey, R.: Extraction of visual features for lipreading. TPAMI 24(2), 198–213 (2002)
Potamianos, G., Neti, C., Gravier, G., Garg, A., Senior, A.: Recent Advances in the Automatic Recognition of Audio-Visual Speech. Proceedings of the IEEE 91(9), 1306–1326 (2003)
Rahman, M., Ferdous, S., Ahmed, S.: Increasing Intelligibility in the Speech of the Autistic Children by an Interactive Computer Game. In: IEEE International Symposium on Multimedia (2010)
Riella, R., Linarth, A., Lippmann, L., Nohama, P.: Computerized System to Aid Deaf Children in Speech Learning. In: IEEE EMBS International Conference (2001)
Schipor, O., Pentiuc, S., Schipor, M.: Towards a Multimodal Emotion Recognition Framework to Be Integrated in a Computer Based Speech Therapy System. In: IEEE Conference on Speech Technology and Human Computer Dialogue, SpeD (2011)
Wei, Y.: Research on Facial Expression Recognition and Synthesis, Master Thesis (2009) Software available at http://code.google.com/p/asmlibrary
Zhao, G., Barnard, M., Pietikainen, M.: Lipreading with local spatialtemporal descriptors. TMM 11(7), 1254–1265 (2009)
Zhou, Z., Zhao, G., Pietikainen, M.: Toward a Practical Lipreading System. In: CVPR (2011)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Chen, S., Quintian, D.M., Tian, Y. (2012). Towards a Visual Speech Learning System for the Deaf by Matching Dynamic Lip Shapes. In: Miesenberger, K., Karshmer, A., Penaz, P., Zagler, W. (eds) Computers Helping People with Special Needs. ICCHP 2012. Lecture Notes in Computer Science, vol 7382. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31522-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-31522-0_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31521-3
Online ISBN: 978-3-642-31522-0
eBook Packages: Computer ScienceComputer Science (R0)