Abstract
The proposed system relates to an interactive scoring system for learning a language, in which a means such as a web camera is used to capture the learners lip movements and then a score is given by making a comparison with images stored in the database. The images stored in the database are those previously recorded by a teacher. By means of the scoring system, the learner can identify and rectify pronunciation problems concerning the lips and tongue. The system also records sounds as well as images from the student. The proposed system processes this data with multimedia processing techniques. With regard to the interactive perspective, a user-friendly visual interface was constructed to help learners use the system. The learners can choose the words they want to practice by capturing their lip image sequences and speech. The lip region image sequences are extracted automatically as visual feature parameters. Combining the visual and voice parameters, the proposed system calculates the similarity between a learners and a teachers pronunciation. An evaluation score is suggested by the proposed system through the previous similarity computation. By this learning process, learners can see the corresponding lip movement of both themselves and a teacher, and correct their pronunciation accordingly. The learners can use the proposed system to practice their pronunciation as many times as they like, without troubling the human teacher, and thus they are able to take more control of improving their pronunciation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Deng, L., Yu, D., Li, X., Acero, A.: A Long-Contextual-Span Model of Resonance Dynamics for Speech Recognition: Parameter Learning and Recognizer Evaluation. In: IEEE workshop on Automatic Speech Recognition and Understanding, pp. 384–389. IEEE Computer Society Press, Los Alamitos (2005)
Kaynak, M.N.: Analysis of lip geometric features for audio-visual speech recognition. IEEE Transactions Systems, Man and Cybernetics, Part A, 564–570 (2004)
Lee, K.D., Lee, M.J., Lee, S.Y.: Extraction of frame-difference features based on PCA and ICA for lip-reading. In: IEEE International Joint Conference on Neural Networks, pp. 232–237. IEEE Computer Society Press, Los Alamitos (2005)
Leung, S.H., Wang, S.L., Lau, W.H.: Lip Image Segmentation Using Fuzzy Clustering Incorporating an Elliptic Shape Function. IEEE Transactions on Image Processing, 51–62 (2004)
Liang, W., Liu, J., Liu, R.: Automatic Spoken English Test for @Chinese Learners. In: International Conference on Communications, Circuits and Systems, vol. 2, pp. 27–30 (2005)
Littlefield, K., Hashemi-Sakhtsari, J.A.: Performance evaluation of an automatic speech recognizer incorporating a fast adaptive speech separation algorithm. In: IEEE International Conference on acoustic, speech and signal processings, vol. 6, pp. 6–10 (2003)
Mattews, I., Cootes, T.F., Bangham, J.A., Cox, S., Harvey, R.: Extraction of visual features for lipreading. IEEE Trans on Pattern Analysis and Machine Intelligence 24(2), 198–213 (2002)
Mok, L.L.: Visual speech features representation for automatic lip-reading. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 397–400. IEEE Computer Society Press, Los Alamitos (2004)
Ou, G.B., Li, X.: Speaker identification using speech and lip features. In: IEEE International Joint Conference on Neural Network, 2565–2570 (2005)
Silsbee, P.L., Bovik, A.C.: Computer lipreading for improved accuracy in automatic speech recognition, IEEE Trans. Speech Audio Processing, 4(2), 337–351 (1996). Tamburini, F.: A multimedia framework for second language teaching in self-access environments. Computers and Education 32, 137–149 (1999)
Tsou, W., Wang, W., Tzeng, Y.: Applying a multimedia storytelling website in foreign language learning. Computers and Education 47, 17–28 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Huang, WC., Chang-Chien, TL., Lin, HP. (2007). An Intelligent Multimedia E-Learning System for Pronunciations. In: Okuno, H.G., Ali, M. (eds) New Trends in Applied Artificial Intelligence. IEA/AIE 2007. Lecture Notes in Computer Science(), vol 4570. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73325-6_9
Download citation
DOI: https://doi.org/10.1007/978-3-540-73325-6_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-73322-5
Online ISBN: 978-3-540-73325-6
eBook Packages: Computer ScienceComputer Science (R0)