Abstract
A system is described for extracting and matching mathematical formulas presented orally during a lecture with those simultaneously displayed on the lecture room screen. Each mathematical formula spoken by the lecturer and displayed on the screen is extracted and shown to the transcriber. Investigation showed that, in a lecture in which many mathematical formulas were presented, about 80% of them were both spoken and pointed to on the screen, meaning that the system can help a transcriber correctly transcribe up to 80% of the formulas presented. A speech recognition system is used to extract the formulas from the lecturer’s speech, and a system that analyzes the trajectory of the end of the stick pointer is used to extract the formulas from the projected images. This information is combined and used to match the pointed-to formulas with the spoken ones. In testing using actual lectures, this system extracted and matched 71.4% of the mathematical formulas both spoken and displayed and presented them for transcription with a precision of 89.4%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Kato, N., Kawano, S., Miyoshi, S., Nishioka, T., Murakami, H., Minagawa, H., Wakatsuki, D., Shirasawa, M., Ishihara, Y., Naito, I.: Subjective Evaluation of Displaying Keywords for Speech to Text Service Operators. The Transactions of Human Interface Society 9(2), 195–203 (2007) (in Japanese)
Miyoshi, S., Kawano, S., Nishioka, T., Kato, N., Shirasawa, M., Murakami, H., Minagawa, H., Ishihara, Y., Naito, I., Wakatsuki, D., Kuroki, H., Kobayashi, M.: A Basic Study on Supplementary Visual Information for Real-Time Captionists in the Lecture of Information Science. IEICE Transactions on Information and Systems (Japanese Edition) J91-D(9), 2236–2246 (2008)
Miyoshi, S., Kuroki, H., Kawano, S., Shirasawa, M., Ishihara, Y., Kobayashi, M.: Support Technique for Real-Time Captionist to Use Speech Recognition Software. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 647–650. Springer, Heidelberg (2008)
Wald, M., Bain, K.: Universal access to communication and learning: role of automatic speech recognition. Universal Access in the Information Society 6(4), 435–447 (2007)
Sakiyama, T., Mukunoki, M., Katsuo, I.: Detection of the Indicated Area with an Indication Stick. In: Int. Conf. on Multimodal Interfaces, pp. 480–487 (2000)
Marutani, T., Nishiguchi, S., Kakusho, K., Minoh, M.: Making a lecture content with deictic information about indicated objects in lecture materials. In: AEARU Workshop on Network Education, pp. 70–75 (2005)
Takeuchi, Y., Saito, K., Ito, A., Ohnishi, N., Iizuka, S., Nakajima, S.: Extracting Pointing Object with Demonstrative Speech Phrase for Remote Transcription in Lecture. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 624–631. Springer, Heidelberg (2008)
Ito, A., Saito, K., Takeuchi, Y., Ohnishi, N., Iizuka, S., Nakajima, S.: A Study on Demonstrative Words Extraction in Instructor Utterance on Communication Support for Hearing Impaired Persons. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 632–639. Springer, Heidelberg (2008)
Takeuchi, Y., Ohta, H., Ohnishi, N., Wakatsuki, D., Minagawa, H.: Extraction of Displayed Objects Corresponding to Demonstrative Words for Use in Remote Transcription. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A.I. (eds.) ICCHP 2010. LNCS, vol. 6180, pp. 152–159. Springer, Heidelberg (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Takeuchi, Y., Kawaguchi, H., Ohnishi, N., Wakatsuki, D., Minagawa, H. (2012). A System for Matching Mathematical Formulas Spoken during a Lecture with Those Displayed on the Screen for Use in Remote Transcription. In: Miesenberger, K., Karshmer, A., Penaz, P., Zagler, W. (eds) Computers Helping People with Special Needs. ICCHP 2012. Lecture Notes in Computer Science, vol 7382. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31522-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-31522-0_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31521-3
Online ISBN: 978-3-642-31522-0
eBook Packages: Computer ScienceComputer Science (R0)