Abstract
We examined the effectiveness of multisensory methods in learning animal sounds and the onomatopoeia that describe them. Hearing-impaired individuals find it difficult to learn onomatopoeia because they have limited or no access to auditory information. To address this problem, we developed a device that converts audio information into vibrations, and attempted to broaden the experience of sound by using multisensory methods to stimulate the senses of sight, hearing, and touch. In the rhythm discrimination test using the device, the hearing-impaired group showed better performance with vibration added to the audio information than without vibration. In addition, we designed a science lesson to help hearing-impaired individuals learn cicada songs. This lesson used multiple media including text, images, sounds, sound waveforms, onomatopoeia, and vibrations. The evaluation results showed a significant difference between the students who were able to distinguish the vibrations and those who could not. The former reported that they found the lesson enjoyable and that it was a useful way to learn science.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This study was accepted by the ethics committee of the Tsukuba University of Technology.
References
Kohler, W.: Gestalt Psychology. Liveright Google Scholar, New York (1929)
Köhler, W.: Gestalt psychology. Psychologische Forschung 31(1), 28–30 (1967)
Nakamura, K.: Use of the auditory modality and language acquisitiony. J. Logopedics Phoniatrics 48, 254–262 (2007)
Neville, H.J., et al.: Cerebral organization for language in deaf and hearing subjects. Proc. Natl. Acad. Sci. USA 95(3), 922–929 (1998). https://doi.org/10.1073/pnas.95.3.922. National Center for Biotechnology Information
Okada, R., et al.: The deaf utilize phonological representations in visually presented verbal memory tasks. Neurosci. Res. National Center Biotechnol. Inf. 90, 83–89 (2015). https://doi.org/10.1016/j.neures.2014.11.004
Arata, M., Imai, M., Namatame, M., Okuda, J., Okada, H., Matsuda, T.: Semantic processing of mimetic words in deaf individuals. An fMRI study. In: 26th O2-3, Japanese Cognitive Science Society (2009). (in Japanese)
Hinton, L., Nichols, J., Ohala, J.J.: Sound Symbolism. Cambridge University Press, Cambridge (2006). Digitally printed first paperback version
Yamaguchi, N.: Giongo gitaigo jiten (Onomatopoeia Mimetic words Dictionary). Kodansha, Tokyo (2003). (in Japanese)
Ono, M.: Nihongo onomatope jiten (Japanese Mimetics Dictionary). Shogakkan, Tokyo (2007). (in Japanese)
Miyazaki, A., Tomimatsu, K.: Onomato planets: physical computing of Japanese onomatopoeia. In: Proceedings of the 3rd International Conference on Tangible and Embedded Interaction. ACM, Cambridge (2009)
Imai, M., Kita, S., Nagumo, M., Okada, H.: Sound symbolism facilitates early verb learning. Cognition 109, 54–65 (2008). https://doi.org/10.1016/j.cognition.2008.07.015
Yusuf, M., Asaga, C., Watanabe, C.: Onomatopeta!: developing a Japanese onomatopoeia learning-support system utilizing native speakers cooperation. In: Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, pp. 173–177. IEEE Computer Society Washington, DC (2008). https://doi.org/10.1109/WIIAT.2008.300
Alves, R.L., Lima, C.S., Freire, R.: Correction of the fundamental voice frequency using tactile and visual feedback. In: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM (2015). https://doi.org/10.1145/2700648.2811378
Honda, T., Okamoto, M.: User interface design of sound tactile. In: Miesenberger, K., Fels, D., Archambault, D., Peňáz, P., Zagler, W. (eds.) ICCHP 2014. LNCS, vol. 8548, pp. 382–385. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08599-9_57
Henriquez, P., Travieso, C.M.: Review of automatic fault diagnosis systems using audio and vibration signals. Trans. Syst. Man Cybern.: Syst. 44(5), 642–652 (2013). https://doi.org/10.1109/TSMCC.2013.2257752
Kanebako, J., Yamanaka, T., Namatame, M.: Effect of vibration on listening sound for a person with hearing loss. In: Miesenberger, K., Bühler, C., Penaz, P. (eds.) ICCHP 2016. LNCS, vol. 9759, pp. 419–423. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41267-2_59
Heidenreich, M., et al.: KCNQ4 K+ channels tune mechanoreceptors for normal touch sensation in mouse and man. Nat. Neurosci. 15, 138–145 (2012). https://doi.org/10.1038/nn.2985
Shinohara, K., Kawahara, S.: A cross-linguistic study of sound symbolism: the images of size. In: Berkeley Linguistics Society, the Linguistic Society of America, pp. 396–410 (2010) https://doi.org/10.3765/bls.v36i1.3926
Eberhardt, M.: A study of phonetic symbolism of deaf children. Psychol. Monogr. 52, 23–42 (1940)
Kanebako, J., kusunoki, F., Inagaki, S., Namatame, M.: Proposal for science learning materials using a “VibGrip”. In: Proceedings of the 12th International Conference on Advances in Computer Entertainment Technology, no. 36, pp. 1–3 (2015). https://doi.org/10.1145/2832932.2832935
Spector, F., Maurer, D.: Early sound symbolism for vowel sounds. i-Perception 4(4), 239–241 (2013). https://doi.org/10.1068/i0535
WHO news deafness and hearing loss, 15 March 2018. http://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss
Namatame, M., Kusunoki, F., Inagaki, S.: Basic research on multisensory methods for teaching onomatopoeia to the hearing-impaired - broadening the experience of sound. In: Proceedings of the 10th International Conference on Computer Supported Education, vol. 1, pp. 22–27. SciTePress, INSTICC (2018). https://doi.org/10.5220/0006665800220027
Acknowledgements
This paper is an extension of our presentation [23] at the CSEDU2018 conference. We sincerely thank the CSEDU2018 committee.
We thank the students of the Tsukuba University of Technology who cooperated on the experiment and Kitamura M. who supported the experimentFootnote 1.
This work was supported by JSPS KAKENHI Grant Numbers 18H01046 and 15K12122.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Namatame, M., Kanebako, J., Kusunoki, F., Inagaki, S. (2019). Effectiveness of Multisensory Methods in Learning Onomatopoeia for the Hearing-Impaired. In: McLaren, B., Reilly, R., Zvacek, S., Uhomoibhi, J. (eds) Computer Supported Education. CSEDU 2018. Communications in Computer and Information Science, vol 1022. Springer, Cham. https://doi.org/10.1007/978-3-030-21151-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-21151-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-21150-9
Online ISBN: 978-3-030-21151-6
eBook Packages: Computer ScienceComputer Science (R0)