Natsuhiko Hirabayashi
ABSTRACT
Our purpose of this study is to increase efficiency of learning sign language and fingerspelling, the visual language.
In general, illustrations and videos are used as learning materials to study these languages. However, the learning materials are drawn from the view point of someone to talk to. Therefore, learners need to imagine finger shape that looked from the person while they study. Understanding the motion is easier if learners can watch it in three dimensions. Therefore, taking lessons by a teacher who has already mastered it will be the best way. However, the method is not appropriate to study by themselves.
Then, we devise a new method to increase the efficiency of learning by increasing learning opportunities by developing equipment that learners can study alone. We propose learning support equipment using glasses-shaped wearable device and mixed reality. By using mixed reality, learners can watch sign language and fingerspelling motion in three dimensions as if learners looked at these languages in the real world.
In the previous studies, the equipment was developed by using augmented reality. AR marker was used for the operation. With this equipment, learners have controlled movement and rotation of 3D model.
In this study, we develop the equipment used mixed reality by selecting the holographic button with gesture. Therefore, learners can operate the equipment without using the other device and this make them possible to study more intuitive. With glasses-shaped wearable device, learners can also study sign language and fingerspelling mimicking the gesture with both hands.
As we build sign language learning support contents, learners can study sign language and fingerspelling with them. However, the present equipment is short of 3D animation of sign language. Therefore, we are planning to make 3D animation by capturing the motions that hearing-impaired person use.
We had a trial test for hearing-impaired person and a sign language interpreter in order to get real impression. As a result, we got an opinion that some 3D animations are not accurate and complicated to understand. Also, we got an opinion that 3D model direction is not clear after the rotation.
According to these opinions, we corrected as follows: the correction of 3D animation, adding function of displaying the track of the finger, and the function of selecting the holographic reset button for the 3D model direction.
- Yoshinori Fujisawa, Shoichi Ito, Kenya Kobayashi: Development of Learning Support System for Fingerspelling by Augmented Reality, Proceedings of The 5th International Conference on Intelligent Systems and Image Processing 2017, p. 492--495, 2017Google Scholar
Cross Ref
- Japan Audiological Society: https://audiology-japan.jp/audiology-japan/wp-content/uploads/2014/12/a1360e77a580a13ce7e259a406858656.pdfGoogle Scholar
- Ministry of Health, Labour and Welfare: https://www.mhlw.go.jp/file/06-Seisakujouhou-12200000-Shakaiengokyokushougaihokenfukushibu/0000172197.pdfGoogle Scholar
- Ministry of Health, Labour and Welfare: https://www.mhlw.go.jp/toukei/saikin/hw/shintai/06/dl/01.pdfGoogle Scholar
- Microsoft HoloLens: https://www.microsoft.com/ja-jp/hololensGoogle Scholar
- Blender: https://www.blender.org/Google Scholar
- PERCEPTION NEURON: https://neuronmocap.com/jaGoogle Scholar
Index Terms
- Development of Learning Support Equipment for Sign Language and Fingerspelling by Mixed Reality
Recommendations
Teaching American Sign Language in Mixed Reality
This paper presents a holistic system to scale up the teaching and learning of vocabulary words of American Sign Language (ASL). The system leverages the most recent mixed-reality technology to allow the user to perceive her own hands in an immersive ...
Improving realism in automated fingerspelling of American sign language
AbstractFingerspelling is a process of communicating letters of a spoken language alphabet using a person’s hand or hands. Portraying animations of fingerspelling has proved surprisingly resistant to automation because of the collisions that arise from ...
A teaching system of Japanese sign language using sign language recognition and generation
MULTIMEDIA '02: Proceedings of the tenth ACM international conference on MultimediaIn recent years, the number of sign language learners is increasing in Japan. And there are many teaching materials of sign language such as textbooks, videotapes and software for PCs. However, these teaching materials have several problems that ...
Comments