skip to main content
10.1145/3491101.3519697acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
poster

SignFind: A Synchronized Sign Language and Chinese Character Teaching Game for Chinese Deaf Children Using Gesture Recognition

Published: 28 April 2022 Publication History

Abstract

This paper presents a targeted system to help Chinese deaf children learn both sign language and Chinese characters in early language learning. The system combines sign recognition and in-air writing techniques with games so that children can practice sign language and Chinese character writing at the same time. Sign recognition is used to assess the accuracy of sign language and in-air writing records the process of writing Chinese characters. In addition, the game adds to the learning fun and makes children more willing to complete learning tasks. We have developed a prototype to evaluate the effectiveness of a simultaneous Chinese sign language and Chinese character teaching system based on gesture recognition and in-air writing. We expect that this system will increase children's willingness and efficiency to learn sign language and Chinese characters, and eventually be used to assist in early language education for deaf children in the Chinese region.

Supplementary Material

MP4 File (3491101.3519697-video-figure.mp4)
Video Figure
MP4 File (3491101.3519697-talk-video.mp4)
Talk Video

References

[1]
Christoph Amma, Marcus Georgi, and Tanja Schultz. 2014. Airwriting: a wearable handwriting recognition system. Personal Ubiquitous Comput. 18, 1 (January 2014), 191–203.
[2]
Dhruva Bansal, Prerna Ravi, Matthew So, Pranay Agrawal, Ishan Chadha, Ganesh Murugappan, and Colby Duke. 2021. CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 481, 1–10.
[3]
Helene Brashear, Valerie Henderson, Kwang-Hyun Park, Harley Hamilton, Seungyon Lee, and Thad Starner. 2006. American sign language recognition in game development for deaf children. In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility (Assets '06). Association for Computing Machinery, New York, NY, USA, 79–86.
[4]
Bach Duy Khuat, Duong Thai Phung, Ha Thi Thu Pham, Anh Ngoc Bui, and Son Tung Ngo. 2021. Vietnamese sign language detection using Mediapipe. In 2021 10th International Conference on Software and Computer Applications (ICSCA 2021). Association for Computing Machinery, New York, NY, USA, 162–165.
[5]
Danielle Bragg, Raja Kushalnagar, and Richard Ladner. 2018. Designing an Animated Character System for American Sign Language. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18). Association for Computing Machinery, New York, NY, USA, 282–294.
[6]
Eyal Krupka, Kfir Karmon, Noam Bloom, Daniel Freedman, Ilya Gurvich, Aviv Hurvitz, Ido Leichter, Yoni Smolin, Yuval Tzairi, Alon Vinnikov, and Aharon Bar-Hillel. 2017. Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, New York, NY, USA, 1887–1898.
[7]
Seungyon Lee, Valerie Henderson, Harley Hamilton, Thad Starner, Helene Brashear, and Steven Hamilton. 2005. A gesture-based american sign language game for deaf children. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). Association for Computing Machinery, New York, NY, USA, 1589–1592.
[8]
Chen Lv. 2021. Exploring the value of Chinese language teaching in special education. Chinese Character Culture (20), 104-105.
[9]
Lu, Ming-Tsan Pierre, Gregory L. Hallman Jr, and John B. Black. 2013. Chinese character learning: Using embodied animations in initial stages. Journal of Technology and Chinese language teaching 4.2 (2013): 1-24.
[10]
Jiajia Li, Zixia Zheng, Xiemin Wei, and Guanyun Wang. 2021. FaceMe: An Augmented Reality Social Agent Game for Facilitating Children's Learning about Emotional Expressions. The Adjunct Publication of the 34th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Machinery, New York, NY, USA, 17–19.
[11]
Shuai Li, Jing Chen, Mengda Li, Jie Lin, and Guanyun Wang. 2017. ColorOdor: Odor Broadens The Color Identification of The Blind. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). Association for Computing Machinery, New York, NY, USA, 2746–2751.
[12]
Goldin-Meadow, S., Mylander, C., & Franklin, A. . 2007. How children make language out of gesture: morphological structure in gesture systems developed by american and chinese deaf children. Cogn Psychol, 55(2), 87-135.
[13]
M. Mohandes, S. Aliyu and M. Deriche. 2014.Arabic sign language recognition using the leap motion controller, 2014 IEEE 23rd International Symposium on Industrial Electronics (ISIE). 960-965.
[14]
Renata Cristina Barros Madeo. 2011. Brazilian sign language multimedia hangman game: a prototype of an educational and inclusive application. In The proceedings of the 13th international ACM SIGACCESS conference on Computers and accessibility (ASSETS '11). Association for Computing Machinery, New York, NY, USA, 311–312.
[15]
Jakob Nielsen. 1994. Enhancing the explanatory power of usability heuristics. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). Association for Computing Machinery, New York, NY, USA, 152–158.
[16]
Jakob Nielsen. 1995. Usability inspection methods. In Conference Companion on Human Factors in Computing Systems (CHI '95). Association for Computing Machinery, New York, NY, USA, 377–378.
[17]
Mateo A. Parreño, Carmen J. Celi, Washington X. Quevedo, David Rivas, and Víctor H. Andaluz. 2017. Teaching-Learning of Basic Language of Signs through Didactic Games. In Proceedings of the 2017 9th International Conference on Education Technology and Computers (ICETC 2017). Association for Computing Machinery, New York, NY, USA, 46–51.
[18]
Ning Xu, Weiqiang Wang, and Xiwen Qu. 2015. A discriminative classifier for in-air handwritten Chinese characters recognition. In Proceedings of the 7th International Conference on Internet Multimedia Computing and Service (ICIMCS '15). Association for Computing Machinery, New York, NY, USA, Article 16, 1–4.
[19]
Songhua Xie, Hailiang Yang, and Hui Nie. 2020. Design of Chinese Character Recognition Based on AlexNet Convolution Neural Network. In Proceedings of the 2020 3rd International Conference on Artificial Intelligence and Pattern Recognition (AIPR 2020). Association for Computing Machinery, New York, NY, USA, 68–73.
[20]
Itaguchi Y, Yamada C, Fukuzawa K. 2015. Writing in the Air: Contributions of Finger Movement to Cognitive Processing. PLoS ONE 10(6): e0128419.
[21]
Itaguchi Y, Yamada C, Yoshihara M, Fukuzawa K. 2017. Writing in the air: A visualization tool for written languages. PLoS ONE 12(6): e0178735.
[22]
Yifan Yang, Leijing Zhou, Rujian Li, Hang Yao, Jialu Song, and Fangtian Ying. 2019. Chinese Character Learning System. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (CHI EA '19). Association for Computing Machinery, New York, NY, USA, Paper LBW2218, 1–5.
[23]
Zhan, Hong, and Hsiu-Jen Cheng. 2014. The role of technology in teaching and learning Chinese characters. International Journal of Technology in Teaching and Learning,10.2:147.
[24]
Zahoor Zafrulla, Helene Brashear, Thad Starner, Harley Hamilton, and Peter Presti. 2011. American sign language recognition with the kinect. In Proceedings of the 13th international conference on multimodal interfaces (ICMI '11). Association for Computing Machinery, New York, NY, USA, 279–286.
[25]
Zhai Zhou, Yanke Wei. 2021. An analysis of the real-life dilemma of sign language teaching in deaf schools in China. Journal of Suihua College.

Cited By

View all
  • (2024)Augmented Reality Application for Improving Writing and Motoric Skills in Children With Disabilities2024 47th MIPRO ICT and Electronics Convention (MIPRO)10.1109/MIPRO60963.2024.10569369(718-723)Online publication date: 20-May-2024
  • (2024)An Efficient Bidirectional Android Translation Prototype for Yemeni Sign Language Using Fuzzy logic and CNN Transfer Learning ModelsIEEE Access10.1109/ACCESS.2024.3512455(1-1)Online publication date: 2024
  • (2023)Supporting ASL Communication Between Hearing Parents and Deaf ChildrenThe 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614511(1-5)Online publication date: 22-Oct-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems
April 2022
3066 pages
ISBN:9781450391566
DOI:10.1145/3491101
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Chinese Characters
  2. Chinese Sign Language
  3. Gesture Interaction
  4. Gesture Recognition
  5. Gesture-based Gaming
  6. In-air Writing
  7. Teaching Methods

Qualifiers

  • Poster
  • Research
  • Refereed limited

Conference

CHI '22
Sponsor:
CHI '22: CHI Conference on Human Factors in Computing Systems
April 29 - May 5, 2022
LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)104
  • Downloads (Last 6 weeks)6
Reflects downloads up to 27 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Augmented Reality Application for Improving Writing and Motoric Skills in Children With Disabilities2024 47th MIPRO ICT and Electronics Convention (MIPRO)10.1109/MIPRO60963.2024.10569369(718-723)Online publication date: 20-May-2024
  • (2024)An Efficient Bidirectional Android Translation Prototype for Yemeni Sign Language Using Fuzzy logic and CNN Transfer Learning ModelsIEEE Access10.1109/ACCESS.2024.3512455(1-1)Online publication date: 2024
  • (2023)Supporting ASL Communication Between Hearing Parents and Deaf ChildrenThe 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614511(1-5)Online publication date: 22-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media