Abstract
The current study examined how to use crowdsourcing to convert sign language-to-text. Generally in Japan, a sign language interpreter reads and vocalizes the sign language of the speaker, and caption typists generate captions from the vocalization. However, this method doubles labor costs and delays caption provision. Therefore, we developed a system that interprets sign language-to-caption text via crowdsourcing, with non-experts performing interpretations. While many individuals classified as deaf/hard-of-hearing (DHH) who can read sign language are suitable for this task, not all of them possess adequate typing skills. To address this, our system divides live sign language video into shorter segments, distributing them to workers. After the worker interprets and types the segments to text, the system generates captions through integration of these texts. Furthermore, we provide a user interface for playback speed control and one second rewinding in order to improve the ease with which tasks are completed. Our system can establish an environment that not only allows the interpretation of sign language-to-caption text, but also provides an opportunity for DHH individuals to assist those that are unable read sign language. We conducted a test using our prototype system for sign language-to-text interpretation. The mean time it took a worker to finish a task was 26 s for a 9 s segment. The combined total rate of missing text and collision between segments was 66%. Analysis of questionnaire responses found that workers assigned fewer tasks considered the tasks more enjoyable.
This study was partially supported by JSPS KAKENHI Grant Number JP15K01056 and JP19K02996.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Preminger, J.E., Levitt, J.E.: Computer-Assisted Remote Transcription (CART): a tool to aid people who are deaf or hard of hearing in the workplace. Volta Rev. 99(4), 219–230 (1997)
Elliot, L.B., Stinson, M.S., McKee, B.G., Everhart, V.S., Francis, P.J.: College students’ perceptions of the c-print speech-to-text transcription system. J. Deaf Stud. Deaf Educ. 6(4), 285–298 (2001)
Kurita, S.: Development of CART software “IPtalk” and captioning services via personal computers: Communication support for the hearing-impaired. J. Inf. Process. Manag. 59(6), 366–376 (2016). https://doi.org/10.1241/johokanri.59.366
Wakatsuki, D., Kato, N., Shionome, T., Kawano, S., Nishioka, T., Naito, I.: Development of web-based remote speech-to-text interpretation system captionline. JACIII 21(2), 310–320 (2017). https://doi.org/10.20965/jaciii.2017.p0310
Lasecki, W.S., et al.: Real-time captioning by groups of non-experts. In: Proceedings of the ACM Symposium on User Interface Software and Technology (UIST 2012), pp. 23–34 (2012)
Lasecki, W.S., Miller, C.D., Bigham, J.P.: Warping time for more effective real-time crowdsourcing. In: Proceedings of the International ACM Conference on Human Factors in Computing Systems (CHI 2013), pp. 2033–2036 (2013)
Shiraishi, Y., Zhang, J., Wakatsuki, D., Kumai, K., Morishima, A.: Crowdsourced real-time captioning of sign language by deaf and hard-of-hearing people. Int. J. Pervasive Comput. Commun. 13(1), 2–25 (2017)
Kumai, K., Zhang, J., Shiraishi, Y., Wakatsuki, D., Kitagawa, H., Morishima, A.: Group rotation management in real-time crowdsourcing. In: Proceedings of 19th International Conference on Information Integration and Web-based Applications & Services (iiWAS 2017), pp. 23–31 (2017)
Kumai, K., et al.: Skill-and-stress-aware assignment of crowd-worker groups to task streams. In: The sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP2018), pp. 88–97 (2018)
Shionome, T., et al.: Complement of incomplete task results for real-time crowdsourcing interpretation. In: Proceedings of 21th International Conference on Asian Language Processing (IALP 2017), pp. 359–362 (2017)
Hashimoto, H., Matsubara, M., Shiraishi, Y., Wakatsuki, D., Zhang J., Morishima, A.: A task assignment method considering inclusiveness and activity degree. In: Proceedings of The Second IEEE Workshop on Human-in-the-loop Methods and Human Machine Collaboration in BigData (IEEE HMData2018), pp. 3498–3503 (2018)
YouTube Live. https://www.youtube.com/live?gl=JP&hl=ja. Accessed 4 June 2020
Node.js. https://nodejs.org/. Accessed 4 June 2020
YouTube Player API. https://developers.google.com/youtube/iframe_api_reference?hl=ja. Accessed 4 June 2020
Acknowledgements
We would like to thank Editage (www.editage.com) for English language editing.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Tanaka, K., Wakatsuki, D., Minagawa, H. (2020). A Study Examining a Real-Time Sign Language-to-Text Interpretation System Using Crowdsourcing. In: Miesenberger, K., Manduchi, R., Covarrubias Rodriguez, M., Peňáz, P. (eds) Computers Helping People with Special Needs. ICCHP 2020. Lecture Notes in Computer Science(), vol 12377. Springer, Cham. https://doi.org/10.1007/978-3-030-58805-2_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-58805-2_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58804-5
Online ISBN: 978-3-030-58805-2
eBook Packages: Computer ScienceComputer Science (R0)