Skip to main content

Advertisement

SpringerLink
Log in

Automated 3D sign language caption generation for video

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Efforts to make online media accessible to a regional audience have picked up pace in recent years with multilingual captioning and keyboards. However, techniques to extend this access to people with hearing loss are limited. Further, owing to a lack of structure in the education of hearing impaired and regional differences, the issue of standardization of Indian Sign Language (ISL) has been left unaddressed, forcing educators to rely on the local language to support the ISL structure, thereby creating an array of correlations for each object, hindering the language building skills of a student. This paper aims to present a useful technology that can be used to leverage online resources and make them accessible to the hearing-impaired community in their primary mode of communication. Our tool presents an avenue for the early development of language learning and communication skills essential for the education of children with a profound hearing loss. With the proposed technology, we aim to provide a standardized teaching and learning medium to a classroom setting that can utilize and promote ISL. The goals of our proposed system involve reducing the burden of teachers to act as a valuable teaching aid. The system allows for easy translation of any online video and correlation with ISL captioning using a 3D cartoonish avatar aimed to reinforce classroom concepts during the critical period. First, the video gets converted to text via subtitles and speech processing methods. The generated text is understood through NLP algorithms and then mapped to avatar captions which are then rendered to form a cohesive video alongside the original content. We validated our results through a 6-month period and a consequent 2-month study, where we recorded a 37% and 70% increase in performance of students taught using Sign captioned videos against student taught with English captioned videos. We also recorded a 73.08% increase in vocabulary acquisition through signed aided videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Agarwal, A., Thakur, M.K.: Sign language recognition using microsoft kinect. In: 2013 Sixth International Conference on Contemporary Computing (IC3), pp. 181–185 (2013). https://doi.org/10.1109/IC3.2013.6612186

  2. Ahmed, M.A., Zaidan, B.B., Zaidan, A.A., Salih, M.M., Lakulu, M.M.B.: A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017. Sensors 18(7), 2208 (2018). https://doi.org/10.3390/s18072208

    Article  Google Scholar 

  3. Ahmed, S.: Real time American sign language video captioning using deep neural networks [PowerPoint presentation]. http://on-demand.gputechconf.com/gtc/2017/presentation/s7346-syed-ahmed-real-time-american-sign-language-video-caption.pdf (2018)

  4. Cannon, J.E., Fredrick, L.D., Easterbrooks, S.R.: Vocabulary instruction through books read in American sign language for English-language learners with hearing loss. Commun. Disord. Q. 31(2), 98–112 (2010). https://doi.org/10.1177/1525740109332832

    Article  Google Scholar 

  5. Center, L.C.N.D.E.: Learning american sign language: Books, media, and classes. http://www3.gallaudet.edu/clerc-center/info-to-go/asl/learning-asl-books_media_classes.html. [Online]

  6. Chambers, B., Abrami, P.C., McWhaw, K., Therrien, M.C.: Developing a computer-assisted tutoring program to help children at risk learn to read. Educ. Res. Eval. 7(2–3), 223–239 (2001). https://doi.org/10.1076/edre.7.2.223.3863

    Article  Google Scholar 

  7. Chandramouli, C.: Census of India 2011, provisional population totals, government of India. http://censusindia.gov.in/2011-prov-results/paper2/data_files/india/paper2_1.pdf (2011). [Online]

  8. Division, S.S.: Disabled persons in India, a statistical profile 2016. http://mospi.nic.in/sites/default/files/publication_reports/Disabled_persons_in_India_2016.pdf (2016). [Online]

  9. Easterbrooks, S.R., Huston, S.G.: The signed reading fluency of students who are deaf/hard of hearing. J. Deaf Stud. Deaf Educ. 13(1), 37–54 (2007). https://doi.org/10.1093/deafed/enm030

    Article  Google Scholar 

  10. Fung, P.C., Chow, B.W.Y., McBride-Chang, C.: The impact of a dialogic reading program on deaf and hard-of-hearing kindergarten and early primary school-aged students in Hong Kong. J. Deaf Educ. Deaf Stud. 10(1), 82–95 (2005)

    Article  Google Scholar 

  11. Goyal, L., Goyal, V.: Automatic translation of English text to Indian sign language synthetic animations. In: Proceedings of the 13th International Conference on Natural Language Processing, pp. 144–153 (2016)

  12. Hermans, D., Knoors, H., Ormel, E., Verhoeven, L.: The relationship between the reading and signing skills of deaf children in bilingual education programs. J. Deaf Stud. Deaf Educ. 13(4), 518–530 (2008). https://doi.org/10.1093/deafed/enn009

    Article  Google Scholar 

  13. Huenerfauth, M.: Generating American sign language animation: overcoming misconceptions and technical challenges. Univers. Access Inf. Soc. 6(4), 419–434 (2008). https://doi.org/10.1007/s10209-007-0095-7

    Article  Google Scholar 

  14. Journal, C.W.R.: Oralism and how it affects the development of the deaf child. http://www.csus.edu/wac/journal/2010/hood_research_paper.pdf (2010)

  15. Kafle, S., Huenerfauth, M.: Evaluating the usability of automatically generated captions for people who are deaf or hard of hearing. In: Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, ASSETS’17, pp. 165–174. ACM, New York, NY, USA (2017). https://doi.org/10.1145/3132525.3132542

  16. Lang, S., Block, M., Rojas, R.: Sign language recognition using kinect. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing, pp. 394–402. Springer Berlin Heidelberg, Berlin (2012)

    Chapter  Google Scholar 

  17. Lazar, J., Feng, J.H., Hochheiser, H.: Research Methods in Human–Computer Interaction. Wiley, New York (2010)

    Google Scholar 

  18. Justice, L.M.: Word exposure conditions and preschoolers’ novel word learning during shared storybook reading. Read. Psychol. 23(2), 87–106 (2002). https://doi.org/10.1080/027027102760351016

    Article  Google Scholar 

  19. Massaro, D.W., Light, J.: Improving the vocabulary of children with hearing loss. Volta Rev. 104(3), 141–174 (2004)

    Google Scholar 

  20. Mayberry, R.I., Eichen, E.B.: The long-lasting advantage of learning sign language in childhood: another look at the critical period for language acquisition. J. Mem. Lang. 30(4), 486–512 (1991). https://doi.org/10.1016/0749-596X(91)90018-F

    Article  Google Scholar 

  21. Michael Strong, P.M.P.: A study of the relationship between American sign language and English literacy. J. Deaf Stud. Deaf Educ. 2(1), 37–46 (1997)

    Article  Google Scholar 

  22. NCERT: Chapter-wise video lessons (2018). http://www.ncert.nic.in/new_ncert/ncert/publication/publication_list/list_of_publication3.html. [Online]

  23. Capirci, O., Cattani, A., Rossini, P., Volterra, V.: Teaching sign language to hearing children as a possible factor in cognitive enhancement. J. Deaf Stud. Deaf Educ. 3(2), 135–142 (1998). https://doi.org/10.1093/oxfordjournals.deafed.a014343

    Article  Google Scholar 

  24. Pettengill, R.: Motionsavvy uni (2011). http://www.motionsavvy.com/about.html. [Online]

  25. RKMVERI: Isl dictionary (2004). http://www.indiansignlanguage.org/. [Online]

  26. Robbins, C., Ehri, L.C.: Reading storybooks to kindergartners helps them learn new vocabulary words. J. Educ. Psychol. 86(1), 54–64 (1994). https://doi.org/10.1037/0022-0663.86.1.54

    Article  Google Scholar 

  27. Santorini, B.: Part-Of-Speech tagging guidelines for the Penn Treebank project (3rd revision, 2nd printing). Tech. rep., Department of Linguistics, University of Pennsylvania, Philadelphia, PA, USA (1990)

  28. Schleper, D.R.: Read it again... and again and again. Perspect. Educ. Deaf. 14(2), 16–19 (1995)

    Google Scholar 

  29. Society, D.A.: Sheila Kothavala Institute for the Deaf. http://deafaidsociety.in/ (2018)

  30. Sousa, L., Rodrigues, J.M.F., Monteiro, J., Cardoso, P.J.S., Lam, R.: Gygsla: a portable glove system for learning sign language alphabet. In: Antona, M., Stephanidis, C. (eds.) Universal Access in Human-Computer Interaction. Users and Context Diversity, pp. 159–170. Springer, Cham (2016)

    Chapter  Google Scholar 

  31. Strong, M., Prinz, P.: A study of the relationship between American sign language and English literacy. J. Deaf Stud. Deaf Educ. 2(1), 37–46 (1997)

    Article  Google Scholar 

  32. Talk, S.: Signtalk: The resource for sign language interpreters. http://signtalk.org/ (2018)

  33. Talkinghands: Indian sign language dictionary (2013). http://www.talkinghands.co.in/. [Online]

  34. Tenório, R.: Virtual interpreter for Brazillian sign language. https://www.handtalk.me/ (2012). [Online]

  35. Verma, D., Dash, P., Bhaskar, S., Pal, R.P., Jain, K., Srivastava, R.P., Hansraj, N.: Disabled persons in India: a statistical profile 2016. http://mospi.nic.in/sites/default/files/publication_reports/Disabled_persons_in_India_2016.pdf (2017). [Online]

  36. Karnabadhir Vidyalaya, P.: PADSAD-the unique school aiming to main stream every hearing impairment and divyang student. http://www.padsad.org/ (2018)

  37. Wideman, C.: Vcom3d. http://www.vcom3d.com/ (2002)

  38. Wiggins, E.: Total communication as an education philosophy. http://www.deaflinx.com/DeafEd/OptionsGuide/TC.html (1998). [Online]

  39. Wobbrock, J.O., Kientz, J.A.: Research contributions in human–computer interaction. Interactions 23(3), 38–44 (2016). https://doi.org/10.1145/2907069

    Article  Google Scholar 

  40. Ye, Y., Tian, Y., Huenerfauth, M., Liu, J.: Recognizing American sign language gestures from within continuous videos. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2018)

  41. Zeshan, U., Vasishta, M., Sethna, M.: Implementation of Indian sign language in educational setting. Asia Pacific Disab. Rehabil. J. 16(1), 16–40 (2005)

    Google Scholar 

  42. Zheng, L., Liang, B., Jiang, A.: Recent advances of deep learning for sign language recognition. In: 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), Sydney, Australia, pp. 1–7 (2017). https://doi.org/10.1109/DICTA.2017.8227483

Download references

Acknowledgements

The authors would like to acknowledge Mrs. Sucheta S Saundankar, Principal, Padsad Karanbadhir Vidyalaya Nashik, and Mrs. P Jessy, Principal, Sheila Kothavala Institute for the Deaf, Bangalore, India, for allowing us to conduct the validation study at their respective schools. We are grateful to the Principal, Jessy Samuel and Speech Therapist, Ms. Suvasini Isaac for personally verifying the signs used in the validation stage for accuracy and consistency. We thank the anonymous reviewers whose insightful comments and suggestions have significantly improved this paper.

Author information

Authors and Affiliations

  1. Department of Information and Communication Technology, Manipal Institute of Technology, MAHE, Manipal, 576104, India

    Nayan Mehta & Sanjay Singh

  2. Department of Electronics and Communication Engineering, Manipal Institute of Technology, MAHE, Manipal, 576104, India

    Suraj Pai

Authors
  1. Nayan Mehta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suraj Pai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanjay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sanjay Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mehta, N., Pai, S. & Singh, S. Automated 3D sign language caption generation for video. Univ Access Inf Soc 19, 725–738 (2020). https://doi.org/10.1007/s10209-019-00668-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-019-00668-9

Keywords

Search

Navigation