Shyam Krishna
ABSTRACT
We present a complete system to automatically translate English content to Indian Sign Language, using animated avatars to perform the output. This is the first work of this scale in ISL, including our own animation and translation system. Our work is further intended to be completely open sourced to foster collaboration from the general public and interested HI members, and also serve as a starting point for future research in the field.
Sign language is the predominant means of communication of the significant population of Hearing Impaired (HI) in India, who are very disadvantaged with regards to access to media and education due to lack of resources. Our work seeks to bridge this gap by providing a translation system that can be used by the HI.
The pipeline of the system consists of a rule based English-to-ISL gloss Machine Translation framework followed by an animation module using hand-crafted animations playing the output sequentially. The process is aided by a English-ISL bilingual translation corpus and a video corpus of ISL performance at both sentence and individual sign levels.
- "Directory of ISL Interpreters.." Indian Sign Language Research and Training Centre, Ministry of Social Justice & Empowerment, Government of India, 2019, www.islrtc.nic.in/directory-isl-interpreters.Google Scholar
- "Hand Talk App." Hand Talk, handtalk.me/en/app.Google Scholar
- "ISL Dictionary Launch." Indian Sign Language Research and Training Centre, Ministry of Social Justice & Empowerment, Government of India, 2019, www.islrtc.nic.in/isl-dictionary-launch.Google Scholar
- Almeida, Inês Rodrigues. Exploring challenges in avatar-based translation from european portuguese to portuguese sign language. Diss. Master's thesis, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal, 2104, 2014.Google Scholar
- Bangham, J. Andrew, et al. "Virtual signing: Capture, animation, storage and transmission-an overview of the visicast project." (2000): 6--6.Google Scholar
- Bungeroth, Jan, and Hermann Ney. "Statistical sign language translation." Workshop on representation and processing of sign languages, LREC. Vol. 4. 2004.Google Scholar
- Cihan Camgoz, Necati, et al. "Neural sign language translation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.Google Scholar
- Courty, Nicolas, and Sylvie Gibet. "Why is the creation of a virtual signer challenging computer animation?." International Conference on Motion in Games. Springer, Berlin, Heidelberg, 2010.Google Scholar
- d'Armond, L. Speers. Representation of American Sign Language for machine translation. Diss. Georgetown University, 2002.Google Scholar
- Dasgupta, Tirthankar, and Anupam Basu. "Prototype machine translation system from text-to-Indian sign language." Proceedings of the 13th international conference on Intelligent user interfaces. 2008.Google Scholar
- Eberhard, David M., Gary F. Simons, and Charles D. Fennig (eds.). 2020. Ethnologue: Languages of the World. Twenty-third edition. Dallas, Texas: SIL International. Online version: http://www.ethnologue.com.Google Scholar
- Ebling, Sarah, and John Glauert. "Exploiting the full potential of JASigning to build an avatar signing train announcements." (2013).Google Scholar
- Elliott, Ralph, et al. "An overview of the sigml notation and sigml signing software system." Fourth International Conference on Language Resources and Evaluation, LREC. 2004.Google Scholar
- Grieve-Smith, Angus B. "SignSynth: A sign language synthesis application using Web3D and Perl." International Gesture Workshop. Springer, Berlin, Heidelberg, 2001.Google Scholar
- Gu, Jiatao, et al. "Universal neural machine translation for extremely low resource languages." arXiv preprint arXiv:1802.05368 (2018).Google Scholar
- Hanke, Thomas. "HamNoSys-representing sign language data in language resources and language processing contexts." LREC. Vol. 4. 2004.Google Scholar
- Kalakar, "Indian Male Character", Sketchfab, sketchfab.com/3d-models/indian-male-character-a311adcb82a3457db16a548a8d261114Google Scholar
- Kaur, Khushdeep, and Parteek Kumar. "HamNoSys to SiGML conversion system for sign language automation." Procedia Computer Science 89.10.1016 (2016).Google Scholar
- Kausar, Sumaira, M. Younus Javed, and Shaleeza Sohail. "Recognition of gestures in Pakistani sign language using fuzzy classifier." Proceedings of the 8th conference on Signal processing, computational geometry and artificial vision. World Scientific and Engineering Academy and Society (WSEAS), 2008.Google Scholar
- Kennaway, J. R., John RW Glauert, and Inge Zwitserlood. "Providing signed content on the Internet by synthesized animation." ACM Transactions on Computer-Human Interaction (TOCHI) 14.3 (2007): 15-es.Google ScholarDigital Library
- Khan, Nabeel Sabir, Adnan Abid, and Kamran Abid. "A Novel Natural Language Processing (NLP)--Based Machine Translation Model for English to Pakistan Sign Language Translation." Cognitive Computation: 1--18.Google Scholar
- Kyle, Fiona E., and Kate Cain. "A comparison of deaf and hearing children's reading comprehension profiles." Topics in Language Disorders 35.2 (2015): 144--156.Google ScholarCross Ref
- Lample, Guillaume, et al. "Phrase-based & neural unsupervised machine translation." arXiv preprint arXiv:1804.07755 (2018).Google Scholar
- López-Colino, Fernando, and José Colás. "Spanish sign language synthesis system." Journal of Visual Languages & Computing 23.3 (2012): 121--136.Google ScholarDigital Library
- Marshall, Ian, and Éva Sáfár. "A prototype text to British Sign Language (BSL) translation system." The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics. 2003.Google Scholar
- McDonald, John, et al. "An automated technique for real-time production of lifelike animations of American Sign Language." Universal Access in the Information Society 15.4 (2016): 551--566.Google ScholarDigital Library
- Morrissey, Sara. Data-driven machine translation for sign languages. Diss. Dublin City University, 2008.Google Scholar
- Ott, Myle, et al. "fairseq: A fast, extensible toolkit for sequence modeling." arXiv preprint arXiv:1904.01038 (2019).Google Scholar
- Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.Google Scholar
- Punch, Studio New, "Britney-Human Female", Unity Asset Store, Unity Technologies, assetstore.unity.com/packages/3d/characters/humanoids/humans/britney-human-female-58766.Google Scholar
- Sarkar, Biswajit, et al. "A translator for bangla text to sign language." 2009 Annual IEEE India Conference. IEEE, 2009.Google Scholar
- Singha, Joyeeta, and Karen Das. "Recognition of Indian sign language in live video." arXiv preprint arXiv:1306.1301 (2013).Google Scholar
- Stoll, Stephanie, et al. "Text2Sign: towards sign language production using neural machine translation and generative adversarial networks." International Journal of Computer Vision (2020): 1--18.Google Scholar
- Tripathi, Kumud, and Neha Baranwal GC Nandi. "Continuous Indian sign language gesture recognition and sentence formation." Procedia Computer Science 54 (2015): 523--531.Google ScholarCross Ref
- Uddin, Md Azher, and Shayhan Ameen Chowdhury. "Hand sign language recognition for bangla alphabet using support vector machine." 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET). IEEE, 2016.Google Scholar
- Veale, Tony, Alan Conway, and Bróna Collins. "The challenges of cross-modal translation: English-to-Sign-Language translation in the Zardoz system." Machine Translation 13.1 (1998): 81--106.Google ScholarDigital Library
- Zeshan, Ulrike. "Indo-Pakistani Sign Language grammar: a typological outline." Sign Language Studies (2003): 157--212.Google Scholar
- Zhao, Liwei, et al. "A machine translation system from English to American Sign Language." Conference of the Association for Machine Translation in the Americas. Springer, Berlin, Heidelberg, 2000.Google Scholar
Index Terms
- Virtual Indian Sign Language Interpreter
Recommendations
Comparative Analysis of Translation Systems from Indian Languages to Indian Sign Language
AbstractThe language exclusively used by hearing-impaired people to interact with one another is classified as sign language. It is a gesture-based language used by hearing-impaired people to share their ideas, thoughts, feelings, etc. Research into ...
Mapping Hindi Text To Indian sign language with Extension Using Wordnet
AICTC '16: Proceedings of the International Conference on Advances in Information Communication Technology & ComputingSign language serves as a primary mode of communication for deaf people. It is a major linguistic challenge which researchers all over the globe have taken up and built systems to facilitate the process of understanding the natural language by means of ...
Text to Sign Language Translation System: A Review of Literature
Many machine translation systems for spoken languages are available, but the translation system between the spoken and Sign Language are limited. The translation from Text to Sign Language is different from the translation between spoken languages ...
Comments