research-article

Enhanced BERT-based Multi-Head Self-Attention Transformer for Transformation of Marathi Text to Marathi Sign Language Gloss

Authors:

Prachi Pramod Waghmare,

Ashwini Mangesh DeshpandeAuthors Info & Claims

ACM Transactions on Asian and Low-Resource Language Information Processing, Volume 23, Issue 10

Article No.: 140, Pages 1 - 16

https://doi.org/10.1145/3687304

Published: 23 October 2024 Publication History

Abstract

One recent advancement in the field of machine learning is the translation of text into sign language gloss, which is a form of natural language for the deaf community. The work presented is a new method to translate Marathi text to Marathi sign language gloss by combining salient features of Bidirectional Encoder Representation from Transformer (BERT) for tokenization and complementing the tokenized frame with Encoder with Attention mechanism and decoding with the LSTM decoder. The work conducted experiments on the created corpora of Marathi text and Marathi sign language gloss sentence pairs. The experiments that employed three models show that the suggested model performs better than the existing approaches. The results show that the translation of Marathi text to sign gloss achieves improved performances with an accuracy of 91.5% and the BLEU scores BLEU-1: 85, BLEU-2: 75, BLEU-3: 65, and BLEU-4: 57.

Statement of Declaration of Competing Interest

The authors declare that they have no competing interest for the research work reported in this article.

References

[1]

Rubén San-Segundo, R. Barra, L. F. D'haro, Juan Manuel Montero, R. Córdoba, and Javier Ferreiros 2006. A Spanish speech-to-sign language translation system for assisting deaf-mute people. In 9th International Conference on Spoken Language Processing.

[2]

Marshall Ian and Éva Sáfár. 2003. A prototype text to British sign language (BSL) translation system. In 41st Annual Meeting of the Association for Computational Linguistic. 113–116.

[3]

Morrissey Sara. 2008. Data-driven Machine Translation for Sign Languages. PhD dissertation, Dublin City University.

[4]

Othman Achraf and Mohamed Jemni. 2011. Statistical sign language machine translation: From English written text to American sign language gloss. arXiv preprint arXiv:1112.0168.

[5]

Khan Nadeem Jadoon, Waqas Anwar, and Nadir Durrani. 2017. Machine translation approaches and survey for Indian languages. arXiv preprint arXiv:1701.04290.

[6]

Orbay Alptekin and Lale Akarun. 2020. Neural sign language translation by learning tokenization. In 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG’20). 222–228.

[7]

Amin Mohamed, Hesahm Hefny, and Mohammed Ammar. 2021. Sign language gloss translation using deep learning models. International Journal of Advanced Computer Science and Applications 12, 11 (2021), 686--692.

[8]

L. Chin-Yew. 2004. ROUGE: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out.

[9]

Jang Jin Yea, Han-Mu Park, Saim Shin, Suna Shin, Byungcheon Yoon, and Gahgene Gweon. 2022. Automatic gloss-level data augmentation for sign language translation. In 13th Language Resources and Evaluation Conference. 6808–6813.

[10]

Camgoz Necati Cihan, Simon Hadfield, Oscar Koller, Hermann Ney, and Richard Bowden. 2018. Neural sign language translation. In IEEE Conference on Computer Vision and Pattern Recognition. 7784–7793.

[11]

Stoll Stephanie, Necati Cihan Camgoz, Simon Hadfield, and Richard Bowden. 2020. Text2Sign: Towards sign language production using neural machine translation and generative adversarial networks. Int. J. Comput. Vis. 128, 4 (2020), 891–908.

Digital Library

[12]

Yin Kayo. 2020. Sign language translation with transformers. arXiv preprint arXiv:2004.00588.

[13]

Luqman Hamzah and Sabri A. Mahmoud. 2019. Automatic translation of Arabic text-to-Arabic sign language. Univ. Access Inf. Soc. 18 (2019), 939–951.

[14]

D. Das Chakladar, P. Kumar, S. Mandal, P. P. Roy, M. Iwamura, and B. G. Kim. 2021. 3D avatar approach for continuous sign movement using speech/text. Appl. Sci. 11, 8, 3439 (2021), 1–13. DOI:

[15]

Sugandhi, Parteek Kumar, and Sanmeet Kaur. 2020. Sign language generation system based on Indian sign language grammar. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP) 19, 4 (2020), 1–26.

Digital Library

[16]

S. R. Bhagwat, R. P. Bhavsar, and B. V. Pawar. 2021. Translation from simple Marathi sentences to Indian sign language using a phrase-based approach. In International Conference on Emerging Smart Computing and Informatics (ESCI’21). 367–373.

[17]

M. S. Nair, A. P. Nimitha, and Sumam Mary Idicula. 2016. Conversion of Malayalam text to Indian sign language using synthetic animation. In International Conference on Next Generation Intelligent Systems (ICNGIS’16). 1–4.

[18]

Dhanjal Amandeep Singh and Williamjeet Singh. 2018. An automatic conversion of Punjabi text to Indian sign language. EAI Endor. Trans. Scalab. Inf. Syst. 7 (2018), e9.

[19]

Natural language toolkit. Retrieved online: https://www.nltk.org/ for Categorizing and Tagging Words (Accessed on: January - October 2023)

[20]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In 40th Annual Meeting of the Association for Computational Linguistics. 311–318.

[21]

F. A. Gers and E. Schmidhuber. 2001. LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Netw. 12, 6 (2001), 1333–1340.

Digital Library

[22]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems. Curran Associates Inc., Red Hook, NY, USA, 6000–6010.

[23]

N. Arvanitis, C. Constantinopoulos, and D. Kosmopoulos. 2019. Translation of sign language glosses to text using sequence-to-sequence attention models. In 15th International Conference on Signal-Image Technology & Internet-based Systems (SITIS’19). 296–302.

[24]

H. Walsh, B. Saunders, and R. Bowden. 2022. Changing the representation: Examining language representation for neural sign language production. arXiv preprint arXiv:2210.06312.

[25]

A. Voskou, K. P. Panousis, D. Kosmopoulos, D. N. Metaxas, and S. Chatzis. 2021. Stochastic transformer networks with linear competing units: Application to end-to-end SL translation. In IEEE/CVF International Conference on Computer Vision. 11946–11955.

Cited By

Waghmare PDeshpande A(2024)Marathi Text to ISL Gloss Translation Using LSTM, GRU and Transformer Neural Models2024 International Conference on Intelligent Computing and Sustainable Innovations in Technology (IC-SIT)10.1109/IC-SIT63503.2024.10862857(1-5)Online publication date: 21-Nov-2024
https://doi.org/10.1109/IC-SIT63503.2024.10862857

Index Terms

Enhanced BERT-based Multi-Head Self-Attention Transformer for Transformation of Marathi Text to Marathi Sign Language Gloss

Recommendations

Marathi to Indian Sign Language Machine Translation
Machine translation has been a prominent field of research, contributing significantly to human life enhancement. Sign language machine translation, a subfield, focuses on translating spoken language content into sign language and vice versa, thereby ...
Sentence Boundary Detection For Marathi Language

Detecting the sentence boundary forms the basic step for many natural language applications. A lot of work has been done in this direction for English and other foreign languages. But not much work has been done for Indian languages. This paper proposes ...
Multi Task Learning Based Shallow Parsing for Indian Languages
Shallow Parsing is an important step for many Natural Language Processing tasks. Although shallow parsing has a rich history for resource rich languages, it is not the case for most Indian languages. Shallow Parsing consists of POS Tagging and Chunking. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Asian and Low-Resource Language Information Processing

ACM Transactions on Asian and Low-Resource Language Information Processing Volume 23, Issue 10

October 2024

189 pages

EISSN:2375-4702

DOI:10.1145/3613658

Editor:
Imed Zitouni
Google, USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2024

Online AM: 08 August 2024

Accepted: 18 July 2024

Revised: 29 April 2024

Received: 16 December 2023

Published in TALLIP Volume 23, Issue 10

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
171
Total Downloads

Downloads (Last 12 months)171
Downloads (Last 6 weeks)12

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Waghmare PDeshpande A(2024)Marathi Text to ISL Gloss Translation Using LSTM, GRU and Transformer Neural Models2024 International Conference on Intelligent Computing and Sustainable Innovations in Technology (IC-SIT)10.1109/IC-SIT63503.2024.10862857(1-5)Online publication date: 21-Nov-2024
https://doi.org/10.1109/IC-SIT63503.2024.10862857

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents