Skip to main content
Springer Nature Link
Log in

A multi-stack RNN-based neural machine translation model for English to Pakistan sign language translation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Sign languages are gesture-based languages used by the deaf community of the world. Every country has a different sign language and there are more than 200 sign languages in the world. American Sign Language (ASL), British Sign Language (BSL), and German Sign Language (DGS) are well-studied sign languages. Due to their different grammatical structure and word order the deaf people feel difficulty in reading and understanding the written text in natural languages. In order to enhance the cognitive ability of the deaf subjects some translation models have been developed for these languages that translate natural language text into corresponding sign language gestures. Most of the earlier natural to sign language translation models rely on rule-based approaches. Recently, some neural machine translation models have been proposed for ASL, BSL, DGS, and Arabic sign language. However, most of these models have low accuracy scores. This research provides an improved and novel multi-stack RNN-based neural machine translation model for natural to sign language translation. The proposed model is based on encoder–decoder architecture and incorporates attention mechanism and embeddings to improve the quality of results. Rigorous experimentation has been performed to compare the proposed multi-stack RNN-based model with baseline models. The experiments have been conducted using a sizeable translation corpus comprising of nearly 50,000 sentences for Pakistan Sign Language (PSL). The performance of the proposed neural machine translation model for PSL has been evaluated with the help of well-established evaluation measures including Bilingual Evaluation Understudy Score (BLEU), and Word Error Rate (WER). The results show that multi-stacked gated recurrent unit-based RNN model that employs Bahdanau attention mechanism and GloVe embedding performed the best showing the BLEU score of 0.83 and WER 0.17, which outperform the existing translation models. The proposed model has been exposed through a software system that converts the translated sentences into PSL gestures using an avatar. The evaluation of the usability has also been performed to see how effectively the avatar-based output helps compensating the cognitive hearing deficit for the deaf people. The results show that it works well for different granularity levels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Not applicable.

References

  1. Sofiane B, Antonio N, Elodie T, Filippo M, Giovanni P, David C, Mohamed C (2014) Interactive technologies for autistic children: a review. Cognit Comput 6(4):722

    Article  Google Scholar 

  2. Luqman H, Mahmoud SA (2017) Transform-based Arabic sign language recognition. Procedia Comput Sci 117:2–9

    Article  Google Scholar 

  3. Khan NS et al (2015) Speak Pakistan: challenges in developing Pakistan sign language using information technology. South Asian Stud 30(2):367

    Google Scholar 

  4. Farooq U, Rahim MSM, Sabir N, Hussain A, Abid A (2021) Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Comput Appl 33(21):14357–14399

    Article  Google Scholar 

  5. Kahlon NK, Singh W (2021) Machine translation from text to sign language: a systematic review. Univ Access Inf Soc 1–35

  6. Othman A, Jemni M (2011) Statistical sign language machine translation: from English written text to American sign language gloss. arXiv:1112.0168

  7. Cate H, Hussain Z (2017) Bidirectional American sign language to English translation. arXiv:1701.02795

  8. Manzano DM (2018) English to ASL Translator for Speech2Signs preprint arXiv:1112.0168

  9. Mocialov B, Turner G, Hastie H (2020) Transfer learning for British sign language modelling. arXiv:2006.02144

  10. Stoll S, Camgoz NC, Hadfield S, Bowden R (2020) Text2Sign: towards sign language production using neural machine translation and generative adversarial networks. Int J Comput Vis 128:891–908

    Article  Google Scholar 

  11. Saunders B, et al. (2020) Everybody sign now: translating spoken language to photo realistic sign language video

  12. Brour M, Benabbou A (2019) Atlaslang NMT: arabic text language into Arabic sign language neural machine translation. J King Saud Univ-Comput Inf Sci. Bonham, M. E. (2015). English to ASL Gloss Machine Translation

  13. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

  14. Khan NS, Abid A, Abid K (2020) A novel natural language processing (NLP)–based machine translation model for English to Pakistan sign language translation. Cogn Comput 12:748–765

    Article  Google Scholar 

  15. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078

  16. Chen MX, Firat O, Bapna A, Johnson M, Macherey W, Foster G, Hughes M (2018) The best of both worlds: combining recent advances in neural machine translation. arXiv:1804.09849

  17. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Proceedings of advances neural information processing systems, pp 3104–3112

  18. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555

  19. Bahdanau DC (2014) Neural machine translation by jointly learning to align and translate. arXiv:1409.0473

  20. Luong MT, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025

  21. Farooq U, Rahim MSM, Khan NS, Rasheed S, Abid A (2021) A crowdsourcing-based framework for the development and validation of machine readable parallel corpus for sign languages. IEEE Access 9:91788–91806

    Article  Google Scholar 

  22. Bragg D, Koller O, Bellard M, Berke L, Boudreault P, Braffort A, Vogler C (2019) Sign language recognition, generation, and translation: an interdisciplinary perspective. In: The 21st international ACM SIGACCESS conference on computers and accessibility, pp 16–31

  23. Mohandes M, Deriche M, Liu J (2014) Image-based and sensor-based approaches to Arabic sign language recognition. IEEE Trans Hum-Mach Syst 44(4):551–557

    Article  Google Scholar 

  24. Khan N, Shahzada A, Ata S, Abid A, Khan Y, Shoaib Farooq M (2014) A vision based approach for Pakistan sign language alphabets recognition. Pensee 76(3)

  25. Farooq U, Asmat A, Rahim MSBM, Khan NS, Abid A (2019) A comparison of hardware based approaches for sign language gesture recognition systems. In: 2019 international conference on innovative computing (ICIC). IEEE, pp 1–6

  26. Peisong W, Qiang S, Hua H, Jian C (2016) Sequentially supervised long short-term memory for gesture recognition. Cognit Comput 8(5):982

    Article  Google Scholar 

  27. Bernd JK, Peter B, Jim K, Emily K, Irene M (2011) Movements and holds in fluent sentence production of American sign language: the action-based approach. Cognit Comput 3(3):449

    Article  Google Scholar 

  28. Galván-Ruiz J, Travieso-González CM, Tejera-Fettmilch A, Pinan-Roescher A, Esteban-Hernández L, Domínguez-Quintana L (2020) Perspective and evolution of gesture recognition for sign language: a review. Sensors 20(12):3571

    Article  Google Scholar 

  29. McCowan IM-P (2004) On the use of information retrieval measures for speech recognition evaluation

  30. Othman A, Tmar Z (2012) English-ASL gloss parallel corpus 2012: ASLG-PC12. In: 5th workshop on the representation and processing of sign languages {Data Set ArSL]

  31. Ferreira MAM, García LS, Bueno J, Felipe TA (2019) Requirements for a framework of a virtual learning environment for deaf people mediated by avatar. In: International conference on human-computer interaction. Springer, Cham, pp 255–262

  32. Huenerfauth M (2009) A linguistically motivated model for speed and pausing in animations of American sign language. ACM Trans Access Comput (TACCESS) 2(2):1–31

    Article  Google Scholar 

  33. Hutchinson J (2012) Literature review: analysis of sign language notations for parsing in machine translation of SASL. Rhodes University, Grahamstown

    Google Scholar 

  34. Stokoe WC Jr (2005) Sign language structure: an outline of the visual communication systems of the American deaf. J Deaf Stud Deaf Educ 10(1):3–37

    Article  Google Scholar 

  35. Gonàlez M, Giménez J, Màrquez L (2012) A graphical interface for MT evaluation and error analysis. In: The 50th annual meeting of the association for computational linguistics

  36. Nießen S, Och FJ, Leusch G, Ney H, Informatik LF (2000) An evaluation tool for machine translation: fast evaluation for MT research. In: Proceedings of the 2nd international conference on language resources and evaluation (LREC-2000)

  37. Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of the 7th conference of the association for machine translation in the Americas: technical papers, pp 223–231

  38. Cox S, Lincoln M, Tryggvason J, Nakisa M, Wells M, Tutt M, Abbott S (2003) The development and evaluation of a speech-to-sign translation system to assist transactions. Int J Hum Comput Interact 16(2):141–161

    Article  Google Scholar 

  39. Veale T, Conway A, Collins B (1998) The challenges of cross-modal translation: English-to-sign-language translation in the zardoz sys- tem. Mach Transl 13(1):81–106

    Article  Google Scholar 

  40. d’Armond LS (2002) Representation of American sign language for machine translation, Ph.D. thesis, Georgetown University

  41. Marshall I, Sáfár É (2001) Extraction of semantic representations from syntactic SMU link grammar linkages. In: Proceedings of recent advances in natural language processing, pp 154–159

  42. Van Zijl L, Andries C (2006) The South African sign language machine translation project: issues on non-manual sign generation. In: Proceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries. South African Institute for Computer Scientists and Information Technologists (2006)

  43. Shoaib U et al (2014) Integrating multiwordnet with Italian sign language lexical resources. Expert Syst Appl 41(5):2300–2308

    Article  Google Scholar 

  44. Bungeroth J, Ney H (2004) Statistical sign language translation. In: Workshop on representation and processing of sign languages, LREC, Vol 4, pp 105–108

  45. Zhao L et al. (2000) A machine translation system from English to American sign language. In: Conference of the association for machine translation in the Americas. Springer, Berlin, Heidelberg

  46. Marshall I, Éva S (2003) A prototype text to British Sign Language (BSL) translation system. In: Proceedings of the 41st annual meeting on association for computational linguistics-volume 2. Association for Computational Linguistics

  47. Shaalan K (2010) Rule-based approach in Arabic natural language processing. Int J Inf Commun Technol (IJICT) 3(3):11–19

    Google Scholar 

  48. San Segundo Hernández R, Lopez Ludeña V, Martin Maganto R, Sánchez D, García A (2010) Language resources for Spanish–Spanish sign language (LSE) translation

  49. Bonham ME (2015) English to ASL gloss machine translation. M. Art thesis, Brigham Young University

  50. Othman A, Jemni M (2019) Designing high accuracy statistical machine translation for sign language using parallel corpus: case study English and American sign language. J Inf Technol Res (JITR) 12(2):134–158

    Article  Google Scholar 

  51. Luqman H, Sabri AM (2018) Automatic translation of Arabic text-to-Arabic sign language. Univ Access Inf Soc 18:939–951

    Article  Google Scholar 

  52. Hadla LS, Hailat TM, Al-Kabi MN (2014) Evaluating Arabic to English machine translation. Int J Adv Comput Sci Appl (IJACSA) 5(11):68–73

    Google Scholar 

  53. Filhol M, Hadjadj MN, Testu B (2016) A rule triggering system for automatic text-to-sign translation. Univ Access Inf Soc 15(4):487–498

    Article  Google Scholar 

  54. Kouremenos D, Ntalianis K, Kollias S (2018) A novel rule based machine translation scheme from Greek to Greek Sign Language: production of different types of large corpora and Language Models evaluation. Comput Speech Lang 51:110–135

    Article  Google Scholar 

  55. Porta J, López-Colino F, Tejedor J, Colás J (2014) A rule-based translation from written Spanish to Spanish Sign Language glosses. Comput Speech Lang 28(3):788–811

    Article  Google Scholar 

  56. San-Segundo R, Montero JM, Macías-Guarasa J, Córdoba R, Ferreiros J, Pardo JM (2008) Proposing a speech to gesture translation architecture for Spanish deaf people. J Vis Lang Comput 19(5):523–538

    Article  Google Scholar 

  57. Othman A, Jemni M (2011) Statistical sign language machine translation: from English written text to American sign language gloss. Int J Comput Sci Issues 8(5):65–73

    Google Scholar 

  58. Nguyen TBD, Phung TN, Vu TT (2018) A rule-based method for text shortening in Vietnamese sign language translation. in information systems design and intelligent applications. Springer, Singapore, pp 655–662

  59. Verma VK, Srivastava S (2018) Toward machine translation linguistic issues of Indian sign language. In: Speech and language processing for human-machine communications. Springer, Singapore, pp 129–135

  60. Yasir F, Prasad PWC, Alsadoon A, Elchouemi A, Sreedharan S (2017) Bangla sign language recognition using convolutional neural network. In: 2017 international conference on intelligent computing, instrumentation and control technologies (ICICICT), pp 49–53. IEEE

  61. Tumsri J, Kimpan W (2017) Thai sign language translation using leap motion controller. In: Proceedings of the international multi conference of engineers and computer scientists, pp 46–51

  62. Camgoz NC, Hadfield S, Koller O, Ney H, Bowden R (2018) Neural sign language translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7784–7793

  63. Brour M, Benabbou A (2019) ATLASLang MTS 1: Arabic text language into Arabic sign language machine translation system. Procedia Comput Sci 148:236–245

  64. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  65. Zeshan U (2000) Sign language in Indo-Pakistan: A description of a signed language. John Benjamins Publishing, Amsterdam

    Book  Google Scholar 

  66. Zeshan U (2003) Indo-Pakistani Sign Language grammar: a typological outline. Sign Lang Stud 157–212

  67. Abbas A, Sarfraz S (2018) Developing a prototype to translate text and speech to Pakistan sign language with bilingual subtitles: a framework. J Educ Technol Syst 47:248–266

    Article  Google Scholar 

  68. Abid K, Khan NS, Farooq U, Farooq MS, Naeem MA, Abid A (2018) A roadmap to elevate Pakistan sign language among regional sign languages. South Asian Studies (1026–678X) 33(2)

  69. Papineni K, Roukos S, Ward T, Zhu W-J (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguistics, pp 311–318

  70. Sun J, Zhang H, Xu S, Liu Y (2021) Full information control for switched neural networks subject to fault and disturbance. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3100143

    Article  Google Scholar 

Download references

Acknowledgements

We are thankful to the sign language experts and students who helped in this study.

Funding

There is no funding available for this research.

Author information

Authors and Affiliations

  1. Faculty of Computing, Universiti Teknologi Malaysia, Johor Bahru, Malaysia

    Uzma Farooq & Mohd Shafry Mohd Rahim

  2. University of Management and Technology, Lahore, Pakistan

    Uzma Farooq & Adnan Abid

  3. Faculty of Computer Science and Information Technology, Virtual University of Pakistan, Lahore, 54000, Pakistan

    Adnan Abid

Authors
  1. Uzma Farooq
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mohd Shafry Mohd Rahim
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Adnan Abid
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Adnan Abid.

Ethics declarations

Conflict of interest

The authors do not have any conflict of interest or any competing interests.

Code availability

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Farooq, U., Mohd Rahim, M.S. & Abid, A. A multi-stack RNN-based neural machine translation model for English to Pakistan sign language translation. Neural Comput & Applic 35, 13225–13238 (2023). https://doi.org/10.1007/s00521-023-08424-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08424-0

Keywords