skip to main content
10.1145/3378904.3378915acmotherconferencesArticle/Chapter ViewAbstractPublication PagesbdetConference Proceedingsconference-collections
research-article

2-way Arabic Sign Language Translator using CNNLSTM Architecture and NLP

Published: 09 April 2020 Publication History

Abstract

Over 466 million (5%) people across the world are suffering from hearing impairment, according to the World Health Organization. There is a great need to bridge the communication gap between the deaf and the general population. In our research work, recent developments such as Natural Language Processing (NLP) and Deep Learning Neural Network (DLNN) are utilized to bridge this gap. We developed a 2-way sign language translator for the Arabic language, which translates text to sign and vice versa. The NLP such as parsing, part of speech tagging, tokenization and translation are developed to achieve text to sign translation. The Convolutional Neural Network (CNN) along with Long Short-Term Memory (LSTM) is used to perform sign to text translation.

References

[1]
United Nations. "Sign Language, Deaf, Advocacy, Human Rights, Disability." United Nations, https://www.un.org/en/events/signlanguagesday.
[2]
Sarji, David K. 2008. "HandTalk: Assistive Technology for the Deaf." Computer, vol. 41, no. 7, pp. 84--86.
[3]
P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz. 2016. Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural network. In Proc. CVPR.
[4]
N. Neverova, C. Wolf, G. Taylor, and F. Nebout. 2014. Multi-scale deep learning for gesture detection and localization. In ECCV Workshops
[5]
D. Wu, L. Pigou, P.-J. Kindermans, N. Le, L. Shao, J. Dambre, and J.-M. Odobez. 2016. Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(8):1583--1597
[6]
Camgoz, Necati Cihan, et al. 2017. "SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition." 2017 IEEE International Conference on Computer Vision (ICCV).
[7]
A.E.E.El Alfi, M.M.R.El Basuony, S.M.El Atawy. 2014. Intelligent Arabic text to Arabic sign language translation for easy deaf communication Int J Comput Appl, 92, pp. 22--29
[8]
S.M. Halawani, A.B. Zaitun. 2012. An avatar based translation system from Arabic speech to Arabic sign language for deaf people Int J Inf Sci Educ ISSN, 2, pp. 13--20 ISSN 2231-1262
[9]
H. Al-Khalifa. 2010. Introducing Arabic sign language for mobile phones Comput Help People Spec Needs, 6180, pp. 213--220, Springer Berlin Heidelberg
[10]
El-Gayyar, Mahmoud M., et al. 2016. "Translation from Arabic Speech to Arabic Sign Language Based on Cloud Computing." Egyptian Informatics Journal, vol. 17, no. 3, pp. 295--303.
[11]
Stamp R, Schembri A, Fenlon J, Rentelis R, Woll B, Cormier K. 2014. Lexical Variation and Change in British Sign Language. PLoS ONE 9(4): e94053. https://doi.org/10.1371/journal.pone.0094053
[12]
Stamp, Rose, et al. 2014. "Lexical Variation and Change in British Sign Language." PLoS ONE, vol. 9, no. 4.
[13]
Minisi MN. 2015. Arabic sign language dictionary; http://www.menasy.com/
[14]
@esl_zayed. 2016. "Emirati Sign Language (ESL)", Instagram Photos and Videos." Instagram, https://www.instagram.com/esl_zayed/?hl=en
[15]
"The Stanford NLP Group." The Stanford Natural Language Processing Group, https://nlp.stanford.edu/projects/arabic.shtml.
[16]
Mohataher. 2017. "Mohataher/Arabic-Stop-Words." GitHub, 22 Jan. https://github.com/mohataher/arabic-stop-words.
[17]
Oudalab. "Oudalab/Arabic-NER." GitHub, https://github.com/oudalab/Arabic-NER.
[18]
Tsironi, Eleni, et al. 2017. "An Analysis of Convolutional Long Short-Term Memory Recurrent Neural Networks for Gesture Recognition." Neurocomputing, vol. 268, pp. 76--86.
[19]
Lin, Tsung-Yi & Maire, Michael & Belongie, Serge & Hays, James & Perona, Pietro & Ramanan, Deva & Dollár, Piotr & Lawrence Zitnick, 2014. C. Microsoft COCO: Common Objects in Context. 8693. 10.1007/978-3-319-10602-1_48.
[20]
Lisa Yuan. "Hearing Loss Facts and Demographics." HLAA, http://hlaa-la.org/better-hearing/hearing-loss-statistics-and-demographics/.
[21]
Joanne Cripps. 2017. "What Is Deaf Culture?" DEAF CULTURE CENTRE, 27 Dec., https:// deafculturecentre.ca /what-is-deaf-culture/.

Cited By

View all
  • (2024)An Efficient Bidirectional Android Translation Prototype for Yemeni Sign Language Using Fuzzy logic and CNN Transfer Learning ModelsIEEE Access10.1109/ACCESS.2024.3512455(1-1)Online publication date: 2024
  • (2024)Motion Images With Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent ModesIEEE Access10.1109/ACCESS.2024.348513112(160728-160740)Online publication date: 2024
  • (2024)Adopting machine translation in the healthcare sectorComputer Speech and Language10.1016/j.csl.2023.10158284:COnline publication date: 4-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
BDET '20: Proceedings of the 2020 2nd International Conference on Big Data Engineering and Technology
January 2020
126 pages
ISBN:9781450376839
DOI:10.1145/3378904
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

  • Natl University of Singapore: National University of Singapore
  • Southwest Jiaotong University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 April 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Deep Learning
  2. Gesture recognition
  3. Language translation
  4. Machine Translation
  5. Natural Language Processing

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

BDET 2020

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)19
  • Downloads (Last 6 weeks)1
Reflects downloads up to 14 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)An Efficient Bidirectional Android Translation Prototype for Yemeni Sign Language Using Fuzzy logic and CNN Transfer Learning ModelsIEEE Access10.1109/ACCESS.2024.3512455(1-1)Online publication date: 2024
  • (2024)Motion Images With Positioning Information and Deep Learning for Continuous Arabic Sign Language Recognition in Signer Dependent and Independent ModesIEEE Access10.1109/ACCESS.2024.348513112(160728-160740)Online publication date: 2024
  • (2024)Adopting machine translation in the healthcare sectorComputer Speech and Language10.1016/j.csl.2023.10158284:COnline publication date: 4-Mar-2024
  • (2023)Robot Assist Sign Language Recognition for Hearing Impaired Persons Using Deep LearningVAWKUM Transactions on Computer Sciences10.21015/vtcs.v11i1.149111:1(245-267)Online publication date: 19-Jun-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media