Skip to main content
SpringerLink
Log in

An optimized automated recognition of infant sign language using enhanced convolution neural network and deep LSTM

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In the world, several sign languages (SL) are used, and BSL (Baby Sign Language) is the process of communication between the parents and baby using gestures. Communication by gestures is a non-verbal process that utilizes motion to pass on realities, expressions and feelings to people. SL is the communication mode in which the information is conveyed via movement of body parts like cheeks, eyebrows and head. Even though many research works based on SL are available, research in BSL remains a challenge. Hence, this paper presents an optimization-based automated recognition of the deep BSL system, which determines the gesture signalled by the kids. Initially, the image frames are extracted from the videos and data augmentation processes are performed. After pre-processing, the features are extracted from the frames using the Enhanced Convolution Neural Network (ECNN). The optimal characteristics are then selected by a new Life Choice Based Optimizer (LCBO). Finally, the classification is carried out by the Deep Long Short-Term Memory (DLSTM) scheme. The implementation is performed on the Python platform, and the performances are evaluated using several performance metrics such as accuracy, precision, kappa, f1-score and recall. The performance of the proposed approach (ECNN-DLSTM) is compared with several deep and machine learning approaches and obtains an accuracy of 99% and a kappa of 96%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

Data sharing does not apply to this article

References

  1. Albanie S, Varol G, Momeni L, Afouras T, Chung JS, Fox N and Zisserman A (2020) BSL-1K: scaling up co-articulated sign language recognition using mouthing cues. arXiv preprint arXiv:2007.12131

  2. Aly W, Aly S, Almotairi S (2019) User-independent American sign-language alphabet recognition based on depth image and PCANet features. IEEE Access 7:123138–123150

    Article  Google Scholar 

  3. Arora M, Mehta P, Mittal D, Bajaj P (2020) Word-level sign language gesture prediction under different conditions. In: international conference on innovative computing and communications. Springer, Singapore: pp. 427-435

  4. Asadi-Aghbolaghi M, Clapés A, Bellantonio M, Jair Escalante H, Ponce-López V, Baró X, Guyon I, Kasaei S, Escalera S (2017) Deep learning for action and gesture recognition in image sequences: a survey. Gesture Recog:539–578

  5. Bragg D, Koller O, Bellard M, Berke L, Boudreault P, Braffort A, Caselli N, Huenerfauth M, Kacorri H, Verhoef T and Vogler C (2019) Sign language recognition, generation, and translation: An Interdisciplinary Perspective Computers and Accessibility 16-31

  6. Cai W, Liu D, Ning X, Wang C, Xie G (2021) Voxel-based three-view hybrid parallel network for 3D object classification. Displays 69:102076

    Article  Google Scholar 

  7. Cui R, Liu H, Zhang C (2019) A deep neural framework for continuous sign language recognition by iterative training. IEEE Transac Multimed 21(7):1880–1891

    Article  Google Scholar 

  8. Deng X, Yang S, Zhang Y, Tan P, Chang L, Wang H (2017) Hand3D: Hand Pose Estimation using 3D Neural Network. arXiv:1704.02224

  9. Farooq U, Rahim MSM, Sabir N, Hussain A, Abid A (2021) Advances in machine translation for sign language: approaches, limitations, and challenges. Neural Comput Applic 33(21):14357–14399

    Article  Google Scholar 

  10. Ferreira P, Cardoso J, Rebelo A (2019) On the role of multi-modal learning in the recognition of sign language. Multimed Tools Appl 78:10035–10056

    Article  Google Scholar 

  11. Gao L, Li H, Liu Z, Liu Z, Wan L, Feng W (2021) RNN-transducer based Chinese sign language recognition. Neurocomputing 434:45–54

    Article  Google Scholar 

  12. Guo H, Wang G, Chen X (2017) Towards good practices for deep 3D hand pose estimation. arXiv:1707.07248

  13. Imran J, Raman B (2020) Deep motion templates and extreme learning machine for sign language recognition. Vis Comput 36(6):1233–1246

    Article  Google Scholar 

  14. Kamruzzaman MM (2020) Arabic sign language recognition and generating Arabic speech using convolutional neural network. Wirel Commun Mob Comput 2020:1–9

    Article  Google Scholar 

  15. Khatri A, Gaba A, Rana KPS and Kumar V (2019) A novel life choice-based optimizer. Soft computing 1-21

  16. Koller O, Zargaran S, Ney H, Bowden R (2018) Deep sign: enabling robust statistical continuous sign language recognition via hybrid CNN-HMMs. Int J Comput Vis 126(12):1311–1325

    Article  Google Scholar 

  17. Kowdiki M, Khaparde A (2021) Automatic hand gesture recognition using hybrid meta-heuristic-based feature selection and classification with dynamic time warping. Comput Sci Rev 39:100320

    Article  Google Scholar 

  18. Li D, Rodriguez C, Yu X, Li H (2020) Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. Appl Comput Vis:1459–1469

  19. Liao Y, Xiong P, Min W, Min W, Lu J (2019) Dynamic sign language recognition based on video sequence with BLSTM-3D residual networks. IEEE Access 7:38044–38054

    Article  Google Scholar 

  20. Lim K, Tan A, Lee C, Tan S (2019) Isolated sign language recognition using convolutional neural network hand modelling and hand energy image. Multimed Tools Appl 78:19917–19944

    Article  Google Scholar 

  21. Masood S, Srivastava A, Thuwal HC and Ahmad M (2018) Real-time sign language gesture (word) recognition from video sequences using CNN and RNN. In intelligent engineering informatics. Springer, Singapore 623-632

  22. Nadgeri S, Kumar A (2019, July) An image texture based approach in understanding and classifying baby sign language. In 2019 2nd international conference on intelligent computing, instrumentation and control technologies (ICICICT). IEEE 1:854–858

    Google Scholar 

  23. Nadgeri S, Kumar D (2020) An analytical study of signs used in baby sign language using Mobilenet framework. In proceedings of the international conference on recent advances in computational techniques (IC-RACT)

  24. Naranjo-Zeledón L, Peral J, Ferrández A, Chacón-Rivas M (2019) A systematic mapping of translation-enabling technologies for sign languages. Electronics 8(9):1047

    Article  Google Scholar 

  25. Neiva DH, Zanchettin C (2018) Gesture recognition: a review focusing on sign language in a mobile context. Expert Syst Appl 103:159–183

    Article  Google Scholar 

  26. Ning X, Gong K, Li W, Zhang L (2021) JWSAA: joint weak saliency and attention aware for person re-identification. Neurocomputing 453:801–811

    Article  Google Scholar 

  27. Prietch SS, Pineda IO, Paim PDS, Calleros JMG, García JG, Resmin R (2019) Discussion on image processing for sign language recognition: an overview of the problem complexity. Res Develop Technol:112–127

  28. Rao GA, Kishore PVV (2018) Selfie video based continuous Indian sign language recognition system. Ain Shams Eng J 9(4):1929–1939

    Article  Google Scholar 

  29. Rastgoo R, Kiani K, Escalera S (2018) Multi-modal deep hand sign language recognition in still images using restricted Boltzmann machine. Entropy

  30. Rastgoo R, Kiani K, Escalera S (2020) Sign language recognition: a deep survey. Expert systems with Applications113794

  31. Saunders B, Camgoz NC, Bowden R (2021) Continuous 3d multi-channel sign language production via progressive transformers and mixture density networks. Int J Comput Vis 129(7):2113–2135

    Article  Google Scholar 

  32. Sullivan AL, Thayer AJ, Farnsworth EM, Susman-Stillman A (2019) Effects of child care subsidy on school readiness of young children with or at-risk for special needs. Early Child Res Q 47:496–506

    Article  Google Scholar 

  33. Wadhawan A, Kumar P (2020) Deep learning-based sign language recognition system for static signs. Neural computing and applications 1–12. https://doi.org/10.1007/s00521-019-04691-y

  34. Wang C, Wang X, Zhang J, Zhang L, Bai X, Ning X, Zhou J, Hancock E (2022) Uncertainty estimation for stereo matching based on evidential deep learning. Pattern Recogn 124:108498

    Article  Google Scholar 

  35. Wangchuk K, Riyamongkol P, Waranusast R (2021) Real-time bhutanese sign language digits recognition system using convolutional neural network. ICT Express 7(2):215–220

    Article  Google Scholar 

  36. Wei S, Chen X, Yang X, Cao S, Zhang X (2016) A component-based vocabulary-extensible sign language gesture recognition framework. Sensors 16(4):556

    Article  Google Scholar 

  37. Wu F, Jing XY, Dong X, Hu R, Yue D, Wang L, Ji YM, Wang R, Chen G (2018) Intraspectrum discrimination and interspectrum correlation analysis deep network for multispectral face recognition. IEEE Transac Cyber 50(3):1009–1022

    Article  Google Scholar 

  38. Yang S, Zhu Q (2017) Continuous Chinese sign language recognition with CNN-LSTM”, Proc. SPIE 10420, Digital Image Processing (ICDIP 2017), 104200F. https://doi.org/10.1117/12.2281671

  39. Zheng L, Liang B, Jiang A (2017) Recent Advances of Deep Learning for Sign Language Recognition. 2017 International conference on digital image computing: techniques and applications (DICTA), Sydney, NSW, Australia

Download references

Funding

No funding is provided for the preparation of the manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, (D), Andhra Pradesh, Guntur, 522502, India

    Vamsidhar Enireddy

  2. Department of Computer Science and Engineering, Malla Reddy Engineering College (Autonomous), Hyderabad, Telangana, 500100, India

    J. Anitha

  3. Miracle Educational Society Group of Institutions, Miracle City, Andhra Pradesh, 535216, India

    N. Mahendra

  4. Department of CSE, RISE Krishna Sai Prakasam Group of Institutions, Ongole, Andhra Pradesh, 523272, India

    G. Kishore

Authors
  1. Vamsidhar Enireddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Anitha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. N. Mahendra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. G. Kishore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vamsidhar Enireddy.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Enireddy, V., Anitha, J., Mahendra, N. et al. An optimized automated recognition of infant sign language using enhanced convolution neural network and deep LSTM. Multimed Tools Appl 82, 28043–28065 (2023). https://doi.org/10.1007/s11042-023-14428-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14428-8

Keywords

Search

Navigation