Skip to main content
SpringerLink
Log in

A Spatio-Temporal Framework for Dynamic Indian Sign Language Recognition

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

A sign language recognition system is a boon to the signer community as it eases the flow of information between the signer and non-signer communities. However, extracting timely detail from the video data is still a challenging task. In this paper, a deep learning based model consisting of trainable CNN and trainable stacked 2 bidirectional long short term memory (S2B-LSTM) has been proposed and tested to recognise the dynamic gestures of Indian sign language (ISL). The CNN architecture has been used as feature extractor to extract the spatial features from the input video data, whereas the temporal relation between the consecutive frames of input video is extracted using S2B-LSTM. This model has been trained and tested on self-developed dataset consisting of 360 videos of ISL dynamic gestures. The CNN-S2B-LSTM model outperforms the existing techniques of sign language recognition with best recognition accuracy of 97.6%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The authors declare that no data or material was taken illegally. However, a publically available dataset was taken for implementation. The dataset generated in the study is a part of ongoing research work; hence, copyrights are reserved to the institute. Upon completing the ongoing project, this dataset can be made available.

References

  1. AL-Rousan, M., Assaleh, K., & Tala'a, A. (2009).Video-based signer-independent Arabic sign language recognition using hidden Markov models. Appl. Soft Comput., 9(3), 990–999. https://doi.org/10.1016/j.asoc.2009.01.002.

  2. Wadhawan, A., & Kumar, P. (2021). Sign language recognition systems: a decade systematic literature review. Archives of Computational Methods in Engineering, 28(3), 785–813. https://doi.org/10.1007/s11831-019-09384-2

    Article  Google Scholar 

  3. Moreira Almeida, S. G., Guimarães, F. G.& Arturo Ramírez, J. (2014). Feature extraction in Brazilian Sign Language Recognition based on phonological structure and using RGB-D sensors. Expert Systems with Applications, 41(16), 7259–7271. https://doi.org/10.1016/j.eswa.2014.05.024.

  4. Sharma, S., & Singh, S. (2021). Recognition of indian sign language (ISL) using deep learning model. Wireless Personal Communications. https://doi.org/10.1007/s11277-021-09152-1

    Article  Google Scholar 

  5. Abhishek, K. S., Qubeley, L. C. F., & Ho, D. (2016). Glove-based hand gesture recognition sign language translator using capacitive touch sensor. In: 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC), Aug. 2016, pp. 334–337. https://doi.org/10.1109/EDSSC.2016.7785276.

  6. Rautaray, S. S., & Agrawal, A. (2015). Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43(1), 1–54. https://doi.org/10.1007/s10462-012-9356-9

    Article  Google Scholar 

  7. S. Sharma and S. Singh, 'Vision-based hand gesture recognition using deep learning for the interpretation of sign language', Expert Syst. Appl., vol. 182, p. 115657, Nov. 2021, doi: https://doi.org/10.1016/j.eswa.2021.115657.

  8. Rekha, J., Bhattacharya, J., & Majumder, S. (2011). Shape, texture and local movement hand gesture features for Indian Sign Language recognition. In: 3rd International Conference on Trendz in Information Sciences Computing (TISC2011), Dec. 2011, pp. 30–35. https://doi.org/10.1109/TISC.2011.6169079.

  9. Kishore, P. V. V., Prasad, M. V. D., Kumar, D. A., & Sastry, A. S. C. S. (2016). Optical flow hand tracking and active contour hand shape features for continuous sign language recognition with artificial neural networks. In: 2016 IEEE 6th International Conference on Advanced Computing (IACC), Feb. 2016, pp. 346–351. https://doi.org/10.1109/IACC.2016.71.

  10. Ahmed, W., Chanda, K., Mitra, S.( 2016). Vision based hand gesture recognition using dynamic time warping for indian sign language. In: 2016 International Conference on Information Science (ICIS), Aug. 2016, pp. 120–125. https://doi.org/10.1109/INFOSCI.2016.7845312.

  11. Naglot, D., & Kulkarni, M. (2016). ANN based Indian Sign Language numerals recognition using the leap motion controller. In: 2016 International Conference on Inventive Computation Technologies (ICICT), Aug. 2016, vol. 2, pp. 1–6. https://doi.org/10.1109/INVENTIVE.2016.7824830.

  12. Kumar, A., Thankachan, K., & Dominic, M. M. (Mar). Sign language recognition. In: 2016 3rd International Conference on Recent Advances in Information Technology (RAIT), Mar. 2016, pp. 422–428. https://doi.org/10.1109/RAIT.2016.7507939.

  13. Ibrahim, N. B., Selim, M. M., & Zayed, H. H. (2018). An Automatic arabic sign language recognition system (ArSLRS). Journal of King Saud University: Computer and Information Sciences, 30(4), 470–477, Oct. 2018. https://doi.org/10.1016/j.jksuci.2017.09.007.

  14. Kim, S. Y., Han, H. G., Kim, J. W., Lee, S., & Kim, T. W. (2017). A hand gesture recognition sensor using reflected impulses. IEEE Sensors Journal, 17(10), 2975–2976. https://doi.org/10.1109/JSEN.2017.2679220

    Article  Google Scholar 

  15. Athira, P. K., Sruthi, C. J., & Lijiya, A. (2019). A signer independent sign language recognition with co-articulation elimination from live videos: An indian scenario. Journal of King Saud University: Computer and Information Sciences. https://doi.org/10.1016/j.jksuci.2019.05.002.

  16. P. M. Ferreira, J. S. Cardoso, and A. Rebelo, 'Multimodal Learning for Sign Language Recognition', in Pattern Recognition and Image Analysis, Cham, 2017, pp. 313–321. doi: https://doi.org/10.1007/978-3-319-58838-4_35.

  17. Darwish, S. M. (2017). Man-machine interaction system for subject independent sign language recognition. In: Proceedings of the 9th International Conference on Computer and Automation Engineering—ICCAE '17, Sydney, Australia, pp. 121–125. https://doi.org/10.1145/3057039.3057040.

  18. Kumar, P., Roy, P. P., & Dogra, D. P. (2018). Independent Bayesian classifier combination based sign language recognition using facial expression. Information Sciences, 428, 30–48. https://doi.org/10.1016/j.ins.2017.10.046

    Article  MathSciNet  Google Scholar 

  19. Vo, D.-H., Huynh, H.-H., Doan, P.-M., & Meunier, J. (2017). Dynamic gesture classification for vietnamese sign language recognition. International Journal of Advanced Computer Science and Applications, 8(3). https://doi.org/10.14569/IJACSA.2017.080357.

  20. Liao, Y., Xiong, P., Min, W., Min, W., & Lu, J. (2019). Dynamic sign language recognition based on video sequence with BLSTM-3D residual networks. IEEE Access, 7, 38044–38054. https://doi.org/10.1109/ACCESS.2019.2904749

    Article  Google Scholar 

  21. Liu, Z., Chai, X., Liu, Z., & Chen, X. (2017). Continuous Gesture Recognition with Hand-Oriented Spatiotemporal Feature. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, Oct. 2017, pp. 3056–3064. https://doi.org/10.1109/ICCVW.2017.361.

  22. Hagemann, T., & Katsarou, K. (2020). A Systematic review on anomaly detection for cloud computing environments.In: 2020 3rd Artificial Intelligence and Cloud Computing Conference, New York, NY, USA, Dec. 2020, pp. 83–96. https://doi.org/10.1145/3442536.3442550.

  23. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  24. Funahashi, K., & Nakamura, Y. (1993). Approximation of dynamical systems by continuous time recurrent neural networks. Neural Networks, 6(6), 801–806.

    Article  Google Scholar 

  25. Luo, S., Rao, Y., Chen, J., Wang, H., & Wang, Z. (2020). Short-term load forecasting model of distribution transformer based on CNN and LSTM. IEEE International Conference on High Voltage Engineering and Application (ICHVE), 2020, 1–4.

    Google Scholar 

  26. Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, S. W. (2018). Action recognition in video sequences using deep bi-directional LSTM With CNN features. IEEE Access, 6, 1155–1166. https://doi.org/10.1109/ACCESS.2017.2778011

    Article  Google Scholar 

  27. Bhuyan, M. K., Ghosh, D., & Bora, P. K. (2005). Co-articulation Detection in Hand Gestures. In: TENCON 2005—2005 IEEE Region 10 Conference, Nov. 2005, pp. 1–4. https://doi.org/10.1109/TENCON.2005.300947.

Download references

Funding

Authors declare that no funding was received for this research work.

Author information

Authors and Affiliations

  1. ECE Department, Punjab Engineering College(Deemed to Be University), Chandigarh, India

    Sakshi Sharma & Sukhwinder Singh

Authors
  1. Sakshi Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sukhwinder Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sakshi Sharma.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, S., Singh, S. A Spatio-Temporal Framework for Dynamic Indian Sign Language Recognition. Wireless Pers Commun 132, 2527–2541 (2023). https://doi.org/10.1007/s11277-023-10730-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-023-10730-8

Keywords

Search

Navigation