Skip to main content
Springer Nature Link
Log in

Tsi-cnn-net: truly shift-invariant convolutional neural network for Indian sign language recognition system

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

The majority of Indian sign language (ISL) recognition systems applied convolutional neural network (CNN) based deep neural networks. However, the output of CNN image classifiers may vary significantly with a little shift in input images. This shortcoming can be partially addressed by data augmentation, anti-aliasing, or blurring that do not work with different input patterns the network trained on and non-linear activation functions like ReLU, respectively. To deal with this short-coming, an ISL recognition approach has been presented using truly shift-invariant CNN. A sub-sampling strategy i.e. adaptive polyphase sampling (APS) has been applied to allow CNN truly shift-invariant. The proposed system is completely consistent to classification task. Furthermore, it offers significantly outstanding classification accuracy not only on Indian sign language datasets but also on datasets of other sign languages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Data (Two Indian Sign Language datasets) is available. VUCS_ISL_I - (https://github.com/UtpalNandi/VUCS_ISL_I) VUCS_ISL_II - (https://github.com/UtpalNandi/VUCS_ISL_II)

References

  1. Sandler W, Lillo-Martin D (2006) Sign language and linguistic universals. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  2. Puranik V, Gawande VS, Gujarathi JV, Patani A, Rane T (2022) Video-based sign language recognition using recurrent neural networks. 2022 2nd Asian Conference on Innovation in Technology (ASIANCON), 1–6

  3. Badhe PC, Kulkarni V (2020) Artificial neural network based indian sign language recognition using hand crafted features. In: 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6. https://doi.org/10.1109/ICCCNT49239.2020.9225294

  4. Gao L, Li H, Liu Z, Liu Z, Wan L, Feng W (2021) Rnn-transducer based Chinese sign language recognition. Neurocomputing 434:45–54. https://doi.org/10.1016/j.neucom.2020.12.006

    Article  MATH  Google Scholar 

  5. Ibrahim NB, Selim MM, Zayed HH (2018) An automatic Arabic sign language recognition system (ARSLRS). J King Saud Univ Comput Inform Sci 30(4):470–477. https://doi.org/10.1016/j.jksuci.2017.09.007

    Article  MATH  Google Scholar 

  6. Varsha M, Nair CS (2021) Indian sign language gesture recognition using deep convolutional neural network. In: 2021 8th International Conference on Smart Computing and Communications (ICSCC), pp. 193–197. https://doi.org/10.1109/ICSCC51209.2021.9528246

  7. Singh R, Jangid M (2021) Indian sign language recognition using color space model and thresholding. In: 2021 Asian Conference on Innovation in Technology (ASIANCON), pp. 1–4. https://doi.org/10.1109/ASIANCON51346.2021.9544615

  8. Likhar P, N, R.G (2021) Indian sign language translation using deep learning. In: 2021 IEEE 9th Region 10 Humanitarian Technology Conference (R10-HTC), pp. 1–4. https://doi.org/10.1109/R10-HTC53172.2021.9641599

  9. Sharma S, Singh S (2023) Isl recognition system using integrated mobile-net and transfer learning method. Exp Syst Appl 221:119772. https://doi.org/10.1016/j.eswa.2023.119772

    Article  MATH  Google Scholar 

  10. Matlani R, Dadlani R, Dumbre S, Mishra S, Tewari MA (2022) Real-time sign language recognition using machine learning and neural network. In: 2022 International Conference on Electronics and Renewable Systems (ICEARS), pp. 1381–1386. https://doi.org/10.1109/ICEARS53579.2022.9752213

  11. Samek W, Montavon G, Lapuschkin S, Anders CJ, Müller K-R (2021) Explaining deep neural networks and beyond: a review of methods and applications. Proc IEEE 109(3):247–278. https://doi.org/10.1109/JPROC.2021.3060483

    Article  MATH  Google Scholar 

  12. Li J, Chen J, Sheng B, Li P, Yang P, Feng DD, Qi J (2022) Automatic detection and classification system of domestic waste via multimodel cascaded convolutional neural network. IEEE Trans Ind Inform 18(1):163–173. https://doi.org/10.1109/TII.2021.3085669

    Article  MATH  Google Scholar 

  13. Ghorai A, Nandi U, Changdar C, Si T, Singh MM, Mondal JK (2023) Indian sign language recognition system using network deconvolution and spatial transformer network. Neural Computi Appl 35(28):20889–20907

    Article  Google Scholar 

  14. Chaman A, Dokmanic I (2021) Truly shift-invariant convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3773–3783

  15. Ikram S, Dhanda N (2021) American sign language recognition using convolutional neural network. In: 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON), pp. 1–12. https://doi.org/10.1109/GUCON50781.2021.9573782

  16. Lee CKM, Ng KKH, Chen C-H, Lau HCW, Chung SY, Tsoi T (2021) American sign language recognition and training method with recurrent neural network. Exp Syst Appl 167:114403. https://doi.org/10.1016/j.eswa.2020.114403

    Article  MATH  Google Scholar 

  17. Sanil A, Lal HS, Krishnan R, M, S., R, S., A, A (2022) Smart american sign language recognition for deaf. In: 2022 International Conference on Innovations in Science and Technology for Sustainable Development (ICISTSD), pp. 209–214. https://doi.org/10.1109/ICISTSD55159.2022.10010566

  18. Aggarwal D, Ahirwar S, Srivastava S, Verma S, Goel Y (2023) Sign language prediction using machine learning techniques: A review. In: 2023 Second International Conference on Electronics and Renewable Systems (ICEARS), pp. 1296–1300. https://doi.org/10.1109/ICEARS56392.2023.10084924

  19. Deshpande A, Shriwas A, Deshmukh V, Kale S (2023) Sign language recognition system using cnn. In: 2023 International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE), pp. 906–911. https://doi.org/10.1109/IITCEE57236.2023.10091051

  20. Li L, Liu D, Shen C, Sun J (2022) American sign language recognition based on machine learning and neural network. In: 2022 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), pp. 452–457. https://doi.org/10.1109/MLISE57402.2022.00096

  21. Sood K, Navdiya B, Hernandez A (2023) American sign language interpreter: A bridge between the two worlds. In: 2023 International Conference on Artificial Intelligence and Smart Communication (AISC), pp. 42–47. https://doi.org/10.1109/AISC56616.2023.10085066

  22. J, M, Krishna BV, S, S.N., K, S (2022) Sign language recognition using machine learning. In: 2022 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES), pp. 1–5. https://doi.org/10.1109/ICSES55317.2022.9914155

  23. Senanayaka SAMAS, Perera RADBS, Rankothge W, Usgalhewa SS, Hettihewa HD, Abeygunawardhana PKW (2022) Continuous american sign language recognition using computer vision and deep learning technologies. In: 2022 IEEE Region 10 Symposium (TENSYMP), pp. 1–6. https://doi.org/10.1109/TENSYMP54529.2022.9864539

  24. Chandrasekhar A, Singh L, Tripathi S, Malik N (2023) Sign language recognition system using deep learning. In: 2023 13th International Conference on Cloud Computing, Data Science & Engineering (Confluence), pp. 67–71. https://doi.org/10.1109/Confluence56041.2023.10048804

  25. Jamwal A, Vasukidevi G, Malleswari TN, Vijayakumar T, Reddy LCS, Gupta ASALGG (2022) Real time conversion of american sign language to text with emotion using machine learning. In: 2022 Sixth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), pp. 603–609. https://doi.org/10.1109/I-SMAC55078.2022.9987362

  26. Al-Barham M, Jamal A, Al-Yaman M (2023) Design of Arabic Sign Language Recognition Model

  27. Aldhahri E, Aljuhani R, Alfaidi A, Alshehri B, Alwadei H, Aljojo N, Alshutayri A, Almazroi A (2023) Arabic sign language recognition using convolutional neural network and mobilenet. Arabian J Sci Eng 48(2):2147–2154. https://doi.org/10.1007/s13369-022-07144-2

    Article  Google Scholar 

  28. Boukdir A, Benaddy M, Ellahyani A, Meslouhi O, Kardouchi M (2021) Isolated video-based Arabic sign language recognition using convolutional and recursive neural networks. Arabian J Sci Eng. https://doi.org/10.1007/s13369-021-06167-5

    Article  Google Scholar 

  29. Dabwan B, Jadhav M (2023) A CNN-LSTM model for Arabic sign language recognition. Atlantis Press, Amsterdam. https://doi.org/10.2991/978-94-6463-196-8_35

    Book  MATH  Google Scholar 

  30. Yuan S, Wang Y, Wang X, Deng H, Sun S, Wang H, Huang P, Li G (2020) Chinese sign language alphabet recognition based on random forest algorithm. In: 2020 IEEE International Workshop on Metrology for Industry 4.0 & IoT, pp. 340–344. https://doi.org/10.1109/MetroInd4.0IoT48571.2020.9138285

  31. Cheng S, Huang C, Wang Z, Wang J, Zeng Z, Wang F, Ding Q (2021) Real-time vision-based chinese sign language recognition with pose estimation and attention network. In: 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 1210–1215. https://doi.org/10.1109/ROBIO54168.2021.9739638

  32. Xu X, Meng K, Chen C, Lu L (2023) Isolated word sign language recognition based on improved SKRESNET-TCN network. J Sens 2023:9503961. https://doi.org/10.1155/2023/9503961

    Article  Google Scholar 

  33. Cheng J, Wei F, Liu Y, Li C, Chen Q, Chen X (2020) Chinese sign language recognition based on DTW-distance-mapping features. Math Prob Eng 2020:8953670. https://doi.org/10.1155/2020/8953670

    Article  MATH  Google Scholar 

  34. Rodríguez-Moreno I, Martínez-Otzeta JM, Sierra B (2023) A hierarchical approach for spanish sign language recognition: From weak classification to robust recognition system. In: Arai K (ed) Intelligent Systems and Applications. Springer, Cham, pp 37–53

  35. Martinez-Martin E, Morillas-Espejo F (2021) Deep learning techniques for Spanish sign language interpretation. Comput Intel Neurosci 2021:5532580. https://doi.org/10.1155/2021/5532580

    Article  Google Scholar 

  36. Saldaña G, Cerezo J, Bustillo M, Pérez A (2018) Recognition and classification of sign language for Spanish. Computación y Sistemas. https://doi.org/10.13053/cys-22-1-2780

    Article  MATH  Google Scholar 

  37. Chiruzzo L, McGill E, Egea-Gómez S, Saggion H (2022) Translating Spanish into Spanish sign language: Combining rules and data-driven approaches

  38. Sridhar A, Ganesan RG, Kumar P, Khapra MM (2020) Include: A large scale dataset for indian sign language recognition. Proceedings of the 28th ACM International Conference on Multimedia

  39. Joshi A, Bhat A, S, P., Gole P, Gupta S, Agarwal S, Modi A (2022) Cislr: Corpus for indian sign language recognition. In: Conference on Empirical Methods in Natural Language Processing

  40. Bansal M, Gupta S (2021) Detection and recognition of hand gestures for indian sign language recognition system. In: 2021 6th International Conference on Signal Processing, Computing and Control (ISPCC), pp. 136–140. https://doi.org/10.1109/ISPCC53510.2021.9609448

  41. Sajanraj TD, Beena M (2018) Indian sign language numeral recognition using region of interest convolutional neural network. In: 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), pp. 636–640. https://doi.org/10.1109/ICICCT.2018.8473141

  42. S, A., Potluri A, George SM, R, G., S, A (2021) Indian sign language recognition using random forest classifier. In: 2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), pp. 1–6. https://doi.org/10.1109/CONECCT52877.2021.9622672

  43. Kumar A, Madaan M, Kumar S, Saha A, Yadav S (2021) Indian sign language gesture recognition in real-time using convolutional neural networks. In: 2021 8th International Conference on Signal Processing and Integrated Networks (SPIN), pp. 562–568. https://doi.org/10.1109/SPIN52536.2021.9566005

  44. Sharma K, Aaryan KA, Dhangar U, Sharma R, Taneja S (2022) Automated indian sign language recognition system using lstm models. In: 2022 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), pp. 461–466. https://doi.org/10.1109/ICCCIS56430.2022.10037711

  45. Sarma N, Talukdar AK, Sarma KK (2021) Real-time indian sign language recognition system using yolov3 model. In: 2021 Sixth International Conference on Image Information Processing (ICIIP), vol. 6, pp. 445–449. https://doi.org/10.1109/ICIIP53038.2021.9702611

  46. Munnaluri V, Pandey V, Singh P (2022) Machine learning based approach for indian sign language recognition. In: 2022 7th International Conference on Communication and Electronics Systems (ICCES), pp. 1128–1132. https://doi.org/10.1109/ICCES54183.2022.9835908

  47. Prathap KB, Swaroop GD, Kumar BP, Kamble V, Parate, M (2023) Islr: Indian sign language recognition. In: 2023 2nd International Conference on Paradigm Shifts in Communications Embedded Systems, Machine Learning and Signal Processing (PCEMS), pp. 1–6. https://doi.org/10.1109/PCEMS58491.2023.10136062

  48. Mistry P, Jotaniya V, Patel P, Patel N, Hasan M (2021) Indian sign language recognition using deep learning. In: 2021 International Conference on Artificial Intelligence and Machine Vision (AIMV), pp. 1–6. https://doi.org/10.1109/AIMV53313.2021.9670933

  49. Das S, Biswas SK, Purkayastha B (2023) Automated Indian sign language recognition system by fusing deep and handcrafted feature. Multimed Tools Appl 82(11):16905–16927. https://doi.org/10.1007/s11042-022-14084-4

    Article  Google Scholar 

  50. Sruthi CJ, Lijiya A (2023) Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (mmpe) for indian sign language. Vis Comput 39(12):6183–6203. https://doi.org/10.1007/s00371-022-02720-x

    Article  MATH  Google Scholar 

  51. Singla V, Bawa S, Singh J (2024) Enhancing Indian sign language recognition through data augmentation and visual transformer. Neural Comput Appl 36(24):15103–15116. https://doi.org/10.1007/s00521-024-09845-1

    Article  MATH  Google Scholar 

  52. Chaman A, Dokmanić I (2021) Truly shift-equivariant convolutional neural networks with adaptive polyphase upsampling. In: 55th Asilomar Conference on Signals, Systems, and Computers

  53. Azulay A, Weiss Y (2019) Why do deep convolutional networks generalize so poorly to small image transformations? J Mach Learn Res 20(184):1–25

    MathSciNet  MATH  Google Scholar 

  54. Zhang R (2019) Making convolutional networks shiftinvariant again. In: Proceedings of Machine Learning Research, pp. 7324–7334

  55. Latif G, Mohammad N, Alghazo J, AlKhalaf R, AlKhalaf R (2019) Arasl: Arabic alphabets sign language dataset. Data Brief 23:103777. https://doi.org/10.1016/j.dib.2019.103777

    Article  Google Scholar 

  56. PRATHUM ARIKERI: American Sign Language (ASL) Dataset. https://www.kaggle.com/datasets/prathumarikeri/american-sign-language-09az

  57. KIRLE LEA: Spanish Sign Language Alphabet (Static). https://www.kaggle.com/datasets/kirlelea/spanish-sign-language-alphabet-static (2022)

  58. Ye C, Evanusa M, He H, Mitrokhin A, Goldstein T, Yorke JA, Fermuller C, Aloimonos Y (2020) Network deconvolution. In: International Conference on Learning Representations. https://openreview.net/forum?id=rkeu30EtvS

  59. Agrawal SC, Jalal AS, Bhatnagar C (2012) Recognition of indian sign language using feature fusion. 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI), 1–5

  60. Shenoy K, Dastane T, Rao V, Vyavaharkar D (2021) Real-time Indian Sign Language (ISL) Recognition

  61. Rahaman MA, Jasim M, Ali MH, Hasanuzzaman M (2014) Real-time computer vision-based bengali sign language recognition. 2014 17th International Conference on Computer and Information Technology (ICCIT), 192–197

  62. C.J., S., A., L (2019) Signet: A deep learning based indian sign language recognition system. In: 2019 International Conference on Communication and Signal Processing (ICCSP), pp. 0596–0600. https://doi.org/10.1109/ICCSP.2019.8698006

  63. Yasir F, Prasad PWC, Alsadoon A, Elchouemi A (2015) Sift based approach on bangla sign language recognition. 2015 IEEE 8th International Workshop on Computational Intelligence and Applications (IWCIA), 35–39

  64. Uddin MA, Chowdhury SA (2016) Hand sign language recognition for bangla alphabet using support vector machine. 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET), 1–4

  65. Katoch S, Singh V, Tiwary US (2022) Indian sign language recognition system using surf with SVM and CNN. Array 14:100141. https://doi.org/10.1016/j.array.2022.100141

    Article  MATH  Google Scholar 

  66. Rao GA, Kishore PVV (2017) Selfie video based continuous indian sign language recognition system. Ain Shams Engineering Journal 9: https://doi.org/10.1016/j.asej.2016.10.013

  67. Ganpatye A, Mane S (2022) Motion based indian sign language recognition using deep learning. In: 2022 2nd International Conference on Intelligent Technologies (CONIT), pp. 1–6. https://doi.org/10.1109/CONIT55038.2022.9848275

  68. Kumar P, Saini R, Behera SK, Dogra DP, Roy PP (2017) Real-time recognition of sign language gestures and air-writing using leap motion. In: 2017 Fifteenth IAPR International Conference on Machine Vision Applications (MVA), pp. 157–160. https://doi.org/10.23919/MVA.2017.7986825

  69. Kumar P, Saini R, Roy PP, Dogra DP (2017) A position and rotation invariant framework for sign language recognition (slr) using Kinect. Multimed Tools Appl 77:8823–8846

    Article  MATH  Google Scholar 

Download references

Acknowledgements

We would like to express our sincere gratitude for continued support from the Department of Computer Science, Vidyasagar University, India and to give access the Lab. to continue research works.

Funding

Not applicable.

Author information

Authors and Affiliations

  1. Department of Computer Science, Vidyasagar University, Paschim Medinipur, West Bengal, 721102, India

    Anudyuti Ghorai, Utpal Nandi, Bachchu Paul & Partha Chowdhuri

  2. Department of Computer Science and Engineering, North Eastern Regional Institute of Science and Technology, Itanagar, Arunachal Pradesh, India

    Moirangthem Marjit Singh

  3. Department of Computer Science, Belda College, Belda, West Bengal, 721424, India

    Chiranjit Changdar

  4. Department of Computer Applications, Maulana Abul Kalam Azad University of Technology, Haringhata, West Bengal, 741249, India

    Pabitra Pal

Authors
  1. Anudyuti Ghorai
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Utpal Nandi
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Moirangthem Marjit Singh
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Chiranjit Changdar
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Bachchu Paul
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Partha Chowdhuri
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Pabitra Pal
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

A. G.: Implementation and Drafting; U. N.: Conceptualization, Investigation, Methodology, Analysis, Supervision, Review and Editing; Others: Review and Editing.

Corresponding author

Correspondence to Utpal Nandi.

Ethics declarations

Conflict of interest

There are no Conflict of interest/Conflict of interest.

Ethics approval

The authors approve that the research presented in this paper is conducted following the principles of ethical and professional conduct.

Consent to participate

Not applicable.

Consent for publication

Not applicable, the authors used publicly available data only and provided the corresponding references.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghorai, A., Nandi, U., Singh, M.M. et al. Tsi-cnn-net: truly shift-invariant convolutional neural network for Indian sign language recognition system. Pattern Anal Applic 28, 52 (2025). https://doi.org/10.1007/s10044-025-01428-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10044-025-01428-7

Keywords