Skip to main content
Springer Nature Link
Log in

User adaptive hand gesture recognition for ISL using multiscale and attention embedded residual densenet with adaptive gesture segmentation framework

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Verbal communication is the predominant and most essential form of interaction in daily life. Sign language, particularly Indian Sign Language (ISL) serves as the main communication method. This visual language incorporates an extensive variety of gestures and visual cues for expression. The lack of widespread knowledge about the meaning behind these gestures leads to a communication gap between the hearing-impaired communities and hearing. Consequently, there is a pressing need for an automated system to bridge this divide. Although American Sign Language has been the subject of extensive research, ISL has unfortunately not been given equal consideration. Individuals with hearing impairments rely on hand gestures to communicate. Regrettably, the vast majority of people do not understand the significance of these gestures. Therefore, a new deep model-based hand gesture recognition of ISL is proposed in this paper to perform the hand gesture recognition on ISL. The aim of the developed model is accomplished by executing the following steps such as image collection, segmentation, and edge detection and recognition. The required image for hand gesture recognition is garnered from the benchmark dataset. Thereafter, the collected images are passed to the segmentation and edge detection phase which is performed by the Adaptive thresholding based region growing and canny edge detection (ATRG-CED). Here, the threshold value and seed value of the ATRG-CEDis optimized by the Modified Tasmanian devil optimization (MTDO). The segmented and edge images along with the input image is given to the Multiscale and attention embedded residual densenet (MAERDNet) for getting the recognized outcome. The experimental analysis is carried out on the proposed deep model-based hand gesture recognition on ISL to confirm its effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1:
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The data underlying this article are available in Dataset 1: https://www.kaggle.com/datasets/vaishnaviasonawane/indian-sign-language-dataset. Dataset 2: https://www.kaggle.com/datasets/vaishnaviasonawane/indian-sign-language-dataset/data.

References

  1. Kabir, M.H., Hasan, M.A., Shin, W.: CSI-DeepNet: a lightweight deep convolutional neural network based hand gesture recognition system using Wi-Fi CSI signal. IEEE Access 10, 114787–114801 (2022)

    Article  MATH  Google Scholar 

  2. Hu, Y., Wong, Y., Dai, Q., Kankanhalli, M., Geng, W., Li, X.: sEMG-based gesture recognition with embedded virtual hand poses and adversarial learning. IEEE Access 7, 104108–104120 (2019)

    Article  Google Scholar 

  3. Zeng, X., Wu, C., Ye, W.-B.: User-definable dynamic hand gesture recognition based on doppler radar and few-shot learning. IEEE Sens. J. 21(20), 23224–23233 (2021)

    Article  Google Scholar 

  4. Zabihi, S., Rahimian, E., Asif, A., Mohammadi, A.: TraHGR: transformer for hand gesture recognition via electromyography. IEEE Trans. Neural Syst. Rehabil. Eng. 31, 4211–4224 (2023)

    Article  MATH  Google Scholar 

  5. Gao, Q., Chen, Y., Ju, Z., Liang, Y.: Dynamic hand gesture recognition based on 3D hand pose estimation for human-robot interaction. IEEE Sens. J. 22(18), 17421–17430 (2022)

    Article  MATH  Google Scholar 

  6. Lee, M., Bae, J.: Deep learning based real-time recognition of dynamic finger gestures using a data glove. IEEE Access 8, 219923–219933 (2020)

    Article  MATH  Google Scholar 

  7. He, J., Niu, X., Zhao, P., Lin, C., Jiang, N.: From forearm to wrist: deep learning for surface electromyography-based gesture recognition. IEEE Trans. Neural Syst. Rehabil. Eng. 32, 102–111 (2024)

    Article  MATH  Google Scholar 

  8. Chen, X., Li, Y., Hu, R., Zhang, X., Chen, X.: Hand gesture recognition based on surface electromyography using convolutional neural network with transfer learning method. IEEE J. Biomed. Health Inform. 25(4), 1292–1304 (2021)

    Article  MATH  Google Scholar 

  9. Pan, T.-Y., Tsai, W.-L., Chang, C.-Y., Yeh, C.-W., Hu, M.-C.: A hierarchical hand gesture recognition framework for sports referee training-based EMG and accelerometer sensors. IEEE Trans. Cybernet. 52(5), 3172–3183 (2022)

    Article  MATH  Google Scholar 

  10. Stadelmayer, T., Santra, A., Weigel, R., Lurz, F.: Radar-based gesture recognition using a variational autoencoder with deep statistical metric learning. IEEE Trans. Microw. Theory Tech. 70(11), 5051–5062 (2022)

    Article  MATH  Google Scholar 

  11. Zhang, H., Qu, H., Teng, L., Tang, C.-Y.: LSTM-MSA: a novel deep learning model with dual-stage attention mechanisms forearm EMG-based hand gesture recognition. IEEE Trans. Neural Syst. Rehabil. Eng. 31, 4749–4759 (2023)

    Article  MATH  Google Scholar 

  12. Li, Q., Luo, Z., Qi, R., Zheng, J.: DeepTPA-Net: a deep triple attention network for sEMG-based hand gesture recognition. IEEE Access 11, 96797–96807 (2023)

    Article  Google Scholar 

  13. Zou, Y., Cheng, L., Han, L., Li, Z., Song, L.: Decoding electromyographic signal with multiple labels for hand gesture recognition. IEEE Signal Process. Lett. 30, 483–487 (2023)

    Article  MATH  Google Scholar 

  14. Azad, R., Asadi-Aghbolaghi, M., Kasaei, S., Escalera, S.: Dynamic 3D hand gesture recognition by learning weighted depth motion maps. IEEE Trans. Circuits Syst. Video Technol. 29(6), 1729–1740 (2019)

    Article  Google Scholar 

  15. Chen, G., et al.: A novel illumination-robust hand gesture recognition system with event-based neuromorphic vision sensor. IEEE Trans. Autom. Sci. Eng. 18(2), 508–520 (2021)

    Article  MATH  Google Scholar 

  16. Yuan, G., Liu, X., Yan, Q., Qiao, S., Wang, Z., Yuan, L.: Hand gesture recognition using deep feature fusion network based on wearable sensors. IEEE Sens. J. 21(1), 539–547 (2021)

    MATH  Google Scholar 

  17. Song, J.-H., Kong, K., Kang, S.-J.: Dynamic hand gesture recognition using improved spatio-temporal graph convolutional network. IEEE Trans. Circuits Syst. Video Technol. 32(9), 6227–6239 (2022)

    Article  MATH  Google Scholar 

  18. Sharma, S., Singh, S.: Vision-based hand gesture recognition using deep learning for the interpretation of sign language. Expert Syst. Appl. 182, 657 (2021)

    Article  MATH  Google Scholar 

  19. Kunjumon, J., and Megalingam, R.K.: Hand gesture recognition system for translating indian sign language into text and speech, 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli. India, pp. 14–18, (2019)

  20. Moyya, P.D., Neelamsetti, K.K., and Thirumoorthy, P.: Deep attention network for enhanced hand gesture recognition system. SN Computer Science, Vol. 4, article number 324, (2023)

  21. Miah, A.S.M., Hasan, M.A.M., Shin, J.: Dynamic hand gesture recognition using multi-branch attention based graph and general deep learning model. IEEE Access 11, 4703–4716 (2023)

    Article  MATH  Google Scholar 

  22. Aggarwal, A., Bhutani, N., Kapur, R., Dhand, G., Sheoran, A.: Real-time hand gesture recognition using multiple deep learning architectures. SIViP 17, 63–3971 (2023)

    Article  MATH  Google Scholar 

  23. Al-Hammadi, M., Muhammad, G., Abdul, W.: Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access 8, 192527–192542 (2020)

    Article  MATH  Google Scholar 

  24. Dong, Y., Liu, J., and Yan, W.: Dynamic Hand gesture recognition based on signals from specialized data glove and deep learning algorithms. IEEE Transactions on Instrumentation and Measurement. vol 70, pp. 1–14, Art no. 2509014, (2021)

  25. Cruz, P.J., Vásconez, J.P., Romero, R., Chico, A., Benalcázar, M.E., Álvarez, R., Barona López, L.I., Valdivieso Caraguay, Á.L.: A deep Q-network based hand gesture recognition system for control of robotic platforms. Sci. Rep. 13(1), 7956 (2023)

    Article  Google Scholar 

  26. Dehghani, M., Hubálovský, Š., and Trojovský, P.: Tasmanian devil optimization: a new bio-inspired optimization algorithm for solving optimization algorithm. IEEE access, vol. 10, (2022)

  27. Bhavani, H. R., and Champa, H. N.: A multilevel thresholding method based on HPSO for the segmentation of various objective functions. 2022 International Conference on Communication, Computing and Internet of Things (IC3IoT), Chennai, India, pp. 1–5, (2022).

  28. Song, R., Zhang, Z., Liu, H.: Edge connection based canny edge detection algorithm. Pattern Recogn. Image Anal. 27, 740–747 (2017)

    Article  MATH  Google Scholar 

  29. Zhang, W.: Combination of SIFT and canny edge detection for registration between SAR and optical images. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2022)

    Google Scholar 

  30. Humeau, A., Mahe, G., Chapeau-Blondeau, F., Rousseau, D., Abraham, P.: Multiscale analysis of microvascular blood flow: a multiscale entropy study of laser doppler flowmetry time series. IEEE Trans. Biomed. Eng. 58(10), 2970–2973 (2011)

    Article  MATH  Google Scholar 

  31. Luo, Z., Li, J., Zhu, Y.: A deep feature fusion network based on multiple attention mechanisms for joint iris-periocular biometric recognition. IEEE Signal Process. Lett. 28, 1060–1064 (2021)

    Article  MATH  Google Scholar 

  32. Kumar, E.K., Kishore, P.V., Kumar, M.T., Kumar, D.A., Sastry, A.S.: Three-dimensional sign language recognition with angular velocity maps and connived feature ResNet. IEEE Signal Process. Lett. 25(12), 1860–1864 (2018)

    Article  MATH  Google Scholar 

  33. Sadollah, A., Bahreininejad, A., Eskandar, H., Hamdi, M.: Mine blast algorithm: a new population based algorithm for solving constrained engineering optimization problems. Appl. Soft Comput. 13(5), 2592–2612 (2013)

    Article  MATH  Google Scholar 

  34. Desuky, A.S., Hussain, S., Kausar, S., Islam, M.A., Bakrawy, L.M.E.: EAOA: an enhanced archimedes optimization algorithm for feature selection in classification. IEEE Access 9, 120795–120814 (2021)

    Article  Google Scholar 

  35. El-Dabah, M.A., Hassan, M.H., Kamel, S., Zawbaa, H.M.: Robust parameters tuning of different power system stabilizers using a quantum artificial gorilla troops optimizer. IEEE Access 10, 82560–82579 (2022)

    Article  Google Scholar 

  36. Ronald, M., Poulose, A., Han, D.S.: iSPLInception: an inception-resnet deep learning architecture for human activity recognition. IEEE Access 9, 68985–69001 (2021)

    Article  Google Scholar 

  37. Pan, H., Pang, Z., Wang, Y., Wang, Y., Chen, L.: A new image recognition and classification method combining transfer learning algorithm and MobileNet model for welding defects. IEEE Access 8, 119951–119960 (2020)

    Article  Google Scholar 

  38. Cui, X., Yu, Z., Li, J., Jiang, B., Li, S., Liu, J.: Research on RFID localization algorithm of refrigerator foods based on improved DenseNet model. IEEE Netw. Lett. 5(2), 135–139 (2023)

    Article  MATH  Google Scholar 

  39. Li, H., Gong, M., Liu, J.: A local statistical fuzzy active contour model for change detection. IEEE Geosci. Remote Sens. Lett. 12(3), 582–586 (2015)

    Article  MATH  Google Scholar 

  40. Meiyan, P., Jun, S., Yuhao, Y., Dasheng, L., Junpeng, Y.: M-FCN based sea-surface weak target detection. J. Syst. Eng. Electron. 32(5), 1111–1118 (2021)

    Article  Google Scholar 

  41. Sinaga, K.P., Yang, M.S.: Unsupervised K-means clustering algorithm. IEEE Access 8, 80716–80727 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

I would like to express my very great appreciation to the co-authors of this manuscript for their valuable and constructive suggestions during the planning and development of this research work.

Funding

This research did not receive any specific funding.

Author information

Authors and Affiliations

  1. Department of Electronics and Communication Engineering, MIT School of Engineering, MIT ADT University, Pune, Maharashtra, 412201, India

    Snehal Abhijeet Gaikwad & Virendra Shete

Authors
  1. Snehal Abhijeet Gaikwad
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Virendra Shete
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

All authors have made substantial contributions to conception and design, revising the manuscript, and the final approval of the version to be published. Also, all authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to Snehal Abhijeet Gaikwad.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gaikwad, S.A., Shete, V. User adaptive hand gesture recognition for ISL using multiscale and attention embedded residual densenet with adaptive gesture segmentation framework. SIViP 19, 210 (2025). https://doi.org/10.1007/s11760-024-03668-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11760-024-03668-2

Keywords