Skip to main content
SpringerLink
Log in

Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (MMPE) for Indian Sign Language

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The ability to communicate in verbal language is one of the greatest gifts of humankind. The people who do not have this ability feel isolated and struggle to convey their part in society. Sign language or gesture communication is the only method they can rely upon, but most of our community cannot understand this language without the help of a translator. The paper presents a dynamic Indian Sign Language recognition system without complicated sensors or costly devices to sense the movements of the hands. The paper proposes a problem-specific contour-based hand tracking algorithm that can track both hands simultaneously, solving the ambiguity caused by merging the hands while gesturing. The paper also proposes a maximum mean probability ensembling that combines the classification probabilities of three different classification models for better accuracy. The proposed model recognizes the double-handed dynamic gestures with an accuracy of 89.83%. The paper discusses the performance of scale-invariant feature transform, tracked image feature and their combination feature for dynamic gesture classification, and tests the discriminating power of different classifiers on these features. The support vector machine classifier showed the best performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The datasets generated and analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Abdelnasser, H., Youssef, M., Harras, K.A.: Wigest: A ubiquitous wifi-based gesture recognition system. In: 2015 IEEE Conference on Computer Communications (INFOCOM), pp. 1472–1480. IEEE (2015)

  2. Abreu, J.G., Teixeira, J.M., Figueiredo, L.S., Teichrieb, V.: Evaluating sign language recognition using the myo armband. In: 2016 XVIII symposium on virtual and augmented reality (SVR), pp. 64–70. IEEE (2016)

  3. Al Farid, F., Hashim, N., Abdullah, J., Bhuiyan, M.R., Shahida Mohd Isa, W.N., Uddin, J., Haque, M.A., Husen, M.N.: A structured and methodological review on vision-based hand gesture recognition system. J. Imaging 8(6), 153 (2022)

  4. Al-Hammadi, M., Bencherif, M.A., Alsulaiman, M., Muhammad, G., Mekhtiche, M.A., Abdul, W., Alohali, Y.A., Alrayes, T.S., Mathkour, H., Faisal, M., et al.: Spatial attention-based 3d graph convolutional neural network for sign language recognition. Sensors 22(12), 4558 (2022)

    Article  Google Scholar 

  5. Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M.A., Alrayes, T.S., Mathkour, H., Mekhtiche, M.A.: Deep learning-based approach for sign language gesture recognition with efficient hand gesture representation. IEEE Access 8, 192527–192542 (2020). https://doi.org/10.1109/ACCESS.2020.3032140

    Article  Google Scholar 

  6. Aly, S., Aly, W.: Deeparslr: A novel signer-independent deep learning framework for isolated arabic sign language gestures recognition. IEEE Access 8, 83199–83212 (2020). https://doi.org/10.1109/ACCESS.2020.2990699

    Article  Google Scholar 

  7. Athira, P.K., Sruthi, C.J., Lijiya, A.: A signer independent sign language recognition with co-articulation elimination from live videos: an Indian scenario. J. King Saud Univ. Comput. Inf. Sci. 34(3), 771–781 (2022). https://doi.org/10.1016/j.jksuci.2019.05.002

  8. Bantupalli, K., Xie, Y.: American sign language recognition using deep learning and computer vision. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 4896–4899. IEEE (2018)

  9. Bastos, I.L., Angelo, M.F., Loula, A.C.: Recognition of static gestures applied to brazilian sign language (libras). In: 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, pp. 305–312. IEEE (2015)

  10. Bhaumik, G., et al.: ExtriDeNet: an intensive feature extrication deep network for hand gesture recognition. Vis. Comput. 38, 3853–3866 (2022)

  11. Bukhari, J., Rehman, M., Malik, S.I., Kamboh, A.M., Salman, A.: American sign language translation through sensory glove; signspeak. Int. J. u- e-Serv. Sci. Technol. 8(1), 131–142 (2015)

    Article  Google Scholar 

  12. Cheok, M.J., Omar, Z., Jaward, M.H.: A review of hand gesture and sign language recognition techniques. Int. J. Mach. Learn. Cybern. 10(1), 131–153 (2019)

    Article  Google Scholar 

  13. Falvo, V., Scatalon, L.P., Barbosa, E.F.: The role of technology to teaching and learning sign languages: A systematic mapping. In: 2020 IEEE Frontiers in Education Conference (FIE), pp. 1–9. IEEE (2020)

  14. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis, pp. 363–370. Springer (2003)

  15. Gadekallu, T.R., Srivastava, G., Liyanage, M., Iyapparaja, M., Chowdhary, C.L., Koppu, S., Maddikunta, P.K.R.: Hand gesture recognition based on a harris hawks optimized convolution neural network. Comput. Electr. Eng. 100, 107836 (2022)

    Article  Google Scholar 

  16. Gurbuz, S.Z., Gurbuz, A.C., Malaia, E.A., Griffin, D.J., Crawford, C.S., Rahman, M.M., Kurtoglu, E., Aksu, R., Macks, T., Mdrafi, R.: American sign language recognition using rf sensing. IEEE Sensors J. 21(3), 3763–3775 (2020)

    Google Scholar 

  17. Hartanto, R., Susanto, A., Santosa, P.I.: Real time static hand gesture recognition system prototype for indonesian sign language. In: 2014 6th International Conference on Information Technology and Electrical Engineering (ICITEE), pp. 1–6. IEEE (2014)

  18. Imran, J., Raman, B.: Deep motion templates and extreme learning machine for sign language recognition. Vis. Comput. 36(6), 1233–1246 (2020)

    Article  Google Scholar 

  19. Jain, R., Karsh, R.K., Barbhuiya, A.A.: Encoded motion image-based dynamic hand gesture recognition. Vis. Comput. 38(6), 1957–1974 (2022)

    Article  Google Scholar 

  20. Johnson, R.J., Johnson, J.E.: Distinction between west bengal sign language and indian sign language based on statistical assessment. Sign Lang. Stud. 16(4), 473–499 (2016)

    Article  Google Scholar 

  21. Karami, A., Zanj, B., Sarkaleh, A.K.: Persian sign language (psl) recognition using wavelet transform and neural networks. Expert Syst. Appl. 38(3), 2661–2667 (2011)

    Article  Google Scholar 

  22. Kishore, P., Kumar, D.A., Sastry, A.C.S., Kumar, E.K.: Motionlets matching with adaptive kernels for 3-d indian sign language recognition. IEEE Sensors J. 18(8), 3327–3337 (2018)

    Article  Google Scholar 

  23. Kumar, A., Thankachan, K., Dominic, M.M.: Sign language recognition. In: 2016 3rd International Conference on Recent Advances in Information Technology (RAIT), pp. 422–428. IEEE (2016)

  24. Kumar, D.A., et al.: Indian sign language recognition using graph matching on 3D motion captured signs. Multimedia Tools Appl. 77(24), 32063–32091 (2018)

  25. Kumar, P., Gauba, H., Roy, P.P., Dogra, D.P.: Coupled hmm-based multi-sensor data fusion for sign language recognition. Pattern Recogn. Lett. 86, 1–8 (2017)

    Article  Google Scholar 

  26. Kumar, P., Gauba, H., Roy, P.P., Dogra, D.P.: A multimodal framework for sensor based sign language recognition. Neurocomputing 259, 21–38 (2017)

    Article  Google Scholar 

  27. Liao, Y., Xiong, P., Min, W., Min, W., Lu, J.: Dynamic sign language recognition based on video sequence with blstm-3d residual networks. IEEE Access 7, 38044–38054 (2019)

    Article  Google Scholar 

  28. Lindeberg, T.: Scale invariant feature transform. Scholarpedia 7(5), 10491 (2012). https://doi.org/10.4249/scholarpedia.10491

    Article  Google Scholar 

  29. Liu, T., Zhou, W., Li, H.: Sign language recognition with long short-term memory. In: 2016 IEEE international conference on image processing (ICIP), pp. 2871–2875. IEEE (2016)

  30. Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision, vol. 2, pp. 1150–1157. Ieee (1999)

  31. Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. Vancouver, British Columbia (1981)

    Google Scholar 

  32. Mandke, K., Chandekar, P.: Deaf education in india. In: Deaf Education Beyond the Western World: Context, Challenges, and Prospects pp. 261–84 (2019)

  33. Mittal, A., Kumar, P., Roy, P.P., Balasubramanian, R., Chaudhuri, B.B.: A modified lstm model for continuous sign language recognition using leap motion. IEEE Sensors J. 19(16), 7056–7063 (2019)

    Article  Google Scholar 

  34. Oliveira, V., Conci, A.: Skin detection using hsv color space. In: H. Pedrini, & J. Marques de Carvalho, Workshops of Sibgrapi, pp. 1–2. Citeseer (2009)

  35. Oudah, M., Al-Naji, A., Chahl, J.: Hand gesture recognition based on computer vision: a review of techniques. journal of Imaging 6(8), 73 (2020)

  36. Pan, T.Y., Lo, L.Y., Yeh, C.W., Li, J.W., Liu, H.T., Hu, M.C.: Real-time sign language recognition in complex background scene based on a hierarchical clustering classification method. In: 2016 IEEE Second International Conference on Multimedia Big Data (BigMM), pp. 64–67. IEEE (2016)

  37. Patil, S.B., Sinha, G.: Distinctive feature extraction for indian sign language (isl) gesture using scale invariant feature transform (sift). J. Inst. Eng. (India) Ser. B 98(1), 19–26 (2017)

    Article  Google Scholar 

  38. Rastgoo, R., Kiani, K., Escalera, S.: Hand sign language recognition using multi-view hand skeleton. Expert Syst. Appl. 150, 113336 (2020)

    Article  Google Scholar 

  39. Rekha, J., Bhattacharya, J., Majumder, S.: Hand gesture recognition for sign language: A new hybrid approach. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), p. 1. The Steering Committee of The World Congress in Computer Science, Computer ... (2011)

  40. Rekha, J., Bhattacharya, J., Majumder, S.: Shape, texture and local movement hand gesture features for indian sign language recognition. In: Trendz in Information Sciences and Computing (TISC), 2011 3rd International Conference on, pp. 30–35. IEEE (2011)

  41. Sahoo, J.P., Prakash, A.J., Pławiak, P., Samantray, S.: Real-time hand gesture recognition using fine-tuned convolutional neural network. Sensors 22(3), 706 (2022)

    Article  Google Scholar 

  42. Sethi, A., Hemanth, S., Kumar, K., Bhaskara Rao, N., Krishnan, R.: Signpro-an application suite for deaf and dumb. IJCSET 2(5), 1203–1206 (2012)

    Google Scholar 

  43. Sharma, K., Joshi, G., Dutta, M.: Analysis of shape and orientation recognition capability of complex zernike moments for signed gestures. In: 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), pp. 730–735. IEEE (2015)

  44. Shi, J., et al.: Good features to track. In: 1994 Proceedings of IEEE conference on computer vision and pattern recognition, pp. 593–600. IEEE (1994)

  45. Shurong, L., Yuanyuan, H., Zuojin, H., Qun, D.: Key frame detection algorithm based on dynamic sign language video for the non specific population. Int. J. Signal Process. Image Process. Pattern Recogn. 8(12), 135–148 (2015)

    Google Scholar 

  46. Sruthi, C., Lijiya, A.: Signet: A deep learning based indian sign language recognition system. In: 2019 International conference on communication and signal processing (ICCSP), pp. 0596–0600. IEEE (2019)

  47. Sruthi, C., Soni, K., Lijiya, A.: Automatic recognition of isl dynamic signs with facial cues. In: Congress on Intelligent Systems, pp. 369–381. Springer (2020)

  48. Tariq, M., Iqbal, A., Zahid, A., Iqbal, Z., Akhtar, J.: Sign language localization: Learning to eliminate language dialects. In: 2012 15th International Multitopic Conference (INMIC), pp. 17–22. IEEE (2012)

  49. Tripathi, K., Baranwal, N., Nandi, G.C.: Continuous dynamic indian sign language gesture recognition with invariant backgrounds. In: Advances in Computing, Communications and Informatics (ICACCI), 2015 International Conference on, pp. 2211–2216. IEEE (2015)

  50. Venugopalan, A., Reghunadhan, R.: Applying deep neural networks for the automatic recognition of sign language words: A communication aid to deaf agriculturists. Expert Syst. Appl. 185, 115601 (2021)

    Article  Google Scholar 

  51. Wang, S., Zhang, S., Zhang, X., et al.: A two-branch hand gesture recognition approach combining atrous convolution and attention mechanism. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02602-2

  52. Yasen, M., Jusoh, S.: A systematic review on hand gesture recognition techniques, challenges and applications. PeerJ Comput. Sci. 5, e218 (2019)

    Article  Google Scholar 

  53. Yun, L., Peng, Z.: An automatic hand gesture recognition system based on viola-jones method and svms. In: Second International Workshop on Computer Science and Engineering pp. 72–77 (2009)

Download references

Acknowledgements

We extend special thanks to Mr Muhammad Niyas T T K, Mr Vineeth V, and Mr Rishad C for their sincere help in dataset creation and the initial coding phase. We also thank all the students of the National Institute of Technology Calicut who were associated with us in creating the dynamic Indian Sign Language dataset. We thank Rahmania Higher Secondary School for Handicapped, Karuna Speech & Hearing Higher Secondary School, and the National Institute of Speech and Hearing (NISH) for their valuable suggestions and support in understanding Indian Sign Language.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology Calicut, Kozhikode, Kerala, India

    C. J. Sruthi & A. Lijiya

Authors
  1. C. J. Sruthi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Lijiya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. J. Sruthi.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sruthi, C.J., Lijiya, A. Double-handed dynamic gesture recognition using contour-based hand tracking and maximum mean probability ensembling (MMPE) for Indian Sign Language. Vis Comput 39, 6183–6203 (2023). https://doi.org/10.1007/s00371-022-02720-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02720-x

Keywords

Search

Navigation