Skip to main content
SpringerLink
Log in

CNN based feature extraction and classification for sign language

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Hand gesture is one of the most prominent ways of communication since the beginning of the human era. Hand gesture recognition extends human-computer interaction (HCI) more convenient and flexible. Therefore, it is important to identify each character correctly for calm and error-free HCI. Literature survey reveals that most of the existing hand gesture recognition (HGR) systems have considered only a few simple discriminating gestures for recognition performance. This paper applies deep learning-based convolutional neural networks (CNNs) for robust modeling of static signs in the context of sign language recognition. In this work, CNN is employed for HGR where both alphabets and numerals of ASL are considered simultaneously. The pros and cons of CNNs used for HGR are also highlighted. The CNN architecture is based on modified AlexNet and modified VGG16 models for classification. Modified pre-trained AlexNet and modified pre-trained VGG16 based architectures are used for feature extraction followed by a multiclass support vector machine (SVM) classifier. The results are evaluated based on different layer features for best recognition performance. To examine the accuracy of the HGR schemes, both the leave-one-subject-out and a random 70–30 form of cross-validation approach were adopted. This work also highlights the recognition accuracy of each character, and their similarities with identical gestures. The experiments are performed in a simple CPU system instead of high-end GPU systems to demonstrate the cost-effectiveness of this work. The proposed system has achieved a recognition accuracy of 99.82%, which is better than some of the state-of-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Badi H (2016) Recent methods in vision-based hand gesture recognition. Int J Data Sci Anal 1(2):77–87

    Article  Google Scholar 

  2. Barczak AL, Reyes NH, Abastillas M, Piccio A, Susnjak T (2011) A new 2D static hand gesture colour image dataset for ASL gestures. Res Lett Inf Math Sci 15:12–20

    Google Scholar 

  3. Bheda V, Radpour D (2018) Using deep convolutional networks for gesture recognition in American sign language. arXiv preprint arXiv:1710.06836.

  4. Chevtchenko SF, Vale RF, Macario V, Cordeiro FR (2018) A convolutional neural network with feature fusion for real-time hand posture recognition. Appl Soft Comput 73:748–766

    Article  Google Scholar 

  5. Czuszyński K, Rumiński J, Kwaśniewska A (2018) Gesture recognition with the linear optical sensor and recurrent neural networks. IEEE Sensors J 18(13):5429–5438

    Article  Google Scholar 

  6. Dadashzadeh A, Targhi AT, Tahmasbi M, Mirmehdi M (2019) HGR-net: a fusion network for hand gesture segmentation and recognition. IET Comput Vis 13(8):700–707

    Article  Google Scholar 

  7. Dehankar AV, Jain S, Thakare VM (2017) Using AEPI method for hand gesture recognition in varying background and blurred images. IEEE Int Conf Electron Commun Aerospace Technol 1:404–409

    Google Scholar 

  8. Fang Y, Liu H, Li G, Zhu X (2015) A multichannel surface EMG system for hand motion recognition. Int J Humanoid Robotics 12(02):1550011

    Article  Google Scholar 

  9. Fang L, Liang N, Kang W, Wang Z, Feng DD (2020) Real-time hand posture recognition using hand geometric features and fisher vector. Signal Process Image Commun 82:115729

    Article  Google Scholar 

  10. Gupta B, Shukla P, Mittal A (2016) K-nearest correlated neighbor classification for Indian sign language gesture recognition using feature fusion. In: IEEE International Conference on Computer Communication and Informatics, pp 1–5

  11. Hasan HS, Kareem SA (2012) Human computer interaction for vision based hand gesture recognition: a survey. In: IEEE International Conference on Advanced Computer Science Applications and Technologies, pp, 55–60

  12. Hassene BA (2019) End-to-end multiview gesture recognition for autonomous Car parking system. MS thesis. University of Waterloo

  13. Jadooki S, Mohamad D, Saba T, Almazyad AS, Rehman A (2017) Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput & Applic 28(11):3285–3294

    Article  Google Scholar 

  14. Jiang D, Zheng Z, Li G, Sun Y, Kong J, Jiang G, Xiong H, Tao B, Xu S, Yu H, Liu H (2019) Gesture recognition based on binocular vision. Clust Comput 22(6):13261–13271

    Article  Google Scholar 

  15. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks In: Adv Neural Inf Process Syst: 1097–1105

  16. Lamar MV (2001) Hand Gesture Recognition using T-CombNET-A Neural Network Model dedicated to Temporal Information Processing. Ph.D. thesis, Nagoya Institute of Technology, Japan

  17. Li SZ, Yu B, Wu W, Su SZ, Ji RR (2015) Feature learning based on SAE–PCA network for human gesture recognition in RGBD images. Neurocomputing 151:565–573

    Article  Google Scholar 

  18. Li SZ, Yu B, Wu W, Su SZ, Ji RR (2015) Feature learning based on SAE–PCA network for human gesture recognition in RGBD images. Neurocomputing 151:565–573

    Article  Google Scholar 

  19. Li Y, Wang X, Liu W, Feng B (2018) Deep attention network for joint hand gesture localization and recognition using static RGB-D images. Inf Sci 441:66–78

    Article  MathSciNet  Google Scholar 

  20. Li G, Zhang L, Sun Y, Kong J (2019) Towards the sEMG hand: internet of things sensors and haptic feedback application. Multimed Tools Appl 78(21):29765–29782

    Article  Google Scholar 

  21. Lin HI, Hsu MH, Chen WK (2014) Human hand gesture recognition using a convolution neural network. In: IEEE Int Conf Automation Sci Eng, pp 1038–1043

  22. Liu P, Li X, Cui H, Li S, Yuan Y (2019) Hand gesture recognition based on single-shot multibox detector deep learning. Mob Inf Syst 2019:1–7

    Google Scholar 

  23. Nagarajan S, Subashini TS (2013) Static hand gesture recognition for sign language alphabets using edge oriented histogram and multi class SVM. Int J Comput Appl 82(4):28–35

    Google Scholar 

  24. Neethu PS, Suguna R, Sathish D (2020) An efficient method for human hand gesture detection and recognition using deep learning convolutional neural networks. Soft Comput 23:1–10

    Google Scholar 

  25. Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput & Applic 28(12):3941–3951

    Article  Google Scholar 

  26. Ozcan T, Basturk A (2019) Transfer learning-based convolutional neural networks with heuristic optimization for hand gesture recognition. Neural Comput & Applic 31(12):8955–8970

    Article  Google Scholar 

  27. Pavlovic VI, Sharma R, Huang TS (1997) Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Trans Pattern Anal Mach Intell 19(7):677–695

    Article  Google Scholar 

  28. Plouffe G, Cretu AM (2015) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316

    Article  Google Scholar 

  29. Ranga V, Yadav N, Garg P (2018) American sign language fingerspelling using hybrid discrete wavelet transform-gabor filter and convolutional neural network. J Eng Sci Technol 13(9):2655–2669

    Google Scholar 

  30. Rathi, P, Kuwar Gupta, R and Agarwal, S., Shukla, A (2019) Sign Language Recognition Using ResNet50 Deep Neural Network Architecture. 5th International Conference on Next Generation Computing Technologies, Available at SSRN 3545064

  31. Rautaray SS, Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54

    Article  Google Scholar 

  32. Shanthakumar VA, Peng C, Hansberger J, Cao L, Meacham S, Blakely V (2020) Design and evaluation of a hand gesture recognition approach for real-time interactions. Multimed Tools Appl 21:1–24

    Google Scholar 

  33. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  34. Sun Y, Li C, Li G, Jiang G, Jiang D, Liu H, Zheng Z, Shu W (2018) Gesture recognition based on kinect and sEMG signal fusion. Mobile Networks Appl 23(4):797–805

    Article  Google Scholar 

  35. Von Hardenberg C, Bérard F (2001) Bare-hand human-computer interaction. In: Proceedings of the 2001 workshop on Perceptive user interfaces, pp 1–8

  36. Wadhawan A, Kumar P (2020) Deep learning-based sign language recognition system for static signs. Neural Comput & Applic 1:1–12

    Google Scholar 

  37. Wang C, Liu Z, Chan SC (2014) Superpixel-based hand gesture recognition with kinect depth camera. IEEE Trans Multimed 17(1):29–39

    Article  Google Scholar 

  38. Zhao J, Allison RS (2019) Comparing head gesture, hand gesture and gamepad interfaces for answering yes/no questions in virtual environments. Virtual Reality 10:1–0

    Google Scholar 

  39. Zhong X, Chen Y, Yu H, Yang X, Hu Z (2018) Context-aware information based ultrasonic gesture recognition method. J Comput-Aided Design Comput Graphics 30(1):173

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge people of Speech and Image Processing Laboratory, National Institute of Technology, Silchar, India, for providing support and necessary facilities for carrying out this work. The authors are thankful to Dr. Amarjit Roy, School of Electronics, VIT-AP University, for the constructive criticism and suggestions.

Author information

Authors and Affiliations

  1. Speech and Image Processing Group, Electronics and Communication Engineering Department, National Institute of Technology, Silchar, Assam, 788010, India

    Abul Abbas Barbhuiya, Ram Kumar Karsh & Rahul Jain

Authors
  1. Abul Abbas Barbhuiya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ram Kumar Karsh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rahul Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abul Abbas Barbhuiya.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barbhuiya, A.A., Karsh, R.K. & Jain, R. CNN based feature extraction and classification for sign language. Multimed Tools Appl 80, 3051–3069 (2021). https://doi.org/10.1007/s11042-020-09829-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09829-y

Keywords

Search

Navigation