Skip to main content

Advertisement

Springer Nature Link
Log in

HandGCNN model for gesture recognition based voice assistance

  • 1220: Visual and Sensory Data Processing for Real Time Intelligent Surveillance System
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Communication plays an important role in today’s world. Before the evolution of the verbal communication, sign language was the only way of communication used by our ancestors. Later on, the verbal communication started evolving and different people from different region started to speak different languages. But there are some groups of people who cannot express themselves with verbal language; instead they use sign language to communicate. To bridge the gap between those people who use sign language for communication with those who use verbal language, a system is designed that recognizes the gestures of the sign language, interprets it and converts it into verbal language. Various researches have been carried out by capturing the hand signs of the speech impaired people through sensors like leap motion sensors and camera. This research works focusses on improving the gesture capturing through camera and process them through deep learning models. This work focussed on creating a hand gesture dataset “HandG” that includes 20,600 images for 10 classes (2060 images per category) using digital camera and image augmentation. A novel Convolution Neural Networks (CNN) based model, termed as “HandGCNN”, is proposed achieving a high prediction accuracy of 99.13%. A real-time system with webcam being the input receptor unit is built which recognises the signal and generates the audio relevant to that. The generated audio will serve as voice assistance for impaired people.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Algorithm 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Apoorva A, Mishra GK, Sahoo RR, Bhoi SK, Mallick C (2021) Deep learning-based ship detection in remote sensing imagery using TensorFlow. In Advances in machine learning and computational intelligence (pp 165–177). Springer, Singapore. https://doi.org/10.1007/978-981-15-5243-4_14

  2. Badi H (2016) RETRACTED ARTICLE: a survey on recent vision-based gesture recognition. Intell Ind Syst 2(2):179–191. https://doi.org/10.1007/s40903-016-0046-9

    Article  Google Scholar 

  3. Bhoi SK, Panda SK, Patra B, Pradhan B, Priyadarshinee P, Tripathy S, … Khilar PM (2018) FallDS-IoT: a fall detection system for elderly healthcare based on IoT data analytics. In 2018 International Conference on Information Technology (ICIT) (pp 155–160). IEEE. https://doi.org/10.1109/ICIT.2018.00041

  4. Dong Y, Liu J, Yan W (2021) Dynamic hand gesture recognition based on signals from specialized data glove and deep learning algorithms. IEEE Trans Instrum Meas 70:1–14. https://doi.org/10.1109/TIM.2021.3077967

    Article  Google Scholar 

  5. Gullapalli S, Karthik P, Sathish P (2020) A comparative analysis of cloud based Watson system and CNN for gesture recognition systems. In 2020 IEEE international Students' conference on electrical, electronics and computer science (SCEECS) (pp 1–5). IEEE. https://doi.org/10.1109/SCEECS48394.2020.66

  6. He K, Sun J (2015) Convolutional neural networks at constrained time cost. In proceedings of the IEEE conference on computer vision and pattern recognition (pp 5353–5360)

  7. Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) MobileNets: efficient convolutional neural networks for Mobile vision applications. ArXiv, abs/1704.04861

  8. Jain A, Jain L, Sharma I, Chauhan A (2018) Image processing based speaking system for mute people using hand gesture. Int J Eng Sci Res Technol (IJESRT) 368–374

  9. Jain R, Karsh RK, Barbhuiya AA (2021) Encoded motion image-based dynamic hand gesture recognition. The visual computer, 1–18. https://doi.org/10.1016/j.cag.2021.04.017

  10. Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick RB, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM international conference on multimedia. https://doi.org/10.1145/2647868.2654889

  11. Kepuska V, Bohouta G (2018) Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC) (pp 99–103). IEEE. https://doi.org/10.1109/CCWC.2018.8301638

  12. Khan T, Pathan AH (2015) Hand gesture recognition based on digital image processing using MATLAB. Int J Sci Eng Res 6(9):338–346

    Google Scholar 

  13. Khari M, Garg AK, Crespo RG, Verdú E (2019) Gesture recognition of RGB and RGB-D static images using convolutional neural networks. Int J Interact Multim Artif Intell 5(7):22–27. https://doi.org/10.9781/ijimai.2019.09.002

    Article  Google Scholar 

  14. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. Commun ACM 60:84–90. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  15. Li Y, Ma D, Yu Y, Wei G, Zhou Y (2021) Compact joints encoding for skeleton-based dynamic hand gesture recognition. Comput Graph 97:191–199

    Article  Google Scholar 

  16. Lingyun G, Lin Z, Zhaokui W (2020) Hierarchical attention-based astronaut gesture recognition: a dataset and CNN model. IEEE Access 8:68787–68798. https://doi.org/10.1109/ACCESS.2020.2986473

    Article  Google Scholar 

  17. Nikam AS, Ambekar AG (2016) Sign language recognition using image based hand gesture recognition techniques. In 2016 online international conference on green engineering and technologies (IC-GET) (pp 1–5). IEEE. https://doi.org/10.1109/GET.2016.7916786

  18. Oudah M, Al-Naji A, Chahl J (2020) Hand gesture recognition based on computer vision: a review of techniques. J Imaging 6(8):73. https://doi.org/10.3390/jimaging6080073

    Article  Google Scholar 

  19. Plouffe G, Cretu AM (2015) Static and dynamic hand gesture recognition in depth data using dynamic time warping. IEEE Trans Instrum Meas 65(2):305–316. https://doi.org/10.1109/TIM.2015.2498560

    Article  Google Scholar 

  20. Poornima N, Murugan M (2019) Improved gesture precision virtual personal assistant (IGP-VPA) system for speech impaired people. I-manager's J Pattern Recogn 6(2):17. https://doi.org/10.26634/jpr.6.2.16754

    Article  Google Scholar 

  21. Qian Y, Dong J, Wang W, Tan T (2015) Deep learning for steganalysis via convolutional neural networks. In media watermarking, security, and forensics 2015 (Vol 9409, p 94090J). International Society for Optics and Photonics. https://doi.org/10.1117/12.2083479

  22. Rai P, Alva A, Mahale GK, Shetty JS, Manjushree AN (2018) Gesture recognition system. Int J Comput Sci Mob Comput 7(5):164–175

    Google Scholar 

  23. Reda MM, Mohammed NG, Seoud RAAAA (2018) SVBiComm: Sign-Voice Bidirectional Communication System for Normal, “Deaf/Dumb” and Blind People based on Machine Learning. In 2018 1st International Conference on Computer Applications & Information Security (ICCAIS) (pp 1–8). IEEE

  24. Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FC, … Zito T (2017) Sustainable computational science: the ReScience initiative. PeerJ Comput Sci 3:e142

    Article  Google Scholar 

  25. Saha HN, Tapadar S, Ray S, Chatterjee SK, Saha S (2018) A machine learning based approach for hand gesture recognition using distinctive feature extraction. In 2018 IEEE 8th annual computing and communication workshop and conference (CCWC) (pp 91–98). IEEE. https://doi.org/10.1109/CCWC.2018.8301631

  26. Saxena A, Jain DK, Singhal A (2014) Hand gesture recognition using an android device. In 2014 fourth international conference on communication systems and network technologies (pp 819–822). IEEE. https://doi.org/10.1109/CSNT.2014.170

  27. Shinde SS, Autee R (2016) Real time hand gesture recognition and voice conversion system for deaf and dumb person based on image processing. JournalNX 2(9):39–43

    Google Scholar 

  28. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556

  29. Zhang W, Wang J, Lan F (2020) Dynamic hand gesture recognition based on short-term sampling neural networks. IEEE/CAA J Autom Sin 8(1):110–120. https://doi.org/10.1109/JAS.2020.1003465

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India

    Rena Stellin, P. Rukmani & L. Jani Anbarasi

  2. Centre for Advanced Data Science, Vellore Institute of Technology, Chennai, India

    Sathiya Narayanan

Authors
  1. Rena Stellin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. P. Rukmani
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. L. Jani Anbarasi
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Sathiya Narayanan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Sathiya Narayanan.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix 1

Appendix 1

1. Summary of the proposed HandGCNN model

Fig. 5
figure 5

Summary of HandGCNN Model

Full size image

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stellin, R., Rukmani, P., Anbarasi, L.J. et al. HandGCNN model for gesture recognition based voice assistance. Multimed Tools Appl 81, 42353–42369 (2022). https://doi.org/10.1007/s11042-022-13497-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13497-5

Keywords