Skip to main content
SpringerLink
Log in

Application Development for Hand Gestures Recognition with Using a Depth Camera

  • Conference paper
  • First Online:
Databases and Information Systems (DB&IS 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1243))

Included in the following conference series:

  • 591 Accesses

Abstract

The aim of the work is to develop an application for hand gestures identification based on a convolutional neural network using the TensorFlow & Keras deep learning frameworks. The gesture recognition system consists of a gesture presentation, a gesture capture device (sensor), the preprocessing and image segmentation algorithms, the features extraction algorithm, and gestures classification. As a sensor, Intel® Real Sense™ depth camera D435 with USB 3.0 support for connecting to a computer was used. For video processing and extraction both RGB images and depth information from the input data, functions from the Intel Real Sense library are applied. For pre-processing and image segmentation algorithms computer vision methods from the OpenCV library are implemented. The subsystem for the features extracting and gestures classification is based on the modified VGG-16, with weights previously trained on the ImageNet database. Performance of the gesture recognition system is evaluated using a custom dataset. Experimental results show that the proposed model, trained on a database of 2000 images, provides high recognition accuracy (99.4%).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jaimes, A., Sebe, N.: Multimodal human–computer interaction: a survey. Comput. Vis. Image Underst. 108, 116–134 (2007)

    Article  Google Scholar 

  2. Bhame, V., Sreemathy, R., Dhumal, H.: Vision based hand gesture recognition using eccentric approach for human computer interaction In: Proceedings of 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 949–953. IEEE (2014)

    Google Scholar 

  3. Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2015)

    Article  Google Scholar 

  4. Pisharady, P.K., Saerbeck, M.: Recent methods and databases in vision-based hand gesture recognition: a review. Comput. Vis. Image Underst. 141, 152–165 (2015)

    Article  Google Scholar 

  5. Chakraborty, B.K., Sarma, D., Bhuyan, M.K., MacDorman, K.F.: Review of constraints on vision-based gesture recognition for human–computer interaction. IET Comput. Vision 12(1), 3–15 (2017)

    Article  Google Scholar 

  6. Kevin, N.Y.Y., Ranganath, S., Ghosh, D.: Trajectory modeling in gesture recognition using CyberGloves and magnetic trackers. In: Proceedings of IEEE Region 10 Conference TENCON, pp. 571–574 (2004)

    Google Scholar 

  7. Zhao, H.-C., Liu, Xi-Yu.: An improved DNA computing method for elevator scheduling problem. In: Zu, Q., Hu, B., Elçi, A. (eds.) ICPCA/SWS 2012. LNCS, vol. 7719, pp. 869–875. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-37015-1_76

    Chapter  Google Scholar 

  8. Sree, S.R., Vyshnavi, S.B., Jayapandian, N.: Real-world application of machine learning and deep learning. In: 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), pp. 1069–1073. IEEE (2019)

    Google Scholar 

  9. Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 37(3), 311–324 (2007)

    Article  Google Scholar 

  10. Kendon, A.: Current issues in the study of gesture. In: The Biological Foundation of Gestures: Motor and Semiotic Aspects, pp. 23–47. Psychology Press (1986)

    Google Scholar 

  11. Chiu, C.Y., Thelwell, M., Senior, T., Choppin, S., Hart, J., Wheat, J.: Comparison of depth cameras for 3D reconstruction in medicine. J. Eng. Med. 233(9), 938–947 (2019)

    Article  Google Scholar 

  12. Fang, Q., Kyrarini, M., Ristic-Durrant, D., Gräser, A.: RGB-D camera based 3D human mouth detection and tracking towards robotic feeding assistance. In: Proceedings of the 11th Pervasive Technologies Related to Assistive Environments Conference, pp. 391–396 (2018)

    Google Scholar 

  13. Satybaldina, D.Zh., Kalymova, K.A.: Development a gesture-based application using Microsoft Kinect sensor. In: 21th International Conference Proceedings on Digital signal processing, DSPA 2019, Moscow, Russia, pp. 525–529 (2019). (in Russian)

    Google Scholar 

  14. Zhang, S.: High-speed 3D shape measurement with structured light methods: a review. Opt. Lasers Eng. 106, 119–131 (2018)

    Article  Google Scholar 

  15. Keselman, L., Woodfill, J.I., Grunnet-Jepsen, A., Bhowmik, A.: Intel R RealSense TM stereoscopic depth cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–10 (2017)

    Google Scholar 

  16. Intel® RealSense™ SDK 2.0. https://www.intelrealsense.com/developers/. Accessed 24 Jan 2020

  17. Intel RealSense D400 Series Product Family. Datasheet. 2019 Intel Corporation. Document Number: 337029-007. https://www.intel.com. Accessed 24 Jan 2020

  18. Bock, R.D.: Low-cost 3D security camera. In: Autonomous Systems: Sensors, Vehicles, Security, and the Internet of Everything. International Society for Optics and Photonics, vol. 10643, pp. 106430E (2018)

    Google Scholar 

  19. Aoki, H., Suzuki, A., Shiga, T.: Study on non-contact heart beat measurement method by using depth sensor. In: Lhotska, L., Sukupova, L., Lacković, I., Ibbott, Geoffrey S. (eds.) World Congress on Medical Physics and Biomedical Engineering 2018. IP, vol. 68/1, pp. 341–345. Springer, Singapore (2019). https://doi.org/10.1007/978-981-10-9035-6_62

    Chapter  Google Scholar 

  20. Syed, T.N., et al.: Seedling-lump integrated non-destructive monitoring for automatic transplanting with Intel RealSense depth camera. Artif. Intell. Agricult. 3, 18–32 (2019)

    Google Scholar 

  21. Holte, M.B., Moeslund, T.B., Fihl, P.: Fusion of range and intensity information for view invariant gesture recognition. In: 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–7. IEEE (2008)

    Google Scholar 

  22. Van den Bergh M., et al.: Real-time 3D hand gesture interaction with a robot for understanding directions from humans. In: 2011 Ro-Man, pp. 357–362. IEEE (2011)

    Google Scholar 

  23. Ren, Z., Yuan, J., Zhang, Z.: Robust hand gesture recognition based on finger-earth mover’s distance with a commodity depth camera. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 1093–1096 (2011)

    Google Scholar 

  24. Wu, D., Zhu, F., Shao, L.: One shot learning gesture recognition from RGBD images. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 7–12. IEEE (2012)

    Google Scholar 

  25. Keskin, C., Kirac, F., Kara, Y., Akarun, L.: Randomized decision forests for static and dynamic hand shape classification. In: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 31–36. IEEE (2012)

    Google Scholar 

  26. Liao, B., Li, J., Ju, Z., Ouyang, G.: Hand gesture recognition with generalized hough transform and DC-CNN using RealSense. In: 2018 Eighth International Conference on Information Science and Technology (ICIST), pp. 84–90. IEEE (2018)

    Google Scholar 

  27. Chernov, V., Alander, J., Bochko, V.: Integer-based accurate conversion between RGB and HSV color spaces. Comput. Electr. Eng. 46, 328–337 (2015)

    Article  Google Scholar 

  28. Simonyan, K., Zisserman. A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556. (2014)

    Google Scholar 

  29. Rezende, E., Ruppert, G., Carvalho, T., Theophilo, A., Ramos, F., Geus, Pd: Malicious software classification using VGG16 deep neural network’s bottleneck features. In: Latifi, S. (ed.) Information Technology - New Generations. AISC, vol. 738, pp. 51–59. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-77028-4_9

    Chapter  Google Scholar 

  30. Liu, Z., et al.: Improved kiwifruit detection using pre-trained VGG16 with RGB and NIR information fusion. IEEE Access, pp. 2327–2336 (2019)

    Google Scholar 

  31. Mantecón, T., del-Blanco, Carlos R., Jaureguizar, F., García, N.: Hand gesture recognition using infrared imagery provided by leap motion controller. In: Blanc-Talon, J., Distante, C., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2016. LNCS, vol. 10016, pp. 47–57. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48680-2_5

    Chapter  Google Scholar 

  32. Visa, S., Ramsay, B., Ralescu, A.L., Van Der Knaap, E.: Confusion matrix-based feature selection. MAICS 710, 120–127 (2011)

    Google Scholar 

Download references

Acknowledgements

This work has been supported by the Ministry of Digital Development, Innovations and Aerospace Industry of the Kazakhstan Republic under project № AP06850817.

Author information

Authors and Affiliations

  1. National Research Nuclear University “MEPhI”, 31 Kashirskoe Road, 115409, Moscow, Russian Federation

    Dina Satybaldina

  2. L.N. Gumilyov Eurasian National University, 11 Pushkin Street, 010008, Nur-Sultan, Kazakhstan

    Dina Satybaldina, Gulziya Kalymova & Natalya Glazyrina

Authors
  1. Dina Satybaldina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gulziya Kalymova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Natalya Glazyrina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dina Satybaldina .

Editor information

Editors and Affiliations

  1. Department of Computer Systems, Tallinn University of Technology, Tallinn, Estonia

    Tarmo Robal

  2. Department of Software Science, Tallinn University of Technology, Tallinn, Estonia

    Hele-Mai Haav

  3. Department of Software Science, Tallinn University of Technology, Tallinn, Estonia

    Jaan Penjam

  4. Institute of Computer Science, University of Tartu, Tartu, Estonia

    Raimundas Matulevičius

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Satybaldina, D., Kalymova, G., Glazyrina, N. (2020). Application Development for Hand Gestures Recognition with Using a Depth Camera. In: Robal, T., Haav, HM., Penjam, J., Matulevičius, R. (eds) Databases and Information Systems. DB&IS 2020. Communications in Computer and Information Science, vol 1243. Springer, Cham. https://doi.org/10.1007/978-3-030-57672-1_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-57672-1_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-57671-4

  • Online ISBN: 978-3-030-57672-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation