Skip to main content
SpringerLink
Log in

Hand Gesture Recognition Exploiting Handcrafted Features and LSTM

  • Conference paper
  • First Online:
Image Analysis and Processing – ICIAP 2023 (ICIAP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14233))

Included in the following conference series:

Abstract

Hand gesture recognition finds application in several heterogeneous fields, such as Human-Computer Interaction, serious games, sign language interpretation, and more. Modern recognition approaches use Deep Learning methods due to their ability in extracting features without human intervention. The drawback of this approach is the need for huge datasets which, depending on the task, are not always available. In some cases, handcrafted features increase the capability of a model in achieving the proposed task, and usually require fewer data with respect to Deep Learning approaches. In this paper, we propose a method that synergistically makes use of handcrafted features and Deep Learning for performing hand gesture recognition. Concerning the features, they are engineered from hand joints, while for Deep Learning, a simple LSTM together with a multilayer perceptron is used. The tests were performed on the DHG dataset, comparing the proposed method with both state-of-the-art methods that use handcrafted features and methods that use learned features. Our approach overcomes the state-of-the-art handcrafted features methods in both 14 and 28 gestures recognition tests, while we overcome the state-of-the-art learned features methods for the 14 gesture recognition test, proving that it is possible to use a simpler model with well engineered features.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alrowais, F., et al.: Modified earthworm optimization with deep learning assisted emotion recognition for human computer interface. IEEE Access 11, 35089–35096 (2023)

    Google Scholar 

  2. Au, S., Dilworth, P., Herr, H.: An ankle-foot emulation system for the study of human walking biomechanics. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 2939–2945 (2006)

    Google Scholar 

  3. Avola, D., Cinque, L., Foresti, G.L., Martinel, N., Pannone, D., Piciarelli, C.: Low-Level Feature Detectors and Descriptors for Smart Image and Video Analysis: A Comparative Study, pp. 7–29 (2018)

    Google Scholar 

  4. Avola, D., Bernardi, M., Cinque, L., Foresti, G.L., Massaroni, C.: Adaptive bootstrapping management by keypoint clustering for background initialization. Pattern Recogn. Lett. 100, 110–116 (2017)

    Article  Google Scholar 

  5. Avola, D., Bernardi, M., Cinque, L., Foresti, G.L., Massaroni, C.: Exploiting recurrent neural networks and leap motion controller for the recognition of sign language and semaphoric hand gestures. IEEE Trans. Multimedia 21(1), 234–245 (2019)

    Article  Google Scholar 

  6. Avola, D., Caschera, M.C., Ferri, F., Grifoni, P.: Ambiguities in sketch-based interfaces. In: 40th Annual Hawaii International Conference on System Sciences (HICSS’07), p. 290b (2007)

    Google Scholar 

  7. Avola, D., Cinque, L., De Marsico, M., Fagioli, A., Foresti, G.L.: Lietome: preliminary study on hand gestures for deception detection via fisher-LSTM. Pattern Recogn. Lett. 138, 455–461 (2020)

    Article  Google Scholar 

  8. Avola, D., et al.: Low-altitude aerial video surveillance via one-class SVM anomaly detection from textural features in UAV images. Information 13(1) (2022)

    Google Scholar 

  9. Avola, D., Cinque, L., Fagioli, A., Foresti, G.L., Fragomeni, A., Pannone, D.: 3D hand pose and shape estimation from RGB images for keypoint-based hand gesture recognition. Pattern Recogn. 129, 108762 (2022)

    Article  Google Scholar 

  10. Avola, D., et al.: Medicinal boxes recognition on a deep transfer learning augmented reality mobile application. In: Sclaroff, S., Distante, C., Leo, M., Farinella, G.M., Tombari, F. (eds.) Image Analysis and Processing - ICIAP 2022, pp. 489–499 (2022)

    Google Scholar 

  11. Avola, D., Cinque, L., Foresti, G.L., Marini, M.R.: An interactive and low-cost full body rehabilitation framework based on 3D immersive serious games. J. Biomed. Inform. 89, 81–100 (2019)

    Article  Google Scholar 

  12. Avola, D., Cinque, L., Foresti, G.L., Marini, M.R., Pannone, D.: VRheab: a fully immersive motor rehabilitation system based on recurrent neural network. Multimedia Tools Appl. 77(19), 24955–24982 (2018)

    Article  Google Scholar 

  13. Avola, D., Foresti, G.L., Cinque, L., Massaroni, C., Vitale, G., Lombardi, L.: A multipurpose autonomous robot for target recognition in unknown environments. In: 14th International Conference on Industrial Informatics (INDIN), pp. 766–771 (2016)

    Google Scholar 

  14. Bai, X., Li, C., Tian, L., Song, H.: Dynamic hand gesture recognition based on depth information. In: International Conference on Control, Automation and Information Sciences), pp. 216–221 (2018)

    Google Scholar 

  15. Boulahia, S.Y., Anquetil, E., Multon, F., Kulpa, R.: Dynamic hand gesture recognition based on 3D pattern assembled trajectories. In: Seventh International Conference on Image Processing Theory, Tools and Applications, pp. 1–6 (2017)

    Google Scholar 

  16. Boutaleb., Y., Soladie., C., Duong., N., Kacete., A., Royan., J., Seguier., R.: Efficient multi-stream temporal learning and post-fusion strategy for 3D skeleton-based hand activity recognition. In: Proceedings of the 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2021) - Volume 4: VISAPP, pp. 293–302 (2021)

    Google Scholar 

  17. De Smedt, Q., Wannous, H., Vandeborre, J.P.: Skeleton-based dynamic hand gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1206–1214 (2016)

    Google Scholar 

  18. De Smedt, Q., Wannous, H., Vandeborre, J.P.: Heterogeneous hand gesture recognition using 3d dynamic skeletal data. Comput. Vis. Image Underst. 181, 60–72 (2019)

    Article  Google Scholar 

  19. Deng, Z., Gao, Q., Ju, Z., Yu, X.: Skeleton-based multifeatures and multistream network for real-time action recognition. IEEE Sens. J. 23(7), 7397–7409 (2023)

    Article  Google Scholar 

  20. Devineau, G., Moutarde, F., Xi, W., Yang, J.: Deep learning for hand gesture recognition on skeletal data. In: 13th IEEE International Conference on Automatic Face and Gesture Recognition, pp. 106–113 (2018)

    Google Scholar 

  21. Fernando, T., Fookes, C., Gammulle, H., Denman, S., Sridharan, S.: Toward on-board panoptic segmentation of multispectral satellite images. IEEE Trans. Geosci. Remote Sens. 61, 1–12 (2023)

    Article  Google Scholar 

  22. Hou, J., Wang, G., Chen, X., Xue, J.-H., Zhu, R., Yang, H.: Spatial-temporal attention Res-TCN for skeleton-based dynamic hand gesture recognition. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 273–286. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_18

    Chapter  Google Scholar 

  23. Khokher, M.R., et al.: Early yield estimation in viticulture based on grapevine inflorescence detection and counting in videos. IEEE Access 11, 37790–37808 (2023)

    Article  Google Scholar 

  24. Lee, J., Olsman, W., Triebel, R.: Learning fluid flow visualizations from in-flight images with tufts. IEEE Robot. Autom. Lett. 8(6), 3677–3684 (2023)

    Article  Google Scholar 

  25. Li, Y., He, Z., Ye, X., He, Z., Han, K.: Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition. EURASIP J. Image Video Process. 2019(1), 78 (2019)

    Article  Google Scholar 

  26. Liu, J., Liu, Y., Wang, Y., Prinet, V., Xiang, S., Pan, C.: Decoupled representation learning for skeleton-based gesture recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5750–5759 (2020)

    Google Scholar 

  27. Lu, W., Tong, Z., Chu, J.: Dynamic hand gesture recognition with leap motion controller. IEEE Sig. Process. Lett. 23(9), 1188–1192 (2016)

    Article  Google Scholar 

  28. Luna, M.A., et al.: Spiral coverage path planning for multi-UAV photovoltaic panel inspection applications. In: International Conference on Unmanned Aircraft Systems (ICUAS), pp. 679–686 (2023)

    Google Scholar 

  29. Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12028–12037 (2019)

    Google Scholar 

  30. Nian, B., Jiang, B., Shi, H., Zhang, Y.: Local contrast attention guide network for detecting infrared small targets. IEEE Trans. Geosci. Remote Sens. 61, 1–13 (2023)

    Article  Google Scholar 

  31. Núñez, J.C., Cabido, R., Pantrigo, J.J., Montemayor, A.S., Vélez, J.F.: Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recogn. 76, 80–94 (2018)

    Article  Google Scholar 

  32. Oudah, M., Al-Naji, A., Chahl, J.: Hand gesture recognition based on computer vision: a review of techniques. J. Imaging 6(8) (2020)

    Google Scholar 

  33. Yang, K., Li, R., Qiao, P., Wang, Q., Li, D., Dou, Y.: Temporal pyramid relation network for video-based gesture recognition. In: 25th IEEE International Conference on Image Processing, pp. 3104–3108 (2018)

    Google Scholar 

  34. Yao, L., Fu, C., Li, S., Zheng, G., Ye, J.: SGDViT: saliency-guided dynamic vision transformer for UAV tracking. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3353–3359 (2023)

    Google Scholar 

  35. Zhang, W., Lin, Z., Cheng, J., Ma, C., Deng, X., Wang, H.: STA-GCN: two-stream graph convolutional network with spatial-temporal attention for hand gesture recognition. Vis. Comput. 36(10), 2433–2444 (2020)

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported by “Smart unmannEd AeRial vehiCles for Human likE monitoRing (SEARCHER)” project of the Italian Ministry of Defence (CIG: Z84333EA0D); and “A Brain Computer Interface (BCI) based System for Transferring Human Emotions inside Unmanned Aerial Vehicles (UAVs)” Sapienza Research Projects (Protocol number: RM1221816C1CF63B); and “DRrone Aerial imaGe SegmentatiOn System (DRAGONS)” (CIG: Z71379B4EA); and Departmental Strategic Plan (DSP) of the University of Udine - Interdepartmental Project on Artificial Intelligence (2020-25); and “A proactive counter-UAV system to protect army tanks and patrols in urban areas (PROACTIVE COUNTER-UAV)” project of the Italian Ministry of Defence (Number 2066/16.12.2019), and the MICS (Made in Italy - Circular and Sustainable) Extended Partnership and received funding from Next-Generation EU (Italian PNRR - M4 C2, Invest 1.3 - D.D. 1551.11-10-2022, PE00000004). CUP MICS B53C22004130001

Author information

Authors and Affiliations

  1. Department of Computer Science, Sapienza University, Via Salaria 113, 00198, Rome, Italy

    Danilo Avola, Luigi Cinque, Emad Emam, Federico Fontana, Marco Raoul Marini & Daniele Pannone

  2. Department of Mathematics, Computer Science and Physics, University of Udine, Via delle Scienze 206, 33100, Udine, Italy

    Gian Luca Foresti

Authors
  1. Danilo Avola
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luigi Cinque
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emad Emam
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Federico Fontana
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gian Luca Foresti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marco Raoul Marini
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Daniele Pannone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danilo Avola .

Editor information

Editors and Affiliations

  1. University of Udine, Udine, Italy

    Gian Luca Foresti

  2. University of Udine, Udine, Italy

    Andrea Fusiello

  3. University of York, York, UK

    Edwin Hancock

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Avola, D. et al. (2023). Hand Gesture Recognition Exploiting Handcrafted Features and LSTM. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing – ICIAP 2023. ICIAP 2023. Lecture Notes in Computer Science, vol 14233. Springer, Cham. https://doi.org/10.1007/978-3-031-43148-7_42

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43148-7_42

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43147-0

  • Online ISBN: 978-3-031-43148-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation