Skip to main content
SpringerLink
Log in

VRGestures: Controller and Hand Gesture Datasets for Virtual Reality

  • Conference paper
  • First Online:
Advances in Computer Graphics (CGI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14497))

Included in the following conference series:

  • 214 Accesses

Abstract

Gesture Recognition is attracting increasingly more attention over the years and has been adopted in main applications in the real world and the Virtual one. New generation Virtual Reality (VR) headsets like the Meta Quest 2 support hand tracking very efficiently and are challenging the research community for more breakthrough discoveries in Hand Gesture Recognition. What has also been quietly improved recently are the VR controllers, which have become wireless and also more practical to use. However, when it comes to VR gesture datasets, and especially controller gesture datasets there are limited data available. Point-And-Click methods are widely accepted, which is why gestures are being neglected, combined with the shortage of available datasets. To address this gap we provide two datasets one with controller gestures and one with hand gestures, capable of recording with either controller or hand and even with both hands simultaneously. We created two VR applications to record for controllers and hands the position and the orientation and also each timestamp that we record data. Then we trained off-the-shelf time series classifiers to test our data, export metrics, and compare different subsets of our datasets between them. Hand gesture recognition is far more complicated than controller gesture recognition as we take almost thrice input and the difference is being analyzed and discussed with findings and metrics. The datasets are available online https://doi.org/10.5281/zenodo.8027807.

Centre for Research and Technology Hellas - Information Technologies Institute.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    meta.com/quest/products/quest-2.

  2. 2.

    store.steampowered.com/app/250820/SteamVR.

  3. 3.

    assetstore.unity.com/packages/tools/integration/oculus-integration-82022.

  4. 4.

    developer.oculus.com/downloads/package/oculus-developer-hub-win.

  5. 5.

    docs.unity3d.com/Manual/index.html.

  6. 6.

    mlflow.org.

References

  1. Arendttorp, E.M.N., Rodil, K., Winschiers-Theophilus, H., Magoath, C.: Overcoming legacy bias: re-designing gesture interactions in virtual reality with a san community in Namibia. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI 2022. Association for Computing Machinery, New York (2022)

    Google Scholar 

  2. Bhaumik, G., Verma, M., Govil, M.C., Vipparthi, S.K.: HyFiNet: hybrid feature attention network for hand gesture recognition. Multimedia Tools Appl. 82(4), 4863–4882 (2023)

    Article  Google Scholar 

  3. D’Eusanio, A., Simoni, A., Pini, S., Borghi, G., Vezzani, R., Cucchiara, R.: A transformer-based network for dynamic hand gesture recognition. In: 2020 International Conference on 3D Vision (3DV), pp. 623–632 (2020). https://doi.org/10.1109/3DV50981.2020.00072

  4. Fronteddu, G., Porcu, S., Floris, A., Atzori, L.: A dynamic hand gesture recognition dataset for human-computer interfaces. Comput. Netw. 205, 108781 (2022)

    Article  Google Scholar 

  5. Gnanapriya, S., Rahimunnisa, K.: A hybrid deep learning model for real time hand gestures recognition. Intell. Autom. Soft Comput. 36(1) (2023)

    Google Scholar 

  6. Guo, L., Lu, Z., Yao, L.: Human-machine interaction sensing technology based on hand gesture recognition: a review. IEEE Trans. Hum.-Mach. Syst. 51(4), 300–309 (2021). https://doi.org/10.1109/THMS.2021.3086003

    Article  Google Scholar 

  7. Gupta, P., Kautz, K., et al.: Online detection and classification of dynamic hand gestures with recurrent 3D convolutional neural networks. In: CVPR, vol. 1, p. 3 (2016)

    Google Scholar 

  8. Huang, Y.J., Liu, K.Y., Lee, S.S., Yeh, I.C.: Evaluation of a hybrid of hand gesture and controller inputs in virtual reality. Int. J. Hum.-Comput. Interact. 37(2), 169–180 (2021)

    Article  Google Scholar 

  9. Jiang, S., Kang, P., Song, X., Lo, B.P., Shull, P.B.: Emerging wearable interfaces and algorithms for hand gesture recognition: a survey. IEEE Rev. Biomed. Eng. 15, 85–102 (2022). https://doi.org/10.1109/RBME.2021.3078190

    Article  Google Scholar 

  10. Jiang, X., Zhu, L., Liu, J., Song, A.: A slam-based 6dof controller with smooth auto-calibration for virtual reality. Vis. Comput. 1–14 (2022)

    Google Scholar 

  11. Köpüklü, O., Gunduz, A., Kose, N., Rigoll, G.: Real-time hand gesture detection and classification using convolutional neural networks. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–8. IEEE (2019)

    Google Scholar 

  12. Kurakin, A., Zhang, Z., Liu, Z.: A real time system for dynamic hand gesture recognition with a depth sensor. In: 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp. 1975–1979. IEEE (2012)

    Google Scholar 

  13. Lai, H.Y., Lai, H.J.: Real-time dynamic hand gesture recognition. In: 2014 International Symposium on Computer, Consumer and Control, pp. 658–661 (2014)

    Google Scholar 

  14. Li, J., Wei, L., Wen, Y., Liu, X., Wang, H.: An approach to continuous hand movement recognition using sEMG based on features fusion. Vis. Comput. 39(5), 2065–2079 (2023)

    Article  Google Scholar 

  15. Löning, M., Bagnall, A., Ganesh, S., Kazakov, V., Lines, J., Király, F.J.: sktime: a unified interface for machine learning with time series. arXiv preprint arXiv:1909.07872 (2019)

  16. Löning, M., et al.: sktime/sktime: v0.13.4 (2022)

    Google Scholar 

  17. Materzynska, J., Berger, G., Bax, I., Memisevic, R.: The jester dataset: a large-scale video dataset of human gestures. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2019)

    Google Scholar 

  18. Mujahid, A., et al.: Real-time hand gesture recognition based on deep learning YOLOv3 model. Appl. Sci. 11(9), 4164 (2021). https://doi.org/10.3390/app11094164. https://www.mdpi.com/2076-3417/11/9/4164

  19. Nguyen, X.S., Brun, L., Lezoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  20. Ntovas, A., Lazaridis, L., Papadimitriou, A., Psaltis, A., Axenopoulos, A., Daras, P.: Data-driven haptic feedback utilizing an object manipulation data-set. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2021)

    Google Scholar 

  21. Núñez-Fernández, D.: Development of a hand gesture based control interface using deep learning. In: Lossio-Ventura, J.A., Condori-Fernandez, N., Valverde-Rebaza, J.C. (eds.) SIMBig 2019. CCIS, vol. 1070, pp. 143–150. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46140-9_14

    Chapter  Google Scholar 

  22. Rahimian, E., Zabihi, S., Asif, A., Farina, D., Atashzar, S.F., Mohammadi, A.: FS-HGR: few-shot learning for hand gesture recognition via electromyography. IEEE Trans. Neural Syst. Rehabil. Eng. 29, 1004–1015 (2021). https://doi.org/10.1109/TNSRE.2021.3077413

    Article  Google Scholar 

  23. Rapin, J., Teytaud, O.: Nevergrad - A gradient-free optimization platform (2018). https://GitHub.com/FacebookResearch/Nevergrad

  24. Schäfer, A., Reis, G., Stricker, D.: Anygesture: arbitrary one-handed gestures for augmented, virtual, and mixed reality applications. Appl. Sci. 12(4), 1888 (2022)

    Article  Google Scholar 

  25. Shi, L., Zhang, Y., Cheng, J., Lu, H.: Skeleton-based action recognition with directed graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  26. Terreran, M., Lazzaretto, M., Ghidoni, S.: Skeleton-based action and gesture recognition for human-robot collaboration. In: Petrovic, I., Menegatti, E., Marković, I. (eds.) IAS 2017, vol. 577, pp. 29–45. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-22216-0_3

    Chapter  Google Scholar 

  27. Toro-Ossaba, A., et al.: LSTM recurrent neural network for hand gesture recognition using EMG signals. Appl. Sci. 12(19), 9700 (2022)

    Article  Google Scholar 

  28. Wang, Y., Hu, Z., Yao, S., Liu, H.: Using visual feedback to improve hand movement accuracy in confined-occluded spaces in virtual reality. Vis. Comput. 39(4), 1485–1501 (2023)

    Google Scholar 

  29. Yadan, O.: Hydra - a framework for elegantly configuring complex applications. Github (2019). https://github.com/facebookresearch/hydra

  30. Zabihi, S., Rahimian, E., Asif, A., Mohammadi, A.: TraHGR: transformer for hand gesture recognition via electromyography (2022)

    Google Scholar 

  31. Zeghoud, S., et al.: Real-time spatial normalization for dynamic gesture classification. Vis. Comput. 1345–1357 (2022)

    Google Scholar 

  32. Zou, Y., Cheng, L.: A transfer learning model for gesture recognition based on the deep features extracted by CNN. IEEE Trans. Artif. Intell. 2(5), 447–458 (2021). https://doi.org/10.1109/TAI.2021.3098253

    Article  Google Scholar 

Download references

Acknowledgements

We also acknowledge financial support by the H2020 EC project INFINITY under contract 883293. We thank Athanasios Ntovas for his help with the experiment setup, and the subjects recorded in our datasets for their participation.

Author information

Authors and Affiliations

  1. Centre for Research and Technology Hellas, Thessaloniki, Greece

    Georgios Papadopoulos, Alexandros Doumanoglou & Dimitrios Zarpalas

Authors
  1. Georgios Papadopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandros Doumanoglou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dimitrios Zarpalas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georgios Papadopoulos .

Editor information

Editors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, China

    Bin Sheng

  2. Shanghai Jiao Tong University, Shanghai, China

    Lei Bi

  3. University of Sydney, Sydney, NSW, Australia

    Jinman Kim

  4. MIRALab-CUI, University of Geneva, Carouge, Geneve, Switzerland

    Nadia Magnenat-Thalmann

  5. Swiss Federal Institute of Technology, Lausanne, Switzerland

    Daniel Thalmann

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Papadopoulos, G., Doumanoglou, A., Zarpalas, D. (2024). VRGestures: Controller and Hand Gesture Datasets for Virtual Reality. In: Sheng, B., Bi, L., Kim, J., Magnenat-Thalmann, N., Thalmann, D. (eds) Advances in Computer Graphics. CGI 2023. Lecture Notes in Computer Science, vol 14497. Springer, Cham. https://doi.org/10.1007/978-3-031-50075-6_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-50075-6_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-50074-9

  • Online ISBN: 978-3-031-50075-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation