Abstract
Augmented Reality (AR) has gained a lot of attraction in the recent past. Arguably, the most important tool that can make AR a household gadget is its interaction with the user. This may lead to two possible interaction methodologies: (i) Using an extra device for interaction; (ii) Using human hands as interaction. Former is probably the easy method, but it may increase the cost of the AR device, limiting its target users. Therefore, hand gestures are a feasible and efficient mode of interaction. However, for accurate and pleasant interaction, the AR device should be capable of hand gesture understanding. In this direction, we propose a hand gesture classification method, based on Convolutional Neural Networks (CNNs) that takes advantage of the pre-trained network weights for faster and efficient training, which also helps improve the quality of gesture classification. Moreover, the proposed approach takes advantage of hand detection for background elimination and efficient gesture recognition. The proposed approach is evaluated on the Hand gesture classification task for three datasets that differ in terms of the number of data samples, amount of gestures, and data quality. The obtained results show that our method outperforms state-of-the-art methods in most of the experimentation cases.
This work is partially supported by Sidia institute of science and technology, and Samsung Eletrônica da Amazônia Ltda, under the auspice of the Brazilian informatics law no 8.387/91.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Agrawal, M., Ainapure, R., Agrawal, S., Bhosale, S., Desai, S.: Models for hand gesture recognition using deep learning. In: 2020 IEEE 5th International Conference on Computing Communication and Automation (ICCCA), pp. 589–594 (2020). https://doi.org/10.1109/ICCCA49541.2020.9250846
Bao, P., Maqueda, A.I., del Blanco, C.R., García, N.: Tiny hand gesture recognition without localization via a deep convolutional network. IEEE Trans. Consum. Electron. 63(3), 251–257 (2017)
Devineau, G., Moutarde, F., Xi, W., Yang, J.: Deep learning for hand gesture recognition on skeletal data. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 106–113. IEEE (2018)
Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
Grzejszczak, T., Kawulok, M., Galuszka, A.: Hand landmarks detection and localization in color images. Multimedia Tools Appl. 75(23), 16363–16387 (2015). https://doi.org/10.1007/s11042-015-2934-5
Jiang, F., Wu, S., Yang, G., Zhao, D., Kung, S.: Independent hand gesture recognition with Kinect. SIViP 8(1), 163–172 (2014)
John, V., Boyali, A., Mita, S., Imanishi, M., Sanma, N.: Deep learning-based fast hand gesture recognition using representative frames. In: 2016 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–8 (2016). https://doi.org/10.1109/DICTA.2016.7797030
Kawulok, M., Kawulok, J., Nalepa, J., Papiez, M.: Skin detection using spatial analysis with adaptive seed. In: 2013 IEEE International Conference on Image Processing, pp. 3720–3724 (2013). https://doi.org/10.1109/ICIP.2013.6738767
Kawulok, M.: Database for hand gesture recognition (2013). http://sun.aei.polsl.pl/~mkawulok/gestures/. Accessed 30 Oct 2020
Kawulok, M., Kawulok, J., Nalepa, J., Smolka, B.: Self-adaptive algorithm for segmenting skin regions. EURASIP J. Adv. Sig. Process. 2014(1), 1–22 (2014). https://doi.org/10.1186/1687-6180-2014-170
Khurshid, A., Cleger, S., Grunitzki, R.: A scene classification approach for augmented reality devices. In: Stephanidis, C., Chen, J.Y.C., Fragomeni, G. (eds.) HCII 2020. LNCS, vol. 12428, pp. 164–177. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59990-4_14
Khurshid, A., Tamayo, S.C., Fernandes, E., Gadelha, M.R., Teofilo, M.: A robust and real-time face anti-spoofing method based on texture feature analysis. In: International Conference on Human-Computer Interaction. pp. 484–496. Springer (2019). https://doi.org/10.1007/978-3-030-30033-3_37
Kourbane, I., Genc, Y.: Skeleton-aware multi-scale heatmap regression for 2D hand pose estimation. arXiv preprint arXiv:2105.10904 (2021)
Li, G., et al.: Hand gesture recognition based on convolution neural network. Clust. Comput. 22(2), 2719–2729 (2017). https://doi.org/10.1007/s10586-017-1435-x
Memo, A., Minto, L., Zanuttigh, P.: Exploiting Silhouette descriptors and synthetic data for hand gesture recognition. In: Giachetti, A., Biasotti, S., Tarini, M. (eds.) Smart Tools and Apps for Graphics - Eurographics Italian Chapter Conference. The Eurographics Association (2015). https://doi.org/10.2312/stag.20151288
Memo, A., Zanuttigh, P.: Head-mounted gesture controlled interface for human-computer interaction. Multimedia Tools Appl. 77(1), 27–53 (2016). https://doi.org/10.1007/s11042-016-4223-3
Nalepa, J., Kawulok, M.: Fast and accurate hand shape classification. In: Kozielski, S., Mrozek, D., Kasprowski, P., Małysiak-Mrozek, B., Kostrzewa, D. (eds.) BDAS 2014. CCIS, vol. 424, pp. 364–373. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06932-6_35
Narasimhaswamy, S., Wei, Z., Wang, Y., Zhang, J., Hoai, M.: Contextual attention for hand detection in the wild. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9567–9576 (2019)
organization, T.: Model Optimization. https://www.tensorflow.org/lite/performance/model_optimization. Accessed 01 Dec 2001
Oudah, M., Al-Naji, A., Chahl, J.: Hand gesture recognition based on computer vision: a review of techniques. J. Imag. 6(8), 73 (2020)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28, 91–99 (2015)
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Szegedy, C., et al.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015). https://doi.org/10.1109/CVPR.2015.7298594
Teeparthi, S., Jatla, V., Pattichis, M.S., Celedón-Pattichis, S., LópezLeiva, C.: Fast hand detection in collaborative learning environments. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds.) CAIP 2021. LNCS, vol. 13052, pp. 445–454. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89128-2_43
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Khurshid, A., Grunitzki, R., Estrada Leyva, R.G., Marinho, F., Matthaus Maia Souto Orlando, B. (2022). Hand Gesture Recognition for User Interaction in Augmented Reality (AR) Experience. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality: Design and Development. HCII 2022. Lecture Notes in Computer Science, vol 13317. Springer, Cham. https://doi.org/10.1007/978-3-031-05939-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-031-05939-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05938-4
Online ISBN: 978-3-031-05939-1
eBook Packages: Computer ScienceComputer Science (R0)