Abstract
Recent works on skeleton-based hand gesture recognition proposed graph-based representation of hand skeleton. In this paper, we propose a new attention-based directed rotation graph fusion method for skeleton-based hand gesture recognition. First, we present a novel double-stream directed rotation graph feature to jointly capture the spatiotemporal dynamics and hand structural information. We utilize the bone direction and rotation information to model the kinematic dependency and relative geometry in hand skeleton. The spatial stream employs a spatial directed rotation graph (SDRG) containing joint position and rotation information to model spatial dependencies between joints. The temporal stream employs a temporal directed rotation graph (TDRG), containing joint displacement and rotation between frames to model temporal dependencies. We design a new attention-based double-stream fusion framework ADF-DGNN, in which the two streams are fed into two directed graph neural networks (DGNNs), and the encoded graphs are concatenated and fused by a fusion module with multi-head attention to generate expressive and discriminative characteristics for identifying hand gesture. The experiments on DHG-14/28 dataset demonstrate the effectiveness of the components of the proposed method and its superiority compared with state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. 43(1), 1–54 (2012). https://doi.org/10.1007/s10462-012-9356-9
Devanne, M., et al.: Human action recognition by shape analysis of motion trajectories on Riemannian manifold. IEEE Trans. Cybern. 45(7), 1340–1352 (2014)
Ohn-Bar, E., Trivedi, M.: Joint angles similarities and HOG2 for action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 465–470. IEEE (2013)
De Smedt, Q., Wannous, H., Vandeborre, J.P.: Skeleton-based dynamic hand gesture recognition. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1–9. IEEE (2016)
De Smedt, Q., Wannous, H., Vandeborre, J.P.: SHREC’17 track: 3D hand gesture recognition using a depth and skeletal dataset. In: Eurographics Workshop on 3D Object Retrieval (2017)
Chen, X., et al. Motion feature augmented recurrent neural network for skeleton-based dynamic hand gesture recognition. In: IEEE International Conference on Image Processing. IEEE (2017)
Ma, C., Wang, A., Chen, G., Xu, C.: Hand joints-based gesture recognition for noisy dataset using nested interval unscented Kalman filter with LSTM network. Vis. Comput. 34(6–8), 1053–1063 (2018). https://doi.org/10.1007/s00371-018-1556-0
Nunez, J.C., et al.: Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recogn. 76, 80–94 (2018)
Weng, J., Liu, M., Jiang, X., Yuan, J.: Deformable pose traversal convolution for 3D action and gesture recognition. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 142–157. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_9
Hou, J., Wang, G., Chen, X., Xue, J.-H., Zhu, R., Yang, H.: Spatial-temporal attention res-TCN for skeleton-based dynamic hand gesture recognition. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11134, pp. 273–286. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_18
Devineau, G., et al.: Deep learning for hand gesture recognition on skeletal data. In: IEEE International Conference on Automatic Face and Gesture Recognition. IEEE (2018)
Ma, C., et al.: Skeleton-based dynamic hand gesture recognition using an enhanced network with one-shot learning. Appl. Sci. 10(11), 3680 (2020)
Do, N.T., et al.: Robust hand shape features for dynamic hand gesture recognition using multi-level feature LSTM. Appl. Sci. 10(18), 6293 (2020)
Yan, S., Xiong, Y., et al.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: 32th AAAI Conference on Artificial Intelligence (2018)
Wen, Y.H., et al.: Graph CNNs with motif and variable temporal block for skeleton-based action recognition. In: 33th AAAI Conference on Artificial Intelligence, vol. 33, pp. 8989–8996. Association for Computing Machinery, New York (2019)
Shi, L., et al.: Skeleton-based action recognition with directed graph neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7912–7921. IEEE (2019)
Li, Y., et al.: Spatial temporal graph convolutional networks for skeleton-based dynamic hand gesture recognition. EURASIP J. Image Video Process. 1, 1–7 (2019). https://doi.org/10.1186/s13640-019-0476-x
Chen, Y., et al.: Construct dynamic graphs for hand gesture recognition via spatial-temporal attention. arXiv, arXiv:1907.08871 (2019)
Shi, L., et al.: Nonlocal graph convolutional networks for skeleton-based action recognition. arXiv, arXiv:1805.07694 (2018)
Slabaugh, G.G.: Computing Euler angles from a rotation matrix, pp. 39–63 (1999). Accessed 6 Aug 2000
Vemulapalli, R., et al.: Rolling rotations for recognizing human actions from 3D skeletal data. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4471–4479. IEEE (2016)
Shi, L., et al.: Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 12026–12035. IEEE (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Li, C., et al.: Skeleton-based action recognition using LSTM and CNN. In: IEEE International Conference on Multimedia and Expo Workshops, pp. 585–590. IEEE (2017)
Xie, N., et al.: Sequential gesture learning for continuous labanotation generation based on the fusion of graph neural networks. IEEE Trans. Circ. Syst. Video Technol. 32, 3722–3734 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xie, N., Yu, W., Yang, L., Guo, M., Li, J. (2022). Attention-Based Fusion of Directed Rotation Graphs for Skeleton-Based Dynamic Hand Gesture Recognition. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13534. Springer, Cham. https://doi.org/10.1007/978-3-031-18907-4_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-18907-4_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18906-7
Online ISBN: 978-3-031-18907-4
eBook Packages: Computer ScienceComputer Science (R0)