Skip to main content
SpringerLink
Log in

GECM: graph embedded convolution model for hand mesh reconstruction

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Hand mesh reconstruction from a single RGB image is one of the popular research topic in human understanding field with applications such as virtual/augmented reality and robot operating system. To reconstruct a hand mesh with good quality, we propose a new mesh vertex feature aggregation network module GEC. The current vertex’ features are generated by aggregating the features of the adjacent vertices according to the topological connections of the mesh vertices. Different from the traditional graph convolution structure, the GEC module circumvents the feature vectorization operation, but constructing the topological nodes with the full convolution operation. It has the advantages of avoiding destroying the spatial structure of feature maps and reducing the interference of features in the pseudo-neighborhood. Taking the GEC module as the core module, a new hand mesh reconstruction model GECM is presented. The FreiHAND dataset and the HO-3D dataset are used to evaluate the performance of the proposed GECM model. The experimental results indicate that the GECM model is superior to or on par with the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Serkan, G., Muhammet, B., Ugur, G., et al.: HandVR: a hand-gesture-based interface to a video retrieval system. Signal Image Video Process. 9(7), 1717–1726 (2015)

    Article  Google Scholar 

  2. Nurettin, Ç., Ugur, G.: A hand gesture recognition technique for human-computer interaction. J. Vis. Commun. Image Represent. 28, 97–104 (2015)

    Article  Google Scholar 

  3. Xiong, F., Zhang, B., Xiao, Y., et al.: A2j.: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image, pp. 793–802. ICCV (2019)

  4. Li, X., Zhou, Y., Sun, Y., et al.: A multi-branch hand pose estimation network with joint-wise feature extraction and fusion. Signal Process. Image Commun. 81, 115692 (2020)

    Article  Google Scholar 

  5. Mueller, F., Davis, M., Bernard, F., et al.: Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Trans. Graphics (TOG) 38(4), 1–3 (2019)

    Article  Google Scholar 

  6. Zhang, J., Jiao, J., Chen, M., et al.: 3D Hand Pose Tracking and Estimation Using Stereo Matching. arXiv preprint arXiv:1610.07214

  7. Zhang, X., Li, Q., Mo, H., et al.: End-to-End Hand Mesh Recovery from a Monocular rgb Image, pp. 2354–2364. ICCV (2019)

  8. Zhou, Y., Habermann, M., Xu, W., et al.: Monocular Real-time Hand Shape and Motion Capture Using Multi-modal Data, pp. 5346–5355. CVPR (2020)

  9. Kulon, D., Güler, R.A., Kokkinos, I., et al.: Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild, pp. 4990–5000. CVPR (2020)

  10. Ge, L., Ren, Z., Li, Y., et al.: 3D Hand Shape and Pose Estimation from a Single rgb Image, pp. 10833–10842. CVPR (2019)

  11. Choi, H., Moon, G., Lee, KM.: Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose, pp. 769–787. ECCV (2020)

  12. Bogo, F., Kanazawa, A., Lassner, C., et al.: Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image, pp. 561–578. ECCV (2016)

  13. Zhang, X., Zhang, F.: Pixel-Wise Regression.: 3d Hand Pose Estimation via Spatial-form Representation and Differentiable Decoder. arXiv preprint arXiv:1905.02085 (2019)

  14. Moon, G., Lee, K M.: I2L-MeshNet.: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation From a Single RGB Image. arXiv preprint arXiv:2008.03713 (2020)

  15. Romero, J., Tzionas, D., Black, M.: Embodied hands: modeling and capturing hands and bodies together. ACM Trans. Graphics (ToG) 36, 1–17 (2017)

    Article  Google Scholar 

  16. Loper, M., Mahmood, N., Romero, J., et al.: SMPL: a skinned multi-person linear model. ACM Trans. Graphics (TOG) 34, 1–16 (2015)

    Article  Google Scholar 

  17. Pishchulin, L., Insafutdinov, E., Tang, S., et al.: Deepcut.: Joint Subset Partition and Labeling for Multi Person Pose Estimation, pp. 4929–4937. CVPR (2016)

  18. Hasson, Y., Varol, G., Tzionas, D., et al.: Learning Joint Reconstruction of Hands and Manipulated Objects, pp. 11807–11816. CVPR (2019)

  19. Xiang, D.L., Joo, H., Sheikh, Y.: Monocular Total Capture: Posing Face, Body, and Hands in the Wild, pp. 10965–10974. CVPR (2019)

  20. Bouritsas, G., Bokhnyak, S., Ploumpis, S., et al.: Neural 3d Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation, pp. 7213–7222. ICCV (2019)

  21. Tang, X., Wang, T., Fu, C.W.: Towards Accurate Alignment in Real-time 3D Hand-mesh Reconstruction, pp. 11698–11707. ICCV (2021)

  22. Peng, H., Xian, C., Zhang, Y.: 3D hand mesh reconstruction from a monocular RGB image. Vis. Comput. 36(10), 2227–39 (2020)

    Article  Google Scholar 

  23. Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. arXiv preprint arXiv:1606.09375 (2016)

  24. Lim, I., Dielen, A., Campen, M., Kobbelt, L.: A Simple Approach to Intrinsic Correspondence Learning on Unstructured 3d Meshes. ECCV Workshops. (2018)

  25. Liu, K., Ding, R., Zou, Z., et al.: A Comprehensive Study of Weight Sharing in Graph Networks for 3D Human Pose Estimation, pp. 318–334. ECCV (2020)

  26. Wang, J., Long, X., Gao, Y., et al.: Graph-pcnn: Two Stage Human Pose Estimation with Graph Pose Refinement, pp. 492–508. ECCV (2020)

  27. Yuan, S., Garcia-Hernando, G., Stenger, B., et al.: Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals, pp. 2636–2645. CVPR (2018)

  28. Iqbal, U., Molchanov, P., Gall, T., et al.: Hand Pose Estimation via Latent 2.5 d Heatmap Regression, pp. 118–134. ECCV (2018)

  29. Wan, C., Probst, T., Van, G.L., Yao, A.: Dense 3D Regression for Hand Pose Estimation, pp. 5147–5156. CVPR (2018)

  30. Iskakov, K., Burkov, E., Lempitsky, V., Malkov, Y.: Learnable Triangulation of Human Pose, pp. 7718–7727. ICCV (2019)

  31. Iqbal, U., Molchanov, P., Kautz, J.: Weakly-Supervised 3D Human Pose Learning via Multi-View Images in the Wild. CVPR (2020)

  32. Miki, D., Abe, S., Chen, S., et al.: Robust human pose estimation from distorted wide-angle images through iterative search of transformation parameters. SIViP 14, 693–700 (2020)

    Article  Google Scholar 

  33. Zhang, F., Zhu, X., Dai, H., et al.: Distribution-Aware Coordinate Representation for Human Pose Estimation. CVPR (2020)

  34. Tekin, B., Bogo, F., Pollefeys, M.: H+O: Unified Egocentric Recognition of 3D Hand-object Poses and Interactions, pp. 4511–4520. CVPR (2019)

  35. Zimmermann, C., Ceylan, D., Yang, J., et al.: Freihand: a Dataset for Markerless Capture of Hand Pose and Shape from Single rgb Images, pp. 813–822. ICCV (2019)

  36. Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: A Method for 3D Annotation of Hand and Object Poses, pp. 3196–3206. CVPR (2020)

  37. Boukhayma, A., Bem, R., Rodrigo, D., et al.: 3D Hand Shape and Pose from Images in the Wild, pp. 10843–10852. CVPR (2019)

  38. Li, M., Gao, Y., Sang, N.: Exploiting Learnable Joint Groups for Hand Pose Estimation. arXiv preprint arXiv:2012.09496 (2020)

Download references

Author information

Authors and Affiliations

  1. Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian, China

    Xuefeng Li, Xiangbo Lin & Yi Sun

Authors
  1. Xuefeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiangbo Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiangbo Lin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported by the National Natural Science Foundation of China (Grant No. 61873046 and No. U-1708263)

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 1152 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Lin, X. & Sun, Y. GECM: graph embedded convolution model for hand mesh reconstruction. SIViP 17, 715–723 (2023). https://doi.org/10.1007/s11760-022-02279-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02279-z

Keywords

Search

Navigation