Skip to main content
SpringerLink
Log in

SORCNet: robust non-rigid shape correspondence with enhanced descriptors by Shared Optimized Res-CapsuleNet

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

3D non-rigid shape correspondence, as an important research topic in 3D shape analysis, is useful but challenging in computer graphics, computer vision, and pattern recognition. Despite recent success of several deep neural networks for shape correspondence, those networks cannot achieve robust results on non-rigid objects due to their local deformation complexity. This paper presents a novel and efficient shape correspondence network—Shared Optimized Res-CapsuleNet (SORCNet)—that learns point features based on enhanced descriptors to solve dense correspondence between non-rigid 3D shapes. To further improve the iterative efficiency and accuracy of the model, we design an optimized residual network structure, based on the stochastic gradient descent algorithm with momentum and weight decay (SGDW). Moreover, as the convolutional neural network does not perform well when the shape has directional variance, we present a shared capsule network structure with dual routings, which correlates the hierarchical geometric relationships of the semantic parts well to extract more informative point features. We proved that the primary capsule has a greater influence on feature extraction than the routing and decoder parts. The entire network, SORCNet, is integrated and trained/tested by taking the descriptors and Laplacian eigenbases of two shapes as input. The experiments on public datasets, such as FAUST, SCAPE, TOSCA and KIDS, demonstrate the better effectiveness, accuracy, and adaptability of our method than those of the state of the art in 3D shape correspondence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Sansoni, G., Trebeschi, M., Docchio, F.: State-of-the-art and applications of 3D imaging sensors in industry, cultural heritage, medicine, and criminal investigation. Sensors 9(1), 568–601 (2009)

    Article  Google Scholar 

  2. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Natl. Acad. Sci. 103(5), 1168–1172 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  3. Kim, V.G., Lipman, Y., Funkhouser, T.: Blended intrinsic maps. ACM Trans. Graphics 30(4), 1–12 (2011)

    Article  Google Scholar 

  4. Ovsjanikov, M., Ben-Chen, M., Solomon, J., Butscher, A., Guibas, L.: Functional maps: a flexible representation of maps between shapes. ACM Trans. Graphics 31(4), 1–11 (2012)

    Article  Google Scholar 

  5. Aubry, M., Schlickewei, U., Cremers, D.: The wave kernel signature: a quantum mechanical approach to shape analysis. In: Proceedings of IEEE International Conference on Computer Vision Workshops, pp. 1626–1633 (2011)

  6. Li, P., Ma, H., Ming, A.: A non-rigid 3D model retrieval method based on scale-invariant heat kernel signature features. Multimedia Tools Appl. 76(7), 10207–10230 (2017)

    Article  Google Scholar 

  7. Bronstein, M.M., Bronstein, A.M., Kimmel, R., Yavneh, I.: Multigrid multidimensional scaling. Numer. Linear Algebra Appl. 13(2–3), 149–171 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  8. Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: diffusion maps. Proc. Natl. Acad. Sci. 102(21), 7426–7431 (2005)

    Article  MATH  Google Scholar 

  9. Lipman, Y., Daubechies, I.: Conformal wasserstein distances: comparing surfaces in polynomial time. Adv. Math. 227(3), 1047–1077 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  10. Mateus, D., Horaud, R., Knossow, D., Cuzzolin, F., Boyer, E.: Articulated shape matching using Laplacian eigenfunctions and unsupervised point registration. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  11. Shtern, A., Kimmel, R.: Matching the LBO eigenspace of non-rigid shapes via high order statistics. Axioms 3(3), 300–319 (2014)

    Article  MATH  Google Scholar 

  12. Corman, É., Ovsjanikov, M., Chambolle, A.: Supervised descriptor learning for non-rigid shape matching. In: European Conference on Computer Vision, pp. 283–298. Springer (2014)

  13. Ginzburg, D., Raviv, D.: Cyclic functional mapping: self-supervised correspondence between non-isometric deformable shapes. arXiv preprint arXiv:1912.01249 (2019)

  14. Litany, O., Remez, T., Rodolà, E., Bronstein, A., Bronstein, M.: Deep functional maps: structured prediction for dense shape correspondence. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5659–5667 (2017)

  15. Tombari, F., Salti, S., Di Stefano, L.: Unique signatures of histograms for local surface description. In: Proceedings of European conference on computer vision, pp. 356–369. Springer (2010)

  16. Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. In: Computer Graphics Forum, vol. 28, pp. 1383–1392 (2009)

  17. Litman, R., Bronstein, A.M.: Learning spectral descriptors for deformable shape correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 36(1), 171–180 (2013)

    Article  Google Scholar 

  18. Dai, G., Xie, J., Zhu, F., Fang, Y.: Learning a discriminative deformation-invariant 3d shape descriptor via many-to-one encoder. Pattern Recognit. Lett. 83, 330–338 (2016)

    Article  Google Scholar 

  19. Papadakis, P., Pratikakis, I., Theoharis, T., Perantonis, S.: Panorama: a 3D shape descriptor based on panoramic views for unsupervised 3d object retrieval. Int. J. Comput. Vision 89(2–3), 177–192 (2010)

    Article  Google Scholar 

  20. Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Proceedings of Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)

  21. Duarte, K., Rawat, Y., Shah, M.: Videocapsulenet: A simplified network for action detection. In: Proceedings of Advances in Neural Information Processing Systems, pp. 7610–7619 (2018)

  22. Lin, A., Li, J., Ma, Z.: On learning and learned representation with dynamic routing in capsule networks. arXiv preprint arXiv:1810.040412(7) (2018)

  23. Zhao, Y., Birdal, T., Deng, H., Tombari, F.: 3D point capsule networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1009–1018 (2019)

  24. Biasotti, S., Cerri, A., Bronstein, A., Bronstein, M.: Recent trends, applications, and perspectives in 3d shape similarity assessment. In: Computer Graphics Forum, vol. 35, pp. 87–119. Wiley Online Library (2016)

  25. Tam, G.K., Cheng, Z.Q., Lai, Y.K., Langbein, F.C., Liu, Y., Marshall, D., Martin, R.R., Sun, X.F., Rosin, P.L.: Registration of 3D point clouds and meshes: a survey from rigid to nonrigid. IEEE Trans. Visual Comput. Graphics 19(7), 1199–1217 (2012)

    Article  Google Scholar 

  26. Rodolà, E., Rota Bulò, S., Windheuser, T., Vestner, M., Cremers, D.: Dense non-rigid shape correspondence using random forests. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4177–4184 (2014)

  27. Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs and manifolds using mixture model cnns. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5115–5124 (2017)

  28. Wang, H., Guo, J., Yan, D.M., Quan, W., Zhang, X.: Learning 3d keypoint descriptors for non-rigid shape matching. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19 (2018)

  29. Fey, M., Lenssen, J.E., Weichert, F., Müller, H.: Splinecnn: fast geometric deep learning with continuous b-spline kernels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 869–877 (2018)

  30. Ovsjanikov, M., Corman, E., Bronstein, M., Rodolà, E., Ben-Chen, M., Guibas, L., Chazal, F., Bronstein, A.: Computing and processing correspondences with functional maps. In: SIGGRAPH ASIA 2016 Courses, pp. 1–60 (2016)

  31. Maron, H., Dym, N., Kezurer, I., Kovalsky, S., Lipman, Y.: Point registration via efficient convex relaxation. ACM Trans. Graphics 35(4), 1–12 (2016)

    Article  Google Scholar 

  32. Halimi, O., Litany, O., Rodolà, E., Bronstein, A., Kimmel, R.: Self-supervised learning of dense shape correspondence. arXiv preprint arXiv:1812.02415 (2018)

  33. Roufosse, J.M., Sharma, A., Ovsjanikov, M.: Unsupervised deep learning for structured shape matching. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1617–1627 (2019)

  34. Hinton, G.E., Krizhevsky, A., Wang, S.D.: Transforming auto-encoders. In: Proceedings of International Conference on Artificial Neural Networks, pp. 44–51. Springer (2011)

  35. Hinton, G.E., Sabour, S., Frosst, N.: Matrix capsules with em routing. In: International Conference on Learning Representations (2018)

  36. Chen, Z., Crandall, D.: Generalized capsule networks with trainable routing procedure. arXiv preprint arXiv:1808.08692 (2018)

  37. Cheraghian, A., Petersson, L.: 3DCapsule: extending the capsule architecture to classify 3D point clouds. In: Proceedings of IEEE Winter Conference on Applications of Computer Vision, pp. 1194–1202 (2019)

  38. Leung, F.H.F., Lam, H.K., Ling, S.H., Tam, P.K.S.: Tuning of the structure and parameters of a neural network using an improved genetic algorithm. IEEE Trans. Neural Netw. 14(1), 79–88 (2003)

    Article  Google Scholar 

  39. Domhan, T., Springenberg, J.T., Hutter, F.: Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)

  40. Ma, L., Khorasani, K.: A new strategy for adaptively constructing multilayer feedforward neural networks. Neurocomputing 51, 361–385 (2003)

    Article  Google Scholar 

  41. Cortes, C., Gonzalvo, X., Kuznetsov, V., Mohri, M., Yang, S.: Adanet: adaptive structural learning of artificial neural networks. In: International Conference on Machine Learning, pp. 874–883 (2017)

  42. Li, H., Yang, Y., Chen, D., Lin, Z.: Optimization algorithm inspired deep neural network structure design. arXiv preprint arXiv:1810.01638 (2018)

  43. Mangasarian, O.L.: Nonlinear Programming. SIAM (1994)

  44. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

  45. Umehara, M., Yamada, K.: Differential geometry of curves and surfaces (2017)

  46. Jia, B., Huang, Q.: De-capsnet: a diverse enhanced capsule network with disperse dynamic routing. Appl. Sci. 10(3), 884 (2020)

    Article  Google Scholar 

  47. Tay, Y., Bahri, D., Metzler, D., Juan, D.C., Zhao, Z., Zheng, C.: Synthesizer: Rethinking self-attention in transformer models. arXiv preprint arXiv:2005.00743 (2020)

  48. Bogo, F., Romero, J., Loper, M., Black, M.J.: FAUST: Dataset and evaluation for 3D mesh registration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3794–3801 (2014)

  49. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)

  50. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  51. Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., Davis, J.: SCAPE: shape completion and animation of people. In: ACM SIGGRAPH, pp. 408–416 (2005)

  52. Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-rigid Shapes. Springer (2008)

  53. Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3D-CODED: 3D correspondences by deep deformation. In: Proceedings of the European Conference on Computer Vision, pp. 230–246 (2018)

  54. Marin, R., Melzi, S., Rodolà, E., Castellani, U.: Farm: Functional automatic registration method for 3d human bodies. In: Computer Graphics Forum, vol. 39, pp. 160–173. Wiley Online Library (2020)

  55. Halimi, O., Litany, O., Rodola, E., Bronstein, A.M., Kimmel, R.: Unsupervised learning of dense shape correspondence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4370–4379 (2019)

  56. Zuffi, S., Black, M.J.: The stitched puppet: a graphical model of 3D human shape and pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3537–3546 (2015)

  57. Chen, Q., Koltun, V.: Robust nonrigid registration by convex optimization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2039–2047 (2015)

  58. Wang, D., Liu, Q.: An optimization view on dynamic routing between capsules (2018)

  59. Xi, E., Bing, S., Jin, Y.: Capsule network performance on complex data. arXiv preprint arXiv:1712.03480 (2017)

Download references

Acknowledgements

This work was partially supported by the grants: NSFC 61972353, NSF IIS-1816511 and OAC-1910469.

Author information

Authors and Affiliations

  1. Beijing Key Lab of Petroleum Data Mining, Department of Computer Science and Technology, China University of Petroleum, Beijing, China

    Yuanfeng Lian & Dingru Gu

  2. Wayne State University, Detroit, MI, USA

    Jing Hua

Authors
  1. Yuanfeng Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dingru Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lian, Y., Gu, D. & Hua, J. SORCNet: robust non-rigid shape correspondence with enhanced descriptors by Shared Optimized Res-CapsuleNet. Vis Comput 39, 749–763 (2023). https://doi.org/10.1007/s00371-021-02372-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02372-3

Keywords

Search

Navigation