skip to main content
research-article

Convolutional neural networks on surfaces via seamless toric covers

Published: 20 July 2017 Publication History

Abstract

The recent success of convolutional neural networks (CNNs) for image processing tasks is inspiring research efforts attempting to achieve similar success for geometric tasks. One of the main challenges in applying CNNs to surfaces is defining a natural convolution operator on surfaces.
In this paper we present a method for applying deep learning to sphere-type shapes using a global seamless parameterization to a planar flat-torus, for which the convolution operator is well defined. As a result, the standard deep learning framework can be readily applied for learning semantic, high-level properties of the shape. An indication of our success in bridging the gap between images and surfaces is the fact that our algorithm succeeds in learning semantic information from an input of raw low-dimensional feature vectors.
We demonstrate the usefulness of our approach by presenting two applications: human body segmentation, and automatic landmark detection on anatomical surfaces. We show that our algorithm compares favorably with competing geometric deep-learning algorithms for segmentation tasks, and is able to produce meaningful correspondences on anatomical surfaces where hand-crafted features are bound to fail.

Supplementary Material

MP4 File (papers-0164.mp4)

References

[1]
2016. Adobe Fuse 3D Characters. https://www.mixamo.com. (2016). Accessed: 2016-10-15.
[2]
2016. Yobi3d - free 3d model search engine. https://www.yobi3d.com. (2016). Accessed: 2016-10-15.
[3]
Noam Aigerman and Yaron Lipman. 2015. Orbifold tutte embeddings. ACM Trans. Graph 34, 6 (2015), 190.
[4]
Noam Aigerman and Yaron Lipman. 2016. Hyperbolic orbifold tutte embeddings. ACM Transactions on Graphics (TOG) 35, 6 (2016), 217.
[5]
Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: shape completion and animation of people. In ACM Transactions on Graphics (TOG), Vol. 24. ACM, 408--416.
[6]
Mathieu Aubry, Ulrich Schlickewei, and Daniel Cremers. 2011. The wave kernel signature: A quantum mechanical approach to shape analysis. In ICCV Workshops. IEEE, 1626--1633. http://dblp.uni-trier.de/db/conf/iccvw/iccvw2011.html#AubrySC11
[7]
Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black. 2014. FAUST: Dataset and evaluation for 3D mesh registration. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, Piscataway, NJ, USA.
[8]
Davide Boscaini, Jonathan Masci, Simone Melzi, Michael M Bronstein, Umberto Castellani, and Pierre Vandergheynst. 2015. Learning class-specific descriptors for deformable shapes using localized spectral convolutional networks. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 13--23.
[9]
Davide Boscaini, Jonathan Masci, Emanuele Rodolà, and Michael M. Bronstein. 2016. Learning shape correspondence with anisotropic convolutional neural networks. In NIPS.
[10]
Doug M Boyer, Yaron Lipman, Elizabeth St Clair, Jesus Puente, Biren A Patel, Thomas Funkhouser, Jukka Jernvall, and Ingrid Daubechies. 2011. Algorithms to automatically quantify the geometric similarity of anatomical surfaces. Proceedings of the National Academy of Sciences 108, 45 (2011), 18221--18226.
[11]
Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5--32.
[12]
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. 2013. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203 (2013).
[13]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2014. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 (2014).
[14]
E Brian Davies. 2007. Linear operators and their spectra. Vol. 106. Cambridge University Press.
[15]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems. 3837--3845.
[16]
Daniela Giorgi, Silvia Biasotti, and Laura Paraboschi. 2007. Shape retrieval contest 2007: Watertight models track. SHREC competition 8 (2007).
[17]
Xianfeng Gu, Steven Gortler, and Hugues Hoppe. 2002. Geometry Images. In SIGGRAPH.
[18]
Kan Guo, Dongqing Zou, and Xiaowu Chen. 2015. 3D Mesh Labeling via Deep Convolutional Neural Networks. ACM Trans. Graph. 35, 1 (2015).
[19]
Mikael Henaff, Joan Bruna, and Yann LeCun. 2015. Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 (2015).
[20]
Tao Ju. 2004. Robust repair of polygonal models. ACM Transactions on Graphics (TOG) 23, 3 (2004), 888--895.
[21]
Felix Kälberer, Matthias Nieser, and Konrad Polthier. 2007. QuadCover-Surface Parameterization using Branched Coverings. In Computer Graphics Forum, Vol. 26. Wiley Online Library, 375--384.
[22]
Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri. 2016. 3D Shape Segmentation with Projective Convolutional Networks. arXiv preprint arXiv:1612.02808 (2016).
[23]
Evangelos Kalogerakis, Aaron Hertzmann, and Karan Singh. 2010. Learning 3D mesh segmentation and labeling. ACM Transactions on Graphics (TOG) 29, 4 (2010), 102.
[24]
Vladimir G Kim, Yaron Lipman, and Thomas Funkhouser. 2011. Blended intrinsic maps. In ACM Transactions on Graphics (TOG), Vol. 30. ACM, 79.
[25]
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[26]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097--1105.
[27]
Dmitry Laptev, Nikolay Savinov, Joachim M Buhmann, and Marc Pollefeys. 2016. TIPOOLING: transformation-invariant pooling for feature learning in Convolutional Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 289--297.
[28]
Roee Litman and Alexander M Bronstein. 2014. Learning spectral descriptors for deformable shape correspondence. IEEE transactions on pattern analysis and machine intelligence 36, 1 (2014), 171--180.
[29]
Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431--3440.
[30]
Wolfgang Maier. 1984. Tooth morphology and dietary specialization. In Food acquisition and processing in primates. Springer, 303--330.
[31]
Jonathan Masci, Davide Boscaini, Michael Bronstein, and Pierre Vandergheynst. 2015. Geodesic convolutional neural networks on riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 37--45.
[32]
J Milnor. 1965. Topology from a differentiable viewpoint(University of Virginia Press, Charlottesville, VA). (1965).
[33]
Emil Praun and Hugues Hoppe. 2003. Spherical parametrization and remeshing. In ACM Transactions on Graphics (TOG), Vol. 22. ACM, 340--349.
[34]
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2016a. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. arXiv preprint arXiv:1612.00593 (2016).
[35]
Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. 2016b. Volumetric and Multi-View CNNs for Object Classification on 3D Data. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE.
[36]
Emanuele Rodolà, Samuel Rota Bulo, Thomas Windheuser, Matthias Vestner, and Daniel Cremers. 2014. Dense non-rigid shape correspondence using random forests. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4177--4184.
[37]
Ayan Sinha, Jing Bai, and Karthik Ramani. 2016. Deep learning 3D shape surfaces using geometry images. In European Conference on Computer Vision. Springer, 223--240.
[38]
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE International Conference on Computer Vision. 945--953.
[39]
Jian Sun, Maks Ovsjanikov, and Leonidas Guibas. 2009. A Concise and Provably Informative Multi-scale Signature Based on Heat Diffusion. In Proceedings of the Symposium on Geometry Processing (SGP '09). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 1383--1392. http://dl.acm.org/citation.cfm?id=1735603.1735621
[40]
Federico Tombari, Samuele Salti, and Luigi Di Stefano. 2010. Unique signatures of histograms for local surface description. In European Conference on Computer Vision. Springer, 356--369.
[41]
W. T. Tutte. 1963. How to draw a graph. Proc. London Math. Soc. 13, 3 (1963), 743--768.
[42]
Andrea Vedaldi and Karel Lenc. 2015. Matconvnet: Convolutional neural networks for matlab. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, 689--692.
[43]
Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 97.
[44]
Lingyu Wei, Qixing Huang, Duygu Ceylan, Etienne Vouga, and Hao Li. 2016. Dense Human Body Correspondences Using Convolutional Networks. In Proc. CVPR.
[45]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1912--1920.
[46]
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. 2016. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Advances in Neural Information Processing Systems. 1696--1704.
[47]
Li Yi, Hao Su, Xingwen Guo, and Leonidas Guibas. 2016. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation. arXiv preprint arXiv:1612.00606 (2016).
[48]
Hong-Kai Zhao, Stanley Osher, Barry Merriman, and Myungjoo Kang. 2000. Implicit and nonparametric shape reconstruction from unorganized data using a variational level set method. Computer Vision and Image Understanding 80, 3 (2000), 295--314.

Cited By

View all
  • (2025)3D Shape Segmentation With Potential Consistency Mining and EnhancementIEEE Transactions on Multimedia10.1109/TMM.2024.352167427(133-144)Online publication date: 1-Jan-2025
  • (2025)A Novel SO(3) Rotational Equivariant Masked Autoencoder for 3D Mesh Object AnalysisIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.346504135:1(329-342)Online publication date: Jan-2025
  • (2024)CUS3D: A New Comprehensive Urban-Scale Semantic-Segmentation 3D Benchmark DatasetRemote Sensing10.3390/rs1606107916:6(1079)Online publication date: 19-Mar-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 36, Issue 4
August 2017
2155 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3072959
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017
Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. convolutional neural network
  2. geometric deep learning
  3. shape analysis
  4. shape segmentation

Qualifiers

  • Research-article

Funding Sources

  • Israel PBC and ISF
  • European Research Council (ERC)

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)56
  • Downloads (Last 6 weeks)5
Reflects downloads up to 30 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)3D Shape Segmentation With Potential Consistency Mining and EnhancementIEEE Transactions on Multimedia10.1109/TMM.2024.352167427(133-144)Online publication date: 1-Jan-2025
  • (2025)A Novel SO(3) Rotational Equivariant Masked Autoencoder for 3D Mesh Object AnalysisIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.346504135:1(329-342)Online publication date: Jan-2025
  • (2024)CUS3D: A New Comprehensive Urban-Scale Semantic-Segmentation 3D Benchmark DatasetRemote Sensing10.3390/rs1606107916:6(1079)Online publication date: 19-Mar-2024
  • (2024)Relation Constrained Capsule Graph Neural Networks for Non-Rigid Shape CorrespondenceACM Transactions on Intelligent Systems and Technology10.1145/368885115:6(1-26)Online publication date: 16-Aug-2024
  • (2024)Data-Driven Car Drag Prediction With Depth and Normal RenderingsJournal of Mechanical Design10.1115/1.4065063146:5Online publication date: 28-Mar-2024
  • (2024)Laplacian2Mesh: Laplacian-Based Mesh UnderstandingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325904430:7(4349-4361)Online publication date: 1-Jul-2024
  • (2024)Mesh Neural Networks Based on Dual Graph PyramidsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325703530:7(4211-4224)Online publication date: 1-Jul-2024
  • (2024)Efficient Pooling Operator for 3D Morphable ModelsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.325582030:7(4225-4233)Online publication date: 1-Jul-2024
  • (2024)Dynamic 3D Point Cloud Sequences as 2D VideosIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.342135946:12(9371-9386)Online publication date: Dec-2024
  • (2024)Mesh Convolution With Continuous Filters for 3-D Surface ParsingIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2023.328187135:10(14863-14877)Online publication date: Oct-2024
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media