Skip to main content
Springer Nature Link
Log in

NPTC-net: Narrow-Band Parallel Transport Convolutional Neural Networks on Point Clouds

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Convolution plays a crucial role in various applications in signal and image processing, analysis, and recognition. It is also the main building block of convolution neural networks (CNNs). Designing appropriate convolution neural networks on manifold-structured point clouds can inherit and empower recent advances of CNNs to analyzing and processing point cloud data. However, one of the major challenges is to define a proper way to “sweep”filters through the point cloud as a natural generalization of the planar convolution and to reflect the point cloud’s geometry at the same time. In this paper, we consider generalizing convolution by adapting parallel transport on the point cloud. Inspired by a triangulated surface-based method [46], we propose the Narrow-Band Parallel Transport Convolution (NPTC) using a specifically defined connection on a voxel-based narrow-band approximation of point cloud data. With that, we further propose a deep convolutional neural network based on NPTC (called NPTC-net) for point cloud classification and segmentation. Comprehensive experiments show that the proposed NPTC-net achieves similar or better results than current state-of-the-art methods on point cloud classification and segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al.: Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016)

  2. Adalsteinsson, D., Sethian, J.A.: A fast level set method for propagating interfaces. J. Comput. Phys. 118(2), 269–277 (1995)

    Article  MathSciNet  Google Scholar 

  3. Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M., Savarese, S.: 3d semantic parsing of large-scale indoor spaces. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1534–1543. IEEE Computer Society, Los Alamitos, CA, USA (2016).10.1109/CVPR.2016.170. https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.170

  4. Bindu, V., Nair, K.R.: A fast narrow band level set formulation for shape extraction. In: The Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT 2014), pp. 137–142. IEEE (2014)

  5. Boscaini, D., Masci, J., Rodolà, E., Bronstein, M.: Learning shape correspondence with anisotropic convolutional neural networks. In: D.D. Lee, M. Sugiyama, U.V. Luxburg, I. Guyon, R. Garnett (eds.) Advances in Neural Information Processing Systems 29, pp. 3189–3197. Curran Associates, Inc. (2016). http://papers.nips.cc/paper/6045-learning-shape-correspondence-with-anisotropic-convolutional-neural-networks.pdf

  6. Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond euclidean data. IEEE Signal Process. Mag. 34(4), 18–42 (2017). https://doi.org/10.1109/MSP.2017.2693418

    Article  Google Scholar 

  7. Bruna, J., Zaremba, W., Szlam, A., Lecun, Y.: Spectral networks and locally connected networks on graphs. In: International Conference on Learning Representations (ICLR2014), CBLS, April 2014 (2014)

  8. Cao, W., Yan, Z., He, Z., He, Z.: A comprehensive survey on geometric deep learning. IEEE Access 8, 35929–35949 (2020)

    Article  Google Scholar 

  9. Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: D.D. Lee, M. Sugiyama, U.V. Luxburg, I. Guyon, R. Garnett (eds.) Advances in Neural Information Processing Systems 29, pp. 3844–3852. Curran Associates, Inc. (2016). http://papers.nips.cc/paper/6081-convolutional-neural-networks-on-graphs-with-fast-localized-spectral-filtering.pdf

  10. Dong, B.: Sparse representation on graphs by tight wavelet frames and applications. Appl. Comput. Harmonic Anal. 42(3), 452–479 (2017)

    Article  MathSciNet  Google Scholar 

  11. Eldar, Y., Lindenbaum, M., Porat, M., Zeevi, Y.Y.: The farthest point strategy for progressive image sampling. IEEE Trans. on Image Process. 6(9), 1305–1315 (1997)

    Article  Google Scholar 

  12. Franke, R., Nielson, G.M.: Scattered data interpolation and applications: a tutorial and survey. Geomet. Model. pp. 131–160 (1991)

  13. Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016)

  14. Grevera, G.J., Udupa, J.K.: An objective comparison of 3-d image interpolation methods. IEEE Trans. Med. Imaging 17(4), 642–652 (1998)

    Article  Google Scholar 

  15. Hammond, D.K., Vandergheynst, P., Gribonval, R.: Wavelets on graphs via spectral graph theory. Appl. Comput. Harmonic Anal. 30(2), 129–150 (2011)

    Article  MathSciNet  Google Scholar 

  16. Helgason, S.: Differential geometry, Lie groups, and symmetric spaces, vol. 80. Academic press (1979)

  17. Henaff, M., Bruna, J., LeCun, Y.: Deep convolutional networks on graph-structured data. arXiv preprint arXiv:1506.05163 (2015)

  18. Hojjatoleslami, S., Kittler, J.: Region growing: a new approach. IEEE Trans. Image process. 7(7), 1079–1084 (1998)

    Article  Google Scholar 

  19. Jiang, M., Zhong, Y., Wang, X., Huang, X., Guo, R.: Improve the nonparametric image segmentation with narrowband levelset and fast gauss transformation (2012)

  20. Katznelson, Y.: An introduction to harmonic analysis. Cambridge University Press (2004)

  21. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Y. Bengio, Y. LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015). arXiv:1412.6980

  22. Klokov, R., Lempitsky, V.: Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 863–872. IEEE Computer Society, Los Alamitos, CA, USA (2017). 10.1109/ICCV.2017.99. https://doi.ieeecomputersociety.org/10.1109/ICCV.2017.99

  23. Knebelman, M.: Spaces of relative parallelism. Ann. Math. pp. 387–399 (1951)

  24. Kobayashi, S., Nomizu, K.: Foundations of differential geometry, vol. 2. Interscience publishers New York (1969)

  25. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009)

    Article  Google Scholar 

  26. Lai, R., Liang, J., Zhao, H.: A local mesh method for solving pdes on point clouds. Inverse Prob. Imaging 7(3), 737–755 (2013)

    Article  MathSciNet  Google Scholar 

  27. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. nature 521(7553), 436 (2015)

  28. Levie, R., Monti, F., Bresson, X., Bronstein, M.M.: Cayleynets: Graph convolutional neural networks with complex rational spectral filters. IEEE Trans. Sig. Process. 67(1), 97–109 (2019). https://doi.org/10.1109/TSP.2018.2879624

    Article  MathSciNet  MATH  Google Scholar 

  29. Li, J., Chen, B.M., Hee Lee, G.: So-net: Self-organizing network for point cloud analysis. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  30. Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: Pointcnn: Convolution on x-transformed points. In: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (eds.) Advances in Neural Information Processing Systems 31, pp. 820–830. Curran Associates, Inc. (2018). http://papers.nips.cc/paper/7362-pointcnn-convolution-on-x-transformed-points.pdf

  31. Li, Y., Ma, L., Zhong, Z., Liu, F., Chapman, M.A., Cao, D., Li, J.: Deep learning for lidar point clouds in autonomous driving: a review. IEEE Transactions on Neural Networks and Learning Systems (2020)

  32. Liu, W., Sun, J., Li, W., Hu, T., Wang, P.: Deep learning on point clouds and its application: A survey. Sensors 19(19), 4188 (2019)

    Article  Google Scholar 

  33. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3d surface construction algorithm. ACM siggraph computer graphics 21(4), 163–169 (1987)

    Article  Google Scholar 

  34. Mao, J., Wang, X., Li, H.: Interpolated convolutional networks for 3d point cloud understanding. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1578–1587 (2019)

  35. Masci, J., Boscaini, D., Bronstein, M.M., Vandergheynst, P.: Geodesic convolutional neural networks on riemannian manifolds. In: The IEEE International Conference on Computer Vision (ICCV) Workshops (2015)

  36. Monti, F., Boscaini, D., Masci, J., Rodola, E., Svoboda, J., Bronstein, M.M.: Geometric deep learning on graphs and manifolds using mixture model cnns. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  37. Morgan, H.L.: The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. J. Chem. Document. 5(2), 107–113 (1965)

    Article  Google Scholar 

  38. Osher, S., Fedkiw, R.: Level set methods and dynamic implicit surfaces, vol. 153. Springer Science & Business Media (2006)

  39. Osher, S., Sethian, J.A.: Fronts propagating with curvature-dependent speed: algorithms based on hamilton-jacobi formulations. J. Comput. Phys. 79(1), 12–49 (1988)

    Article  MathSciNet  Google Scholar 

  40. Pal, S.K., Mitra, S.: Multilayer perceptron, fuzzy sets, classifiaction (1992)

  41. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

  42. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, R. Garnett (eds.) Advances in Neural Information Processing Systems 30, pp. 5099–5108. Curran Associates, Inc. (2017). http://papers.nips.cc/paper/7095-pointnet-deep-hierarchical-feature-learning-on-point-sets-in-a-metric-space.pdf

  43. Robbins, H., Monro, S.: A stochastic approximation method. The annals of mathematical statistics pp. 400–407 (1951)

  44. Rosenthal, P., Linsen, L.: Smooth surface extraction from unstructured point-based volume data using pdes. IEEE Trans. Visual. Comput. Graph. 14(6), 1531–1546 (2008)

    Article  Google Scholar 

  45. Rosenthal, P., Molchanov, V., Linsen, L.: A narrow band level set method for surface extraction from unstructured point-based volume data

  46. Schonsheck, S.C., Dong, B., Lai, R.: Parallel transport convolution: A new tool for convolutional neural networks on manifolds. CoRR abs/1805.07857 (2018). arXiv:1805.07857

  47. Sethian, J.A.: A fast marching level set method for monotonically advancing fronts. Proceed. Natl. Acad. Sci. 93(4), 1591–1595 (1996)

    Article  MathSciNet  Google Scholar 

  48. Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. J. Big Data 6(1), 1–48 (2019)

    Article  Google Scholar 

  49. Tatarchenko, M., Park, J., Koltun, V., Zhou, Q.Y.: Tangent convolutions for dense prediction in 3d. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

  50. Tchapmi, L., Choy, C., Armeni, I., Gwak, J., Savarese, S.: Segcloud: Semantic segmentation of 3d point clouds. In: 2017 International Conference on 3D Vision (3DV), pp. 537–547 (2017). 10.1109/3DV.2017.00067

  51. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6411–6420 (2019)

  52. Wang, C., Samari, B., Siddiqi, K.: Local spectral graph convolution for point set feature learning. In: The European Conference on Computer Vision (ECCV) (2018)

  53. Wang, S., Suo, S., Ma, W.C., Pokrovsky, A., Urtasun, R.: Deep parametric continuous convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2589–2597 (2018)

  54. Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph CNN for learning on point clouds. CoRR abs/1801.07829 (2018). arXiv:1801.07829

  55. Wu, W., Qi, Z., Li, F.: Pointconv: Deep convolutional networks on 3d point clouds. CoRR abs/1811.07246 (2018). arXiv:1811.07246

  56. Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3d shapenets: A deep representation for volumetric shapes. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  57. Xu, Y., Fan, T., Xu, M., Zeng, L., Qiao, Y.: Spidercnn: Deep learning on point sets with parameterized convolutional filters. In: The European Conference on Computer Vision (ECCV) (2018)

  58. Yang, Y., Liu, S., Pan, H., Liu, Y., Tong, X.: Pfcnn: convolutional neural networks on 3d surfaces using parallel frames. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13578–13587 (2020)

  59. Yang, Z., Litany, O., Birdal, T., Sridhar, S., Guibas, L.: Continuous geodesic convolutions for learning on 3d shapes. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 134–144 (2021)

  60. Yi, L., Kim, V.G., Ceylan, D., Shen, I.C., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., Guibas, L.: A scalable active framework for region annotation in 3d shape collections. ACM Trans. Graph. 35(6), 210:1–210:12 (2016). 10.1145/2980179.2980238. http://doi.acm.org/10.1145/2980179.2980238

  61. Zhang, C., Luo, W., Urtasun, R.: Efficient convolutions for real-time semantic segmentation of 3d point clouds. In: 2018 International Conference on 3D Vision (3DV), pp. 399–408. IEEE (2018)

  62. Zhao, H.: A fast sweeping method for eikonal equations. Math. Comput. 74(250), 603–627 (2005)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Beijing International Center for Mathematical Research, Peking University, Beijing, China

    Pengfei Jin, Tianhao Lai & Bin Dong

  2. Department of Mathematics, Rensselaer Polytechnic Institute, Troy, NY, USA

    Rongjie Lai

  3. Center for Data Science, Peking University, Beijing, China

    Bin Dong

  4. Beijing Institute of Big Data Research, Beijing, China

    Bin Dong

Authors
  1. Pengfei Jin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Tianhao Lai
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Rongjie Lai
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Bin Dong
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Rongjie Lai or Bin Dong.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

R. Lai’s work is supported in part by an NSF Career Award DMS-1752934. Bin Dong is supported in part by Beijing Natural Science Foundation (Z180001), NSFC 12090022 and Beijing Academy of Artificial Intelligence (BAAI)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, P., Lai, T., Lai, R. et al. NPTC-net: Narrow-Band Parallel Transport Convolutional Neural Networks on Point Clouds. J Sci Comput 90, 39 (2022). https://doi.org/10.1007/s10915-021-01699-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10915-021-01699-2

Keywords

Mathematics Subject Classification