Skip to main content
Log in

Convolutional neural networks with hybrid weights for 3D point cloud classification

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The classification of 3D point clouds is a regular task, but remains a highly challenging problem because 3D point clouds usually contain a large amount of information on irregular shapes. Several recent studies have shown the excellent performance of deep learning in 3D point cloud classification. Convolutional neural network (CNN)-based 3D point cloud classification methods are also increasingly used owing to their efficient and convenient feature extraction capability. However, most of these methods do not take much prior information and local structural information into consideration, often resulting in their inability to extract sufficient information to improve the classification accuracy. In this study, we present a novel convolution operation named HyConv, which includes two key components. First, inspired by 2D convolution, we design a feature transformation module to capture more local structural information. Second, to extract the prior information, a hybrid weight module is introduced to estimate two types of weights on the basis of the distribution information of the spatial and feature domains. Additionally, we propose an adaptive method to learn hybrid weights to obtain hybrid distribution information. Finally, based on the proposed convolutional operator HyConv, we build a deep neural network Hybrid-CNN and conduct experiments on two commonly used datasets. The results show that our hybrid network outperforms most existing methods on ModelNet40. Furthermore, state-of-the-art performance is achieved with ScanObjectNN, which is a great improvement compared with existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Liang Z, Guo Y, Feng Y, Chen W, Qiao L, Zhou L, Zhang J, Liu H (2019) Stereo matching using multi-level cost volume and multi-scale feature constancy. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2019.2928550

  2. Chen X, Ma H, Wan J, Li B, Xia T (2017) Multi-view 3D object detection network for autonomous driving. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1907–1915

  3. Zhang C, Cheng J, Tian Q (2020) Multi-view image classification with visual, semantic and view consistency. IEEE Trans Image Process 29:617–627. https://doi.org/110.1109/TIP.2019.2934576

    Article  MathSciNet  Google Scholar 

  4. Charles RQ, Su H, Kaichun M, Guibas LJ (2017a) PointNet: Deep learning on point sets for 3D classification and segmentation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 652–660

  5. Charles RQ, Yi L, Su H, Guibas LJ (2017b) PointNet++: Deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems (NeurIPS), pp 5099–5108

  6. Guo Y, Wang H, Hu Q, Liu H, Liu L, Bennamoun M (2020) Deep learning for 3D point clouds: A survey. IEEE Trans Pattern Anal Mach Intell. https://doi.org/110.1109/TPAMI.2020.3005434

  7. Sun J, Ovsjanikov M, Guibas L (2009) A concise and provably informative multiscale signature based on heat diffusion. Comput Graph Forum 28(5):1383–1392

    Article  Google Scholar 

  8. Aubry M, Schlickewei U, Cremers D (2011) The wave kernel signature: A quantum mechanical approach to shape analysis. In: IEEE International conference on computer vision(ICCV), pp 1626–1633

  9. Bronstein MM, Kokkinos I (2010) Scale-invariant heat kernel signatures for non-rigid shape recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1704–1711

  10. Xie J, Dai G, Fang Y (2017) Deep multimetric learning for shape-based 3D model retrieval. IEEE Trans Multimed 19(11):2463–2474

    Article  Google Scholar 

  11. Chen Q, Fang B, Yu Y, Tang Y (2015) 3D CAD model retrieval based on the combination of features. Multimed Tools Appl 74(13):4907–4925

    Article  Google Scholar 

  12. Gao S, Cheng MM, Zhao K, Zhang XY, Yang MH, Torr PH (2019) Res2Net: A new multi-scale backbone architecture. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2019.2938758

  13. Jing L, Tian Y (2020) Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans Pattern Anal Mach Intell. https://doi.org/110.1109/TPAMI.2020.2992393

  14. Maturana D, Scherer S (2015) VoxNet: A 3D convolutional neural network for real-time object recognition. In: IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 922–928

  15. Le T, Duan Y (2018) PointGrid: A deep network for 3D shape understanding. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 9204–9214

  16. Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-View convolutional neural networks for 3D shape recognition, pp 945–953

  17. Han Z, Shang M, Liu Z, Vong CM, Liu YS, Zwicker M, Han J, Chen CP (2018) SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by RNN with attention. IEEE Trans Image Process 28(2):658–672

    Article  MathSciNet  Google Scholar 

  18. Sun K, Zhang J, Liu J, Yu R, Song Z (2021) DRCNN: Dynamic routing convolutional neural network for multi-view 3D object recognition. IEEE Trans Image Process 30:868–877

    Article  Google Scholar 

  19. Zhao H, Jiang L, Fu CW, Jia J (2019) PointWeb: Enhancing local neighborhood features for point cloud processing. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5560–5568

  20. Duan Y, Zheng Y, Lu J, Zhou J, Tian Q (2019) Structural relational reasoning of point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 949–958

  21. Li M, Hu Y, Zhao N, Guo L (2019) LPCCNet: A lightweight network for point cloud classification. IEEE Geosci Remote Sens Lett 16(6):962–966

    Article  Google Scholar 

  22. Boulch A (2020) ConvPoint: continuous convolutions for point cloud processing. Comput Graph 88:24–34. https://doi.org/10.1016/j.cag.2020.02.005

    Article  Google Scholar 

  23. Lan S, Yu R, Yu G, Davis LS (2019) Modeling local geometric structure of 3D point clouds using Geo-CNN. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 998–1008

  24. Liang P, Fang Z, Huang B, Zhou H, Tang X, Zhong C (2020) PointFusionNet: Point feature fusion network for 3D point clouds analysis. Appl Intell. https://doi.org/10.1007/s10489-020-02004-8

  25. Wu W, Qi Z, Fuxin L (2019) PointConv: deep convolutional networks on 3D point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 9621–9630

  26. Hermosilla P, Ritschel T, Vȧzquez P, Vinacua Á, Ropinski T (2018) Monte Carlo convolution for learning on non-uniformly sampled point clouds. ACM Trans Graph 37(6):1–12

    Article  Google Scholar 

  27. Ma C, Guo Y, Lei Y, An W (2018) Binary volumetric convolutional neural networks for 3-D object recognition. IEEE Trans Instrum Meas 68(1):38–48

    Article  Google Scholar 

  28. Klokov R, Lempitsky V (2017) Escape from cells: Deep kd-networks for the recognition of 3D point cloud models. In: Proceedings of the IEEE International conference on computer vision(ICCV), pp 863–872

  29. Riegler G, Osman Ulusoy A, Geiger A (2017) OctNet: Learning deep 3D representations at high resolutions. In: Proceedings of the IEEE Conference on computer vision and pattern recognition(CVPR), pp 3577–3586

  30. Yang Z, Wang L (2019) Learning relationships for multi-view 3D object recognition. In: IEEE International conference on computer vision (ICCV), pp 7505–7514

  31. Feng Y, Zhang Z, Zhao X, Ji R, Gao Y (2018) GVCNN: group-view convolutional neural networks for 3D shape recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 264–272

  32. Simonovsky M, Komodakis N (2017) Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 3693–3702

  33. Zhang Y, Rabbat M (2018) A Graph-CNN for 3D point cloud classification. In: IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 6279–6283. https://doi.org/10.1109/ICASSP.2018.8462291

  34. Wang C, Samari B, Siddiqi K (2018) Local spectral graph convolution for point set feature learning. In: Proceedings of the European conference on computer vision (ECCV), pp 52–66

  35. Atzmon M, Maron H, Lipman Y (2018) Point convolutional neural networks by extension operators. ACM Trans Graph 37(4):1–12

    Article  Google Scholar 

  36. Xu Y, Fan T, Xu M, Zeng L, Qiao Y (2018) SpiderCNN: Deep learning on point sets with parameterized convolutional filters. In: IEEE Conference on European conference on computer vision (ECCV), pp 87–102

  37. Li Y, Bu R, Sun M, Wu W, Di X, Chen B (2018) PointCNN: Convolution on \(\mathcal {X}\)-transformed points. In: Advances in Neural Information Processing Systems(NeurIPS), pp 820–830

  38. Boulch A, Puy G, Marlet R (2020) FKAConv: Feature-kernel alignment for point cloud convolution. In: Proceedings of the Asian conference on computer vision (ACCV)

  39. Ben-Shabat Y, Lindenbaum M, Fischer A (2018) 3DmFV: Three-dimensional point cloud classification in real-time using convolutional neural networks. IEEE Robot Autom Lett 3(4):3145–3152

    Article  Google Scholar 

  40. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D ShapeNets: A deep representation for volumetric shapes. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1912–1920

  41. Wu W, Qi Z, Fuxin L (2019) Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In: IEEE International conference on computer vision (ICCV), pp 1588–1597

  42. Hua B, Pham Q, Nguyen DT, Tran M, Yu L, Yeung S (2016) SceneNN: a scene meshes dataset with annotations. In: International conference on 3D vision (3DV), pp 92–101

  43. Dai A, Chang AX, Savva M, Halber M, Funkhouser T, Niessner M (2017) ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5828–5839

  44. Rao Y, Lu J, Zhou J (2020) Global-Local bidirectional reasoning for unsupervised representation learning of 3D point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5376–5385

  45. Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2019) Dynamic graph CNN for learning on point clouds. ACM Trans Graph 38(5):1–12

    Article  Google Scholar 

  46. Qiu S, Anwar S, Barnes N (2020). Dense-Resolution network for point cloud classification and segmentation. arXiv:200506734

  47. Maaten LVD, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11):2579–2605

    MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under grant 62032022, and the Zhejiang Provincial Natural Science Foundation of China under grant LZ20F030001. (Corresponding author: Feilong Cao.)

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Feilong Cao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, M., Ye, H. & Cao, F. Convolutional neural networks with hybrid weights for 3D point cloud classification. Appl Intell 51, 6983–6996 (2021). https://doi.org/10.1007/s10489-021-02240-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-02240-6

Keywords

Navigation