Abstract
The classification of 3D point clouds is a regular task, but remains a highly challenging problem because 3D point clouds usually contain a large amount of information on irregular shapes. Several recent studies have shown the excellent performance of deep learning in 3D point cloud classification. Convolutional neural network (CNN)-based 3D point cloud classification methods are also increasingly used owing to their efficient and convenient feature extraction capability. However, most of these methods do not take much prior information and local structural information into consideration, often resulting in their inability to extract sufficient information to improve the classification accuracy. In this study, we present a novel convolution operation named HyConv, which includes two key components. First, inspired by 2D convolution, we design a feature transformation module to capture more local structural information. Second, to extract the prior information, a hybrid weight module is introduced to estimate two types of weights on the basis of the distribution information of the spatial and feature domains. Additionally, we propose an adaptive method to learn hybrid weights to obtain hybrid distribution information. Finally, based on the proposed convolutional operator HyConv, we build a deep neural network Hybrid-CNN and conduct experiments on two commonly used datasets. The results show that our hybrid network outperforms most existing methods on ModelNet40. Furthermore, state-of-the-art performance is achieved with ScanObjectNN, which is a great improvement compared with existing methods.
Similar content being viewed by others
References
Liang Z, Guo Y, Feng Y, Chen W, Qiao L, Zhou L, Zhang J, Liu H (2019) Stereo matching using multi-level cost volume and multi-scale feature constancy. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2019.2928550
Chen X, Ma H, Wan J, Li B, Xia T (2017) Multi-view 3D object detection network for autonomous driving. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1907–1915
Zhang C, Cheng J, Tian Q (2020) Multi-view image classification with visual, semantic and view consistency. IEEE Trans Image Process 29:617–627. https://doi.org/110.1109/TIP.2019.2934576
Charles RQ, Su H, Kaichun M, Guibas LJ (2017a) PointNet: Deep learning on point sets for 3D classification and segmentation. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 652–660
Charles RQ, Yi L, Su H, Guibas LJ (2017b) PointNet++: Deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems (NeurIPS), pp 5099–5108
Guo Y, Wang H, Hu Q, Liu H, Liu L, Bennamoun M (2020) Deep learning for 3D point clouds: A survey. IEEE Trans Pattern Anal Mach Intell. https://doi.org/110.1109/TPAMI.2020.3005434
Sun J, Ovsjanikov M, Guibas L (2009) A concise and provably informative multiscale signature based on heat diffusion. Comput Graph Forum 28(5):1383–1392
Aubry M, Schlickewei U, Cremers D (2011) The wave kernel signature: A quantum mechanical approach to shape analysis. In: IEEE International conference on computer vision(ICCV), pp 1626–1633
Bronstein MM, Kokkinos I (2010) Scale-invariant heat kernel signatures for non-rigid shape recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1704–1711
Xie J, Dai G, Fang Y (2017) Deep multimetric learning for shape-based 3D model retrieval. IEEE Trans Multimed 19(11):2463–2474
Chen Q, Fang B, Yu Y, Tang Y (2015) 3D CAD model retrieval based on the combination of features. Multimed Tools Appl 74(13):4907–4925
Gao S, Cheng MM, Zhao K, Zhang XY, Yang MH, Torr PH (2019) Res2Net: A new multi-scale backbone architecture. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2019.2938758
Jing L, Tian Y (2020) Self-supervised visual feature learning with deep neural networks: a survey. IEEE Trans Pattern Anal Mach Intell. https://doi.org/110.1109/TPAMI.2020.2992393
Maturana D, Scherer S (2015) VoxNet: A 3D convolutional neural network for real-time object recognition. In: IEEE/RSJ International conference on intelligent robots and systems (IROS), pp 922–928
Le T, Duan Y (2018) PointGrid: A deep network for 3D shape understanding. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 9204–9214
Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-View convolutional neural networks for 3D shape recognition, pp 945–953
Han Z, Shang M, Liu Z, Vong CM, Liu YS, Zwicker M, Han J, Chen CP (2018) SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by RNN with attention. IEEE Trans Image Process 28(2):658–672
Sun K, Zhang J, Liu J, Yu R, Song Z (2021) DRCNN: Dynamic routing convolutional neural network for multi-view 3D object recognition. IEEE Trans Image Process 30:868–877
Zhao H, Jiang L, Fu CW, Jia J (2019) PointWeb: Enhancing local neighborhood features for point cloud processing. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5560–5568
Duan Y, Zheng Y, Lu J, Zhou J, Tian Q (2019) Structural relational reasoning of point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 949–958
Li M, Hu Y, Zhao N, Guo L (2019) LPCCNet: A lightweight network for point cloud classification. IEEE Geosci Remote Sens Lett 16(6):962–966
Boulch A (2020) ConvPoint: continuous convolutions for point cloud processing. Comput Graph 88:24–34. https://doi.org/10.1016/j.cag.2020.02.005
Lan S, Yu R, Yu G, Davis LS (2019) Modeling local geometric structure of 3D point clouds using Geo-CNN. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 998–1008
Liang P, Fang Z, Huang B, Zhou H, Tang X, Zhong C (2020) PointFusionNet: Point feature fusion network for 3D point clouds analysis. Appl Intell. https://doi.org/10.1007/s10489-020-02004-8
Wu W, Qi Z, Fuxin L (2019) PointConv: deep convolutional networks on 3D point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 9621–9630
Hermosilla P, Ritschel T, Vȧzquez P, Vinacua Á, Ropinski T (2018) Monte Carlo convolution for learning on non-uniformly sampled point clouds. ACM Trans Graph 37(6):1–12
Ma C, Guo Y, Lei Y, An W (2018) Binary volumetric convolutional neural networks for 3-D object recognition. IEEE Trans Instrum Meas 68(1):38–48
Klokov R, Lempitsky V (2017) Escape from cells: Deep kd-networks for the recognition of 3D point cloud models. In: Proceedings of the IEEE International conference on computer vision(ICCV), pp 863–872
Riegler G, Osman Ulusoy A, Geiger A (2017) OctNet: Learning deep 3D representations at high resolutions. In: Proceedings of the IEEE Conference on computer vision and pattern recognition(CVPR), pp 3577–3586
Yang Z, Wang L (2019) Learning relationships for multi-view 3D object recognition. In: IEEE International conference on computer vision (ICCV), pp 7505–7514
Feng Y, Zhang Z, Zhao X, Ji R, Gao Y (2018) GVCNN: group-view convolutional neural networks for 3D shape recognition. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 264–272
Simonovsky M, Komodakis N (2017) Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proceedings of the IEEE Conference on computer vision and pattern recognition, pp 3693–3702
Zhang Y, Rabbat M (2018) A Graph-CNN for 3D point cloud classification. In: IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 6279–6283. https://doi.org/10.1109/ICASSP.2018.8462291
Wang C, Samari B, Siddiqi K (2018) Local spectral graph convolution for point set feature learning. In: Proceedings of the European conference on computer vision (ECCV), pp 52–66
Atzmon M, Maron H, Lipman Y (2018) Point convolutional neural networks by extension operators. ACM Trans Graph 37(4):1–12
Xu Y, Fan T, Xu M, Zeng L, Qiao Y (2018) SpiderCNN: Deep learning on point sets with parameterized convolutional filters. In: IEEE Conference on European conference on computer vision (ECCV), pp 87–102
Li Y, Bu R, Sun M, Wu W, Di X, Chen B (2018) PointCNN: Convolution on \(\mathcal {X}\)-transformed points. In: Advances in Neural Information Processing Systems(NeurIPS), pp 820–830
Boulch A, Puy G, Marlet R (2020) FKAConv: Feature-kernel alignment for point cloud convolution. In: Proceedings of the Asian conference on computer vision (ACCV)
Ben-Shabat Y, Lindenbaum M, Fischer A (2018) 3DmFV: Three-dimensional point cloud classification in real-time using convolutional neural networks. IEEE Robot Autom Lett 3(4):3145–3152
Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D ShapeNets: A deep representation for volumetric shapes. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 1912–1920
Wu W, Qi Z, Fuxin L (2019) Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In: IEEE International conference on computer vision (ICCV), pp 1588–1597
Hua B, Pham Q, Nguyen DT, Tran M, Yu L, Yeung S (2016) SceneNN: a scene meshes dataset with annotations. In: International conference on 3D vision (3DV), pp 92–101
Dai A, Chang AX, Savva M, Halber M, Funkhouser T, Niessner M (2017) ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5828–5839
Rao Y, Lu J, Zhou J (2020) Global-Local bidirectional reasoning for unsupervised representation learning of 3D point clouds. In: IEEE Conference on computer vision and pattern recognition (CVPR), pp 5376–5385
Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2019) Dynamic graph CNN for learning on point clouds. ACM Trans Graph 38(5):1–12
Qiu S, Anwar S, Barnes N (2020). Dense-Resolution network for point cloud classification and segmentation. arXiv:200506734
Maaten LVD, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11):2579–2605
Acknowledgements
This work was supported by the National Natural Science Foundation of China under grant 62032022, and the Zhejiang Provincial Natural Science Foundation of China under grant LZ20F030001. (Corresponding author: Feilong Cao.)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Hu, M., Ye, H. & Cao, F. Convolutional neural networks with hybrid weights for 3D point cloud classification. Appl Intell 51, 6983–6996 (2021). https://doi.org/10.1007/s10489-021-02240-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-021-02240-6