Skip to main content
SpringerLink
Log in

PointFusionNet: Point feature fusion network for 3D point clouds analysis

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The 3D point clouds is an important type of geometric data structure, and the analysis of 3D point clouds based on deep learning is a very challenging task due to the disorder and irregularity. In existing research, RS-CNN provides an effective and promising method to obtain shape features on disordered point clouds directly, which encodes local features effectively. However, RS-CNN fails to consider point-wise features and global features, which are conducive to point clouds better. In this paper, we proposed PointFusionNet, which solves these problems effectively by fusing point-wise features, local features, and global features. We have designed Feature Fusion Convolution (FF-Conv) and Global Relationship Reasoning Module (GRRM) to build PointFusionNet. The point-wise features were fused with their corresponding local features in the FF-Conv and then mapped into a high-dimensional space to extract richer local features. The GRRM inferred the relationship between various parts, in order to capture global features for enriching the content of the feature descriptor. Therefore the PointFusionNet is suitable for point clouds classification and semantic segmentation by using the two distinctive modules. The PointFusionNet has been tested on ModelNet40 and ShapeNet part datasets, and the experiments show that PointFusionNet has a competitive advantage in shape classification and part segmentation tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Feng M, Wang Y, Liu J, Zhang L, Zaki HF, Mian A (2018) Benchmark data set and method for depth estimation from light field images. IEEE Trans Image Process 27(7):3586–3598

    Article  MathSciNet  Google Scholar 

  2. Zhou Y, Tuzel O (2018) Voxelnet: End-to-end learning for point cloud based 3d object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4490–4499

  3. Qi CR, Liu W, Wu C, Su H, Guibas L (2018) Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 918–927

  4. Ku J, Mozifian M, Lee J, Harakeh A, Waslander SL (2018) Joint 3d proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, pp 1–8

  5. Liu Z, Chen H, Di H, Tao Y, Gong J, Xiong G, Qi J (2018) Real-time 6d lidar slam in large scale natural terrains for ugv. In: 2018 IEEE Intelligent vehicles symposium (IV). IEEE, pp 662–667

  6. Zhu Y, Mottaghi R, Kolve E, Lim JJ, Gupta A, Fei-Fei L, Farhadi A (2017) Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: 2017 IEEE International conference on robotics and automation (ICRA). IEEE, pp 3357–3364

  7. Biswas J, Veloso M (2012) Depth camera based indoor mobile robot localization and navigation. In: 2012 IEEE International conference on robotics and automation. IEEE, pp 1697–1702

  8. Golovinskiy A, Kim VG, Funkhouser T (2009) Shape-based recognition of 3d point clouds in urban environments. In: 2009 IEEE 12Th international conference on computer vision. IEEE, pp 2154–2161

  9. Wang Z, Lu F (2019) Voxsegnet: Volumetric cnns for semantic part segmentation of 3d shapes. IEEE transactions on visualization and computer graphics

  10. Wang DZ, Posner I (2015) Voting for voting in online point cloud object detection. In: Robotics: Science and systems, vol 1, pp 10–15607

  11. Riegler G, Osman Ulusoy A, Geiger A (2017) Octnet: Learning deep 3d representations at high resolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3577–3586

  12. Maturana D, Scherer S (2015) Voxnet: a 3d convolutional neural network for real-time object recognition. In: 2015 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, pp 922–928

  13. Qi CR, Su H, Mo K, Guibas L (2017) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 652–660

  14. Qi CR, Yi L, Su H, Guibas L (2017) Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In: Advances in neural information processing systems, pp 5099–5108

  15. Wang Y, Sun Y, Liu Z, Sarma SE, Bronstein MM, Solomon JM (2019) Dynamic graph cnn for learning on point clouds. ACM Trans Graph (TOG) 38(5):1–12

    Article  Google Scholar 

  16. Liu Y, Fan B, Xiang S, Pan C (2019) Relation-shape convolutional neural network for point cloud analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8895–8904

  17. Zhou H, Fang Z, Gao Y, Huang B, Zhong C, Shang R (2020) Feature fusion network based on attention mechanism for 3d semantic segmentation of point clouds. Pattern Recognition Letters

  18. Li X, Wang W, Hu X, Yang J (2019) Selective kernel networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 510–519

  19. Cao Y, Xu J, Lin S, Wei F, Hu H (2019) Gcnet: Non-local networks meet squeeze-excitation networks and beyond. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp 0–0

  20. Chen Y, Rohrbach M, Yan Z, Shuicheng Y, Feng J, Kalantidis Y (2019) Graph-based global reasoning networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 433–442

  21. Qi CR, Su H (2016) Nießner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and multi-view cnns for object classification on 3d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5648–5656

  22. Gadelha M, Wang R, Maji S (2018) Multiresolution tree networks for 3d point cloud processing. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 103–118

  23. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3d shapenets: a deep representation for volumetric shapes. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1912–1920

  24. Klokov R, Lempitsky V (2017) Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In: Proceedings of the IEEE International Conference on Computer Vision, pp 863–872

  25. Wang PS, Liu Y, Guo YX, Sun CY, Tong X (2017) O-cnn: Octree-based convolutional neural networks for 3d shape analysis. ACM Trans Graph (TOG) 36(4):1–11

    Google Scholar 

  26. Tatarchenko M, Dosovitskiy A, Brox T (2017) Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2088–2096

  27. Wang PS, Sun CY, Liu Y, Tong X (2018) Adaptive o-cnn: a patch-based deep representation of 3d shapes. ACM Trans Graph (TOG) 37(6):1–11

    Google Scholar 

  28. Le T, Duan Y (2018) Pointgrid: a deep network for 3d shape understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9204–9214

  29. Feng Y, Zhang Z, Zhao X, Ji R, Gao Y (2018) Gvcnn: Group-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 264–272

  30. Guo H, Wang J, Gao Y, Li J, Lu H (2016) Multi-view 3d object retrieval with deep embedding network. IEEE Trans Image Process 25(12):5526–5537

    Article  MathSciNet  Google Scholar 

  31. Han Z, Shang M, Liu Z, Vong CM, Liu YS, Zwicker M, Han J, Chen CP (2018) Seqviews2seqlabels: Learning 3d global features via aggregating sequential views by rnn with attention. IEEE Trans Image Process 28(2):658–672

    Article  MathSciNet  Google Scholar 

  32. Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE international conference on computer vision, pp 945–953

  33. Xie J, Dai G, Zhu F, Wong EK, Fang Y (2016) Deepshape: Deep-learned shape descriptor for 3d shape retrieval. IEEE Trans Pattern Anal Mach Intell 39(7):1335–1345

    Article  Google Scholar 

  34. Shen Y, Feng C, Yang Y, Tian D (2018) Mining point cloud local structures by kernel correlation and graph pooling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4548–4557

  35. Li J, Chen B, Hee Lee G (2018) So-net: Self-organizing network for point cloud analysis. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 9397–9406

  36. Komarichev A, Zhong Z, Hua J (2019) A-cnn: Annularly convolutional neural networks on point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7421–7430

  37. Wang L, Huang Y, Hou Y, Zhang S, Shan J (2019) Graph attention convolution for point cloud semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 10296–10305

  38. Zhang Y, Rabbat M (2018) A graph-cnn for 3d point cloud classification. In: 2018 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 6279–6283

  39. Simonovsky M, Komodakis N (2017) Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3693–3702

  40. Song L, Li Y, Li Z, Yu G, Sun H, Sun J, Zheng N (2019) Learnable tree filter for structure-preserving feature transform. In: Advances in neural information processing systems, pp 1709–1719

  41. Yi L, Kim VG, Ceylan D, Shen IC, Yan M, Su H, Lu C, Huang Q, Sheffer A, Guibas L (2016) A scalable active framework for region annotation in 3d shape collections. ACM Trans Graph (TOG) 35(6):1–12

    Article  Google Scholar 

  42. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

  43. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034

  44. Hua BS, Tran MK, Yeung SK (2018) Pointwise convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 984–993

  45. Xie S, Liu S, Chen Z, Tu Z (2018) Attentional shapecontextnet for point cloud recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4606–4615

  46. Wang C, Samari B, Siddiqi K (2018) Local spectral graph convolution for point set feature learning. In: Proceedings of the European conference on computer vision (ECCV), pp 52–66

  47. Li Y, Bu R, Sun M, Wu W, Di X, Chen B (2018) Pointcnn: Convolution on x-transformed points. In: Advances in neural information processing systems, pp 820–830

  48. Atzmon M, Maron H, Lipman Y (2018) Point convolutional neural networks by extension operators. arXiv:1803.10091

  49. Xu Y, Fan T, Xu M, Zeng L, Qiao Y (2018) Spidercnn: Deep learning on point sets with parameterized convolutional filters. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 87–102

  50. Chang AX, Funkhouser T, Guibas L, Hanrahan P, Huang Q, Li Z, Savarese S, Savva M, Song S, Su H et al (2015) Shapenet: An information-rich 3d model repository. arXiv:1512.03012

  51. Huang Q, Wang W, Neumann U (2018) Recurrent slice networks for 3d segmentation of point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2626–2635

  52. Su H, Jampani V, Sun D, Maji S, Kalogerakis E, Yang MH, Kautz J (2018) Splatnet: Sparse lattice networks for point cloud processing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2530–2539

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61772328).

Author information

Authors and Affiliations

  1. Shanghai University of Engineering Science, 333 Longteng Road, Songjiang District, Shanghai, Shanghai, 201620, China

    Pan Liang, Zhijun Fang, Bo Huang, Heng Zhou, Xianhua Tang & Cengsi Zhong

Authors
  1. Pan Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhijun Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heng Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xianhua Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cengsi Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pan Liang.

Ethics declarations

Conflict of interests

We wish to confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liang, P., Fang, Z., Huang, B. et al. PointFusionNet: Point feature fusion network for 3D point clouds analysis. Appl Intell 51, 2063–2076 (2021). https://doi.org/10.1007/s10489-020-02004-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02004-8

Keywords

Search

Navigation