Skip to main content
SpringerLink
Log in

PSR-GAT: Arbitrary point cloud super-resolution using graph attention networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract 

Point cloud super-resolution plays a central role in the mesh's quality in 3D reconstruction, while the feature extractor is vital for the learning-based point cloud upsampling pipelines. In this paper, we propose an arbitrary 3D point cloud upsampling network (PSR-GAT), which comprises the feature extraction module, GAT module, and upsampling module. For the input point cloud, the feature extraction module locates k nearest points of each point in 3D space by k-NN algorithm, then converts the local geometry information into high dimensional feature space through a multi-layer point-wise convolution. The GAT module converts the local geometry feature of each point into the semantic feature through a multi-layer graph attention network. The module dynamically adjusts the neighbor space of the point in each layer to increase the receptive field range and effectively fuses the semantic information of different levels through residual connection. This makes the local geometric in- formation extraction efficient. The upsampling module adds the number of points and maps them from feature space to 3D space. Extensive experimental results show that PSR-GAT exhibits a better performance than the existing state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

All data included in this study are available upon request by contact with the corresponding author.

References

  1. Alexa M, Behr J, Cohen-Or D, Fleishman S, Levin D, Silva CT (2003) Computing and rendering point set surfaces. IEEE Trans Visual Comput Graphics 9(1):3–15

    Article  Google Scholar 

  2. Amirkhani D, Bastanfard A (2019) Inpainted image quality evaluation based on saliency map features. In: 2019 5th Iranian Conference on Signal Processing and Intelligent Systems (ICSPIS), pp. 1–6. IEEE

  3. Bastanfard A, Amirkhani D, Mohammadi M (2022) Toward image super- resolution based on local regression and nonlocal means. Multimed Tools Appl 81(16):23473–23492

    Article  Google Scholar 

  4. Boltz S, Nielsen F, Soatto S (2010) Earth mover distance on superpixels. In: International Conference on Image Processing, HongKong, China, pp. 4597–4600

  5. Butt MA, Maragos P (1998) Optimum design of chamfer distance transforms. IEEE Trans Image Process 7(10):1477–1484

    Article  ADS  Google Scholar 

  6. Chen TY, Hsiao CC, Huang C-C (2022) Density-imbalance-eased lidar point cloud upsampling via feature consistency learning. IEEE Trans Intell Veh 1–1. https://doi.org/10.1109/TIV.2022.3162672

  7. Chen Y, Liu L, Phonevilay V, Gu K, Xia R, Xie J, Zhang Q, Yang K (2021) Image super-resolution reconstruction based on feature map attention mechanism. Appl Intell 51(7):4367–4380

    Article  Google Scholar 

  8. Cuturi M (2013) Sinkhorn distances: Lightspeed computation of optimal transport. Advances in Neural Information Processing Systems 26.

  9. Dong Y, Cordonnier J-B, Loukas A (2021) Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In: International Conference on Machine Learning, pp. 2793–2803

  10. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307

    Article  Google Scholar 

  11. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: The KITTI dataset. Int J Robotics Res 32(11):1231–1237. https://doi.org/10.1177/0278364913491297

    Article  Google Scholar 

  12. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Providence, RI, USA, pp. 3354–3361

  13. Guennebaud G, Barthe L, Paulin M (2004) Real-time point cloud refinement. In: PBG, pp. 41–48

  14. Guo MH, Cai JX, Liu ZN, Mu TJ, Martin RR, Hu SM (2021) Pct: Point cloud transformer. Comput Vis Media 7(2):187–199

  15. Guo F, Ren Q, Tang J, Li Z (2022) Dilated multi-scale fusion for point cloud classification and segmentation. Multimed Tools Appl 81:6069–6090. https://doi.org/10.1007/s11042-021-11825-9

    Article  Google Scholar 

  16. Han B, Zhang X, Ren S (2022) Pu-gacnet: Graph attention convolution net- work for point cloud upsampling. Image Vis Comput 104371

  17. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Las Vegas, NV, USA, pp. 770–778

  18. Hu X, Mu H, Zhang X, Wang Z, Tan T, Sun J (2019) Meta-sr: A magnification-arbitrary network for super-resolution. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Long Beach, USA, pp. 1575–1584

  19. Huang H, Wu S, Gong M, Cohen-Or D, Ascher U, Zhang H (2013) Edge- aware point set resampling. ACM Trans Graph (TOG) 32(1):1–12

    Article  Google Scholar 

  20. Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, San Diego, CA, USA. http://arxiv.org/abs/1412.6980

  21. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  22. Li D, He K, Wang L, Zhang D (2022) Local feature extraction network with high correspondences for 3d point cloud registration. Appl Intell 52:9638–9649. https://doi.org/10.1007/s10489-021-03055-1

    Article  Google Scholar 

  23. Li R, Li X, Fu C-W, Cohen-Or D, Heng P-A (2019) Pu-gan: a point cloud upsampling adversarial network. In: Proc. IEEE/CVF International Conference on Computer Vision, Long Beach, CA, USA, pp. 7203–7212

  24. Li R, Li X, Heng P, Fu C (2021) Point cloud upsampling via disentangled refinement. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, pp. 344–353. https://openaccess.thecvf.com/content/CVPR2021/html/Li_Point_Cloud_Upsampling_via_Disentangled_Refinement_CVPR_2021_paper.html

  25. Li R, Li X, Heng P-A, Fu C-W (2021) Point cloud upsampling via dis- entangled refinement. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, pp. 344–353

  26. Lipman Y, Cohen-Or D, Levin D, Tal-Ezer H (2007) Parameterization-free projection for geometry reconstruction. ACM Trans Graph (TOG) 26(3):22

    Article  Google Scholar 

  27. Luo C, Chen Y, Wang N, Zhang Z (2019) Spectral feature transformation for person re-identification. In: Proc. IEEE/CVF International Conference on Computer Vision, Seoul, Korea, pp. 4976–4985

  28. Luo L, Tang L, Zhou W, Wang S, Yang Z (2021) PU-EVA: an edge-vector based approximation solution for flexible-scale point cloud upsampling. In: Proc. International Conference on Computer Vision, pp. 16188–16197. https://doi.org/10.1109/ICCV48922.2021.01590

  29. Mansourifar H, Bastanfard A (2011) A novel practical approach for weight manipulation of conic splines. In: 2011 Eighth International Conference Computer Graphics, Imaging and Visualization, pp. 65–69. IEEE

  30. Mansourifar H, Bastanfard A (2011) Fast conic spline data fitting of noise- free data points. In: 2011 Eighth International Conference Computer Graphics, Imaging and Visualization, pp. 59–64. IEEE

  31. Mao J, Xue Y, Niu M, Bai H, Feng J, Liang X, Xu H, Xu C (2021) Voxel transformer for 3d object detection. In: Proc. IEEE/CVF International Conference on Computer Vision, pp. 3164–3173

  32. Maturana D, Scherer S (2015) Voxnet: A 3d convolutional neural network for real-time object recognition. In: Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, Hamburg, Germany, pp. 922–928

  33. Mei Y, Fan Y, Zhou Y (2021) Image super-resolution with non-local sparse attention. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, pp. 3517–3526. https://openaccess.thecvf.com/content/CVPR2021/html/MeiImageSuper-ResolutionWithNon-LocalSparseAttentionCVPR2021paper.html

  34. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang EZ, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, Vancouver, Canada, pp. 8024–8035. https://proceedings.neurips.cc/paper_files/paper/2019/file/bdbca288fee7f92f2bfa9f7012727740-Paper.pdf. Accessed 14 Dec 2019

  35. Qi CR, Su H, Mo K, Guibas LJ (2017) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Honolulu, HI, USA, pp. 652–660

  36. Qi CR, Yi L, Su H, Guibas LJ (2017) Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Adv Neural Inf Process Syst 30

  37. Qian G, Abualshour A, Li G, Thabet A, Ghanem B (2021) Pu-gcn: Point cloud upsampling using graph convolutional networks. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, pp. 11683–11692

  38. Qian Y, Hou J, Kwong S, He Y (2020) Pugeo-net: A geometry-centric net- work for 3d point cloud upsampling. In: Proc. European Conference of Computer Vision. Lecture Notes in Computer Science, vol. 12364, pp. 752–769. https://doi.org/10.1007/978-3-030-58529-7_44

  39. Qiao C, Li D, Guo Y, Liu C, Jiang T, Dai Q, Li D (2021) Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nat Methods 18(2):194–202

    Article  CAS  PubMed  Google Scholar 

  40. Simonovsky M, Komodakis N (2017) Dynamic edge-conditioned filters in convolutional neural networks on graphs. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Honolulu, HI, USA, pp. 3693–3702

  41. Sokolova M, Japkowicz N, Szpakowicz S (2006) Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation. In: Australasian Joint Conference on Artificial Intelligence, Hobart, Australasian, pp. 1015–1021

  42. Song W, Xiao Z, Wang Y, Charlin L, Zhang M, Tang J (2019) Session- based social recommendation via dynamic graph attention networks. In: Proc. ACM International Conference on Web Search and Data Mining, Melbourne, Australia, pp. 555–563. https://doi.org/10.1145/3289600.3290989

  43. Tang H, Liu Z, Zhao S, Lin Y, Lin J, Wang H, Han S (2020) Searching efficient 3d architectures with sparse point-voxel convolution. In: Vedaldi A, Bischof H, Brox T, Frahm J (eds.) Proc. European Conference of Computer Vision. Lecture Notes in Computer Science, vol. 12373, pp. 685–702. https://doi.org/10.1007/978-3-030-58604-1_41

  44. Te G, Hu W, Zheng A, Guo Z (2018) Rgcnn: Regularized graph cnn for point cloud segmentation. In: Proc. 26th ACM International Conference on Multimedia, Seoul, Korea, pp. 746–754

  45. van Erven T, Harremoes P (2014) Renyi divergence and kullback-leibler divergence. IEEE Trans Inf Theory 60(7):3797–3820. https://doi.org/10.1109/TIT.2014.2320500

    Article  Google Scholar 

  46. Velickovic P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y (2017) Graph attention networks. arXiv preprint arXiv:1710.10903

  47. Wang H, Liu X, Kang W, Yan Z, Wang B, Ning Q (2022) Multi-features guidance network for partial-to-partial point cloud registration. Neural Comput Appl 34:1623–1634. https://doi.org/10.1007/s00521-021-06464-y

    Article  CAS  Google Scholar 

  48. Wang G, Xu G, Wu Q, Wu X (2021) Two-stage point cloud super resolution with local interpolation and readjustment via outer-product neural network. J Syst Sci Complexity 34(1):68–82

    Article  Google Scholar 

  49. Wu H, Zhang J, Huang K (2019) Point cloud super resolution with adver- sarial residual graph networks. arXiv preprint arXiv:1908.02111

  50. Xu J, Le X, Chen C, Guan X (2022) Spinet: self-supervised point cloud frame interpolation network. Neural Comput Appl. https://doi.org/10.1007/s00521-022-06939-6

    Article  PubMed  PubMed Central  Google Scholar 

  51. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  ADS  MathSciNet  PubMed  Google Scholar 

  52. Ye S, Chen D, Han S, Wan Z, Liao J (2021) Meta-pu: An arbitrary-scale upsampling network for point cloud. IEEE Trans Vis Comput Graphics 28(9):3206–3218

  53. Yu L, Li X, Fu C, Cohen Or D, Heng P (2018) Pu-net: Point cloud upsampling network. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Salt Lake City, UT, USA, pp. 2790–2799

  54. Yu X, Rao Y, Wang Z, Liu Z, Lu J, Zhou J (2021) Pointr: Diverse point cloud completion with geometry-aware transformers. In: Proc. IEEE/CVF International Conference on Computer Vision, pp. 12498–12507

  55. Zhang Z (2012) Microsoft kinect sensor and its effect. IEEE Multim 19(2):4–10. https://doi.org/10.1109/MMUL.2012.24

    Article  Google Scholar 

  56. Zhang W, Fu X, Li W (2020) 3d modeling system of lidar point cloud pro- cessing algorithm based on artificial intelligence. In: Proc. Applications and Techniques in Cyber Intelligence. Advances in Intelligent Systems and Computing, vol. 1244, pp. 764–770. Springer, Fuyang, China

  57. Zhang F, Nauata N, Furukawa Y (2020) Conv-mpn: Convolutional message passing neural network for structured outdoor architecture reconstruction. In: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognition, Seattle, WA, USA, pp. 2795–2804. https://doi.org/10.1109/CVPR42600.2020.00287

  58. Zhao Y, Hui L, Xie J (2021) Sspu-net: Self-supervised point cloud upsampling via differentiable rendering. In: Proc. Multimedia Conference, pp. 2214–2223. https://doi.org/10.1145/3474085.3475381

  59. Zolanvari SMI, Ruano S, Rana A, Cummins A, da Silva RE, Rahbar M, Smolic A (2019) Dublincity: Annotated lidar point cloud and its applications. In: Proc. British Machine Vision Conferenc, Cardiff, UK, p. 44. https://bmvc2019.org/wp-content/uploads/papers/0644-paper.pdf. Accessed 2019-09-06

Download references

Acknowledgements

This work was supported by the Yunnan Major Scien- tific and Technological Special Project under Grant 202002AD080001.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Yunnan University, Kunming, 650500, Yunnan, China

    Fan Zhong & Zhengyao Bai

Authors
  1. Fan Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhengyao Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhengyao Bai.

Ethics declarations

Conflicts of interests/Competing interests

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhong, F., Bai, Z. PSR-GAT: Arbitrary point cloud super-resolution using graph attention networks. Multimed Tools Appl 83, 26213–26232 (2024). https://doi.org/10.1007/s11042-023-16525-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16525-0

Keywords

Search

Navigation