Skip to main content
SpringerLink
Log in

SPINet: self-supervised point cloud frame interpolation network

  • S.I. :Interpretation of Deep Learning
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

For autonomous vehicles, the acquisition frequency difference between LiDAR (10–20 Hz) and camera (over 100 Hz) makes simultaneous update of two perceptive systems (2D/3D) less efficient. Nowadays, frame interpolation is in urgent need for increasing frame rate of point cloud sequences obtained by LiDAR. However, a major limitation of current full supervised methods is that high frame rate ground truth sequences are hard to access. We propose a novel Self-supervised Point Cloud Frame Interpolation Network (SPINet) accommodating with variable motion situation, retaining geometric consistency, but without the necessity of utilizing G.T. data. Extensive experiments show that our proposed SPINet outperforms the current full supervised methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Qi CR, Liu W, Wu C, Su H, Guibas LJ (2018) Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 918–927

  2. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention, pp 424–432

  3. Wang G, Wu X, Liu Z, Wang H (2021) Pwclo-net: Deep lidar odometry in 3d point clouds using hierarchical embedding mask optimization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15910–15919

  4. Liu H, Liao K, Lin C, Zhao Y, Liu M (2020) Plin: a network for pseudo-lidar point cloud interpolation. Sensors 20(6):1573

    Article  Google Scholar 

  5. Lu F, Chen G, Qu S, Li, Z, Liu Y, Knoll A (2020) Pointinet: point cloud frame interpolation network. arXiv preprint arXiv:2012.10066

  6. Liu X, Qi C.R, Guibas L.J (2019) Flownet3d: learning scene flow in 3d point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 529–537

  7. Huang Z, Yu Y, Xu J, Ni F, Le X (2020) Pf-net: point fractal network for 3d point cloud completion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7662–7670

  8. Yu Y, Huang Z, Li F, Zhang H, Le X (2020) Point encoder gan: a deep learning model for 3d point cloud inpainting. Neurocomputing 384:192–199

    Article  Google Scholar 

  9. Zhao B, Le X, Xi J (2019) A novel sdass descriptor for fully encoding the information of a 3d local surface. Inf. Sci. 483:363–382

    Article  Google Scholar 

  10. Wang Z, Li S, Howard-Jenkins H, Prisacariu V, Chen M (2020) Flownet3d++: Geometric losses for deep scene flow estimation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 91–98

  11. Gu X, Wang Y, Wu C, Lee YJ, Wang P (2019) Hplflownet: hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3254–3263

  12. Wu W, Wang Z, Li Z, Liu W, Fuxin L (2019) Pointpwc-net: a coarse-to-fine network for supervised and self-supervised scene flow estimation on 3d point clouds. arXiv preprint arXiv:1911.12408

  13. Mittal H, Okorn B, Held D (2020) Just go with the flow: self-supervised scene flow estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11177–11185

  14. Puy G, Boulch A, Marlet R (2020) Flot: scene flow on point clouds guided by optimal transport. arXiv preprint arXiv:2007.11142

  15. Wang G, Wu X, Liu Z, Wang H (2021) Hierarchical attention learning of scene flow in 3d point clouds. IEEE Trans Image Process 30:5168–5181

    Article  Google Scholar 

  16. Yang J, Zhang Q, Ni B, Li L, Liu J, Zhou M, Tian Q (2019) Modeling point clouds with self-attention and gumbel subset sampling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3323–3332

  17. Wang L, Huang Y, Hou Y, Zhang S, Shan J (2019) Graph attention convolution for point cloud semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10296–10305

  18. Li R, Li X, Fu C.-W, Cohen-Or D, Heng P.-A (2019) Pu-gan: a point cloud upsampling adversarial network. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7203–7212

  19. Wei Y, Wang Z, Rao Y, Lu J, Zhou J (2021) Pv-raft: point-voxel correlation fields for scene flow estimation of point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6954–6963

  20. Liu H, Liao K, Lin C, Zhao Y, Guo Y (2021) Pseudo-lidar point cloud interpolation based on 3d motion representation and spatial supervision. IEEE Trans Intell Transp Syst

  21. Tishchenko I, Lombardi S, Oswald M.R, Pollefeys M (2020) Self-supervised learning of non-rigid residual flow and ego-motion. In: 2020 international conference on 3D vision (3DV), pp 150–159

  22. Pontes J.K, Hays J, Lucey S (2020) Scene flow from point clouds with or without learning. In: 2020 international conference on 3D vision (3DV), pp 261–270

  23. Li R, Lin G, Xie L (2021) Self-point-flow: self-supervised scene flow estimation from point clouds with optimal transport and random walk. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15577–15586

  24. Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3354–3361

  25. Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) Nuscenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11621–11631

  26. Menze M, Geiger A (2015) Object scene flow for autonomous vehicles. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3061–3070

  27. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst, 8026–8037

  28. Fan H, Su H, Guibas L.J (2017) A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 605–613

Download references

Acknowledgements

The work described in the paper is supported part by the National Key R&D Program of China (No. 2021YFB1716000) and the National Natural Science Foundation of China (No. 62176152).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Jiawen Xu

  2. Department of Automation, Shanghai Jiao Tong University, Shanghai, 200240, China

    Xinyi Le, Cailian Chen & Xinping Guan

  3. Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, 200240, China

    Xinyi Le, Cailian Chen & Xinping Guan

  4. Shanghai Engineering Research Center of Intelligent Control and Management, Shanghai, 200240, China

    Xinyi Le, Cailian Chen & Xinping Guan

Authors
  1. Jiawen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinyi Le
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cailian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinping Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinyi Le.

Ethics declarations

Conflict of interest

The authors of the manuscript confirm that there are no conflicts of interest regarding its publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, J., Le, X., Chen, C. et al. SPINet: self-supervised point cloud frame interpolation network. Neural Comput & Applic 35, 9951–9960 (2023). https://doi.org/10.1007/s00521-022-06939-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-06939-6

Keywords

Search

Navigation