Abstract
For autonomous vehicles, the acquisition frequency difference between LiDAR (10–20 Hz) and camera (over 100 Hz) makes simultaneous update of two perceptive systems (2D/3D) less efficient. Nowadays, frame interpolation is in urgent need for increasing frame rate of point cloud sequences obtained by LiDAR. However, a major limitation of current full supervised methods is that high frame rate ground truth sequences are hard to access. We propose a novel Self-supervised Point Cloud Frame Interpolation Network (SPINet) accommodating with variable motion situation, retaining geometric consistency, but without the necessity of utilizing G.T. data. Extensive experiments show that our proposed SPINet outperforms the current full supervised methods.







Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Qi CR, Liu W, Wu C, Su H, Guibas LJ (2018) Frustum pointnets for 3d object detection from rgb-d data. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 918–927
Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention, pp 424–432
Wang G, Wu X, Liu Z, Wang H (2021) Pwclo-net: Deep lidar odometry in 3d point clouds using hierarchical embedding mask optimization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15910–15919
Liu H, Liao K, Lin C, Zhao Y, Liu M (2020) Plin: a network for pseudo-lidar point cloud interpolation. Sensors 20(6):1573
Lu F, Chen G, Qu S, Li, Z, Liu Y, Knoll A (2020) Pointinet: point cloud frame interpolation network. arXiv preprint arXiv:2012.10066
Liu X, Qi C.R, Guibas L.J (2019) Flownet3d: learning scene flow in 3d point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 529–537
Huang Z, Yu Y, Xu J, Ni F, Le X (2020) Pf-net: point fractal network for 3d point cloud completion. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7662–7670
Yu Y, Huang Z, Li F, Zhang H, Le X (2020) Point encoder gan: a deep learning model for 3d point cloud inpainting. Neurocomputing 384:192–199
Zhao B, Le X, Xi J (2019) A novel sdass descriptor for fully encoding the information of a 3d local surface. Inf. Sci. 483:363–382
Wang Z, Li S, Howard-Jenkins H, Prisacariu V, Chen M (2020) Flownet3d++: Geometric losses for deep scene flow estimation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 91–98
Gu X, Wang Y, Wu C, Lee YJ, Wang P (2019) Hplflownet: hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3254–3263
Wu W, Wang Z, Li Z, Liu W, Fuxin L (2019) Pointpwc-net: a coarse-to-fine network for supervised and self-supervised scene flow estimation on 3d point clouds. arXiv preprint arXiv:1911.12408
Mittal H, Okorn B, Held D (2020) Just go with the flow: self-supervised scene flow estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11177–11185
Puy G, Boulch A, Marlet R (2020) Flot: scene flow on point clouds guided by optimal transport. arXiv preprint arXiv:2007.11142
Wang G, Wu X, Liu Z, Wang H (2021) Hierarchical attention learning of scene flow in 3d point clouds. IEEE Trans Image Process 30:5168–5181
Yang J, Zhang Q, Ni B, Li L, Liu J, Zhou M, Tian Q (2019) Modeling point clouds with self-attention and gumbel subset sampling. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3323–3332
Wang L, Huang Y, Hou Y, Zhang S, Shan J (2019) Graph attention convolution for point cloud semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10296–10305
Li R, Li X, Fu C.-W, Cohen-Or D, Heng P.-A (2019) Pu-gan: a point cloud upsampling adversarial network. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7203–7212
Wei Y, Wang Z, Rao Y, Lu J, Zhou J (2021) Pv-raft: point-voxel correlation fields for scene flow estimation of point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6954–6963
Liu H, Liao K, Lin C, Zhao Y, Guo Y (2021) Pseudo-lidar point cloud interpolation based on 3d motion representation and spatial supervision. IEEE Trans Intell Transp Syst
Tishchenko I, Lombardi S, Oswald M.R, Pollefeys M (2020) Self-supervised learning of non-rigid residual flow and ego-motion. In: 2020 international conference on 3D vision (3DV), pp 150–159
Pontes J.K, Hays J, Lucey S (2020) Scene flow from point clouds with or without learning. In: 2020 international conference on 3D vision (3DV), pp 261–270
Li R, Lin G, Xie L (2021) Self-point-flow: self-supervised scene flow estimation from point clouds with optimal transport and random walk. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 15577–15586
Geiger A, Lenz P, Urtasun R (2012) Are we ready for autonomous driving? the kitti vision benchmark suite. In: 2012 IEEE conference on computer vision and pattern recognition, pp 3354–3361
Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) Nuscenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11621–11631
Menze M, Geiger A (2015) Object scene flow for autonomous vehicles. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3061–3070
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L et al (2019) Pytorch: An imperative style, high-performance deep learning library. Adv Neural Inf Process Syst, 8026–8037
Fan H, Su H, Guibas L.J (2017) A point set generation network for 3d object reconstruction from a single image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 605–613
Acknowledgements
The work described in the paper is supported part by the National Key R&D Program of China (No. 2021YFB1716000) and the National Natural Science Foundation of China (No. 62176152).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors of the manuscript confirm that there are no conflicts of interest regarding its publication.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xu, J., Le, X., Chen, C. et al. SPINet: self-supervised point cloud frame interpolation network. Neural Comput & Applic 35, 9951–9960 (2023). https://doi.org/10.1007/s00521-022-06939-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-022-06939-6