PI-Net: An End-to-End Deep Neural Network for Bidirectionally and Directly Fusing Point Clouds With Images | IEEE Journals & Magazine | IEEE Xplore

PI-Net: An End-to-End Deep Neural Network for Bidirectionally and Directly Fusing Point Clouds With Images


Abstract:

We present a novel network, PI-Net, for the fusion between point clouds and images in this letter. Most existing fusion methods project point clouds into pseudo images an...Show More

Abstract:

We present a novel network, PI-Net, for the fusion between point clouds and images in this letter. Most existing fusion methods project point clouds into pseudo images and then fuse the pseudo and RGB images with 2D CNNs. To get rid of structuring the pseudo images as the preprocessing, we propose a new fusion module, PI-fusion module. It can directly fuse points with pixels or grids which refer to the cells in the feature maps extracted from RGB images. Specifically, every point is aligned with a certain pixel/grid based on the pre-calibrated camera-LiDAR external parameters. The features aggregated from the points and pixels/grids are transferred to each other, which achieves the bidirectional fusion of the information. Then, we construct the PI-Net's fusion backbone by plugging the PI-fusion modules between a certain 2D CNN and a point-based network to fuse the features at every scale. The detection and the segmentation head networks further utilize the fused information to complete various tasks. The PI-Net unifies the point-cloud-to-image and image-to-point-cloud fusion into one network and achieves state-of-the-art results on the various tasks, such as the road detection, 3D object detection, and point cloud segmentation tasks on the KITTI benchmark.
Published in: IEEE Robotics and Automation Letters ( Volume: 6, Issue: 4, October 2021)
Page(s): 8647 - 8654
Date of Publication: 22 September 2021

ISSN Information:

Funding Agency:


Contact IEEE to Subscribe

References

References is not available for this document.