Content-Aware Feature Upsampling for Voxel-Based 3D Semantic Segmentation

Abstract

Voxel-based sparse convolutional networks(sparse CNNs) are widely used in 3D point cloud semantic segmentation. In particular, feature upsampling, as one of the fundamental operations in the sparse CNNs, has been under-explored compared with other basic operations such as sparse convolution and pooling. Therefore, we dive deep into this area and focus on the upsampling design in sparse CNNs. 3D sparse deconvolution is the most representative feature unsampling in sparse CNNs. However, it applies the same kernel across the point cloud, regardless of the content of each point. To this end, we propose 3D Content-Aware Feature Upsampling(3DCAFU), a universal and effective module beyond sparse deconvolution in sparse CNNs. 3DCAFU has three appealing properties: (1) Content-aware processing. Instead of a fixed kernel for the point cloud feature, 3DCAFU generates point-wise kernels specific to each point for adaptive upsampling. (2) Context aggregation. Since the generation of the point-wise kernels aggregates the context of local neighborhoods, it makes the upsampled feature of 3DCAFU contain richer semantic information compared with sparse deconvolution. (3) Lightweight and efficient. 3DCAFU introduces little extra parameters and accelerates the computation on GPUs by gather-scatter paradigm. Extensive experiments on the SemanticKITTI, SemanticPOSS, nuScenes, and Waymo benchmarks validate the effectiveness of our approach. For instance, it outperforms the baseline by 1.7$\%$ mIoU in the SemanticKITTI dataset. SphereFormer with 3DCAFU has achieved state-of-the-art performance among voxel-based methods for 3D semantic segmentation. The code will be made publicly available soon.