PCNet: Paired channel feature volume network for accurate and efficient depth estimation

Abstract

Most state-of-the-art deep learning based depth estimation methods follow the pipeline of firstly forming a 4D cost volume (feature dimension, max disparity, height, and width) and then regressing disparity from the cost volume by several 3D convolutional layers. Applying 3D operations on the 4D tensor leads to unacceptable computational complexity and memory cost. To solve the problem, we aim at replacing the 4D cost volume with 3D cost volume so that the disparity can be regressed by 2D convolutions to achieve a good balance between efficiency and effectiveness. To this end, a light-weighted network, called PCNet, is proposed to generate 3D cost volume. The main novelty lies in the proposed Paired Channel Feature Volume (PCFV) which is capable of combining the features of stereo pairs with specially designed 3D filters to preliminarily encode the relationship between each pair of the channels. Moreover, a densely connected aggregation on the outputs of PCFV is performed to exploit much richer contextual information. Experimental results on the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate that the proposed PCNet achieves comparable accuracy with state-of-the-art methods and keeps high efficiency as well.

Introduction

Depth estimation is a fundamental and important task in field of computer vision. Accurate depth sensing plays an important role in applications such as odometry, robot navigation, and autonomous driving. Meanwhile, the depth information is also helpful in high-level object detection and semantic segmentation tasks [12], [13], [14], [15], [16], [17].

The depth estimation methods can be divided into two categories: active depth estimation and passive depth estimation. In active depth estimation, depth information is generated by active depth sensor, such as MMW radar (millimeter-wave radar) and LiDAR (Light Detection and Ranging) which are widely used in autonomous driving. Depth map can be extracted by the point clouds generated by active depth sensor. As for passive depth estimation, depth information is generated by passive depth sensor, such as stereo camera. Unlikely active depth estimation methods, stereo-based depth estimation methods gain advantage on low cost and rich contextual information which makes it a common solution to deal with depth estimation problem.

With the development of deep learning, great progress has been achieved in computer vision[1], [2], [3], [18], [19], [20], [21], [7], [6], [8], [9], [10], [11]. CNN (Convolutional Neural Network), as the most representative method of deep learning, has been introduced to stereo matching task to achieve good results on both accuracy and efficiency. Existing CNN based stereo methods can be classified into feature based methods and cost volume based methods. In feature based methods, disparity information is directly predicted by the deep CNN without special design. However, feature based methods fail in complex scene because structural information of the stereo inputs is not taken into account. In the recent few years, the cost volume based methods follow the pipeline: feature extractor, cost volume construction, cost aggregation, and refinement. The cost volume based methods achieve promising results even in challenging situations by taking the geometric constraints in stereo matching into account. However, the cost volume based methods suffer high computational cost compared with feature based methods for the construction of cost volume.

Two typical cost volumes are summarized in Fig. 1. Fig. 1(a) shows the concat based cost volume [22], [23], [24]. Left and right features are aligned in $h$ dimension and slide in $w$ dimension from 0 to $D$ ($D$ is the max disparity to predict). Then the overlap parts of the stereo features are concatenated to obtain a 4D cost volume with the size of $[D,2C,H,W]$. However, the cost volume based methods suffer from high computational cost because 3D convolutional operations are involved in the aggregation module. The correlation based cost volume [25], [26] is shown in Fig. 1(b). Left and right features are aligned in $h$ dimension and slide in $w$ dimension from 0 to $D$. Then the overlap parts of left and right features are fused by inner product to obtain $D$ tensors. Finally, the averages of all the tensors in channel dimension are concatenated together to obtain a 3D cost volume with the size of $[D,H,W]$. The correlation based cost volume gains advantage on efficiency but lacks of contextual information which results in low accuracy.

To balance the accuracy and efficiency, a Paired Channel Feature Volume Network (PCNet) is proposed. The key of the proposed method is a novel Paired Channel Feature Volume (PCFV) module (Fig. 1(c)). The left and right feature maps are interweaved in channel dimension. Then small 3D convolutional filters, called Paired Channel 3D Convolution, are introduced to learn relations between each channel-paired feature maps. Finally, the averages of the feature volume in channel dimension are computed to generate the Paired Channel Feature Volume. Details of all the steps of the proposed PCFV are summarized in Alg. 1. Because the Paired Channel Feature Volume is a 3D tensor, we use 2D convolutions in the aggregation module and the refinement module instead of 3D convolutions. The PCFV module can avoid high computational cost caused by 3D convolutional operations.

The contributions of the paper can be summarized as follows:

• A novel feature volume module called PCFV module is proposed to describe the relations between stereo feature maps. Compared with 4D cost volume, the 3D tensor generated by PCFV module can be handled by light-weighted aggregation module and refinement module.

• A densely connected aggregation module is proposed to exploit abundant contextual information and enhance the feature representation.

• Experiments are conducted on one synthetic dataset (SceneFlow Dataset [28]) and two real-world driving datasets (KITTI 2012 Dataset [29] and KITTI 2015 Dataset [30]). The proposed PCNet achieves state-of-the-art accuracy with high efficiency.