SRI-Net: Similarity retrieval-based inference network for light field salient object detection

Highlights

• We deploy the FSRM to explicitly dig complementary information from all focal slices.

• The similarity computation is utilized to retrieve the most valuable focal slice.

• To aggregate the retrieved focal slice and the RGB image, we deploy the two-stream SIM to extract and fuse the deep features of them.

Abstract

The cutting-edge RGB saliency models are prone to fail for some complex scenes, while RGB-D saliency models are often affected by inaccurate depth maps. Fortunately, light field images can provide a sufficient spatial layout depiction of 3D scenes. Therefore, this paper focuses on salient object detection of light field images, where a Similarity Retrieval-based Inference Network (SRI-Net) is proposed. Due to various focus points, not all focal slices extracted from light field images are beneficial for salient object detection, thus, the key point of our model lies in that we attempt to select the most valuable focal slice, which can contribute more complementary information for the RGB image. Specifically, firstly, we design a focal slice retrieval module (FSRM) to choose an appropriate focal slice by measuring the foreground similarity between the focal slice and RGB image. Secondly, in order to combine the original RGB image and the selected focal slice, we design a U-shaped saliency inference module (SIM), where the two-stream encoder is used to extract multi-level features, and the decoder is employed to aggregate multi-level deep features. Extensive experiments are conducted on two widely used light field datasets, and the results firmly demonstrate the superiority and effectiveness of the proposed SRI-Net.

Introduction

Salient object detection (SOD) attempts to imitate human visual attention, and it is used to highlight the most attractive objects in the scene. It can be treated as a pre-processing step of many computer vision tasks, such as visual tracking [1], image retrieval [2], image/video segmentation [3], image/video compression [4], [5], and so on.

Generally, according to the input type of saliency models, the salient object detection models can be divided into 2D (RGB), 3D (RGB-D), 4D (light field). In the early stage, many traditional models were developed based on the heuristic prior [6], [7] and the traditional machine learning models [8], [9]. In recent years, convolutional neural networks (CNN) [10] have been widely used in computer vision applications. Especially fully convolutional neural networks (FCN) [11] largely promote salient object detection performance [12], [13], [14], [15]. However, there are also many shortcomings in 2D and 3D models, among which 2D models cannot accurately segment the salient object when encountering the image with low contrast and complex backgrounds. For the 3D models, they often present failure cases when confronting low-quality depth maps of RGB-D scenes. Fortunately, the light field imaging can effectively capture the spatial layout information of the scene, where the focal slices are treated as one type of light field data and can be obtained by adjusting the focal length of camera. According to Fig. 1, it can be seen that the light field data consists of a sequence of focal slices which focus on the different depths of scene. Thus, the light field salient object detection has attracted more and more attention. Over the past few decades, the light field saliency detection has been pushed forward significantly. For example, in [16], Zhang et al. tried to imitate the human memory mechanism to fuse information and excavate spatial correlations of focal slices. Piao et al. [17] designed asymmetrical two-stream architecture to exploit the focal slices and produce focal knowledge tailored for the student network. Zhang et al. [18] implemented the feature fusion of the focal stack by using 3D convolution [19] and aggregated the RGB feature with the fused focal stack feature by deploying the synergistic attention module. We can find that most light field saliency models try to aggregate all focal slices, which largely promotes the performance of light field salient object detection. However, the aggregation of all focal slices will inevitably introduce redundant information [17], [18]. Especially, each focal slice focuses on different depths of the scene, salient objects in some focal slices may be out of focus, and thus the stacked focal slices may introduce redundancy information which will interfere with the concerns on salient regions. Meanwhile, we notice that the focal slice can provide complementary information for the RGB image. Therefore, we attempt to select the most valuable focal slice, which has the similar concerns on foreground regions as the RGB image. In this way, we can filter out the disturbances from other focal slices, and pay more concerns on salient regions by fusing features from the retrieved focal slice and the RGB image.

Motivated by the descriptions above, we propose a novel Similarity Retrieval-based Inference Network (SRI-Net) for light field salient object detection, as shown in Fig. 2, which contains the focal slice retrieval module (FSRM) and the saliency inference module (SIM). To be specific, firstly, some focal slices are with interfering information, leading to the performance degradation of saliency models. In contrast, the most valuable focal slice can provide enough salient cues, and we can retrieve it from all focal slices by deploying FSRM shown in Fig. 3. At the beginning, the RGB image ${\mathbf{I}}^R$ is fed into the coarse saliency map prediction network, which is used to generate the coarse saliency map ${\mathbf{S}}_o$. After that, the coarse saliency map is used to guide the focal slice retrieval process. Concretely, the RGB image ${\mathbf{I}}^R$ and all focal slices ${\left\{{\mathbf{I}}_i^F\right\}}_{i=1}^{12}$ multiply by the coarse saliency map ${\mathbf{S}}_o$, where both kinds of images only contain the foreground regions. Subsequently, to measure the similarity between the RGB image and each focal slice, we compute the mean absolute error (MAE) value between them. Here, the minimal MAE value corresponds to the most valuable focal slice ${\mathbf{I}}^F$, which focuses on the foreground regions and thus presents a clear appearance of salient objects along with a blurry background. Following this way, the retrieved focal slice ${\mathbf{I}}^F$ provides prominent contrast information (namely clear foreground and blurry background) to the RGB image ${\mathbf{I}}^R$, complementing the RGB image.

Secondly, after retrieving the effective focal slice, the RGB image ${\mathbf{I}}^R$ and the retrieved focal slice ${\mathbf{I}}^F$ are fed into a two-stream encoder–decoder network shown in Fig. 2. Concretely, we first deploy the encoder to extract the multi-scale deep features from the RGB image and the retrieved focal slice, and then fuse the two types of deep features. After that, the decoder tries to progressively integrate the fused deep feature into the final saliency map.

Overall, our main contributions are summarized as follows:

• We propose a novel light field salient object detection model, namely Similarity Retrieval-based Inference Network (SRI-Net), where the proposed SRI-Net consists of the focal slice retrieval module (FSRM) and the saliency inference module (SIM).

• To sufficiently dig the complementary information from the focal slices, we deploy the FSRM to retrieve the most valuable focal slice by comparing the similarity of foreground regions between the RGB image and each focal slice, where the selected focal slice is with a strong contrast between clear salient regions and blurred backgrounds.

• To aggregate the retrieved focal slice and the RGB image, we deploy the two-stream SIM to extract and fuse the deep features from the images. The generated saliency maps can highlight the salient objects completely and are with clear boundary details.

The remaining of this article is organized as follows. In Section 2, the related works about salient object detection are briefly reviewed. A detailed description of the proposed model is given in Section 3. Comprehensive experiments and detailed analyses are reported in Section 4. Finally, we give a conclusion in Section 5.