Stereoscopic saliency model using contrast and depth-guided-background prior

Abstract

Many successful models of saliency have been proposed to detect salient regions for 2D images. Because stereopsis, with its distinctive depth information, influences human viewing, it is necessary for stereoscopic saliency detection to consider depth information as an additional cue. In this paper, we propose a 3D stereoscopic saliency model based on both contrast and depth-guided-background prior. First, a depth-guided-background prior is specifically detected from a disparity map apart from the conventional prior, assuming boundary super-pixels as background. Then, saliency based on disparity with the help of the proposed prior is proposed to prioritize the contrasts among super-pixels. In addition, a scheme to combine the contrast of disparity and the contrast of color is presented. Finally, 2D spatial dissimilarity features are further employed to refine the saliency map. Experimental results on the PSU stereo saliency benchmark dataset (SSB) show that the proposed method performs better than existing saliency models.

Introduction

The human visual system (HVS) is miraculous in determining the important aspects present in the visual when the eyes acquire information. To simulate the HVS mechanism, two distinct saliency models are applied: bottom-up (stimulus-driven) and top-down (task-driven) [1], [2], [3], [4]. Bottom-up models extract the center-surrounding difference (local or global) features with low-level information in different channels and then combine these feature maps to build a saliency map [5], [6], [7]. Though many image and video saliency models are used in vision tasks [8], [9], [10], [11], [12], with the development of 3D technologies, stereoscopic visual saliency models have attracted increasing attention for diverse applications such as synthetic vision [13], retargeting [14], rendering [15], quality assessment [16], [17], visual discomfort [18], [19], stereoscopic thumbnail creation [20] and disparity control [21]. However, thus far, research [22], [23], [24] that focuses on stereoscopic saliency models is lacking when compared with the rapid expansion of saliency models for 2D images. Overall, much work remains to be done in 3D stereoscopic saliency model research before it can approach the capabilities of the human visual system.

Because the sensation of impression [25] is enhanced by the binocular parallax generated by a stereo-channel-separated display, binocular depth cues introduced for 3D displays have changed human viewing behavior [26], [27]. Therefore, this additional depth cue should be considered in a stereoscopic saliency model. Although current bottom-up stereoscopic saliency models are effective, questions remain that must be addressed:

• * How can a salient object be detected in a scene where the colors are similar? Sometimes, the colors of both salient and non-salient objects are similar in natural images; therefore, methods based on color contrast tend to obtain low values inside salient objects. In this situation, disparity cues could help to complete the whole salient object (e.g., the first row in Fig. 1(e)–(g)). Color and disparity are both useful in computing saliency and are, to some extent, relevant to each other. Because the interaction of disparity and 2D information is ignored, we attempt to introduce compactness [28] from the color map to measure the relationship while calculating contrast (e.g., the first row in Fig. 1(d)).

• * How can a salient object be detected in a scene with a cluttered background? Current bottom-up saliency methods have difficulties in coping with images where the non-salient part of the image is cluttered (e.g., the second row of Fig. 1(a)). Previous models sometimes render non-salient parts as salient because of their high contrast to the surroundings (e.g., the second row in Fig. 1(e) and (f)). This phenomenon may be avoided by using a disparity map (e.g., Fig. 1(g)). Although boundary background prior [29], [30] has been verified as effective, we find that a prior could be exploited by a disparity map based on two observations: 1) the non-salient areas distant from viewers tend to form a smooth surface on a disparity map; 2) there is an obvious discontinuity between the salient and non-salient parts. Hence, taking multiple cues into consideration, including compactness and background priors, can be useful in forming a complete stereoscopic saliency analysis (e.g., the second row in Fig. 1(d)).

• * How can a salient object be detected according to the disparity cue in a cluttered scene? For a simple scene, a good saliency map shown in the first row in Fig. 1(g), could be obtained by a disparity map shown in the first row in Fig. 1(c)). However, the saliency detection method cannot cope with one image very well, especially in a cluttered scene shown in the second row in Fig. 1(g). Some regions in background are defined as high saliency values due to being close to salient ones in depth. Then, color feature (compactness [28]), and background priors may be considered together, in this way, non-salient background could be suppressed to a certain degree (e.g., Fig. 1(d)).

In this paper, we propose a united stereoscopic saliency model, called Saliency with Contrast and Depth-Guided-Background (SCDGB), which combines the saliency obtained from a disparity map with low-level contrast. Saliency based on disparity takes advantage of compactness and background priors to not only highlight the salient object but also eliminate non-salient objects. As for background priors, the depth-guided-background prior is specially explored on a disparity map in addition to the conventional boundary background prior [29]. The contrasts are composed of color contrast and disparity contrast measured by compactness. Moreover, the 2D saliency map and the center-bias preference of human vision are also employed to refine the final saliency map. The results of experiments show that the proposed stereoscopic saliency model achieves superior performance compared with existing saliency approaches. The contributions of this paper are as follows:

• We propose a stereoscopic saliency model that unites contrast and saliency based on disparity.

• We develop a saliency based on disparity using the proposed depth-guided-background prior.

• We present a strategy to represent the contrast of stereoscopic images by fusing the multichannel contrasts.

The rest of this paper is organized as follows. A review of the related saliency models is given in Section 2. The proposed stereoscopic saliency detection model is elaborated in Section 3. An experiment and an evaluation of the proposed model are presented in Section 4. Conclusions are provided in Section 5.