Binocular energy response based quality assessment of stereoscopic images

Abstract

Perceptual quality assessment in three-dimensional (3D) is challenging. In this research, we propose a binocular energy response based quality assessment metric of stereoscopic images. To be more specific, we first construct binocular energy responses of the original and distorted images, and measure the similarity between them as Image Quality Metric (IQM). Then, the binocular response and the binocular masking components are used to modulate the IQM, respectively. Finally, two evaluation results are nonlinearly integrated into an overall score by considering the importance of each component. Experimental results show that compared with the relevant existing metrics, the proposed metric can achieve higher consistency with the subjective assessment of stereoscopic image.

Introduction

Three-dimensional (3D) imaging technologies have been researched widely recently [1], the application of which ranges from content creation, video coding, network transmission and stereoscopic display. Therefore, designing for objective perceptual 3D image quality assessment (3D-IQA) approach is increasingly important [2], since such perceptual issue is hardly considered in the traditional 2D image quality assessment (2D-IQA). Following the research of 2D-IQA, 3D-IQA approaches fall into two categories: subjective assessment and objective assessment. Specifically, the development of objective quality assessment models has been a fruitful area of work.

In the aspect of objective/subjective assessment, the term ‘quality of experience (QoE)’ should be considered to capture the various factors that contribute to the overall visual experience of the 3D visual signal [3]. In contrast to the 2D case, QoE of 3D involves not only evaluating 2D image quality, but also additional aspects of quality, e.g., depth perception, visual comfort, and other visual experience. Many 2D-IQA metrics were proposed during the last decade, such as Structural SIMilarity (SSIM) [4], visual signal-to-noise ratio (VSNR) [5], visual information fidelity (VIF) [6], etc. However, the direct use of 2D-IQA in measuring 3D-IQA may not be straightforward, since the above 3D perceptual attributes needed to be considered. Lambooij et al. constructed a 3D quality model as a weighted sum of 2D image quality and perceived depth, and the model was validated by subjective experiments [7]. Chen et al. explored 3D QoE by constructing the visual experience as a weight sum of image quality, depth quantity and visual comfort, and subjective experiments were conducted to test the model [8]. However, these methods remained on a subjective level to explore the combination of various perceptual scales.

Currently, some publicly available 3D databases were provided, such as LIVE 3D IQA database [9], EPFL 3D image database [10], IRCCyN/IVC 3D image database [11], etc., by adding different types of stimuli (e.g., image distortion or camera distance) on both left and right images. Some objective 3D-IQA metrics were proposed by verifying on the databases. Research on objective 3D-IQA can be divided into two categories based on the involved information for evaluation. The most direct use of state-of-the-art 2D-IQA approaches in 3D-IQA is to evaluate the two views of the stereoscopic images, disparity/depth images separately by 2D metrics, and then combined into an overall score. Benoit et al. presented a linear combination solution for disparity distortion and 2D image quality on both views [11]. You et al. integrated the disparity information into quality assessment, and investigated the capabilities of some combination schemes [12]. Ha et al. designed a quality assessment method by considering the factors of temporal variation and disparity distribution [13]. Hewage et al. performed the evaluation for 3D video by using the extracted information from the depth maps and color images [14]. Obviously, it is not effective to assess the quality of perceived depth using image quality assessment methods (e.g., SSIM), because stimuli toward perceived depth are different with those for 2D image quality.

From another point of view, visual perceptual properties (e.g., monocular and binocular properties) were other important cues in 3D-IQA. Maalouf et al. computed the ‘Cyclopean’ image from left and right images to simulate the brain perception, and used contrast sensitivity coefficients of cyclopean image as the basis of evaluation [15]. Lin et al. utilized binocular integration (i.e., binocular combination and the binocular frequency integration) behaviors as the bases for measuring the quality of stereoscopic 3D images [16]. Ryu et al. formulated a model for stereoscopic images based on binocular perception model considering asymmetric properties of stereoscopic images [17]. Ko et al. proposed a structural distortion parameter based binocular perception model for 3D image quality assessment [18]. Wang et al. proposed a metric by considering the binocular spatial sensitivity to reflect the binocular fusion and suppression properties [19]. Bensalma et al. proposed a Binocular Energy Quality Metric (BEQM) by modeling the simple cells responsible for the local spatial frequency analysis and the complex cells responsible for the generation of the binocular energy [20]. However, these methods are simple extensions of the monocular visual properties into the binocular vision, and how these monocular visual properties affect the binocular vision is still not accounted.

From the observation of the existing 3D-IQA metrics, both image quality and depth perception are expected to measure the QoE of 3D. However, the combination of these two parts is somewhat ill-defined in the existing metrics, since disparity map is estimated from the stereoscopic image, while the perception of depth from disparity is generally not well understood. In order to tackle the problem, we propose a binocular energy response based stereoscopic image quality assessment metric in this paper. The main contributions of this work are as follows: (1) We construct binocular energy responses of the original and distorted images based on Gabor filters and disparity information, and measure the similarity between them as Image Quality Metric (IQM); (2) By considering the binocular response and binocular masking characteristics, we use the binocular energy and the binocular just noticeable difference (BJND) components to modulate the IQM, respectively; (3) Two evaluation results are nonlinearly integrated into an overall score by considering the importance of each component. The rest of the paper is organized as follows. Section 2 discusses the background and motivation of the proposed metric. Section 3 presents the proposed quality assessment metric. The experimental results are analyzed in Section 4, and finally conclusions are drawn in Section 5.