No-reference quality index of depth images based on statistics of edge profiles for view synthesis
Introduction
Multi-view and free-viewpoint videos have become increasingly popular. As the most important technique, view synthesis has been drawing more and more attention recently. The quality of synthesized view is critical to the success of these applications. Objective quality models can be used to optimize and benchmark view synthesis algorithms, so they are highly desired for developing advanced view synthesis technique.
In view synthesis, texture and depth images are utilized to render the new viewpoint. In this process, the quality of texture image, depth image and the rendering operation are all crucial for obtaining a high quality synthesized view [2]. Generally, the distortions in texture images are straightforwardly transferred to the synthesized image. Therefore, the impact of distortions in texture images on the synthesized view is intuitive, and the traditional image quality indices can be adopted to measure the quality of texture images [19], [30], [32], [36]. Most of the existing view synthesis quality metrics are dedicated to evaluating the influence of rendering distortions on the perceptual quality of the synthesized images [1], [4], [6], [10], [13], [29]. Typically, they are tested on the IRCCyN/IVC database [2], which consists of synthesized images rendered from distortion-free texture and depth images. In practice, depth image is used to guide the warping process in view synthesis, including a forward mapping from reference view to the 3D space and an inverse mapping to the target view [18]. As a result, depth image plays a paramount role in the process of view synthesis, which has significant influence on the perceptual quality of synthesized images. Although a great number of quality assessment models have been proposed in the past few years, they are mainly designed for natural images and are not readily applicable to depth images. The underlying reason is that a pixel in depth image represents the distance between the target object and the camera in the scene, which is essentially different from natural images. The objective quality assessment of depth images remains a topic largely untouched.
Up to now, very little effort has been done on the quality evaluation of depth images. Farid et al. [7] proposed a Blind Depth Quality Metric (BDQM) for measuring the compression distortions in depth images. The BDQM model is based on the analysis of compression sensitive pixels in depth maps. Specifically, the authors found that the histogram of pixels around the depth transitions becomes flatter when the compression ratio increases. Therefore, the sharpness of the compression sensitive regions was quantified and used to evaluate the quality of depth images. Xiang et al. [33] presented a no-reference (NR) depth quality index by measuring the misalignment errors between the edges of texture and depth images. The misalignments were detected from three aspects, including the edge orientation similarity, the spatial similarity and the segment length similarity. Finally, the bad point rate (BPR) was computed as the overall quality score. Le et al. [15] proposed a reduced-reference (RR) depth quality model by jointly measuring the interactions between the local depth distortion and the local texture characteristics. Specifically, local depth distortions were first detected by calculating the difference of depth values in a local neighborhood. Then the texture image was filtered using the Gabor filter, which was then employed to weight the local depth distortions adaptively. The final quality score was computed by pooling all the local distortion values after removing the outlier regions. The above depth quality metrics have achieved notable advances in measuring the distortions in depth images. Meantime, it should be noted that the BDQM metric [7] is specifically designed for compression distortions in depth images. However, depth images may be subject to different kinds of distortions in real applications. The metrics in [15], [33] both require the corresponding texture images for measuring the quality of depth images, which are RR depth quality metrics. In practice, depth images are usually acquired by depth cameras or depth estimation algorithms, so the reference depth images are often unavailable. Therefore, NR quality metrics for depth images are urgently needed.
In this paper, we present a new NR depth image quality index by measuring the Statistics of Edge Profiles (SEP) in the scale space domain, aiming to model the impact of depth distortions on the perceptual quality of view synthesis. Since the edge regions in depth images have more significant impact on the overall quality of view synthesis [18], we first propose a method to construct the edge profiles. Then we analyze the statistical distributions of edge profiles and use the distribution parameters for building the quality evaluation model for depth images. Specifically, we model the first-order and second-order statistical characteristics of the edge profiles using Weibull Distribution and Asymmetric Generalized Gaussian Distribution (AGGD), respectively. Finally, the extracted statistical features are input into the random forest (RF) regressor for training the quality model. The performance of the proposed SEP index is verified on depth images from two public view synthesis image/video quality databases, and also compared with the relevant state-of-the-arts. The experimental results demonstrate the advantages of the proposed SEP metric in terms of both the consistency with human ratings and the generalization ability.
Section snippets
Statistical characteristics of depth images
The proposed depth quality metric is inspired by the natural scene statistics (NSS) theory, which has been frequently adopted for traditional image quality assessment (IQA) [22], [25], [27], [34]. The idea of NSS-based IQA is that high-quality natural images follow specific statistical regularities, which are prone to change under distortions. While notable success has been achieved in applying NSS in the quality evaluation of natural images, the existing NSS models are not applicable to depth
Proposed NR depth image quality metric
The proposed depth image quality metric is based on the fact that the quality of edge regions in depth images are crucial for generating high-quality synthesized view. Specifically, we have found that although the statistics of the whole depth image do not follow specific distributions, the statistical distribution of edge regions can be well modeled, and the model parameters can be effectively used for quantifying the quality of depth images. The flowchart of the proposed depth quality metric
Experimental protocols
We evaluate the performance of our proposed depth quality metric based on two public view synthesis image/video quality databases, including MCL-3D [28] and SIAT [20]. (1) The MCL-3D database is generated based on nine image-plus-depth sources, each with three viewpoints. Six different types of distortions are added to the texture and/or depth images, including Gaussian blurring, down-sampling blurring, white noise, JPEG compression, JPEG2000 compression and transmission error. For each kind of
Conclusion
In this paper, we have presented a NR quality assessment metric of depth images for view synthesis application. The proposed metric is based on the observation that the edge profiles, i.e., neighboring regions around depth image edges, follow specific statistics. Inspired by this, we have constructed edge profiles from depth images, and extract quality-aware statistical features by fitting the gradient magnitude and LoG distributions. The proposed metric operates in the scale space
Declaration of Competing Interest
The authors declare that they do not have any financial or nonfinancial conflict of interests.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (61771473 and 61379143), the Natural Science Foundation of Jiangsu Province (BK20181354), the Six Talent Peaks High-level Talents in Jiangsu Province (XYDXX-063) and the Qing Lan Project of Jiangsu Province.
References (38)
- et al.
Objective image quality assessment of 3d synthesized views
Signal Process. Image Commun.
(2015) - et al.
Subjective and objective video quality assessment of 3d synthesized views with texture/depth compression distortion
IEEE Trans. Image Process.
(2015) - et al.
Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features
IEEE Trans. Image Process.
(2014) - et al.
Towards a new quality metric for 3-d synthesized view assessment
IEEE J. Select. Top. Signal Process.
(2011) A computational approach to edge detection
IEEE Transp. Anal. Mach. Intell.
(1986)- et al.
Objective view synthesis qualityassessment
Proc. Electron. Imag. Int. Society Opt. Photon.
(2012) - et al.
Decision forests for classification, regression, density estimation, manifold learning and semisupervised learning
Comput. Gr. Vis.
(2012) - et al.
Perceptual quality assessment of 3D synthesized images
Proceedings of the IEEE International Conference Multimedia and Expo (ICME)
(2017) - et al.
Blind depth quality assessment using histogram shape analysis
Proceedings of the 3DTV-Conference: The True Vision- Capture, Transmission and Display of 3D Video (3DTV-CON)
(2015) - et al.
Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation
Proceedings of the IEEE International Conference Computer Vision (ICCV)
(2013)
A six-stimulus theory for stochastic texture
Int. J. Comput. Vis.
Model-based referenceless quality metric of 3d synthesized images using local image description
IEEE Trans. Image Process.
Quality assessment considering viewing distance and image resolution
IEEE Trans. Broadcast
Using free energy principle for blind image quality assessment
IEEE Trans. Multimed.
Critical binocular asymmetry measure for perceptual quality assessment of synthesized stereo 3d images in view synthesis
IEEE Trans. Circ. Syst. Video Technol.
Orientation-selective adaptation to first- and second-order patterns in human visual cortex
J. Neurophysiol.
A new depth image quality metric using a pair of color and depth images
Multimed. Tools Appl.
Image sharpness assessment by sparse representation
IEEE Trans. Multimed.
No-reference and robust image sharpness evaluation based on multi-scale spatial and spectral features
IEEE Trans. Multimed.
Cited by (10)
Two-stage unsupervised facial image quality measurement
2022, Information SciencesQuality assessment of DIBR-synthesized views: An overview
2021, NeurocomputingCitation Excerpt :Finally, the misalignments are used to calculate the final quality scores. Li et al. proposed a NR depth image quality index based on the statistics of edge profiles (SEP) [58]. The first-order and second-order statistical features are firstly extracted based on edge profiles which are the neighbouring regions around the depth edges.
Depth Perception Assessment of 3D Videos Based on Stereoscopic and Spatial Orientation Structural Features
2023, IEEE Transactions on Circuits and Systems for Video TechnologyA Novel Reduced Reference Image Quality Assessment Based on Formal Concept Analysis
2023, Computer JournalParallel depth buffer algorithm based on a ternary optical computer
2022, Applied OpticsQuality Assessment of DIBR-Synthesized Views Based on Sparsity of Difference of Closings and Difference of Gaussians
2022, IEEE Transactions on Image Processing