Elsevier

Information Sciences

Volume 516, April 2020, Pages 205-219
Information Sciences

No-reference quality index of depth images based on statistics of edge profiles for view synthesis

https://doi.org/10.1016/j.ins.2019.12.061Get rights and content

Abstract

Virtual view synthesis has been increasingly popular due to the wide applications of multi-view and free-viewpoint videos. In view synthesis, texture images are rendered to generate the new viewpoint with the guidance of the depth images. The quality of depth images is vital for generating high-quality synthesized views. While the impact of texture image and the rendering process on the quality of the synthesized view has been extensively studied, the quality evaluation of depth images remains largely unexplored. With this motivation, this paper presents a no-reference image quality index for depth maps by modeling the statistics of edge profiles (SEP) in a multi-scale framework. The Canny operator is first utilized to locate the edges in depth images. Then the edge profiles are constructed, based on which the first-order and second-order statistical features are extracted for portraying the distortions in depth images. Finally, the random forest is employed for building the quality assessment model for depth maps. Experiments are conducted on two annotated view synthesis image/video quality databases. The experimental results and comparisons demonstrate that the proposed metric outperforms the relevant state-of-the-art quality metrics by a large margin. Furthermore, it has better generalization ability.

Introduction

Multi-view and free-viewpoint videos have become increasingly popular. As the most important technique, view synthesis has been drawing more and more attention recently. The quality of synthesized view is critical to the success of these applications. Objective quality models can be used to optimize and benchmark view synthesis algorithms, so they are highly desired for developing advanced view synthesis technique.

In view synthesis, texture and depth images are utilized to render the new viewpoint. In this process, the quality of texture image, depth image and the rendering operation are all crucial for obtaining a high quality synthesized view [2]. Generally, the distortions in texture images are straightforwardly transferred to the synthesized image. Therefore, the impact of distortions in texture images on the synthesized view is intuitive, and the traditional image quality indices can be adopted to measure the quality of texture images [19], [30], [32], [36]. Most of the existing view synthesis quality metrics are dedicated to evaluating the influence of rendering distortions on the perceptual quality of the synthesized images [1], [4], [6], [10], [13], [29]. Typically, they are tested on the IRCCyN/IVC database [2], which consists of synthesized images rendered from distortion-free texture and depth images. In practice, depth image is used to guide the warping process in view synthesis, including a forward mapping from reference view to the 3D space and an inverse mapping to the target view [18]. As a result, depth image plays a paramount role in the process of view synthesis, which has significant influence on the perceptual quality of synthesized images. Although a great number of quality assessment models have been proposed in the past few years, they are mainly designed for natural images and are not readily applicable to depth images. The underlying reason is that a pixel in depth image represents the distance between the target object and the camera in the scene, which is essentially different from natural images. The objective quality assessment of depth images remains a topic largely untouched.

Up to now, very little effort has been done on the quality evaluation of depth images. Farid et al. [7] proposed a Blind Depth Quality Metric (BDQM) for measuring the compression distortions in depth images. The BDQM model is based on the analysis of compression sensitive pixels in depth maps. Specifically, the authors found that the histogram of pixels around the depth transitions becomes flatter when the compression ratio increases. Therefore, the sharpness of the compression sensitive regions was quantified and used to evaluate the quality of depth images. Xiang et al. [33] presented a no-reference (NR) depth quality index by measuring the misalignment errors between the edges of texture and depth images. The misalignments were detected from three aspects, including the edge orientation similarity, the spatial similarity and the segment length similarity. Finally, the bad point rate (BPR) was computed as the overall quality score. Le et al. [15] proposed a reduced-reference (RR) depth quality model by jointly measuring the interactions between the local depth distortion and the local texture characteristics. Specifically, local depth distortions were first detected by calculating the difference of depth values in a local neighborhood. Then the texture image was filtered using the Gabor filter, which was then employed to weight the local depth distortions adaptively. The final quality score was computed by pooling all the local distortion values after removing the outlier regions. The above depth quality metrics have achieved notable advances in measuring the distortions in depth images. Meantime, it should be noted that the BDQM metric [7] is specifically designed for compression distortions in depth images. However, depth images may be subject to different kinds of distortions in real applications. The metrics in [15], [33] both require the corresponding texture images for measuring the quality of depth images, which are RR depth quality metrics. In practice, depth images are usually acquired by depth cameras or depth estimation algorithms, so the reference depth images are often unavailable. Therefore, NR quality metrics for depth images are urgently needed.

In this paper, we present a new NR depth image quality index by measuring the Statistics of Edge Profiles (SEP) in the scale space domain, aiming to model the impact of depth distortions on the perceptual quality of view synthesis. Since the edge regions in depth images have more significant impact on the overall quality of view synthesis [18], we first propose a method to construct the edge profiles. Then we analyze the statistical distributions of edge profiles and use the distribution parameters for building the quality evaluation model for depth images. Specifically, we model the first-order and second-order statistical characteristics of the edge profiles using Weibull Distribution and Asymmetric Generalized Gaussian Distribution (AGGD), respectively. Finally, the extracted statistical features are input into the random forest (RF) regressor for training the quality model. The performance of the proposed SEP index is verified on depth images from two public view synthesis image/video quality databases, and also compared with the relevant state-of-the-arts. The experimental results demonstrate the advantages of the proposed SEP metric in terms of both the consistency with human ratings and the generalization ability.

Section snippets

Statistical characteristics of depth images

The proposed depth quality metric is inspired by the natural scene statistics (NSS) theory, which has been frequently adopted for traditional image quality assessment (IQA) [22], [25], [27], [34]. The idea of NSS-based IQA is that high-quality natural images follow specific statistical regularities, which are prone to change under distortions. While notable success has been achieved in applying NSS in the quality evaluation of natural images, the existing NSS models are not applicable to depth

Proposed NR depth image quality metric

The proposed depth image quality metric is based on the fact that the quality of edge regions in depth images are crucial for generating high-quality synthesized view. Specifically, we have found that although the statistics of the whole depth image do not follow specific distributions, the statistical distribution of edge regions can be well modeled, and the model parameters can be effectively used for quantifying the quality of depth images. The flowchart of the proposed depth quality metric

Experimental protocols

We evaluate the performance of our proposed depth quality metric based on two public view synthesis image/video quality databases, including MCL-3D [28] and SIAT [20]. (1) The MCL-3D database is generated based on nine image-plus-depth sources, each with three viewpoints. Six different types of distortions are added to the texture and/or depth images, including Gaussian blurring, down-sampling blurring, white noise, JPEG compression, JPEG2000 compression and transmission error. For each kind of

Conclusion

In this paper, we have presented a NR quality assessment metric of depth images for view synthesis application. The proposed metric is based on the observation that the edge profiles, i.e., neighboring regions around depth image edges, follow specific statistics. Inspired by this, we have constructed edge profiles from depth images, and extract quality-aware statistical features by fitting the gradient magnitude and LoG distributions. The proposed metric operates in the scale space

Declaration of Competing Interest

The authors declare that they do not have any financial or nonfinancial conflict of interests.

Acknowledgement

This work was supported by the National Natural Science Foundation of China (61771473 and 61379143), the Natural Science Foundation of Jiangsu Province (BK20181354), the Six Talent Peaks High-level Talents in Jiangsu Province (XYDXX-063) and the Qing Lan Project of Jiangsu Province.

References (38)

  • F. Battisti et al.

    Objective image quality assessment of 3d synthesized views

    Signal Process. Image Commun.

    (2015)
  • X.K. Liu et al.

    Subjective and objective video quality assessment of 3d synthesized views with texture/depth compression distortion

    IEEE Trans. Image Process.

    (2015)
  • W.F. Xue et al.

    Blind image quality assessment using joint statistics of gradient magnitude and Laplacian features

    IEEE Trans. Image Process.

    (2014)
  • E. Bosc et al.

    Towards a new quality metric for 3-d synthesized view assessment

    IEEE J. Select. Top. Signal Process.

    (2011)
  • J. Canny

    A computational approach to edge detection

    IEEE Transp. Anal. Mach. Intell.

    (1986)
  • P.H. Conze et al.

    Objective view synthesis qualityassessment

    Proc. Electron. Imag. Int. Society Opt. Photon.

    (2012)
  • A. Criminisi et al.

    Decision forests for classification, regression, density estimation, manifold learning and semisupervised learning

    Comput. Gr. Vis.

    (2012)
  • M.S. Farid et al.

    Perceptual quality assessment of 3D synthesized images

    Proceedings of the IEEE International Conference Multimedia and Expo (ICME)

    (2017)
  • M.S. Farid et al.

    Blind depth quality assessment using histogram shape analysis

    Proceedings of the 3DTV-Conference: The True Vision- Capture, Transmission and Display of 3D Video (3DTV-CON)

    (2015)
  • D. Ferstl et al.

    Image Guided Depth Upsampling Using Anisotropic Total Generalized Variation

    Proceedings of the IEEE International Conference Computer Vision (ICCV)

    (2013)
  • J.M. Geusebroek et al.

    A six-stimulus theory for stochastic texture

    Int. J. Comput. Vis.

    (2005)
  • K. Gu et al.

    Model-based referenceless quality metric of 3d synthesized images using local image description

    IEEE Trans. Image Process.

    (2018)
  • K. Gu et al.

    Quality assessment considering viewing distance and image resolution

    IEEE Trans. Broadcast

    (2015)
  • K. Gu et al.

    Using free energy principle for blind image quality assessment

    IEEE Trans. Multimed.

    (2015)
  • Y.J. Jung et al.

    Critical binocular asymmetry measure for perceptual quality assessment of synthesized stereo 3d images in view synthesis

    IEEE Trans. Circ. Syst. Video Technol.

    (2016)
  • J. Larsson et al.

    Orientation-selective adaptation to first- and second-order patterns in human visual cortex

    J. Neurophysiol.

    (2006)
  • T.H. Le et al.

    A new depth image quality metric using a pair of color and depth images

    Multimed. Tools Appl.

    (2017)
  • L.D. Li et al.

    Image sharpness assessment by sparse representation

    IEEE Trans. Multimed.

    (2016)
  • L.D. Li et al.

    No-reference and robust image sharpness evaluation based on multi-scale spatial and spectral features

    IEEE Trans. Multimed.

    (2017)
  • Cited by (10)

    • Quality assessment of DIBR-synthesized views: An overview

      2021, Neurocomputing
      Citation Excerpt :

      Finally, the misalignments are used to calculate the final quality scores. Li et al. proposed a NR depth image quality index based on the statistics of edge profiles (SEP) [58]. The first-order and second-order statistical features are firstly extracted based on edge profiles which are the neighbouring regions around the depth edges.

    View all citing articles on Scopus
    View full text