Stereo/multiview picture quality: Overview and recent advances

https://doi.org/10.1016/j.image.2013.07.008Get rights and content

Highlights

  • We review various perceptual and quality issues for stereoscopic viewing of 3D content.

  • Technical issues on 3D format, production, coding, and displays technologies are described extensively.

  • Several issues with 3D viewing (depth, multiview, display, and viewer-specific issues) are discussed and analyzed.

  • We identify various Quality of Experience (QoE) indicators for 3D quality assessment.

  • We present current efforts for standardization of 3D quality measurement methods.

Abstract

Stereoscopic 3D content brings with it a variety of complex technological and perceptual issues. For the percept of depth to be convincing, consistent, and comfortable, a large number of parameters throughout the imaging and processing pipeline need to be matched correctly. In practice, tradeoffs are inevitable, which may then affect the quality or comfort of the 3D viewing experience.

This paper reviews stereo/multiview picture quality from an engineering perspective. With a focus on recent or emerging approaches and technologies used in 3D systems, it addresses in particular depth issues, multiview issues, display issues, viewer issues, as well as possible measurements and standards for 3D quality.

Introduction

Hardware upgrading costs, lack of 3D content, and the need for glasses have long been limiting factors for the widespread acceptance of stereo/3D technology outside of special settings such as virtual reality or gaming. With the arrival of more affordable equipment, a surge in 3D content production, and autostereoscopic displays, 3D viewing may receive a significant boost. As the technology becomes more widely adopted and mature, quality issues rise to the forefront of concerns. For 3D to become a widespread success, high quality of the presentation and comfortable 3D percepts are essential.

Quality issues for images and 2D video have been studied quite extensively [1], and commercial quality assurance (QA) tools are already being deployed to monitor video quality in real time. Most of these tools are designed to pick out common spatial and temporal distortions of the video resulting from compression and transmission.

Stereoscopic 3D adds another layer of complexity on top of the common 2D impairments from video compression and transmission issues [2]. Furthermore, stereoscopic content may have potential psychological and physiological effects [3], [4], especially when 3D is not produced, processed and presented correctly. Symptoms such as eye strain, headache, or dizziness are commonly reported by viewers. This underlines that 3D viewing comes with more severe concerns than 2D. These effects need to be better understood, and one of the primary practical goals must be to minimize or prevent possible discomfort caused by 3D content.

There are some excellent reviews on the perception of stereoscopic pictures [5], [6] as well as a 3-volume treatise on the subject [7]. A special issue [8] delves into many other 3DTV-related topics. Building on an earlier short compendium [9], our aim here is to provide a coherent overview of 3D picture quality and in particular its inter-relationship with recent algorithms and emerging techniques for 3D production, processing, and display.

The paper is organized as follows. We first give a brief overview of stereoscopic viewing basics as well as various technologies used in the 3D imaging pipeline. We then discuss specific issues with 3D viewing, in particular depth, multiview, display, and viewer-specific issues. Finally, we identify a number of possible Quality of Experience (QoE) indicators for 3D and review current standardization efforts for subjective and objective quality measurement.

Section snippets

Depth cues

3D is all about the perception of depth. There is a multitude of depth cues that the human visual system (HVS) uses to understand a 3D scene [10]. These can be classified into oculomotor cues coming from the eye muscles, and visual cues from the scene content itself. They can also be classified into monocular and binocular cues [11].

Oculomotor cues include accommodation and vergence. Accommodation, a monocular cue, refers to the variation of the lens shape and thickness (and thus its focal

Technical issues

In this section, we describe various technical issues with regard to 3D formats, 3D production, 3D video coding, and 3D display. Table 1 provides a summary.

Vergence–accommodation conflict

In normal 3D viewing conditions in the real world, when the eyes fixate on something, that point is brought into focus using accommodation of the lens, while the eyes converge on the same point. The fixation point falls on the horopter, a set of locations in space that project onto corresponding retinal points. Its shape is discussed in detail in [71]. Points located in front of or behind the horopter create negative or positive disparities on the retina, respectively, which is used as a

Dual- and multiview issues

Good correspondences between views are essential to creating a high-quality 3D viewing experience. Various possibilities for mismatch are presented in this section. Table 2 summarizes challenges and possible solutions of these issues, which apply to both dual- and multiview 3D presentations.

Display issues

Presenting two or more views on a display usually takes its toll, leading to reduced brightness and reduced spatial or temporal resolution, compared to showing a single 2D view on the same display. Two other important issues that affect all stereoscopic display technologies are discussed in the following.

Viewer issues

About 5% of viewers have defective stereo vision [142]. Furthermore, each viewer has differences in optimal viewing conditions due to individual differences in (depth) perception. Aside from differences in visual acuity or optical corrections between left and right eyes [95], this is determined by several other factors such as age, gender, and degree of previous 3D viewing experience [143], [144], [145]. Therefore, viewer idiosyncracies must be taken into account for an accurate evaluation of

3D quality measurement and standardization

Many of the issues discussed above can affect the 3D viewing experience. Both physiological and psychological effects can be observed:

  • Physiological effects can be measured objectively. Visual fatigue5 is the most common physiological symptom associated with stereoscopic 3D [75]. It comes with a measurable decrease in the performance of the

Conclusions

We discussed various quality issues of stereoscopy that need to be quantified and monitored. In many cases, this still requires determining the appropriate parameter ranges and acceptable thresholds for a comfortable viewing experience. Quality assurance (QA) is important in three different aspects:

  • Technical issues, such as idiosyncrasies of the various display types. QA for these technical issues is generally done in the lab, when a technology is evaluated. As technologies become more mature,

References (160)

  • L.M.J. Meesters et al.

    A survey of perceptual evaluations and requirements of three-dimensional TV

    IEEE Transactions on Circuits and Systems for Video Technology

    (2004)
  • S. Kim, J. Yoshitake, H. Morikawa, T. Kawai, O. Yamada, A. Iguchi, Psycho-physiological effects of visual artifacts by...
  • A.G. Solimini et al.

    A survey of visually induced symptoms and associated factors in spectators of three dimensional stereoscopic movies

    BMC Public Health

    (2012)
  • S.J. Daly et al.

    Perceptual issues in stereoscopic signal processing

    IEEE Transactions on Broadcasting

    (2011)
  • M.S. Banks et al.

    Stereoscopy and the human visual system

    SMPTE Motion Imaging Journal

    (2012)
  • I.P. Howard et al.

    Perceiving in Depth

    (2012)
  • L. Zhang et al.

    Special issue on 3D-TV horizoncontents, systems, and visual perception

    IEEE Transactions on Broadcasting

    (2011)
  • S. Winkler, D. Min, Stereoscopic image quality compendium, in: Proceedings of International Conference on Information,...
  • R. Blake et al.

    Perception

    (2005)
  • S. Reichelt, R. Häussler, G. Fütterer, N. Leister, Depth cues in human visual perception and their realization in 3D...
  • J. Cutting, P. Vishton, Perceiving layout and knowing distances: The integration, relative potency, and contextual use...
  • P. Lebreton et al.

    Evaluating depth perception of 3D stereoscopic videos

    IEEE Journal of Selected Topics in Signal Processing

    (2012)
  • M. Mikkola et al.

    Stereoscopic depth cues outperform monocular ones on autostereoscopic display

    IEEE Journal of Selected Topics in Signal Processing

    (2012)
  • D. Hoffman et al.

    Focus information is used to interpret binocular images

    Journal of Vision

    (2010)
  • A. Vetro, Frame-compatible formats for 3D video distribution, in: Proceedings of IEEE International Conference on Image...
  • W. Chen, J. Fournier, M. Barkowsky, P. Le Callet, New requirements of subjective video quality assessment methodologies...
  • P. Merkle, A. Smolic, K. Müller, T. Wiegand, Multi-view video plus depth representation and coding, in: Proceedings of...
  • K.-J. Yoon et al.

    Adaptive support-weight approach for correspondence search

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2006)
  • D. Min et al.

    Cost aggregation and occlusion handling with WLS in stereo matching

    IEEE Transactions on Image Processing

    (2008)
  • M.F. Tappen, W.T. Freeman, Comparison of graph cuts with belief propagation for stereo using identical MRF parameters,...
  • C. Fehn, Depth image based rendering (DIBR), compression and transmission for a new approach on 3D-TV, in: Proceedings...
  • K.-J. Oh, S. Yea, Y.-S. Ho, Hole-filling method using depth based in-painting for view synthesis in free viewpoint...
  • S. Choi et al.

    Space-time hole filling with random walks in view extrapolation for 3-D video

    IEEE Transactions on Image Processing

    (2013)
  • J. Shade, S.J. Gortler, L.-W. He, R. Szeliski, Layered depth images, in: Proceedings of the ACM SIGGRAPH, 1998, pp....
  • L. Zitnick et al.

    High-quality video view interpolation using a layered representation

    ACM Transactions on Graphics

    (2004)
  • S.-U. Yoon et al.

    Multiple color and depth video coding using a hierarchical representation

    IEEE Transactions on Circuits and Systems for Video Technology

    (2007)
  • K. Müller, A. Smolic, K. Dix, P. Kauff, T. Wiegand, Reliability-based generation and view synthesis in layered depth...
  • S.M. Seitz et al.

    Photorealistic scene reconstruction by voxel coloring

    International Journal of Computer Vision

    (1999)
  • D. Mahajan et al.

    Moving gradientsa path-based method for plausible image interpolation

    ACM Transactions on Graphics

    (2009)
  • M. Lang et al.

    Nonlinear disparity mapping for stereoscopic 3D

    ACM Transactions on Graphics

    (2010)
  • W. Matusik et al.

    3D TVA scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes

    ACM Transactions on Graphics

    (2004)
  • B. Wilburn et al.

    High performance imaging using large camera arrays

    ACM Transactions on Graphics

    (2005)
  • Z. Zhang

    A flexible new technique for camera calibration

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2000)
  • J.Y. Bouguet, Camera Calibration Toolbox for Matlab,...
  • T. Svoboda et al.

    A convenient multicamera self-calibration for virtual environments

    Presence

    (2005)
  • Point Grey Research,...
  • G. Litos, X. Zabulis, G. Triantafyllidis, Synchronous image acquisition based on network synchronization, in:...
  • D. Bradley, B. Atcheson, I. Ihrke, W. Heidrich, Synchronization and rolling shutter compensation for consumer video...
  • A. Elhayek, C. Stoll, K.I. Kim, H.-P. Seidel, C. Theobalt, Feature-based multi-video synchronization with subframe...
  • J. Kopf et al.

    Joint bilateral upsampling

    ACM Transactions on Graphics

    (2007)
  • Cited by (27)

    • Evaluating virtual image quality using the side-views information fusion and depth maps

      2018, Information Fusion
      Citation Excerpt :

      View Synthesis Quality Assessment (VSQA) [39] combines SSIM with three weighting functions derived from contrast, orientation and texture maps of the reference and synthesized views to assess the quality of virtual pictures. A good literature on 3D-IQA and various 3D quality artifacts can be found in [40,41]. Most existing quality assessment algorithms for DIBR synthesized images are full-reference and rely on the conventional 2D-IQA algorithms.

    • No reference stereo video quality assessment based on motion feature in tensor decomposition domain

      2018, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Stereo video has board application prospect due to its excellent immersive experience [1].

    • Methods for reducing visual discomfort in stereoscopic 3D: A review

      2016, Signal Processing: Image Communication
      Citation Excerpt :

      Due to the number of recent assessment methods, there is increasing need to standardise QoE across labs [10]. For an overview of quality assessment methods for stereo 3D, we refer to [195]. In this section, we focus on methods that explicitly incorporate visual discomfort.

    • PinchLens: Applying Spatial Magnification and Adaptive Control-Display Gain for Precise Selection in Virtual Reality

      2023, Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2023
    • Geometric Analysis of 3D Facial Image Data: A Survey

      2022, Recent Patents on Engineering
    View all citing articles on Scopus
    1

    This work is supported by the research grant for ADSC's Human Sixth Sense Programme from Singapore's Agency for Science, Technology and Research (ASTAR).

    View full text