Stereo/multiview picture quality: Overview and recent advances
Introduction
Hardware upgrading costs, lack of 3D content, and the need for glasses have long been limiting factors for the widespread acceptance of stereo/3D technology outside of special settings such as virtual reality or gaming. With the arrival of more affordable equipment, a surge in 3D content production, and autostereoscopic displays, 3D viewing may receive a significant boost. As the technology becomes more widely adopted and mature, quality issues rise to the forefront of concerns. For 3D to become a widespread success, high quality of the presentation and comfortable 3D percepts are essential.
Quality issues for images and 2D video have been studied quite extensively [1], and commercial quality assurance (QA) tools are already being deployed to monitor video quality in real time. Most of these tools are designed to pick out common spatial and temporal distortions of the video resulting from compression and transmission.
Stereoscopic 3D adds another layer of complexity on top of the common 2D impairments from video compression and transmission issues [2]. Furthermore, stereoscopic content may have potential psychological and physiological effects [3], [4], especially when 3D is not produced, processed and presented correctly. Symptoms such as eye strain, headache, or dizziness are commonly reported by viewers. This underlines that 3D viewing comes with more severe concerns than 2D. These effects need to be better understood, and one of the primary practical goals must be to minimize or prevent possible discomfort caused by 3D content.
There are some excellent reviews on the perception of stereoscopic pictures [5], [6] as well as a 3-volume treatise on the subject [7]. A special issue [8] delves into many other 3DTV-related topics. Building on an earlier short compendium [9], our aim here is to provide a coherent overview of 3D picture quality and in particular its inter-relationship with recent algorithms and emerging techniques for 3D production, processing, and display.
The paper is organized as follows. We first give a brief overview of stereoscopic viewing basics as well as various technologies used in the 3D imaging pipeline. We then discuss specific issues with 3D viewing, in particular depth, multiview, display, and viewer-specific issues. Finally, we identify a number of possible Quality of Experience (QoE) indicators for 3D and review current standardization efforts for subjective and objective quality measurement.
Section snippets
Depth cues
3D is all about the perception of depth. There is a multitude of depth cues that the human visual system (HVS) uses to understand a 3D scene [10]. These can be classified into oculomotor cues coming from the eye muscles, and visual cues from the scene content itself. They can also be classified into monocular and binocular cues [11].
Oculomotor cues include accommodation and vergence. Accommodation, a monocular cue, refers to the variation of the lens shape and thickness (and thus its focal
Technical issues
In this section, we describe various technical issues with regard to 3D formats, 3D production, 3D video coding, and 3D display. Table 1 provides a summary.
Vergence–accommodation conflict
In normal 3D viewing conditions in the real world, when the eyes fixate on something, that point is brought into focus using accommodation of the lens, while the eyes converge on the same point. The fixation point falls on the horopter, a set of locations in space that project onto corresponding retinal points. Its shape is discussed in detail in [71]. Points located in front of or behind the horopter create negative or positive disparities on the retina, respectively, which is used as a
Dual- and multiview issues
Good correspondences between views are essential to creating a high-quality 3D viewing experience. Various possibilities for mismatch are presented in this section. Table 2 summarizes challenges and possible solutions of these issues, which apply to both dual- and multiview 3D presentations.
Display issues
Presenting two or more views on a display usually takes its toll, leading to reduced brightness and reduced spatial or temporal resolution, compared to showing a single 2D view on the same display. Two other important issues that affect all stereoscopic display technologies are discussed in the following.
Viewer issues
About 5% of viewers have defective stereo vision [142]. Furthermore, each viewer has differences in optimal viewing conditions due to individual differences in (depth) perception. Aside from differences in visual acuity or optical corrections between left and right eyes [95], this is determined by several other factors such as age, gender, and degree of previous 3D viewing experience [143], [144], [145]. Therefore, viewer idiosyncracies must be taken into account for an accurate evaluation of
3D quality measurement and standardization
Many of the issues discussed above can affect the 3D viewing experience. Both physiological and psychological effects can be observed:
- •
Physiological effects can be measured objectively. Visual fatigue5 is the most common physiological symptom associated with stereoscopic 3D [75]. It comes with a measurable decrease in the performance of the
Conclusions
We discussed various quality issues of stereoscopy that need to be quantified and monitored. In many cases, this still requires determining the appropriate parameter ranges and acceptable thresholds for a comfortable viewing experience. Quality assurance (QA) is important in three different aspects:
- •
Technical issues, such as idiosyncrasies of the various display types. QA for these technical issues is generally done in the lab, when a technology is evaluated. As technologies become more mature,
References (160)
- et al.
Blur and disparity are complementary cues to depth
Current Biology
(2012) Depth map creation and image-based rendering for advanced 3DTV services providing interoperability and scalability
Signal Processing: Image Communication
(2007)- et al.
2D/3D freeview video generation for 3DTV system
Signal Processing: Image Communication
(2009) - et al.
View synthesis prediction for multiview video coding
Signal Processing: Image Communication
(2009) - et al.
The effects of multiview depth video compression on multiview rendering
Signal Processing: Image Communication
(2009) - et al.
Visual fatigue caused by viewing stereoscopic motion imagesbackground, theories, and observations
Displays
(2008) - et al.
A study of visual fatigue and visual comfort for 3D HDTV/HDTV images
Displays
(2002) - et al.
Two factors in visual fatigue caused by stereoscopic HDTV images
Displays
(2004) - et al.
Spatio-temporal properties of Panum's fusional area
Vision Research
(1981) - et al.
The evolution of video quality measurement: from PSNR to hybrid metrics
IEEE Transactions on Broadcasting
(2008)
A survey of perceptual evaluations and requirements of three-dimensional TV
IEEE Transactions on Circuits and Systems for Video Technology
A survey of visually induced symptoms and associated factors in spectators of three dimensional stereoscopic movies
BMC Public Health
Perceptual issues in stereoscopic signal processing
IEEE Transactions on Broadcasting
Stereoscopy and the human visual system
SMPTE Motion Imaging Journal
Perceiving in Depth
Special issue on 3D-TV horizoncontents, systems, and visual perception
IEEE Transactions on Broadcasting
Perception
Evaluating depth perception of 3D stereoscopic videos
IEEE Journal of Selected Topics in Signal Processing
Stereoscopic depth cues outperform monocular ones on autostereoscopic display
IEEE Journal of Selected Topics in Signal Processing
Focus information is used to interpret binocular images
Journal of Vision
Adaptive support-weight approach for correspondence search
IEEE Transactions on Pattern Analysis and Machine Intelligence
Cost aggregation and occlusion handling with WLS in stereo matching
IEEE Transactions on Image Processing
Space-time hole filling with random walks in view extrapolation for 3-D video
IEEE Transactions on Image Processing
High-quality video view interpolation using a layered representation
ACM Transactions on Graphics
Multiple color and depth video coding using a hierarchical representation
IEEE Transactions on Circuits and Systems for Video Technology
Photorealistic scene reconstruction by voxel coloring
International Journal of Computer Vision
Moving gradientsa path-based method for plausible image interpolation
ACM Transactions on Graphics
Nonlinear disparity mapping for stereoscopic 3D
ACM Transactions on Graphics
3D TVA scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes
ACM Transactions on Graphics
High performance imaging using large camera arrays
ACM Transactions on Graphics
A flexible new technique for camera calibration
IEEE Transactions on Pattern Analysis and Machine Intelligence
A convenient multicamera self-calibration for virtual environments
Presence
Joint bilateral upsampling
ACM Transactions on Graphics
Cited by (27)
Evaluating virtual image quality using the side-views information fusion and depth maps
2018, Information FusionCitation Excerpt :View Synthesis Quality Assessment (VSQA) [39] combines SSIM with three weighting functions derived from contrast, orientation and texture maps of the reference and synthesized views to assess the quality of virtual pictures. A good literature on 3D-IQA and various 3D quality artifacts can be found in [40,41]. Most existing quality assessment algorithms for DIBR synthesized images are full-reference and rely on the conventional 2D-IQA algorithms.
No reference stereo video quality assessment based on motion feature in tensor decomposition domain
2018, Journal of Visual Communication and Image RepresentationCitation Excerpt :Stereo video has board application prospect due to its excellent immersive experience [1].
Methods for reducing visual discomfort in stereoscopic 3D: A review
2016, Signal Processing: Image CommunicationCitation Excerpt :Due to the number of recent assessment methods, there is increasing need to standardise QoE across labs [10]. For an overview of quality assessment methods for stereo 3D, we refer to [195]. In this section, we focus on methods that explicitly incorporate visual discomfort.
PinchLens: Applying Spatial Magnification and Adaptive Control-Display Gain for Precise Selection in Virtual Reality
2023, Proceedings - 2023 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2023Geometric Analysis of 3D Facial Image Data: A Survey
2022, Recent Patents on Engineering
- 1
This work is supported by the research grant for ADSC's Human Sixth Sense Programme from Singapore's Agency for Science, Technology and Research (A⋆STAR).