Technical SectionA perceptual approach for stereoscopic rendering optimization
Introduction
Technologies underlying 3D autostereoscopic displays have matured to the point that several commercial products are now available in the mass market. These displays are an extension of the conventional 2D displays, by their ability to emit a different image for each eye. Binocular head-mounted displays have also matured to the level that they are widely used in a number of applications. The main difficulty of these stereoscopic and autostereoscopic displays is that they require a rendering phase for each view, which multiplies the rendering time by the number of views. Consequently, there is a need to optimize solutions for stereoscopic rendering.
The traditional way of stereoscopic rendering is to handle the left and the right eye views separately, which is still the model in use in graphics APIs such as OpenGL [1]. A number of stereoscopic and multi-view rendering techniques have recently been proposed. These approaches can be categorized as pipeline-based solutions, which aim to optimize the rendering on the rasterization stage of the rendering pipeline [2], [3], [4]; and image-based solutions, where one view is rendered using the graphics rendering pipeline, and the other view is generated from this image, using the correspondences of the two views [5], [6], [7].
In this paper, we propose a new perceptually-based solution for optimization, by utilizing the suppression theory of binocular vision. According to the Binocular Suppression Theory, the less dominant view will be suppressed by the dominant one; and when the images from corresponding regions differ in an appropriate way, they fuse but the disparities are registered and used for impression of depth. It has been shown that the result of which view suppresses the other depends on the visual properties of the two images [8], [9]. Section 2.2 overviews the Binocular Suppression Theory that our solution is based on.
We investigate how binocular suppression mechanism can be utilized for optimization, by comparing the effects of different graphics rendering and modeling methods. Our aim is to identify the rendering and modeling features that do not affect the overall quality of a stereo image pair when simplified in one view. We applied our approach to a number of representative and commonly used methods used in rendering, including framebuffer upsampling, mixed-level antialiasing, specular highlight, mixed shading, mesh simplification, texture resampling, and mixed shadowing. We performed an experimental study in order to evaluate each method’s perceptual effect on the overall perceived 3D image, in terms of quality, sharpness, depth, and comfort. The experimental results show that the overall perceived stereo image quality is not affected when one of the views is modified by a technique that decreases the intensity contrast. On the other hand, when a modification that increases the intensity contrast is applied on a single view; it will be visible and the overall perceived stereo image quality will be affected towards the modified image.
The main contributions of this study are as follows:
- •
a new approach for stereo rendering optimization, based on the usage of different stimuli, each with different quality, for each eye,
- •
a content creation guideline, describing when it is appropriate to use each optimization,
- •
a formal experimental study to verify the proposed hypothesis.
The rest of the paper is organized as follows. First, we survey previous work on stereoscopic rendering methods and Binocular Suppression Theory. Then, we explain our perceptually-based approach and the graphical methods we have used. Lastly, we provide the experiment design and our analysis of the results.
Section snippets
Stereoscopic rendering optimization
A number of techniques have been proposed to optimize stereoscopic rendering. The first group of solutions follows a graphics pipeline-based approach, by utilizing the coherence between neighboring views. Adelson and Hodges [10] simultaneously render a triangle to both images by using the x-axis coherence in device coordinates to accelerate the stereoscopic rendering process. Kalaiah and Capin [4] propose a GPU-based solution that reduces the number of vertex shader computations needed for
Approach
In this paper, we present a Binocular Suppression Theory based approach to stereoscopic graphics rendering. The proposed method exploits the fact that the overall perception of the stereo pair in a region will be determined by the dominant image on the corresponding region, instead of summation of the effect of two images. Our goal is to explore how the rendering quality can be reduced in the suppressed view, without reducing the overall perceived quality of the rendered 3D image. If such
Experiment
We have implemented the proposed methods, and performed a formal experiment to observe whether the use of each method is perceptible. We have decided to base our work on users’ subjective ratings, instead of objective evaluation in which users perform a task (such as measuring the time and error when placing an object at a depth with respect to another object). Our subjective evaluation may cause rater bias in the results due to the individual characteristics of the subjects; however, our
Experimental results and discussion
To determine the difference between the reference and test content, the Test minus Reference score was used. A score of zero means that the test sequence was rated equivalently to the reference sequence, and a negative score means that the test content was rated lower than the reference content. Error bars in the figures below show the 95% confidence interval of the mean, which corresponds to the range within which the mean is expected to fall with 95% certainty. Data points in the
Performance results
In this section, we further demonstrate the performance gain of the mixed-stereoscopic rendering approach, using the methods indicated as feasible in Table 4.
The performance gain of the mixed stereo approach over traditional stereoscopic rendering depends on the choice of the method for rendering the scene. Using advanced rendering techniques, such as BRDF, area light sources, anisotropic texture filtering, etc. will increase the advantage of our approach over the traditional approach. However,
Conclusion & future work
In this paper, we have presented a perceptually-based optimization approach for stereoscopic rendering, which makes use of the binocular suppression mechanism of the human visual system. The proposed method exploits the fact that the 3D perception of the overall stereo pair in a region is determined by the dominant image on the corresponding region, instead of summation of the effect of two images. We have also introduced an estimate of the strength of a view, called intensity contrast, and
Acknowledgments
The authors were supported by the European Commission FP7-213349 All 3D Imaging Phone project and TUBITAK. Also, we would like to thank all the participants of the subjective experiment described in this work and Onur Kucuktunc for his help on preparation of the test videos.
References (36)
- et al.
Accelerated rendering algorithm for stereoscopic display
Computers & Graphics
(1996) - et al.
The influence of color on binocular rivalry
Visual Psycophysics and Physiology
(1978) - et al.
Saliency-based search mechanism for overt and covert shifts of visual attention
Vision Research
(2000) - et al.
New sharpness metric based on local kurtosis, edge and energy information
Signal Processing: Image Communication
(February 2004) - et al.
Visual comfort of binocular and 3D displays
Displays
(2004) - et al.
A study of visual fatigue and visual comfort for 3D HDTV/HDTV images
Displays
(2002) - Segal M, Akeley K. The OpenGL Graphics System: A Specification (Version 2.0);...
- et al.
Simultaneous generation of stereoscopic views
Computer Graphics Forum
(1991) - et al.
An efficient multi-view rasterization architecture
Eurographics Symposium on Rendering
(2006) - et al.
Unified rendering pipeline for autostereoscopic displays
3DTV Conference
(2007)
Interactive stereoscopic rendering of volumetric environments
IEEE Transactions on Visualization and Computer Graphics
Depth-image based rendering (DIBR), compression and transmission for a new approach on 3DTV
SPIE
Binocular fusion and rivalry
Binocular Vision and Stereopsis
Visual competition
Nature Reviews Neuroscience
Stereoscopic ray tracing
The Visual Computer
Stereoscopic image generation based on depth images for 3D TV
IEEE Transactions on Broadcasting
Fast stereo volume rendering
IEEE Visualization
Cited by (10)
Visual perception of computer-generated stereoscopic pictures: Toward the impact of image resolution
2021, Signal Processing: Image CommunicationCitation Excerpt :The essence of binocular rivalry [5] is that when a stimulus is communicated to the suppressed eye, the activated neurons have an inhibitory effect on the visual center, resulting in one eye suppressing the other visual perception. However, binocular suppression [6] is a special condition of binocular rivalry when the visual stimuli no longer appear alternately between the left and right eyes; when two eyes see a similar image, the eye that sees a high-quality image will dominate, thereby suppressing the image of the other eye. The image reflected in the brain is ultimately a high-quality image, as shown in the right portion of Fig. 1.
Improved gloss depiction using the empirical highlight un-distortion method for 3D image-warping-based stereo rendering
2015, Computers and Graphics (Pergamon)Citation Excerpt :Zhang et al. [8] applied a filtering technique to remove dis-occluded areas in new views resulting from DIBR. Based on the binocular suppression theory of human perception, Bulbul et al. [9] showed that stereo rendering could be accelerated by producing a high quality image for one eye and a low quality one for the other eye. Sorbier et al. [10] indicated that stereoscopic rendering can be accelerated by setting multiple rendering targets after cloning the same primitives for both views in GPUs.
Tile Pair-Based Adaptive Multi-Rate Stereo Shading
2020, IEEE Transactions on Visualization and Computer GraphicsEfficient stereo rendering of large 3D datasets based on binocular suppression
2015, 2015 International Conference on Big Data and Smart Computing, BIGCOMP 2015Stereoscopic image quality in virtual environments
2014, 2014 International Conference on 3D Imaging, IC3D 2014 - ProceedingsVisibility threshold of compressed stereoscopic image: Effects of asymmetrical coding
2013, Imaging Science Journal