skip to main content
research-article
Public Access

Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering

Published: 27 November 2020 Publication History

Abstract

Virtual and augmented reality (VR/AR) displays crucially rely on stereoscopic rendering to enable perceptually realistic user experiences. Yet, existing near-eye display systems ignore the gaze-dependent shift of the no-parallax point in the human eye. Here, we introduce a gaze-contingent stereo rendering technique that models this effect and conduct several user studies to validate its effectiveness. Our findings include experimental validation of the location of the no-parallax point, which we then use to demonstrate significant improvements of disparity and shape distortion in a VR setting, and consistent alignment of physical and digitally rendered objects across depths in optical see-through AR. Our work shows that gaze-contingent stereo rendering improves perceptual realism and depth perception of emerging wearable computing systems.

Supplementary Material

ZIP File (a269-krajancich.zip)
Supplemental material.
MP4 File (a269-krajancich.mp4)
MP4 File (3414685.3417820.mp4)
Presentation video

References

[1]
Kurt Akeley, Simon J. Watt, Ahna R. Girshick, and Martin S. Banks. 2004. A Stereo Display Prototype with Multiple Focal Distances. ACM Trans. Graph. (SIGGRAPH) 23, 3 (2004), 804--813.
[2]
Kaan Akşit, Ward Lopes, Jonghyun Kim, Peter Shirley, and David Luebke. 2017. Near-eye varifocal augmented reality display using see-through screens. ACM Transactions on Graphics 36, 6 (Nov. 2017), 189:1--189:13.
[3]
David Atchison and George Smith. 2000. Optics of the human eye. Butterworth Heinemann.
[4]
David A. Atchison. 2017. Schematic Eyes. In Handbook of Visual Optics, Volume I - Fundamentals and Eye Optics, Pablo Artal (Ed.). CRC Press, Chapter 16.
[5]
Geoffrey P. Bingham. 1993. Optical flow from eye movement with head immobilized: "Ocular occlusion" beyond the nose. Vision Research 33, 5 (March 1993), 777--789.
[6]
Johannes Burge, Charless C. Fowlkes, and Martin S. Banks. 2010. Natural-Scene Statistics Predict How the Figure-Ground Cue of Convexity Affects Human Depth Perception. Journal of Neuroscience 30, 21 (May 2010), 7269--7280.
[7]
Jen-Hao Rick Chang, B. V. K. Vijaya Kumar, and Aswin C. Sankaranarayanan. 2018. Towards multifocal displays with dense focal stacks. ACM Transactions on Graphics 37, 6 (Dec. 2018), 198:1--198:13.
[8]
Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2011. A Perceptual Model for Disparity. ACM Transactions on Graphics (Proceedings SIGGRAPH 2011, Vancouver) 30, 4 (2011).
[9]
Neil A. Dodgson. 2004. Variation and extrema of human interpupillary distance. In Stereoscopic Displays and Virtual Reality Systems XI, Mark T. Bolas, Andrew J. Woods, John O. Merritt, and Stephen A. Benton (Eds.), Vol. 5291. International Society for Optics and Photonics, SPIE, 36 -- 46.
[10]
Andrew T. Duchowski, Nathan Cournia, and Hunter A. Murphy. 2004. Gaze-Contingent Displays: A Review. Cyberpsychology & behavior 7 (2004), 621--34.
[11]
Andrew T. Duchowski, Donald H. House, Jordan Gestring, Rui I. Wang, Krzysztof Krejtz, Izabela Krejtz, Radoslaw Mantiuk, and Bartosz Bazyluk. 2014. Reducing Visual Discomfort of 3D Stereoscopic Displays with Gaze-contingent Depth-of-field. In Proc. ACM Symp. on Appl. Perc. (SAP). 39--46.
[12]
David Dunn, Cary Tippets, Kent Torell, Petr Kellnhofer, Kaan Akşit, Piotr Didyk, Karol Myszkowski, David Luebke, and Henry Fuchs. 2017. Wide Field Of View Varifocal Near-Eye Display Using See-Through Deformable Membrane Mirrors. IEEE TVCG 23, 4 (2017), 1322--1331.
[13]
Wilson S. Geisler and Jeffrey S. Perry. 1998. Real-time foveated multiresolution system for low-bandwidth video communication. In Human Vision and Electronic Imaging III, Vol. 3299. International Society for Optics and Photonics, 294--305.
[14]
Andrew Glennerster, Brian J. Rogers, and Mark F. Bradshaw. 1996. Stereoscopic depth constancy depends on the subject's task. Vision Research 36, 21 (Nov. 1996), 3441--3456.
[15]
Brian Guenter, Mark Finch, Steven Drucker, Desney Tan, and John Snyder. 2012. Foveated 3D Graphics. ACM Trans. Graph. (SIGGRAPH Asia) 31, 6 (2012), 164:1--164:10.
[16]
Philippe Hanhart and Touradj Ebrahimi. 2014. Subjective evaluation of two stereoscopic imaging systems exploiting visual attention to improve 3D quality of experience. In Proc. SPIE vol. 9011. 0D-1--11.
[17]
Sebastien Hillaire, Anatole Lecuyer, Remi Cozot, and Gery Casiez. 2008. Using an Eye-Tracking System to Improve Camera Motions and Depth-of-Field Blur Effects in Virtual Environments. In IEEE Virtual Reality Conference. 47--50.
[18]
Hong Hua and Bahram Javidi. 2014. A 3D integral imaging optical see-through head-mounted display. Optics Express 22, 11 (2014), 13484--13491.
[19]
Fu-Chung Huang, Kevin Chen, and Gordon Wetzstein. 2015. The Light Field Stereoscope: Immersive Computer Graphics via Factored Near-Eye Light Field Display with Focus Cues. ACM Trans. Graph. (SIGGRAPH) 34, 4 (2015).
[20]
Yuta Itoh, Toshiyuki Amano, Daisuke Iwai, and Gudrun Klinker. 2016. Gaussian Light Field: Estimation of Viewpoint-Dependent Blur for Optical See-Through Head-Mounted Displays. IEEE Transactions on Visualization and Computer Graphics 22, 11 (Nov. 2016), 2368--2376.
[21]
Yuta Itoh and Gudrun Klinker. 2014. Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization. In 2014 IEEE Symposium on 3D User Interfaces (3DUI). 75--82.
[22]
David Jacobs, Orazio Gallo, Emily A. Cooper, Kari Pulli, and Marc Levoy. 2015. Simulating the Visual Experience of Very Bright and Very Dark Scenes. ACM Trans. Graph. 34, 3, Article 25 (2015), 15 pages.
[23]
Changwon Jang, Kiseung Bang, Gang Li, and Byoungho Lee. 2018. Holographic Near-Eye Display with Expanded Eye-Box. ACM Trans. Graph. 37, 6, Article 195 (Dec. 2018), 14 pages.
[24]
Changwon Jang, Kiseung Bang, Seokil Moon, Jonghyun Kim, Seungjae Lee, and Byoungho Lee. 2017. Retinal 3D: Augmented Reality near-Eye Display via Pupil-Tracked Light Field Projection on Retina. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017).
[25]
Adam L Janin, David W Mizell, and Thomas P Caudell. 1993. Calibration of head-mounted displays for augmented reality applications. In Proc. IEEE Virtual Reality. 246--255.
[26]
Paul V. Johnson, Jared AQ. Parnell, Joohwan Kim, Christopher D. Saunter, Gordon D. Love, and Martin S. Banks. 2016. Dynamic lens and monovision 3D displays to improve viewer comfort. OSA Opt. Express 24, 11 (2016), 11808--11827.
[27]
Anton S Kaplanyan, Anton Sochenov, Thomas Leimkühler, Mikhail Okunev, Todd Goodall, and Gizem Rufo. 2019. DeepFovea: neural reconstruction for foveated rendering and video compression using learned statistics of natural videos. ACM Transactions on Graphics (TOG) 38, 6 (2019), 1--13.
[28]
Petr Kellnhofer, Piotr Didyk, Karol Myszkowski, Mohamed M. Hefeeda, Hans-Peter Seidel, and Wojciech Matusik. 2016a. GazeStereo3D: Seamless Disparity Manipulations. ACM Transactions on Graphics (Proc. SIGGRAPH) 35, 4 (2016).
[29]
Petr Kellnhofer, Piotr Didyk, Tobias Ritschel, Belen Masia, Karol Myszkowski, and Hans-Peter Seidel. 2016b. Motion parallax in stereo 3D: model and applications. ACM Transactions on Graphics 35, 6 (Nov. 2016), 176:1--176:12.
[30]
Jonghyun Kim, Youngmo Jeong, Michael Stengel, Kaan Akşit, Rachel Albert, Ben Boudaoud, Trey Greer, Joohwan Kim, Ward Lopes, Zander Majercik, et al. 2019. Foveated AR: dynamically-foveated augmented reality display. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--15.
[31]
Robert Konrad, Anastasios Angelopoulos, and Gordon Wetzstein. 2020. Gaze-contingent ocular parallax rendering for virtual reality. ACM Transactions on Graphics (TOG) 39, 2 (2020), 1--12.
[32]
Robert Konrad, Emily Cooper, and Gordon Wetzstein. 2015. Novel Optical Configurations for Virtual Reality: Evaluating User Preference and Performance with Focus-tunable and Monovision Near-eye Displays. In Proc. SIGCHI.
[33]
George A. Koulieris, Kaan Akşit, Michael Stengel, Rafał K. Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. Computer Graphics Forum 38, 2 (2019).
[34]
Hiroaki Kudo and Noboru Ohnishi. 2000. Effect of the sight line shift when a head-mounted display is used. In Proc. EMBS International Conference, Vol. 1. 548--550.
[35]
Hiroaki Kudo, Masaya Saito, Tsuyoshi Yamamura, and Noboru Ohnishi. 1999. Measurement of the ability in monocular depth perception during gazing at near visual target-effect of the ocular parallax cue. In Proc. IEEE International Conference on Systems, Man, and Cybernetics, Vol. 2. 34--37.
[36]
Douglas Lanman and David Luebke. 2013. Near-eye Light Field Displays. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6 (2013), 220:1--220:10.
[37]
Sheng Liu, Dewen Cheng, and Hong Hua. 2008. An Optical See-through Head Mounted Display with Addressable Focal Planes. In Proc. ISMAR. 33--42.
[38]
David Luebke and Benjamin Hallen. 2001. Perceptually driven simplification for interactive rendering. In Rendering Techniques 2001. Springer, 223--234.
[39]
Andrew Maimone, Andreas Georgiou, and Joel S. Kollin. 2017. Holographic near-eye displays for virtual and augmented reality. ACM Transactions on Graphics 36, 4 (July 2017), 85:1--85:16.
[40]
Radosław Mantiuk, Bartosz Bazyluk, and Anna Tomaszewska. 2011. Gaze-dependent depth-of-field effect rendering in virtual environments. In Serious Games Development and Appl. 1--12.
[41]
Radoslaw Mantiuk and Mateusz Markowski. 2013. Gaze-Dependent Tone Mapping. In ICIAR.
[42]
Michael Mauderer, Simone Conte, Miguel A. Nacenta, and Dhanraj Vishwanath. 2014. Depth Perception with Gaze-Contingent Depth of Field. In Proc. SIGCHI. 217--226.
[43]
Michael Mauderer, David R. Flatla, and Miguel A. Nacenta. 2016. Gaze-Contingent Manipulation of Color Perception. Proc. SIGCHI (2016).
[44]
Olivier Mercier, Yusufu Sulai, Kevin Mackenzie, Marina Zannoli, James Hillis, Derek Nowrouzezahrai, and Douglas Lanman. 2017. Fast Gaze-contingent Optimal Decompositions for Multifocal Displays. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017).
[45]
Hunter Murphy and Andrew Duchowski. 2001. Gaze-Contingent Level Of Detail Rendering. EuroGraphics 2001 (01 2001).
[46]
Toshikazu Ohshima, Hiroyuki Yamamoto, and Hideyuki Tamura. 1996. Gaze-directed adaptive rendering for interacting with virtual space. In Proc. IEEE VR. IEEE, 103--110.
[47]
Nitish Padmanaban, Robert Konrad, Tal Stramer, Emily A. Cooper, and Gordon Wetzstein. 2017. Optimizing virtual reality for all users through gaze-contingent and adaptive focus displays. PNAS 114, 9 (2017), 2183--2188.
[48]
Nitish Padmanaban, Yifan Peng, and Gordon Wetzstein. 2019. Holographic Near-Eye Displays Based on Overlap-Add Stereograms. ACM Trans. Graph. (SIGGRAPH Asia) 6 (2019). Issue 38.
[49]
Jae-Hyeung Park and Seong-Bok Kim. 2018. Optical see-through holographic near-eye-display with eyebox steering and depth of field control. Opt. Express 26, 21 (2018), 27076--27088.
[50]
Anjul Patney, Marco Salvi, Joohwan Kim, Anton Kaplanyan, Chris Wyman, Nir Benty, David Luebke, and Aaron Lefohn. 2016. Towards foveated rendering for gaze-tracked virtual reality. ACM Transactions on Graphics (TOG) 35, 6 (Nov. 2016), 179:1--179:12.
[51]
Eli Peli, T Reed Hedges, Jinshan Tang, and Dan Landmann. 2001. A Binocular Stereoscopic Display System with Coupled Convergence and Accommodation Demands. In SID Symposium Digest of Technical Papers, Vol. 32. 1296--1299.
[52]
Yifan Peng, Suyeon Choi, Nitish Padmanaban, Jonghyun Kim, and Gordon Wetzstein. 2020. Neural Holography. In ACM SIGGRAPH Emerging Technologies.
[53]
Alexander Plopski, Yuta Itoh, Christian Nitschke, Kiyoshi Kiyokawa, Gudrun Klinker, and Haruo Takemura. 2015. Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays. IEEE Transactions on Visualization and Computer Graphics 21, 4 (2015), 481--490.
[54]
Joshua Ratcliff, Alexey Supikov, Santiago Alfaro, and Ronald Azuma. 2020. ThinVR: Heterogeneous microlens arrays for compact, 180 degree FOV VR near-eye displays. IEEE Transactions on Visualization and Computer Graphics 26, 5 (May 2020), 1981--1990.
[55]
Whitman Richards and John F Miller. 1969. Convergence as a cue to depth. Perception & Psychophysics 5, 5 (1969), 317--320.
[56]
Jannick P. Rolland, Myron W. Krueger, and Alexei Goon. 2000. Multifocal planes head-mounted displays. OSA Appl. Opt. 39, 19 (2000), 3209--3215.
[57]
Heiko H Schütt, Stefan Harmeling, Jakob H Macke, and Felix A Wichmann. 2016. Painfree and accurate Bayesian estimation of psychometric functions for (potentially) overdispersed data. Vision Research 122 (2016), 105--123.
[58]
Liang Shi, Fu-Chung Huang, Ward Lopes, Wojciech Matusik, and David Luebke. 2017. Near-eye Light Field Holographic Rendering with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6, Article 236 (2017), 236:1--236:17 pages.
[59]
Takashi Shibata, Joohwan Kim, David M. Hoffman, and Martin S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision 11, 8 (2011), 11--11.
[60]
Y Shin, HW Lim, MH Kang, M Seong, H Cho, and JH Kim. 2016. Normal range of eye movement and its relationship to age. Acta Ophthalmologica 94 (2016).
[61]
Qi Sun, Fu-Chung Huang, Joohwan Kim, Li-Yi Wei, David Luebke, and Arie Kaufman. 2017. Perceptually-guided foveation for light field displays. ACM Transactions on Graphics 36, 6 (Nov. 2017), 192:1--192:13.
[62]
Mihran Tuceryan and Nassir Navab. 2000. Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. In Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000). 149--158.
[63]
Jacek Turski. 2016. On binocular vision: The geometric horopter and Cyclopean eye. Vision research 119 (2016), 73--81.
[64]
Margarita Vinnikov and Robert S. Allison. 2014. Gaze-contingent Depth of Field in Realistic Scenes: The User Experience. In Proc. Symp. on Eye Tracking Res. and Appl. (ETRA). 119--126.
[65]
Božo Vojniković and Ettore Tamajo. 2013. Horopters-Definition and Construction. Collegium antropologicum 37, 1 (2013), 9--12.
[66]
Felix A Wichmann and N Jeremy Hill. 2001. The psychometric function: I. Fitting, sampling, and goodness of fit. Perception & psychophysics 63, 8 (2001), 1293--1313.
[67]
Albert Yonas, Lincoln G. Craton, and William B. Thompson. 1987. Relative motion: Kinetic information for the order of depth at an edge. Perception & Psychophysics 41, 1 (01 Jan 1987), 53--59.

Cited By

View all
  • (2025)Impact of Background, Foreground, and Manipulated Object Rendering on Egocentric Depth Perception in Virtual and Augmented Indoor EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338261631:4(2155-2166)Online publication date: Apr-2025
  • (2024)Integrating eye rotation and contrast sensitivity into image quality evaluation of virtual reality head-mounted displaysOptics Express10.1364/OE.52766032:14(24968)Online publication date: 25-Jun-2024
  • (2024)Relating visual and pictorial space: Integration of binocular disparity and motion parallaxJournal of Vision10.1167/jov.24.13.724:13(7)Online publication date: 9-Dec-2024
  • Show More Cited By

Index Terms

  1. Optimizing depth perception in virtual and augmented reality through gaze-contingent stereo rendering

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 39, Issue 6
      December 2020
      1605 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3414685
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 27 November 2020
      Published in TOG Volume 39, Issue 6

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. applied perception
      2. augmented reality
      3. rendering
      4. virtual reality

      Qualifiers

      • Research-article

      Funding Sources

      • Okawa Research Grant
      • NSF
      • ARO
      • Stanford Knight-Hennessy Fellowship
      • Sloan Fellowship

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)433
      • Downloads (Last 6 weeks)43
      Reflects downloads up to 05 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Impact of Background, Foreground, and Manipulated Object Rendering on Egocentric Depth Perception in Virtual and Augmented Indoor EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.338261631:4(2155-2166)Online publication date: Apr-2025
      • (2024)Integrating eye rotation and contrast sensitivity into image quality evaluation of virtual reality head-mounted displaysOptics Express10.1364/OE.52766032:14(24968)Online publication date: 25-Jun-2024
      • (2024)Relating visual and pictorial space: Integration of binocular disparity and motion parallaxJournal of Vision10.1167/jov.24.13.724:13(7)Online publication date: 9-Dec-2024
      • (2024)Identifying Behavioral Correlates to Visual DiscomfortACM Transactions on Graphics10.1145/368792943:6(1-10)Online publication date: 19-Dec-2024
      • (2024)A Calibration Interface for 3D Gaze Depth Disambiguation in Virtual Environments2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00144(695-696)Online publication date: 16-Mar-2024
      • (2024)Measuring and Predicting Multisensory Reaction Latency: A Probabilistic Model for Visual-Auditory IntegrationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345618530:11(7364-7374)Online publication date: 1-Nov-2024
      • (2024)Low-Latency Ocular Parallax Rendering and Investigation of Its Effect on Depth Perception in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337207830:5(2228-2238)Online publication date: 5-Mar-2024
      • (2024)Eye-Centered Low-Latency Motion Parallax Displays2024 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII58957.2024.10417227(1326-1327)Online publication date: 8-Jan-2024
      • (2024)ARVA: An Augmented Reality-Based Visual Aid for Mobility Enhancement Through Real-Time Video Stream TransformationIEEE Access10.1109/ACCESS.2024.346262812(137268-137283)Online publication date: 2024
      • (2024)Gaze analysisImage and Vision Computing10.1016/j.imavis.2024.104961144:COnline publication date: 1-Apr-2024
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Full Access

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media