Sean J. Liu
Tovi Grossman
Michael Glueck
ABSTRACT
In cinematic VR, viewers can only see a limited portion of the scene at any time. As a result, they may miss important events outside their field of view. While there are many techniques which offer spatial guidance (where to look), there has been little work on temporal guidance (when to look). Temporal guidance offers viewers a look-ahead time and allows viewers to plan their head motion for important events. This paper introduces spatiotemporal visual guidance and presents a new widget, RadarVR, which shows both spatial and temporal information of regions of interest (ROIs) in a video. Using RadarVR, we conducted a study to investigate the impact of temporal guidance and explore trade-offs between spatiotemporal and spatial-only visual guidance. Results show spatiotemporal feedback allows users to see a greater percentage of ROIs, with 81% more seen from their initial onset. We discuss design implications for future work in this space.
Supplemental Material
Available for Download
zip
UIST23_RadarVR_Supplemental_Material.zip (178.7 MB)
Video Presentation (4.5 min), Video Preview (30 sec), and Supplemental Material
zip
3606734.zip (361.3 MB)
Supplemental File
- Activision. 2007. Call of Duty 4: Modern Warfare. [Playstation 3].Google Scholar
- Alcove VR. 2022. 360 3D MARSEILLE, FRANCE - Guided Tour - Virtual Travel 4K (Full series on Alcove for Quest). https://www.youtube.com/watch?v=gURIhqukTSQ. Copyright 2022 by Alcove VR. Reprinted with permission.Google Scholar
- Patrick Baudisch and Ruth Rosenholtz. 2003. Halo: a technique for visualizing off-screen objects. In Proceedings of the SIGCHI conference on Human factors in computing systems. 481–488.Google ScholarDigital Library
- Carlo Bonferroni. 1936. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8 (1936), 3–62.Google Scholar
- Felix Bork, Christian Schnelzer, Ulrich Eck, and Nassir Navab. 2018. Towards Efficient Visual Guidance in Limited Field-of-View Head-Mounted Displays. IEEE Transactions on Visualization and Computer Graphics 24, 11 (2018), 2983–2992. https://doi.org/10.1109/TVCG.2018.2868584Google ScholarCross Ref
- Jessica Brillhart. 2016. In the Blink of a Mind — Attention. https://medium.com/the-language-of-vr/in-the-blink-of-a-mind-attention-1fdff60fa045 [Online; posted 05-Feb-2016].Google Scholar
- Gerd Bruder, Frank Steinicke, Phil Wieland, and Markus Lappe. 2012. Tuning Self-Motion Perception in Virtual Reality with Visual Illusions. IEEE Transactions on Visualization and Computer Graphics 18, 7 (July 2012), 1068–1078. https://doi.org/10.1109/TVCG.2011.274Google ScholarDigital Library
- Stefano Burigat, Luca Chittaro, and Silvia Gabrielli. 2006. Visualizing Locations of Off-Screen Objects on Mobile Devices: A Comparative Evaluation of Three Approaches. In Proceedings of the 8th Conference on Human-Computer Interaction with Mobile Devices and Services (Helsinki, Finland) (MobileHCI ’06). Association for Computing Machinery, New York, NY, USA, 239–246. https://doi.org/10.1145/1152215.1152266Google ScholarDigital Library
- Chong Cao, Zhaowei Shi, and Miao Yu. 2020. Automatic Generation of Diegetic Guidance in Cinematic Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 600–607. https://doi.org/10.1109/ISMAR50242.2020.00087Google ScholarCross Ref
- Nina Doerr, Katrin Angerbauer, Melissa Reinelt, and Michael Sedlmair. 2023. Bees, Birds and Butterflies: Investigating the Influence of Distractors on Visual Attention Guidance Techniques. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, Article 51, 7 pages. https://doi.org/10.1145/3544549.3585816Google ScholarDigital Library
- Beat Games. 2018. Beat Saber. [Meta Quest].Google Scholar
- Beat Games. 2019. Beat Saber. [Meta Quest].Google Scholar
- Rockstar Games. 2008. Grand Theft Auto IV. [Playstation 3].Google Scholar
- Google. 2008. Google Maps. https://www.google.com/mapsGoogle Scholar
- Google Spotlight Stories. 2016. 360 Google Spotlight Stories: HELP. https://www.youtube.com/watch?v=G-XZhKqQAHU.Google Scholar
- Steve Grogorick, Georgia Albuquerque, and Marcus A. Magnor. 2018. Comparing Unobtrusive Gaze Guiding Stimuli in Head-Mounted Displays. In 2018 IEEE International Conference on Image Processing, ICIP 2018, Athens, Greece, October 7-10, 2018. IEEE, Athens, Greece, 2805–2809. https://doi.org/10.1109/ICIP.2018.8451784Google ScholarCross Ref
- Steve Grogorick, Jan-Philipp Tauscher, Nikkel Heesen, Susana Castillo, and Marcus Magnor. 2020. Stereo Inverse Brightness Modulation for Guidance in Dynamic Panorama Videos in Virtual Reality. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 542–553.Google Scholar
- Uwe Gruenefeld, Abdallah El Ali, Susanne Boll, and Wilko Heuten. 2018. Beyond Halo and Wedge: Visualizing out-of-View Objects on Head-Mounted Virtual and Augmented Reality Devices(MobileHCI ’18). Association for Computing Machinery, New York, NY, USA, Article 40, 11 pages. https://doi.org/10.1145/3229434.3229438Google ScholarDigital Library
- Uwe Gruenefeld, Dag Ennenga, Abdallah El Ali, Wilko Heuten, and Susanne Boll. 2017. EyeSee360: Designing a Visualization Technique for out-of-View Objects in Head-Mounted Augmented Reality. In Proceedings of the 5th Symposium on Spatial User Interaction (Brighton, United Kingdom) (SUI ’17). Association for Computing Machinery, New York, NY, USA, 109–118. https://doi.org/10.1145/3131277.3132175Google ScholarDigital Library
- Uwe Gruenefeld, Ilja Koethe, Daniel Lange, Sebastian Weiß, and Wilko Heuten. 2019. Comparing Techniques for Visualizing Moving Out-of-View Objects in Head-mounted Virtual Reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 742–746. https://doi.org/10.1109/VR.2019.8797725Google ScholarCross Ref
- Jan Gugenheimer, Dennis Wolf, Gabriel Haas, Sebastian Krebs, and Enrico Rukzio. 2016. SwiVRChair: A Motorized Swivel Chair to Nudge Users’ Orientation for 360 Degree Storytelling in Virtual Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1996–2000. https://doi.org/10.1145/2858036.2858040Google ScholarDigital Library
- Sean Gustafson, Patrick Baudisch, Carl Gutwin, and Pourang Irani. 2008. Wedge: clutter-free visualization of off-screen locations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 787–796.Google ScholarDigital Library
- Harmonix. 2005. Guitar Hero. [Playstation 2].Google Scholar
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183. https://doi.org/10.1016/S0166-4115(08)62386-9Google ScholarCross Ref
- Indie Film School. 2016. Pokèmon 360 - CATCH ’EM ALL in VR!https://www.youtube.com/watch?v=pHUVS_GrIeM.Google Scholar
- Benjamin Jakobs. 2019. Die Entwickler des VR-Hits Beat Saber gehören jetzt zu Facebook. https://www.eurogamer.de/die-entwickler-des-vr-hits-beat-saber-gehoeren-jetzt-zu-facebookGoogle Scholar
- Oliver Beren Kaul and Michael Rohs. 2017. HapticHead: A Spherical Vibrotactile Grid around the Head for 3D Guidance in Virtual and Augmented Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3729–3740. https://doi.org/10.1145/3025453.3025684Google ScholarDigital Library
- Joona Kinnunen. 2018. Spatial guiding through haptic cues in omnidirectional video. Master’s thesis.Google Scholar
- Konami. 1998. Dance Dance Revolution. [Arcade].Google Scholar
- Daniel Lange, Tim Claudius Stratmann, Uwe Gruenefeld, and Susanne Boll. 2020. HiveFive: Immersion Preserving Attention Guidance in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376803Google ScholarDigital Library
- Jiannan Li, Jiahe Lyu, Mauricio Sousa, Ravin Balakrishnan, Anthony Tang, and Tovi Grossman. 2021. Route tapestries: Navigating 360 virtual tour videos using slit-scan visualizations. In The 34th Annual ACM Symposium on User Interface Software and Technology. 223–238.Google ScholarDigital Library
- Klemen Lilija, Henning Pohl, and Kasper Hornbæk. 2020. Who put that there? temporal navigation of spatial recordings by direct manipulation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–11.Google ScholarDigital Library
- Yen-Chen Lin, Yung-Ju Chang, Hou-Ning Hu, Hsien-Tzu Cheng, Chi-Wen Huang, and Min Sun. 2017. Tell Me Where to Look: Investigating Ways for Assisting Focus in 360° Video. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 2535–2545. https://doi.org/10.1145/3025453.3025757Google ScholarDigital Library
- Yung-Ta Lin, Yi-Chi Liao, Shan-Yuan Teng, Yi-Ju Chung, Liwei Chan, and Bing-Yu Chen. 2017. Outside-In: Visualizing Out-of-Sight Regions-of-Interest in a 360° Video Using Spatial Picture-in-Picture Previews. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 255–265. https://doi.org/10.1145/3126594.3126656Google ScholarDigital Library
- Sean J. Liu, Maneesh Agrawala, Stephen DiVerdi, and Aaron Hertzmann. 2019. View-dependent video textures for 360° video. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 249–262.Google ScholarDigital Library
- Sandy Louchart and Ruth Aylett. 2003. Solving the Narrative Paradox in VEs – Lessons from RPGs. In Intelligent Virtual Agents, Thomas Rist, Ruth S. Aylett, Daniel Ballin, and Jeff Rickel (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 244–248.Google Scholar
- Andrew MacQuarrie and Anthony Steed. 2017. Cinematic virtual reality: Evaluating the effect of display type on the viewing experience for panoramic video. In 2017 IEEE Virtual Reality (VR). 45–54. https://doi.org/10.1109/VR.2017.7892230Google ScholarCross Ref
- Carlos Marañes, Diego Gutierrez, and Ana Serrano. 2020. Exploring the impact of 360 movie cuts in users’ attention. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 73–82.Google ScholarCross Ref
- Carlos Marañes, Diego Gutierrez, and Ana Serrano. 2022. Towards assisting the decision-making process for content creators in cinematic virtual reality through the analysis of movie cuts and their influence on viewers’ behavior. International Transactions in Operational Research (2022).Google Scholar
- Belen Masia, Javier Camon, Diego Gutierrez, and Ana Serrano. 2021. Influence of Directional Sound Cues on Users’ Exploration Across 360° Movie Cuts. IEEE Computer Graphics and Applications 41, 4 (2021), 64–75.Google ScholarCross Ref
- James McCrae, Michael Glueck, Tovi Grossman, Azam Khan, and Karan Singh. 2010. Exploring the Design Space of Multiscale 3D Orientation. In Proceedings of the International Conference on Advanced Visual Interfaces (Roma, Italy) (AVI ’10). Association for Computing Machinery, New York, NY, USA, 81–88. https://doi.org/10.1145/1842993.1843008Google ScholarDigital Library
- James McCrae, Igor Mordatch, Michael Glueck, and Azam Khan. 2009. Multiscale 3D Navigation. In Proceedings of the 2009 Symposium on Interactive 3D Graphics and Games (Boston, Massachusetts) (I3D ’09). Association for Computing Machinery, New York, NY, USA, 7–14. https://doi.org/10.1145/1507149.1507151Google ScholarDigital Library
- National Geographic. 2018. Elephant Encounter in 360 - Ep. 2 | The Okavango Experience. https://www.youtube.com/watch?v=HI7mTIxNotQ.Google Scholar
- Cuong Nguyen, Stephen DiVerdi, Aaron Hertzmann, and Feng Liu. 2017. Vremiere: In-headset virtual reality video editing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5428–5438.Google ScholarDigital Library
- Lasse T. Nielsen, Matias B. Møller, Sune D. Hartmeyer, Troels C. M. Ljung, Niels C. Nilsson, Rolf Nordahl, and Stefania Serafin. 2016. Missing the Point: An Exploration of How to Guide Users’ Attention during Cinematic Virtual Reality. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology (Munich, Germany) (VRST ’16). Association for Computing Machinery, New York, NY, USA, 229–232. https://doi.org/10.1145/2993369.2993405Google ScholarDigital Library
- Orbitian Media. 2022. One Day in London - VR/360° guided city tour (8K resolution). https://www.youtube.com/watch?v=v_4_gdbltyM. Copyright 2022 by Orbitian Media: https://orbitian.media. Reprinted with permission.Google Scholar
- Amy Pavel, Björn Hartmann, and Maneesh Agrawala. 2017. Shot orientation controls for interactive cinematography with 360 video. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 289–297.Google ScholarDigital Library
- RenderHeads. 2022. AVPro Video. https://www.renderheads.com/content/docs/AVProVideo/articles/intro.htmlGoogle Scholar
- Sylvia Rothe, Felix Althammer, and Mohamed Khamis. 2018. GazeRecall: Using Gaze Direction to Increase Recall of Details in Cinematic Virtual Reality. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (Cairo, Egypt) (MUM 2018). Association for Computing Machinery, New York, NY, USA, 115–119. https://doi.org/10.1145/3282894.3282903Google ScholarDigital Library
- Sylvia Rothe, Daniel Buschek, and Heinrich Hußmann. 2019. Guidance in Cinematic Virtual Reality-Taxonomy, Research Status and Challenges. Multimodal Technologies and Interaction 3, 1 (2019). https://doi.org/10.3390/mti3010019Google ScholarCross Ref
- Sylvia Rothe and Heinrich Hußmann. 2018. Guiding the viewer in cinematic virtual reality by diegetic cues. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics. Springer, 101–117.Google ScholarCross Ref
- School of Rock the Musical. 2015. SCHOOL OF ROCK: The Musical – “You’re in the Band” (360 Video). https://www.youtube.com/watch?v=GFRPXRhBYOI.Google Scholar
- Ana Serrano, Vincent Sitzmann, Jaime Ruiz-Borau, Gordon Wetzstein, Diego Gutierrez, and Belen Masia. 2017. Movie Editing and Cognitive Event Segmentation in Virtual Reality Video. ACM Trans. Graph. 36, 4, Article 47 (jul 2017), 12 pages. https://doi.org/10.1145/3072959.3073668Google ScholarDigital Library
- Alia Sheikh, Andy Brown, Zillah Watson, and Michael Evans. 2016. Directing attention in 360-degree video. (2016).Google Scholar
- Shakeeb Shirazi. 2018. Timeline visualization of omnidirectional videos. Master’s thesis.Google Scholar
- Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2017. How do people explore virtual environments?IEEE Transactions on Visualization and Computer Graphics (2017).Google Scholar
- Marco Speicher, Christoph Rosenberg, Donald Degraen, Florian Daiber, and Antonio Krúger. 2019. Exploring Visual Guidance in 360-Degree Videos. In Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video (Salford (Manchester), United Kingdom) (TVX ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3317697.3323350Google ScholarDigital Library
- TL;DR Media. 2023. Simple Radar for Counter-Strike. https://readtldr.gg/simpleradarGoogle Scholar
- Jan Oliver Wallgrün, Mahda M. Bagher, Pejman Sajjadi, and Alexander Klippel. 2020. A Comparison of Visual Attention Guiding Approaches for 360° Image-Based VR Tours. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 83–91. https://doi.org/10.1109/VR46266.2020.00026Google ScholarCross Ref
- Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials (2007), 1–3.Google Scholar
- Stefanie Zollmann, Tobias Langlotz, Raphael Grasset, Wei Hong Lo, Shohei Mori, and Holger Regenbrecht. 2021. Visualization Techniques in Augmented Reality: A Taxonomy, Methods and Patterns. IEEE Transactions on Visualization and Computer Graphics 27, 9 (2021), 3808–3825. https://doi.org/10.1109/TVCG.2020.2986247Google ScholarDigital Library
Index Terms
- RadarVR: Exploring Spatiotemporal Visual Guidance in Cinematic VR
Recommendations
Exploring Visual Guidance in 360-degree Videos
TVX '19: Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video360-degree videos offer a novel viewing experience with the ability to explore virtual environments much more freely than before. Technologies and aesthetics behind this approach of film-making are not yet fully developed. The newly gained freedom ...
Narrative Perspectives and Embodiment in Cinematic Virtual Reality
Extended RealityAbstractAlong with the technological advancements of virtual reality over the years, has come the emergence of Cinematic Virtual Reality (CVR), where immersive 360-degree video approaches the high-quality found in feature film. Extensive research has been ...
The Transmutation of Perception: Research of Attention and Visual Guidance in Virtual Reality Context
CHI PLAY '18 Extended Abstracts: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended AbstractsAs the panoramic experience of virtual reality (VR) breaks the 'frame' of traditional film, the old audio-visual language is dissolving and a new system has not yet been established. So far, developers and researchers have not reached a conclusion about ...
Comments