skip to main content
10.1145/3544549.3585853acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
Work in Progress

Spatiality and Semantics - Towards Understanding Content Placement in Mixed Reality

Published: 19 April 2023 Publication History

Abstract

Mixed Reality (MR) popularizes numerous situated applications where virtual content is spatially integrated into our physical environment. However, we only know little about what properties of an environment influence the way how people place digital content and perceive the resulting layout. We thus conducted a preliminary study (N = 8) examining how physical surfaces affect organizing virtual content like documents or charts, focusing on user perception and experience. We found, among others, that the situated layout of virtual content in its environment can be characterized by the level of spatial as well as semantic coupling. Consequently, we propose a two-dimensional design space to establish the vocabularies and detail their parameters for content organization. With our work, we aim to facilitate communication between designers or researchers, inform general MR interface design, and provide a first step towards future MR workspaces empowered by blending digital content and its real-world context.

Supplementary Material

MP4 File (3544549.3585853-talk-video.mp4)
Pre-recorded Video Presentation
MP4 File (3544549.3585853-video-preview.mp4)
Video Preview

References

[1]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa
[2]
Andrea Bravo and Anja M. Maier. 2020. Immersive visualisations in design: Using augmented reality (AR) for information presentation. Proceedings of the Design Society: DESIGN Conference 1, 1215–1224. https://doi.org/10.1017/dsd.2020.33
[3]
Andrea Bravo, Anja M. Maier, and Philip J. Cash. 2021. Watch that seam! Designing hybrid presentations with data visualisation in augmented reality. International Journal of Design 15, 2 (2021). Num Pages: 15 Number: 2.
[4]
Wolfgang Büschel, Annett Mitschick, and Raimund Dachselt. 2018. Here and Now: Reality-Based Information Retrieval: Perspective Paper. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval (New Brunswick, NJ, USA) (CHIIR ’18). Association for Computing Machinery, New York, NY, USA, 171–180. https://doi.org/10.1145/3176349.3176384
[5]
Han Joo Chae, Youli Chang, Minji Kim, Gwanmo Park, and Jinwook Seo. 2020. ARphy: Managing Photo Collections Using Physical Objects in AR. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI EA ’20). Association for Computing Machinery, New York, NY, USA, 1–7. https://doi.org/10.1145/3334480.3382885
[6]
Yifei Cheng, Yukang Yan, Xin Yi, Yuanchun Shi, and David Lindlbauer. 2021. SemanticAdapt: Optimization-Based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic Connections. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 282–297. https://doi.org/10.1145/3472749.3474750
[7]
Kylie Davidson, Lee Lisle, Kirsten Whitley, Doug A. Bowman, and Chris North. 2022. Exploring the Evolution of Sensemaking Strategies in Immersive Space to Think. IEEE Transactions on Visualization and Computer Graphics (2022), 1–15. https://doi.org/10.1109/TVCG.2022.3207357
[8]
Neven ElSayed, Bruce Thomas, Kim Marriott, Julia Piantadosi, and Ross Smith. 2015. Situated Analytics. In 2015 Big Data Visual Analytics (BDVA). 1–8. https://doi.org/10.1109/BDVA.2015.7314302
[9]
Barrett Ens, Benjamin Bach, Maxime Cordeil, Ulrich Engelke, Marcos Serrano, Wesley Willett, Arnaud Prouzeau, Christoph Anthes, Wolfgang Büschel, Cody Dunne, Tim Dwyer, Jens Grubert, Jason H. Haga, Nurit Kirshenbaum, Dylan Kobayashi, Tica Lin, Monsurat Olaosebikan, Fabian Pointecker, David Saffo, Nazmus Saquib, Dieter Schmalstieg, Danielle Albers Szafir, Matt Whitlock, and Yalong Yang. 2021. Grand Challenges in Immersive Analytics. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 459, 17 pages. https://doi.org/10.1145/3411764.3446866
[10]
Barrett Ens and Pourang Irani. 2017. Spatial Analytic Interfaces: Spatial User Interfaces for In Situ Visual Analytics. IEEE Computer Graphics and Applications 37, 2 (2017), 66–79. https://doi.org/10.1109/MCG.2016.38
[11]
Barrett Ens, Eyal Ofek, Neil Bruce, and Pourang Irani. 2015. Spatial Constancy of Surface-Embedded Layouts across Multiple Environments. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction (Los Angeles, California, USA) (SUI ’15). Association for Computing Machinery, New York, NY, USA, 65–68. https://doi.org/10.1145/2788940.2788954
[12]
Barrett M. Ens, Rory Finnegan, and Pourang P. Irani. 2014. The Personal Cockpit: A Spatial Interface for Effective Task Switching on Head-Worn Displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3171–3180. https://doi.org/10.1145/2556288.2557058
[13]
João Marcelo Evangelista Belo, Mathias N. Lystbæk, Anna Maria Feit, Ken Pfeuffer, Peter Kán, Antti Oulasvirta, and Kaj Grønbæk. 2022. AUIT - the Adaptive User Interfaces Toolkit for Designing XR Applications. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology(UIST ’22). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3526113.3545651
[14]
Zhenyi He, Ruofei Du, and Ken Perlin. 2020. CollaboVR: A Reconfigurable Framework for Creative Collaboration in Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 542–554. https://doi.org/10.1109/ISMAR50242.2020.00082
[15]
Dominik Herr, Jan Reinhardt, Guido Reina, Robert Krüger, Rafael V. Ferrari, and Thomas Ertl. 2018. Immersive Modular Factory Layout Planning using Augmented Reality. Procedia CIRP 72 (2018), 1112–1117. https://doi.org/10.1016/j.procir.2018.03.200
[16]
Daekun Kim and Daniel Vogel. 2022. Everywhere Cursor: Extending Desktop Mouse Interaction into Spatial Augmented Reality: Extended Abstract. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22). Association for Computing Machinery, New York, NY, USA, Article 389, 7 pages. https://doi.org/10.1145/3491101.3519796
[17]
Benjamin Lee, Xiaoyun Hu, Maxime Cordeil, Arnaud Prouzeau, Bernhard Jenny, and Tim Dwyer. 2021. Shared Surfaces and Spaces: Collaborative Data Visualisation in a Co-located Immersive Environment. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2021), 1171–1181. https://doi.org/10.1109/TVCG.2020.3030450
[18]
Zhen Li, Michelle Annett, Ken Hinckley, Karan Singh, and Daniel Wigdor. 2019. HoloDoc: Enabling Mixed Reality Workspaces That Harness Physical and Digital Content. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300917
[19]
Jiazhou Liu, Arnaud Prouzeau, Barrett Ens, and Tim Dwyer. 2020. Design and Evaluation of Interactive Small Multiples Data Visualisation in Immersive Spaces. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 588–597. https://doi.org/10.1109/VR46266.2020.00081
[20]
Jiazhou Liu, Arnaud Prouzeau, Barrett Ens, and Tim Dwyer. 2022. Effects of Display Layout on Spatial Memory for Immersive Environments. Proc. ACM Hum.-Comput. Interact. 6, ISS, Article 576, 21 pages. https://doi.org/10.1145/3567729
[21]
Feiyu Lu and Doug A. Bowman. 2021. Evaluating the Potential of Glanceable AR Interfaces for Authentic Everyday Uses. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 768–777. https://doi.org/10.1109/VR50410.2021.00104
[22]
Weizhou Luo, Anke Lehmann, Hjalmar Widengren, and Raimund Dachselt. 2022. Where Should We Put It? Layout and Placement Strategies of Documents in Augmented Reality for Collaborative Sensemaking. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 627, 16 pages. https://doi.org/10.1145/3491102.3501946
[23]
Weizhou Luo, Anke Lehmann, Yushan Yang, and Raimund Dachselt. 2021. Investigating Document Layout and Placement Strategies for Collaborative Sensemaking in Augmented Reality(CHI EA ’21). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411763.3451588
[24]
Weizhou Luo, Zhongyuan Yu, Rufat Rzayev, Marc Satkowski, Stefan Gumhold, Matthew McGinity, and Raimund Dachselt. 2023. PEARL: Physical Environment based Augmented Reality Lenses for In-Situ Human Movement Analysis. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, 15 pages. https://doi.org/10.1145/3544548.3580715
[25]
Fabrice Matulic, Wolfgang Büschel, Michael Ying Yang, Stephan Ihrke, Anmol Ramraika, Carsten Rother, and Raimund Dachselt. 2016. Smart Ubiquitous Projection: Discovering Surfaces for the Projection of Adaptive Content. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (San Jose, California, USA) (CHI EA ’16). Association for Computing Machinery, New York, NY, USA, 2592–2600. https://doi.org/10.1145/2851581.2892545
[26]
Daniel Medeiros, Mark McGill, Alexander Ng, Robert McDermid, Nadia Pantidi, Julie Williamson, and Stephen Brewster. 2022. From Shielding to Avoidance: Passenger Augmented Reality and the Layout of Virtual Displays for Productivity in Shared Transit. IEEE Transactions on Visualization and Computer Graphics 28, 11 (2022), 3640–3650. https://doi.org/10.1109/TVCG.2022.3203002
[27]
Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D. Wilson. 2016. SnapToReality: Aligning Augmented Reality to the Real World. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 1233–1244. https://doi.org/10.1145/2858036.2858250
[28]
Leonardo Pavanatto, Chris North, Doug A. Bowman, Carmen Badea, and Richard Stoakley. 2021. Do we still need physical monitors? An evaluation of the usability of AR virtual monitors for productivity work. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 759–767. https://doi.org/10.1109/VR50410.2021.00103
[29]
Patrick Reipschläger and Raimund Dachselt. 2019. DesignAR: Immersive 3D-Modeling Combining Augmented Reality with Interactive Displays. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS ’19). Association for Computing Machinery, New York, NY, USA, 29–41. https://doi.org/10.1145/3343055.3359718
[30]
Patrick Reipschlager, Tamara Flemisch, and Raimund Dachselt. 2021. Personal Augmented Reality for Information Visualization on Large Interactive Displays. IEEE Transactions on Visualization and Computer Graphics 27, 2 (2021), 1182–1192. https://doi.org/10.1109/TVCG.2020.3030460
[31]
Marc Satkowski and Raimund Dachselt. 2021. Investigating the Impact of Real-World Environments on the Perception of 2D Visualizations in Augmented Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 522, 15 pages. https://doi.org/10.1145/3411764.3445330
[32]
Marc Satkowski, Rufat Rzayev, Eva Goebel, and Raimund Dachselt. 2022. ABOVE & BELOW: Investigating Ceiling and Floor for Augmented Reality Content Placement. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 518–527. https://doi.org/10.1109/ISMAR55827.2022.00068
[33]
Kadek Ananta Satriadi, Andrew Cunningham, Bruce H. Thomas, Adam Drogemuller, Antoine Odi, Niki Patel, Cathlyn Aston, and Ross T. Smith. 2022. Augmented Scale Models: Presenting Multivariate Data Around Physical Scale Models in Augmented Reality. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 54–63. https://doi.org/10.1109/ISMAR55827.2022.00019
[34]
Kadek Ananta Satriadi, Barrett Ens, Maxime Cordeil, Tobias Czauderna, and Bernhard Jenny. 2020. Maps Around Me: 3D Multiview Layouts in Immersive Spaces. Proc. ACM Hum.-Comput. Interact. 4, ISS, Article 201 (nov 2020), 20 pages. https://doi.org/10.1145/3427329
[35]
Kadek Ananta Satriadi, Jim Smiley, Barrett Ens, Maxime Cordeil, Tobias Czauderna, Benjamin Lee, Ying Yang, Tim Dwyer, and Bernhard Jenny. 2022. Tangible Globes for Data Visualisation in Augmented Reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 505, 16 pages. https://doi.org/10.1145/3491102.3517715
[36]
Dominik Schmidt, Johannes Frohnhofen, Sven Knebel, Florian Meinel, Mariya Perchyk, Julian Risch, Jonathan Striebel, Julia Wachtel, and Patrick Baudisch. 2015. Ergonomic Interaction for Touch Floors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3879–3888. https://doi.org/10.1145/2702123.2702254
[37]
Ryo Suzuki, Rubaiat Habib Kazi, Li-yi Wei, Stephen DiVerdi, Wilmot Li, and Daniel Leithinger. 2020. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 166–181. https://doi.org/10.1145/3379337.3415892
[38]
Xiyao Wang, Lonni Besançon, David Rousseau, Mickael Sereno, Mehdi Ammi, and Tobias Isenberg. 2020. Towards an Understanding of Augmented Reality Extensions for Existing 3D Data Analysis Tools. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376657
[39]
Zhen Wen, Wei Zeng, Luoxuan Weng, Yihan Liu, Mingliang Xu, and Wei Chen. 2023. Effects of View Layout on Situated Analytics for Multiple-View Representations in Immersive Visualization. IEEE Transactions on Visualization and Computer Graphics 29, 1 (2023), 440–450. https://doi.org/10.1109/TVCG.2022.3209475
[40]
Wesley Willett, Yvonne Jansen, and Pierre Dragicevic. 2017. Embedded Data Representations. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 461–470. https://doi.org/10.1109/TVCG.2016.2598608

Cited By

View all
  • (2025)Towards gamification for spatial digital learning environmentsEntertainment Computing10.1016/j.entcom.2024.10089352(100893)Online publication date: Jan-2025
  • (2024)G-VOILA: Gaze-Facilitated Information Querying in Daily ScenariosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596238:2(1-33)Online publication date: 15-May-2024
  • (2024)The Jamais Vu Effect: Understanding the Fragile Illusion of Co-Presence in Mixed RealityProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661574(2227-2246)Online publication date: 1-Jul-2024
  • Show More Cited By

Index Terms

  1. Spatiality and Semantics - Towards Understanding Content Placement in Mixed Reality

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
      April 2023
      3914 pages
      ISBN:9781450394222
      DOI:10.1145/3544549
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 April 2023

      Check for updates

      Author Tags

      1. Augemented Reality
      2. Content Organization
      3. Design Space
      4. Layout
      5. Mixed Reality
      6. User Study

      Qualifiers

      • Work in progress
      • Research
      • Refereed limited

      Funding Sources

      Conference

      CHI '23
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

      Upcoming Conference

      CHI 2025
      ACM CHI Conference on Human Factors in Computing Systems
      April 26 - May 1, 2025
      Yokohama , Japan

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)674
      • Downloads (Last 6 weeks)109
      Reflects downloads up to 08 Mar 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Towards gamification for spatial digital learning environmentsEntertainment Computing10.1016/j.entcom.2024.10089352(100893)Online publication date: Jan-2025
      • (2024)G-VOILA: Gaze-Facilitated Information Querying in Daily ScenariosProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596238:2(1-33)Online publication date: 15-May-2024
      • (2024)The Jamais Vu Effect: Understanding the Fragile Illusion of Co-Presence in Mixed RealityProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661574(2227-2246)Online publication date: 1-Jul-2024
      • (2024)Exploring Spatial Organization Strategies for Virtual Content in Mixed Reality EnvironmentsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638181(1-6)Online publication date: 11-May-2024
      • (2024)Gaze on the Go: Effect of Spatial Reference Frame on Visual Target Acquisition During Physical Locomotion in Extended RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642915(1-16)Online publication date: 11-May-2024
      • (2024)A Systematic Literature Review of User Evaluation in Immersive AnalyticsComputer Graphics Forum10.1111/cgf.1511143:3Online publication date: 10-Jun-2024
      • (2024)Tangible Sumeru: Enhancing AR Interactive Narrative Experience with 3D-Printed Assemblies for Dunhuang Grotto MuralsHCI International 2024 – Late Breaking Posters10.1007/978-3-031-78561-0_31(308-318)Online publication date: 30-Dec-2024
      • (2023)How Does Explainability Look in Hybrid User Interfaces?2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct60411.2023.00058(251-256)Online publication date: 16-Oct-2023

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      HTML Format

      View this article in HTML Format.

      HTML Format

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media