skip to main content
10.1145/3411764.3445246acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

Do You Really Need to Know Where “That” Is? Enhancing Support for Referencing in Collaborative Mixed Reality Environments

Published: 07 May 2021 Publication History

Abstract

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks. Conversations during these tasks are heavily anchored around task objects and their spatial relationships in the real world, making referencing - the ability to refer to an object in a way that is understood by others - a crucial process that warrants explicit support in collaborative Mixed Reality systems. This paper presents a 2x2 mixed factorial experiment that explores the effects of providing spatial information and system-generated guidance to task objects. It also investigates the effects of such guidance on the remote collaborator’s need for spatial information. Our results show that guidance increases performance and communication efficiency while reducing the need for spatial information, especially in unfamiliar environments. Our results also demonstrate a reduced need for remote experts to be in immersive environments, making guidance more scalable, and expertise more accessible.

References

[1]
Matt Adcock, Stuart Anderson, and Bruce Thomas. 2013. RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks. In Proceedings of the 12th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry. ACM, 235–242.
[2]
Ferran Argelaguet, Alexander Kulik, André Kunert, Carlos Andujar, and Bernd Froehlich. 2011. See-through techniques for referential awareness in collaborative virtual reality. International Journal of Human-Computer Studies 69, 6 (2011), 387–400.
[3]
Huidong Bai, Prasanth Sasikumar, Jing Yang, and Mark Billinghurst. 2020. A User Study on Mixed Reality Remote Collaboration with Eye Gaze and Hand Gesture Sharing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376550
[4]
Martin Bauer, Gerd Kortuem, and Zary Segall. 1999. ” Where are you pointing at?” A study of remote collaboration in a wearable videoconference system. In Digest of Papers. Third International Symposium on Wearable Computers. IEEE, 151–158.
[5]
Mark Billinghurst and Hirokazu Kato. 2002. Collaborative augmented reality. Commun. ACM 45, 7 (2002), 64–70.
[6]
Frank Biocca, Arthur Tang, Charles Owen, and Fan Xiao. 2006. Attention funnel: omnidirectional 3D cursor for mobile augmented reality platforms. In Proceedings of the SIGCHI conference on Human Factors in computing systems. 1115–1122.
[7]
Sébastien Bottecchia, Jean-Marc Cieutat, and Jean-Pierre Jessel. 2010. TAC: augmented reality system for collaborative tele-assistance in the field of maintenance through internet. In Proceedings of the 1st Augmented Human International Conference. 1–7.
[8]
Yuan-Chia Chang, Hao-Chuan Wang, Hung-kuo Chu, Shung-Ying Lin, and Shuo-Ping Wang. 2017. AlphaRead: Support unambiguous referencing in remote collaboration with readable object annotation. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 2246–2259.
[9]
Jeffrey W Chastine, Kristine Nagel, Ying Zhu, and Luca Yearsovich. 2007. Understanding the design space of referencing in collaborative augmented reality environments. In Proceedings of graphics interface 2007. ACM, 207–214.
[10]
Jeffrey W Chastine, Ying Zhu, and Jon A Preston. 2006. A framework for inter-referential awareness in collaborative environments. In 2006 International Conference on Collaborative Computing: Networking, Applications and Worksharing. IEEE, 1–5.
[11]
Long Chen, Thomas W Day, Wen Tang, and Nigel W John. 2017. Recent developments and future challenges in medical mixed reality. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 123–135.
[12]
Herbert H Clark and Susan E Brennan. 1991. Grounding in communication.(1991).
[13]
Herbert H Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition 22, 1 (1986), 1–39.
[14]
Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika 16, 3 (1951), 297–334.
[15]
Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz studies: why and how. In Proceedings of the 1st international conference on Intelligent user interfaces. 193–200.
[16]
Archi Dasgupta, Mark Manuel, Rifat Sabbir Mansur, Nabil Nowak, and Denis Gračanin. 2020. Towards Real Time Object Recognition For Context Awareness in Mixed Reality: A Machine Learning Approach. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 262–268.
[17]
Ryan Anthony J de Belen, Huyen Nguyen, Daniel Filonik, Dennis Del Favero, and Tomasz Bednarz. 2019. A systematic review of the current state of collaborative mixed reality technologies: 2013–2018. (2019).
[18]
Alan Dix. 1994. Computer supported cooperative work: A framework. In Design issues in CSCW. Springer, 9–26.
[19]
Paul Dourish and Victoria Bellotti. 1992. Awareness and coordination in shared workspaces. In CSCW, Vol. 92. 107–114.
[20]
Steven Dow, Jaemin Lee, Christopher Oezbek, Blair MacIntyre, Jay David Bolter, and Maribeth Gandy. 2005. Wizard of Oz interfaces for mixed reality applications. In CHI’05 extended abstracts on human factors in computing systems. 1339–1342.
[21]
MR Ensley. 1995. Toward a theory of situation awareness in dynamic systems. Human factors 37(1995), 85–104.
[22]
Olivier Friard and Marco Gamba. 2016. BORIS: a free, versatile open-source event-logging software for video/audio coding and live observations. Methods in Ecology and Evolution 7, 11 (2016), 1325–1330.
[23]
Susan R Fussell, Robert E Kraut, and Jane Siegel. 2000. Coordination of communication: Effects of shared visual context on collaborative work. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 21–30.
[24]
Susan R Fussell, Leslie D Setlock, and Robert E Kraut. 2003. Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 513–520.
[25]
Lei Gao, Huidong Bai, Rob Lindeman, and Mark Billinghurst. 2017. Static local environment capturing and sharing for MR remote collaboration. In SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. 1–6.
[26]
Steffen Gauglitz, Cha Lee, Matthew Turk, and Tobias Höllerer. 2012. Integrating the physical environment into mobile remote collaboration. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services. 241–250.
[27]
William W Gaver, Abigail Sellen, Christian Heath, and Paul Luff. 1993. One is not enough: Multiple views in a media space. In Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems. ACM, 335–341.
[28]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
[29]
Jim Hollan and Scott Stornetta. 1992. Beyond being there. In Proceedings of the SIGCHI conference on Human factors in computing systems. 119–125.
[30]
Weidong Huang and Leila Alem. 2011. Supporting hand gestures in mobile remote collaboration: a usability evaluation. In Proceedings of the 25th BCS Conference on Human-Computer Interaction. British Computer Society, 211–216.
[31]
Hiroshi Ishii, Minoru Kobayashi, and Kazuho Arita. 1994. Iterative design of seamless collaboration media. Commun. ACM 37, 8 (1994), 83–97.
[32]
Dongsik Jo and Gerard Jounghyun Kim. 2016. ARIoT: scalable augmented reality framework for interacting with Internet of Things appliances everywhere. IEEE Transactions on Consumer Electronics 62, 3 (2016), 334–340.
[33]
Michal Joachimczak, Juan Liu, and Hiroshi Ando. 2017. Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors. In Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, 514–515.
[34]
Janet G Johnson, Danilo Gasques Rodrigues, Madhuri Gubbala, and Nadir Weibel. 2018. HoloCPR: Designing and Evaluating a Mixed Reality Interface for Time-Critical Emergencies. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. ACM, 67–76.
[35]
Seungwon Kim, Gun Lee, Nobuchika Sakata, and Mark Billinghurst. 2014. Improving co-presence with augmented visual communication cues for sharing experience through video conference. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 83–92.
[36]
David Kirk and Danae Stanton Fraser. 2006. Comparing remote gesture technologies for supporting collaborative physical tasks. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, 1191–1200.
[37]
Robert E Kraut, Susan R Fussell, and Jane Siegel. 2003. Visual information as a conversational resource in collaborative physical tasks. Human–Computer Interaction 18, 1-2 (2003), 13–49.
[38]
Robert E Kraut, Mark D Miller, and Jane Siegel. 1996. Collaboration in performance of physical tasks: Effects on outcomes and communication. In Proceedings of the 1996 ACM conference on Computer supported cooperative work. ACM, 57–66.
[39]
Morgan Le Chénéchal, Thierry Duval, Valérie Gouranton, Jérôme Royan, and Bruno Arnaldi. 2016. Vishnu: virtual immersive support for HelpiNg users an interaction paradigm for collaborative remote guiding in mixed reality. In 2016 IEEE Third VR International Workshop on Collaborative Virtual Environments (3DCVE). IEEE, 9–12.
[40]
Gun A Lee, Theophilus Teo, Seungwon Kim, and Mark Billinghurst. 2018. A user study on mr remote collaboration using live 360 video. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 153–164.
[41]
Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321–1329.
[42]
Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, 2016. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 741–754.
[43]
Jiazhi Ou, Susan R Fussell, Xilin Chen, Leslie D Setlock, and Jie Yang. 2003. Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks. In Proceedings of the 5th international conference on Multimodal interfaces. ACM, 242–249.
[44]
Thomas Pechmann and Werner Deutsch. 1982. The development of verbal and nonverbal devices for reference. Journal of experimental child psychology 34, 2 (1982), 330–341.
[45]
Thammathip Piumsomboon, Gun A Lee, Andrew Irlitti, Barrett Ens, Bruce H Thomas, and Mark Billinghurst. 2019. On the shoulder of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–17.
[46]
Thammathip Piumsomboon, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. CoVAR: a collaborative virtual and augmented reality system for remote collaboration. In SIGGRAPH Asia 2017 Emerging Technologies. 1–2.
[47]
Abhishek Ranjan, Jeremy P Birnholtz, and Ravin Balakrishnan. 2006. An exploratory analysis of partner action and camera control in a video-mediated collaborative task. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work. ACM, 403–412.
[48]
Troels A Rasmussen and Weidong Huang. 2019. SceneCam: Using AR to improve Multi-Camera Remote Collaboration. In SIGGRAPH Asia 2019 XR. 36–37.
[49]
Mauro Del Rio, Vittorio Meloni, Francesca Frexia, Francesco Cabras, Roberto Tumbarello, Sabrina Montis, Andrea Marini, and Gianluigi Zanetti. 2018. Augmented Reality for Supporting Real Time Telementoring: an Exploratory Study Applied to Ultrasonography. In Proceedings of the 2nd International Conference on Medical and Health Informatics. 218–222.
[50]
Edgar Rojas-Muñoz, Maria E Cabrera, Chengyuan Lin, Daniel Andersen, Voicu Popescu, Kathryn Anderson, Ben L Zarzaur, Brian Mullis, and Juan P Wachs. 2020. The System for Telementoring with Augmented Reality (STAR): A head-mounted display to improve surgical coaching and confidence in remote areas. Surgery (2020).
[51]
Somaiieh Rokhsaritalemi, Abolghasem Sadeghi-Niaraki, and Soo-Mi Choi. 2020. A Review on Mixed Reality: Current Trends, Challenges and Prospects. Applied Sciences 10, 2 (2020), 636.
[52]
Rajinder S Sodhi, Brett R Jones, David Forsyth, Brian P Bailey, and Giuliano Maciocci. 2013. BeThere: 3D mobile collaboration with spatial input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 179–188.
[53]
Maximilian Speicher, Jingchen Cao, Ao Yu, Haihua Zhang, and Michael Nebeling. 2018. 360anywhere: Mobile ad-hoc collaboration in any environment using 360 video and augmented reality. Proceedings of the ACM on Human-Computer Interaction 2, EICS(2018), 1–20.
[54]
Aaron Stafford, Wayne Piekarski, and Bruce H Thomas. 2006. Implementation of god-like interaction techniques for supporting collaboration between outdoor AR and indoor tabletop users. In 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, 165–172.
[55]
Ohan Oda Carmine Elvezio Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual Replicas for Remote Assistance in Virtual and Augmented Reality. (2015).
[56]
Zsolt Szalavári, Dieter Schmalstieg, Anton Fuhrmann, and Michael Gervautz. 1998. “Studierstube”: An environment for collaboration in augmented reality. Virtual Reality 3, 1 (1998), 37–48.
[57]
Matthew Tait and Mark Billinghurst. 2015. The effect of view independence in a collaborative AR system. Computer Supported Cooperative Work (CSCW) 24, 6 (2015), 563–589.
[58]
Franco Tecchia, Leila Alem, and Weidong Huang. 2012. 3D helping hands: a gesture based MR system for remote collaboration. In Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry. 323–328.
[59]
Theophilus Teo, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2019. 360Drops: Mixed Reality Remote Collaboration using 360 Panoramas within the 3D Scene. In SIGGRAPH Asia 2019 Emerging Technologies. 1–2.
[60]
Theophilus Teo, Ashkan F. Hayati, Gun A. Lee, Mark Billinghurst, and Matt Adcock. 2019. A Technique for Mixed Reality Remote Collaboration using 360 Panoramas in 3D Reconstructed Scenes. In 25th ACM Symposium on Virtual Reality Software and Technology. 1–11.
[61]
Theophilus Teo, Louise Lawrence, Gun A Lee, Mark Billinghurst, and Matt Adcock. 2019. Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 201.
[62]
Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019. Loki: Facilitating remote instruction of physical tasks using bi-directional mixed-reality telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161–174.
[63]
Peng Wang, Xiaoliang Bai, Mark Billinghurst, Shusheng Zhang, Sili Wei, Guangyao Xu, Weiping He, Xiangyu Zhang, and Jie Zhang. 2020. 3DGAM: using 3D gesture and CAD models for training on mixed reality remote collaboration. Multimedia Tools and Applications(2020), 1–26.
[64]
Peng Wang, Shusheng Zhang, Xiaoliang Bai, Mark Billinghurst, Weiping He, Shuxia Wang, Xiaokun Zhang, Jiaxiang Du, and Yongxing Chen. 2019. Head Pointer or Eye Gaze: Which Helps More in MR Remote Collaboration?. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 1219–1220.
[65]
Shiyao Wang, Michael Parsons, Jordan Stone-McLean, Peter Rogers, Sarah Boyd, Kristopher Hoover, Oscar Meruvia-Pastor, Minglun Gong, and Andrew Smith. 2017. Augmented reality as a telemedicine platform for remote procedural training. Sensors 17, 10 (2017), 2294.
[66]
Tzu-Yang Wang, Yuji Sato, Mai Otsuki, Hideaki Kuzuoka, and Yusuke Suzuki. 2020. Developing an AR Remote Collaboration System with Semantic Virtual Labels and a 3D Pointer. In International Conference on Human-Computer Interaction. Springer, 407–417.
[67]
Nadir Weibel, Danilo Gasques, Janet Johnson, Thomas Sharkey, Zhuoqun Robin Xu, Xinming Zhang, Enrique Zavala, Michael Yip, and Konrad Davis. 2020. ARTEMIS: Mixed-Reality Environment for Immersive Surgical Telementoring. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–4.
[68]
Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. 143–146.
[69]
Nelson Wong and Carl Gutwin. 2010. Where are you pointing? The accuracy of deictic pointing in CVEs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1029–1038.
[70]
Nelson Wong and Carl Gutwin. 2014. Support for deictic pointing in CVEs: still fragmented after all these years’. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1377–1387.
[71]
Jing Yang, Prasanth Sasikumar, Huidong Bai, Amit Barde, Gábor Sörös, and Mark Billinghurst. 2020. The effects of spatial auditory and visual cues on mixed reality remote collaboration. Journal on Multimodal User Interfaces(2020), 1–16.
[72]
George Yule. 2016. The study of language. Cambridge university press. 128–140 pages.

Cited By

View all
  • (2024)LeARn at Home: Comparing Augmented Reality and Video Conferencing Remote TutoringProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701577(255-263)Online publication date: 1-Dec-2024
  • (2024)Improving Video Navigation for Spatial Task Tutorials by Spatially Segmenting and Situating How-To VideosProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682103(1-13)Online publication date: 7-Oct-2024
  • (2024)Communication Models for Remote GuidanceComputer‐Supported Collaboration10.1002/9781119719830.ch2(7-40)Online publication date: 24-May-2024
  • Show More Cited By

Index Terms

  1. Do You Really Need to Know Where “That” Is? Enhancing Support for Referencing in Collaborative Mixed Reality Environments
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
          May 2021
          10862 pages
          ISBN:9781450380966
          DOI:10.1145/3411764
          This work is licensed under a Creative Commons Attribution International 4.0 License.

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 07 May 2021

          Check for updates

          Author Tags

          1. Collaboration
          2. Mixed Reality
          3. Referencing
          4. Remote Guidance

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Conference

          CHI '21
          Sponsor:

          Acceptance Rates

          Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

          Upcoming Conference

          CHI 2025
          ACM CHI Conference on Human Factors in Computing Systems
          April 26 - May 1, 2025
          Yokohama , Japan

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)377
          • Downloads (Last 6 weeks)39
          Reflects downloads up to 17 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2024)LeARn at Home: Comparing Augmented Reality and Video Conferencing Remote TutoringProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701577(255-263)Online publication date: 1-Dec-2024
          • (2024)Improving Video Navigation for Spatial Task Tutorials by Spatially Segmenting and Situating How-To VideosProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682103(1-13)Online publication date: 7-Oct-2024
          • (2024)Communication Models for Remote GuidanceComputer‐Supported Collaboration10.1002/9781119719830.ch2(7-40)Online publication date: 24-May-2024
          • (2023)PaperToPlace: Transforming Instruction Documents into Spatialized and Context-Aware Mixed Reality ExperiencesProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606832(1-21)Online publication date: 29-Oct-2023
          • (2023)A Survey on Remote Assistance and Training in Mixed Reality EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324708129:5(2291-2303)Online publication date: 1-May-2023
          • (2022)Effect of View Sharing on Spatial Knowledge Acquisition in Remote CollaborationProceedings of the 2022 ACM Symposium on Spatial User Interaction10.1145/3565970.3567699(1-9)Online publication date: 1-Dec-2022
          • (2022)Designing eXtended reality experiences for the future of hybrid meetingsXRDS: Crossroads, The ACM Magazine for Students10.1145/355819029:1(20-25)Online publication date: 6-Oct-2022
          • (2022)Imitative CollaborationJournal of Visual Communication and Image Representation10.1016/j.jvcir.2022.10360088:COnline publication date: 1-Oct-2022
          • (2021)Socially Distanced: Have user evaluation methods for Immersive Technologies changed during the COVID-19 pandemic?2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct54149.2021.00094(415-420)Online publication date: Oct-2021

          View Options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Login options

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media