ABSTRACT
Mixed Reality (MR) remote collaboration provides an enhanced immersive experience where a remote user can provide verbal and nonverbal assistance to a local user to increase the efficiency and performance of the collaboration. This is usually achieved by sharing the local user's environment through live 360 video or a 3D scene, and using visual cues to gesture or point at real objects allowing for better understanding and collaborative task performance. While most of prior work used one of the methods to capture the surrounding environment, there may be situations where users have to choose between using 360 panoramas or 3D scene reconstruction to collaborate, as each have unique benefits and limitations. In this paper we designed a prototype system that combines 360 panoramas into a 3D scene to introduce a novel way for users to interact and collaborate with each other. We evaluated the prototype through a user study which compared the usability and performance of our proposed approach to live 360 video collaborative system, and we found that participants enjoyed using different ways to access the local user's environment although it took them longer time to learn to use our system. We also collected subjective feedback for future improvements and provide directions for future research.
- M. Adcock, S. Anderson, and B. Thomas, RemoteFusion. Hong Kong, Hong Kong: ACM, 2013, pp. 235–242.Google Scholar
- M. Adcock and C. Gunn, “Using Projected Light for Mobile Remote Guidance,” Comput. Support. Coop. Work, vol. 24, no. 6, pp. 591–611, Dec. 2015.Google ScholarDigital Library
- M. Billinghurst, A. Nassani, and C. Reichherzer, “Social panoramas,” in SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications on - SA ’14, 2014, pp. 1–1.Google ScholarDigital Library
- J. Brooke, “SUS: a quick and dirty usability scale,” In P.W.Jordan, B. Thomas, B.A. Weerdmeester, and I.L. McClelland (Eds.) Usability Evaluation in Industry. London: Taylor and Francis, pp. 189–194, 1996.Google Scholar
- S. E. Chen and S. Eric, “QuickTime VR,” in Proceedings of the 22nd annual conference on Computer graphics and interactive techniques - SIGGRAPH ’95, 1995, pp. 29–38.Google ScholarDigital Library
- A. Dai, M. Nienerbner, M. Zollhofer, S. Izadi, and C. Theobalt, “BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration,” ACM Trans. Graph., vol. 36, no. 4, p. 1, 2017.Google ScholarDigital Library
- C. Elvezio, M. Sukan, O. Oda, S. Feiner, and B. Tversky, “Remote collaboration in AR and VR using virtual replicas,” ACM SIGGRAPH 2017 VR Village on - SIGGRAPH ’17. ACM, Los Angeles, California, pp. 1–2, 2017.Google ScholarDigital Library
- O. Fakourfar, K. Ta, R. Tang, S. Bateman, and A. Tang, “Stabilized Annotations for Mobile Remote Assistance,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16, 2016, pp. 1548–1560.Google ScholarDigital Library
- D. Farin, W. Effelsberg, and P. H. N. de With, “Floor-plan reconstruction from panoramic images,” Proceedings of the 15th ACM international conference on Multimedia. ACM, Augsburg, Germany, pp. 823–826, 2007.Google ScholarDigital Library
- S. Gauglitz, C. Lee, M. Turk, and T. Höllerer, “Integrating the physical environment into mobile remote collaboration,” in Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services - MobileHCI ’12, 2012, p. 241.Google ScholarDigital Library
- S. Gauglitz, B. Nuernberger, M. Turk, and T. Höllerer, “In touch with the remote world,” Proc. 20th ACM Symp. Virtual Real. Softw. Technol. - VRST ’14, pp. 197–205, 2014.Google ScholarDigital Library
- S. Gauglitz, B. Nuernberger, M. Turk, and T. Höllerer, “World-stabilized annotations and virtual scene navigation for remote collaboration,” in Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST ’14, 2014, pp. 449–459.Google ScholarDigital Library
- S. Izadi, R. A. Newcombe, D. Kim, O. Hilliges, D. Molyneaux, S. Hodges, P. Kohli, J. Shotton, A. J. Davison and A. Fitzgibbon ”KinectFusion: real-time dynamic 3D surface reconstruction and interaction,” ACM SIGGRAPH 2011 Talks. ACM, Vancouver, British Columbia, Canada, p. 1, 2011.Google ScholarDigital Library
- M. Joachimczak, J. Liu, and H. Ando, “Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors,” Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, Glasgow, UK, pp. 514–515, 2017.Google ScholarDigital Library
- S. Kasahara and J. Rekimoto, “JackIn head: immersive visual telepresence system with omnidirectional wearable camera for remote collaboration,” Proc. 21st ACM Symp. Virtual Real. Softw. Technol., vol. 23, no. 3, pp. 217–225, 2015.Google ScholarDigital Library
- S. Kim, G. A. Lee, S. Ha, N. Sakata, and M. Billinghurst, “Automatically Freezing Live Video for Annotation during Remote Collaboration,” in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15, 2015, pp. 1669–1674.Google ScholarDigital Library
- S. Kim, G. Lee, W. Huang, H. Kim, W. Woo, and M. Billinghurst, “Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–13.Google ScholarDigital Library
- G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “A User Study on MR Remote Collaboration Using Live 360 Video,” in2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2018, pp. 153–164.Google Scholar
- G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “Mixed reality collaboration through sharing a live panorama,” SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. ACM, Bangkok, Thailand, pp. 1–4, 2017.Google ScholarDigital Library
- G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “Sharedsphere: MR collaboration through shared live panorama,” SIGGRAPH Asia 2017 Emerging Technologies. ACM, Bangkok, Thailand, pp. 1–2, 2017.Google ScholarDigital Library
- B. Nuernberger, K.-C. Lien, L. Grinta, C. Sweeney, M. Turk, and T. Höllerer, “Multi-view gesture annotations in image-based 3D reconstructed scenes,” in Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology - VRST ’16, 2016, pp. 129–138.Google ScholarDigital Library
- B. Nuernberger, K. C. Lien, T. Hollerer, and M. Turk, “Interpreting 2D gesture annotations in 3D augmented reality,” in2016 IEEE Symposium on 3D User Interfaces, 3DUI 2016 - Proceedings, 2016, pp. 149–158.Google ScholarCross Ref
- O. Oda, C. Elvezio, M. Sukan, S. Feiner, and B. Tversky, “Virtual Replicas for Remote Assistance in Virtual and Augmented Reality,” Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST ’15. ACM, Charlotte, NC, USA, pp. 405–415, 2015.Google ScholarDigital Library
- S. Orts-Escolano, M. Dou, V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep, S. Wang, S. B. Kang, C. Rhemann, P. Kohli, Y. Lutchyn, C. Keskin, S. Izadi, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson and S. Khamis, “Holoportation: Virtual 3D Teleportation in Real-time,” Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo, Japan, pp. 741–754, 2016.Google ScholarDigital Library
- T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. D. Wilson, “Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment,” Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW ’16. ACM, San Francisco, California, USA, pp. 1714–1723, 2016.Google ScholarDigital Library
- T. Piumsomboon, A. Day, B. Ens, Y. Lee, G. Lee, and M. Billinghurst, “Exploring enhancements for remote mixed reality collaboration,” SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA ’17. ACM, Bangkok, Thailand, pp. 1–5, 2017.Google Scholar
- T. Piumsomboon, G. A. Lee, A. Irlitti, B. Ens, B. H. Thomas, and M. Billinghurst, “On the Shoulder of the Giant,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–17.Google ScholarDigital Library
- T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H. Thomas and M. Billinghurst, “Mini-Me,” Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM, Montreal QC, Canada, pp. 1–13, 2018.Google ScholarDigital Library
- T. Piumsomboon, Y. Lee, G. Lee, and M. Billinghurst, “CoVAR,” SIGGRAPH Asia 2017 Emerging Technologies on - SA ’17. ACM, Bangkok, Thailand, pp. 1–2, 2017.Google Scholar
- K. M. Rahimi, C. Banigan, and E. D. Ragan, “Scene Transitions and Teleportation in Virtual Reality and the Implications for Spatial Awareness and Sickness,” IEEE Trans. Vis. Comput. Graph., pp. 1–1, 2018.Google ScholarCross Ref
- M. Taitand M. Billinghurst, “The Effect of View Independence in a Collaborative AR System,” Comput. Support. Coop. Work, vol. 24, no. 6, pp. 563–589, 2015.Google ScholarDigital Library
- A. Tang and O. Fakourfar, “Watching 360° Videos Together,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17, 2017, pp. 4501–4506.Google ScholarDigital Library
- A. Tang, O. Fakourfar, C. Neustaedter, and S. Bateman, “Collaboration in 360° Videochat: Challenges and Opportunities,” Proc. 2017 Conf. Des. Interact. Syst., pp. 1327–1339, 2017.Google ScholarDigital Library
- T. Teo, L. Lawrence, G. A. Lee, M. Billinghurst, and M. Adcock, “Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–14.Google ScholarDigital Library
- T. Teo, G. A. Lee, M. Billinghurst, and M. Adcock, “Hand gestures and visual annotation in live 360 panorama-based mixed reality remote collaboration,” in Proceedings of the 30th Australian Conference on Computer-Human Interaction - OzCHI ’18, 2018, pp. 406–410.Google ScholarDigital Library
Recommendations
Exploring enhancements for remote mixed reality collaboration
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive ApplicationsIn this paper, we explore techniques for enhancing remote Mixed Reality (MR) collaboration in terms of communication and interaction. We created CoVAR, a MR system for remote collaboration between an Augmented Reality (AR) and Augmented Virtuality (AV) ...
Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing SystemsRemote Collaboration using Virtual Reality (VR) and Augmented Reality (AR) has recently become a popular way for people from different places to work together. Local workers can collaborate with remote helpers by sharing 360-degree live video or 3D ...
Merging environments for shared spaces in mixed reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologyIn virtual reality a real walking interface limits the extent of a virtual environment to our local walkable space. As local spaces are specific to each user, sharing a virtual environment with others for collaborative work or games becomes complicated. ...
Comments