skip to main content
10.1145/3359996.3364238acmconferencesArticle/Chapter ViewAbstractPublication PagesvrstConference Proceedingsconference-collections
research-article

A Technique for Mixed Reality Remote Collaboration using 360 Panoramas in 3D Reconstructed Scenes

Published:12 November 2019Publication History

ABSTRACT

Mixed Reality (MR) remote collaboration provides an enhanced immersive experience where a remote user can provide verbal and nonverbal assistance to a local user to increase the efficiency and performance of the collaboration. This is usually achieved by sharing the local user's environment through live 360 video or a 3D scene, and using visual cues to gesture or point at real objects allowing for better understanding and collaborative task performance. While most of prior work used one of the methods to capture the surrounding environment, there may be situations where users have to choose between using 360 panoramas or 3D scene reconstruction to collaborate, as each have unique benefits and limitations. In this paper we designed a prototype system that combines 360 panoramas into a 3D scene to introduce a novel way for users to interact and collaborate with each other. We evaluated the prototype through a user study which compared the usability and performance of our proposed approach to live 360 video collaborative system, and we found that participants enjoyed using different ways to access the local user's environment although it took them longer time to learn to use our system. We also collected subjective feedback for future improvements and provide directions for future research.

References

  1. M. Adcock, S. Anderson, and B. Thomas, RemoteFusion. Hong Kong, Hong Kong: ACM, 2013, pp. 235–242.Google ScholarGoogle Scholar
  2. M. Adcock and C. Gunn, “Using Projected Light for Mobile Remote Guidance,” Comput. Support. Coop. Work, vol. 24, no. 6, pp. 591–611, Dec. 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. M. Billinghurst, A. Nassani, and C. Reichherzer, “Social panoramas,” in SIGGRAPH Asia 2014 Mobile Graphics and Interactive Applications on - SA ’14, 2014, pp. 1–1.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. J. Brooke, “SUS: a quick and dirty usability scale,” In P.W.Jordan, B. Thomas, B.A. Weerdmeester, and I.L. McClelland (Eds.) Usability Evaluation in Industry. London: Taylor and Francis, pp. 189–194, 1996.Google ScholarGoogle Scholar
  5. S. E. Chen and S. Eric, “QuickTime VR,” in Proceedings of the 22nd annual conference on Computer graphics and interactive techniques - SIGGRAPH ’95, 1995, pp. 29–38.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. A. Dai, M. Nienerbner, M. Zollhofer, S. Izadi, and C. Theobalt, “BundleFusion: Real-Time Globally Consistent 3D Reconstruction Using On-the-Fly Surface Reintegration,” ACM Trans. Graph., vol. 36, no. 4, p. 1, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. C. Elvezio, M. Sukan, O. Oda, S. Feiner, and B. Tversky, “Remote collaboration in AR and VR using virtual replicas,” ACM SIGGRAPH 2017 VR Village on - SIGGRAPH ’17. ACM, Los Angeles, California, pp. 1–2, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. O. Fakourfar, K. Ta, R. Tang, S. Bateman, and A. Tang, “Stabilized Annotations for Mobile Remote Assistance,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI ’16, 2016, pp. 1548–1560.Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. D. Farin, W. Effelsberg, and P. H. N. de With, “Floor-plan reconstruction from panoramic images,” Proceedings of the 15th ACM international conference on Multimedia. ACM, Augsburg, Germany, pp. 823–826, 2007.Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. S. Gauglitz, C. Lee, M. Turk, and T. Höllerer, “Integrating the physical environment into mobile remote collaboration,” in Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services - MobileHCI ’12, 2012, p. 241.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. S. Gauglitz, B. Nuernberger, M. Turk, and T. Höllerer, “In touch with the remote world,” Proc. 20th ACM Symp. Virtual Real. Softw. Technol. - VRST ’14, pp. 197–205, 2014.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. S. Gauglitz, B. Nuernberger, M. Turk, and T. Höllerer, “World-stabilized annotations and virtual scene navigation for remote collaboration,” in Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST ’14, 2014, pp. 449–459.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. S. Izadi, R. A. Newcombe, D. Kim, O. Hilliges, D. Molyneaux, S. Hodges, P. Kohli, J. Shotton, A. J. Davison and A. Fitzgibbon ”KinectFusion: real-time dynamic 3D surface reconstruction and interaction,” ACM SIGGRAPH 2011 Talks. ACM, Vancouver, British Columbia, Canada, p. 1, 2011.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. Joachimczak, J. Liu, and H. Ando, “Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors,” Proceedings of the 19th ACM International Conference on Multimodal Interaction. ACM, Glasgow, UK, pp. 514–515, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. S. Kasahara and J. Rekimoto, “JackIn head: immersive visual telepresence system with omnidirectional wearable camera for remote collaboration,” Proc. 21st ACM Symp. Virtual Real. Softw. Technol., vol. 23, no. 3, pp. 217–225, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. S. Kim, G. A. Lee, S. Ha, N. Sakata, and M. Billinghurst, “Automatically Freezing Live Video for Annotation during Remote Collaboration,” in Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA ’15, 2015, pp. 1669–1674.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. S. Kim, G. Lee, W. Huang, H. Kim, W. Woo, and M. Billinghurst, “Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–13.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “A User Study on MR Remote Collaboration Using Live 360 Video,” in2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2018, pp. 153–164.Google ScholarGoogle Scholar
  19. G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “Mixed reality collaboration through sharing a live panorama,” SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications. ACM, Bangkok, Thailand, pp. 1–4, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. G. A. Lee, T. Teo, S. Kim, and M. Billinghurst, “Sharedsphere: MR collaboration through shared live panorama,” SIGGRAPH Asia 2017 Emerging Technologies. ACM, Bangkok, Thailand, pp. 1–2, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. B. Nuernberger, K.-C. Lien, L. Grinta, C. Sweeney, M. Turk, and T. Höllerer, “Multi-view gesture annotations in image-based 3D reconstructed scenes,” in Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology - VRST ’16, 2016, pp. 129–138.Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. B. Nuernberger, K. C. Lien, T. Hollerer, and M. Turk, “Interpreting 2D gesture annotations in 3D augmented reality,” in2016 IEEE Symposium on 3D User Interfaces, 3DUI 2016 - Proceedings, 2016, pp. 149–158.Google ScholarGoogle ScholarCross RefCross Ref
  23. O. Oda, C. Elvezio, M. Sukan, S. Feiner, and B. Tversky, “Virtual Replicas for Remote Assistance in Virtual and Augmented Reality,” Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST ’15. ACM, Charlotte, NC, USA, pp. 405–415, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. S. Orts-Escolano, M. Dou, V. Tankovich, C. Loop, Q. Cai, P. A. Chou, S. Mennicken, J. Valentin, V. Pradeep, S. Wang, S. B. Kang, C. Rhemann, P. Kohli, Y. Lutchyn, C. Keskin, S. Izadi, S. Fanello, W. Chang, A. Kowdle, Y. Degtyarev, D. Kim, P. L. Davidson and S. Khamis, “Holoportation: Virtual 3D Teleportation in Real-time,” Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo, Japan, pp. 741–754, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. T. Pejsa, J. Kantor, H. Benko, E. Ofek, and A. D. Wilson, “Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment,” Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing - CSCW ’16. ACM, San Francisco, California, USA, pp. 1714–1723, 2016.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. T. Piumsomboon, A. Day, B. Ens, Y. Lee, G. Lee, and M. Billinghurst, “Exploring enhancements for remote mixed reality collaboration,” SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications on - SA ’17. ACM, Bangkok, Thailand, pp. 1–5, 2017.Google ScholarGoogle Scholar
  27. T. Piumsomboon, G. A. Lee, A. Irlitti, B. Ens, B. H. Thomas, and M. Billinghurst, “On the Shoulder of the Giant,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–17.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. T. Piumsomboon, G. A. Lee, J. D. Hart, B. Ens, R. W. Lindeman, B. H. Thomas and M. Billinghurst, “Mini-Me,” Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM, Montreal QC, Canada, pp. 1–13, 2018.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. T. Piumsomboon, Y. Lee, G. Lee, and M. Billinghurst, “CoVAR,” SIGGRAPH Asia 2017 Emerging Technologies on - SA ’17. ACM, Bangkok, Thailand, pp. 1–2, 2017.Google ScholarGoogle Scholar
  30. K. M. Rahimi, C. Banigan, and E. D. Ragan, “Scene Transitions and Teleportation in Virtual Reality and the Implications for Spatial Awareness and Sickness,” IEEE Trans. Vis. Comput. Graph., pp. 1–1, 2018.Google ScholarGoogle ScholarCross RefCross Ref
  31. M. Taitand M. Billinghurst, “The Effect of View Independence in a Collaborative AR System,” Comput. Support. Coop. Work, vol. 24, no. 6, pp. 563–589, 2015.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. A. Tang and O. Fakourfar, “Watching 360° Videos Together,” in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI ’17, 2017, pp. 4501–4506.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. A. Tang, O. Fakourfar, C. Neustaedter, and S. Bateman, “Collaboration in 360° Videochat: Challenges and Opportunities,” Proc. 2017 Conf. Des. Interact. Syst., pp. 1327–1339, 2017.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. T. Teo, L. Lawrence, G. A. Lee, M. Billinghurst, and M. Adcock, “Mixed Reality Remote Collaboration Combining 360 Video and 3D Reconstruction,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, pp. 1–14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. T. Teo, G. A. Lee, M. Billinghurst, and M. Adcock, “Hand gestures and visual annotation in live 360 panorama-based mixed reality remote collaboration,” in Proceedings of the 30th Australian Conference on Computer-Human Interaction - OzCHI ’18, 2018, pp. 406–410.Google ScholarGoogle ScholarDigital LibraryDigital Library

Recommendations

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format .

View HTML Format