Sungchul Jung
Robert W. Lindeman
Skip Abstract Section
Abstract
Physical face-to-face collaboration with someone gives a higher-quality experience compared to mediated communication options, such as a phone- or video-based chat. Participants can share rich sensory cues to multiple human senses in a physical space. Also, the perceptual sensing of the surrounding environment including other peoples' reactions can influence human communication and emotion, and thus collaborative performance. Shared spaces in virtual environments provide degraded sensory experiences because most commercial virtual reality systems typically provide only visual and audio feedback. The impact of richer, multi-sensory feedback on joint decision-making tasks in VR is still an open area of research. Two independent studies exploring this topic are presented in this paper. We implemented a multi-sensory system that delivers vision, audio, tactile, and smell feedback, and we compared the system to a typical VR system. The scenario placed two users in a virtual theme-park safari ride with a number of non-player character (NPC) passengers to simulate realistic scenarios compared to the real-world and we varied the type and complexity of NPCs reactions to participants.
In Experiment 1, we provided both users with either multi-sensory or typical sensory feedback symmetrically as a between-subjects factor, and used NPC reaction type as a within-subjects factor. In Experiment 2, we provided sensory feedback asymmetrically to each user (i.e., one had multi-sensory cues and the other had typical sensory cues) as a between-subjects factor, and used NPC reaction type as a within-subjects factor. We found that the number of sensory channels and NPC reactions did not influence user perception significantly under either symmetric or asymmetric sensory feedback conditions. However, after accounting for individual personality traits (e.g., assertive, passive), as well as any existing relationship between the pairs, we found that increasing the number of sensory channels can significantly improve subjective responses.
- Jeremy N. Bailenson, Jim Blascovich, Andrew C. Beall, and Jack M. Loomis. 2003. Interpersonal Distance in Immersive Virtual Environments. Personality and Social Psychology Bulletin, Vol. 29, 7 (2003), 819--833. https://doi.org/10.1177/0146167203029007002 PMID: 15018671.Google Scholar
Cross Ref
- Aryabrata Basu, Andrew Raij, and Kyle Johnsen. 2012. Ubiquitous Collaborative Activity Virtual Environments. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (Seattle, Washington, USA) (CSCW '12). Association for Computing Machinery, New York, NY, USA, 647--650. https://doi.org/10.1145/2145204.2145302Google ScholarDigital Library
- Mark Billinghurst, Suzanne Weghorst, and Thomas A. Furness. 1998. Shared space: An augmented reality approach for computer supported collaborative work. Virtual Reality, Vol. 3 (1998), 25--36.Google ScholarDigital Library
- Anna C. Bocar. 2017. Aggressive, Passive, and Assertive: Which Communication Style Is Commonly Used by College Students? SSRN (2017).Google Scholar
- Ernest Cline. 2011. Ready player one. New York: Crown Publishers.Google Scholar
- Benjamin H. Detenber, Robert F. Simons, and Jason E. Reiss. 2000. The Emotional Significance of Color in Television Presentations. Media Psychology, Vol. 2 (2000), 331 -- 355.Google ScholarCross Ref
- A. Dey, H. Chen, C. Zhuang, M. Billinghurst, and R. W. Lindeman. 2018. Effects of Sharing Real-Time Multi-Sensory Heart Rate Feedback in Different Immersive Collaborative Virtual Environments. In 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 165--173. https://doi.org/10.1109/ISMAR.2018.00052Google ScholarCross Ref
- Ecorys. 2017. Ecorys report | VR & AR set to create up to 480,000 jobs by 2020. https://ec.europa.eu/futurium/en/connect-university/vr-ar-set-create-480000-jobs-2020.Google Scholar
- Johnathon P Ehsani, Fearghal O'Brien, and Bruce Simmons-Morton. 2017. Comparing G-Force Measurement Between a Smartphone App and an In-Vehicle Accelerometer. In Proceedings of the Ninth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. 221--227.Google ScholarCross Ref
- M. Feng, A. Dey, and R. W. Lindeman. 2016a. The effect of multi-sensory cues on performance and experience during walking in immersive virtual environments. In 2016 IEEE Virtual Reality (VR). 173--174. https://doi.org/10.1109/VR.2016.7504709Google ScholarCross Ref
- M. Feng, A. Dey, and R. W. Lindeman. 2016b. An initial exploration of a multi-sensory design space: Tactile support for walking in immersive virtual environments. In 2016 IEEE Symposium on 3D User Interfaces (3DUI). 95--104. https://doi.org/10.1109/3DUI.2016.7460037Google ScholarCross Ref
- M. Feng, R. W. Lindeman, H. Abdel-Moati, and J. C. Lindeman. 2015. Haptic ChairIO: A system to study the effect of wind and floor vibration feedback on spatial orientation in VEs. In 2015 IEEE Symposium on 3D User Interfaces (3DUI). 149--150. https://doi.org/10.1109/3DUI.2015.7131744Google ScholarCross Ref
- Carlos Flavián, Sergio Ibáñez-Sánchez, and Carlos Orús. 2019. The impact of virtual, augmented and mixed reality technologies on the customer experience. Journal of Business Research, Vol. 100 (2019), 547--560. https://doi.org/10.1016/j.jbusres.2018.10.050Google ScholarCross Ref
- Faul Franz, Erdfelder Edgar, Lang Albert-Georg, and Buchner Axel. 2007. G*Power 3: A Flexible Statistical Power Analysis Program for The Social, Behavioral, and Biomedical Sciences. Behavior Research Methods, Vol. 39, 175--191 (2007).Google ScholarCross Ref
- Rebecca Fribourg, Ferran Argelaguet, Ludovic Hoyet, and Anatole Lécuyer. 2018. Studying the Sense of Embodiment in VR Shared Experiences. 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (2018), 273--280.Google Scholar
- Susan R. Fussell, Robert E. Kraut, and Jane Siegel. 2000. Coordination of Communication: Effects of Shared Visual Context on Collaborative Work (CSCW '00). Association for Computing Machinery, New York, NY, USA, 21--30. https://doi.org/10.1145/358916.358947Google ScholarDigital Library
- Jolene Galegher and Robert E. Kraut. 1992. Computer-Mediated Communication and Collaborative Writing: Media Influence and Adaptation to Communication Constraints. In Proceedings of the 1992 ACM Conference on Computer-Supported Cooperative Work (Toronto, Ontario, Canada) (CSCW '92). Association for Computing Machinery, New York, NY, USA, 155--162. https://doi.org/10.1145/143457.143474Google ScholarDigital Library
- Gheorghita Ghinea and Oluwakemi Ademoye. 2012. The sweet smell of success: Enhancing multimedia applications with olfaction. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 8, 1 (2012), 2.Google ScholarDigital Library
- William Gibson. 1984. Neuromancer. New York :Ace Science Fiction Books.Google Scholar
- J. G. Grandi, H. G. Debarba, and A. Maciel. 2019. Characterizing Asymmetric Collaborative Interactions in Virtual and Augmented Realities. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 127--135.Google Scholar
- Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (2017).Google ScholarDigital Library
- John Harris and Mark Hancock. 2019. To Asymmetry and Beyond! Improving Social Connectedness by Increasing Designed Interdependence in Cooperative Play. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, Article Paper 9, bibinfonumpages12 pages. https://doi.org/10.1145/3290605.3300239Google ScholarDigital Library
- Heilig, Morton L. 1961. Sensorama simulator. https://patents.google.com/patent/US3050870A/enGoogle Scholar
- Jim Hollan and Scott Stornetta. 1992. Beyond Being There. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Monterey, California, USA) (CHI '92). ACM, New York, NY, USA, 119--125. https://doi.org/10.1145/142750.142769Google ScholarDigital Library
- Ying Hua, Vivian Loftness, Robert Kraut, and Kevin M Powell. 2010. Workplace Collaborative Space Layout Typology and Occupant Perception of Collaboration Environment. Environment and Planning B: Planning and Design, Vol. 37, 3 (2010), 429--448. https://doi.org/10.1068/b35011Google ScholarCross Ref
- Matthieu Ischer, Naëm Baron, Christophe Mermoud, Isabelle Cayeux, Christelle Porcherot, David Sander, and Sylvain Delplanque. 2014. How incorporation of scents could enhance immersive virtual experiences. Frontiers in Psychology, Vol. 5 (July 2014). https://doi.org/10.3389/fpsyg.2014.00736Google ScholarCross Ref
- Hiroshi Ishii, Minoru Kobayashi, and Jonathan Grudin. 1993. Integration of Interpersonal Space and Shared Workspace: ClearBoard Design and Experiments. ACM Trans. Inf. Syst., Vol. 11, 4 (Oct. 1993), 349--375. https://doi.org/10.1145/159764.159762Google ScholarDigital Library
- Randolph L. Jackson and Eileen Fagan. 2000. Collaboration and Learning within Immersive Virtual Reality. In Proceedings of the Third International Conference on Collaborative Virtual Environments (San Francisco, California, USA) (CVE '00). Association for Computing Machinery, New York, NY, USA, 83--92. https://doi.org/10.1145/351006.351018Google ScholarDigital Library
- B. Johanson, A. Fox, and T. Winograd. 2002. The Interactive Workspaces project: experiences with ubiquitous computing rooms. IEEE Pervasive Computing, Vol. 1, 2 (2002), 67--74.Google ScholarDigital Library
- Sungchul Jung and Charles E. Hughes. 2016. The Effects of Indirect Real Body Cues of Irrelevant Parts on Virtual Body Ownership and Presence. In ICAT-EGVE 2016 - International Conference on Artificial Reality and Telexistence and Eurographics Symposium on Virtual Environments. 107--114.Google ScholarDigital Library
- Sungchul Jung, Christian Sandor, Pamela J Wisniewski, and Charles E. Hughes. 2017. RealME: The Influence of Body and Hand Representations on Body Ownership and Presence. In Proceedings of the 5th Symposium on Spatial User Interaction, ACM SUI 2017. 3--11.Google ScholarDigital Library
- Sungchul Jung, Pamela J Wisniewski, and Charles E. Hughes. 2018. In Limbo: The Effect of Gradual Visual Transition between Real and Virtual on Virtual Body Ownership Illusion and Presence. In Proceedings of the 25th IEEE Conference on Virtual Reality and 3D User Interfaces, IEEE VR 2018.Google Scholar
- Sungchul Jung, Andrew L. Wood, Simon Hoermann, Pramuditha L. Abhayawardhana, and Robert W. Lindeman. 2020. The Impact of Multi-sensory Stimuli on Confidence Levels for Perceptual-cognitive Tasks in VR. In Proceedings of the 27th IEEE Conference on Virtual Reality and 3D User Interfaces, IEEE VR 2020.Google Scholar
- Konstantina Kilteni, Antonella Maselli, Konrad P. Kording, and Mel Slater. 2015. Over My Fake Body: Body Ownership Illusions for Studying The Multisensory Basis of Own-Body Perception. Frontiers in Human Neuroscience, Vol. 9, 4 (MARCH 2015).Google ScholarCross Ref
- Ernst Kruijff, Alexander Marquardt, Christina Trepkowski, Jonas Schild, and André Hinkenjann. 2017. Designed emotions: challenges and potential methodologies for improving multisensory cues to enhance user engagement in immersive systems. The Visual Computer, Vol. 33, 4 (April 2017), 471--488. https://doi.org/10.1007/s00371-016-1294-0Google ScholarDigital Library
- Ernst Kruijff, Christina Trepkowski, and Robert W. Lindeman. 2015. The Effect of Vibration and Low-frequency Audio on Full-body Haptic Sensations. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology (VRST '15). ACM, New York, NY, USA, 194--194. https://doi.org/10.1145/2821592.2821626 event-place: Beijing, China.Google ScholarDigital Library
- William Little. 2016. Introduction to Sociology 2nd canadian edition ed.). https://opentextbc.ca/introductionto.Google Scholar
- Takamichi Nakamoto and Hai Pham Dinh Minh. 2007. Improvement of olfactory display using solenoid valves. IEEE, 179--186.Google Scholar
- Eleuda Nunez, Soichiro Matsuda, Masakazu Hirokawa, Junichi Yamamoto, and Kenji Suzuki. 2018. Effect of Sensory Feedback on Turn-Taking Using Paired Devices for Children with ASD. Multimodal Technologies and Interaction, Vol. 2, 4 (Sep 2018), 61. https://doi.org/10.3390/mti2040061Google ScholarCross Ref
- World Health Organization. 2020. Coronavirus disease (COVID-19) outbreak. https://www.who.int/emergencies/diseases/novel-coronavirus-2019.Google Scholar
- Ye Pan and Anthony Steed. 2017. The impact of self-avatars on trust and collaboration in shared virtual environments. PLOS ONE, Vol. 12, 12 (12 2017), 1--20. https://doi.org/10.1371/journal.pone.0189078Google ScholarCross Ref
- Kasey Panetta. 2020. Gartner Top 10 Strategic Technology Trends for 2020. https://www.gartner.com/smarterwithgartner/gartner-top-10-strategic-technology-trends-for-2020/.Google Scholar
- Palacín R Powell, J.P. 2015. Passenger Stability Within Moving Railway Vehicles: Limits on Maximum Longitudinal Acceleration. Urban Rail Transit, Vol. 1, 95--103 (2015).Google ScholarCross Ref
- Jun Rekimoto and Masanori Saitoh. 1999. Augmented Surfaces: A Spatially Continuous Work Space for Hybrid Computing Environments. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI '99). Association for Computing Machinery, New York, NY, USA, 378--385. https://doi.org/10.1145/302979.303113Google ScholarDigital Library
- Daniel Roth, Marc Erich Latoschik, Kai Vogeley, and Gary Bente. 2015. Hybrid Avatar-Agent Technology -- A Conceptual Step Towards Mediated "Social" Virtual Reality and its Respective Challenges. i-com, Vol. 14, 2 (2015).Google Scholar
- Daniel Roth, David Mal, Christian Felix Purps, Peter Kullmann, and Marc Erich Latoschik. 2018. Injecting Nonverbal Mimicry with Hybrid Avatar-Agent Technologies: A NaÏVe Approach. In Proceedings of the Symposium on Spatial User Interaction (Berlin, Germany) (SUI '18). ACM, New York, NY, USA, 69--73. https://doi.org/10.1145/3267782.3267791Google ScholarDigital Library
- Junta Sato, Kaori Ohtsu, Yuichi Bannai, and Ken-ichi Okada. 2009. Effective presentation technique of scent using small ejection quantities of odor. IEEE, 151--158.Google Scholar
- Richard Skarbez, Frederick P. Brooks, Jr., and Mary C. Whitton. 2017. A Survey of Presence and Related Concepts. ACM Comput. Surv., Vol. 50, 6 (Nov. 2017), 96:1--96:39. https://doi.org/10.1145/3134301Google ScholarDigital Library
- Mel Slater. 2009. Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, Vol. 364, 1535 (December 2009), 3549--3557. https://doi.org/10.1098/rstb.2009.0138Google ScholarCross Ref
- Diane H. Sonnenwald, Kelly L. Maglaughlin, and Mary C. Whitton. 2004. Designing to Support Situation Awareness Across Distances: An Example from a Scientific Collaboratory. Inf. Process. Manage., Vol. 40, 6 (Nov. 2004), 989--1011. https://doi.org/10.1016/j.ipm.2003.10.002Google ScholarDigital Library
- Anthony Steed, Ye Pan, Fiona Zisch, and William Steptoe. 2016. The Impact of a Self-Avatar on Cognitive Load in Immersive Virtual Reality. In In IEEE Conference on Virtual Reality. 67--76.Google ScholarCross Ref
- Anthony Steed, Mel Slater, Amela Sadagic, Adrian Bullock, and Jolanda G. Tromp. 1999. Leadership and collaboration in shared virtual environments. Proceedings IEEE Virtual Reality (Cat. No. 99CB36316) (1999), 112--115.Google ScholarCross Ref
- A. Steed, W. Steptoe, W. Oyekoya, F. Pece, T. Weyrich, J. Kautz, D. Friedman, A. Peer, M. Solazzi, F. Tecchia, M. Bergamasco, and M. Slater. 2012. Beaming: An Asymmetric Telepresence System. IEEE Computer Graphics and Applications, Vol. 32, 6 (Nov 2012), 10--17. https://doi.org/10.1109/MCG.2012.110Google ScholarDigital Library
- Zsolt Szalavari, Dieter Schmalstieg, Anton Fuhrmann, and Michael Gervautz. 1998. "Studierstube" - An Environment for Collaboration in Augmented Reality 1 ed.).., Seite von - bis: pp. 37--49.Google Scholar
- Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoper. Virtual Environ., Vol. 9, 5 (Oct. 2000), 497--503. https://doi.org/10.1162/105474600566989Google ScholarDigital Library
- Amy Voida, Stephen Voida, Saul Greenberg, and Helen Ai He. 2008. Asymmetry in Media Spaces. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (San Diego, CA, USA) (CSCW '08). Association for Computing Machinery, New York, NY, USA, 313--322. https://doi.org/10.1145/1460563.1460615Google ScholarDigital Library
Index Terms
- On the Use of Multi-sensory Cues in Symmetric and Asymmetric Shared Collaborative Virtual Spaces
Recommendations
Poker in Virtual Reality
This study examines virtual reality (VR) poker and how the platform affects poker players' experience. Players use a self-customized avatar and other features of a computer platform that differs from in-person poker. Data were collected through ...
Virtual space and place: theory and test
Little is known about how individuals come to relate to settings in virtual worlds (VWs), which are defined as digital environments in which individuals, groups, and even organizations interact in virtual (that is to say, nonphysical) spaces. This ...
Using a preamble to increase presence in digital virtual environments
Immersion in a digital virtual environment (DVE) increases the likelihood that individuals will feel present in the DVE and hence respond as they would in a similar physically grounded environment. Previous research utilizing high-fidelity technology ...
Comments