ABSTRACT
Many user studies are now conducted outside laboratories to increase the number and heterogeneity of participants. These studies are conducted in diverse settings, with the potential to give research greater external validity and statistical power at a lower cost. The feasibility of conducting virtual reality (VR) studies outside laboratories remains unclear because these studies often use expensive equipment, depend critically on the physical context, and sometimes study delicate phenomena concerning body awareness and immersion. To investigate, we explore pointing, 3D tracing, and body-illusions both in-lab and out-of-lab. The in-lab study was carried out as a traditional experiment with state-of-the-art VR equipment; 31 completed the study in our laboratory. The out-of-lab study was conducted by distributing commodity cardboard VR glasses to participants; 57 completed the study anywhere they saw fit. The effects found in-lab were comparable to those found out-of-lab, with much larger variations in the settings in the out-of-lab condition. A follow-up study showed that performance metrics are mostly governed by the technology used, where more complex VR phenomena depend more critically on the internal control of the study. We argue that conducting VR studies outside the laboratory is feasible, and that certain types of VR studies may advantageously be run this way. From the results, we discuss the implications and limitations of running VR studies outside the laboratory.
- Kevin W. Arthur, Kellogg S. Booth, and Colin Ware. 1993. Evaluating 3D Task Performance for Fish Tank Virtual Worlds. ACM Trans. Inf. Syst. 11, 3 (July 1993), 239--265. Google ScholarDigital Library
- Cedric Bach and Dominique L Scapin. 2010. Comparing inspections and user testing for the evaluation of virtual environments. Internat. J. Hum.-Comput. Interact. 26, 8 (2010), 786--824. Google ScholarCross Ref
- Domna Banakou, Raphaela Groten, and Mel Slater. 2013. Illusory Ownership of a Virtual Child Body Causes Overestimation of Object Sizes and Implicit Attitude Changes. Proc. Natl. Acad. Sci. USA 110, 31 (2013), 12846--12851. Google ScholarCross Ref
- John Bolton, Mike Lambert, Denis Lirette, and Ben Unsworth. 2014. PaperDude: A Virtual Reality Cycling Exergame (CHIEA '14). ACM, New York, NY, USA, 475--478.Google Scholar
- Barry Brown, Stuart Reeves, and Scott Sherwood. 2011. Into the Wild: Challenges and Opportunities for Field Trial Methods (CHI '11). ACM, New York, NY, USA, 1657--1666. Google ScholarDigital Library
- Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk a new Source of Inexpensive, yet High-quality, Data? Perspect. Psychol. Sci. 6, 1 (2011), 3--5. Google ScholarCross Ref
- Kelly Caine. 2016. Local Standards for Sample Size at CHI (CHI '16). ACM, New York, NY, USA, 981--992.Google ScholarDigital Library
- Scott Carter, Jennifer Mankoff, and Jeffrey Heer. 2007. Momento: Support for Situated Ubicomp Experimentation (CHI '07). ACM, New York, NY, USA, 125--134. Google ScholarDigital Library
- Matthew J. C. Crump, John V. McDonnell, and Todd M. Gureckis. 2013. Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE 8, 3 (03 2013), 1--18.Google Scholar
- Yvonne A. W. de Kort, Wijnand A. Ijsselsteijn, Jolien Kooijman, and Yvon Schuurmans. 2003. Virtual Laboratories: Comparability of Real and Virtual Environments for Environmental Psychology. Presence: Teleoper. Virtual Environ. 12, 4 (Aug. 2003), 360--373. Google ScholarDigital Library
- Azucena Garcia-Palacios, Hunter G. Hoffman, Albert Carlin, Thomas A. Furness, and Christina Botella. 2002. Virtual Reality in the Treatment of Spider Phobia: a Controlled Study. Behav. Res. Ther. 40, 9 (2002), 983 -- 993. Google ScholarCross Ref
- Laura Germine, Ken Nakayama, Bradley C. Duchaine, Christopher F. Chabris, Garga Chatterjee, and Jeremy B. Wilmer. 2012. Is the Web as Good as the Lab? Comparable Performance from Web and Lab in Cognitive/Perceptual Experiments. Psych. Bul. & Rev. 19, 5 (2012), 847--857. Google ScholarCross Ref
- Joseph K. Goodman, Cynthia E. Cryder, and Amar Cheema. 2013. Data Collection in a Flat World: the Strengths and Weaknesses of Mechanical Turk Samples. J. Behav. Dec. Making 26, 3 (2013), 213--224. Google ScholarCross Ref
- Sandy J. J. Gould, Anna L. Cox, and Duncan P. Brumby. 2016. Diminished Control in Crowdsourcing: An Investigation of Crowdworker Multitasking Behavior. ACM Trans. Comput.-Hum. Interact. 23, 3, Article 19 (June 2016), 29 pages.Google ScholarDigital Library
- Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design (CHI '10). ACM, New York, NY, USA, 203--212. Google ScholarDigital Library
- Niels Henze, Martin Pielot, Benjamin Poppinga, Torben Schinke, and Susanne Boll. 2011. My App is an Experiment: Experience from User Studies in Mobile App Stores. Int. J. Mob. Hum. Comput. Interact. 3, 4 (Oct. 2011), 71--91. Google ScholarDigital Library
- Kasper Hornbæk. 2013. Some Whys and Hows of Experiments in Human-Computer Interaction. Found. Trends Hum.-Comput. Interact. 5, 4 (June 2013), 299--373.Google Scholar
- Kasper Hornbæk, Søren S. Sander, Javier Andrés Bargas-Avila, and Jakob Grue Simonsen. 2014. Is Once Enough?: On the Extent and Content of Replications in Human-computer Interaction (CHI '14). ACM, New York, NY, USA, 3523--3532. Google ScholarDigital Library
- Konstantina Kilteni, Antonella Maselli, Konrad P. Kording, and Mel Slater. 2015. Over my Fake Body: Body Ownership Illusions for Studying the Multisensory Basis of Own-body Perception. Front. Hum. Neurosci. 9 (2015), 141. Google ScholarCross Ref
- Konstantina Kilteni, Jean-Marie Normand, Maria V. Sanchez-Vives, and Mel Slater. 2012. Extending Body Space in Immersive Virtual Reality: A Very Long Arm Illusion. PLoS ONE 7, 7 (07 2012), 1--15.Google Scholar
- Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk (CHI '08). ACM, New York, NY, USA, 453--456.Google Scholar
- Jesper Kjeldskov and Mikael B. Skov. 2014. Was It Worth the Hassle?: Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations (MobileHCI '14). ACM, New York, NY, USA, 43--52. Google ScholarDigital Library
- Sally A. Linkenauger, Markus Leyrer, Heinrich H. Bülthoff, and Betty J. Mohler. 2013. Welcome to Wonderland: The Influence of the Size and Shape of a Virtual Hand On the Perceived Size and Shape of Virtual Objects. PLoS ONE 8, 7 (07 2013).Google Scholar
- Lara Maister, Natalie Sebanz, Günther Knoblich, and Manos Tsakiris. 2013. Experiencing Ownership Over a Dark-skinned Body Reduces Implicit Racial Bias. Cognition 128, 2 (2013), 170 -- 178. Google ScholarCross Ref
- Tim Marsh. 1999. Evaluation of Virtual Reality Systems for Usability (CHI EA '99). ACM, New York, NY, USA, 61--62.Google Scholar
- Winter Mason and Siddharth Suri. 2012. Conducting Behavioral Research on Amazon's Mechanical Turk. Behav. Res. Methods 44, 1 (2012), 1--23. Google ScholarCross Ref
- Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays (CHI '15). ACM, New York, NY, USA, 2143--2152.Google ScholarDigital Library
- Joseph E. McGrath. 1995. Human-computer Interaction. Morgan Kaufmann Publishers, San Francisco, CA, USA, Chapter Methodology Matters: Doing Research in the Behavioral and Social Sciences, 152--169.Google Scholar
- Aske Mottelson and Kasper Hornbæk. 2016. An Affect Detection Technique Using Mobile Commodity Sensors in the Wild (UbiComp '16). ACM, New York, NY, USA 781--792.Google Scholar
- Xueni Pan, Mel Slater, Alejandro Beacco, Xavi Navarro, Anna I. Bellido Rivas, David Swapp, Joanna Hale, Paul Alexander George Forbes, Catrina Denvir, Antonia F. de C. Hamilton, and Sylvie Delacroix. 2016. The Responses of Medical General Practitioners to Unreasonable Patient Demand for Antibiotics - A Study of Medical Ethics Using Immersive Virtual Reality. PLoS ONE 11, 2 (02 2016).Google Scholar
- Gabriele Paolacci and Jesse Chandler. 2014. Inside the Turk: Understanding Mechanical Turk as a Participant Pool. Curr. Dir. Psychol. 23, 3 (2014), 184--188. Google ScholarCross Ref
- Katharina Reinecke and Krzysztof Z. Gajos. 2015. LabintheWild: Conducting Large-Scale Online Experiments With Uncompensated Samples (CSCW '15). ACM, New York, NY, USA, 1364--1378. Google ScholarDigital Library
- Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who Are the Crowdworkers?: Shifting Demographics in Mechanical Turk (CHI EA '10). ACM, New York, NY, USA, 2863--2872. Google ScholarDigital Library
- Steven V. Rouse. 2015. A Reliability Analysis of Mechanical Turk Data. Comput. Human Behav. 43 (2015), 304 -- 307. Google ScholarDigital Library
- Maria V. Sanchez-Vives, Bernhard Spanlang, Antonio Frisoli, Massimo Bergamasco and Mel Slater. 2010. Virtual Hand Illusion Induced by Visuomotor Correlations. PLoS ONE 5, 4 (04 2010), 1--6.Google Scholar
- William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi Experimental Designs for Generalized Causal Inference. Houghton Mifflin Company, Boston, MA, USA, Chapter Statistical Conclusion Validity and Internal Validity, 33--102.Google Scholar
- Mel Slater, Angus Antley, Adam Davison, David Swapp, Christoph Guger, Chris Barker, Nancy Pistrang, and Maria V. Sanchez-Vives. 2006. A Virtual Reprise of the Stanley Milgram Obedience Experiments. PLoS ONE 1, 1 (12 2006), 1--10.Google Scholar
- Mel Slater, Daniel Pérez Marcos, Henrik Ehrsson, and Maria V. Sanchez-Vives. 2008. Towards a Digital Body: The Virtual Arm Illusion. Front. Hum. Neurosci. 2, 6 (2008). Google ScholarCross Ref
- Mel Slater, Daniel Perez-Marcos, H. Henrik Ehrsson, and Maria V. Sanchez-Vives. 2009. Inducing Illusory Ownership of a Virtual Body. Front. Hum. Neurosci. 3, 2 (15 Sep 2009), 214--220. Google ScholarCross Ref
- Mel Slater, Aitor Rovira, Richard Southern, David Swapp, Jian J. Zhang, Claire Campbell, and Mark Levine. 2013. Bystander Responses to a Violent Incident in an Immersive Virtual Environment. PLoS ONE 8, 1 (01 2013), 1--13.Google Scholar
- Mel Slater, Bernhard Spanlang, Maria V. Sanchez-Vives, and Olaf Blanke. 2010. First Person Experience of Body Transfer in Virtual Reality. PLoS ONE 5, 5 (05 2010), 1--9.Google Scholar
- Mel Slater, Martin Usoh, and Anthony Steed. 1994. Depth of Presence in Virtual Environments. Presence-Teleop. Virt. 3, 2 (Jan. 1994), 130--144.Google ScholarDigital Library
- Mel Slater, Martin Usoh, and Anthony Steed. 1995. Taking Steps: The Influence of a Walking Technique on Presence in Virtual Reality. ACM Trans. Comput.-Hum. Interact. 2, 3 (Sept. 1995), 201--219. Google ScholarDigital Library
- Mel Slater and Sylvia Wilbur. 1997. A Framework for Immersive Virtual Environments Five: Speculations on the Role of Presence in Virtual Environments. Presence-Teleop. Virt. 6, 6 (Dec. 1997), 603--616. Google ScholarDigital Library
- R. William Soukoreff and I. Scott MacKenzie. 2004. Towards a Standard for Pointing Device Evaluation, Perspectives on 27 Years of Fitts' Law Research in HCI. Int. J. Hum. Comput. Stud. 61, 6 (Dec. 2004), 751--789. Google ScholarDigital Library
- STEAM. 2017. Steam Hardware & Software Survey: September 2017. http://store.steampowered.com/hwsurvey. (2017). [Accessed 01-September-2017].Google Scholar
- Anthony Steed, Sebastian Frlston, Maria M. López, Jason Drummond, Ye Pan, and David Swapp. 2016. An 'In the Wild' Experiment on Presence and Embodiment using Consumer Virtual Reality Equipment. IEEE Trans. Vis. Comput. Graph. 22, 4 (April 2016), 1406--1414. Google ScholarDigital Library
- Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell Gabriele Paolacci, and Jesse Chandler. 2015. The Average Laboratory Samples a Population of 7,300 Amazon Mechanical Turk Workers. Jud. Dec. Mak. 10, 5 (2015), 479--491.Google Scholar
- Alistair Sutcliffe and Brian Gault. 2004. Heuristic Evaluation of Virtual Reality Applications. Interact. Comput. 16, 4 (2004), 831--849. Google ScholarCross Ref
- Martin Usoh, Kevin Arthur, Mary C. Whitton, Rui Bastos, Anthony Steed, Mel Slater and Frederick P. Brooks, Jr. 1999. Walking > Walking-in-place > Flying, in Virtual Environments (SIGGRAPH '99). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 359--364.Google Scholar
- Ulrich von Zadow, Sandra Buron, Tina Harms, Florian Behringer, Kai Sostmann, and Raimund Dachselt. 2013. SimMed: Combining Simulation and Interactive Tabletops for Medical Education (CHI '13). ACM, New York, NY, USA, 1469--1478. Google ScholarDigital Library
Index Terms
- Virtual reality studies outside the laboratory
Recommendations
Crowdsourcing Virtual Reality Experiments using VRChat
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing SystemsResearch involving Virtual Reality (VR) headsets is becoming more and more popular. However, scaling VR experiments is challenging as researchers are often limited to using one or a small number of headsets for in-lab studies. One general way to scale ...
Remote VR Studies: A Framework for Running Virtual Reality Studies Remotely Via Participant-Owned HMDs
We investigate opportunities and challenges of running virtual reality (VR) studies remotely. Today, many consumers own head-mounted displays (HMDs), allowing them to participate in scientific studies from their homes using their own equipment. ...
Immersive virtual reality with multimodal interaction and streaming technology
ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal InteractionIn this demo, we present an immersive virtual reality (VR) system which integrates multimodal interaction sensors (i.e., smartphone, Kinect v2, and Myo armband) and streaming technology to improve the VR experience. The integrated system solves the ...
Comments