Andrew Irlitti
Thuong Hoang
Frank Vetere
ABSTRACT
Virtual and augmented reality offer comparative performance in terms of remote usability testing to lab-based co-located settings. However, direct contact with a researcher is still required to provide setup, troubleshoot, and training. In this paper, we present Surrogate-Aloud as a remote ideation and usability method that establishes a surrogate relationship between participants and a facilitating researcher through video conferencing. The researcher wears a VR headset and shares their viewpoint through video conferencing to a remote participant, who applies think-aloud protocol to express movement and interaction commands to be executed by the researcher, alongside their thought process as they interact with virtual prototypes or scenarios. We conducted a preliminary study to evaluate the Surrogate-Aloud method for remote usability evaluation and ideation of a new instructional technique with volumetric recordings. Results show that Surrogate-Aloud leverages the surrogate’s technical expertise and enables sufficient capability to conduct truly remote usability evaluation and ideation.
- Julian Abich IV, Lauren E. Reinerman-Jones, Gerald Matthews, Gregory F. Welch, Stephanie J. Lackey, Charles E. Hughes, and Arjun Nagendran. 2014. Good enough yet? A preliminary evaluation of human-surrogate interaction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8525 LNCS, PART 1(2014), 239–250. https://doi.org/10.1007/978-3-319-07458-0_23Google Scholar
Digital Library
- Morten Sieker Andreasen, Henrik Villemann Nielsen, Simon Ormholt Schrøder, and Jan Stage. 2007. What happened to remote usability testing?: An empirical study of three methods. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, New York, NY, USA, 1405–1414. https://doi.org/10.1145/1240624.1240838Google ScholarDigital Library
- T Boren and J Ramey. 2000. Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3(2000), 261–278. https://doi.org/10.1109/47.867942Google ScholarCross Ref
- Kapil Chalil Madathil and Joel S. Greenstein. 2017. An investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testing. Applied Ergonomics 65(2017), 501–514. https://doi.org/10.1016/j.apergo.2017.02.011Google ScholarCross Ref
- Lynne Cooke. 2010. Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. IEEE Transactions on Professional Communication 53, 3 (sep 2010), 202–215. https://doi.org/10.1109/TPC.2010.2052859Google ScholarCross Ref
- Jing Du, Yangming Shi, Zhengbo Zou, and Dong Zhao. 2018. CoVR: Cloud-Based Multiuser Virtual Reality Headset System for Project Communication of Remote Users. Journal of Construction Engineering and Management 144, 2(2018), 1–19. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001426Google ScholarCross Ref
- Sanne Elling, Leo Lentz, and Menno De Jong. 2012. Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences. IEEE Transactions on Professional Communication 55, 3(2012), 206–220. https://doi.org/10.1109/TPC.2012.2206190Google ScholarCross Ref
- K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data.the MIT Press, Cambridge, MA, USA.Google Scholar
- F Fahmi, K Tanjung, F Nainggolan, B Siregar, N Mubarakah, and M Zarlis. 2020. Comparison study of user experience between virtual reality controllers, leap motion controllers, and senso glove for anatomy learning systems in a virtual reality environment. IOP Conference Series: Materials Science and Engineering 851 (May 2020), 012–024. https://doi.org/10.1088/1757-899x/851/1/012024Google ScholarCross Ref
- Mingming Fan, Jinglan Lin, Christina Chung, and Khai N. Truong. 2019. Concurrent think-aloud verbalizations and usability problems. ACM Transactions on Computer-Human Interaction 26, 5(2019), 35 pages. https://doi.org/10.1145/3325281Google ScholarDigital Library
- Charith Lasantha Fernando, Masahiro Furukawa, Tadatoshi Kurogi, Kyo Hirota, Sho Kamuro, Katsunari Sato, Kouta Minamizawa, and Susumu Tachi. 2012. TELESAR V: TELExistence Surrogate Anthropomorphic Robot. In ACM SIGGRAPH 2012 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’12). Association for Computing Machinery, New York, NY, USA, Article 23, 1 pages. https://doi.org/10.1145/2343456.2343479Google ScholarDigital Library
- Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, and Sean Rintel. 2020. VROOM: Virtual Robot Overlay for Online Meetings. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/3334480.3382820Google ScholarDigital Library
- Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments 21, 4(2012), 373–387.Google ScholarDigital Library
- Kangsoo Kim and Greg Welch. 2015. Maintaining and enhancing human-surrogate presence in augmented reality. In Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, ISMARW 2015 (Fukuoka, Japan). IEEE, New York, NY, USA, 15–19. https://doi.org/10.1109/ISMARW.2015.13Google ScholarDigital Library
- E Krahmer and N Ummelen. 2004. Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication 47, 2(2004), 105–117. https://doi.org/10.1109/TPC.2004.828205Google ScholarCross Ref
- P Lincoln, G Welch, A Nashel, A Ilie, A State, and H Fuchs. 2009. Animatronic Shader Lamps Avatars. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, NY, USA, 27–33. https://doi.org/10.1109/ISMAR.2009.5336503Google ScholarDigital Library
- Bethany Ann MacKey, Paul A. Bremner, and Manuel Giuliani. 2020. The effect of virtual reality control of a robotic surrogate on presence and social presence in comparison to telecommunications software. ACM/IEEE International Conference on Human-Robot Interaction (2020), 349–351. https://doi.org/10.1145/3371382.3378268Google ScholarDigital Library
- Kapil Chalil Madathil and Joel S. Greenstein. 2011. Synchronous remote usability testing - A new approach facilitated by virtual worlds. Conference on Human Factors in Computing Systems - Proceedings (2011), 2225–2234. https://doi.org/10.1145/1978942.1979267Google ScholarDigital Library
- Jesus Mayor, Laura Raya, and Alberto Sanchez. 2019. A comparative study of virtual reality methods of interaction and locomotion based on presence, cybersickness and usability. IEEE Transactions on Emerging Topics in Computing (2019), 1–12. https://doi.org/10.1109/TETC.2019.2915287Google ScholarCross Ref
- Sharon Mcdonald, Tingting Zhao, and Helen M. Edwards. 2016. Look who’s talking: Evaluating the utility of interventions during an interactive think-aloud. Interacting with Computers 28, 3 (2016), 387–403. https://doi.org/10.1093/iwc/iwv014Google ScholarCross Ref
- Arjun Nagendran, Gregory Welch, Charles Hughes, and Remo Pillat. 2015. Technical Report: Exploring Human Surrogate Characteristics. In Virtual Realities. Lecture Notes in Computer Science, vol 8844, Guido Brunnett, Sabine Coquillart, Robert van Liere, Gregory Welch, and Libor Váša (Eds.). Springer International Publishing, Cham, 215–228. https://doi.org/10.1007/978-3-319-17043-5_12Google ScholarCross Ref
- Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes on in people’s heads?. In Proceedings of the second Nordic conference on Human-computer interaction - NordiCHI ’02. ACM Press, New York, New York, USA, 101. https://doi.org/10.1145/572021.572033Google ScholarDigital Library
- Daniel Roth, Jean-Luc Lugrin, Marc Erich Latoschik, and Stephan Huber. 2017. Alpha IVBO-construction of a scale to measure the illusion of virtual body ownership. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM Press, NY, USA, 2875–2883.Google ScholarDigital Library
- Microsoft Teams. 2020. Video Conferencing, Web Conferencing, Webinars, Screen Sharing. https://www.microsoft.com/en-us/microsoft-teams/group-chat-software.Google Scholar
- Katherine E. Thompson, Evelyn P. Rozanski, and Anne R. Haake. 2004. Here, there, anywhere: Remote usability testing that works. SIGITE 2004 Conference(2004), 132–137.Google ScholarDigital Library
- T Tullis, S Fleischman, and M McNulty. 2002. An empirical comparison of lab and remote usability testing of web sites. Usability Professionals ...(2002), 6 pages. http://www.testapic.com/dl/RemoteVsLab.pdfGoogle Scholar
- Michael E. Walker, Hooman Hedayati, and Daniel Szafir. 2019. Robot Teleoperation with Augmented Reality Virtual Surrogates. ACM/IEEE International Conference on Human-Robot Interaction 2019-March(2019), 202–210. https://doi.org/10.1109/HRI.2019.8673306Google ScholarCross Ref
- David Whitney, Eric Rosen, Elizabeth Phillips, George Konidaris, and Stefanie Tellex. 2020. Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality BT. In Robotics Research, Nancy M Amato, Greg Hager, Shawna Thomas, and Miguel Torres-Torriti (Eds.). Springer International Publishing, Cham, 335–350.Google Scholar
- Bob G Witmer and Michael J Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (1998), 225–240. https://doi.org/10.1162/105474698565686Google ScholarDigital Library
- Zoom. 2020. Video Conferencing, Web Conferencing, Webinars, Screen Sharing. https://zoom.us/.Google Scholar
Index Terms
- Surrogate-Aloud: A Human Surrogate Method for Remote Usability Evaluation and Ideation in Virtual Reality
Index terms have been assigned to the content through auto-classification.
Recommendations
Synchronous remote usability testing: a new approach facilitated by virtual worlds
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThis study proposes a new methodology for conducting synchronous remote usability studies using a three-dimensional virtual usability testing laboratory built using the Open Wonderland toolkit. This virtual laboratory method is then compared with two ...
A Comparison of Laboratory and Synchronous Remote Usability Testing Methods Using AR
Human-Computer Interaction. Theoretical Approaches and Design MethodsAbstractStructured AR usability testing in the product development phase enables obtaining early user feedback, allowing companies to focus on UX from the early design phase. Therefore, several studies of usability testing using AR have been conducted, ...
Maintaining and Enhancing Human-Surrogate Presence in Augmented Reality
ISMARW '15: Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality WorkshopsWe present some background and ideas related to a human's sense of presence with a human surrogate (a stand-in for a human) in an augmented reality (AR) setting. In particular we examine several factors related to human surrogates that are common to ...
Comments