skip to main content
10.1145/3411763.3451764acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
poster

Surrogate-Aloud: A Human Surrogate Method for Remote Usability Evaluation and Ideation in Virtual Reality

Published: 08 May 2021 Publication History

Abstract

Virtual and augmented reality offer comparative performance in terms of remote usability testing to lab-based co-located settings. However, direct contact with a researcher is still required to provide setup, troubleshoot, and training. In this paper, we present Surrogate-Aloud as a remote ideation and usability method that establishes a surrogate relationship between participants and a facilitating researcher through video conferencing. The researcher wears a VR headset and shares their viewpoint through video conferencing to a remote participant, who applies think-aloud protocol to express movement and interaction commands to be executed by the researcher, alongside their thought process as they interact with virtual prototypes or scenarios. We conducted a preliminary study to evaluate the Surrogate-Aloud method for remote usability evaluation and ideation of a new instructional technique with volumetric recordings. Results show that Surrogate-Aloud leverages the surrogate’s technical expertise and enables sufficient capability to conduct truly remote usability evaluation and ideation.

References

[1]
Julian Abich IV, Lauren E. Reinerman-Jones, Gerald Matthews, Gregory F. Welch, Stephanie J. Lackey, Charles E. Hughes, and Arjun Nagendran. 2014. Good enough yet? A preliminary evaluation of human-surrogate interaction. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 8525 LNCS, PART 1(2014), 239–250. https://doi.org/10.1007/978-3-319-07458-0_23
[2]
Morten Sieker Andreasen, Henrik Villemann Nielsen, Simon Ormholt Schrøder, and Jan Stage. 2007. What happened to remote usability testing?: An empirical study of three methods. In Conference on Human Factors in Computing Systems - Proceedings. Association for Computing Machinery, New York, NY, USA, 1405–1414. https://doi.org/10.1145/1240624.1240838
[3]
T Boren and J Ramey. 2000. Thinking aloud: reconciling theory and practice. IEEE Transactions on Professional Communication 43, 3(2000), 261–278. https://doi.org/10.1109/47.867942
[4]
Kapil Chalil Madathil and Joel S. Greenstein. 2017. An investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testing. Applied Ergonomics 65(2017), 501–514. https://doi.org/10.1016/j.apergo.2017.02.011
[5]
Lynne Cooke. 2010. Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. IEEE Transactions on Professional Communication 53, 3 (sep 2010), 202–215. https://doi.org/10.1109/TPC.2010.2052859
[6]
Jing Du, Yangming Shi, Zhengbo Zou, and Dong Zhao. 2018. CoVR: Cloud-Based Multiuser Virtual Reality Headset System for Project Communication of Remote Users. Journal of Construction Engineering and Management 144, 2(2018), 1–19. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001426
[7]
Sanne Elling, Leo Lentz, and Menno De Jong. 2012. Combining concurrent think-aloud protocols and eye-tracking observations: An analysis of verbalizations and silences. IEEE Transactions on Professional Communication 55, 3(2012), 206–220. https://doi.org/10.1109/TPC.2012.2206190
[8]
K Anders Ericsson and Herbert A Simon. 1984. Protocol analysis: Verbal reports as data.the MIT Press, Cambridge, MA, USA.
[9]
F Fahmi, K Tanjung, F Nainggolan, B Siregar, N Mubarakah, and M Zarlis. 2020. Comparison study of user experience between virtual reality controllers, leap motion controllers, and senso glove for anatomy learning systems in a virtual reality environment. IOP Conference Series: Materials Science and Engineering 851 (May 2020), 012–024. https://doi.org/10.1088/1757-899x/851/1/012024
[10]
Mingming Fan, Jinglan Lin, Christina Chung, and Khai N. Truong. 2019. Concurrent think-aloud verbalizations and usability problems. ACM Transactions on Computer-Human Interaction 26, 5(2019), 35 pages. https://doi.org/10.1145/3325281
[11]
Charith Lasantha Fernando, Masahiro Furukawa, Tadatoshi Kurogi, Kyo Hirota, Sho Kamuro, Katsunari Sato, Kouta Minamizawa, and Susumu Tachi. 2012. TELESAR V: TELExistence Surrogate Anthropomorphic Robot. In ACM SIGGRAPH 2012 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’12). Association for Computing Machinery, New York, NY, USA, Article 23, 1 pages. https://doi.org/10.1145/2343456.2343479
[12]
Brennan Jones, Yaying Zhang, Priscilla N. Y. Wong, and Sean Rintel. 2020. VROOM: Virtual Robot Overlay for Online Meetings. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/3334480.3382820
[13]
Konstantina Kilteni, Raphaela Groten, and Mel Slater. 2012. The sense of embodiment in virtual reality. Presence: Teleoperators and Virtual Environments 21, 4(2012), 373–387.
[14]
Kangsoo Kim and Greg Welch. 2015. Maintaining and enhancing human-surrogate presence in augmented reality. In Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality Workshops, ISMARW 2015 (Fukuoka, Japan). IEEE, New York, NY, USA, 15–19. https://doi.org/10.1109/ISMARW.2015.13
[15]
E Krahmer and N Ummelen. 2004. Thinking about thinking aloud: a comparison of two verbal protocols for usability testing. IEEE Transactions on Professional Communication 47, 2(2004), 105–117. https://doi.org/10.1109/TPC.2004.828205
[16]
P Lincoln, G Welch, A Nashel, A Ilie, A State, and H Fuchs. 2009. Animatronic Shader Lamps Avatars. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, NY, USA, 27–33. https://doi.org/10.1109/ISMAR.2009.5336503
[17]
Bethany Ann MacKey, Paul A. Bremner, and Manuel Giuliani. 2020. The effect of virtual reality control of a robotic surrogate on presence and social presence in comparison to telecommunications software. ACM/IEEE International Conference on Human-Robot Interaction (2020), 349–351. https://doi.org/10.1145/3371382.3378268
[18]
Kapil Chalil Madathil and Joel S. Greenstein. 2011. Synchronous remote usability testing - A new approach facilitated by virtual worlds. Conference on Human Factors in Computing Systems - Proceedings (2011), 2225–2234. https://doi.org/10.1145/1978942.1979267
[19]
Jesus Mayor, Laura Raya, and Alberto Sanchez. 2019. A comparative study of virtual reality methods of interaction and locomotion based on presence, cybersickness and usability. IEEE Transactions on Emerging Topics in Computing (2019), 1–12. https://doi.org/10.1109/TETC.2019.2915287
[20]
Sharon Mcdonald, Tingting Zhao, and Helen M. Edwards. 2016. Look who’s talking: Evaluating the utility of interventions during an interactive think-aloud. Interacting with Computers 28, 3 (2016), 387–403. https://doi.org/10.1093/iwc/iwv014
[21]
Arjun Nagendran, Gregory Welch, Charles Hughes, and Remo Pillat. 2015. Technical Report: Exploring Human Surrogate Characteristics. In Virtual Realities. Lecture Notes in Computer Science, vol 8844, Guido Brunnett, Sabine Coquillart, Robert van Liere, Gregory Welch, and Libor Váša (Eds.). Springer International Publishing, Cham, 215–228. https://doi.org/10.1007/978-3-319-17043-5_12
[22]
Janni Nielsen, Torkil Clemmensen, and Carsten Yssing. 2002. Getting access to what goes on in people’s heads?. In Proceedings of the second Nordic conference on Human-computer interaction - NordiCHI ’02. ACM Press, New York, New York, USA, 101. https://doi.org/10.1145/572021.572033
[23]
Daniel Roth, Jean-Luc Lugrin, Marc Erich Latoschik, and Stephan Huber. 2017. Alpha IVBO-construction of a scale to measure the illusion of virtual body ownership. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM Press, NY, USA, 2875–2883.
[24]
Microsoft Teams. 2020. Video Conferencing, Web Conferencing, Webinars, Screen Sharing. https://www.microsoft.com/en-us/microsoft-teams/group-chat-software.
[25]
Katherine E. Thompson, Evelyn P. Rozanski, and Anne R. Haake. 2004. Here, there, anywhere: Remote usability testing that works. SIGITE 2004 Conference(2004), 132–137.
[26]
T Tullis, S Fleischman, and M McNulty. 2002. An empirical comparison of lab and remote usability testing of web sites. Usability Professionals ...(2002), 6 pages. http://www.testapic.com/dl/RemoteVsLab.pdf
[27]
Michael E. Walker, Hooman Hedayati, and Daniel Szafir. 2019. Robot Teleoperation with Augmented Reality Virtual Surrogates. ACM/IEEE International Conference on Human-Robot Interaction 2019-March(2019), 202–210. https://doi.org/10.1109/HRI.2019.8673306
[28]
David Whitney, Eric Rosen, Elizabeth Phillips, George Konidaris, and Stefanie Tellex. 2020. Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality BT. In Robotics Research, Nancy M Amato, Greg Hager, Shawna Thomas, and Miguel Torres-Torriti (Eds.). Springer International Publishing, Cham, 335–350.
[29]
Bob G Witmer and Michael J Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoper. Virtual Environ. 7, 3 (1998), 225–240. https://doi.org/10.1162/105474698565686
[30]
Zoom. 2020. Video Conferencing, Web Conferencing, Webinars, Screen Sharing. https://zoom.us/.

Cited By

View all
  • (2023)The Think-Aloud Method for Evaluating the Usability of a Regional AtlasISPRS International Journal of Geo-Information10.3390/ijgi1203009512:3(95)Online publication date: 26-Feb-2023
  • (2022)Watch The Videos Whenever You Have TimeProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3531181(1-5)Online publication date: 6-Jun-2022
  • (2022)Usability engineering in practice: developing an intervention for post-stroke therapy during a global pandemicJournal of Medical Engineering & Technology10.1080/03091902.2022.208925746:6(433-447)Online publication date: 24-Aug-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems
May 2021
2965 pages
ISBN:9781450380959
DOI:10.1145/3411763
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Remote Usability Testing
  2. Think-Aloud
  3. Virtual Reality Surrogate

Qualifiers

  • Poster
  • Research
  • Refereed limited

Funding Sources

Conference

CHI '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)2
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)The Think-Aloud Method for Evaluating the Usability of a Regional AtlasISPRS International Journal of Geo-Information10.3390/ijgi1203009512:3(95)Online publication date: 26-Feb-2023
  • (2022)Watch The Videos Whenever You Have TimeProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3531181(1-5)Online publication date: 6-Jun-2022
  • (2022)Usability engineering in practice: developing an intervention for post-stroke therapy during a global pandemicJournal of Medical Engineering & Technology10.1080/03091902.2022.208925746:6(433-447)Online publication date: 24-Aug-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media