skip to main content
10.1145/3290605.3300737acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Online, VR, AR, Lab, and In-Situ: Comparison of Research Methods to Evaluate Smart Artifacts

Published: 02 May 2019 Publication History

Abstract

Empirical studies are a cornerstone of HCI research. Technical progress constantly enables new study methods. Online surveys, for example, make it possible to collect feedback from remote users. Progress in augmented and virtual reality enables to collect feedback with early designs. In-situ studies enable researchers to gather feedback in natural environments. While these methods have unique advantages and disadvantages, it is unclear if and how using a specific method affects the results. Therefore, we conducted a study with 60 participants comparing five different methods (online, virtual reality, augmented reality, lab setup, and in-situ) to evaluate early prototypes of smart artifacts. We asked participants to assess four different smart artifacts using standardized questionnaires. We show that the method significantly affects the study result and discuss implications for HCI research. Finally, we highlight further directions to overcome the effect of the used methods.

Supplementary Material

ZIP File (paper507.zip)
MP4 File (paper507p.mp4)
Preview video

References

[1]
Mark Altosaar, Roel Vertegaal, Changuk Sohn, and Daniel Cheng. 2006. AuraOrb: Social Notifcation Appliance. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (CHI EA '06). ACM, New York, NY, USA, 381--386.
[2]
Javier A. Bargas-Avila and Kasper Hornbæk. 2011. Old Wine in New Bottles or Novel Challenges: A Critical Analysis of Empirical Studies of User Experience. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2689--2698.
[3]
Ann Blandford, Dominic Furniss, and Stephann Makri. 2016. Qualitative Hci Research: Going Behind the Scenes. Morgan & Claypool Publishers. 1--115 pages.
[4]
Stephen Brewster, David McGookin, and Christopher Miller. 2006. Olfoto: Designing a Smell-based Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, New York, NY, USA, 653--662.
[5]
John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.
[6]
Barry Brown, Stuart Reeves, and Scott Sherwood. 2011. Into the Wild: Challenges and Opportunities for Field Trial Methods. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1657--1666.
[7]
Scott Cliford and Jennifer Jerit. 2014. Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies. Journal of Experimental Political Science 1, 2 (2014), 120--131.
[8]
Ashley Colley, Jani Väyrynen, and Jonna Häkkilä. 2015. Exploring the use of virtual environments in an industrial site design process. In Human-Computer Interaction (Interact'15). Springer, 363--380.
[9]
Mary Czerwinski, Ran Gilad-Bachrach, Shamsi Iqbal, and Gloria Mark. 2016. Challenges for Designing Notifcations for Afective Computing Systems. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 1554--1559.
[10]
Frédéric Dandurand, Thomas R. Shultz, and Kristine H. Onishi. 2008. Comparing online and lab methods in a problem-solving experiment. Behavior Research Methods 40, 2 (01 May 2008), 428--434.
[11]
Alan Dix, Janet E. Finlay, Gregory D. Abowd, and Russell Beale. 2003. Human-Computer Interaction (3rd Edition). Prentice-Hall, Inc., Upper Saddle River, NJ, USA.
[12]
Steven Dow, Jaemin Lee, Christopher Oezbek, Blair MacIntyre, Jay David Bolter, and Maribeth Gandy. 2005. Wizard of Oz Interfaces for Mixed Reality Applications. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, New York, NY, USA, 1339--1342.
[13]
Henry Been-Lirn Duh, Gerald C. B. Tan, and Vivian Hsueh-hua Chen. 2006. Usability Evaluation for Mobile Device: A Comparison of Laboratory and Field Tests. In Proceedings of the 8th Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '06). ACM, New York, NY, USA, 181--186.
[14]
Yiannis Georgiou and Eleni A. Kyza. 2017. The development and validation of the ARI questionnaire: An instrument for measuring immersion in location-based augmented reality settings. International Journal of Human-Computer Studies 98 (2017), 24 -- 37.
[15]
Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDif: A questionnaire to measure perceived hedonic and pragmatic quality. In Mensch & Computer (MuC '03). 187--196.
[16]
Marc Hassenzahl, Michael Burmester, and Franz Koller. 2003. AttrakDif: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. Vieweg+Teubner Verlag, Wiesbaden, 187--196.
[17]
Niels Henze, Enrico Rukzio, and Susanne Boll. 2011. 100,000,000 Taps: Analysis and Improvement of Touch Performance in the Large. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 133--142.
[18]
H. Hofman, J. Groen, S. Rousseau, Ari Hollander, W. Winn, M. Wells, and Thomas Furness. 1996. Tactile Augmentation: Enhancing presence in virtual reality with tactile feedback from real objects. (01 1996).
[19]
Kasper Hornbæk. 2010. Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology 29, 1 (2010), 97--111.
[20]
Kasper Hornbæk. 2013. Some Whys and Hows of Experiments in Human-Computer Interaction. Foundations and Trends in HumanComputer Interaction 5, 4 (2013), 299--373.
[21]
Eva Hornecker and Emma Nicol. 2012. What Do Lab-based User Studies Tell Us About In-the-wild Behavior?: Insights from a Study of Museum Interactives. In Proceedings of the Designing Interactive Systems Conference (DIS '12). ACM, New York, NY, USA, 358--367.
[22]
Anne Kaikkonen, Aki Kekäläinen, Mihael Cankar, Titti Kallio, and Anu Kankainen. 2005. Usability Testing of Mobile Applications: A Comparison Between Laboratory and Field Testing. J. Usability Studies 1, 1 (Nov. 2005), 4--16. http://dl.acm.org/citation.cfm?id=2835525.
[23]
Jesper Kjeldskov and Mikael B. Skov. 2014. Was It Worth the Hassle?: Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices and Services (MobileHCI '14). ACM, New York, NY, USA, 43--52.
[24]
Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als, and Rune T. Høegh. 2004. Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. In Mobile Human-Computer Interaction (MobileHCI '04). Springer, Berlin, Heidelberg, 61--73.
[25]
Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Efects of Avatar Hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 345, 9 pages.
[26]
Andrew L. Kun, Hidde van der Meulen, and Christian P. Janssen. 2017. Calling While Driving: An Initial Experiment with HoloLens. In Proceedings of the Ninth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design. 200--206.
[27]
Andrii Matviienko, Maria Rauschenberger, Vanessa Cobus, Janko Timmermann, Heiko Müller, Jutta Fortmann, Andreas Löcken, Christoph Trappe, Wilko Heuten, and Susanne Boll. 2015. Deriving Design Guidelines for Ambient Light Systems. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM '15). ACM, New York, NY, USA, 267--277.
[28]
Sarah Mennicken, Jonas Hofer, Anind Dey, and Elaine M. Huang. 2014. Casalendar: A Temporal Interface for Automated Homes. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (CHI EA '14). ACM, New York, NY, USA, 2161--2166.
[29]
Aske Mottelson and Kasper Hornbæk. 2017. Virtual Reality Studies Outside the Laboratory. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (VRST '17). ACM, New York, NY, USA, Article 9, 10 pages.
[30]
Christian Monrad Nielsen, Michael Overgaard, Michael Bach Pedersen, Jan Stage, and Sigge Stenild. 2006. It's Worth the Hassle!: The Added Value of Evaluating the Usability of Mobile Systems in the Field. In Proceedings of the 4th Nordic Conference on Human-computer Interaction: Changing Roles (NordiCHI '06). ACM, New York, NY, USA, 272--280.
[31]
Donald A. Norman and Stephen W. Draper. 1986. User Centered System Design; New Perspectives on Human-Computer Interaction. L. Erlbaum Associates Inc., Hillsdale, NJ, USA.
[32]
Thomas Olsson and Markus Salo. 2012. Narratives of Satisfying and Unsatisfying Experiences of Current Mobile Augmented Reality Applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 2779--2788.
[33]
Shane R. Porter, Michael R. Marner, Ross T. Smith, Joanne E. Zucco, and Bruce H. Thomas. 2010. Validating Spatial Augmented Reality for interactive rapid prototyping. In IEEE International Symposium on Mixed and Augmented Reality. 265--266.
[34]
Yvonne Rogers, Kay Connelly, Lenore Tedesco, William Hazlewood, Andrew Kurtz, Robert E. Hall, Josh Hursey, and Tammy Toscos. 2007. Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. In International Conference on Ubiquitous Computing (UbiComp'07). Springer, Berlin, Heidelberg, 336--353.
[35]
Yvonne Rogers, Helen Sharp, and Jenny Preece. 2009. Interaction Design: Beyond Human - Computer Interaction (2nd ed.). Wiley Publishing.
[36]
Antti Salovaara, Antti Oulasvirta, and Giulio Jacucci. 2017. Evaluation of Prototypes and the Problem of Possible Futures. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 2064--2077.
[37]
Valerie M. Sue and Lois A. Ritter. 2012. Conducting online surveys (2nd ed.). Thousand Oaks, California.
[38]
Xu Sun and Andrew May. 2013. A Comparison of Field-based and Lab-based Experiments to Evaluate User Experience of Personalised Mobile Devices. Adv. in Hum.-Comp. Int. 2013, Article 2 (Jan. 2013), 1 pages.
[39]
Leena Ventä-Olkkonen, Jonna Häkkilä, and Kaisa Väänänen-VainioMattila. 2014. Exploring the Augmented Home Window: User Perceptions of the Concept. In Proceedings of the 13th International Conference on Mobile and Ubiquitous Multimedia (MUM '14). ACM, New York, NY, USA, 190--198.
[40]
Alexandra Voit, Tonja Machulla, Dominik Weber, Valentin Schwind, Stefan Schneegass, and Niels Henze. 2016. Exploring Notifcations in Smart Home Environments. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (MobileHCI '16). ACM, New York, NY, USA, 942--947.
[41]
Alexandra Voit, Dominik Weber, Elizabeth Stowell, and Niels Henze. 2017. Caloo: An Ambient Pervasive Smart Calendar to Support Aging in Place. In Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia (MUM '17). ACM, New York, NY, USA, 25--30.
[42]
Ina Wechsung and Anja B. Naumann. 2008. Evaluation methods for multimodal systems: A comparison of standardized usability questionnaires. In International Tutorial and Research Workshop on Perception and Interactive Technologies for Speech-Based Systems. Springer, 276-- 284.
[43]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 143--146.

Cited By

View all
  • (2025)A comparative study of VR and AR heritage applications on visitor emotional experiences: a case study from a peripheral Spanish destinationVirtual Reality10.1007/s10055-025-01109-029:1Online publication date: 6-Feb-2025
  • (2024)Longitudinal Effects of External Communication of Automated Vehicles in the USA and Germany: A Comparative Study in Virtual Reality and Via a BrowserProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997788:4(1-33)Online publication date: 21-Nov-2024
  • (2024)Construction of SVS: Scale of Virtual Twin's Similarity to Physical Counterpart in Simple EnvironmentsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682100(1-9)Online publication date: 7-Oct-2024
  • Show More Cited By

Index Terms

  1. Online, VR, AR, Lab, and In-Situ: Comparison of Research Methods to Evaluate Smart Artifacts

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
        May 2019
        9077 pages
        ISBN:9781450359702
        DOI:10.1145/3290605
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 02 May 2019

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. empirical methods
        2. prototype evaluation
        3. smart artifacts
        4. surveys
        5. user studies

        Qualifiers

        • Research-article

        Funding Sources

        Conference

        CHI '19
        Sponsor:

        Acceptance Rates

        CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)218
        • Downloads (Last 6 weeks)14
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2025)A comparative study of VR and AR heritage applications on visitor emotional experiences: a case study from a peripheral Spanish destinationVirtual Reality10.1007/s10055-025-01109-029:1Online publication date: 6-Feb-2025
        • (2024)Longitudinal Effects of External Communication of Automated Vehicles in the USA and Germany: A Comparative Study in Virtual Reality and Via a BrowserProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997788:4(1-33)Online publication date: 21-Nov-2024
        • (2024)Construction of SVS: Scale of Virtual Twin's Similarity to Physical Counterpart in Simple EnvironmentsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682100(1-9)Online publication date: 7-Oct-2024
        • (2024)I Did Not Notice: A Comparison of Immersive Analytics with Augmented and Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651085(1-7)Online publication date: 11-May-2024
        • (2024)Immersive In-Situ Prototyping: Influence of Real-World Context on Evaluating Future Pedestrian Interfaces in Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651071(1-8)Online publication date: 11-May-2024
        • (2024)Sitting Posture Recognition and Feedback: A Literature ReviewProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642657(1-20)Online publication date: 11-May-2024
        • (2024)Evaluating Transferable Emotion Expressions for Zoomorphic Social Robots using VR Prototyping2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR62088.2024.00125(1087-1096)Online publication date: 21-Oct-2024
        • (2024)“Moving with the story”: the haptics of reader experience and response to digital comicsNew Review of Hypermedia and Multimedia10.1080/13614568.2024.2374291(1-20)Online publication date: 22-Jul-2024
        • (2023)Virtual Urban Field Studies: Evaluating Urban Interaction Design Using Context-Based Interface PrototypesMultimodal Technologies and Interaction10.3390/mti70800827:8(82)Online publication date: 18-Aug-2023
        • (2023)Simulating Wearable Urban Augmented Reality Experiences in VR: Lessons Learnt from Designing Two Future Urban InterfacesMultimodal Technologies and Interaction10.3390/mti70200217:2(21)Online publication date: 16-Feb-2023
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media