skip to main content
10.1145/1978942.1979185acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Into the wild: challenges and opportunities for field trial methods

Published: 07 May 2011 Publication History

Abstract

Field trials of experimental systems in the wild have developed into a standard method within HCI - testing new systems with groups of users in relatively unconstrained settings outside of the laboratory. In this paper we discuss methodological challenges in running user trials. Using a trial of trials we examined the practices of investigators and participants - documenting demand characteristics, where users adjust their behaviour to fit the expectations of those running the trial, the interdependence of how trials are run and the result they produce, and how trial results can be dependent on the insights of a subset of trial participants. We develop three strategies that researchers can use to leverage these challenges to run better trials.

References

[1]
Barkhuus, L., et al. Picking Pockets on the Lawn: The Development of Tactics and Strategies in a Mobile Game. In Proceedings of UbiComp 2005, Tokyo, Japan., Springer Verlag (2005).
[2]
Barkhuus, L. and Rode, J. From Mice to Men: 24 years of Evaluation in CHI. Extended Proceedings of CHI 2007 (alt.chi)(2007).
[3]
Bell, M., et al. Interweaving mobile games with everyday life. In Proceedings of CHI 2006, ACM Press (2006), 417--426.
[4]
Benford, S., et al. Experiments in inhabited TV. In Proceedings of CHI 98, ACM Press (1998).
[5]
Benford, S., et al. The Error of our Ways: The experience of Self-Reported Position in a Location-Based Game. In Ubicomp 2004, Springer (2004).
[6]
Boehner, K., et al. How HCI interprets the probes. In Proceedings of CHI 2007 (2007) ACM.
[7]
Carter, S., et al. Exiting the cleanroom: On ecological validity and ubiquitous computing. Human Computer Interaction, 23, 1 (2008), 47--99.
[8]
Clandinin, D. and Connelly, F. Narrative inquiry. Jossey-Bass San Francisco, 2000.
[9]
Crabtree, A. Taking technomethodology seriously: hybrid change in the ethnomethodology-design relationship. Eur. J. Inf. Syst., 13, 3 (2004), 195--209.
[10]
Davies, N. Proof-of-concept demonstrators and other evils of application-led research. Proceedings of UbiApp Workshop, Ubicomp 2005(2005).
[11]
Dourish, P. and Adler, A. Your place or mine? Learning from long-term use of audio-video communication. Computer Supported Cooperative Work (CSCW), 5, 1 (1996), 33--62.
[12]
Feyerabend, P. Against method. Verso Books, 1993.
[13]
Gaver, B., et al. Design: Cultural probes. interactions, 6, 1 (1999).
[14]
Gaver, W. Cultural commentators: Non-native interpretations as resources for polyphonic assessment. International journal of human-computer studies, 65, 4 (2007), 292--305.
[15]
Gaver, W., et al. Anatomy of a failure: how we knew when our design went wrong, and what we learned from it. Proceedings of CHI 2009(2009).
[16]
Harper, R. H. R., et al. Locating systems at work. Interacting With Computers, 4, 3 (1992), 343--363.
[17]
Hutchinson, H., et al. Technology probes: inspiring design for and with families. CHI '03: Proceedings of the SIGCHI conference on Human factors in computing systems(2003).
[18]
Intille, S., et al. A living laboratory for the design and evaluation of ubiquitous computing technologies. In Proceedings of CHI 2005 (2005),1941--1944 ACM New York, NY, USA.
[19]
Jones, S. Was there a Hawthorne effect? American Journal of Sociology(1992), 451--468.
[20]
Kjeldskov, J., et al. Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In Proceedings of MobileHCI 2004 (2004),61--73.
[21]
Kjeldskov, J., et al. Is it worth the hassle? Exploring the added value of evaluating the usability of context-aware mobile systems in the field. In Proceedings of MobileHCI 2004 (2004),61--73.
[22]
Ljungblad, S. and Holmquist, L. Transfer scenarios: grounding innovation with marginal practices. ACM, City, 2007.
[23]
Mauthner, N., et al. The data are out there, or are they? Implications for archiving and revisiting qualitative data. Sociology, 32, 4 (1998), 733--745.
[24]
McMillan, D., et al. Further into the wild: Running worldwide trials of mobile systems. Pervasive Computing(2010), 210--227.
[25]
Pousman, Z., et al. Living with tableau machine: a longitudinal investigation of a curious domestic intelligence. In Proceedings of Ubicomp 2008 (2008),370--379 ACM.
[26]
Robinson, M. Design for unanticipated use. In Proceedings of ECSCW 1993 (1993),187--202 Kluwer Academic Publishers.
[27]
Rogers, Y., et al. Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. In Proceedings of Ubicomp 2007 (2007).
[28]
Root, R. W. Design of a multi-media vehicle for social browsing. In Proceedings of CSCW 1988 (1988) ACM Press.
[29]
Smith, I. E., et al. Social Disclosure of Place: From Location Technology to Communication Practices. In Pervasive 2005 (2005), 134--151.
[30]
Tolmie, P. and Crabtree, A. Deploying research technology in the home. In Proceedings of the 2008 ACM conference on Computer supported cooperative work (2008),639--648 ACM.
[31]
Whittaker, S., et al. Let's stop pushing the envelope and start addressing it: a reference task agenda for HCI. Hum.-Comput. Interact., 15, 2 (2000), 75--106.

Cited By

View all
  • (2025)Demand characteristics in human–computer experimentsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103379193:COnline publication date: 1-Jan-2025
  • (2024)The Town Crier: A Use-Case Design and Implementation for a Socially Assistive Robot in Retirement HomesRobotics10.3390/robotics1304006113:4(61)Online publication date: 9-Apr-2024
  • (2024)Sound of the Police—Virtual Reality Training for Police Communication for High-Stress OperationsMultimodal Technologies and Interaction10.3390/mti80600468:6(46)Online publication date: 4-Jun-2024
  • Show More Cited By

Index Terms

  1. Into the wild: challenges and opportunities for field trial methods
    Index terms have been assigned to the content through auto-classification.

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2011
    3530 pages
    ISBN:9781450302289
    DOI:10.1145/1978942
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2011

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. demand characteristics
    2. ethnography.
    3. field trials
    4. methods

    Qualifiers

    • Research-article

    Conference

    CHI '11
    Sponsor:

    Acceptance Rates

    CHI '11 Paper Acceptance Rate 410 of 1,532 submissions, 27%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)174
    • Downloads (Last 6 weeks)15
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Demand characteristics in human–computer experimentsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103379193:COnline publication date: 1-Jan-2025
    • (2024)The Town Crier: A Use-Case Design and Implementation for a Socially Assistive Robot in Retirement HomesRobotics10.3390/robotics1304006113:4(61)Online publication date: 9-Apr-2024
    • (2024)Sound of the Police—Virtual Reality Training for Police Communication for High-Stress OperationsMultimodal Technologies and Interaction10.3390/mti80600468:6(46)Online publication date: 4-Jun-2024
    • (2024)Lessons From 3 Longitudinal Sensor-Based Human Behavior Assessment Field Studies and an Approach to Support Stakeholder Management: Content AnalysisJournal of Medical Internet Research10.2196/5046126(e50461)Online publication date: 31-Oct-2024
    • (2024)Stepping on the scale? Only in the morning, of course! - Exploring the Lived Experience of Consumer Self-Tracking Technology in the Context of Citizen ScienceProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685372(1-17)Online publication date: 13-Oct-2024
    • (2024)Tangible Interactions for Pain Assessment in Palliative CareProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685336(1-15)Online publication date: 13-Oct-2024
    • (2024)Facilitating systems thinking in serious game design by highlighting inter-player relationshipsProceedings of the European Conference on Cognitive Ergonomics 202410.1145/3673805.3673810(1-8)Online publication date: 8-Oct-2024
    • (2024)Teaching Affordances: Challenges and Mitigations when integrating the Mechanisms and Conditions framework in HCIProceedings of the 6th Annual Symposium on HCI Education10.1145/3658619.3658631(1-8)Online publication date: 5-Jun-2024
    • (2024)In-the-Wild Exploration of the Impact of the Lunar Cycle on Sleep in a University Cohort with Oura RingsProceedings of the Augmented Humans International Conference 202410.1145/3652920.3653044(259-262)Online publication date: 4-Apr-2024
    • (2024)Prototyping and Evaluation of Emotionally Resonant Vibrotactile Comfort Objects as a Calming Social Anxiety InterventionACM Transactions on Computer-Human Interaction10.1145/364861531:4(1-48)Online publication date: 19-Sep-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media