skip to main content
10.1145/2207676.2208588acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

A hybrid mass participation approach to mobile software trials

Published: 05 May 2012 Publication History

Abstract

User trials of mobile applications have followed a steady march out of the lab, and progressively further ''into the wild', recently involving ''app store'-style releases of software to the general public. Yet from our experiences on these mass participation systems and a survey of the literature, we identify a number of reported difficulties. We propose a hybrid methodology that aims to address these, by combining a global software release with a concurrent local trial. A phone-based game, created to explore the uptake and use of ad hoc peer-to-peer networking, was evaluated using this new hybrid trial method, combining a small-scale local trial (11 users) with a ''mass participation' trial (over 10,000 users). Our hybrid method offers many benefits, allowing locally observed findings to be verified, patterns in globally collected data to be explained and addresses ethical issues raised by the mass participation approach. We note trends in the local trial that did not appear in the larger scale deployment, and which would therefore have led to misleading results were the application trialed using ''traditional' methods alone. Based on this study and previous experience, we provide a set of guidelines to researchers working in this area.

References

[1]
Abowd, G. D., Mynatt, E. D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput.-Hum. Interact. 7, 29--58 (2000).
[2]
Ames, M., Naaman, M.: Why We Tag: Motivations for Annotation in Mobile and Online Media. CHI (2007).
[3]
Bell, M and Chalmers, M., and Barkhuus, L., Hall, M., Sherwood, S., Tennent, P., Brown, B., Rowland, D., Benford, S., Capra, M., & Hampshire, A., Interweaving Mobile Games with Everyday Life, Proc. ACM CHI, pp. 417--426. (2006).
[4]
Campbell, A., Eisenman, S., Fodor, K., Lane, N., Lu, H., Miluzzo, E., Musolesi, M., Peterson, R., Zheng, X. Transforming the social networking experience with sensing presence from mobile phones. Proc ACM Conf on Embedded network sensor systems. 367--368 (2008).
[5]
Chalmers, M., McMillan, D., Morrison, A., Cramer, H., Rost, M., Mackay, W.: Ethics, Logs and Videotape: Ethics in Large Scale User Trials and User Generated Content. Ext. Abd, Proc CHI (2011).
[6]
Crabtree, A., Benford, S., Greenhalgh, C., Tennent, P., Chalmers, M., Brown, B.: Supporting ethnographic studies of ubiquitous computing in the wild. Proc conference on Designing Interactive systems. (2006).
[7]
Cramer, H., Rost, M. and Bentley, F. An Introduction to Research in the Large, Special Issue IJMHCI (2011).
[8]
Davies, N., Mitchell, K., Cheverst, K. and Blair, G. Developing A Context Sensitive Tourist Guide Proc. HCI for Mobile Devices (1998).
[9]
Denscombe, M. The good research guide. Maidenhead: Open University Press. (2003).
[10]
Denzin, N. K. The Research Act, 3rd edn. Englewood Cliffs, NJ: Prentice Hall. (1989).
[11]
FAST Federation Asks: Do you know what you're agreeing to? www.fastiis.org/resources/press/id/304/.
[12]
Feather, N. T., Values and income level, Australian Journal of Psychology, 27(1), (1975).
[13]
Girardello, A. and Michahelles, F.: AppAware: which mobile applications are hot? : Proc. Int. conference on HCI with mobile devices & services. ACM, 431--434.
[14]
Henze, N., Poppinga, B., Boll, S. Experiments in the Wild: Public Evaluation of Off-Screen Visualizations in the Android Market. Proc NordiCHI (2010).
[15]
James, N, Busher, H.: Credibility, authenticity and voice: dilemmas in online interviewing. Qualitative Research 6: 403--420 (2006).
[16]
Jay, T., Stanton Fraser, D.: The role of a cohort in the design and evaluation of pervasive systems. Proc. ACM Conf on Designing interactive systems. 31--39 (2008).
[17]
Khelil, A., Becker, C., Tian, J. and Rothermel, K.: An epidemic model for information diffusion in MANETs. Proc Int workshop on modeling analysis and simulation of wireless and mobile systems. ACM, (2002).
[18]
Kjeldskov, J., Skov, M., Als, B. and Høegh, R.: Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. MobileHCI, 529--535 (2004).
[19]
Kurkowski, S., Camp, T., Colagrosso, M.: MANET simulation studies: the incredibles. SIGMOBILE Mobile Computing and Comms Review 9 50--61 (2005).
[20]
McMillan, D., Morrison, A., Brown, O., Hall, M. and Chalmers, M.: Further into the Wild: Running Worldwide Trials of Mobile Systems. Pervasive Computing, Vol. 6030. 210--227 (2010).
[21]
McMillan, D., Morrison, A., Chalmers, M. Comparison of Distribution Channels for Large-Scale Deployments of iOS Applications, IJMHCI 3(4), 1--17 (2011).
[22]
Morrison. A., Reeves. S., McMillan, D., Chalmers M. Experiences of Mass Participation in Ubicomp Research. Research In The Large Workshop, ACM International Conference Ubiquitous Computing (2010).
[23]
Oliver, E. A survey of platforms for mobile networks research. SIGMOBILE (2009).
[24]
O'Shea, T., Lamming, M., Chalmers, M., Graube, N., Wellner, P. and Wiginton, G,: Expectations and Perceptions of Ubiquitous Computing: Experiments with BirdDog, a Prototype Person Locator, Proc. BCS/IEE Conf on IT and People (1991).
[25]
Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R., Hursey, J., Toscos, T.: Why it's worth the hassle: the value of in-situ studies when designing Ubicomp. International conference on Ubiquitous computing. Springer-Verlag, 336--35 (2007).
[26]
Rosenthal, R. Experimenter effects in behavioral research. New York, Appleton-Century-Crofts, (1966).
[27]
Schleicher, R., Shirazi, A. S., Rohs, M., Kratz, S., & Schmidt, A. WorldCupinion Experiences with an Android App for Real-Time Opinion Sharing During Soccer World Cup Games. IJMHCI, 3(4), 18--35 (2011).
[28]
Strøm, G., Interaction Design for Countries with a Traditional Culture: A Comparative Study of Income Levels and Cultural Values. People And Computers XIX - The Bigger Picture, 2, 301--316 (2006).
[29]
Sturges, J. and Hanrahan,K.: Comparing Telephone and Face-to-Face Qualitative Interviewing: a Research Note. Qualitative Research April 4: 107--118 (2004).
[30]
Want, R., Hopper, A., Falcão, V. and Gibbons, J.:The active badge location system. ACM Trans. Inf. Syst. 10 (1992).

Cited By

View all
  • (2022)A Scoping Review of Ethics Across SIGCHIProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533511(137-154)Online publication date: 13-Jun-2022
  • (2019)An N-of-1 Evaluation Framework for Behaviour Change ApplicationsExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312923(1-6)Online publication date: 2-May-2019
  • (2018)Snooze!Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3229434.3229436(1-13)Online publication date: 3-Sep-2018
  • Show More Cited By

Index Terms

  1. A hybrid mass participation approach to mobile software trials

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2012
    3276 pages
    ISBN:9781450310154
    DOI:10.1145/2207676
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 May 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. ad hoc peer-to-peer networking
    2. manets
    3. mass participation
    4. mobile multiplayer games
    5. user trial methodology

    Qualifiers

    • Research-article

    Conference

    CHI '12
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)8
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)A Scoping Review of Ethics Across SIGCHIProceedings of the 2022 ACM Designing Interactive Systems Conference10.1145/3532106.3533511(137-154)Online publication date: 13-Jun-2022
    • (2019)An N-of-1 Evaluation Framework for Behaviour Change ApplicationsExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290607.3312923(1-6)Online publication date: 2-May-2019
    • (2018)Snooze!Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3229434.3229436(1-13)Online publication date: 3-Sep-2018
    • (2017)AVUIProceedings of the 2017 CHI Conference on Human Factors in Computing Systems10.1145/3025453.3026042(1093-1104)Online publication date: 2-May-2017
    • (2016)Implementing ethics for a mobile app deploymentProceedings of the 28th Australian Conference on Computer-Human Interaction10.1145/3010915.3010919(406-415)Online publication date: 29-Nov-2016
    • (2016)Personal Tracking of Screen Time on Digital DevicesProceedings of the 2016 CHI Conference on Human Factors in Computing Systems10.1145/2858036.2858055(284-296)Online publication date: 7-May-2016
    • (2016)Work or leisure? Designing a user-centered approach for researching activity "in the wild"Personal and Ubiquitous Computing10.1007/s00779-016-0935-720:4(487-515)Online publication date: 1-Aug-2016
    • (2015)From in the wild to in vivoProceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/2785830.2785883(494-503)Online publication date: 24-Aug-2015
    • (2015)Caring for BatteriesProceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/2785830.2785864(383-392)Online publication date: 24-Aug-2015
    • (2014)Tube starProceedings of the 11th International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services10.4108/icst.mobiquitous.2014.257846(161-170)Online publication date: 2-Dec-2014
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media