Alistair Morrison
Stuart Reeves
Matthew Chalmers
ABSTRACT
User trials of mobile applications have followed a steady march out of the lab, and progressively further ''into the wild', recently involving ''app store'-style releases of software to the general public. Yet from our experiences on these mass participation systems and a survey of the literature, we identify a number of reported difficulties. We propose a hybrid methodology that aims to address these, by combining a global software release with a concurrent local trial. A phone-based game, created to explore the uptake and use of ad hoc peer-to-peer networking, was evaluated using this new hybrid trial method, combining a small-scale local trial (11 users) with a ''mass participation' trial (over 10,000 users). Our hybrid method offers many benefits, allowing locally observed findings to be verified, patterns in globally collected data to be explained and addresses ethical issues raised by the mass participation approach. We note trends in the local trial that did not appear in the larger scale deployment, and which would therefore have led to misleading results were the application trialed using ''traditional' methods alone. Based on this study and previous experience, we provide a set of guidelines to researchers working in this area.
- Abowd, G. D., Mynatt, E. D.: Charting past, present, and future research in ubiquitous computing. ACM Trans. Comput.-Hum. Interact. 7, 29--58 (2000). Google Scholar
Digital Library
- Ames, M., Naaman, M.: Why We Tag: Motivations for Annotation in Mobile and Online Media. CHI (2007). Google ScholarDigital Library
- Bell, M and Chalmers, M., and Barkhuus, L., Hall, M., Sherwood, S., Tennent, P., Brown, B., Rowland, D., Benford, S., Capra, M., & Hampshire, A., Interweaving Mobile Games with Everyday Life, Proc. ACM CHI, pp. 417--426. (2006). Google ScholarDigital Library
- Campbell, A., Eisenman, S., Fodor, K., Lane, N., Lu, H., Miluzzo, E., Musolesi, M., Peterson, R., Zheng, X. Transforming the social networking experience with sensing presence from mobile phones. Proc ACM Conf on Embedded network sensor systems. 367--368 (2008). Google ScholarDigital Library
- Chalmers, M., McMillan, D., Morrison, A., Cramer, H., Rost, M., Mackay, W.: Ethics, Logs and Videotape: Ethics in Large Scale User Trials and User Generated Content. Ext. Abd, Proc CHI (2011). Google ScholarDigital Library
- Crabtree, A., Benford, S., Greenhalgh, C., Tennent, P., Chalmers, M., Brown, B.: Supporting ethnographic studies of ubiquitous computing in the wild. Proc conference on Designing Interactive systems. (2006). Google ScholarDigital Library
- Cramer, H., Rost, M. and Bentley, F. An Introduction to Research in the Large, Special Issue IJMHCI (2011).Google Scholar
- Davies, N., Mitchell, K., Cheverst, K. and Blair, G. Developing A Context Sensitive Tourist Guide Proc. HCI for Mobile Devices (1998).Google Scholar
- Denscombe, M. The good research guide. Maidenhead: Open University Press. (2003).Google Scholar
- Denzin, N. K. The Research Act, 3rd edn. Englewood Cliffs, NJ: Prentice Hall. (1989).Google Scholar
- FAST Federation Asks: Do you know what you're agreeing to? www.fastiis.org/resources/press/id/304/.Google Scholar
- Feather, N. T., Values and income level, Australian Journal of Psychology, 27(1), (1975).Google ScholarCross Ref
- Girardello, A. and Michahelles, F.: AppAware: which mobile applications are hot? : Proc. Int. conference on HCI with mobile devices & services. ACM, 431--434. Google ScholarDigital Library
- Henze, N., Poppinga, B., Boll, S. Experiments in the Wild: Public Evaluation of Off-Screen Visualizations in the Android Market. Proc NordiCHI (2010). Google ScholarDigital Library
- James, N, Busher, H.: Credibility, authenticity and voice: dilemmas in online interviewing. Qualitative Research 6: 403--420 (2006).Google ScholarCross Ref
- Jay, T., Stanton Fraser, D.: The role of a cohort in the design and evaluation of pervasive systems. Proc. ACM Conf on Designing interactive systems. 31--39 (2008). Google ScholarDigital Library
- Khelil, A., Becker, C., Tian, J. and Rothermel, K.: An epidemic model for information diffusion in MANETs. Proc Int workshop on modeling analysis and simulation of wireless and mobile systems. ACM, (2002). Google ScholarDigital Library
- Kjeldskov, J., Skov, M., Als, B. and Høegh, R.: Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. MobileHCI, 529--535 (2004).Google Scholar
- Kurkowski, S., Camp, T., Colagrosso, M.: MANET simulation studies: the incredibles. SIGMOBILE Mobile Computing and Comms Review 9 50--61 (2005). Google ScholarDigital Library
- McMillan, D., Morrison, A., Brown, O., Hall, M. and Chalmers, M.: Further into the Wild: Running Worldwide Trials of Mobile Systems. Pervasive Computing, Vol. 6030. 210--227 (2010). Google ScholarDigital Library
- McMillan, D., Morrison, A., Chalmers, M. Comparison of Distribution Channels for Large-Scale Deployments of iOS Applications, IJMHCI 3(4), 1--17 (2011). Google ScholarDigital Library
- Morrison. A., Reeves. S., McMillan, D., Chalmers M. Experiences of Mass Participation in Ubicomp Research. Research In The Large Workshop, ACM International Conference Ubiquitous Computing (2010).Google Scholar
- Oliver, E. A survey of platforms for mobile networks research. SIGMOBILE (2009). Google ScholarDigital Library
- O'Shea, T., Lamming, M., Chalmers, M., Graube, N., Wellner, P. and Wiginton, G,: Expectations and Perceptions of Ubiquitous Computing: Experiments with BirdDog, a Prototype Person Locator, Proc. BCS/IEE Conf on IT and People (1991).Google Scholar
- Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R., Hursey, J., Toscos, T.: Why it's worth the hassle: the value of in-situ studies when designing Ubicomp. International conference on Ubiquitous computing. Springer-Verlag, 336--35 (2007). Google ScholarDigital Library
- Rosenthal, R. Experimenter effects in behavioral research. New York, Appleton-Century-Crofts, (1966).Google Scholar
- Schleicher, R., Shirazi, A. S., Rohs, M., Kratz, S., & Schmidt, A. WorldCupinion Experiences with an Android App for Real-Time Opinion Sharing During Soccer World Cup Games. IJMHCI, 3(4), 18--35 (2011). Google ScholarDigital Library
- Strøm, G., Interaction Design for Countries with a Traditional Culture: A Comparative Study of Income Levels and Cultural Values. People And Computers XIX - The Bigger Picture, 2, 301--316 (2006).Google Scholar
- Sturges, J. and Hanrahan,K.: Comparing Telephone and Face-to-Face Qualitative Interviewing: a Research Note. Qualitative Research April 4: 107--118 (2004).Google Scholar
- Want, R., Hopper, A., Falcão, V. and Gibbons, J.:The active badge location system. ACM Trans. Inf. Syst. 10 (1992). Google ScholarDigital Library
Index Terms
- A hybrid mass participation approach to mobile software trials
Recommendations
A hybrid probabilistic counter-based broadcast approach protocol for Mobile Ad hoc Networks
Broadcast has been widely used in Mobile Ad Hoc Networks (MANETs) to disseminate information to all reachable nodes. However, the conventional broadcast scheme using simple flooding mechanism suffers from broadcast storm problem characterised by ...
A hybrid multiagent routing approach for wireless ad hoc networks
Wireless ad-hoc networks are infrastructureless networks that comprise wireless mobile nodes able to communicate each other outside wireless transmission range. Due to frequent network topology changes in one hand and the limited underlying bandwidth in ...
Hybrid routing protocol with Quality of Service support for Ad Hoc Wireless Networks
In Mobile Ad Hoc Networks (MANETs), mobile nodes must collaborate amongst themselves to interconnect and establish communication. Moreover, their networking characteristics highlight a new breed of unresolved routing and Quality of Service problems. In ...
Comments