Evan Welbourne
ABSTRACT
In this paper we explore a scalable data collection methodology that simultaneously achieves low cost and a high degree of control. We use popular online crowdsourcing platforms to recruit 63 subjects for a 90-day data collection that resulted in over 75K hours of data. The total cost of data collection was dramatically lower than for alternative methodologies, with total subject compensation under $3.5K US, and a total of less than 10 hours/week spent by researchers managing the study. At the same time, our methodology enhances control and enables richer study protocols by allowing direct contact with subjects. We were able to conduct surveys, exchange messages, and debug remotely with feedback from subjects. In addition to reporting on study details, we also discuss interesting findings and offer lessons learned.
- ACRA https://github.com/ACRA Oct 2013Google Scholar
- Aharony, N. et al. Social fMRI: Investigating and shaping social mechanisms in the real world. Pervasive and Mobile Computing 7 643--659 (2011) Google ScholarDigital Library
- Amazon Mechanical Turk. http://www.mturk.com Oct 2013Google Scholar
- Amazon EC2. http://aws.amazon.com/ec2/ Oct 2013Google Scholar
- Apache. http://httpd.apache.org/ Oct 2013Google Scholar
- CastingWords. https://castingwords.com/ Oct 2013Google Scholar
- Django. https://www.djangoproject.com/ Oct 2013Google Scholar
- Elance. http://www.elance.com Oct 2013Google Scholar
- Funf Open Sensing Framework. http://funf.org/ Oct 2013Google Scholar
- Kocchar, S. et al. The Anatomy of a Large-Scale Human Computation Engine. KDD-HCOMP '10, 2010 Google ScholarDigital Library
- Laurila, J. K. The Mobile Data Challenge: Big Data for Mobile Computing Research. Mobile Data Challenge Workshop in conjunction with Pervasive '12, 2012Google Scholar
- Miluzzo, E. et al. Research in the App Store Era: Experiences from the CenceMe App Deployment on the iPhone. Ubicomp '10, 2010Google Scholar
- Nielsen. Who's Winning the U.S. Smartphone Market? http://www.nielsen.com/us/en/newswire.html Aug 2013Google Scholar
- ODesk. http://www.odesk.com Oct 2013Google Scholar
- Paolacci, G. et al. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 2010Google Scholar
- PhoneLab. http://www.phone-lab.org Jan 2014Google Scholar
- Ra, M. et al. Medusa: A Programming Framework for Crowd-Sensing Applications. MobiSys'12, 2012 Google ScholarDigital Library
- Ruby on Rails. http://rubyonrails.org/ Oct 2013Google Scholar
- Schmidt, L. A. Crowdsourcing for Human Subjects Research. CrowdConf 2010, 2010Google Scholar
- Smith, A. Smartphone Ownership -- 2013 Update. Pew Research Center Report, Jun 2013Google Scholar
Index Terms
- Crowdsourced mobile data collection: lessons learned from a new study methodology
Recommendations
Age-based garbage collection
Modern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Age-based garbage collection
OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applicationsModern generational garbage collectors look for garbage among the young objects, because they have high mortality; however, these objects include the very youngest objects, which clearly are still live. We introduce new garbage collection algorithms, ...
Garbage collection hints
HiPEAC'05: Proceedings of the First international conference on High Performance Embedded Architectures and CompilersThis paper shows that Appel-style garbage collectors often make suboptimal decisions both in terms of when and how to collect. We argue that garbage collection should be done when the amount of live bytes is low (in order to minimize the collection cost)...
Comments