Tom Gross
Skip Abstract Section
Abstract
The Experience Sampling Method (ESM) is a popular research method found in many fields to gather rich insights into participants' thoughts, emotions, and daily routines near the moment they happen. During the last decade, many technologically advanced ESM tools emerged that combine manually entered self-reports with automatically collected data from device sensors. In fact, it became difficult to keep track of them. We compiled a survey of ESM and its tools addressing technological capabilities for developers, study design opportunities for study administrators, and answering usability for study participants. It comprises data from 30 systems, applications, and toolkits, which are used for ESM studies. We present our results on the current state of the art from these main user perspectives, list general shortcomings, and give recommendations for future ESM tools.
- ACM. ACM Digital Library. ACM Inc., https://dl.acm.org/, 2022. (Accessed 2/1/2023).Google Scholar
- Apache. Apache Cordova. https://cordova.apache.org/, 2022. (Accessed 20/04/2021).Google Scholar
- Apple. ResearchKit -- Create Powerful Apps for Medical Research. http://researchkit.org/, 2022. (Accessed 2/1/2023).Google Scholar
- Arslan, Ruben C, Walther, Matthias P and Tata, Cyril S. (2020). form: A Study Framework Allowing for Automated Feedback Generation and Complex Longitudinal Experience-Sampling Studies using R. Behavior Research Methods 52, 1. pp. 376--387.Google ScholarCross Ref
- Bachmann, Anja, Zetzsche, Robert, Schankin, Andrea, Riedel, Till, Beigl, Michael, Reichert, Markus, Santangelo, Philip and Ebner-Priemer, Ulrich. (2015). ESMAC: A Web-Based Configurator for Context-Aware Experience Sampling Apps in Ambulatory Assessment. In Proceedings of the 5th EAI International Conference on Wireless Mobile Communication and Healthcare (London, Great Britain). ICST. pp. 15--18.Google ScholarDigital Library
- Batalas, Nikolaos, aan het Rot, Marije, Khan, Vassilis Javed and Markopoulos, Panos. (2018). Using TEMPEST: End-User Programming of Web-Based Ecological Momentary Assessment Protocols. Proceedings of the ACM on Human-Computer Interaction 2, EICS. pp. 1--24.Google Scholar
- Batalas, Nikoloas and Markopoulos, Panos. (2012). Considerations for Computerised In Situ Data Collection Platforms. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2012 (June 25--26, Copenhagen, Denmark). ACM, N.Y. pp. 231--236.Google Scholar
- Behavio. Funf | Open Sensing Framework. https://www.funf.org/about.html, 2022. (Accessed 2/1/2023).Google Scholar
- BlueJayVentures. PIEL Survey | Experience Sampling Method (ESM) | Ecological Momentary Assessment (EMA) | Survey Software. https://pielsurvey.org/, 2022. (Accessed 2/1/2023).Google Scholar
- Broda, Michael. (2017). Using Multilevel Models to Explore Predictors of High School Students' Nonresponse in Experience Sampling Method (ESM) Studies. Social Science Computer Review 35, 6. pp. 733--750.Google ScholarDigital Library
- Carter, Scott, Mankoff, Jennifer and Heer, Jeffrey. (2007). Momento: Support for Situated Ubicomp Experimentation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA). Association for Computing Machinery. pp. 125--134.Google ScholarDigital Library
- Clarivate. Web of Science. https://webofknowledge.com, 2022. (Accessed 2/1/2023).Google Scholar
- Django. The Web Framework for Perfectionists with Deadlines | Django. https://www.djangoproject.com/, 2022. (Accessed 2/1/2023).Google Scholar
- Enketo. Web Forms Evolved. Deploy and Conduct Surveys that Work without a Connection, on Any Device. https://enketo.org/, 2022. (Accessed 2/1/2023).Google Scholar
- ESMCapture. Experience Sampling Made Simple. http://esmcapture.com/, 2022. (Accessed 2/1/2023).Google Scholar
- EsperTech. Complex Event Processing, Streaming Analytics, Streaming SQL - EsperTech. https://www.espertech.com/, 2022. (Accessed 2/1/2023).Google Scholar
- EU. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the Protection of Natural Persons with Regard to the Processing of Personal Data and on the Free Movement of Such Data, and Repealing Directive 95/46/EC (General Data Protection Regulation). https://eur-lex.europa.eu/eli/reg/2016/679/oj, 2022.Google Scholar
- Evans, Bob. (2016). Paco-Applying Computational Methods to Scale Qualitative Methods. In Ethnographic Praxis in Industry Conference Proceedings (Minneapolis, Minnesota, USA). Wiley Online Library. pp. 348--368.Google ScholarCross Ref
- Feldman Barrett, Lisa and Barrett, Daniel. (Summer 2001). An Introduction to Computerised Experience Sampling in Psychology. Social Science Computer Review 19, 2. pp. 175--185.Google Scholar
- Ferreira, Denzil, Kostakos, Vassilis and Dey, Anind K. (2015). AWARE: Mobile Context Instrumentation Framework. Frontiers in ICT 2. pp. 6.Google Scholar
- Fischer, Joel E. (2009). Experience-sampling tools: a critical review. Mobile living labs 09: Methods and tools for evaluation in the wild. pp. 35.Google Scholar
- Froehlich, Jon, Chen, Mike Y, Consolvo, Sunny, Harrison, Beverly and Landay, James A. (2007). MyExperience: A System for In Situ Tracing and Capturing of User Feedback on Mobile Phones. In Proceedings of the 5th international conference on Mobile systems, applications and services (San Juan, Puerto Rico). Association for Computing Machinery. pp. 57--70.Google ScholarDigital Library
- Funk, Matthias, Liang, Rong-Hao, Palanque, Philippe, Hu, Jun and Markopoulos, Panos. (June 2021). Engineering Interactive Computing Systems 2022: Editorial Introduction. Proceedings of the ACM on Human-Computer Interaction 5, EICS, 193. pp. 193:1--193:4.Google Scholar
- Gamma, Erich, Helm, Richard, Johnson, Ralph and Vlissides, John. (1995). Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Longman Publishing Co., Inc.,Google ScholarDigital Library
- Google. Flutter - Beautiful Native Apps in Record Time. https://flutter.dev/, 2022. (Accessed 2/1/2023).Google Scholar
- Google. Google Scholar. https://scholar.google.com/, 2022. (Accessed 2/1/2023).Google Scholar
- Google. Protocol Buffers | Google Developers. https://developers.google.com/protocol-buffers, 2022. (Accessed 2/1/2023).Google Scholar
- Hall, Malcolm, Bell, Marek, Morrison, Alistair, Reeves, Stuart, Sherwood, Scott and Chalmers, Matthew. (2009). Adapting Ubicomp Software and its Evaluation. In Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems - EICS 2009 (July 15--17, Pittsburgh, PA). ACM, N.Y. pp. 141--148.Google ScholarDigital Library
- Harari, Gabriella M, Lane, Nicholas D, Wang, Rui, Crosier, Benjamin S, Campbell, Andrew T and Gosling, Samuel D. (2016). Using Smartphones to Collect Behavioral Data in Psychological Science: Opportunities, Practical Considerations, and Challenges. Perspectives on Psychological Science 11, 6. pp. 838--854.Google ScholarCross Ref
- Hektner, Joel M., Csikszentmihalyi, Mihaly and Schmidt, Jennifer A. (2006). Experience Sampling Method: Measuring the Quality of Everyday Life. Sage Publications Inc., Thousand Oaks, CA.Google Scholar
- Hofmann, Wilhelm and Patel, Paresh V. (2015). SurveySignal: A Convenient Solution for Experience Sampling Research Using Participants' Own Smartphones Social Science Computer Review 33, 2. pp. 235--253.Google Scholar
- iDialogPad. This is an EMA (Ecological Momentary Assessment) and General Questionaire Universal App for iOS. https://github.com/gemu2015/iDialogPad, 2022. (Accessed 2/1/2023).Google Scholar
- IEEE. IEEE Xplore. https://ieeexplore.ieee.org, 2022. (Accessed 03/04/2021).Google Scholar
- ilumivu. mEMA App : ilumivu. https://ilumivu.com/solutions/ecological-momentary-assessment-app/, 2022. (Accessed 2/1/2023).Google Scholar
- Intille, Stephen S, Rondoni, John, Kukla, Charles, Ancona, Isabel and Bao, Ling. (2003). A Context-Aware Experience Sampling Tool. In CHI '03 Extended Abstracts on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA). Association for Computing Machinery. pp. 972--973.Google ScholarDigital Library
- Intille, Stephen S, Stone, AA and Shiffman, S. (2007). Technological Innovations Enabling Automatic, Context-Sensitive Ecological Momentary Assessment. The Science of Real-Time Data Capture: Self-Reports in Health Research. pp. 308--337.Google Scholar
- ISO. Ergonomics of Human-System Interaction - Part 11: Usability: Definitions and Concepts. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en, 2018. (Accessed 20/04/2021).Google Scholar
- Karr, Chris. Purple Robot is an Android Application for Creating Automated Experiences on the Android Platform with Sensing and Inference Capabilities. https://github.com/cbitstech/Purple-Robot, 2022. (Accessed 19/07/2021).Google Scholar
- Keusch, Florian and Conrad, Frederick G. (2021). Using Smartphones to Capture and Combine Self-Reports and Passively Measured Behavior in Social Research. Journal of Survey Statistics and Methodology.Google Scholar
- Koval, Peter, Hinton, J, Dozo, N, Gleeson, John, Alvarez, Mario, Harrison, A, Vu, D, Susanto, R, Jayaputera, G and Sinnott, Richard. SEMA3: Smartphone Ecological Momentary Assessment, Version 3. https://sema3.com/, 2019. (Accessed 2/1/2023).Google Scholar
- Larson, Reed and Csikszentmihalyi, Mihaly. (2014). The Experience Sampling Method. In Csikszentmihalyi, Mihaly, ed. Flow and the Foundations of Positive Psychology. Springer-Verlag, Heidelberg. pp. 21--34.Google ScholarCross Ref
- Lathia, Neal, Rachuri, Kiran, Mascolo, Cecilia and Roussos, George. (2013). Open Source Smartphone Libraries for Computational Social Science. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication (Zurich, Switzerland). Association for Computing Machinery. pp. 911--920.Google ScholarDigital Library
- Lawes, Mario, Hetschko, Clemens, Sakshaug, Joseph W and Grießemer, Stephan. (2021). Contact Modes and Participation in App-Based Smartphone Surveys: Evidence From a Large-Scale Experiment. Social Science Computer Review. pp. 1--17.Google Scholar
- LifeData. LifeData Experience Sampling App | LifeData Experience Sampling App for Research. https://www.lifedatacorp.com/, 2022. (Accessed 2/1/2023).Google Scholar
- Ludwig, Thomas, Dax, Julian, Pipek, Volkmar and Randall, Dave. (2016). Work or Leisure? Designing a User-Centered Approach for Researching Activity ?in the Wild". Personal and Ubiquitous Computing 20, 4. pp. 487--515.Google ScholarDigital Library
- Luyten, Kris, Palanque, Philippe, Quigley, Aaron and Winckler, Marco. (June 2022). Engineering Interactive Computing Systems 2022: Editorial Introduction. Proceedings of the ACM on Human-Computer Interaction 6, EICS, 149. pp. 149:1--149:3.Google ScholarDigital Library
- Meers, Kristof, Dejonckheere, Egon, Kalokerinos, Elise K, Rummens, Koen and Kuppens, Peter. (2020). mobileQ: A Free User-Friendly Application for Collecting Experience Sampling Data. Behaviour Research Methods 52, 4. pp. 1510--1515.Google ScholarCross Ref
- Metricwire. Metricwire Inc. | Real-World Intelligence | Value Based-Decisions. https://metricwire.com/, 2022. (Accessed 19/07/2021).Google Scholar
- Microsoft. GitHub: Where the World Builds Software · GitHub. https://github.com/, 2022. (Accessed 2/1/2023).Google Scholar
- Microsoft. Xamarin | Open-Source Mobile App Platform for .NET. https://dotnet.microsoft.com/apps/xamarin, 2022. (Accessed 2/1/2023).Google Scholar
- movisensXS. movisensXS -- Experience Sampling Solution. movisens GmbH. https://www.movisens.com/en/products/movisensxs/, 2022. (Accessed 2/1/2023).Google Scholar
- Mozilla. MDN Web Docs. https://developer.mozilla.org/en-US/, 2022. (Accessed 20/04/2021).Google Scholar
- Nielson, Jakob. 10 Usability Heuristics for User Interface Design. https://www.nngroup.com/articles/ten-usability-heuristics/, 1994. (Accessed 20/04/2021).Google Scholar
- O'Reilly-Shah, Vikas and Mackey, Sean. (Apr-Jun 2016). Survalytics: An Open-Source Cloud-Integrated Experience Sampling, Survey, and Analytics and Metadata Collection Module for Android Operating System Apps. JMIR mHealth and uHealth 4, 2. pp. 17--26.Google Scholar
- ODK-X. Build your Custom Data Management App. https://odk-x.org/, 2022. (Accessed 2/1/2023).Google Scholar
- ODK. Collect Data Anywhere. https://getodk.org/index.html, 2022. (Accessed 2/1/2023).Google Scholar
- OpenCPU. OpenCPU - Producing and Reproducing Results. https://www.opencpu.org/, 2022. (Accessed 2/1/2023).Google Scholar
- Page, Matthew J., McKenzie, Joanne E., Bossuyt, Patrick M., Boutron, Isabelle, Hoffmann, Tammy C., Mulrow, Cynthia D., Shamseer, Larissa, Tetzlaff, Jennifer M., Akl, Elie A., Brennan, Sue E., Chou, Roger, Glanville, Julie, Grimshaw, Jeremy M., Hróbjartsson, Asbjørn, Lalu, Manoj M., Li, Tianjing, Loder, Elizabeth W., Mayo-Wilson, Evan, McDonald, Steve, McGuinness, Luke A., Stewart, Lesley A., Thomas, James, Tricco, Andrea C., Welch, Vivian A., Whiting, Penny and Moher, David (2021). The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews. Systematic Reviews 10, 89. pp. 2--11.Google Scholar
- Pejovic, Veliko, Lathia, Neal, Mascolo, Cecilia and Musolesi, Mirco. (2016). Mobile-Based Experience Sampling for Behaviour Research. In Tkalcic, Marko, De Carolis, Berardina, de Gemmis, Marco, Odic, Ante and Kosir, Andrej, eds. Emotions and Personality in Personalised Services. Springer-Verlag, Heidelberg. pp. 141--161.Google ScholarCross Ref
- Pejovic, Veljko, Lathia, Neal, Mascolo, Cecilia and Musolesi, Mirco. (2016). Mobile-Based Experience Sampling for Behaviour Research. In Emotions and personality in personalised services. Springer. pp. 141--161.Google ScholarCross Ref
- Qualtrics. Online Survey Software - 1B+ Surveys Annually | Qualtrics. https://www.qualtrics.com/uk/core-xm/survey-software/, 2022. (Accessed 2/1/2023).Google Scholar
- Rough, Daniel J and Quigley, Aaron. (2020). End-User Development of Experience Sampling Smartphone Apps-Recommendations and Requirements. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2. pp. 1--19.Google ScholarDigital Library
- Rough, Daniel John and Quigley, Aaron John. (2017). Jeeves - An Experience Sampling Study Creation Tool. In BCS Health Informatics Scotland (HIS) (Edinburgh, United Kingdom). BCS Electronic Workshops in Computing.Google Scholar
- Salganik, Matthew J. (2019). Bit by Bit: Social Research in the Digital Age. Princeton University Press,Google Scholar
- Silvia, Paul J, Kwapil, Thomas R, Eddington, Kari M and Brown, Leslie H. (2013). Missed Beeps and Missing Data: Dispositional and Situational Predictors of Nonresponse in Experience Sampling Research. Social Science Computer Review 31, 4. pp. 471--481.Google ScholarCross Ref
- Stier, Sebastian, Breuer, Johannes, Siegers, Pascal and Thorson, Kjerstin. (2020). Integrating Survey Data and Digital Trace Data: Key Issues in Developing an Emerging Field. Social Science Computer Review 38, 5. pp. 503--516.Google ScholarDigital Library
- Struminskaya, Bella, Lugtig, Peter, Keusch, Florian and Höhne, Jan Karem. (2020). Augmenting Surveys With Data From Sensors and Apps: Opportunities and Challenges. Social Science Computer Review. pp. 1--13.Google Scholar
- Tangmunarunkit, Hongsuda, Hsieh, Cheng-Kang, Longstaff, Brent, Nolen, S, Jenkins, John, Ketcham, Cameron, Selsky, Joshua, Alquaddoomi, Faisal, George, Dony and Kang, Jinha. (2015). Ohmage: A General and Extensible End-to-End Participatory Sensing Platform. ACM Transactions on Intelligent Systems and Technology (TIST) 6, 3. pp. Article 38, 01--21.Google Scholar
- Taylor, John T and Taylor, Wayne T. (2021). Software Architecture: Foundations, Theory, and Practice. Springer,Google Scholar
- Teamscope. Teamscope: Data Collection App for Research | Secure, Offline. https://www.teamscopeapp.com/, 2022. (Accessed 2/1/2023).Google Scholar
- Thai, Sabrina and Page-Gould, Elizabeth. (Dec 2018). ExperienceSampler: An Open-Source Scaffold for Building Smartphone Apps for Experience Sampling. Psychological Methods 23, 4. pp. 729--739.Google ScholarCross Ref
- Urbandroid Team. Don't Kill My App. https://dontkillmyapp.com/, 2022. (Accessed 3/1/2023).Google Scholar
- Van Berkel, Niels, Ferreira, Denzil and Kostakos, Vassilis. (2017). The Experience Sampling Method on Mobile Devices. ACM Computing Surveys (CSUR) 50, 6. pp. Article 93, 01--40.Google Scholar
- Van Berkel, Niels, Ferreira, Denzil and Kostakos, Vassilis. (Dec. 2017). The Experience Sampling Method on Mobile Devices. ACM Computing Surveys 50, 6. pp. 93:1--93:40.Google Scholar
- W3C. Geolocation API. https://www.w3.org/TR/geolocation/, 2022. (Accessed 2/1/2023).Google Scholar
- Wenz, Alexander, Jäckle, Annette, Burton, Jonathan and Couper, Mick P. (2022). The Effects of Personalized Feedback on Participation and Reporting in Mobile App Data Collection. Social Science Computer Review 40, 1. pp. 165--178.Google ScholarDigital Library
- Xiong, Haoyi, Huang, Yu, Barnes, Laura E and Gerber, Matthew S. (2016). Sensus: A Cross-Platform, General-Purpose System for Mobile Crowdsensing in Human-Subject Studies. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany). Association for Computing Machinery. pp. 415--426.Google ScholarDigital Library
Index Terms
- The Experience Sampling Method and its Tools: A Review for Developers, Study Administrators, and Participants
Recommendations
The Experience Sampling Method on Mobile Devices
The Experience Sampling Method (ESM) is used by scientists from various disciplines to gather insights into the intra-psychic elements of human life. Researchers have used the ESM in a wide variety of studies, with the method seeing increased ...
Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports
Research highlights- We investigate the effect of random, interval, and smartphone-unlock based questionnaires
AbstractThe Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of ...
End-User Development of Experience Sampling Smartphone Apps -Recommendations and Requirements
Professional programmers are significantly outnumbered by end-users of software, making it problematic to predict the diverse, dynamic needs of these users in advance. An end-user development (EUD) approach, supporting the creation and modification of ...
Comments