Niels van Berkel
Jorge Goncalves
Tilman Dingler
Denzil Ferreira
Vassilis Kostakos
ABSTRACT
Mobile self-reports are a popular technique to collect participant labelled data in the wild. While literature has focused on increasing participant compliance to self-report questionnaires, relatively little work has assessed response accuracy. In this paper, we investigate how participant context can affect response accuracy and help identify strategies to improve the accuracy of mobile self-report data. In a 3-week study we collect over 2,500 questionnaires containing both verifiable and non-verifiable questions. We find that response accuracy is higher for questionnaires that arrive when the phone is not in ongoing or very recent use. Furthermore, our results show that long completion times are an indicator of a lower accuracy. Using contextual mechanisms readily available on smartphones, we are able to explain up to 13% of the variance in participant accuracy. We offer actionable recommendations to assist researchers in their future deployments of mobile self-report studies.
- Sally Andrews, David A. Ellis, Heather Shaw, and Lukasz Piwek. 2015. Beyond Self-Report: Tools to Compare Estimated and Real-World Smartphone Use. PLOS ONE 10, 10 (2015), 1--9.Google Scholar
Cross Ref
- Alan Baddeley. 1992. Working memory. Science 255, 5044 (1992), 556--559.Google ScholarCross Ref
- Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1--48.Google ScholarCross Ref
- Daniel J. Beal and Howard M. Weiss. 2003. Methods of Ecological Momentary Assessment in Organizational Research. Organizational Research Methods 6, 4 (2003), 440--464.Google ScholarCross Ref
- S. L. Beilock and M. S. Decaro. 2007. From poor performance to success under stress: working memory, strategy selection, and mathematical problem solving under pressure. Journal of Experimental Psychology: Learning, Memory, and Cognition 33, 6 (2007), 983--998.Google ScholarCross Ref
- Niels van Berkel, Matthias Budde, Senuri Wijenayake, and Jorge Goncalves. 2018. Improving Accuracy in Mobile Human Contributions: An Overview. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (UbiComp '18). ACM, New York, NY, USA, 594--599. Google ScholarDigital Library
- Niels van Berkel, Denzil Ferreira, and Vassilis Kostakos. 2017. The Experience Sampling Method on Mobile Devices. Comput. Surveys 50, 6, Article 93 (2017), 40 pages. Google ScholarDigital Library
- Niels van Berkel, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2017. Gamification of Mobile Experience Sampling Improves Data Quality and Quantity. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 107 (2017), 21 pages. Google ScholarDigital Library
- Niels van Berkel, Jorge Goncalves, Lauri Lovén, Denzil Ferreira, Simo Hosio, and Vassilis Kostakos. 2019. Effect of Experience Sampling Schedules on Response Rate and Recall Accuracy of Objective SelfReports. International Journal of Human-Computer Studies (2019).Google Scholar
- Benjamin M. Bolker, Mollie E. Brooks, Connie J. Clark, Shane W. Geange, John R. Poulsen, M. Henry H. Stevens, and Jada-Simone S. White. 2009. Generalized linear mixed models: a practical guide for ecology and evolution. Trends in Ecology & Evolution 24, 3 (2009), 127--135.Google ScholarCross Ref
- Catherine E. Connelly, David Zweig, Jane Webster, and John P. Trougakos. 2011. Knowledge hiding in organizations. Journal of Organizational Behavior 33, 1 (2011), 64--88.Google ScholarCross Ref
- S. Consolvo and M. Walker. 2003. Using the experience sampling method to evaluate ubicomp applications. IEEE Pervasive Computing 2, 2 (2003), 24--31. Google ScholarDigital Library
- N. Cowan. 2005. Working Memory Capacity. Psychology Press.Google Scholar
- Nelson Cowan. 2010. The Magical Mystery Four: How Is Working Memory Capacity Limited, and Why? Current Directions in Psychological Science 19, 1 (2010), 51--57. PMID: 20445769.Google ScholarCross Ref
- M. Csikszentmihalyi, R. Larson, and S. Prescott. 1977. The ecology of adolescent activity and experience. Journal of Youth and Adolescence 6, 3 (1977), 281--294.Google ScholarCross Ref
- Mihaly Csikszentmihalyi and Reed Larson. 2014. Validity and Reliability of the Experience-Sampling Method. Springer Netherlands, Dordrecht, 35--54.Google Scholar
- Fred J. Damerau. 1964. A Technique for Computer Detection and Correction of Spelling Errors. Commun. ACM 7, 3 (1964), 171--176. Google ScholarDigital Library
- Tilman Dingler, Albrecht Schmidt, and Tonja Machulla. 2017. Building Cognition-Aware Systems: A Mobile Toolkit for Extracting Time-ofDay Fluctuations of Cognitive Performance. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 47 (2017), 15 pages. Google ScholarDigital Library
- Carsten Eickhoff and Arjen P. de Vries. 2013. Increasing cheat robustness of crowdsourcing tasks. Information Retrieval 16, 2 (2013), 121--137. Google ScholarDigital Library
- R. W. Engle, S. W. Tuholski, J. E. Laughlin, and A. R. Conway. 1999. Working memory, short-term memory, and general fluid intelligence: a latent-variable approach. Journal of Experimental Psychology: General 128, 3 (1999), 309--331.Google ScholarCross Ref
- Hossein Falaki, Ratul Mahajan, Srikanth Kandula, Dimitrios Lymberopoulos, Ramesh Govindan, and Deborah Estrin. 2010. Diversity in Smartphone Usage. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services (MobiSys '10). ACM, New York, NY, USA, 179--194. Google ScholarDigital Library
- R. Frank Falk and Nancy B Miller. 1992. A primer for soft modeling. University of Akron Press.Google Scholar
- Joyce Ho and Stephen S. Intille. 2005. Using Context-aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '05). ACM, New York, NY, USA, 909--918. Google ScholarDigital Library
- Gary Hsieh, Ian Li, Anind Dey, Jodi Forlizzi, and Scott E. Hudson. 2008. Using Visualizations to Increase Compliance in Experience Sampling. In Proceedings of the 10th International Conference on Ubiquitous Computing (UbiComp '08). ACM, New York, NY, USA, 164--167. Google ScholarDigital Library
- Ira E. Hyman, S. Matthew Boss, Breanne M. Wise, Kira E. McKenzie, and Jenna M. Caggiano. 2009. Did you see the unicycling clown? Inattentional blindness while walking and talking on a cell phone. Applied Cognitive Psychology 24, 5 (2009), 597--607.Google ScholarCross Ref
- M Iida, P. E. Shrout, J.-P Laurenceau, and Niall Bolger. 2012. Using diary methods in psychological research. (2012), 277--305.Google Scholar
- Shamsi T. Iqbal and Brian P. Bailey. 2005. Investigating the Effectiveness of Mental Workload As a Predictor of Opportune Moments for Interruption. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, New York, NY, USA, 1489--1492. Google ScholarDigital Library
- Chang-Jae Kim, Sang-hyun Hong, Byung-Sam Kim, Joon-Pyo Cheon, Yoonki Lee, Hyun-Jung Koh, and Jaemin Lee. 2008. Comparison of various tests designed to assess the recovery of cognitive and psychomotor function after ambulatory anesthesia. Korean Journal of Anesthesiology 55, 3 (2008), 291--297.Google ScholarCross Ref
- Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing User Studies with Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 453--456. Google ScholarDigital Library
- Predrag Klasnja, Beverly L. Harrison, Louis LeGrand, Anthony LaMarca, Jon Froehlich, and Scott E. Hudson. 2008. Using Wearable Sensors and Real Time Inference to Understand Human Recall of Routine Activities. In Proceedings of the 10th International Conference on Ubiquitous Computing (UbiComp '08). ACM, New York, NY, USA, 154--163. Google ScholarDigital Library
- Kostadin Kushlev, Jason Proulx, and Elizabeth W. Dunn. 2016. "Silence Your Phones": Smartphone Notifications Increase Inattention and Hyperactivity Symptoms. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1011--1020. Google ScholarDigital Library
- Donald A. Laird. 1925. Relative Performance of College Students as Conditioned by Time of Day and Day of Week. Journal of Experimental Psychology 8, 1 (1925), 50.Google ScholarCross Ref
- Reed Larson and Mihaly Csikszentmihalyi. 2014. The Experience Sampling Method. Springer Netherlands, Dordrecht, 21--34.Google Scholar
- Neal Lathia, Kiran K. Rachuri, Cecilia Mascolo, and Peter J. Rentfrow. 2013. Contextual Dissonance: Design Bias in Sensor-based Experience Sampling Methods. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 183--192. Google ScholarDigital Library
- V. R. LeBlanc, M. M. McConnell, and S. D. Monteiro. 2015. Predictable chaos: a review of the effects of emotions on attention, memory and decision making. Advances in Health Sciences Education. Theory and Practice 20, 1 (2015), 265--282.Google Scholar
- K. O. McCabe, L. Mack, and W. Fleeson. 2012. A guide for data cleaning in experience sampling studies. Guilford Press, New York, NY, US, 321--338.Google Scholar
- Abhinav Mehrotra, Jo Vermeulen, Veljko Pejovic, and Mirco Musolesi. 2015. Ask, but Don't Interrupt: The Case for Interruptibility-aware Mobile Experience Sampling. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers (UbiComp/ISWC'15 Adjunct). ACM, New York, NY, USA, 723--732. Google ScholarDigital Library
- M. R. U. Meyer, C. Wu, and S. M. Walsh. 2016. Theoretical Antecedents of Standing at Work: An Experience Sampling Approach Using the Theory of Planned Behavior. AIMS Public Health 3, 4 (2016), 682--701.Google ScholarCross Ref
- George A Miller. 1956. The Magical Number Seven, Plus or Minus Two: Some limits on our capacity for processing information. Psychological review 63, 2 (1956), 81.Google Scholar
- Minitab. 2014. How to Interpret a Regression Model with Low R-squared and Low P values. https://bit.ly/2otiSw5Google Scholar
- Martin Pielot, Tilman Dingler, Jose San Pedro, and Nuria Oliver. 2015. When Attention is Not Scarce - Detecting Boredom from Mobile Phone Usage. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 825--836. Google ScholarDigital Library
- Suzanne Prescott and Mihaly Csikszentmihalyi. 1981. Environmental effects on cognitive and affective states: The experiential time sampling approach. Social Behavior and Personality: an international journal 9, 1 (1981), 23--32.Google Scholar
- Robert W. Reeder, Adrienne Porter Felt, Sunny Consolvo, Nathan Malkin, Christopher Thompson, and Serge Egelman. 2018. An Experience Sampling Study of User Reactions to Browser Warnings in the Field. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 512, 13 pages. Google ScholarDigital Library
- Harry T. Reis and Shelly L. Gable. 2000. Event-sampling and other methods for studying everyday experience. Handbook of Research Methods in Social and Personality Psychology (2000), 190--222.Google Scholar
- Stephanie Rosenthal, Anind K. Dey, and Manuela Veloso. 2011. Using Decision-Theoretic Experience Sampling to Build Personalized Mobile Phone Interruption Models. In Pervasive Computing, Kent Lyons, Jeffrey Hightower, and Elaine M. Huang (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 170--187. Google ScholarDigital Library
- James A. Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology 39, 6 (1980), 1161.Google ScholarCross Ref
- Ulrich Schimmack. 2003. Affect Measurement in Experience Sampling Research. Journal of Happiness Studies 4, 1 (2003), 79--106.Google ScholarCross Ref
- Christina Schmidt, Fabienne Collette, Christian Cajochen, and Philippe Peigneux. 2007. A time to think: Circadian rhythms in human cognition. Cognitive Neuropsychology 24, 7 (2007), 755--789. PMID: 18066734.Google ScholarCross Ref
- Christie Napa Scollon, Chu-Kim Prieto, and Ed Diener. 2009. Experience Sampling: Promises and Pitfalls, Strength and Weaknesses. Springer Netherlands, Dordrecht, 157--180.Google Scholar
- S. Shiffman, A. A. Stone, and M. R. Hufford. 2008. Ecological Momentary Assessment. Annual Review of Clinical Psychology 4 (2008), 1--32.Google ScholarCross Ref
- A. A. Stone, R. C. Kessler, and J. A. Haythornthwaite. 1991. Measuring daily events and experiences: decisions for the researcher. Journal of Personality 59, 3 (1991), 575--607.Google ScholarCross Ref
- Khai N. Truong, Thariq Shihipar, and Daniel J. Wigdor. 2014. Slide to X: Unlocking the Potential of Smartphone Unlocking. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3635--3644. Google ScholarDigital Library
- Nash Unsworth, Richard P. Heitz, Josef C. Schrock, and Randall W. Engle. 2005. An automated version of the operation span task. Behavior Research Methods 37, 3 (2005), 498--505.Google ScholarCross Ref
- Aku Visuri, Niels van Berkel, Chu Luo, Jorge Goncalves, Denzil Ferreira, and Vassilis Kostakos. 2017. Challenges of quantified-self: encouraging self-reported data logging during recurrent smartphone usage. In Proceedings of the 31st British Computer Society Human Computer Interaction Conference. Google ScholarDigital Library
- Aku Visuri, Niels van Berkel, Chu Luo, Jorge Goncalves, Denzil Ferreira, and Vassilis Kostakos. 2017. Predicting Interruptibility for Manual Data Collection: A Cluster-based User Model. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 12, 14 pages. Google ScholarDigital Library
- R. West, K. J. Murphy, M. L. Armilio, F. I. Craik, and D. T. Stuss. 2002. Effects of time of day on age differences in working memory. Journal of Gerontology 57, 1 (2002), 3--10.Google ScholarCross Ref
- Ladd Wheeler and Harry T. Reis. 1991. Self-Recording of Everyday Life Events: Origins, Types, and Uses. Journal of Personality 59, 3 (1991), 339--354.Google ScholarCross Ref
- David L. Woods, Mark M. Kishiyama, E. William Yund, Timothy J. Herron, Ben Edwards, Oren Poliva, Robert F. Hink, and Bruce Reed. 2011. Improving digit span assessment of short-term verbal memory. Journal of Clinical and Experimental Neuropsychology 33, 1 (2011), 101--111.Google ScholarCross Ref
- J. C. Cassandra Wright, M. Paul Dietze, A. Paul Agius, Emmanuel Kuntsche, Robin Room, Michael Livingston, Margaret Hellard, and S. C. Megan Lim. 2017. An Ecological Momentary Intervention to Reduce Alcohol Consumption in Young Adults Delivered During Drinking Events: Protocol for a Pilot Randomized Controlled Trial. JMIR Research Protocols 6, 5 (2017), e95.Google ScholarCross Ref
- Yulong Yang, Gradeigh D. Clark, Janne Lindqvist, and Antti Oulasvirta. 2016. Free-Form Gesture Authentication in the Wild. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3722--3735. Google ScholarDigital Library
- Zhen Yue, Eden Litt, Carrie J. Cai, Jeff Stern, Kathy K. Baxter, Zhiwei Guan, Nikhil Sharma, and Guangqiang (George) Zhang. 2014. Photographing Information Needs: The Role of Photos in Experience Sampling Method-style Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1545--1554. Google ScholarDigital Library
Index Terms
- Context-Informed Scheduling and Analysis: Improving Accuracy of Mobile Self-Reports
Recommendations
Gamification of Mobile Experience Sampling Improves Data Quality and Quantity
The Experience Sampling Method is used to capture high-quality in situ data from study participants. This method has become popular in studies involving smartphones, where it is often adapted to motivate participation through the use of gamification ...
Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports
Research highlights- We investigate the effect of random, interval, and smartphone-unlock based questionnaires
AbstractThe Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of ...
A large-scale study of daily information needs captured in situ
The goal of this work is to provide a fundamental understanding of the daily information needs of people through a large-scale, in-depth, quantitative investigation. To this end, we have conducted one of the most comprehensive studies of information ...
Comments