research-article

Are You Killing Time? Predicting Smartphone Users’ Time-killing Moments via Fusion of Smartphone Sensor Data and Screenshots

Authors:

Yung-Ju ChangAuthors Info & Claims

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 647, Pages 1 - 19

https://doi.org/10.1145/3544548.3580689

Published: 19 April 2023 Publication History

Abstract

Time-killing on smartphones has become a pervasive activity, and could be opportune for delivering content to their users. This research is believed to be the first attempt at time-killing detection, which leverages the fusion of phone-sensor and screenshot data. We collected nearly one million user-annotated screenshots from 36 Android users. Using this dataset, we built a deep-learning fusion model, which achieved a precision of 0.83 and an AUROC of 0.72. We further employed a two-stage clustering approach to separate users into four groups according to the patterns of their phone-usage behaviors, and then built a fusion model for each group. The performance of the four models, though diverse, yielded better average precision of 0.87 and AUROC of 0.76, and was superior to that of the general/unified model shared among all users. We investigated and discussed the features of the four time-killing behavior clusters that explain why the models’ performance differ.

Supplementary Material

Supplemental Materials (3544548.3580689-supplemental-materials.zip)

Download
59.44 MB

MP4 File (3544548.3580689-talk-video.mp4)

Pre-recorded Video Presentation

Download
215.87 MB

References

[1]

Piotr D. Adamczyk and Brian P. Bailey. 2004. If Not Now, When? The Effects of Interruption at Different Moments within Task Execution. Association for Computing Machinery, New York, NY, USA, 271–278. https://doi.org/10.1145/985692.985727

Digital Library

[2]

Piotr D. Adamczyk, Shamsi T. Iqbal, and Brian P. Bailey. 2005. A method, system, and tools for intelligent interruption management. In Proceedings of the 4th international workshop on Task models and diagrams(TAMODIA ’05). Association for Computing Machinery, New York, NY, USA, 123–126. https://doi.org/10.1145/1122935.1122959

Digital Library

[3]

Elena Agapie, Jaime Teevan, and Andrés Monroy-Hernández. 2015. Crowdsourcing in the field: A case study using local crowds for event reporting. In Third AAAI Conference on Human Computation and Crowdsourcing(HCOMP ’15).

[4]

Yeslam Al-Saggaf, Rachel MacCulloch, and Karl Wiener. 2019. Trait Boredom Is a Predictor of Phubbing Frequency. Journal of Technology in Behavioral Science 4 (09 2019). https://doi.org/10.1007/s41347-018-0080-4

[5]

Yeslam Al-Saggaf and Sarah B O’Donnell. 2019. Phubbing: Perceptions, reasons behind, predictors, and impacts. Human Behavior and Emerging Technologies 1, 2 (2019), 132–140.

[6]

Amanda Baughan, Mingrui Ray Zhang, Raveena Rao, Kai Lukoff, Anastasia Schaadhardt, Lisa D Butler, and Alexis Hiniker. 2022. “I Don’t Even Remember What I Read”: How Design Influences Dissociation on Social Media. In CHI Conference on Human Factors in Computing Systems. 1–13.

[7]

Tony Beltramelli. 2018. Pix2code: Generating Code from a Graphical User Interface Screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems (Paris, France) (EICS ’18). Association for Computing Machinery, New York, NY, USA, Article 3, 6 pages. https://doi.org/10.1145/3220134.3220135

Digital Library

[8]

Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, and Gernot Bauer. 2011. Falling Asleep with Angry Birds, Facebook and Kindle: A Large Scale Study on Mobile Application Usage. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (Stockholm, Sweden) (MobileHCI ’11). Association for Computing Machinery, New York, NY, USA, 47–56. https://doi.org/10.1145/2037373.2037383

Digital Library

[9]

Miriam Brinberg, Nilam Ram, Xiao Yang, Mu-Jung Cho, S Shyam Sundar, Thomas N Robinson, and Byron Reeves. 2021. The idiosyncrasies of everyday digital lives: Using the Human Screenome Project to study user behavior on smartphones. Computers in Human Behavior 114 (2021), 106570.

Digital Library

[10]

Barry Brown, Moira McGregor, and Eric Laurier. 2013. IPhone in Vivo: Video Analysis of Mobile Device Use. Association for Computing Machinery, New York, NY, USA, 1031–1040. https://doi.org/10.1145/2470654.2466132

Digital Library

[11]

Barry Brown, Moira McGregor, and Donald McMillan. 2014. 100 Days of IPhone Use: Understanding the Details of Mobile Device Use. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 223–232. https://doi.org/10.1145/2628363.2628377

Digital Library

[12]

Carrie J. Cai, Anji Ren, and Robert C. Miller. 2017. WaitSuite: Productive Use of Diverse Waiting Moments. ACM Trans. Comput.-Hum. Interact. 24, 1, Article 7 (March 2017), 41 pages. https://doi.org/10.1145/3044534

Digital Library

[13]

Yung-Ju Chang and John C. Tang. 2015. Investigating Mobile Users’ Ringer Mode Usage and Attentiveness and Responsiveness to Communication. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 6–15. https://doi.org/10.1145/2785830.2785852

Digital Library

[14]

Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery D.C.: Design Search and Knowledge Discovery through Auto-Created GUI Component Gallery. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 180 (Nov. 2019), 22 pages. https://doi.org/10.1145/3359282

Digital Library

[15]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 665–676. https://doi.org/10.1145/3180155.3180240

Digital Library

[16]

Yu-Chun Chen, Keui-Chun Kao, Yu-Jen Lee, Faye Shih, Wei-Chen Chiu, and Yung-Ju Chang. 2021. Killing-Time Detection from Smartphone Screenshots. In Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers(Virtual, USA) (UbiComp ’21). Association for Computing Machinery, New York, NY, USA, 15–16. https://doi.org/10.1145/3460418.3479295

Digital Library

[17]

Pei-Yu Peggy Chi, Matthew Long, Akshay Gaur, Abhimanyu Deora, Anurag Batra, and Daphne Luong. 2019. Crowdsourcing Images for Global Diversity. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI ’19). Association for Computing Machinery, New York, NY, USA, Article 79, 10 pages. https://doi.org/10.1145/3338286.3347546

Digital Library

[18]

Chia-En Chiang, Yu-Chun Chen, Fang-Yu Lin, Felicia Feng, Hao-An Wu, Hao-Ping Lee, Chang-Hsuan Yang, and Yung-Ju Chang. 2021. “I Got Some Free Time”: Investigating Task-Execution and Task-Effort Metrics in Mobile Crowdsourcing Tasks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 648, 14 pages. https://doi.org/10.1145/3411764.3445477

Digital Library

[19]

Woohyeok Choi, Sangkeun Park, Duyeon Kim, Youn-kyung Lim, and Uichin Lee. 2019. Multi-Stage Receptivity Model for Mobile Just-In-Time Health Intervention. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 39 (June 2019), 26 pages. https://doi.org/10.1145/3328910

Digital Library

[20]

Mihaly Csikszentmihalyi. 2000. Beyond boredom and anxiety.Jossey-bass.

[21]

Tilman Dingler and Martin Pielot. 2015. I’ll Be There for You: Quantifying Attentiveness towards Mobile Messaging. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 1–5. https://doi.org/10.1145/2785830.2785840

Digital Library

[22]

Tilman Dingler, Benjamin Tag, Sabrina Lehrer, and Albrecht Schmidt. 2018. Reading Scheduler: Proactive Recommendations to Help Users Cope with Their Daily Reading Volume. In Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia (Cairo, Egypt) (MUM 2018). Association for Computing Machinery, New York, NY, USA, 239–244. https://doi.org/10.1145/3282894.3282917

Digital Library

[23]

Tilman Dingler, Dominik Weber, Martin Pielot, Jennifer Cooper, Chung-Cheng Chang, and Niels Henze. 2017. Language Learning On-the-Go: Opportune Moments and Design of Mobile Microlearning Sessions. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI ’17). Association for Computing Machinery, New York, NY, USA, Article 28, 12 pages. https://doi.org/10.1145/3098279.3098565

Digital Library

[24]

Trinh Minh Tri Do, Jan Blom, and Daniel Gatica-Perez. 2011. Smartphone Usage in the Wild: A Large-Scale Analysis of Applications and Context. In Proceedings of the 13th International Conference on Multimodal Interfaces (Alicante, Spain) (ICMI ’11). Association for Computing Machinery, New York, NY, USA, 353–360. https://doi.org/10.1145/2070481.2070550

Digital Library

[25]

John D Eastwood, Alexandra Frischen, Mark J Fenske, and Daniel Smilek. 2012. The unengaged mind: Defining boredom in terms of attention. Perspectives on Psychological Science 7, 5 (2012), 482–495.

[26]

Andreas Elpidorou. 2018. The bored mind is a guiding mind: Toward a regulatory theory of boredom. Phenomenology and the Cognitive Sciences 17 (2018), 455–484.

[27]

Hossein Falaki, Ratul Mahajan, Srikanth Kandula, Dimitrios Lymberopoulos, Ramesh Govindan, and Deborah Estrin. 2010. Diversity in Smartphone Usage. In Proceedings of the 8th International Conference on Mobile Systems, Applications, and Services (San Francisco, California, USA) (MobiSys ’10). Association for Computing Machinery, New York, NY, USA, 179–194. https://doi.org/10.1145/1814433.1814453

Digital Library

[28]

Robert Fisher and Reid Simmons. 2011. Smartphone Interruptibility Using Density-Weighted Uncertainty Sampling with Reinforcement Learning. In 2011 10th International Conference on Machine Learning and Applications and Workshops, Vol. 1. 436–441. https://doi.org/10.1109/ICMLA.2011.128

Digital Library

[29]

Jon Froehlich, Mike Y. Chen, Sunny Consolvo, Beverly Harrison, and James A. Landay. 2007. MyExperience: A System for in Situ Tracing and Capturing of User Feedback on Mobile Phones. In Proceedings of the 5th International Conference on Mobile Systems, Applications and Services (San Juan, Puerto Rico) (MobiSys ’07). Association for Computing Machinery, New York, NY, USA, 57–70. https://doi.org/10.1145/1247660.1247670

Digital Library

[30]

Ralph R Greenson. 1953. On boredom. Journal of the American Psychoanalytic Association 1, 1(1953), 7–21.

[31]

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. In International Joint Conference on Artificial Intelligence (IJCAI).

[32]

Alexis Hiniker, Shwetak N. Patel, Tadayoshi Kohno, and Julie A. Kientz. 2016. Why Would You Do That? Predicting the Uses and Gratifications behind Smartphone-Usage Behaviors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 634–645. https://doi.org/10.1145/2971648.2971762

Digital Library

[33]

Bo-Jhang Ho, Bharathan Balaji, Mehmet Koseoglu, Sandeep Sandha, Siyou Pei, and Mani Srivastava. 2020. Quick Question: Interrupting Users for Microtasks with Reinforcement Learning. arXiv:2007.09515 [cs] (July 2020). http://arxiv.org/abs/2007.09515 arXiv:2007.09515.

[34]

Joyce Ho and Stephen S. Intille. 2005. Using Context-Aware Computing to Reduce the Perceived Burden of Interruptions from Mobile Devices. Association for Computing Machinery, New York, NY, USA, 909–918. https://doi.org/10.1145/1054972.1055100

Digital Library

[35]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.

Digital Library

[36]

Cynthia A. Hoffner and Sangmi Lee. 2015. Mobile Phone Use, Emotion Regulation, and Well-Being. Cyberpsychology, Behavior, and Social Networking 18, 7(2015), 411–416. https://doi.org/10.1089/cyber.2014.0487 arXiv:https://doi.org/10.1089/cyber.2014.0487PMID: 26167841.

[37]

Nanna Inie and Mircea F Lungu. 2021. Aiki - Turning Online Procrastination into Microlearning. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 369, 13 pages. https://doi.org/10.1145/3411764.3445202

Digital Library

[38]

Shamsi T. Iqbal and Brian P. Bailey. 2007. Understanding and developing models for detecting and differentiating breakpoints during interactive tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 697–706. https://doi.org/10.1145/1240624.1240732

Digital Library

[39]

Shamsi T. Iqbal and Brian P. Bailey. 2011. Oasis: A framework for linking notification delivery to the perceptual structure of goal-directed tasks. ACM Transactions on Computer-Human Interaction 17, 4 (Dec. 2011), 15:1–15:28. https://doi.org/10.1145/1879831.1879833

Digital Library

[40]

Ellen Isaacs, Alan Walendowski, Steve Whittaker, Diane J. Schiano, and Candace Kamm. 2002. The Character, Functions, and Styles of Instant Messaging in the Workplace. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (New Orleans, Louisiana, USA) (CSCW ’02). Association for Computing Machinery, New York, NY, USA, 11–20. https://doi.org/10.1145/587078.587081

Digital Library

[41]

Ellen Isaacs, Nicholas Yee, Diane J Schiano, Nathan Good, Nicolas Ducheneaut, and Victoria Bellotti. 2009. Mobile microwaiting moments: The role of context in receptivity to content while on the go. PARC white paper (2009) 10 (2009).

[42]

Chakajkla Jesdabodi and Walid Maalej. 2015. Understanding Usage States on Mobile Devices. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (UbiComp ’15). Association for Computing Machinery, New York, NY, USA, 1221–1225. https://doi.org/10.1145/2750858.2805837

Digital Library

[43]

Simon L. Jones, Denzil Ferreira, Simo Hosio, Jorge Goncalves, and Vassilis Kostakos. 2015. Revisitation Analysis of Smartphone App Use. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Osaka, Japan) (UbiComp ’15). Association for Computing Machinery, New York, NY, USA, 1197–1208. https://doi.org/10.1145/2750858.2807542

Digital Library

[44]

Kleomenis Katevas, Ioannis Arapakis, and Martin Pielot. 2018. Typical Phone Use Habits: Intense Use Does Not Predict Negative Well-Being. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (Barcelona, Spain) (MobileHCI ’18). Association for Computing Machinery, New York, NY, USA, Article 11, 13 pages. https://doi.org/10.1145/3229434.3229441

Digital Library

[45]

Jürgen Kawalek, Annegret Stark, and Marcel Riebeck. 2008. A New Approach to Analyze Human-Mobile Computer Interaction. J. Usability Studies 3, 2 (Feb. 2008), 90–98.

Digital Library

[46]

Ronald C Kessler, Lenard Adler, Minnie Ames, Olga Demler, Steve Faraone, EVA Hiripi, Mary J Howes, Robert Jin, Kristina Secnik, Thomas Spencer, 2005. The World Health Organization Adult ADHD Self-Report Scale (ASRS): a short screening scale for use in the general population. Psychological medicine 35, 2 (2005), 245–256.

[47]

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arxiv:1412.6980 [cs.LG]

[48]

Vassilis Kostakos, Denzil Ferreira, Jorge Goncalves, and Simo Hosio. 2016. Modelling Smartphone Usage: A Markov State Transition Model. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing(Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 486–497. https://doi.org/10.1145/2971648.2971669

Digital Library

[49]

Philipp Krieter. 2019. Can I Record Your Screen? Mobile Screen Recordings as a Long-Term Data Source for User Studies. In Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia (Pisa, Italy) (MUM ’19). Association for Computing Machinery, New York, NY, USA, Article 23, 10 pages. https://doi.org/10.1145/3365610.3365618

Digital Library

[50]

Philipp Krieter and Andreas Breiter. 2018. Analyzing Mobile Application Usage: Generating Log Files from Mobile Screen Recordings. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (Barcelona, Spain) (MobileHCI ’18). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. https://doi.org/10.1145/3229434.3229450

Digital Library

[51]

Nuning Kurniasih. 2017. Internet Addiction, Lifestyle or Mental Disorder? A Phenomenological Study on Social Media Addiction in Indonesia. KnE Social Sciences 2, 4 (Jun. 2017), 135–144. https://doi.org/10.18502/kss.v2i4.879

[52]

Hao-Ping Lee, Kuan-Yin Chen, Chih-Heng Lin, Chia-Yu Chen, Yu-Lin Chung, Yung-Ju Chang, and Chien-Ru Sun. 2019. Does Who Matter? Studying the Impact of Relationship Characteristics on Receptivity to Mobile IM Messages. Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300756

Digital Library

[53]

Jian Li, Andrew Lepp, and Jacob E. Barkley. 2015. Locus of control and cell phone use: Implications for sleep quality, academic performance, and subjective well-being. Computers in Human Behavior 52 (2015), 450–457. https://doi.org/10.1016/j.chb.2015.06.021

Digital Library

[54]

Tong Li, Mingyang Zhang, Hancheng Cao, Yong Li, Sasu Tarkoma, and Pan Hui. 2020. ”What Apps Did You Use?”: Understanding the Long-Term Evolution of Mobile App Usage. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW ’20). Association for Computing Machinery, New York, NY, USA, 66–76. https://doi.org/10.1145/3366423.3380095

Digital Library

[55]

Yu-Hsuan Lin, Li-Ren Chang, Yang-Han Lee, Hsien-Wei Tseng, Terry BJ Kuo, and Sue-Huei Chen. 2014. Development and Validation of the Smartphone Addiction Inventory (SPAI).PloS one 9, 6 (2014), e98312.

[56]

Kai Lukoff, Cissy Yu, Julie Kientz, and Alexis Hiniker. 2018. What Makes Smartphone Use Meaningful or Meaningless?Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 22 (March 2018), 26 pages. https://doi.org/10.1145/3191754

Digital Library

[57]

James MacQueen 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Vol. 1. Oakland, CA, USA, 281–297.

[58]

Donald McMillan, Moira McGregor, and Barry Brown. 2015. From in the Wild to in Vivo: Video Analysis of Mobile Device Use. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 494–503. https://doi.org/10.1145/2785830.2785883

Digital Library

[59]

Abhinav Mehrotra, Robert Hendley, and Mirco Musolesi. 2016. PrefMiner: Mining User’s Preferences for Intelligent Mobile Notification Management. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 1223–1234. https://doi.org/10.1145/2971648.2971747

Digital Library

[60]

Varun Mishra, Florian Künzler, Jan-Niklas Kramer, Elgar Fleisch, Tobias Kowatsch, and David Kotz. 2021. Detecting receptivity for mhealth interventions in the natural environment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1–24.

Digital Library

[61]

Aditi Misra, Aaron Gooze, Kari E. Watkins, M. Asad, and Christopher A. Le Dantec. 2014. Crowdsourcing and Its Application to Transportation Data Collection and Management. Transportation Research Record 2414 (2014), 1 – 8.

[62]

Christopher Monk, Deborah Boehm-Davis, and J. Trafton. 2002. The Attentional Costs of Interrupting Task Performance at Various Stages. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 46 (Sept. 2002). https://doi.org/10.1177/154193120204602210

[63]

Tadashi Okoshi, Julian Ramos, Hiroki Nozaki, Jin Nakazawa, Anind K Dey, and Hideyuki Tokuda. 2015. Attelia: Reducing user’s cognitive load due to interruptive notifications on smart phones. In 2015 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 96–104. https://doi.org/10.1109/PERCOM.2015.7146515

[64]

Tadashi Okoshi, Kota Tsubouchi, Masaya Taji, Takanori Ichikawa, and Hideyuki Tokuda. 2017. Attention and engagement-awareness in the wild: A large-scale study with adaptive notifications. 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom) (2017), 100–110.

[65]

Antti Oulasvirta, Tye Rattenbury, Lingyi Ma, and Eeva Raita. 2012. Habits Make Smartphone Use More Pervasive. Personal Ubiquitous Comput. 16, 1 (Jan. 2012), 105–114. https://doi.org/10.1007/s00779-011-0412-2

Digital Library

[66]

Leysia Palen and Marilyn Salzman. 2002. Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile Conditions. In Proceedings of the 2002 ACM Conference on Computer Supported Cooperative Work (New Orleans, Louisiana, USA) (CSCW ’02). Association for Computing Machinery, New York, NY, USA, 87–95. https://doi.org/10.1145/587078.587092

Digital Library

[67]

Chunjong Park, Junsung Lim, Juho Kim, Sung-Ju Lee, and Dongman Lee. 2017. Don’t Bother Me. I’m Socializing! A Breakpoint-Based Smartphone Notification System. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17). Association for Computing Machinery, New York, NY, USA, 541–554. https://doi.org/10.1145/2998181.2998189

Digital Library

[68]

Veljko Pejovic and Mirco Musolesi. 2014. InterruptMe: Designing Intelligent Prompting Mechanisms for Pervasive Applications. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Seattle, Washington) (UbiComp ’14). Association for Computing Machinery, New York, NY, USA, 897–908. https://doi.org/10.1145/2632048.2632062

Digital Library

[69]

Martin Pielot, Linas Baltrunas, and Nuria Oliver. 2015. Boredom-Triggered Proactive Recommendations. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct(Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 1106–1110. https://doi.org/10.1145/2786567.2794340

Digital Library

[70]

Martin Pielot, Bruno Cardoso, Kleomenis Katevas, Joan Serrà, Aleksandar Matic, and Nuria Oliver. 2017. Beyond Interruptibility: Predicting Opportune Moments to Engage Mobile Phone Users. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (Sept. 2017), 91:1–91:25. https://doi.org/10.1145/3130956

Digital Library

[71]

Martin Pielot, Rodrigo de Oliveira, Haewoon Kwak, and Nuria Oliver. 2014. Didn’t You See My Message? Predicting Attentiveness to Mobile Instant Messages. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 3319–3328. https://doi.org/10.1145/2556288.2556973

Digital Library

[72]

Martin Pielot, Tilman Dingler, Jose San Pedro, and Nuria Oliver. 2015. When attention is not scarce - detecting boredom from mobile phone usage. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing(UbiComp ’15). Association for Computing Machinery, New York, NY, USA, 825–836. https://doi.org/10.1145/2750858.2804252

Digital Library

[73]

Benjamin Poppinga, Wilko Heuten, and Susanne Boll. 2014. Sensor-based identification of opportune moments for triggering notifications. IEEE Pervasive Computing 13, 1 (2014), 22–29.

Digital Library

[74]

Nilam Ram, Xiao Yang, Mu-Jung Cho, Miriam Brinberg, Fiona Muirhead, Byron Reeves, and Thomas N Robinson. 2020. Screenomics: A new approach for observing and studying individuals’ digital lives. Journal of adolescent research 35, 1 (2020), 16–50.

[75]

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779–788.

[76]

Byron Reeves, Nilam Ram, Thomas N Robinson, James J Cummings, C Lee Giles, Jennifer Pan, Agnese Chiatti, Mj Cho, Katie Roehrick, Xiao Yang, 2021. Screenomics: A framework to capture and analyze personal life experiences and the ways that technology shapes them. Human–Computer Interaction 36, 2 (2021), 150–201.

[77]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015), 91–99.

[78]

Hillol Sarker, Moushumi Sharmin, Amin Ahsan Ali, Md. Mahbubur Rahman, Rummana Bari, Syed Monowar Hossain, and Santosh Kumar. 2014. Assessing the Availability of Users to Engage in Just-in-Time Intervention in the Natural Environment. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Seattle, Washington) (UbiComp ’14). Association for Computing Machinery, New York, NY, USA, 909–920. https://doi.org/10.1145/2632048.2636082

Digital Library

[79]

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In IEEE International Conference on Computer Vision (ICCV).

[80]

Jeremiah Smith, Anna Lavygina, Jiefei Ma, Alessandra Russo, and Naranker Dulay. 2014. Learning to Recognise Disruptive Smartphone Notifications. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 121–124. https://doi.org/10.1145/2628363.2628404

Digital Library

[81]

Julian Steil, Philipp Müller, Yusuke Sugano, and Andreas Bulling. 2018. Forecasting user attention during everyday mobile interactions using device-integrated and wearable sensors. Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services (Sep 2018). https://doi.org/10.1145/3229434.3229439

Digital Library

[82]

Andriy A Struk, Jonathan SA Carriere, J Allan Cheyne, and James Danckert. 2017. A short boredom proneness scale: Development and psychometric properties. Assessment 24, 3 (2017), 346–359.

[83]

John C. Tang, Sophia B. Liu, Michael Muller, James Lin, and Clemens Drews. 2006. Unobtrusive but Invasive: Using Screen Recording to Collect Field Data on Computer-Mediated Interaction. In Proceedings of the 2006 20th Anniversary Conference on Computer Supported Cooperative Work (Banff, Alberta, Canada) (CSCW ’06). Association for Computing Machinery, New York, NY, USA, 479–482. https://doi.org/10.1145/1180875.1180948

Digital Library

[84]

Nađa Terzimehić, Luke Haliburton, Philipp Greiner, Albrecht Schmidt, Heinrich Hussmann, and Ville Mäkelä. 2022. MindPhone: Mindful Reflection at Unlock Can Reduce Absentminded Smartphone Use. In Designing Interactive Systems Conference. 1818–1830.

Digital Library

[85]

Naundefineda Terzimehić, Luke Haliburton, Philipp Greiner, Albrecht Schmidt, Heinrich Hussmann, and Ville Mäkelä. 2022. MindPhone: Mindful Reflection at Unlock Can Reduce Absentminded Smartphone Use. In Designing Interactive Systems Conference (Virtual Event, Australia) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1818–1830. https://doi.org/10.1145/3532106.3533575

Digital Library

[86]

Robert L Thorndike. 1953. Who belongs in the family. In Psychometrika. Citeseer.

[87]

Jonathan A. Tran, Katie S. Yang, Katie Davis, and Alexis Hiniker. 2019. Modeling the Engagement-Disengagement Cycle of Compulsive Phone Use. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300542

Digital Library

[88]

Liam D. Turner, Stuart M. Allen, and Roger M. Whitaker. 2017. Reachable but not receptive: Enhancing smartphone interruptibility prediction by modelling the extent of user engagement with notifications. Pervasive and Mobile Computing 40 (2017), 480–494. https://doi.org/10.1016/j.pmcj.2017.01.011

Digital Library

[89]

Niels van Berkel, Chu Luo, Theodoros Anagnostopoulos, Denzil Ferreira, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2016. A Systematic Assessment of Smartphone Usage Gaps. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 4711–4721. https://doi.org/10.1145/2858036.2858348

Digital Library

[90]

Steven Van Canneyt, Marc Bron, Andy Haines, and Mounia Lalmas. 2017. Describing Patterns and Disruptions in Large Scale Mobile App Usage Data. In Proceedings of the 26th International Conference on World Wide Web Companion (Perth, Australia) (WWW ’17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 1579–1584. https://doi.org/10.1145/3041021.3051113

Digital Library

[91]

Wijnand AP Van Tilburg and Eric R Igou. 2012. On boredom: Lack of challenge and meaning as distinct boredom experiences. Motivation and Emotion 36 (2012), 181–194.

[92]

Aku Visuri, Niels van Berkel, Chu Luo, Jorge Goncalves, Denzil Ferreira, and Vassilis Kostakos. 2017. Predicting Interruptibility for Manual Data Collection: A Cluster-Based User Model. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, Austria) (MobileHCI ’17). Association for Computing Machinery, New York, NY, USA, Article 12, 14 pages. https://doi.org/10.1145/3098279.3098532

Digital Library

[93]

Sara Alida Volkmer and Eva Lermer. 2019. Unhappy and addicted to your phone? – Higher mobile phone use is associated with lower well-being. Computers in Human Behavior 93 (2019), 210–218. https://doi.org/10.1016/j.chb.2018.12.015

Digital Library

[94]

Heli Väätäjä and Paul Egglestone. 2012. Briefing news reporting with mobile assignments: perceptions, needs and challenges. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work(CSCW ’12). Association for Computing Machinery, New York, NY, USA, 485–494. https://doi.org/10.1145/2145204.2145280

Digital Library

[95]

Dominik Weber, Alexandra Voit, Gisela Kollotzek, and Niels Henze. 2019. Annotif: A System for Annotating Mobile Notifcations in User Studies. In Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia(Pisa, Italy) (MUM ’19). Association for Computing Machinery, New York, NY, USA, Article 24, 12 pages. https://doi.org/10.1145/3365610.3365611

Digital Library

[96]

Thomas D. White, Gordon Fraser, and Guy J. Brown. 2019. Improving Random GUI Testing with Image-Based Widget Detection. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 307–317. https://doi.org/10.1145/3293882.3330551

Digital Library

[97]

Qiang Xu, Jeffrey Erman, Alexandre Gerber, Zhuoqing Mao, Jeffrey Pang, and Shobha Venkataraman. 2011. Identifying Diverse Usage Behaviors of Smartphone Apps. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference (Berlin, Germany) (IMC ’11). Association for Computing Machinery, New York, NY, USA, 329–344. https://doi.org/10.1145/2068816.2068847

Digital Library

[98]

Xiao Yang, Nilam Ram, Thomas Robinson, and Byron Reeves. 2019. Using Screenshots to Predict Task Switching on Smartphones. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6. https://doi.org/10.1145/3290607.3313089

Digital Library

[99]

Nalingna Yuan, Heidi M Weeks, Rosa Ball, Mark W Newman, Yung-Ju Chang, and Jenny S Radesky. 2019. How much do parents actually use their smartphones? Pilot study comparing self-report to passive sensing. Pediatric research 86, 4 (2019), 416–418.

[100]

Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445186

Digital Library

[101]

Sha Zhao, Julian Ramos, Jianrong Tao, Ziwen Jiang, Shijian Li, Zhaohui Wu, Gang Pan, and Anind K. Dey. 2016. Discovering Different Kinds of Smartphone Users through Their Application Usage Behaviors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp ’16). Association for Computing Machinery, New York, NY, USA, 498–509. https://doi.org/10.1145/2971648.2971696

Digital Library

[102]

Éilish Duke and Christian Montag. 2017. Smartphone addiction, daily interruptions and self-reported productivity. Addictive Behaviors Reports 6 (2017), 90–95. https://doi.org/10.1016/j.abrep.2017.07.002

Cited By

Lee YWu MChang CChang XChang Y(2024)Investigating User-perceived Impacts of Contextual Factors on Opportune MomentsProceedings of the ACM on Human-Computer Interaction10.1145/36765148:MHCI(1-28)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676514
Gao NYu ZXu YYu CWang YSalim FShi YKostakos VKay JHoang T(2024)Leveraging Large Language Models for Generating Mobile Sensing Strategies in Human Behavior ModelingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678423(729-735)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678423
Fang LZhang SJia HGoncalves JKostakos VKostakos VKay JHoang T(2024)ScreenTK: Seamless Detection of Time-Killing Moments Using Continuous Mobile Screen Text and On-Device LLMsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677547(196-200)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677547
Show More Cited By

Index Terms

Are You Killing Time? Predicting Smartphone Users’ Time-killing Moments via Fusion of Smartphone Sensor Data and Screenshots
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools
    2. Ubiquitous and mobile devices
      1. Smartphones

Recommendations

Smart smartphone development: iOS versus android
SIGCSE '11: Proceedings of the 42nd ACM technical symposium on Computer science education

In a remarkably short timeframe, developing apps for smartphones has gone from an arcane curiosity to an essential skill set. Employers are scrambling to find developers capable of transforming their ideas into apps. Educators interested in filling that ...
Characterizing Smartphone Usage Patterns from Millions of Android Users
IMC '15: Proceedings of the 2015 Internet Measurement Conference

he prevalence of smart devices has promoted the popular- ity of mobile applications (a.k.a. apps) in recent years. A number of interesting and important questions remain unan- swered, such as why a user likes/dislikes an app, how an app becomes popular ...
Pro Smartphone Cross-Platform Development: iPhone, Blackberry, Windows Mobile and Android Development and Distribution

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

14911 pages

ISBN:9781450394215

DOI:10.1145/3544548

Editors:
Albrecht Schmidt
LMU Munich, Germany60028717
,
Kaisa Väänänen
Tampere University, Finland60011170
,
Tesh Goyal
Google Research, USA60006191
,
Per Ola Kristensson
University of Cambridge, UK60031101
,
Anicia Peters
University of Namibia, Namibia60072704
,
Stefanie Mueller
Massachusetts Institute of Technology, USA60022195
,
Julie R. Williamson
University of Glasgow, UK60001490
,
Max L. Wilson
University of Nottingham, UK60015138

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science and Technology Council, ROC

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
1,063
Total Downloads

Downloads (Last 12 months)618
Downloads (Last 6 weeks)141

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lee YWu MChang CChang XChang Y(2024)Investigating User-perceived Impacts of Contextual Factors on Opportune MomentsProceedings of the ACM on Human-Computer Interaction10.1145/36765148:MHCI(1-28)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676514
Gao NYu ZXu YYu CWang YSalim FShi YKostakos VKay JHoang T(2024)Leveraging Large Language Models for Generating Mobile Sensing Strategies in Human Behavior ModelingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678423(729-735)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678423
Fang LZhang SJia HGoncalves JKostakos VKostakos VKay JHoang T(2024)ScreenTK: Seamless Detection of Time-Killing Moments Using Continuous Mobile Screen Text and On-Device LLMsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677547(196-200)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3677547
Zheng JGao G(2024)Fragmented Moments, Balanced Choices: How Do People Make Use of Their Waiting Time?Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642608(1-14)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642608
Lin YLin LWang CChang Y(2023)AdJustMoment: Customize Your Ad Watching ExperienceAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610700(103-107)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610700
Lin YSu LChen UWang PLee YChang Y(2023)Automatic, Manual, or Hybrid? A Preliminary Investigation of Users' Perception of Features for Supporting Notification ManagementAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610699(98-102)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594739.3610699

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Table of Conten