Abstract
Passive mobile sensing for the purpose of human state modeling is a fast-growing area. It has been applied to solve a wide range of behavior-related problems, including physical and mental health monitoring, affective computing, activity recognition, routine modeling, etc. However, in spite of the emerging literature that has investigated a wide range of application scenarios, there is little work focusing on the lessons learned by researchers, and on guidance for researchers to this approach. How do researchers conduct these types of research studies? Is there any established common practice when applying mobile sensing across different application areas? What are the pain points and needs that they frequently encounter? Answering these questions is an important step in the maturing of this growing sub-field of ubiquitous computing, and can benefit a wide range of audiences. It can serve to educate researchers who have growing interests in this area but have little to no previous experience. Intermediate researchers may also find the results interesting and helpful for reference to improve their skills. Moreover, it can further shed light on the design guidelines for a future toolkit that could facilitate research processes being used. In this paper, we fill this gap and answer these questions by conducting semi-structured interviews with ten experienced researchers from four countries to understand their practices and pain points when conducting their research. Our results reveal a common pipeline that researchers have adopted, and identify major challenges that do not appear in published work but that researchers often encounter. Based on the results of our interviews, we discuss practical suggestions for novice researchers and high-level design principles for a toolkit that can accelerate passive mobile sensing research.

Similar content being viewed by others
References
Aledavood, T., Torous, J., Hoyos, A.M.T., Naslund, J.A., Onnela, J.P., Keshavan, M.: Smartphone-based tracking of sleep in depression, anxiety, and psychotic disorders. Curr. Psychiatry Rep. 21(7), 49 (2019)
Alexander, B., Karakas, K., Kohout, C., Sakarya, H., Singh, N., Stachtiaris, J., Barnes, L.E., Gerber, M.S.: A behavioral sensing system that promotes positive lifestyle changes and improves metabolic control among adults with type 2 diabetes. In: 2017 Systems and Information Engineering Design Symposium (SIEDS), IEEE, pp 283–288 (2017)
Althoff, T., Hicks, J.L., King, A.C., Delp, S.L., Leskovec, J., et al.: Large-scale physical activity data reveal worldwide activity inequality. Nature 547(7663), 336–339 (2017)
Bae, S., Ferreira, D., Suffoletto, B., Puyana, J.C., Kurtz, R., Chung, T., Dey, A.K.: Detecting drinking episodes in young adults using smartphone-based sensors. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(2), 5 (2017)
Bae, S., Chung, T., Ferreira, D., Dey, A.K., Suffoletto, B.: Mobile phone sensors and supervised machine learning to identify alcohol use events in young adults: Implications for just-in-time adaptive interventions. Addict. behav. 83, 42–47 (2018)
Banovic, N., Buzali, T., Chevalier, F., Mankoff, J., Dey, A.K.: Modeling and understanding human routine behavior. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, pp. 248–260 (2016)
Ben-Zeev, D., Wang, R., Abdullah, S., Brian, R., Scherer, E.A., Mistler, L.A., Hauser, M., Kane, J.M., Campbell, A., Choudhury, T.: Mobile behavioral sensing for outpatients and inpatients with schizophrenia. Psychiatr. Serv. 67(5), 558–561 (2016)
Biel, J.I., Martin, N., Labbe, D., Gatica-Perez, D.: Bites‘n’Bits: inferring eating behavior from contextual mobile data. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(4), 1–33 (2018). https://doi.org/10.1145/3161161
Bilogur, A.: Missingno: a missing data visualization suite. J. Open Source Softw. 3(22), 547 (2018)
Blunck, H., Bouvin, N.O., Franke, T., Grønbæk, K., Kjaergaard, M.B., Lukowicz, P., Wüstenberg, M.: On heterogeneity in mobile sensing applications aiming at representative data collection. In: Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, pp. 1087–1098 (2013)
Bogomolov, A., Lepri, B., Staiano, J., Oliver, N., Pianesi, F., Pentland, A.: Once upon a crime: Towards crime prediction from demographics and mobile data. In: Proceedings of the International Conference on Multimodal Interaction, pp. 427–434 (2014). https://doi.org/10.1145/2663204.2663254
Bogomolov, A., Lepri, B., Staiano, J., Letouzé, E., Oliver, N., Pianesi, F., Pentland, A.: Moves on the street: classifying crime hotspots using aggregated anonymized data on people dynamics. Big data 3(3), 148–158 (2015)
Bulling, A., Blanke, U., Schiele, B.: A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput. Surv. (CSUR) 46(3), 1–33 (2014)
Canzian, L., Musolesi, M.: Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, pp. 1293–1304 (2015)
Carreiro, S., Fang, H., Zhang, J., Wittbold, K., Weng, S., Mullins, R., Smelson, D., Boyer, E.W.: imstrong: deployment of a biosensor system to detect cocaine use. J. Med. Syst. 39(12), 186 (2015)
Carreiro, S., Wittbold, K., Indic, P., Fang, H., Zhang, J., Boyer, E.W.: Wearable biosensors to detect physiologic change during opioid use. J. Med. Toxicol. 12(3), 255–262 (2016)
Carter, S., Mankoff, J., Klemmer, S.R., Matthews, T.: Exiting the cleanroom: on ecological validity and ubiquitous computing. Hum. Comput. Interact. 23(1), 47–99 (2008)
Centellegher, S., Miritello, G., Villatoro, D., Parameshwar, D., Lepri, B., Oliver, N.: Mobile money: understanding and predicting its adoption and use in a developing economy. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 2(4) (2018). http://arxiv.org/abs/1812.03289
Chang, L., Lu, J., Wang, J.U., Chen, X., Fang, D., Tang, Z.O., Wang, J., Tang, Z., Nurmi, P., Wang, Z.: SleepGuard: capturing rich sleep information using smartwatch sensing data. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(98), 34 (2018). https://doi.org/10.1145/3264908
Chen, X., Wang, Y., He, J., Pan, S., Li, Y., Zhang, P.: CAP?: context-aware app usage prediction with heterogeneous graph embedding. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3(1), 4 (2019)
Chintha, K.K., Indic, P., Chapman, B., Boyer, E.W., Carreiro, S.: Wearable biosensors to evaluate recurrent opioid toxicity after naloxone administration: a hilbert transform approach. In: Proceedings of the Annual Hawaii International Conference on System Sciences. Annual Hawaii International Conference on System Sciences, NIH Public Access, vol. 2018, p. 3247 (2018)
Cho, H., Ippolito, D., Yu, Y.W.: Contact tracing mobile apps for covid-19: privacy considerations and related trade-offs (2020). arXiv preprint arXiv:200311511
Choudhury, T., Borriello, G., Consolvo, S., Haehnel, D., Harrison, B., Hemingway, B., Hightower, J., Koscher, K., LaMarca, A., Landay, J.A., et al.: The mobile sensing platform: an embedded activity recognition system. IEEE Pervas. Comput. 7(2), 32–41 (2008)
Deslandes, A., Moraes, H., Ferreira, C., Veiga, H., Silveira, H., Mouta, R., Pompeu, F.A., Coutinho, E.S.F., Laks, J.: Exercise and mental health: many reasons to move. Neuropsychobiology 59(4), 191–198 (2009)
Di Clemente, R., Luengo-Oroz, M., Travizano, M., Xu, S., Vaitla, B., González, M.C.: Sequence of purchases in credit card data reveal life styles in urban populations. Nat. Commun. 2018, 1–8 (2017). https://doi.org/10.1038/s41467-018-05690-8. arXiv:1703.00409
Doryab, A., Dey, A.K., Kao, G., Low, C.: Modeling biobehavioral rhythms with passive sensing in the wild: a case study to predict readmission risk after pancreatic surgery. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3(1), 8 (2019a)
Doryab, A., Villalba, D.K., Chikersal, P., Dutcher, J.M., Tumminia, M., Liu, X., Cohen, S., Creswell, K., Mankoff, J., Creswell, J.D., et al.: Identifying behavioral phenotypes of loneliness and social isolation with passive sensing: statistical analysis, data mining and machine learning of smartphone and fitbit data. JMIR mHealth uHealth 7(7), e13209 (2019b)
Fan, J., Fan, X., Tian, F., Li, Y., Liu, Z., Sun, W., Wang, H.: What is that in your hand? recognizing grasped objects via forearm electromyography sensing. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(4), 1–24 (2018)
Ferreira, E., Ferreira, D.: Towards altruistic data quality assessment for mobile sensing. In: Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers, pp. 464–469 (2017)
Ferreira, D., Kostakos, V., Dey, A.K.: Aware: mobile context instrumentation framework. Front. ICT 2, 6 (2015)
Gonzalez, M.C., Hidalgo, C.A., Barabasi, A.L.: Understanding individual human mobility patterns. Nature 453(7196), 779–782 (2008)
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. ACM SIGKDD Explor. Newslett. 11(1), 10–18 (2009)
Harari, G.M., Müller, S.R., Aung, M.S., Rentfrow, P.J.: Smartphone sensing methods for studying behavior in everyday life. Curr. Opin. Behav. Sci. 18, 83–90 (2017)
Hernández, N., Castro, L.A., Favela, J., Michán, L., Arnrich, B.: Data quality in mobile sensing datasets for pervasive healthcare. In: Handbook of Large-Scale Distributed Computing in Smart Healthcare. Springer, Berlin, pp. 217–238 (2017)
Imai, R., Tsubouchi, K., Konishi, T., Shimosaka, M.: Early destination prediction with spatio-temporal user behavior patterns. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(4), 1–19 (2018). https://doi.org/10.1145/3161197
Klemmer, S.R., Li, J., Lin, J., Landay, J.A.: Papier-mache: Toolkit support for tangible input. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Association for Computing Machinery, New York, NY, USA, CHI ’04, p 399–406. (2004). https://doi.org/10.1145/985692.985743
Lane, N.D., Xu, Y., Lu, H., Hu, S., Choudhury, T., Campbell, AT., Zhao, F.: Enabling large-scale human activity inference on smartphones using community similarity networks (csn). In: Proceedings of the 13th International Conference on Ubiquitous Computing, pp. 355–364 (2011)
Lane, N.D., Miluzzo, E., Lu, H., Peebles, D., Choudhury, T., Campbell, A.T.: A survey of mobile phone sensing. IEEE Commun. Mag. 48(9), 140–150 (2010)
Laport-López, F., Serrano, E., Bajo, J., Campbell, A.T.: A review of mobile sensing systems, applications, and opportunities. Knowl. Inf. Syst. 62(1), 145–174 (2020)
Lee, C., Lee, G.G.: Information gain and divergence-based feature selection for machine learning-based text categorization. Inf. Process. Manag. 42(1), 155–165 (2006)
Lu, J., Bi, J., Shang, C., Yue, C., Morillo, R., Ware, S., Kamath, J., Bamis, A., Russell, A., Wang, B.: Joint modeling of heterogeneous sensing data for depression assessment via multi-task learning. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(1), 1–21 (2018). https://doi.org/10.1145/3191753. arXiv:1505.02818v1
Madan, A., Cebrian, M., Moturu, S., Farrahi, K., Pentland, A.: Sensing the health state of a community. IEEE Pervas. Comput. 11(4), 36–45 (2012). https://doi.org/10.1109/MPRV.2011.79
Manek, A.S., Shenoy, P.D., Mohan, M.C., Venugopal, K.: Aspect term extraction for sentiment analysis in large movie reviews using gini index feature selection method and svm classifier. WWW 20(2), 135–154 (2017)
Matarazzo, T.J., Pakzad, S.N.: Structural identification for mobile sensing with missing observations. J. Eng. Mech. 142(5), 04016021 (2016)
Mazilu, S., Blanke, U., Calatroni, A., Gazit, E., Hausdorff, J.M., Tröster, G.: The role of wrist-mounted inertial sensors in detecting gait freeze episodes in Parkinson’s disease. Pervas. Mob. Comput. 33, 1–16 (2016)
Mehrotra, A., Tsapeli, F., Hendley, R., Musolesi, M.: Mytraces: investigating correlation and causation between users’ emotional states and mobile phone interaction. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(3), 83 (2017)
Migrating apps to android 9: android developers. (2019). https://developer.android.com/about/versions/pie/android-9.0-migration
Min, J.K., Doryab, A., Wiese, J., Amini, S., Zimmerman, J., Hong, J.I.: Toss’n’turn: smartphone as sleep and sleep quality detector. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp. 477–486 (2014)
Min, C., Lee, S., Lee, C., Lee, Y., Kang, S., Choi, S., Kim, W., Song, J.: Pada: Power-aware development assistant for mobile sensing applications. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, Association for Computing Machinery, New York, NY, USA, UbiComp ’16, pp. 946–957. (2016). https://doi.org/10.1145/2971648.2971676
Mirjafari, S., Masaba, K., Grover, T., Wang, W., Audia, P., Campbell, A.T., Chawla, N.V., Swain, V.D., Choudhury, M.D., Dey, A.K., D’Mello, S.K., Gao, G., Gregg, J.M., Jagannath, K., Jiang, K., Lin, S., Liu, Q., Mark, G., Martinez, G.J., Mattingly, S.M., Moskal, E., Mulukutla, R., Nepal, S., Nies, K., Reddy, M.D., Robles-Granda, P., Saha, K., Sirigiri, A., Striegel, A.: Differentiating higher and lower job performers in the workplace using mobile sensing. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3(2), 1–24 (2019). https://doi.org/10.1145/3328908
Morgan, W.P., Goldston, S.E.: Exercise and Mental Health. Taylor & Francis, US (1987)
Naughton, F., Hopewell, S., Lathia, N., Schalbroeck, R., Brown, C., Mascolo, C., McEwen, A., Sutton, S.: A context-sensing mobile phone app (q sense) for smoking cessation: a mixed-methods study. JMIR mHealth uHealth 4(3), e106 (2016)
Obuchi, M., Huckins, J.F., Wang, W., daSilva, A., Rogers, C., Murphy, E., Hedlund, E., Holtzheimer, P., Mirjafari, S., Campbell, A.: Predicting brain functional connectivity using mobile sensing. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 4(1), 1–22 (2020)
Oliver, N., Letouzé, E., Sterly, H., Delataille, S., De Nadai, M., Lepri, B., Lambiotte, R., Benjamins, R., Cattuto, C., Colizza, V., et al.: (2020) Mobile phone data and covid-19: missing an opportunity? arXiv preprint arXiv:200312347
Pedersen, A.B., Mikkelsen, E.M., Cronin-Fenton, D., Kristensen, N.R., Pham, T.M., Pedersen, L., Petersen, I.: Missing data and multiple imputation in clinical epidemiological research. Clin. Epidemiol. 9, 157 (2017)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Pérez-Torres, R., Torres-Huitzil, C., Galeana-Zapién, H.: Power management techniques in smartphone-based mobility sensing systems: a survey. Pervas. Mob. Comput. 31, 1–21 (2016)
Postolache, G., Postolache, O.: Smartphone sensing technologies for tailored parkinson’s disease diagnosis and monitoring. In: Mobile Solutions and Their Usefulness in Everyday Life. Springer, Berlin, pp. 251–273 (2019)
Qin, T., Shangguan, W., Song, G., Tang, J.: Spatio-temporal routine mining on mobile phone data. ACM TKDD 12(5), 56 (2018)
Reis, H.T.: Why researchers should think” real-world”: a conceptual rationale (2012)
Saeb, S., Zhang, M., Karr, C.J., Schueller, S.M., Corden, M.E., Kording, K.P., Mohr, D.C.: Mobile phone sensor correlates of depressive symptom severity in daily-life behavior: an exploratory study. J. Med. Internet Res. 17(7), e175 (2015)
Salekin, A., Eberle, J.W., Glenn, J.J., Teachman, B.A., Stankovic, J.A.: A weakly supervised learning framework for detecting social anxiety and depression. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(2), 26 (2018)
Sarda, A., Munuswamy, S., Sarda, S., Subramanian, V.: Using passive smartphone sensing for improved risk stratification of patients with depression and diabetes: cross-sectional observational study. JMIR mHealth uHealth 7(1), e11041 (2019)
Shoaib, M., Bosch, S., Scholten, H., Havinga, P.J., Incel, O.D.: Towards detection of bad habits by fusing smartphone and smartwatch sensors. In: 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), IEEE, pp. 591–596 (2015)
Srinivasan, V., Koehler, C., Jin, H.: RuleSelector: selecting conditional action rules from user behavior patterns. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(1), 1–34 (2018). https://doi.org/10.1145/3191767
Suhara, Y., Xu, Y., Pentland, A.: Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks. In: Proceedings of the 26th International Conference on World Wide Web, International World Wide Web Conferences Steering Committee, pp. 715–724 (2017)
Sun, X., Qiu, L., Wu, Y., Tang, Y., Cao, G.: SleepMonitor: monitoring respiratory rate and body position during sleep using smartwatch. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(3), 1–22 (2017). https://doi.org/10.1145/3130969
Trifan, A., Oliveira, M., Oliveira, J.L.: Passive sensing of health outcomes through smartphones: systematic review of current solutions and possible limitations. JMIR mHealth uHealth 7(8), e12649 (2019)
Vaizman, Y., Weibel, N., Lanckriet, G.: Context recognition in-the-wild: unified model for multi-modal sensors and multi-label classification. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(4), 1–22 (2017). https://doi.org/10.1145/3161192
Valero-Mora, P., Rodrigo, M.F., Sanchez, M., SanMartin, J.: A plot for the visualization of missing value patterns in multivariate data. Pract. Assess. Res. Eval. 24(1), 9 (2019)
Wang R., Aung, M.S., Abdullah, S., Brian, R., Campbell, A.T., Choudhury, T., Hauser, M., Kane, J., Merrill, M., Scherer, E.A., et al.: Crosscheck: toward passive sensing and detection of mental health changes in people with schizophrenia. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 886–897 (2016b)
Wang, R., Chen, F., Chen, Z., Li, T., Harari, G., Tignor, S., Zhou, X., Ben-Zeev, D., Campbell, A.T.: Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, pp. 3–14 (2014)
Wang, W., HARARI, G.M., Wang, R., Muller, S.R., Mirjafari, S., Masaba, K., Campbell, A.T.: Sensing behavioral change over time : using within-person variability features from mobile sensing to predict personality traits. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 2(3) (2018)
Wang, R., Harari, G., Hao, P., Zhou, X., Campbell, A.T.: Smartgpa: how smartphones can assess and predict academic performance of college students. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, pp. 295–306 (2015)
Wang, D., Weisz, J.D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., Gray, A.: Human-ai collaboration in data science: Exploring data scientists’ perceptions of automated ai. Proc. ACM Hum. Comput. Interact. 3(CSCW):1–24 (2019a)
Wang, L., Zhang, D., Wang, Y., Chen, C., Han, X., M’hamed, A.: Sparse mobile crowdsensing: challenges and opportunities. IEEE Commun. Mag. 54(7), 161–167 (2016a)
Wang, R., Wang, W., Aung, M.S., Ben-Zeev, D., Brian, R., Campbell, A.T., Choudhury, T., Hauser, M., Kane, J., Scherer, E.A., et al.: Predicting symptom trajectories of schizophrenia using mobile sensing. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(3), 110 (2017)
Wang, H., Li, Y., Zeng, S., Wang, G., Zhang, P., Hui, P., Jin, D.: Modeling spatio-temporal app usage for a large user population. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3(1), 1–23 (2019b). https://doi.org/10.1145/3314414
Xu, X., Chikersal, P., Doryab, A., Villalba, D.K., Dutcher, J.M., Tumminia, M.J., Althoff, T., Cohen, S., Creswell, K.G., Creswell, J.D., et al.: Leveraging routine behavior and contextually-filtered features for depression detection among college students. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 3(3), 1–33 (2019)
Yan, Z., Subbaraju, V., Chakraborty, D., Misra, A., Aberer, K.: Energy-efficient continuous activity recognition on mobile phones: an activity-adaptive approach. In: 2012 16th International Symposium on Wearable Computers, Ieee, pp. 17–24 (2012)
Zhang, Z.: Missing data imputation: focusing on single imputation. Ann. Transl. Med. 4(1) (2016)
Zhang, X., Li, W., Chen, X., Lu, S.: Moodexplorer: towards compound emotion detection via smartphone sensing. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1(4), 176 (2018)
Zhou, Y., De, S., Wang, W., Wang, R., Moessner, K.: Missing data estimation in mobile sensing environments. IEEE Access 6, 69869–69882 (2018)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Financial interests
The authors declare they have no financial interests.
Non-financial interests
Author Anind K. Dey is one of the editors-in-chief of “CCF Transactions on Pervasive Computing and Interaction”.
Rights and permissions
About this article
Cite this article
Xu, X., Mankoff, J. & Dey, A.K. Understanding practices and needs of researchers in human state modeling by passive mobile sensing. CCF Trans. Pervasive Comp. Interact. 3, 344–366 (2021). https://doi.org/10.1007/s42486-021-00072-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42486-021-00072-4