Abstract
Affect state of a person has an impact on the intellectual processes that control human behavior. Experiencing negative affect escalates mental problems, and experiencing positive affect states improve imaginative reasoning and thereby enhances one’s behavior and discipline. Hence, this work centers around affect acknowledgment from typing-based context data during the pandemic. In this paper, we present a novel sensing scheme that perceives one’s affect state from their unique contexts. We also aim to study how affect states vary in smartphone users during the pandemic. We collected data from 52 participants over 2 months with an Android application. We exploited the Circumplex Model of Affect (CMA) to infer 25 affect states, leveraging built-in motion and touch sensors on smartphones. We conducted comprehensive experiments by developing machine learning models to predict 25 states. Through our study, we observe that the states of users are heavily pertinent to one’s typing and motion contexts. A thorough evaluation shows that affect prediction model yields an F1-score of 0.90 utilizing diverse contexts. To the best of our knowledge, our work predicts the highest number of affect states (25 states) with better performance compared to state-of-the-art methods.







Similar content being viewed by others
Data availability
The dataset generated and analyzed during the current study are not publicly available due to the privacy concerns of the participants but the anonymized data are available from the corresponding author on reasonable request.
References
Gillan, C.M., Rutledge, R.B.: Smartphones and the neuroscience of mental health. Ann. Rev. Neurosci. 44, 129–151 (2021)
Lex, E., Kowald, D., Seitlinger, P., Tran, T.N.T., Felfernig, A., Schedl, M.: Psychology-Informed Recommender Systems. Now Publishers, Hanover (2021)
Sher, L.: The impact of the COVID-19 pandemic on suicide rates. QJM: Int. J. Med. 113(10), 707–712 (2020)
Rudokaite, D., Indriuniene, V.: Effectiveness of psychodrama for mitigating school fears among senior secondary school students. Zeitschrift für Psychodrama und Soziometrie 18(2), 369–385 (2019)
Huang, P.H.: Boost: improving mindfulness, thinking, and diversity. Wm. & Mary Bus. L. Rev. 10, 139 (2018)
Wu, W., Chen, L., Zhao, Y.: Personalizing recommendation diversity based on user personality. User Model. User Adapt. Interact. 28(3), 237–276 (2018)
Trisha, M., Guhan, P., Bhattacharya, U., Chandra, R., Bera, A., Manocha, D.: EmotiCon: context-aware multi-modal emotion recognition using Frege’s principle. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14234–14243 (2020)
Alexandros, Z., Khan, A., Kalogridis, G., Vatsikas, S., Lewis, T., Sooriyabandara, M.: HealthyOffice: mood recognition at work using smartphones and wearable sensors. In: 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops), pp. 1–6. IEEE (2016)
Díaz-García, A., González-Robles, A., Mor, S., Mira, A., Quero, S., García-Palacios, A., Botella, C.: Positive and Negative Affect Schedule (PANAS): psychometric properties of the online Spanish version in a clinical sample with emotional disorders. BMC Psychiatry 20(1), 1–13 (2020)
Russell, J.A.: A circumplex model of affect. J. Personal. Soc. Psychol. 39(6), 1161 (1980)
Samuel, O., Walker, G., Salmon, P., Filtness, A., Stevens, N., Mulvihill, C., Stanton, N.: Riding the emotional roller-coaster: using the circumplex model of affect to model motorcycle riders’ emotional state-changes at intersections. Transp. Res. Part F Traffic Psychol. Behav. 66, 139–150 (2019)
Stanisławski, K., Cieciuch, J., Strus, W.: Ellipse rather than a circumplex: a systematic test of various circumplexes of emotions. Personal. Individ. Diff. 181, 111052 (2021)
Girardi, D., Lanubile, F., Novielli, N.: Emotion detection using noninvasive low cost sensors. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 125–130. IEEE (2017)
LiKamWa, R., Liu, Y., Lane, N. D., Zhong, L.: Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services, pp. 389–402 (2013)
Evmenenko, A., Teixeira, D.S.: The circumplex model of affect in physical activity contexts: a systematic review. Int. J. Sport Exerc. Psychol. 20(1), 168–201 (2022)
Fernández-Caballero, A., Martínez-Rodrigo, A., Pastor, J.M., Castillo, J.C., Lozano-Monasor, E., López, M.T., Fernández-Sotos, A.: Smart environment architecture for emotion detection and regulation. J. Biomed. Inform. 64, 55–73 (2016)
Morshed, M.B., Saha, K., Li, R., D’Mello, S.K., De Choudhury, M., Abowd, G.D., Plötz, T.: Prediction of mood instability with passive sensing. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 3(3), 1–21 (2019)
Piskioulis, O., Tzafilkou, K., Economides, A.: Emotion detection through smartphone’s accelerometer and gyroscope sensors. In: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pp. 130–137 (2021)
Buda, T.S., Khwaja, M., Matic, A.: Outliers in smartphone sensor data reveal outliers in daily happiness. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 5(1), 1–19 (2021)
Wampfler, R., Klingler, S., Solenthaler, B., Schinazi, V. R., Gross, M.: Affective state prediction based on semi-supervised learning from smartphone touch data. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020). https://doi.org/10.1145/3313831.3376504
Hashmi, M.A., Riaz, Q., Zeeshan, M., Shahzad, M., Fraz, M.M.: Motion reveal emotions: identifying emotions from human walk using chest mounted smartphone. IEEE Sens. J. 20(22), 13511–13522 (2020)
Balducci, F., Impedovo, D., Macchiarulo, N., Pirlo, G.: Affective states recognition through touch dynamics. Multimed. Tools Appl. 79(47), 35909–35926 (2020)
Wang, P., Dong, L., Liu, W., Jing, N.: Clustering-based emotion recognition micro-service cloud framework for mobile computing. IEEE Access 8, 49695–49704 (2020)
Ruensuk, M., Oh, H., Cheon, E., Oakley, I., Hong, H.: Detecting negative emotions during social media use on smartphones. In: Proceedings of Asian CHI Symposium 2019: Emerging HCI Research Collection, pp. 73–79 (2019)
Tikadar, S., Bhattacharya, S.: A novel method to build and validate an affective state prediction model from touch-typing. In: IFIP Conference on Human-Computer Interaction, pp. 99–119. Springer, Cham (2019)
Ghandeharioun, A., McDuff, D., Czerwinski, M., Rowan, K.: Emma: an emotion-aware well-being chatbot. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1–7. IEEE (2019). https://doi.org/10.1109/ACII.2019.8925455
Ghosh, S., Sahu, S., Ganguly, N., Mitra, B., De, P.: EmoKey: an emotion-aware smartphone keyboard for mental health monitoring. In: 2019 11th International Conference on Communication Systems & Networks (COMSNETS), pp. 496–499. IEEE (2019). https://doi.org/10.1109/COMSNETS.2019.8711078
Zhang, X., Li, W., Chen, X., Lu, S.: Moodexplorer: towards compound emotion detection via smartphone sensing. Proc. ACM Interact. Mobile Wearable Ubiquitous Technol. 1(4), 1–30 (2018). https://doi.org/10.1145/3161414
Gloor, P.: Consistent excitement correlates with happiness-Predicting mood through body sensing with smartwatches. ICKN White Pap (2017)
Ma, Y., Xu, B., Bai, Y., Sun, G., Zhu, R.: Daily mood assessment based on mobile phone sensing. In: 2012 Ninth International Conference on Wearable and Implantable Body Sensor Networks, pp. 142–147. IEEE (2012) https://doi.org/10.1109/BSN.2012.3
Yang, N., Samuel, A.: Context-rich detection of user’s emotions using a smartphone. Microsoft Research Internship Report (2011)
Sultana, M., Al-Jefri, M., Lee, J.: Using machine learning and smartphone and smartwatch data to detect emotional states and transitions: exploratory study. JMIR mHealth uHealth 8(9), e17818 (2020)
Budner, P., Eirich, J., Gloor, P.A.: Making you happy makes me happy–Measuring Individual Mood with Smartwatches (2017) arXiv preprint arXiv:1711.06134
Cortes, C., Vapnik, V.: Mach. Learn. Support-vector networks 20(3), 273–297 (1995)
Ali, J., Khan, R., Ahmad, N., Maqsood, I.: Random forests and decision trees. Int. J. Comput. Sci. Issues (IJCSI) 9(5), 272 (2012)
Ghosh, S., Hiware, K., Ganguly, N., Mitra, B., De, P.: Emotion detection from touch interactions during text entry on smartphones. Int. J. Hum. Comput. Stud. 130, 47–57 (2019)
Salido Ortega, M.G., Rodríguez, L.F., Gutierrez-Garcia, J.O.: Towards emotion recognition from contextual information using machine learning. J. Amb. Intell. Humaniz. Comput. 11(8), 3187–3207 (2020). https://doi.org/10.1007/s12652-019-01485-x
https://gs.statcounter.com/os-market-share/mobile/worldwide. Accessed 3 Oct 2020
Breazeal, C.: Social robots for health applications. In: 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5368–5371. IEEE (2011)
Riva, G., Calvo, R A., Lisetti, C.: Cyberpsychology and affective computing. In: The Oxford Handbook of Affective Computing, pp 547–558 (2015)
Marshall, J., Wang, D.: Mood-sensitive truth discovery for reliable recommendation systems in social sensing. In: Proceedings of the 10th ACM conference on recommender systems, pp 167–174 (2016)
Álvarez, P., Zarazaga-Soria, F.J., Baldassarri, S.: Mobile music recommendations for runners based on location and emotions: the dj-running system. Pervasive Mobile Comput. 67, 101242 (2020)
Funding
The authors have not received any funding for the execution of this work.
Author information
Authors and Affiliations
Contributions
SJ and PV wrote, revised, and corrected the main manuscript. AS and SJ did data collection, experiments, and analysis. PV. and VGM supervised the conceptualization, ideation, and implementation of the work. All authors reviewed the manuscript
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no financial or proprietary interest in any material discussed in this article. The authors have no conflicts of interest to declare that are relevant to the content of this article.
Additional information
Communicated by B. Bao.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jacob, S., Vinod, P., Subramanian, A. et al. Affect sensing from smartphones through touch and motion contexts. Multimedia Systems 29, 2495–2509 (2023). https://doi.org/10.1007/s00530-023-01142-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00530-023-01142-6