ABSTRACT
The ubiquity of smartphones and the widespread usage of text entry by soft keyboard in different instant messaging applications (e.g., WhatsApp, FB messenger) have opened the possibilities of inferring emotions from longitudinal typing data. To build this emotion inference engine, we apply machine learning models on features extracted from user’s typing patterns (not content). However, one major challenge encountered while developing the emotion inference model is the requirement of individual training data as typing patterns are often person-specific. In this paper, we investigate the possibility of combining typing pattern with emotion self-reporting to identify a group of similar users so that the training data among these users can be shared to fulfill the requirement of personalized dataset. We develop a framework AffectPro, which quantifies the typing interaction behavior (e.g., typing speed, error rate) and self-reporting pattern (e.g., emotion state transition probability) to construct the affective profiles of users. We evaluated AffectPro in a 6-week in-the-wild study involving 28 users, who used an Android application encompassing a custom keyboard to perform all their typing activities, and to report their instantaneous emotions. We extracted different typing signatures and self-report behavior details from the collected dataset (≈ 5000 typing sessions, ≈ 108 hours of typing data) to construct the affective profile of users. Our results demonstrate similarity across users in terms of typing signature, emotion self-reporting pattern, and a combination of both; which can be leveraged to share training data among similar users to overcome the challenges of personalized data collection.
- [1] Android IME 2022. https://developer.android.com/guide/topics/text/creating-input-method.html.Google Scholar
- Nabil Bin Hannan, Khalid Tearo, Joseph Malloch, and Derek Reilly. 2017. Once More, With Feeling: Expressing Emotional Intensity in Touchscreen Gestures. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 427–437.Google ScholarDigital Library
- Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 1293–1304.Google ScholarDigital Library
- Bokai Cao, Lei Zheng, Chenwei Zhang, Philip S Yu, Andrea Piscitello, John Zulueta, Olu Ajilore, Kelly Ryan, and Alex D Leow. 2017. Deepmood: modeling mobile phone typing dynamics for mood detection. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 747–755.Google ScholarDigital Library
- Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321–357.Google ScholarCross Ref
- Matteo Ciman and Katarzyna Wac. 2016. Individuals’ stress assessment using human-smartphone interaction analysis. IEEE Transactions on Affective Computing 9, 1 (2016), 51–65.Google ScholarCross Ref
- Christopher Ifeanyi Eke, Azah Anir Norman, Liyana Shuib, and Henry Friday Nweke. 2019. A survey of user profiling: State-of-the-art, challenges, and solutions. IEEE Access 7(2019), 144907–144924.Google ScholarCross Ref
- Clayton Epp, Michael Lippold, and Regan L Mandryk. 2011. Identifying emotional states using keystroke dynamics. In Proceedings of ACM SIGCHI.Google ScholarDigital Library
- Alberto Fernández, Salvador Garcia, Francisco Herrera, and Nitesh V Chawla. 2018. SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. Journal of artificial intelligence research 61 (2018), 863–905.Google ScholarCross Ref
- Surjya Ghosh, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2017. Evaluating effectiveness of smartphone typing as an indicator of user emotion. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 146–151.Google ScholarCross Ref
- Surjya Ghosh, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2017. Tapsense: Combining self-report patterns and typing characteristics for smartphone based emotion detection. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–12.Google ScholarDigital Library
- Surjya Ghosh, Kaustubh Hiware, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2019. Does emotion influence the use of auto-suggest during smartphone typing?. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 144–149.Google ScholarDigital Library
- Surjya Ghosh, Kaustubh Hiware, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2019. Emotion detection from touch interactions during text entry on smartphones. International Journal of Human-Computer Studies 130 (2019), 47–57.Google ScholarDigital Library
- Surjya Ghosh, Sumit Sahu, Niloy Ganguly, Bivas Mitra, and Pradipta De. 2019. EmoKey: An emotion-aware smartphone keyboard for mental health monitoring. In 2019 11th International Conference on Communication Systems & Networks (COMSNETS). IEEE, 496–499.Google ScholarCross Ref
- Chieh-Yang Huang, Tristan Labetoulle, Ting-Hao Kenneth Huang, Yi-Pei Chen, Hung-Chen Chen, Vallari Srivastava, and Lun-Wei Ku. 2017. Moodswipe: A soft keyboard that suggests messages based on user-specified emotions. arXiv preprint arXiv:1707.07191(2017).Google Scholar
- Tsvi Kuflik and Peretz Shoval. 2000. Generation of user profiles for information filtering—research agenda. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval. 313–315.Google Scholar
- Hosub Lee, Young Sang Choi, Sunjae Lee, and IP Park. 2012. Towards unobtrusive emotion recognition for affective social communication. In 2012 IEEE Consumer Communications and Networking Conference (CCNC). IEEE, 260–264.Google ScholarCross Ref
- Robert LiKamWa, Yunxin Liu, Nicholas D Lane, and Lin Zhong. 2013. Moodscope: Building a mood sensor from smartphone usage patterns. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services. 389–402.Google Scholar
- Hao Ma. 2014. On measuring social friend interest similarities in recommender systems. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. 465–474.Google ScholarDigital Library
- Iris B Mauss and Michael D Robinson. 2009. Measures of emotion: A review. Cognition and emotion 23, 2 (2009), 209–237.Google Scholar
- Patrick E McKight and Julius Najab. 2010. Kruskal-wallis test. The corsini encyclopedia of psychology(2010), 1–1.Google Scholar
- Patrick E McKnight and Julius Najab. 2010. Mann-Whitney U Test. The Corsini encyclopedia of psychology(2010), 1–1.Google Scholar
- Aske Mottelson and Kasper Hornbæk. 2016. An affect detection technique using mobile commodity sensors in the wild. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 781–792.Google ScholarDigital Library
- Fariba Noori and Mohammad Kazemifard. 2016. AUBUE: An Adaptive User-Interface Based on Users’ Emotions. Journal of Computing and Security 3, 2 (2016), 127–145.Google Scholar
- Massimiliano Orri, Jean-Baptiste Pingault, Alexandra Rouquette, Christophe Lalanne, Bruno Falissard, Catherine Herba, Sylvana M Côté, and Sylvie Berthoz. 2017. Identifying affective personality profiles: a latent profile analysis of the affective neuroscience personality scales. Scientific reports 7, 1 (2017), 1–14.Google Scholar
- Marco Polignano, Fedelucio Narducci, Marco de Gemmis, and Giovanni Semeraro. 2021. Towards Emotion-aware Recommender Systems: an Affective Coherence Model based on Emotion-driven Behaviors. Expert Systems with Applications 170 (2021), 114382.Google ScholarCross Ref
- Andrew Raij, Animikh Ghosh, Santosh Kumar, and Mani Srivastava. 2011. Privacy risks emerging from the adoption of innocuous wearable sensors in the mobile environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 11–20.Google ScholarDigital Library
- James A Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology 39, 6(1980), 1161–1178.Google ScholarCross Ref
- José A Sáez, Julián Luengo, Jerzy Stefanowski, and Francisco Herrera. 2015. SMOTE–IPF: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering. Information Sciences 291(2015), 184–203.Google ScholarDigital Library
- Yoshihiko Suhara, Yinzhan Xu, and Alex’Sandy’ Pentland. 2017. Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks. In Proceedings of the 26th International Conference on World Wide Web. 715–724.Google ScholarDigital Library
- Mark A Thornton and Diana I Tamir. 2017. Mental models accurately predict emotion transitions. Proceedings of the National Academy of Sciences 114, 23(2017), 5982–5987.Google ScholarCross Ref
- Liam D Turner, Stuart M Allen, and Roger M Whitaker. 2015. Push or delay? decomposing smartphone notification response behaviour. In Human Behavior Understanding. Springer, 69–83.Google Scholar
- Rafael Wampfler, Severin Klingler, Barbara Solenthaler, Victor R Schinazi, and Markus Gross. 2020. Affective State Prediction Based on Semi-Supervised Learning from Smartphone Touch Data. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
Index Terms
- AffectPro: Towards Constructing Affective Profile Combining Smartphone Typing Interaction and Emotion Self-reporting Pattern
Recommendations
TapSense: combining self-report patterns and typing characteristics for smartphone based emotion detection
MobileHCI '17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and ServicesTyping based communication applications on smartphones, like WhatsApp, can induce emotional exchanges. The effects of an emotion in one session of communication can persist across sessions. In this work, we attempt automatic emotion detection by jointly ...
Effect of affective profile on communication patterns and affective expressions in interactions with a dialog system
ACII'11: Proceedings of the 4th international conference on Affective computing and intelligent interaction - Volume Part IInterlocutors' affective profile and character traits play an important role in interactions. In the presented study, we apply a dialog system to investigate the effects of the affective profile on user-system communication patterns and users' ...
Does emotion influence the use of auto-suggest during smartphone typing?
IUI '19: Proceedings of the 24th International Conference on Intelligent User InterfacesTyping based interfaces are common across many mobile applications, especially messaging apps. To reduce the difficulty of typing using keyboard applications on smartphones, smartwatches with restricted space, several techniques, such as auto-complete, ...
Comments