skip to main content
research-article

MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing

Published: 08 January 2018 Publication History

Abstract

Social psychology and neuroscience had confirmed that emotion state exerts a significant effect on human communication, perception, social behavior and decision making. With the wide availability of smartphones equipped with microphone, accelerometer, GPS, and other source of sensors, it is worthwhile to explore the possibility of automatic emotion detection via smartphone sensing. Particularly, we focus on a novel research problem that tries to detect the compound emotion (a set of multiple dimensional basic emotions) of smartphone users. We observe that users' self-reported emotional states have high correlation with their smartphone usage patterns and sensing data. Based on the observations, we exploit a feature extraction and selection algorithm to find the most significant features. We further adopt a factor graph model to tackle the correlations between features and emotion labels, and propose a machine learning algorithm for compound emotion detection based on the smartphone sensing data. The proposed mechanism is implemented as an APP called MoodExplorer in Android platform. Extensive experiments conducted on the smartphone data collected from 30 university students show that MoodExplorer can recognize users' compound emotions with 76.0% exact match on average.

Supplementary Material

zhang (zhang.zip)
Supplemental movie, appendix, image and software files for, MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing

References

[1]
J. Ang, R. Dhillon, A. Krupski, E. Shriberg, and A. Stolcke. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In INTERSPEECH. Citeseer, 2002.
[2]
S. Aral and D. Walker. Identifying influential and susceptible members of social networks. Science, 337(6092):337--341, 2012.
[3]
A. B. Ashraf, S. Lucey, J. F. Cohn, T. Chen, Z. Ambadar, K. M. Prkachin, and P. E. Solomon. The painful face--pain expression recognition using active appearance models. Image and vision computing, 27(12):1788--1796, 2009.
[4]
X. Bao, S. Fan, A. Varshavsky, K. Li, and R. Roy Choudhury. Your reactions suggest you liked the movie: Automatic content rating via reaction sensing. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 197--206. ACM, 2013.
[5]
C. Beedie, P. Terry, and A. Lane. Distinctions between emotion and mood. Cognition 8 Emotion, 19(6):847--878, 2005.
[6]
A. Bogomolov, B. Lepri, M. Ferron, F. Pianesi, and A. S. Pentland. Daily stress recognition from mobile phone data, weather conditions and individual traits. In Proceedings of the 22nd ACM international conference on Multimedia, pages 477--486. ACM, 2014.
[7]
L. Canzian and M. Musolesi. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing, pages 1293--1304. ACM, 2015.
[8]
B. Cao, L. Zheng, C. Zhang, P. S. Yu, A. Piscitello, J. Zulueta, O. Ajilore, K. Ryan, and A. D. Leow. Deepmood: Modeling mobile phone typing dynamics for mood detection. 2017.
[9]
J. R. Crawford and J. D. Henry. The positive and negative affect schedule (panas): Construct validity, measurement properties and normative data in a large non-clinical sample. British Journal of Clinical Psychology, 43(3):245--265, 2004.
[10]
E. Diener, S. Oishi, and R. E. Lucas. Personality, culture, and subjective well-being: Emotional and cognitive evaluations of life. Annual review of psychology, 54(1):403--425, 2003.
[11]
S. Du, Y. Tao, and A. M. Martinez. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15):E1454--E1462, 2014.
[12]
P. Ekman, W. V. Friesen, M. O'Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W. A. LeCompte, T. Pitcairn, P. E. Ricci-Bitti, et al. Universals and cultural differences in the judgments of facial expressions of emotion. Journal of personality and social psychology, 53(4):712, 1987.
[13]
A. Exler, M. Urschel, A. Schankin, and M. Beigl. Smartphone-based detection of location changes using wifi data. In International Conference on Wireless Mobile Communication and Healthcare, pages 164--167. Springer, 2016.
[14]
J. H. Fowler, N. A. Christakis, et al. Dynamic spread of happiness in a large social network: longitudinal analysis over 20 years in the framingham heart study. British Medical Journal, 337:a2338, 2008.
[15]
B. J. Frey and D. J. MacKay. A revolution: Belief propagation in graphs with cycles. Advances in neural information processing systems, pages 479--485, 1998.
[16]
T. Geller. How do you feel?: Your computer knows. Communications of the ACM, 57(1):24--26, 2014.
[17]
A. Gluhak, M. Presser, L. Zhu, S. Esfandiyari, and S. Kupschick. Towards mood based mobile services and applications. In European Conference on Smart Sensing and Context, pages 159--174. Springer, 2007.
[18]
J. Golbeck, C. Robles, and K. Turner. Predicting personality with social media. In CHI‘11 extended abstracts on human factors in computing systems, pages 253--262. ACM, 2011.
[19]
J. J. Gross and O. P. John. Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. Journal of personality and social psychology, 85(2):348, 2003.
[20]
J. M. Hammersley and P. Clifford. Markov fields on finite graphs and lattices. 1971.
[21]
K. C. Herdem. Reactions: Twitter based mobile application for awareness of friends' emotions. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 796--797. ACM, 2012.
[22]
J. Hernandez, M. E. Hoque, W. Drevo, and R. W. Picard. Mood meter: counting smiles in the wild. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 301--310. ACM, 2012.
[23]
D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[24]
F. R. Kschischang, B. J. Frey, and H.-A. Loeliger. Factor graphs and the sum-product algorithm. IEEE Transactions on information theory, 47(2):498--519, 2001.
[25]
J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the eighteenth international conference on machine learning (ICML ‘01), volume 1, pages 282--289, 2001.
[26]
H. Leng, Y. Lin, and L. Zanzi. An experimental study on physiological parameters toward driver emotion recognition. In International Conference on Ergonomics and Health Aspects of Work with Computers, pages 237--246. Springer, 2007.
[27]
M. Levandowsky and D. Winter. Distance between sets. Nature, 234(5323):34--35, 1971.
[28]
S. Li, L. Huang, R. Wang, and G. Zhou. Sentence-level emotion classification with label and context dependence. Proceedings of ACL-2015, pages 1045--1053, 2013.
[29]
W. Li, Y. Hu, X. Fu, S. Lu, and D. Chen. Cooperative positioning and tracking in disruption tolerant networks. IEEE Transactions on Parallel and Distributed Systems, 26(2):382--391, 2015.
[30]
R. LiKamWa, Y. Liu, N. D. Lane, and L. Zhong. Moodscope: building a mood sensor from smartphone usage patterns. In Proceeding of the 11th annual international conference on Mobile systems, applications, and services, pages 389--402. ACM, 2013.
[31]
G. MacKerron and S. Mourato. Happiness is greater in natural environments. Global Environmental Change, 23(5):992--1000, 2013.
[32]
A. Mottelson and K. Hornbæk. An affect detection technique using mobile commodity sensors in the wild. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pages 781--792. ACM, 2016.
[33]
K. P. Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
[34]
R. Plutchik. A general psychoevolutionary theory of emotion. Theories of emotion, 1(3-31):4, 1980.
[35]
J. Posner, J. A. Russell, and B. S. Peterson. The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and psychopathology, 17(03):715--734, 2005.
[36]
R. Reisenzein, M. Studtmann, and G. Horstmann. Coherence between emotion and facial expression: Evidence from laboratory experiments. Emotion Review, 5(1):16--23, 2013.
[37]
S. Servia-Rodríguez, K. K. Rachuri, C. Mascolo, P. J. Rentfrow, N. Lathia, and G. M. Sandstrom. Mobile sensing at the service of mental well-being: a large-scale longitudinal study. In Proceedings of the 26th International Conference on World Wide Web, pages 103--112. International World Wide Web Conferences Steering Committee, 2017.
[38]
N. SpolaôR, E. A. Cherman, M. C. Monard, and H. D. Lee. A comparison of multi-label feature selection methods using the problem transformation approach. Electronic Notes in Theoretical Computer Science, 292:135--151, 2013.
[39]
N. Spolaôr, E. A. Cherman, M. C. Monard, and H. D. Lee. Relieff for multi-label feature selection. In Intelligent Systems (BRACIS), 2013 Brazilian Conference on, pages 6--11. IEEE, 2013.
[40]
S. A. Stansfeld and M. P. Matheson. Noise pollution: non-auditory effects on health. British medical bulletin, 68(1):243--257, 2003.
[41]
T. Stütz, T. Kowar, M. Kager, M. Tiefengrabner, M. Stuppner, J. Blechert, F. H. Wilhelm, and S. Ginzinger. Smartphone based stress prediction. In International Conference on User Modeling, Adaptation, and Personalization, pages 240--251. Springer, 2015.
[42]
Y. Suhara, Y. Xu, and A. Pentland. Deepmood: Forecasting depressed mood based on self-reported histories via recurrent neural networks. In Proceedings of the 26th International Conference on World Wide Web, pages 715--724. International World Wide Web Conferences Steering Committee, 2017.
[43]
B. Sun, Q. Ma, S. Zhang, K. Liu, and Y. Liu. iself: Towards cold-start emotion labeling using transfer learning with smartphones. In 2015 IEEE Conference on Computer Communications (INFOCOM), pages 1203--1211. IEEE, 2015.
[44]
C. Sutton and A. McCallum. An introduction to conditional random fields. arXiv preprint arXiv:1011.4088, 2010.
[45]
S. S. Tomkins. Affect, imagery, consciousness: Vol. i. the positive affects. 1962.
[46]
G. Tsoumakas, I. Katakis, and I. Vlahavas. Mining multi-label data. In Data mining and knowledge discovery handbook, pages 667--685. Springer, 2009.
[47]
L. Wasserman. All of statistics: a concise course in statistical inference. Springer Science 8 Business Media, 2013.
[48]
Y. Yang, J. Jia, B. Wu, and J. Tang. Social role-aware emotion contagion in image social networks. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
[49]
Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31(1):39--58, 2009.
[50]
M.-L. Zhang and Z.-H. Zhou. A review on multi-label learning algorithms. IEEE transactions on knowledge and data engineering, 26(8):1819--1837, 2014.
[51]
S. Zhang and P. Hui. A survey on mobile affective computing. ArXiv Prepr. ArXiv14101648, 2014.
[52]
Y. Zhang, J. Tang, J. Sun, Y. Chen, and J. Rao. Moodcast: Emotion prediction via dynamic continuous factor graph model. In Proceedings of the 10th International Conference on Data Mining, pages 1193--1198. IEEE, 2010.
[53]
M. Zhao, F. Adib, and D. Katabi. Emotion recognition using wireless signals. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (MobiCom ‘16), pages 95--108, 2016.
[54]
S. Zhao, H. Yao, and X. Jiang. Predicting continuous probability distribution of image emotions in valence-arousal space. In Proceedings of the 23rd ACM international conference on Multimedia, pages 879--882. ACM, 2015.
[55]
Y. Zhou, H. Xue, and X. Geng. Emotion distribution recognition from facial expressions. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1247--1250. ACM, 2015.

Cited By

View all
  • (2025)ScooterID: Posture-Based Continuous User Identification From Mobility Scooter RidesIEEE Transactions on Mobile Computing10.1109/TMC.2024.347360924:2(970-984)Online publication date: 1-Feb-2025
  • (2025)PUREmotion: Understanding the Impact of Highway Construction on People’s WellbeingSocial Networks Analysis and Mining10.1007/978-3-031-78548-1_5(48-57)Online publication date: 24-Jan-2025
  • (2024)Medusa3D: The Watchful Eye Freezing Illegitimate Users in Virtual Reality InteractionsProceedings of the ACM on Human-Computer Interaction10.1145/36765158:MHCI(1-21)Online publication date: 24-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 1, Issue 4
December 2017
1298 pages
EISSN:2474-9567
DOI:10.1145/3178157
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2018
Accepted: 01 October 2017
Revised: 01 August 2017
Received: 01 May 2017
Published in IMWUT Volume 1, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Compound emotion
  2. Emotion detection
  3. Factor graph
  4. Smartphone sensing

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)157
  • Downloads (Last 6 weeks)16
Reflects downloads up to 03 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)ScooterID: Posture-Based Continuous User Identification From Mobility Scooter RidesIEEE Transactions on Mobile Computing10.1109/TMC.2024.347360924:2(970-984)Online publication date: 1-Feb-2025
  • (2025)PUREmotion: Understanding the Impact of Highway Construction on People’s WellbeingSocial Networks Analysis and Mining10.1007/978-3-031-78548-1_5(48-57)Online publication date: 24-Jan-2025
  • (2024)Medusa3D: The Watchful Eye Freezing Illegitimate Users in Virtual Reality InteractionsProceedings of the ACM on Human-Computer Interaction10.1145/36765158:MHCI(1-21)Online publication date: 24-Sep-2024
  • (2024)Smartphone-based Human Behavior Task Modeling for Explainable Mental Health Detection ModelCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3679003(894-896)Online publication date: 5-Oct-2024
  • (2024)Leveraging Large Language Models for Generating Mobile Sensing Strategies in Human Behavior ModelingCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678423(729-735)Online publication date: 5-Oct-2024
  • (2024)Cross-content User Authentication in Virtual RealityProceedings of the 30th Annual International Conference on Mobile Computing and Networking10.1145/3636534.3696212(2098-2105)Online publication date: 4-Dec-2024
  • (2024)Where Do You Look When Unlocking Your Phone? : A Field Study of Gaze Behaviour During Smartphone UnlockExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3651094(1-7)Online publication date: 11-May-2024
  • (2024)Persuasive strategies and emotional states: towards designing personalized and emotion-adaptive persuasive systemsUser Modeling and User-Adapted Interaction10.1007/s11257-023-09390-x34:4(1175-1225)Online publication date: 1-Sep-2024
  • (2024)A Brief IntroductionIncentive Mechanism for Mobile Crowdsensing10.1007/978-981-99-6921-0_1(1-8)Online publication date: 4-Jan-2024
  • (2023)Human-computer interaction for virtual-real fusionJournal of Image and Graphics10.11834/jig.23002028:6(1513-1542)Online publication date: 2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media