Philip Schmidt
Kristof Van Laerhoven
ABSTRACT
Affective computing aims to detect a person's affective state (e.g. emotion) based on observables. The link between affective states and biophysical data, collected in lab settings, has been established successfully. However, the number of realistic studies targeting affect detection in the wild is still limited. In this paper we present an exploratory field study, using physiological data of 11 healthy subjects. We aim to classify arousal, State-Trait Anxiety Inventory (STAI), stress, and valence self-reports, utilizing feature-based and convolutional neural network (CNN) methods. In addition, we extend the CNNs to multi-task CNNs, classifying all labels of interest simultaneously. Comparing the F1 score averaged over the different tasks and classifiers the CNNs reach an 1.8% higher score than the classical methods. However, the F1 scores barely exceed 45%. In the light of these results, we discuss pitfalls and challenges for physiology-based affective computing in the wild.
- S. Bhattacharya and N. D. Lane. 2016. From smart to deep: Robust activity recognition on smartwatches using deep learning. In PerCom Workshops.Google Scholar
- Gordon H Bower. 1981. Mood and memory. Am Psychol (1981).Google Scholar
- B. Brenner, A.and Ehmke H. Burckhardt, G. and Draguhn, U. Eysel, and D. Singer. 2018. Physiologie.Google Scholar
- Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput Surv. (2014). Google ScholarDigital Library
- J. Choi, B. Ahmed, andR. Gutierrez-Osuna. 2012. Development and Evaluation of an Ambulatory Stress Monitor Based on Wearable Sensors. IEEE Trans. Inf. Technol. Biomed. (2012). Google ScholarDigital Library
- V. Ebrahimi Kahou, S. and Michalski, K. Konda, R. Memisevic, and C. Pal. 2015. Recurrent Neural Networks for Emotion Recognition in Video (ICMI '15). Google ScholarDigital Library
- A Edwards. 1957. The social desirability variable in personality assessment and research. (1957).Google Scholar
- M. Gjoreski, M. Luǎtrek, M. Gams, and H. Gjoreski. 2017. Monitoring stress with a wrist device using context. J. Biomed. Inform. (2017). Google ScholarDigital Library
- A. Grünerbl, A. Muaremi, V. Osmani, G. Bahle, and P. Lukowicz. 2015. Smartphone-Based Recognition of States and State Changes in Bipolar Disorder Patients. IEEE J BIOMED HEALTH (2015).Google Scholar
- N. Hammerla, S. Halloran, and T. Ploetz. 2016. Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables. arXiv preprint arXiv:1604.08880 (2016). Google ScholarDigital Library
- J. Hannink, T. Kautz, C. F. Pasluosta, K. Gaßmann, and B. M. Eskofier. 2017. Sensor-Based Gait Parameter Extraction With Deep Convolutional Neural Networks. IEEE J. Biomed. Health Inform. (2017).Google Scholar
- J. Healey, L. Nachman, S. Subramanian, J. Shahabdeen, and M. Morris. 2010. Out of the Lab and into the Fray: Towards Modeling Emotion in Everyday Life (Pervasive'10). Google ScholarDigital Library
- J. A. Healey and R. W. Picard. 2005. Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. (2005). Google ScholarDigital Library
- J. Hernandez, R. Morris, and R. W. Picard. 2011. Call Center Stress Recognition with Person-Specific Models (ACII'11). Google ScholarDigital Library
- S. Koelstra, C. Muhl, M. Soleymani, J. Lee, T. Ebrahimi, and I. Patras. 2012. Deap: A database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput (2012). Google ScholarDigital Library
- Sylvia D Kreibig. 2010. Autonomic nervous system activity in emotion: A review. Biol. Psychol. (2010).Google Scholar
- M. Lin, Q. Chen, and S. Yan. 2013. Network In Network. arXiv preprint arXiv:1312.4400.Google Scholar
- A. Mattila, J. Salminen, T. Nummi, and M. Joukamaa. 2006. Age is strongly associated with alexithymia in the general population. J Psychosom Res (2006).Google Scholar
- Jon D Morris. 1995. Observations: SAM: the Self-Assessment Manikin; an efficient cross-cultural measurement of emotional response. J Advert Res (1995).Google Scholar
- S. Münzner, P. Schmidt, A. Reiss, M. Hanselmann, R. Stiefelhagen, and R. Dürichen. 2017. CNN-based Sensor Fusion Techniques for Multimodal Human Activity Recognition (ISWC '17).Google Scholar
- F. Pedregosa, G. Varoquaux, A. Gramfort, E. Duchesnay, et al. 2011. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12 (2011). Google ScholarDigital Library
- R. Picard, E. Vyzas, and J. Healey. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. (2001). Google ScholarDigital Library
- K. Plarre, A. Raij, S. Hossain, M. Ali, A. Scott, et al. 2011. Continuous inference of psychological stress from sensory measurements collected in the natural environment (IPSN 2011).Google Scholar
- James A Russell. 1979. Affective space is bipolar. J Pers Soc Psychol (1979).Google Scholar
- A. Sano, A. Yu, A. McHill, A. Phillips, and R. Picard. 2015. Prediction of Happy-Sad mood from daily behaviors and previous sleep history. In Conf Proc IEEE Eng Med Biol Soc.Google Scholar
- P. Schmidt, A. Reiss, R. Dürichen, C. Marberger, and K. Van Laerhoven. 2018. Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection (ICMI '18). Google ScholarDigital Library
- P. Schmidt, A. Reiss, R. Dürichen, and K. Van Laerhoven. 2018. Labelling Affective States "in the Wild": Practical Guidelines and Lessons Learned (UbiComp '18). Google ScholarDigital Library
- P. Schmidt, A. Reiss, R. Dürichen, and K. Van Laerhoven. 2018. Wearable affect and stress recognition: A review. arXiv preprint arXiv:1811.08854 (2018).Google Scholar
- S. Schneegass, B. Pfleging, N. Broy, F. Heinrich, and A. Schmidt. 2013. A data set of real world driving to assess driver workload (AutomotiveUI '13). Google ScholarDigital Library
- C. Spielberger, R. Gorsuch, and R. Lushene. 1983. Manual for the state-trait anxiety inventory. (1983).Google Scholar
- S. A. Taylor, N. Jaques, E. Nosakhare, A. Sano, and R. Picard. 2018. Personalized Multitask Learning for Predicting Tomorrows Mood, Stress, and Health. IEEE Trans. Affect. Comput. (2018).Google Scholar
- X. Zhang, W. Li, X. Chen, and S. Lu. 2018. MoodExplorer: Towards Compound Emotion Detection via Smartphone Sensing. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (2018). Google ScholarDigital Library
- Yi Zheng, Qi Liu, Enhong Chen, Yong Ge, and J. Leon Zhao. 2016. Exploiting multi-channels deep convolutional neural networks for multivariate time series classification. Front. Comput. Sci. (2016). Google ScholarDigital Library
Index Terms
- Multi-target affect detection in the wild: an exploratory study
Recommendations
Contextual modulation of affect: Comparing humans and deep neural networks
ICMI '22 Companion: Companion Publication of the 2022 International Conference on Multimodal InteractionWhen inferring emotions, humans rely on a number of cues, including not only facial expressions, body posture, but also expressor-external, contextual information. The goal of the present study was to compare the impact of such contextual information ...
Affect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
This survey describes recent progress in the field of Affective Computing (AC), with a focus on affect detection. Although many AC researchers have traditionally attempted to remain agnostic to the different emotion theories proposed by psychologists, ...
Group emotion recognition in the wild by combining deep neural networks for facial expression classification and scene-context analysis
ICMI '17: Proceedings of the 19th ACM International Conference on Multimodal InteractionThis paper presents the implementation details of a proposed solution to the Emotion Recognition in the Wild 2017 Challenge, in the category of group-level emotion recognition. The objective of this sub-challenge is to classify a group's emotion as ...
Comments