Tanmay Randhavane
Uttaran Bhattacharya
Pooja Kabra
ABSTRACT
We present a novel data-driven algorithm to learn the perceived emotions of individuals based on their walking motion or gaits. Given an RGB video of an individual walking, we extract their walking gait as a sequence of 3D poses. Our goal is to exploit the gait features to learn and model the emotional state of the individual into one of four categorical emotions: happy, sad, angry, or neutral. Our perceived emotion identification approach uses deep features learned using long short-term memory networks (LSTMs) on datasets with labeled emotive gaits. We combine these features with gait-based affective features consisting of posture and movement measures. Our algorithm identifies both the categorical emotions from the gaits and the corresponding values for the dimensional emotion components - valence and arousal. We also introduce and benchmark a dataset called Emotion Walk (EWalk), consisting of videos of gaits of individuals annotated with emotions. We show that our algorithm mapping the combined feature space to the perceived emotional state provides an accuracy of 80.07% on the EWalk dataset, outperforming the current baselines by an absolute 13–24%.
Supplemental Material
Available for Download
- 2018. CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.edu/.Google Scholar
- Danilo Avola, Marco Cascio, Luigi Cinque, Gian Luca Foresti, Cristiano Massaroni, and Emanuele Rodolà. 2019. 2d skeleton-based action recognition via two-branch stacked lstm-rnns. IEEE Transactions on Multimedia(2019).Google Scholar
- LF Barrett 2019. Emotional expressions reconsidered: challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest (2019).Google Scholar
- F C Benitez-Quiroz, R Srinivasan, 2016. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In CVPR.Google Scholar
- D Bernhardt and P Robinson. 2007. Detecting affect from non-stylised body motions. In ACII 2007. Springer, 59–70.Google ScholarDigital Library
- U Bhattacharya 2020a. STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence(AAAI’20). AAAI Press, 1342–1350.Google ScholarCross Ref
- U Bhattacharya 2020b. Take an Emotion Walk: Perceiving Emotions from Gaits Using Hierarchical Attention Pooling and Affective Mapping. In Proceedings of the European Conference on Computer Vision (ECCV).Google ScholarDigital Library
- J Butepage 2017. Deep representation learning for human motion prediction and classification. In Proceedings of the IEEE CVPR. 6158–6166.Google ScholarCross Ref
- A Camurri 2003. Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques. International journal of human-computer studies 59, 1-2 (2003), 213–225.Google ScholarDigital Library
- Z Cao, T Simon, 2017. Realtime 2D Pose Estimation using Part Affinity Fields. In CVPR.Google Scholar
- L Carli 1995. Nonverbal behavior, gender, and influence.Journal of personality and social psychology 68, 6(1995), 1030.Google Scholar
- S Choi 2021. MobileHumanPose: Toward Real-Time 3D Human Pose Estimation in Mobile Devices. In CVPR Workshops. 2328–2338.Google ScholarCross Ref
- A Crenn, A Khan, 2016. Body expression recognition from anim. 3D skeleton. In IC3D.Google Scholar
- A Crenn, A Meyer, 2017. Toward an efficient body expression recognition based on the synthesis of a neutral movement. In ICMI.Google Scholar
- R Dabral, A Mundhada, 2018. Learning 3d human pose from structure and motion. In ECCV.Google Scholar
- M Daoudi. 2017. Emotion recognition by body movement representation on the manifold of symmetric positive definite matrices. In ICIAP.Google Scholar
- P. Ekman. 1993. Facial Expression and Emotion. American Psychologist 48, 4 (1993), 384–392.Google ScholarCross Ref
- Melissa J Ferguson and John A Bargh. 2004. How social perception can automatically influence behavior. Trends in cognitive sciences 8, 1 (2004), 33–39.Google Scholar
- Joseph L Fleiss and Jacob Cohen. 1973. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement 33, 3 (1973), 613–619.Google Scholar
- J Forlizzi 2007. How interface agents affect interaction between humans and computers. In Proceedings of the 2007 conference on Designing pleasurable products and interfaces. ACM, 209–221.Google ScholarDigital Library
- I Goodfellow 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672–2680.Google Scholar
- K Greff, R K Srivastava, 2017. LSTM: A search space odyssey. IEEE transactions on neural networks and learning systems (2017).Google Scholar
- James J Gross. 2008. Emotion regulation. Handbook of emotions 3, 3 (2008), 497–513.Google Scholar
- M Gross 2012. Effort-shape and kinematic assessment of bodily expression of emotion during gait. Human Movement Science 31, 1 (2012), 202–221.Google ScholarCross Ref
- M Gross, E Crane, 2010. Methodology for assessing bodily expression of emotion. Journal of Nonverbal Behavior(2010).Google Scholar
- Hatice Gunes and Massimo Piccardi. 2008. Automatic temporal segment detection and affect recognition from face and body display. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 39, 1(2008), 64–84.Google ScholarDigital Library
- David J Hauser and Norbert Schwarz. 2016. Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior research methods 48, 1 (2016), 400–407.Google Scholar
- C Ionescu, D Papava, 2014. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. TPAMI (2014).Google Scholar
- M Karg 2013. Body movements for affective expression: A survey of automatic recognition and generation. IEEE Transactions on Affective Computing(2013).Google Scholar
- M Karg, K Kuhnlenz, 2010. Recognition of affect based on gait patterns. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) (2010).Google ScholarDigital Library
- ZH Kilimci 2019. Mood detection from physical and neurophysical data using deep learning models. Complexity 2019(2019).Google Scholar
- Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114(2013).Google Scholar
- A Kleinsmith 2011. Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics 41, 4(2011), 1027–1038.Google ScholarDigital Library
- A Kleinsmith, N Bianchi-Berthouze, 2013. Affective body expression perception and recognition: A survey. IEEE TAC (2013).Google ScholarDigital Library
- N Krämer 2016. Closing the gender gap in STEM with friendly male instructors? On the effects of rapport behavior and gender of a virtual agent in an instructional interaction. Computers & Education 99 (2016), 1–13.Google ScholarDigital Library
- J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174.Google Scholar
- B Li 2016. Identifying Emotions from Non-Contact Gaits Information Based on Microsoft Kinects. IEEE Transactions on Affective Computing 9, 4 (2016), 585–591.Google ScholarDigital Library
- T Lin, M Maire, 2014. Microsoft coco: Common objects in context. In ECCV.Google Scholar
- Y Luo, J Ren, 2018. LSTM Pose Machines. In CVPR.Google Scholar
- Y Ma 2006. A motion capture library for the study of identity, gender, and emotion perception from biological motion. Behavior research methods(2006).Google Scholar
- D Mehta 2017. Vnect: Real-time 3d human pose estimation with a single rgb camera. Acm transactions on graphics (tog) 36, 4 (2017), 1–14.Google ScholarDigital Library
- J Michalak, N Troje, 2009. Embodiment of sadness and depression-gait patterns associated with dysphoric mood. Psychosomatic Medicine(2009).Google Scholar
- J Mikels, B Fredrickson, 2005. Emotional category data on images from the International Affective Picture System. Behavior research methods(2005).Google Scholar
- T Mittal 2020. EmotiCon: Context-Aware Multimodal Emotion Recognition Using Frege’s Principle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14234–14243.Google ScholarCross Ref
- J Montepare, S Goldstein, 1987. The identification of emotions from gait information. Journal of Nonverbal Behavior(1987).Google Scholar
- S Narang, A Best, 2017. Motion recognition of self and others on realistic 3D avatars. Computer Animation & Virtual Worlds(2017).Google Scholar
- K M Naugle 2011. Emotional state affects the initiation of forward gait.Emotion 11, 2 (2011), 267.Google Scholar
- F Noroozi 2018. Survey on emotional body gesture recognition. IEEE transactions on affective computing(2018).Google Scholar
- L Omlor, M Giese, 2007. Extraction of spatio-temporal primitives of emotional body expressions. Neurocomputing (2007).Google Scholar
- S Piana 2013. A set of full-body movement features for emotion recognition to help children affected by autism spectrum condition. In IDGEI International Workshop.Google Scholar
- S Piana and Others. 2014. Real-time automatic emotion recognition from body gestures. arXiv preprint arXiv:1402.5047(2014).Google Scholar
- S Piana and Others. 2016. Adaptive body gesture representation for automatic emotion recognition. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 1(2016), 6.Google Scholar
- T Randhavane 2019. Learning perceived emotion using affective and deep features for mental health applications. In 2019 IEEE ISMAR-Adjunct. IEEE, 395–399.Google Scholar
- S Ribet 2019. Survey on Style in 3D Human Body Motion: Taxonomy, Data, Recognition and its Applications. IEEE Transactions on Affective Computing(2019).Google Scholar
- M. D. Robinson and G. L. Clore. 2002. Belief and feeling: Evidence for an accessibility model of emotional self-report. Psychological Bulletin(2002).Google Scholar
- C Roether, L Omlor, 2009b. Critical features for the perception of emotion from gait. Vision (2009).Google Scholar
- C L Roether 2009a. Features in the recognition of emotions from dynamic bodily expression. In Dynamics of Visual Motion Processing. Springer, 313–340.Google Scholar
- J Russell, J Bachorowski, 2003. Facial & vocal expressions of emotion. Rev. of Psychology (2003).Google Scholar
- Tomasz Sapiński, Dorota Kamińska, Adam Pelikant, and Gholamreza Anbarjafari. 2019. Emotion recognition from skeletal movements. Entropy 21, 7 (2019), 646.Google ScholarCross Ref
- N Savva 2012. Continuous recognition of player’s affective body expression as dynamic quality of aesthetic experience. IEEE Transactions on Computational Intelligence and AI in games 4, 3(2012), 199–212.Google ScholarCross Ref
- J Tilmanne 2012. Stylistic gait synthesis based on hidden Markov models. EURASIP Journal on advances in signal processing (2012).Google Scholar
- Joëlle Tilmanne and Thierry Dutoit. 2010. Expressive gait synthesis using PCA and Gaussian modeling. In MiG. Springer, 363–374.Google Scholar
- Silvan S Tomkins. 1984. Affect theory. Approaches to emotion 163, 163-195 (1984).Google Scholar
- G Venture, H Kadone, 2014. Recognizing emotions conveyed by human gait. International Journal of Social Robotics(2014).Google Scholar
- C Wang 2019. Learning Temporal and Bodily Attention in Protective Movement Behavior Detection. In 2019 8th ACIIW. IEEE, 324–330.Google Scholar
- S Xia, C Wang, 2015. Realtime style transfer unlabeled heterogeneous human motion. TOG (2015).Google Scholar
- S Yan 2018. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Thirty-Second AAAI Conference on Artificial Intelligence.Google ScholarCross Ref
- Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. 2018. Pose guided human video generation. In ECCV. 201–216.Google Scholar
Index Terms
- Learning Gait Emotions Using Affective and Deep Features
Recommendations
The Effect of Different Affective Arousal Levels on Taste Perception
ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal InteractionThe emotions we experience shape our perception, and our emotion is shaped by our perceptions. Taste perception is also influenced by emotions. Positive and negative emotions alter sweetness, sourness, and bitterness perception. However, most previous ...
Affective guidance of intelligent agents: How emotion controls cognition
How do emotions and moods color cognition? In this article, we examine how such reactions influence both judgments and cognitive performance. We argue that many affective influences are due, not to affective reactions themselves, but to the information ...
Affective computing with primary and secondary emotions in a virtual human
We introduce the WASABI ([W]ASABI [A]ffect [S]imulation for [A]gents with [B]elievable [I]nteractivity) Affect Simulation Architecture, in which a virtual human's cognitive reasoning capabilities are combined with simulated embodiment to achieve the ...
Comments