Abstract
Human behaviors and the emotional states that they convey have been studied by psychologist and sociologists. The tracking of behaviors and emotions is becoming more pervasive with the advent of the Internet of Things (IoT), where small and always connected sensors can continuously capture information about human gestures, movements and postures. The captured information about readable behaviors conveys significant information that can be represented as time series. Few studies in emotion recognition and affective computing have explored the connection between the time series sensors data and the emotional behavior they conveys. In this paper, an innovative approach is proposed to study the emotions and behaviors connected to the time series data. A convolutional network augmented with attention-based bidirectional LSTM is introduced to represent the correlations between behaviors and emotions. The advantage of this model is that it can well recognized emotions by exploiting the data captured by sensors. The experimental results show that the proposed deep learning method outperforms separate schemes and achieves a high degree of accuracy for modelling human behaviors and emotions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473 (2014)
Biondi, G., Franzoni, V., Li, Y., Milani, A.: Web-based similarity for emotion recognition in web objects. In: Proceedings of the 9th International Conference on Utility and Cloud Computing, UCC 2016, New York, NY, USA, pp. 327–332. ACM (2016)
Cho, K., Courville, A.C., Bengio, Y.: Describing multimedia content using attention-based encoder-decoder networks. IEEE Trans. Multimedia 17(11), 1875–1886 (2015)
Coulson, M.: Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. J. Nonverbal Behav. 28(2), 117–139 (2004). https://doi.org/10.1023/B:JONB.0000023655.25550.be
Crane, E., Gross, M.: Motion capture and emotion: affect detection in whole body movement. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 95–101. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_9
De Gelder, B.: Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philos. Trans. Roy. Soc. B Biol. Sci. 364(1535), 3475–3484 (2009)
Deng, J., Leung, C., Li, Y.: Beyond big data of human behaviors: Modeling human behaviors and deep emotions. In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pp. 282–286, April 2018
Deng, J.J., Leung, C.H.: Dynamic time warping for music retrieval using time series modeling of musical emotions. IEEE Trans. Affect. Comput. 6(2), 137–151 (2015)
Deng, J.J., Leung, C.H., Milani, A., Chen, L.: Emotional states associated with music: classification, prediction of changes, and consideration in recommendation. ACM Trans. Interact. Intell. Syst. (TiiS) 5(1), 4 (2015)
Deng, J.J., Leung, C.H.C., Mengoni, P., Li, Y.: Emotion recognition from human behaviors using attention model. In: 2018 IEEE First International Conference on Artificial Intelligence and Knowledge Engineering (AIKE), pp. 249–253. IEEE (2018)
Dolan, R.J.: Emotion, cognition, and behavior. Science 298(5596), 1191–1194 (2002)
Franzoni, V., Li, Y., Mengoni, P.: A path-based model for emotion abstraction on facebook using sentiment analysis and taxonomy knowledge. In: Proceedings of the International Conference on Web Intelligence - WI 2017, pp. 947–952 (2017)
Gratch, J., Marsella, S.: A domain-independent framework for modeling emotion. Cogn. Syst. Res. 5(4), 269–306 (2004)
Harrigan, J.A.: Proxemics, kinesics, and gaze. In: Harrigan, J., Rosenthal, R., Scherer, K.R., Scherer, K. (eds.) The New Handbook of Methods in Nonverbal Behavior Research, pp. 137–198. Oxford University Press, Oxford (2005)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
Hicheur, H., Kadone, H., Grezes, J., Berthoz, A.: The combined role of motion-related cues and upper body posture for the expression of emotions during human walking. In: Mombau, K., Berns, K. (eds). Modeling, Simulation and Optimization of Bipedal Walking. Cognitive Systems Monographs, vol 18. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36368-9_6
Hudlicka, E.: Beyond cognition: Modeling emotion in cognitive architectures. In: ICCM, pp. 118–123 (2004)
zard, C.E., Malatesta, C.Z.: Perspectives on emotional development i: differential emotions theory of early emotional development. In: The first draft of this paper was based on an invited address to the Eastern Psychological Association, 1 April 1983. Wiley, Hoboken (1987)
Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1990)
Picard, R.W., et al.: Affective Computing. MIT Press, Cambridge (1995)
Roether, C.L., Omlor, L., Christensen, A., Giese, M.A.: Critical features for the perception of emotion from gait. J. Vis. 9(6), 15–15 (2009)
Thayer, R.E.: The Biopsychology of Mood and Arousal. Oxford University Press, Oxford (1990)
Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Zhang, Z., Song, Y., Cui, L., Liu, X., Zhu, T.: Emotion recognition based on customized smart bracelet with built-in accelerometer. PeerJ 4, e2258 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Deng, J.J., Leung, C.H.C. (2020). Deep Convolutional and Recurrent Neural Networks for Emotion Recognition from Human Behaviors. In: Gervasi, O., et al. Computational Science and Its Applications – ICCSA 2020. ICCSA 2020. Lecture Notes in Computer Science(), vol 12250. Springer, Cham. https://doi.org/10.1007/978-3-030-58802-1_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-58802-1_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58801-4
Online ISBN: 978-3-030-58802-1
eBook Packages: Computer ScienceComputer Science (R0)