Skip to main content
Springer Nature Link
Log in

Computational representation and analysis of emotion dynamics

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Human emotions are dynamic in nature. The intensity with which they are felt changes over time, and they have a natural timescale of expression, from onset to decay. Further, emotions shade from one to another, and many feelings are built up of blends of pure emotions. In order to represent this complex reality visually, a variety of models of the space of emotions have been introduced in the psychology literature. In this paper we present a computational approach to transforming facial expressions of emotions into positions in one of these models, the activation-evaluation space. This enables the study of emotion dynamics from facial expressions, including the transitions between emotion states and changes in emotion intensity through time, based on a set of shape models for different emotions. We consider different ways to build these shape models, and then show how to represent each frame of emotion in the activation-evaluation space, and analyse sequences of emotions from video by following temporal trajectories in that space. We demonstrate the approach on a standard dataset and compare it to human annotations, with promising results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Agbolade O, Nazri A, Yaakob R, Ghani A A, Cheah Y K (2019) 3-dimensional facial expression recognition in human using multi-points warping. BMC Bioinform 20(1):1–15

    Article  Google Scholar 

  2. Bansal A, Chaudhary S, Roy S D (2013) A novel LDA and HMM-based technique for emotion recognition from facial expressions. In: Multimodal pattern recognition of social signals in human-computer-interaction. Springer, pp 19–26

  3. Bartlett M S, Littlewort G, Lainscsek C, Fasel I, Movellan J (2004) Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: IEEE international conference on systems, man and cybernetics, vol 1. IEEE, pp 592–597

  4. Bindu M H, Gupta P, Tiwary US (2007) Cognitive model-based emotion recognition from facial expressions for live human computer interaction. In: 2007 IEEE symposium on computational intelligence in image and signal processing. IEEE, pp 351–356

  5. Blumberg S H, Izard C E (1991) Patterns of emotion experiences as predictors of facial expressions of emotion. Merrill-Palmer Quart (1982-):183–197

  6. Braathen B, Bartlett M S, Littlewort G, Smith E, Movellan J R (2002) An approach to automatic recognition of spontaneous facial actions. In: Proceedings fifth IEEE international conference on automatic face and gesture recognition. IEEE, pp 360–365

  7. Busso C, Bulut M, Lee C, Kazemzadeh A, Mower E, Kim S, Chang J N, Lee S, Narayanan S S (2008) IEMOCAP: Interactive emotional dyadic motion capture database. J Lang Resour Eval 4(42):335–359

    Article  Google Scholar 

  8. Cootes T F, Taylor C J, Cooper D H, Graham J (1995) Active shape models-their training and application. Comput Vis Image Understand 61 (1):38–59

    Article  Google Scholar 

  9. Costantini E, Pianesi F, Prete M (2005) Recognising emotions in human and synthetic faces: The role of the upper and lower parts of the face. In: Proceedings of the 10th international conference on intelligent user interfaces. ACM, pp 20–27

  10. D’Mello S (2012) Monitoring affective trajectories during complex learning. In: Encyclopedia of the sciences of learning. Springer, pp 2325–2328

  11. D’Mello S, Graesser A (2010) Modeling cognitive-affective dynamics with hidden markov models. In: Proceedings of the annual meeting of the cognitive science society, vol 32

  12. Ebrahimi Kahou S, Michalski V, Konda K, Memisevic R, Pal C (2015) Recurrent neural networks for emotion recognition in video. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 467–474

  13. Ekman P, Friesen W V (1978) Facial action coding system: A technique for the measurement of facial actions. Consulting Psychologists Press, Palo Alto

  14. Ekman P (1992) An argument for basic emotions. Cogn Emotion 6(3-4):169–200

    Article  Google Scholar 

  15. Ekman P (1999) Basic emotions. Handb Cogn Emotion 4:5–60

    Google Scholar 

  16. Ekman P (2007) Emotions revealed: Recognizing faces and feelings to improve communication and emotional life. Macmillan

  17. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using cnn-rnn and c3d hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction. ACM, pp 445–450

  18. Fisher N I (1995) Statistical analysis of circular data. Cambridge University Press

  19. Gu S, Wang F, Patel N P, Bourgeois J A, Huang J H (2019) A model for basic emotions using observations of behavior in drosophila. Front Psychol 10

  20. Guha T, Yang Z, Grossman R B, Narayanan S S (2018) A computational study of expressive facial dynamics in children with autism. IEEE Trans Affect Comput 9(1):14–20

    Article  Google Scholar 

  21. Gunes H, Piccardi M (2006) A bimodal face and body gesture database for automatic analysis of human nonverbal affective behavior. In: Pattern recognition, 2006. ICPR 2006. 18th international conference on, vol 1. IEEE, pp 1148–1153

  22. Gunes H, Piccardi M (2009) Automatic temporal segment detection and affect recognition from face and body display. IEEE Trans Syst Man Cybern Part B (Cybern) 39(1):64–84

    Article  Google Scholar 

  23. Gunes H, Schuller B, Pantic M, Cowie R (2011) Emotion representation, analysis and synthesis in continuous space: A survey. In: Automatic face & gesture recognition and workshops (FG 2011), 2011 IEEE international conference on. IEEE, pp 827–834

  24. Hakim A (2018) Predicting and improving the recognition of emotions. J Appl Comput Inf Technol 11(1):1–9

    Google Scholar 

  25. Hakim A, Marsland S, Guesgen H W (2012) A robust joint face model for human emotion recognition. In: Proceedings of the 27th conference on image and vision computing New Zealand. ACM, pp 352–357

  26. Hakim A, Marsland S, Guesgen H W (2013) Computational analysis of emotion dynamics. In: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, pp 185–190

  27. Hakim A, Marsland S, Guesgen H W (2013) Statistical modelling of complex emotions using mixture of von mises distributions. In: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, pp 517–522

  28. Hasani B, Mahoor M H (2017) Facial expression recognition using enhanced deep 3d convolutional neural networks. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. IEEE, pp 2278–2288

  29. Heylen J, Verduyn P, Van Mechelen I, Ceulemans E (2015) Variability in anger intensity profiles: Structure and predictive basis. Cogn Emotion 29 (1):168–177

    Article  Google Scholar 

  30. Hoeksma J B, Oosterlaan J, Schipper E, Koot H (2007) Finding the attractor of anger: Bridging the gap between dynamic concepts and empirical data. Emotion 7(3):638

    Article  Google Scholar 

  31. Hoque M E, McDuff D J, Picard R W (2012) Exploring temporal patterns in classifying frustrated and delighted smiles. IEEE Trans Affect Comput 3(3):323–334

    Article  Google Scholar 

  32. Huang Z (2015) An investigation of emotion changes from speech. In: Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on. IEEE, pp 733–736

  33. Huang Z, Epps J (2017) An investigation of emotion dynamics and kalman filtering for speech-based emotion prediction. Proceedings of Interspeech 2017, pp 3301–3305

  34. Hupont I, Cerezo E, Baldassarri S (2010) Sensing facial emotions in a continuous 2d affective space. In: Systems Man and Cybernetics (SMC), 2010 IEEE International Conference on. IEEE, pp 2045–2051

  35. Hutto D D, Robertson I, Kirchhoff M D (2018) A new, better bet: rescuing and revising basic emotion theory. Front Psychol 9

  36. Izard C E (1991) The psychology of emotions. Springer Science & Business Media

  37. Jenke R, Peer A (2018) A cognitive architecture for modeling emotion dynamics: intensity estimation from physiological signals. Cogn Syst Res 49:128–141

    Article  Google Scholar 

  38. Jin JM, Jjie Z S (1996) Computation of special functions. Wiley

  39. Karg M, Haug S, Kuhnlenz K, Buss M (2009) A dynamic model and system-theoretic analysis of affect based on a piecewise linear system. In: Robot and Human Interactive Communication, 2009. RO-MAN 2009. The 18th IEEE International Symposium on. IEEE, pp 238–244

  40. Kim D H, Baddar W, Jang J, Ro Y M (2017) Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput

  41. Kipp M (2001) ANVIL - A generic annotation tool for multimodal dialogue. In: Proceedings of the 7th European conference on speech communication and technology (Eurospeech), pp 1367–1370

  42. Kirkland T, Cunningham W A (2012) Mapping emotions through time: how affective trajectories inform the language of emotion. Emotion 12(2):268

    Article  Google Scholar 

  43. Krumhuber E G, Küster D, Namba S, Skora L (2020) Human and machine validation of 14 databases of dynamic facial expressions. Behav Res Methods:1–16

  44. Kuiper N H (1960) Tests concerning random points on a circle. In: Nederl. Akad. Wetensch. Proc. Ser. A, vol 63, pp 38–47

  45. Kuppens P, Verduyn P (2017) Emotion dynamics. Curr Opinion Psychol 17:22–26

    Article  Google Scholar 

  46. Mahoor M H, Cadavid S, S.Messinger D, Cohn J F (2009) A framework for automated measurement of the intensity of non-posed facial action units. In: IEEE computer society conference on computer vision and pattern recognition workshops, vol 1 & 2, pp 833–839

  47. Mardia K V (1975) Statistics of directional data. J R Stat Soc Ser B (Methodol):349–393

  48. Mariooryad S, Busso C (2012) Factorizing speaker, lexical and emotional variabilities observed in facial expressions. In: Image Processing (ICIP), 2012 19th IEEE International Conference on. IEEE, pp 2605–2608

  49. Martinez A M, Du S (2017) A model of the perception of facial expressions of emotion by humans: Research overview and perspectives. In: Gesture Recognition. Springer, pp 183–202

  50. Martins P, Batista J (2009) Identity and expression recognition on low dimensional manifolds.. In: ICIP, pp 3341–3344

  51. Moors A, Ellsworth P C, Scherer K R, Frijda N H (2013) Appraisal theories of emotion: State of the art and future development. Emot Rev 5(2):119–124

    Article  Google Scholar 

  52. Nguyen D, Nguyen K, Sridharan S, Ghasemi A, Dean D, Fookes C (2017) Deep spatio-temporal features for multimodal emotion recognition. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, pp 1215–1223

  53. Nicolaou M A, Gunes H, Pantic M (2012) Output-associative rvm regression for dimensional and continuous emotion prediction. Image Vis Comput 30 (3):186–196

    Article  Google Scholar 

  54. Oravecz Z, Tuerlinckx F, Vandekerckhove J (2011) A hierarchical latent stochastic differential equation model for affective dynamics. Psychol Methods 16(4):468

    Article  Google Scholar 

  55. Ortony A, Turner T J (1990) What’s basic about basic emotions. Psychol Rev 97(3):315–331

    Article  Google Scholar 

  56. Picard R W (1997) Affective computing. MIT Press, Cambridge

    Google Scholar 

  57. Picard R W, Klein J (2002) Computers that recognise and respond to user emotion: theoretical and practical implications. Interact Comput 14(2):141–169

    Article  Google Scholar 

  58. Qian B, Rasheed K (2004) Hurst exponent and financial market predictability. In: Proceedings of The 2nd IASTED international conference on financial engineering and applications, pp 203–209

  59. Résibois M, Kalokerinos E K, Verleysen G, Kuppens P, Van Mechelen I, Fossati P, Verduyn P (2018) The relation between rumination and temporal features of emotion intensity. Cogn Emotion 32(2):259–274

    Article  Google Scholar 

  60. Russell J A (1980) A circumplex model of affect. J Person Soc Psychol 39(6):1161

    Article  Google Scholar 

  61. Russell J A, Mehrabian A (1977) Evidence for a three-factor theory of emotions. J Res Person 11(3):273–294

    Article  Google Scholar 

  62. Scherer K R (2009) Emotions are emergent processes: they require a dynamic computational architecture. Philos Trans R Soc B: Biol Sci 364 (1535):3459–3474

    Article  Google Scholar 

  63. Schmidt K L, Cohn J F (2001) Human facial expressions as adaptations: Evolutionary questions in facial expression research. Amer J Phys Anthropol Official Publ Amer Assoc Phys Anthropol 116(S33):3–24

    Google Scholar 

  64. Senecal S, Cuel L, Aristidou A, Magnenat-Thalmann N (2016) Continuous body emotion recognition system during theater performances. Comput Animation Virtual Worlds 27(3-4):311–320

    Article  Google Scholar 

  65. Steephen J E (2013) Hed: A computational model of affective adaptation and emotion dynamics. IEEE Trans Affect Comput 4(2):197–210

    Article  Google Scholar 

  66. Terzopoulos D, Waters K (1990) Physically-based facial modelling, analysis, and animation. J Vis Comput Animation 1(2):73–80

    Article  Google Scholar 

  67. Tian Y-I, Kanade T, Cohn J F (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  68. Uddin M T, Canavan S (2020) Quantified facial temporal-expressiveness dynamics for affect analysis. arXiv:2010.14705

  69. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: Computer Vision and Pattern Recognition Workshop, 2006. CVPRW’06. Conference on. IEEE, pp 149–149

  70. Verduyn P, Van Mechelen I, Tuerlinckx F, Meers K, Van Coillie H (2009) Intensity profiles of emotional experience over time. Cogn Emotion 23 (7):1427–1443

    Article  Google Scholar 

  71. Watson G S (1961) Goodness-of-fit tests on a circle. Biometrika 48 (1/2):109–114

    Article  MathSciNet  MATH  Google Scholar 

  72. Whissell C M (1989) The dictionary of affect in language. In: The measurement of emotions. Elsevier, pp 113–131

  73. Wöllmer M, Metallinou A, Eyben F, Schuller B, Narayanan S (2010) Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. In: Proceedings of INTERSPEECH 2010, Makuhari, Japan, pp 2362–2365

  74. Yik M, Russell J A, Steiger J H (2011) A 12-point circumplex structure of core affect. Emotion 11(4):705

    Article  Google Scholar 

  75. Zhang C, Wang P, Chen K, Kämäräinen J-K (2017) Identity-aware convolutional neural networks for facial expression recognition. J Syst Eng Electron 28(4):784–792

    Article  Google Scholar 

  76. Zhou G, Zhan Y, Zhang J (2006) Facial expression recognition based on selective feature extraction. In: Sixth International Conference on Intelligent Systems Design and Applications, vol 2. IEEE, pp 412–417

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Muhammad Nawaz Shareef University of Agriculture Multan, Multan, Pakistan

    Ayesha Hakim

  2. School of Mathematics and Statistics, Victoria University of Wellington, Wellington, New Zealand

    Stephen Marsland

  3. School of Fundamental Sciences, Massey University, Massey, New Zealand

    Hans W. Guesgen

Authors
  1. Ayesha Hakim
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Stephen Marsland
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hans W. Guesgen
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ayesha Hakim.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hakim, A., Marsland, S. & Guesgen, H.W. Computational representation and analysis of emotion dynamics. Multimed Tools Appl 81, 21111–21133 (2022). https://doi.org/10.1007/s11042-022-12490-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12490-2

Keywords