Skip to main content
Springer Nature Link
Log in

Manual annotation and automatic image processing of multimodal emotional behaviors: validating the annotation of TV interviews

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

There have been a lot of psychological researches on emotion and nonverbal communication. Yet, these studies were based mostly on acted basic emotions. This paper explores how manual annotation and image processing can cooperate towards the representation of spontaneous emotional behavior in low-resolution videos from TV. We describe a corpus of TV interviews and the manual annotations that have been defined. We explain the image-processing algorithms that have been designed for the automatic estimation of movement quantity. Finally, we explore how image processing can be used for the validation of manual annotations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Abrilian S, Devillers L, Buisine S, Martin J-C (2005) EmoTV1: annotation of real-life emotions for the specification of multimodal affective interfaces. In: 11th international conference human–computer interaction (HCII’2005), Las Vegas, Nevada, USA, 22–27 July 2005 LEA

  2. Bakeman R, Gottman JM (1997) Observing interaction. An introduction to sequential analysis, 2nd edn. Cambridge University Press, Cambridge, 0 521 45008 X

  3. Banse R, Scherer K (1996) Acoustic profiles in vocal emotion expression. J Pers Soc Psychol 70(3):614–636

    Article  Google Scholar 

  4. Bänziger T, Pirker H, Scherer K (2006) GEMEP—Geneva multimodal emotion portrayals: a corpus for the study of multimodal emotional expressions. Workshop “Corpora for research on emotion and affect”. In: 5th international conference on language resources and evaluation (LREC’2006), Genova, Italy, 23rd May 2006, pp 15–19. http://www.limsi.fr/Individu/martin/tmp/LREC2006/WS-Emotion/LREC06–WSemotion-proceeding-12.pdf

  5. Boone RT, Cunningham JG (1998) Children’s decoding of emotion in expressive body movement: the development of cue attunement. Dev Psychol 34(5):1007–1016

    Article  Google Scholar 

  6. Caridakis G, Raouzaiou A, Karpouzis K, Kollias S (2006) Synthesizing gesture expressivity based on real sequences. Workshop “Multimodal Corpora. From Multimodal Behaviour Theories to Usable Models”. In: 5th international conference on language resources and evaluation (LREC’2006), Genova, Italy, 27th May 2006, pp 19–23. http://www.limsi.fr/Individu/martin/tmp/LREC2006/WS-Emotion/LREC06-WSemotion-proceeding-12.pdf

  7. Cowie R (2000) Emotional states expressed in speech. ISCA ITRW on speech and emotion: developing a conceptual framework for research, pp 224–231

  8. De Silva PR, Kleinsmith A, Bianchi-Berthouze N (2005) Towards unsupervised detection of affective body posture nuances. In: 1st international conference on affective computing and intelligent interaction (ACII’2005). Springer, Beijing, 22–24 October 2005, pp 32–40

  9. DeMeijer M (1989) The contribution of general features of body movement to the attribution of emotions. J Nonverbal Behav 13:247–268

    Article  Google Scholar 

  10. Devillers L, Abrilian S, Martin J-C (2005) Representing real life emotions in audiovisual data with non basic emotional patterns and context features. In: 1st international conference affective computing and intelligent interaction (ACII’2005), Beijing, China, Spinger, Berlin, 22–24 October 2005, pp 519–526. http://www.affectivecomputing.org/2005

  11. Douglas-Cowie E, Campbell N, Cowie R, Roach P (2003) Emotional speech; Towards a new generation of databases. Speech Commun 40:33–60

    Article  MATH  Google Scholar 

  12. Ekman P (1999) Basic emotions. Handbook of cognition and emotion. In: Dalgleish T, Power MJ (eds) Wiley, New York, pp 301–320

  13. Enos F, Hirschberg J (2006) A framework for eliciting emotional speech: capitalizing on the actor’s process. Workshop “Corpora for research on emotion and affect”. In: 5th international conference on language resources and evaluation (LREC’2006), Genova, Italy, 23 May 2006, pp 6–10. http://www.limsi.fr/Individu/martin/tmp/LREC2006/WS-Emotion/LREC06-WSemotion-proceeding-12.pdf

  14. Gunes H, Piccardi M (2005) Fusing face and body display for bi-modal emotion recognition: single frame analysis and multi-frame post integration. In: 1st international conference on affective computing and intelligent interaction (ACII’2005), Beijing, China. Springer, Heidelberg, 22–24 October 2005, pp 102–110

  15. Hartmann B, Mancini M, Pelachaud C (2005) Implementing expressive gesture synthesis for embodied conversational agents. Gesture Workshop (GW’2005), Vannes, France

  16. Kapur A, Kapur A, Virji-Babul N, Tzanetakis G, Driessen PF (2005) Gesture-based affective computing on motion capture data. In: 1st international conference on affective computing and intelligent interaction (ACII’2005), Springer, Beijing, China, 22–24 October 2005, pp 1–8

  17. Kipp M (2004). Gesture generation by imitation. From human behavior to computer character animation. Florida, Boca Raton, Dissertation.com. 1581122551. http://www.dfki.de/∼kipp/dissertation.html

  18. Martin J-C, Abrilian S, Devillers L (2005) Annotating multimodal behaviors occurring during non basic emotions. In: 1st international conference on affective computing and intelligent interaction (ACII’2005), Beijing, China, Spinger, Berlin, 22–24 October 2005, pp 550–557. http://www.affectivecomputing.org/2005

  19. Martin J-C, Abrilian S, Devillers L, Lamolle M, Mancini M, Pelachaud C (2005) Levels of representation in the annotation of emotion for the specification of expressivity in ECAs. In: 5th international working conference on intelligent virtual agents (IVA’2005), Kos, Greece. Springer, Heidelberg, 12–14 September 2005, pp 405–417. http://www.iva05.unipi.gr/

  20. Martin J-C, Caridakis G, Devillers L, Karpouzis K, Abrilian S (2006) Manual annotation and automatic image processing of multimodal emotional behaviors in tv interviews. In: 3rd IFIP conference on artificial intelligence applications & innovations (AIAI’2006), Athens, Greece, 7–9 June 2006

  21. McNeill D (1992) Hand and mind—what gestures reveal about thoughts. University of Chicago Press, Chicago, 0-226-56132-1

  22. Newlove J (1993) Laban for actors and dancers. Routledge, New York 1 85459 160 6

  23. Schröder M (2003) “Experimental study of affect burst.” speech communication. In: Special issue following the ISCA workshop on speech and emotion 40(1–2):99–116

  24. Wallbott HG (1998) Bodily expression of emotion. Eur J Soc Psychol 28:879–896. http://www3.interscience.wiley.com/cgi-bin/abstract/1863/ABSTRACT

    Article  Google Scholar 

Download references

Acknowledgment

This work was partly funded by the FP6 IST HUMAINE Network of Excellence (http://www.emotion-research.net).

Author information

Authors and Affiliations

  1. LIMSI-CNRS, BP 133, 91403, Orsay Cedex, France

    J. -C. Martin, L. Devillers & S. Abrilian

  2. Image, Video and Multimedia Systems Lab, National Technical, University of Athens, Iroon Polytechniou 9, 157 80, Athens, Greece

    G. Caridakis & K. Karpouzis

Authors
  1. J. -C. Martin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. G. Caridakis
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. L. Devillers
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. K. Karpouzis
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. S. Abrilian
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to J. -C. Martin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Martin, J.C., Caridakis, G., Devillers, L. et al. Manual annotation and automatic image processing of multimodal emotional behaviors: validating the annotation of TV interviews. Pers Ubiquit Comput 13, 69–76 (2009). https://doi.org/10.1007/s00779-007-0167-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-007-0167-y

Keywords