Skip to main content
Springer Nature Link
Log in

The Amount of Information on Emotional States Conveyed by the Verbal and Nonverbal Channels: Some Perceptual Data

  • Chapter
Progress in Nonlinear Speech Processing

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 4391))

Abstract

In a face-to-face interaction, the addressee exploits both the verbal and nonverbal communication modes to infer the speaker’s emotional state. Is such an informational content redundant? Is the amount of information conveyed by each communication mode the same or is it different? How much information about the speaker’s emotional state is conveyed by each mode and is there a preferential red communication mode for a given emotional state? This work attempts to give an answer to the above questions evaluating the subjective perception of emotional states in the single (either visual or auditory channel) and the combined channels (visual and auditory). Results show that vocal expressions convey the same amount of information as the combined channels and that the video alone conveys poorer emotional information than the audio and the audio and video together. Interpretations of these results (that seem to not support the data reported in literature proving the dominance of the visual channel in the emotion’s perception) are given in terms of cognitive load, language expertise and dynamicity. Also, a mathematical model inspired to the information processing theory is hypothesized to support the suggested interpretations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Ali, S.M., Silvey, S.D.: A general class of coefficients of divergence of one distribution from another. Journal of Royal Statistic Society 28, 131–142 (1966)

    MATH  MathSciNet  Google Scholar 

  2. Apple, W., Hecht, K.: Speaking emotionally: The relation between verbal and vocal communication of affect. Journal of Personality and Social Psychology 42, 864–875 (1982)

    Article  Google Scholar 

  3. Argyle, M.: Bodily Communication. Methuen & Co., New York (1988)

    Google Scholar 

  4. Banse, R., Scherer, K.: Acoustic profiles in vocal emotion expression. Journal of Personality & Social Psychology 70(3), 614–636 (1996)

    Article  Google Scholar 

  5. Barnes, J.: Aristotele complete work (Revised Oxford translation in 2 volumes). Princeton University Press, Princeton (1984)

    Google Scholar 

  6. Bachorowski, J.A.: Vocal expression and perception of emotion. Current Directions in Psychological Science 8, 53–57 (1999)

    Article  Google Scholar 

  7. Birdwhistell, R.: Kinesies and context. University of Pennsylvania Press, Philadelphia (1970)

    Google Scholar 

  8. Breitenstein, C., Van Lancker, D., Daum, I.: The contribution of speech rate and pitch variation to the perception of vocal emotions in a German and an American sample. Cognition & Emotion 15, 57–79 (2001)

    Article  Google Scholar 

  9. Burgoon, J.K.: Nonverbal signals. In: Knapp, J.L., Miller, G.R. (eds.) Handbook of interpersonal communication, pp. 229–285. Sage, Thousand Oaks (1994)

    Google Scholar 

  10. Burleson, B.R.: Comforting messages: Features, functions, and outcomes. In: Daly, J.A., Wiemann, J.M. (eds.) Strategic interpersonal communication, pp. 135–161. Lawrence Erlbaum, Hillsdale (1994)

    Google Scholar 

  11. Burns, K.L., Beier, E.G.: Significance of vocal and visual channels in the decoding of emotional meaning. Journal of Communication 23, 118–130 (1973)

    Article  Google Scholar 

  12. Caccioppo, J.T., Klein, D.J., Bernston, G.C., Hatfield, E.: The psychophysiology of emotion. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 119–142. Guilford, New York (1993)

    Google Scholar 

  13. Caccioppo, J.T., Bush, L.K., Tassinary, L.G.: Microexpressive facial actions as a functions of affective stimuli: Replication and extension. Personality and Social Psychology Bullettin 18, 515–526 (1992)

    Article  Google Scholar 

  14. Corraze, G.: Les communications nonverbales. Presses Universitaires de France, Paris (1980)

    Google Scholar 

  15. Cosmides, L.: Invariances in the acoustic expressions of emotions during speech. Journal of Experimental Psycology, Human Perception Performance 9, 864–881 (1983)

    Article  Google Scholar 

  16. Darwin, C.: The expression of the emotions in man and the animals (1872), Reproduced by the University of Chicago, Chicago press (1965)

    Google Scholar 

  17. Davitz, J.R.: Auditory correlates of vocal expression of emotional feeling. In: Davitz, J.R. (ed.) The communication of emotional meaning, pp. 101–112. McGraw-Hill, New York (1964)

    Google Scholar 

  18. Davitz, J.: The communication of emotional meaning. McGraw-Hill, New York (1964)

    Google Scholar 

  19. Davitz, J.R.: Auditory correlates of vocal expression of emotional feeling. In: Davitz, J.R. (ed.) The communication of emotional meaning, pp. 101–112. McGraw-Hill, New York (1964)

    Google Scholar 

  20. Ekman, P., Friesen, W.V., Hager, J.C.: The facial action coding system, 2nd edn. Weidenfeld & Nicolson, London (2002)

    Google Scholar 

  21. Ekman, P.: Facial expression of emotion: New findings, new questions. Psychological Science 3, 34–38 (1992)

    Article  Google Scholar 

  22. Ekman, P.: An argument for basic emotions. Cognition and Emotion 6, 169–200 (1992)

    Article  Google Scholar 

  23. Ekman, P.: The argument and evidence about universals in facial expressions of emotion. In: Wagner, H., Manstead, A. (eds.) Handbook of social psychophysiology, pp. 143–164. Wiley, Chichester (1989)

    Google Scholar 

  24. Ekman, P.: Expression and the nature of emotion. In: Scherer, K., Ekman, P. (eds.) Approaches to emotion, pp. 319–343. Lawrence Erlbaum, Hillsdale (1984)

    Google Scholar 

  25. Ekman, P., Friesen, W.V.: Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  26. Ekman, P., Friesen, W.V.: Manual for the Facial Action Coding System. Consulting Psychologists Press, Palo Alto (1977)

    Google Scholar 

  27. Ekman, P., Friesen, W.V.: Head and body cues in the judgement of emotion: A reformulation. Perceptual Motor Skills 24, 711–724 (1967)

    Google Scholar 

  28. Frick, R.: Communicating emotions: the role of prosodic features. Psychological Bullettin 93, 412–429 (1985)

    Article  Google Scholar 

  29. Fridlund, A.J.: The new ethology of human facial expressions. In: Russell, J.A., Fernandez-Dols, J. (eds.) The psychology of facial expression, pp. 103–129. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  30. Fridlund, A.J.: Human facial expressions: An evolutionary view. CA Academic press, San Diego (1994)

    Google Scholar 

  31. Friend, M.: Developmental changes in sensitivity to vocal paralanguage. Developmental Science 3, 148–162 (2000)

    Article  Google Scholar 

  32. Frijda, N.H.: Moods, emotion episodes, and emotions. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 381–402. Guilford, New York (1993)

    Google Scholar 

  33. Frijda, N.H.: The emotions. Cambridge University Press, Cambridge (1986)

    Google Scholar 

  34. Fulcher, J.A.: Vocal affect expression as an indicator of affective response. Behavior Research Methods, Instruments & Computers 23, 306–313 (1991)

    Google Scholar 

  35. Gallager, R.G.: Information theory and reliable communication. John Wiley, Chichester (1968)

    MATH  Google Scholar 

  36. Goldin-Meadow, S.: Hearing gesture: How our hands help us think. The Belknap Press, Cambridge (2003)

    Google Scholar 

  37. Graham, J., Ricci-Bitti, P.E., Argyle, M.: A cross-cultural study of the communication of emotion by facial and gestural cues. Journal of Human Movement Studies 1, 68–77 (1975)

    Google Scholar 

  38. Haldane, E.L., Ross, G.R.: The philosophical work of Descartes. Dover, New York, current edition (1911)

    Google Scholar 

  39. Izard, C.E.: Innate and universal facial expressions: Evidence from developmental and cross-cultural research. Psychological Bulletin 115, 288–299 (1994)

    Article  Google Scholar 

  40. Izard, C.E.: Organizational and motivational functions of discrete emotions. In: Lewis, M., Haviland, J.M. (eds.) Handbook of emotions, pp. 631–641. Guilford, New York (1993)

    Google Scholar 

  41. Izard, C.E.: Basic emotions, relations among emotions, and emotion–cognition relations. Psychological Review 99, 561–565 (1992)

    Article  Google Scholar 

  42. Izard, C.E., Dougherty, L.M., Hembree, E.A.: A system for identifying affect expressions by holistic judgments. Unpublished manuscript. Available from Instructional Resource Center, University of Delaware (1983)

    Google Scholar 

  43. Izard, C.E.: The maximally discriminative facial movement coding system (MAX). Unpublished manuscript. Available from Instructional Resource Center, University of Delaware (1979)

    Google Scholar 

  44. James, W.: What is an Emotion? First published in Mind 9, 188–205 (1884), See http://psychclassics.asu.edu/James/emotion.htm

  45. Junqua, J.C.: The Lombard reflex and its role on human listeners and automatic speech recognizers. JASA 93(1), 510–524 (1993)

    Google Scholar 

  46. Kendon, A.: Gesture: Visible action as utterance. Cambridge University Press, Cambridge (2004)

    Google Scholar 

  47. Klasmeyer, G., Sendlmeier, W.F.: Objective voice parameters to characterize the emotional content in speech. In: Elenius, K., Branderudf, P. (eds.) Proceedings of ICPhS 1995, vol. 1, pp. 182–185. Arne Strömbergs Grafiska, Stockholm (1995)

    Google Scholar 

  48. Klinerberg, O.: Emotional expression in chinese literature. Journal of Abnormal and Social Psychology 33, 517–520 (1938)

    Article  Google Scholar 

  49. Klinnert, M.D., Campos, J.J., Sorce, J.F., Emde, R.N., Svejda, M.: Emotions as behaviour regulators: Social referencing in infancy. In: Plutchik, R., Kellerman, H. (eds.) Emotions: Theory, research, and experience, pp. 57–86. Academic Press, New York (1983)

    Google Scholar 

  50. La Barre, W.: The cultural basis of emotions and gestures. Journal of Personality 16, 49–68 (1947)

    Article  Google Scholar 

  51. Lumsden, C.J., Wilson, E.O.: Genes mind and culture: The co-evolutionary process. Harvard University Press, Cambridge (1981)

    Google Scholar 

  52. McNeill, D.: Gesture and thought. University of Chicago Press, Chicago (2005)

    Google Scholar 

  53. Mehrabian, A.: Orientation behaviors and non verbal attitude communication. Journal of Communication 17(4), 324–332 (1967)

    Article  Google Scholar 

  54. Millar, F.E., Rogers, L.E.: A relational approach to interpersonal communication. In: Miller, G.R. (ed.) Explorations in interpersonal communication, pp. 87–105. Sage, Beverly Hills (1976)

    Google Scholar 

  55. Miller, G.R.: Explorations in interpersonal communication. Sage, London (1976)

    Google Scholar 

  56. Mozziconacci, S.: Pitch variations and emotions in speech. In: Elenius, K., Branderudf, P. (eds.) Proceedings of ICPhS 1995, vol. 1, pp. 178–181. Arne Strömbergs Grafiska, Stockholm (1995)

    Google Scholar 

  57. Nushikyan, E.A.: Intonational universals in texual context. In: Elenius, K., Branderudf, P. (eds.) Proceedings of ICPhS 1995, vol. 1, pp. 258–261. Arne Strömbergs Grafiska, Stockholm (1995)

    Google Scholar 

  58. Oatley, K., Jenkins, J.M.: Understanding emotions. Blackwell, Oxford (1996)

    Google Scholar 

  59. Panksepp, J.: Affective neuroscience: A conceptual framework for the neurobiological study of emotions. In: Strongman, K. (ed.) International Review of Studies of Emotions, vol. 1, pp. 59–99. Wiley, Chichester (1991)

    Google Scholar 

  60. Plato: The Republic (375 BC). Penguin, Harmondworth, current edition (1955)

    Google Scholar 

  61. Plutchik, R.: Emotion and their vicissitudes: Emotions and Psychopatology. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 53–66. Guilford, New York (1993)

    Google Scholar 

  62. Plutchik, R.: The emotions. University Press of America, Lanham (1991)

    Google Scholar 

  63. Pittam, J., Scherer, K.R.: Vocal expression and communication of emotion. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 185–197. Guilford, New York (1993)

    Google Scholar 

  64. Ricouer, P.: The voluntary and the involuntary (Translation of Kohak, K. Original work 1950). Northwestern University Press, Evanston (1966)

    Google Scholar 

  65. Rosenberg, B.G., Langer, J.: A study of postural-gestural communication. Journal of Personality and Social Psychology 2(4), 593–597 (1965)

    Article  Google Scholar 

  66. Scherer, K.: Vocal communication of emotion: A review of research paradigms. Speech Communication 40, 227–256 (2003)

    Article  MATH  Google Scholar 

  67. Scherer, K.R., Banse, R., Wallbott, H.G.: Emotion inferences from vocal expression correlate across languages and cultures. Journal of Cross-Cultural Psychology 32, 76–92 (2001)

    Article  Google Scholar 

  68. Scherer, K.R., Banse, R., Wallbott, H.G., Goldbeck, T.: Vocal cues in emotion encoding and decoding. Motivation and Emotion 15, 123–148 (1991)

    Article  Google Scholar 

  69. Scherer, K.R.: Vocal correlates of emotional arousal and affective disturbance. In: Wagner, H., Manstead, A. (eds.) Handbook of social Psychophysiology, pp. 165–197. Wiley, New York (1989)

    Google Scholar 

  70. Scherer, K.R.: Experiencing Emotion: A Cross-cultural Study. Cambridge University Press, Cambridge (1982)

    Google Scholar 

  71. Scherer, K.R., Oshinsky, J.S.: Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion 1, 331–346 (1977)

    Article  Google Scholar 

  72. Schwartz, G.E., Weinberger, D.A., Singer, J.A.: Cardiovascular differentiation of happiness, sadness, anger, and fear following imagery and exercise. Psychosomatic Medicine 43, 343–364 (1981)

    Google Scholar 

  73. Schwartz, G.E., Ahern, G.L., Brown, S.: Lateralized facial muscle response to positive and negative emotional stimuli. Psychophysiology 16, 561–571 (1979)

    Article  Google Scholar 

  74. Schwartz, G.E., Fair, P.L., Salt, P., Mandel, M.R., Klerman, G.L.: Facial muscle patterning to affective imagery in depressed and non-depressed subjects. Science 192, 489–491 (1976)

    Article  Google Scholar 

  75. Shannon, C.E., Weaver, W.: Mathematical Theory of Communication. University of Illinois Press, Urbana (1949)

    MATH  Google Scholar 

  76. Sinanović, S., Johnson, D.H.: Toward a theory of information processing. Submitted to Signal Processing (2006), http://www-ece.rice.edu/~dhj/cv.html#publications

  77. Tomkins, S.S.: Affect, imagery, consciousness - vol. 1: The positive affects. Springer, New York (1962)

    Google Scholar 

  78. Tomkins, S.S.: Affect, imagery, consciousness - vol. 2: The negative affects. Springer, New York (1963)

    Google Scholar 

  79. Tomkins, S.S.: Affect theory. In: Scherer, K.R., Ekman, P. (eds.) Approaches to emotion, pp. 163–196. Erlbaum, Hillsdale (1984)

    Google Scholar 

  80. White, G.M.: Emotion inside out The anthropology of affect. In: Haviland, M., Lewis, J.M. (eds.) Handbook of Emotion, pp. 29–40. Guilford, New York (1993)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Psicologia, Seconda Università di Napoli, Via Vivaldi 43, 81100 Caserta, Italy

    Anna Esposito

  2. IIASS, Via Pellegrino 19, 84019, Vietri sul Mare, SA, Italy

    Anna Esposito

Authors
  1. Anna Esposito
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Yannis Stylianou Marcos Faundez-Zanuy Anna Esposito

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Berlin Heidelberg

About this chapter

Cite this chapter

Esposito, A. (2007). The Amount of Information on Emotional States Conveyed by the Verbal and Nonverbal Channels: Some Perceptual Data. In: Stylianou, Y., Faundez-Zanuy, M., Esposito, A. (eds) Progress in Nonlinear Speech Processing. Lecture Notes in Computer Science, vol 4391. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71505-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-71505-4_13

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-71503-0

  • Online ISBN: 978-3-540-71505-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics