Skip to main content
SpringerLink
Log in

The Perceptual and Cognitive Role of Visual and Auditory Channels in Conveying Emotional Information

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

In expressing emotional states, we generate a synaesthesic experience in our interlocutor when we transmit information about our feelings by the simultaneous use of several sensorial channels. These are referred to as verbal (the semantic content of the message) and non-verbal (gesture, gaze, tonal expression) modalities. From an engineering point of view the transmission of the same information by more than one sensorial channel should be considered as redundant. Is this true or does each channel transmit a specific piece of information? How much emotional information is transmitted by each channel and which plays a major role? As an attempt to address these questions, here we present a comparative analysis of the subjective perception of emotional states by the visual and auditory channels considered either singularly or in combination, always in the non-verbal modality. The results reveal that the audio and visual components of emotional messages convey much the same amount of information either separately or in combination, hence suggesting that each channel performs a robust encoding of the emotional features. Redundancy probably facilitates the recovery of emotional information in case one of the channels is impaired. This conclusion is challenged by language cultural specificity, since when tested on a foreign language the addressers rely more on visual information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. In the following it is assumed that “A gesture is a movement of the body, or any part of it, that is considered as expressive of thought or feeling” (Oxford English Dictionary).

  2. The results discussed do not consider all the range of emotional labels. Sarcasm/Irony is discarded because small children were involved in the experiments. Children were less skilled in the use of language lexicon and in perceiving subtle changes of prosodic features that characterize this emotion and therefore they may not have been able to identify it.

    Because of the difficulty to retrieve video-clips where the semantic meaning of the sentences expressed by the protagonists was not clearly expressing “disgust” at a moderate intensity level, also this emotion was discarded. The video clips retrieved were expressing too obvious expressions of disgust and therefore did not allow to verify the perceptual ability of the subjects to catch this emotional feeling.

  3. Considered reactions to goal-relevant changes in the environment and therefore associated to basic survival problems.

References

  1. Banse R, Scherer K. Acoustic profiles in vocal emotion expression. J Pers Soc Psychol. 1996;70(3):614–36. doi:10.1037/0022-3514.70.3.614.

    Article  PubMed  CAS  Google Scholar 

  2. Burns KL, Beier EG. Significance of vocal and visual channels in the decoding of emotional meaning. J Commun. 1973;23:118–30. doi:10.1111/j.1460-2466.1973.tb00936.x.

    Article  PubMed  CAS  Google Scholar 

  3. Caccioppo JT, Bush LK, Tassinary LG. Microexpressive facial actions as a functions of affective stimuli: replication and extension. Pers Soc Psychol Bull. 1992;18:515–26. doi:10.1177/0146167292185001.

    Article  Google Scholar 

  4. Cacioppo JT, Berntson GG, Larsen JT, Poehlmann KM, Ito TA. The psychophysiology of emotion. In: Lewis JM, Haviland-Jones M, editors. Handbook of emotions. 2nd ed. New York: Guilford Press; 2000. p. 173–91.

    Google Scholar 

  5. Ekman P. The argument and evidence about universals in facial expressions of emotion. In: Wagner H, Manstead A, editors. Handbook of social psychophysiology. Chichester: Wiley; 1989. p. 143–64.

    Google Scholar 

  6. Ekman P. Facial expression of emotion: new findings, new questions. Psychol Sci. 1992;3:34–8. doi:10.1111/j.1467-9280.1992.tb00253.x.

    Article  Google Scholar 

  7. Ekman P. An argument for basic emotions. Cogn Emot. 1992;6:169–200. doi:10.1080/02699939208411068.

    Article  Google Scholar 

  8. Ekman P, Friesen WV, Hager JC. The facial action coding system. 2nd ed. London: Weidenfeld & Nicolson; 2002. eBook.

    Google Scholar 

  9. Esposito A. The amount of information on emotional states conveyed by the verbal and nonverbal channels: some perceptual data. In: Stilianou Y, et al., editors. Progress in nonlinear speech processing. Lecture notes in computer science, vol. 4392. New York: Springer-Verlag; 2007. p. 245–64.

  10. Graham J, Ricci-Bitti PE, Argyle M. A cross-cultural study of the communication of emotion by facial and gestural cues. J Hum Mov Stud. 1975;1:68–77.

    Google Scholar 

  11. Izard CE. Basic emotions, relations among emotions, and emotion–cognition relations. Psychol Rev. 1992;99:561–5. doi:10.1037/0033-295X.99.3.561.

    Article  PubMed  CAS  Google Scholar 

  12. Izard CE. Innate and universal facial expressions: evidence from developmental and cross-cultural research. Psychol Bull. 1994;115:288–99. doi:10.1037/0033-2909.115.2.288.

    Article  PubMed  CAS  Google Scholar 

  13. Izard CE, Ackerman BP. Motivational, organizational, and regulatory functions of discrete emotions. In: Lewis JM, Haviland-Jones M, editors. Handbook of emotions. 2nd ed. New York: Guilford Press; 2000. p. 253–64.

    Google Scholar 

  14. Massaro DW. Perceiving talking faces. The MIT Press; 1998.

  15. Oatley K, Jenkins JM. Understanding emotions. 2nd ed. Oxford, England: Blackwell; 2000.

    Google Scholar 

  16. Plutchik R. Emotion and their vicissitudes: emotions and psychopathology. In: Lewis JM, Haviland-Jones M, editors. Handbook of emotions. New York: Guilford Press; 1993. p. 53–66.

    Google Scholar 

  17. Scherer K. Vocal communication of emotion: a review of research paradigms. Speech Commun. 2003;40:227–56. doi:10.1016/S0167-6393(02)00084-5.

    Article  Google Scholar 

  18. Scherer KR, Banse R, Wallbott HG, Goldbeck T. Vocal cues in emotion encoding and decoding. Motiv Emot. 1991;15:123–48. doi:10.1007/BF00995674.

    Article  Google Scholar 

  19. Scherer KR, Banse R, Wallbott HG. Emotion inferences from vocal expression correlate across languages and cultures. J Cross Cult Psychol. 2001;32:76–92. doi:10.1177/0022022101032001009.

    Article  Google Scholar 

  20. Schwartz GE, Fair PL, Salt P, Mandel MR, Klerman GL. Facial muscle patterning to affective imagery in depressed and non-depressed subjects. Science. 1976;192:489–91. doi:10.1126/science.1257786.

    Article  PubMed  CAS  Google Scholar 

  21. Schwartz GE, Ahern GL, Brown S. Lateralized facial muscle response to positive and negative emotional stimuli. Psychophysiology. 1979;16:561–71. doi:10.1111/j.1469-8986.1979.tb01521.x.

    Article  PubMed  CAS  Google Scholar 

  22. Schwartz GE, Weinberger DA, Singer JA. Cardiovascular differentiation of happiness, sadness, anger, and fear following imagery and exercise. Psychosom Med. 1981;43:343–64.

    PubMed  CAS  Google Scholar 

Download references

Acknowledgments

The paper has been partially supported by COST Action 2102: “Cross Modal Analysis of Verbal and Nonverbal Communication” (www.cost.esf.org/domains_actions/ict/Actions/Verbal_and_Non-verbal_Communication) and by Regione Campania, L.R. N.5 del 28.03.2002, Project ID N. BRC1293, Feb. 2006. Deep gratitude goes to Professor Roberto Ligrone for his useful comments and suggestions. Tina Marcella Nappi is also acknowledged for her editorial help.

Author information

Authors and Affiliations

  1. Dipartimento di Psicologia, Seconda Università di Napoli, Via Vivaldi 43, 81100, Caserta, Italy

    Anna Esposito

  2. IIASS, Via Pellegrino 19, 84019, Vietri sul Mare, SA, Italy

    Anna Esposito

Authors
  1. Anna Esposito
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna Esposito.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Esposito, A. The Perceptual and Cognitive Role of Visual and Auditory Channels in Conveying Emotional Information. Cogn Comput 1, 268–278 (2009). https://doi.org/10.1007/s12559-009-9017-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-009-9017-8

Keywords

Search

Navigation