Skip to main content
SpringerLink
Log in

EEG Correlates of Voice and Face Emotional Judgments in the Human Brain

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

The purpose of this study is to clarify the neural correlates of human emotional judgment. This study aimed to induce a controlled perturbation in the emotional system of the brain by multimodal stimuli, and to investigate whether such emotional stimuli could induce reproducible and consistent changes in electroencephalography (EEG) signals. We exposed 12 subjects to auditory, visual, or combined audio–visual stimuli. Audio stimuli consisted of voice recordings of the Japanese word “arigato” (thank you) pronounced with three different intonations (angry—A, happy—H or neutral—N). Visual stimuli consisted of faces of women expressing the same emotional valences (A, H or N). Audio–visual stimuli were composed using either congruent combinations of faces and voices (e.g., H × H) or noncongruent combinations (e.g., A × H). The data were collected using an EEG system, and analysis was performed by computing the topographic distributions of EEG signals in the theta, alpha, and beta frequency ranges. We compared the conditions stimuli (A, H or N), and congruent versus noncongruent. Topographic maps of EEG power differed between those conditions. The obtained results demonstrate that EEG could be used as a tool to investigate emotional valence and discriminate various emotions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Aftanas LI, Lotova NV, Koshkarov VI, Makhnev VP, Mordvintsev YN, Popov SA. Non-linear dynamic complexity of the human EEG during evoked emotions. Int J Psychophysiol. 1998;28(1):63–76.

    Article  CAS  PubMed  Google Scholar 

  2. Biehl M, Matsumoto D, Ekman P, Hearn V, Heider K, Kudoh T, Veronica T. Matsumoto and Ekman’s Japanese and Caucasian facial expressions of emotion (JACFEE): reliability data and cross-national differences. J Nonverbal Behav. 1997;21(1):3–21.

    Article  Google Scholar 

  3. Barrett LF, Mesquita B, Ochsner KN, Gross JJ. The experience of emotion. Annu Rev Psychol. 2007;58:373–403.

    Article  PubMed Central  PubMed  Google Scholar 

  4. Chakrabarti B, Bullmore E, Baron-Cohen S. Empathizing with basic emotions: common and discrete neural substrates. Soc Neurosci. 2006;1(3–4):364–84.

    Article  PubMed  Google Scholar 

  5. Conrad NJ, Schmidt LA, Niccols A, Polak CP, Riniolo TC, Burack JA. Frontal electroencephalogram asymmetry during affective processing in children with Down syndrome: a pilot study. J Intellect Disabil Res. 2007;51(Pt 12):988–95.

    Article  CAS  PubMed  Google Scholar 

  6. de Gelder B, Morris JS, Dolan RJ. Unconscious fear influences emotional awareness of faces and voices. Proc Natl Acad Sci USA. 2005;102(51):18682–7.

    Article  PubMed Central  PubMed  Google Scholar 

  7. Dolan RJ, Morris JS, de Gelder B. Crossmodal binding of fear in voice and face. Proc Natl Acad Sci USA. 2001;98(17):10006–10.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  8. Doronbekov TK, Tokunaga H, Ikejiri Y, Kazui H, Hatta N, Masaki Y, Ogino A, Miyoshi N, Oku N, Nishikawa T, Takeda M. Neural basis of fear conditioning induced by video clip: positron emission tomography study. Psychiatry Clin Neurosci. 2005;59(2):155–62.

    Article  PubMed  Google Scholar 

  9. Duncan S, Barrett LF. Affect is a form of cognition: a neurobiological analysis. Cogn Emot. 2007;21:1184–211.

    Article  PubMed Central  PubMed  Google Scholar 

  10. Ekman P, Friesen WV. Manual for facial action coding system. Palo Alto: Consulting Psychologists; 1978.

    Google Scholar 

  11. Evans JS. Dual-processing accounts of reasoning, judgment, and social cognition. Annu Rev Psychol. 2008;59:255–78.

    Article  PubMed  Google Scholar 

  12. Gepperth AR, Rebhan S, Hasler S, Fritsch J. Biased competition in visual processing hierarchies: a learning approach using multiple cues. Cogn Comput. 2011;3(1):146–66.

    Article  Google Scholar 

  13. Giuliani NR, McRae K, Gross JJ. The up- and down-regulation of amusement: experiential, behavioral, and autonomic consequences. Emotion. 2008;8(5):714–9.

    Article  PubMed Central  PubMed  Google Scholar 

  14. Grandjean D, Scherer KR. Unpacking the cognitive architecture of emotion processes. Emotion. 2008;8(3):341–51.

    Article  PubMed  Google Scholar 

  15. Grimm S, Beck J, Schuepbach D, Hell D, Boesiger P, Bermpohl F, Niehaus L, Boeker H, Northoff G. Imbalance between left and right dorsolateral prefrontal cortex in major depression is linked to negative emotional judgment: an fMRI study in severe major depressive disorder. Biol Psychiatry. 2008;63(4):369–76.

    Article  PubMed  Google Scholar 

  16. Grimm S, Schmidt CF, Bermpohl F, Heinzel A, Dahlem Y, Wyss M. Segregated neural representation of distinct emotion dimensions in the prefrontal cortex—an fMRI study. NeuroImage. 2006;30:325–40.

    Article  PubMed  Google Scholar 

  17. Hoshi Y, Chen SJ. Regional cerebral blood flow changes associated with emotions in children. Pediatr Neurol. 2002;27(4):275–81.

    Article  PubMed  Google Scholar 

  18. Jack RE, Garrod OG, Yu H, Caldara R, Schyns PG. Facial expressions of emotion are not culturally universal. Proc Natl Acad Sci USA. 2012;109(19):7241–4.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  19. Keightley ML, Winocur G, Graham SJ, Mayberg HS, Hevenor SJ, Grady CL. An fMRI study investigating cognitive modulation of brain regions associated with emotional processing of visual stimuli. Neuropsychologia. 2003;41:585–96.

    Article  PubMed  Google Scholar 

  20. Kootstra G, de Boer B, Schomaker LR. Predicting eye fixations on complex visual stimuli using local symmetry. Cogn Comput. 2011;3(1):223–40.

    Article  Google Scholar 

  21. Lang PJ, Bradley MM, Cuthbert BN. Emotion and motivation: measuring affective perception. J Clin Neurophysiol. 1998;15(5):397–408.

    Article  CAS  PubMed  Google Scholar 

  22. Lange K, Williams LM, Young AW, Bullmore ET, Brammer MJ, Williams SCR. Task instructions modulate neural responses to fearful facial expressions. Biol Psychiatry. 2003;53:226–32.

    Article  PubMed  Google Scholar 

  23. Ledoux JE. Emotion circuits in the brain. Annu Rev Neurosci. 2000;23:155–84.

    Article  CAS  PubMed  Google Scholar 

  24. Li A, Dang J. A cross-cultural investigation on emotion expression under vocal and facial conflict—also an observation on emotional McGurk effect. In: International symposium on biomechanical and physiological modeling and speech science, Kanazawa, I. 2009; p. 37–50. Japan.

  25. Martin LL, Stepper S, Strack F. Inhibiting and facilitating conditions of the human smile: a nonobtrusive test of the facial feedback hypothesis. J Pers Soc Psychol. 1988;54(5):768–77.

    Article  PubMed  Google Scholar 

  26. McGurk H, MacDonald J. Hearing lips and seeing voices. Nature. 1976;264(5588):746–8.

    Article  CAS  PubMed  Google Scholar 

  27. Nakamura K, Kawashima R, Ito K, Sugiura M, Kato T, Nakamura A. Activation of the right inferior frontal cortex during assessment of facial emotion. J Neurophysiol. 1999;82:1610–4.

    CAS  PubMed  Google Scholar 

  28. Northoff G, Heinzel A, Bermpohl F, Niese R, Pfennig A, Pascual-Leone A, Schlaug G. Reciprocal modulation and attenuation in the prefrontal cortex: an fMRI study on emotional—cognitive interaction. Hum Brain Mapp. 2004;21:202–12.

    Article  PubMed  Google Scholar 

  29. Ochsner KN, Bunge SA, Gross JJ, Gabrieli JD. Rethinking feelings: an FMRI study of the cognitive regulation of emotion. J Cogn Neurosci. 2002;14:1215–29.

    Article  PubMed  Google Scholar 

  30. Onton J, Makeig S. High-frequency broadband modulations of electroencephalographic spectra. Front Hum Neurosci. 2009;23(3):61.

    Google Scholar 

  31. Rigoulot S, Pell MD. Seeing emotion with your ears: emotional prosody implicitly guides visual attention to faces. PLoS ONE. 2012;7(1):e30740.

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  32. Schachter S, Singer JE. Cognitive, social, and physiological determinants of emotional state. Psychol Rev. 1962;69(5):379–99.

    Article  CAS  PubMed  Google Scholar 

  33. Scherer KR. Appraisal theories. In: Dalgleish T, Power M, editors. Handbook of cognition and emotion. Chichester: Wiley; 1999. p. 637–63.

    Google Scholar 

  34. Scherer KR, Shorr A, Johnstone T. Appraisal processes in emotion: theory, methods, research. Canary, NC: Oxford University Press; 2001.

    Google Scholar 

  35. Tsoi DT, Lee KH, Gee KA, Holden KL, Parks RW, Woodruff PW. Humour experience in schizophrenia: relationship with executive dysfunction and psychosocial impairment. Psychol. 2008;38(6):801–10.

    Google Scholar 

  36. Tsuchiya N, Adolphs R. Emotion and consciousness. Trends Cogn Sci. 2007;11(4):158–67.

    Article  PubMed  Google Scholar 

  37. Van den Stock J, Grèzes J, de Geldera B. Human and animal sounds influence recognition of body language. Brain Res. 2008;1242(25):185–90.

    Article  PubMed  Google Scholar 

  38. van Brakel J. Emotions: a cross-cultural perspective on forms of life. Soc Perspect Emot. 1993;II:179–237.

    Google Scholar 

  39. van Damme S, Crombez G, Spence C. Is visual dominance modulated by the threat value of visual and auditory stimuli? Exp Brain Res. 2009;193(2):197–204.

    Article  PubMed  Google Scholar 

  40. van Hemmen JL, Sejnowski TJ. 23 problems in systems neuroscience. Oxford: Oxford University Press; 2006.

    Book  Google Scholar 

  41. Welch PD. The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans Audio Electroacoust. 1967;AU-15:70–3.

    Article  Google Scholar 

  42. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol. 1988;47:1063–70.

    Article  Google Scholar 

  43. Yanulevskaya V, Marsman JB, Cornelissen F, Geusebroek JM. An image statistics-based model for fixation prediction. Cogn Comput. 2011;3(1):94–104.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory for Advanced Brain Signal Processing, RIKEN Brain Science Institute, Wako-shi, Japan

    K. Hiyoshi-Taniguchi, T. Yokota, H. Bakardjian, A. Cichocki & F. B. Vialatte

  2. Human Brain Research Center, Kyoto University Graduate School of Medicine, Kyoto, Japan

    K. Hiyoshi-Taniguchi & H. Fukuyama

  3. RIKEN BSI-TOYOTA Collaboration Center, RIKEN, Wako-shi, Japan

    M. Kawasaki

  4. IM2A, Groupe Hospitalier Pitié-Salpêtrière, Paris, France

    H. Bakardjian

  5. Laboratoire SIGMA, ESPCI ParisTech, Paris, France

    F. B. Vialatte

Authors
  1. K. Hiyoshi-Taniguchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Kawasaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. T. Yokota
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. H. Bakardjian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. H. Fukuyama
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. A. Cichocki
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. F. B. Vialatte
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. B. Vialatte.

Additional information

A. Cichocki and F.B. Vialatte made equal contributions and should be considered as co-last authors of the present manuscript.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hiyoshi-Taniguchi, K., Kawasaki, M., Yokota, T. et al. EEG Correlates of Voice and Face Emotional Judgments in the Human Brain. Cogn Comput 7, 11–19 (2015). https://doi.org/10.1007/s12559-013-9225-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-013-9225-0

Keywords

Search

Navigation