Affiliations: Centre for Language Technology, University of Copenhagen, Njalsgade 140, build. 25 4th floor, 2300 Copenhagen S, Denmark. E-mail: [email protected] | [x] Laboratory of Speech Acoustics, Department of Telecommunications and Media-Informatics, Budapest University of Technology and Economics, Budapest, Hungary | [y] Dipartimento di Psicologia and IIASS, Seconda Università di Napoli, Vietri Sul Mare, Salerno, Italy
Abstract: This paper deals with multimodal behaviours in an annotated corpus of first encounters. More specifically, it presents a study aimed to determine common relations between facial expressions connected to emotions and co-speech. Emotions are described through emotion labels and bipolar values in three emotional dimensions, Pleasure, Arousal and Dominance, while the transcriptions of speech comprise words, quasi words, pauses, and filled pauses. The study establishes that there is a strong relation between specific communicative aspects, facial expressions and co-speech. We found that some emotions denoting facial expressions always co-occur with speech, and are often related to specific speech tokens, while other often occur uni-modally, that is without co-occurring speech. We have also observed large individual differences in the amount of facial expressions produced, but we did not find a correlation between the amount of speech tokens and facial expressions produced by the participants in the first encounters. Our study also confirms preceding research [31] suggesting that the most common emotions in the first encounters partly depend on the specific social activity. These findings are important for understanding human behaviours in face to face communications, but they also contribute to the construction and evaluation of corpus based models of plausible affective COGINFOCOM systems.
