Abstract
In the field of Embodied Conversational Agent (ECA) one of the main challenges is to generate socially believable agents. The long run objective of the present study is to infer rules for the multimodal generation of agents’ socio-emotional behaviour. In this paper, we introduce the Social Multimodal Association Rules with Timing (SMART) algorithm. It proposes to learn the rules from the analysis of a multimodal corpus composed by audio-video recordings of human-human interactions. The proposed methodology consists in applying a Sequence Mining algorithm using automatically extracted Social Signals such as prosody, head movements and facial muscles activation as an input. This allows us to infer Temporal Association Rules for the behaviour generation. We show that this method can automatically compute Temporal Association Rules coherent with prior results found in the literature especially in the psychology and sociology fields. The results of a perceptive evaluation confirms the ability of a Temporal Association Rules based agent to express a specific stance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Truong, K., Heylen, D., Chetouani, M., Mutlu, B., Salah, A.A.: Workshop on emotion representations and modelling for companion systems. In: ERM4CT@ICMI (2015)
Vinciarelli, A., Pantic, M., Bourlard, H.: Social signal processing: survey of an emerging domain. Image Vision Comput. 27, 1743–1759 (2009)
Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., Schröder, M.: Bridging the gap between social animal, unsocial machine: a survey of social signal processing. Affect. Comput. 3, 69–87 (2012)
Rudovic, O., Nicolaou, M.A., Pavlovic, V.: Machine Learning Methods for Social Signal Processing (2014)
Pentland, A.: Social dynamics: signals and behavior. In: ICDL (2004)
Sandbach, G., Zafeiriou, S., Pantic, M.: Markov random field structures for facial action unit intensity estimation. In: ICCVW (2013)
Savran, A., Cao, H., Nenkova, A., Verma, R.: Temporal Bayesian fusion for affect sensing: combining video, audio, and lexical modalities (2014)
Scherer, K.R.: What are emotions? And how can they be measured? Soc. Sci. Inf. 44, 693–727 (2005)
Keltner, D.: Signs of appeasement: evidence for the distinct displays of embarrassment, amusement, and shame. J. Pers. Soc. Psychol. 68, 441–454 (1995)
Bevacqua, E., Pelachaud, C.: Expressive audio-visual speech. Comput. Anim. Virtual Worlds 15, 297–304 (2004)
Fu, Q., op den Akker, R., Bruijnes, M.: A literature review of typical behavior of different interpersonal attitude. Capita Selecta HMI, University of Twente (2014)
Allwood, J., Cerrato, L.: A study of gestural feedback expressions. In: First Nordic Symposium on Multimodal Communication (2003)
Cafaro, A., Vilhjálmsson, H.H., Bickmore, T., Heylen, D., Jóhannsdóttir, K.R., Valgardsson, G.S.: First impressions: users judgments of virtual agents personality and interpersonal attitude in first encounters. In: IVA (2012)
Cowie, R., Gunes, H., McKeown, G., Armstrong, J., Douglas-Cowie, E.: The emotional and communicative significance of head nods and shakes in a naturalistic database (2010)
Lee, J., Marsella, S.: Modeling speaker behavior: a comparison of two approaches. In: IVA (2012)
Ravenet, B., Ochs, M., Pelachaud, C.: From a user-created corpus of virtual agent’s non-verbal behavior to a computational model of interpersonal attitudes. In: IVA (2013)
Martínez, H.P., Yannakakis, G.N.: Mining multimodal sequential patterns: a case study on affect detection. In: ICMI (2011)
Chollet, M., Ochs, M., Pelachaud, C.: From non-verbal signals sequence mining to bayesian networks for interpersonal attitudes expression. In: Bickmore, T., Marsella, S., Sidner, C. (eds.) IVA 2014. LNCS, vol. 8637, pp. 120–133. Springer, Heidelberg (2014). doi:10.1007/978-3-319-09767-1_15
Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: CVPR (2013)
Mertens, P.: The prosogram: semi-automatic transcription of prosody based on a tonal perception model. In: International Conference on Speech Prosody, 2004 (2004)
Lin, J., Keogh, E., Wei, L., Lonardi, S.: Experiencing SAX: a novel symbolic representation of time series. Data Min. Knowl. Discov. (2007)
Nicolle, J., Rapp, V., Bailly, K., Prevost, L., Chetouani, M.: Robust continuous prediction of human emotions using multiscale dynamic cues. In: ICMI (2012)
Guillame-Bert, M., Crowley, J.L.: Learning temporal association rules on symbolic time sequences. In: ACML (2012)
McKeown, G., Valstar, M., Cowie, R., Pantic, M., Schröder, M.: The semaine database: annotated multimodal records of emotionally colored conversations between a person and a limited agent. Affect. Comput. 3, 5–17 (2012)
Argyle, M.: Bodily Communication. Routledge, London (2013)
Ochs, M., Pelachaud, C.: Model of the perception of smiling virtual character. In: AAMAS (2012)
Guaïtella, I., Santi, S., Lagrue, B., Cavé, C.: Are eyebrow movements linked to voice variations and turn-taking in dialogue? An experimental investigation. Lang. Speech 52, 207–222 (2009)
Roon, K.D., Tiede, M.K., Dawson, K.M., Whalen, D.H.: Coordination of eyebrow movement with speech acoustics and head movement. In: ICPhS (2015)
Pecune, F., Cafaro, A., Chollet, M., Philippe, P., Pelachaud, C.: Suggestions for extending SAIBA with the VIB platform. In: Proceedings of the Workshop on Architectures and Standards for IVA (2014)
Motulsky, H.: Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking. Oxford University Press, New York (2013)
Acknowledgement
This work was performed within the Labex SMART supported by French state funds managed by the ANR within the Investissements dÁvenir programme under reference ANR-11-IDEX-0004-0.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Janssoone, T., Clavel, C., Bailly, K., Richard, G. (2016). Using Temporal Association Rules for the Synthesis of Embodied Conversational Agents with a Specific Stance. In: Traum, D., Swartout, W., Khooshabeh, P., Kopp, S., Scherer, S., Leuski, A. (eds) Intelligent Virtual Agents. IVA 2016. Lecture Notes in Computer Science(), vol 10011. Springer, Cham. https://doi.org/10.1007/978-3-319-47665-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-47665-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-47664-3
Online ISBN: 978-3-319-47665-0
eBook Packages: Computer ScienceComputer Science (R0)