Abstract
Building animated conversational agents requires developing a fine-grained analysis of the motions and meanings available to interlocutors in face-to-face conversation and implementing strategies for using these motions and meanings to communicate effectively. In this paper, we sketch our efforts to characterize people’s facial displays of uncertainty in face-to-face conversation. We analyze empirical data from human–human conversation and extend our platform for conversational animation, including RUTH (the Rutgers University Talking Head), to simulate what we find. This methodology leads to a range of new insights into the structure, timing, expressive content and communicative function of facial actions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
McNeill, D.: Hand and Mind: What Gestures Reveal about Thought. University of Chicago Press, Chicago (1992)
Bavelas, J.B., Chovil, N.: Visible acts of meaning: An integrated message model of language in face-to-face dialogue. Journal of Language and Social Psychology 19(2), 163–194 (2000)
Engle, R.A.: Toward a Theory of Multimodal Communication: Combining Speech, Gestures, Diagrams and Demonstrations in Instructional Explanations. PhD thesis, Stanford University (2000)
Cassell, J., McNeill, D., McCullough, K.E.: Speech-gesture mismatches: evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics and Cognition 6(2) (1999)
Krahmer, E., Ruttkay, Z., Swerts, M., Wesselink, W.: Pitch, eyebrows and the perception of focus. In: Symposium on Speech Prosody (2002)
Cassell, J., Nakano, Y., Bickmore, T.W., Sidner, C.L., Rich, C.: Non-verbal cues for discourse structure. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2001), pp. 106–115 (2001)
Nakano, Y.I., Reinstein, G., Stocky, T., Cassell, J.: Towards a model of face-to-face grounding. In: Dignum, F.P.M. (ed.) ACL 2003. LNCS (LNAI), vol. 2922, pp. 553–561. Springer, Heidelberg (2004)
Oh, I.: Modeling Believable Human-Computer Interaction with an Embodied Conversational Agent: Face-to-face Communication of Uncertainty. PhD thesis, Rutgers University (2006)
Oh, I., Frank, M., Stone, M.: Face-to-face communication of uncertainty: expression and recognition of uncertainty signals by different levels across modalities. In: ICA International Communication Association (2007)
Stone, M., Oh, I.: Understanding RUTH: Creating Believable Behaviors for a Virtual Human Under Uncertainty. In: Duffy, V.G. (ed.) HCII 2007 and DHM 2007. LNCS, vol. 4561, pp. 443–452. Springer, Heidelberg (2007)
DeCarlo, D., Revilla, C., Stone, M., Venditti, J.: Specifying and animating facial signals for discourse in embodied conversational agents. Computer Animation and Virtual Worlds 15(1), 27–38 (2004)
Smith, V.L., Clark, H.H.: On the course of answering questions. Journal of Memory and Language 32, 25–38 (1993)
Brennan, S.E., Willams, M.: The feeling of another’s knowing: prosody and filled pauses as cues to listeners about the metacognitive states of speakers. Journal of Memory and Language 34, 383–398 (1995)
Givens, D.B.: The Nonverbal Dictionary of Gestures, Signs and Body Language Cues. Center for Nonverbal Studies Press (2001)
Darwin, C.: The Expression of the Emotions in Man and Animals, John Murray, London (1872)
Rozin, P., Cohen, A.B.: High frequency of facial expressions corresponding to confusion, concentration and worry in an analysis of naturally occurring facial expresssions in americans. Emotion 3, 68–75 (2003)
Swerts, M., Krahmer, E.: Audiovisual prosody and feeling of knowing. Journal of Memory and Language 53(1), 81–94 (2005)
Krahmer, E., Swerts, M.: How children and adults produce and perceive uncertainty in audiovisual speech. Language and Speech 48(1), 29–53 (2005)
Ellsworth, P.C.: Confusion, concentration and other emotions of interest. Emotion 3, 81–85 (2003)
Brashers, D.E.: Communication and uncertainty management. Journal of Communication 51, 477–497 (2001)
Ekman, P.: About brows: Emotional and conversational signals. In: von Cranach, M., Foppa, K., Lepenies, W., Ploog, D. (eds.) Human Ethology: Claims and Limits of a New Discipline: Contributions to the Colloquium, pp. 169–202. Cambridge University Press, Cambridge (1979)
Clark, H.H., Krych, M.A.: Speaking while monitoring addressees for understanding. Journal of Memory and Language (50), 62–81 (2004)
Paek, T., Horvitz, E.: Conversation as action under uncertainty. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 455–464 (2000)
DeVault, D., Stone, M.: Scorekeeping in an uncertain language game. In: BRANDIAL: Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 64–71 (2006)
Liscombe, J., Hirschberg, J., Venditti, J.J.: Detecting certainness in spoken tutorial dialogues. In: Proceedings of Interspeech (2005)
Forbes-Riley, K., Litman, D.J.: Analyzing dependencies between student certainness states and tutor responses in a spoken dialogue corpus. In: Dybkjaer, L., Minker, W. (eds.) Recent Trends in Discourse and Dialogue, Springer, Heidelberg (2007)
Löfqvist, A.: Speech as audible gestures. In: Hardcastle, W.J., Marchal, A. (eds.) Speech Production and Speech Modeling, pp. 289–322. Kluwer, Dordrecht (1990)
Cohen, M.M., Massaro, D.W.: Modeling coarticulation in synthetic visual speech. In: Thalmann, N.M., Thalmann, D. (eds.) Models and techniques in computer animation, pp. 139–156. Springer, Heidelberg (1993)
King, S.A.: A facial model and animation techniques for animated speech. PhD thesis, The Ohio State University (2001)
Black, A., Taylor, P.: Festival speech synthesis system. Technical Report HCRC/TR-83, Human Communication Research Center (1997)
Silverman, K.E.A., Beckman, M., Pitrelli, J.F., Ostendorf, M., Wightman, C., Price, P., Pierrehumbert, J.: ToBI: A standard for labeling English prosody. In: Proceedings of the International Conference on Spoken Language Processing, pp. 867–870 (1992)
Bull, P., Connelly, G.: Body movement and emphasis in speech. Journal of Nonverbal Behavior 9(3), 169–187 (1985)
Ekman, P., Friesen, W.V., Hager, J.C.: Facial Action Coding System (FACS): Manual and Investigator’s Guide. In: A Human Face, Salt Lake City, UT (2002)
Cohn, J.F., Ekman, P.: Measuring facial action. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) The New Handbook of Nonverbal Behavior Research, Oxford, pp. 9–64 (2005)
Frank, M.: Research methods in detecting deception research. In: Harrigan, J.A., Rosenthal, R., Scherer, K.R. (eds.) The New Handbook of Methods in Nonverbal Behavior, Oxford (2006)
Vilhjálmsson, H., Cassell, J.: BodyChat: autonomous communicative behaviors in avatars. In: Proceedings of the International Conference on Autonomous Agents (Agents 1998), pp. 269–276 (1998)
Ekman, P., Friesen, W.V.: The repertoire of nonverbal behavior: Categories, origins, usage, and coding. Semiotica 1(1), 49–98 (1969)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Stone, M., Oh, I. (2008). Modeling Facial Expression of Uncertainty in Conversational Animation. In: Wachsmuth, I., Knoblich, G. (eds) Modeling Communication with Robots and Virtual Humans. Lecture Notes in Computer Science(), vol 4930. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-79037-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-540-79037-2_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-79036-5
Online ISBN: 978-3-540-79037-2
eBook Packages: Computer ScienceComputer Science (R0)