Abstract
Human interactions are predominantly conducted via verbal communication which allows presentation of sophisticated propositional content. However, much of the interpretation of the utterances and the speaker’s attitudes are conveyed using multimodal cues such as facial expressions, hand gestures, head movements and body posture. This paper reports some observations on multimodal communication and feedback giving activity in first encounter interactions, and discusses how head, hand, and body movements are used in conversational interactions as means ofvisual interaction management, i.e. unobtrusive ways to control the interaction and construct shared understanding among the interlocutors. The observations and results contribute to the models for coordinating communication in human-human conversations as well as in interactions between humans and intelligent situated agents.
Chapter PDF
Similar content being viewed by others
References
Allwood, J.: An Activity Based Approach to Pragmatics. In: Gothenburg Papers In Theoretical Linguistics 76. Department of Linguistics. Göteborg University (2000)
Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The MUMIN Coding Scheme for the Annotation of Feedback, Turn Management and Sequencing Phenomena. Multimodal Corpora for Modelling Human Multimodal Behaviour. Special Issue of the International Journal of Language Resources and Evaluation 41(3-4), 273–287 (2007)
André, E., Pelachaud, C.: Interacting with Embodied Conversational Agents. In: Jokinen, K., Cheng, F. (eds.) Speech-based Interactive Systems: Theory and Applications. Springer (2009)
Argyle, M., Cook, M.: Gaze and Mutual Gaze. Cambridge University Press (1976)
Battersby, S.: Moving Together: the organization of Non-verbal cues during multiparty conversation. PhD Thesis, Queen Mary, University of London (2011)
Clark, H.H., Schaefer, E.F.: Contributing to Discourse. Cognitive Science 13, 259–294 (1989)
Feldman, R.S., Rim, B.: Fundamentals of Nonverbal Behavior. Cambridge University Press (1991)
Hart, C.L., Fillmore, D.G., Griffith, J.D.: Indirect Detection of Deception: Looking for Change. Current Research in Social Psychology 1(9), 134–142 (2009)
Jokinen, K.: Rational communication and affordable natural language interaction for ambient environments. In: Lee, G.G., Mariani, J., Minker, W., Nakamura, S. (eds.) IWSDS 2010. LNCS, vol. 6392, pp. 163–168. Springer, Heidelberg (2010)
Jokinen, K.: Pointing gestures and synchronous communication management. In: Esposito, A., Campbell, N., Vogel, C., Hussain, A., Nijholt, A. (eds.) COST 2102 Int. Training School 2009. LNCS, vol. 5967, pp. 33–49. Springer, Heidelberg (2010)
Jokinen, K., Furukawa, H., Nishida, M., Yamamoto, S.: Gaze and Turn-taking behaviour in Casual Conversational Interactions. ACM Trans. Interactive Intelligent Systems (2013)
Jokinen, K., Pärkson, S.: Synchrony and copying in conversational interactions. In: Proceedings of the 3rd Nordic Symposium on Multimodal Communication, vol. 15, pp. 18–24 (2011)
Jokinen, K., Wilcock, G.: Multimodal Signals and Holistic Interaction Structuring. In: Proceedings of the 24th International Conference on Computational Linguistics (COLING 2012), Mumbai, India (2012)
Kendon, A.: Gesture. Visible Action as Utterance. Cambridge University Press (2005)
Kipp, M.: Anvil - A Generic Annotation Tool for Multimodal Dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology (Eurospeech), pp. 1367–1370 (2001)
Lee, J., Marsella, S.C., Traum, D.R., Gratch, J., Lance, B.: The Rickel Gaze Model: A Window on the Mind of a Virtual Human. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 296–303. Springer, Heidelberg (2007)
Levitan, R., Gravano, A., Hirschberg, J.: Entrainment in Speech Preceding Back-channels. In: Proceedings of ACL 2011, pp. 113–117 (2011)
Mancini, M., Castellano, G., Bevacqua, E., Peters, C.: Copying Behaviour of Expressive Motion. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 180–191. Springer, Heidelberg (2007)
Nakano, Y., Nishida, T.: Attentional Behaviours as Nonverbal Communicative Signals in Situated Interactions with Conversational Agents. In: Nishida, T. (ed.) Engineering Approaches to Conversational Informatics. John Wiley (2007)
Navarretta, C., Ahlsén, E., Allwood, J., Jokinen, K., Paggio, P.: Feedback in Nordic First-Encounters: a Comparative Study. In: Proceedings of the Language Resources and Evaluation Conference (LREC-2012), Istanbul, Turkey (2012)
Norman, D.A.: The Psychology of Everyday Things. Basic Books, New York (1988)
Pickering, M., Garrod, S.: Towards a mechanistic psychology of dialogue. Behavioral and Brain Sciences 27, 169–226 (2004)
Toivio, E., Jokinen, K.: Multimodal Feedback Signaling in Finnish. In: Proceedings of the Human Language Technologies – The Baltic Perspective (2012); Published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License
Tomasello, M.: First verbs: A case study of early grammatical development. Cambridge University Press, Cambridge (1992)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Jokinen, K. (2013). Multimodal Feedback in First Encounter Interactions. In: Kurosu, M. (eds) Human-Computer Interaction. Interaction Modalities and Techniques. HCI 2013. Lecture Notes in Computer Science, vol 8007. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39330-3_28
Download citation
DOI: https://doi.org/10.1007/978-3-642-39330-3_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39329-7
Online ISBN: 978-3-642-39330-3
eBook Packages: Computer ScienceComputer Science (R0)