Skip to main content
SpringerLink
Log in

Nonverbal Feedback in Interactions

  • Chapter
Affective Information Processing

Abstract

Understanding how nonverbal aspects of communication support, complement, and in some cases, override verbal communication is necessary for human interactions. It is also crucial for designing and implementing interactive systems that aim at supporting flexible interaction management using natural language with users. In particular, the need for more comprehensive communication has become obvious in theubiquitous computing context where context-aware applications and automatic services require sophisticated knowledge management and adaptation to various user needs. Interactions with smart objects, services, and environments need to address challenges concerning natural, intuitive, easy, and friendly interaction. For instance, the Roadmap for Smart Human Environments (Plomp et al., 2002) envisages that smart environments will be populated by several context-aware devices which communicate with each other and with the users. The systems will identify their current context of use, adapt their behaviour accordingly, and also allow natural interaction.

In this chapter, I discuss verbal and nonverbal communication especially for the purposes of designing and developing interactive systems. I report on machinelearning experiments conducted on annotated gesture and facial expression data, and focus on the feedback and turn-taking processes that are important in building shared understanding of the semantics and flow of interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 54.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Allwood, J. (2001). The structure of dialog. In M. Taylor, D. Bouwhuis, & F. Nel (Eds.), The structure of multimodal dialogue II (pp. 3–24). Amsterdam: Benjamins.

    Google Scholar 

  2. Allwood, J, Cerrato, L., Jokinen, K., Navarretta, K., & Paggio, P. (2007). The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. In J.C. Martin, P. Paggio, P. Kuenlein, R. Stiefelhagen, & F. Pianesi (Eds), Multimodal corpora for modelling human multimodal behaviour. Special issue of the International Journal of Language Resources and Evaluation, 41(3–4), 273–287. http://www.springer.com/journal/10579/.

  3. Allwood, J., Traum, D., & Jokinen, K. (2000). Cooperation, dialogue and ethics. Special issue on collaboration, cooperation and conflict in dialogue systems, International Journal of Human— Computer Studies, 53(6), 871–914.

    Article  MATH  Google Scholar 

  4. André, E., & Pelachaud, C. (forthcoming). Interacting with embodied conversational agents. In K. Jokinen & F. Cheng (Eds.), New trends in speech-based interactive systems. New York: Springer.

    Google Scholar 

  5. Arbib, M. (2003). The evolving mirror system: A neural basis for language readiness. In M. Christiansen and S. Kirby (Eds.), Language evolution (pp. 182–200).Oxford: Oxford University Press,

    Google Scholar 

  6. Barsalou, L. W. (1999) Perceptual symbol systems. Behavioral and Brain Sciences, 22, 577–660.

    Google Scholar 

  7. Beira, R., Lopes, M., Praca, M., Santos-Victor, J., Bernardino, A., Mettay, G., Becchiz, F., & Saltar, R. (2006). Design of the robot-cub (iCub) head. In Proceedings of the IEEE International Conference on Robotics and Automation, Orlando, FL (pp. 94–100).

    Google Scholar 

  8. Campbell, N. (2007). On the use of nonverbal speech sounds in human communication. In N. Campbell (Ed.), Verbal and Nonverbal Communication Behaviors (LNAI 4775, pp.117– 128). New York: Springer.

    Chapter  Google Scholar 

  9. Campbell, N., & Jokinen, K. (2008). Non-verbal information resources for constructive dialogue management. Tutorial at the LREC 2008. Marrakech, Marocco.

    Google Scholar 

  10. Campbell, N., & Ohara, R. (2005). How far can non-verbal information help us follow a conversation? Preliminary experiments with speech-style and gesture tracking. In Proceedings of the ATR Symposium on the Cross-Modal Processing of Faces & Voices. No laughing matter.

    Google Scholar 

  11. Cassell, J., Sullivan J., Prevost, S., & Churchill, E. (Eds.)(2000). Embodied conversational agents. Cambridge, MA: MIT Press.

    Google Scholar 

  12. Clark, H., & Wilkes-Gibbs, D. (1986). Referring as a collaborative process. Cognition 22, 1–39.

    Article  Google Scholar 

  13. Douglas, C. E., Campbell, N., Cowie, R., & Roach, P. (2003). Emotional speech: Towards a new generation of databases. Speech Communication, 40, 33–60.

    Article  MATH  Google Scholar 

  14. Duncan, S., Jr., & Fiske, D.W. (1977). Face-to-face interaction: Research, methods and theory. Hillsdale, NJ: Lawrence Erlbaum. Distributed by John Wiley & Sons.

    Google Scholar 

  15. Garrod, S., & Doherty, G. (1994). Conversation, co-ordination and Convention: An empirical investigation of how groups establish linguistic conventions. Cognition, 53,181–215.

    Article  Google Scholar 

  16. Gibson, J. J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.

    Google Scholar 

  17. Harnard, S. (1990). The symbol grounding problem. Physica D, 42, 335–346.

    Article  Google Scholar 

  18. Jokinen, K. (2000). Learning dialogue systems. In Proceedings of the LREC Workshop from Spoken Dialogue to Full Natural Interactive Dialogue. Athens (pp. 13–17).

    Google Scholar 

  19. Jokinen, K. (2007). Interaction and mobile route navigation application. In L. Meng, A. Zipf, & S. Winter (Eds.), Map-based mobile services—Usage context, interaction and application. New York: Springer Series on Geoinformatics.

    Google Scholar 

  20. Jokinen, K. (2008). Constructive Dialogue Management — Speech interaction and rational agents. Hoboken, NJ: John Wiley & Sons.

    Google Scholar 

  21. Jokinen, K., & Hurtig, T. (2006). User expectations and real experience on a multimodal interactive system. In Proceedings of the Interspeech 2006, Pittsburgh, PA.

    Google Scholar 

  22. Jokinen, K., Paggio, P., & Navarretta, C. (2008). Distinguishing the communicative functions of gestures — An experiment with annotated gesture data. In Proceedings of the Conference on Machine-Learning in Multimodal Interaction.

    Google Scholar 

  23. Jokinen, K., & Ragni, A. (2007). On the annotation and analysis of multimodal corpus. In Proceedings of the 3rd Baltic Conference on Human Technology. Kaunas, Lithuania.

    Google Scholar 

  24. Katagiri, Y. (2005). Interactional alignment in collaborative problem solving dialogues, In Proceedings of the 9th International Pragmatics Conference, Riva del Garda Italy.

    Google Scholar 

  25. Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.

    Google Scholar 

  26. Kipp, M. (2001). Anvil — A generic annotation tool for multimodal dialogue. In Proceedings of the Seventh European Conference on Speech Communication and Technology (pp. 1367–1370).

    Google Scholar 

  27. Maes, P. (Ed.) (1990). Designing autonomous agents: Theory and practice from biology to engineering and back. Cambridge, MA: MIT Press.

    Google Scholar 

  28. Mandler, J. (2004). The foundations of mind: Origins of conceptual thought. Oxford: Oxford University Press.

    Google Scholar 

  29. Mavridis, N., & Roy, D. (2006). Grounded situation models for robots: where words and percepts meet. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

    Google Scholar 

  30. McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press.

    Google Scholar 

  31. Norman, D. A. (1988). The psychology of everyday things. New York: Basic Books.

    Google Scholar 

  32. Norros, L., Kaasinen, E., Plomp, J., & Rämä, P. (2003) Human—technology interaction research and design. VTT roadmaps. VTT Research Notes 2220. Espoo: VTT Industrial Systems,

    Google Scholar 

  33. Peirce, C. S. (1931). Elements of logic. Collected papers of Charles Sanders Peirce. C. Hartshorne and P. Weiss (Eds.) (vol. 2). Cambridge, MA: Harvard University Press.

    Google Scholar 

  34. Pickering, M. & Garrod, S. (2004). Towards a mechanistic psychology of dialogue, Behavioral and Brain Sciences 27, 169–226.

    Google Scholar 

  35. Plomp, J., Ahola, J., Alahuhta, P., Kaasinen, E., Korhonen, I., Laikari, A., Lappalainen, V., Pakanen, J., Rentto, K., & Virtanen, A. (2002). Smart human environments. In: Sipilä, M. (Ed.). Communications Technologies. The VTT Roadmaps. Espoo: Technical Research Centre of Finland, VTT Research Notes 2146, pp. 61–81.

    Google Scholar 

  36. Steels, L. (2003). Evolving grounded communication for robots. Trends in Cognitive Science, 7(7), 308–312.

    Article  Google Scholar 

  37. Swerts, M., & Krahmer, E. (2005). Audiovisual prosody and feeling of knowing. Journal of Memory and Language, 53, 81–94.

    Article  Google Scholar 

  38. Tao, J. (forthcoming). Multimodal information processing for affective computing. In K. Jokinen and F. Cheng (Eds.), New trends in speech-based interactive systems. New York: Springer.

    Google Scholar 

  39. Thompson, E. (2001). Empathy and consciousness. Journal of Consciousness Studies, 8, 1–32.

    Google Scholar 

  40. Tomasello, M. (1992). First verbs: A case study of early grammatical development. Cambridge: Cambridge University Press.

    Google Scholar 

  41. Traum, D. (1999). Computational models of grounding in collaborative systems. In Working Papers of the AAAI}Fall Symposium on Psychological Models of Communication in Collaborative Systems, AAAI, Menlo Park, CA (pp. 124–131).

    Google Scholar 

  42. Weiser, M. (1991). The computer for the twenty-first century. Scientific American, 265(3): 94–10.

    Article  Google Scholar 

  43. Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques (2nd ed.). San Francisco: Morgan Kaufmann.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Helsinki and University of Tampere, Finland

    Kristiina Jokinen

Authors
  1. Kristiina Jokinen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Automation, Chinese Academy of Sciences, 95 Zhongguancun East Road, Haidian, Beijing, 100080, P.R. China

    Jianhua Tao  & Tieniu Tan BSc, MSc, PhD  & 

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag London Limited

About this chapter

Cite this chapter

Jokinen, K. (2009). Nonverbal Feedback in Interactions. In: Tao, J., Tan, T. (eds) Affective Information Processing. Springer, London. https://doi.org/10.1007/978-1-84800-306-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-1-84800-306-4_13

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-84800-305-7

  • Online ISBN: 978-1-84800-306-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation