Skip to main content
Springer Nature Link
Log in

Situated Dialogue Processing for Human-Robot Interaction

  • Chapter
Cognitive Systems

Part of the book series: Cognitive Systems Monographs ((COSMOS,volume 8))

  • 1517 Accesses

Introduction

In CoSy, our robots were to be able to interact with human. These interactions served to help the robot learn more about its environment, or to plan and carry out actions. For a robot to make sense of such dialogues, it needs to understand how a dialogue can relate to, and refer to, “the world” – local visuo-spatial scenes, as in the Playmate scenario (9), or the spatial organization of an indoor environment in the Explorer scenario (10).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Roy, D., Reiter, E.: Connecting language to the world. Artificial Intelligence 167(1-2), 1–12 (2005)

    Article  Google Scholar 

  2. Winograd, T.: A process model of language understanding. In: Schank, R., Colby, K. (eds.) Computer Models of Thought and Language, pp. 152–186. Freeman, New York (1973)

    Google Scholar 

  3. Gorniak, P., Roy, D.: Grounded semantic composition for visual scenes. Journal of Artificial Intelligence Research 21, 429–470 (2004)

    Google Scholar 

  4. Gorniak, P., Roy, D.: Probabilistic grounding of situated speech using plan recognition and reference resolution. In: Proceedings of the Seventh International Conference on Multimodal Interfaces, ICMI 2005 (2005)

    Google Scholar 

  5. Gorniak, P., Roy, D.: Situated language understanding as filtering perceived affordances. Cognitive Science 31(2), 197–231 (2007)

    Google Scholar 

  6. Steels, L., Baillie, J.-C.: Shared grounding of event descriptions by autonomous robots. Robotics and Autonomous Systems 43(2-3), 163–173 (2003)

    Article  Google Scholar 

  7. Steels, L.: Semiotic dynamics for embodied agents. IEEE Intelligent Systems 21(3), 32–38

    Google Scholar 

  8. Steels, L.: The symbol grounding problem has been solved. so what’s next? In: De Vega, M., Glennberg, G., Graesser, G. (eds.) Symbols, embodiment and meaning. Academic Press, New Haven (2008)

    Google Scholar 

  9. Scheutz, M., Eberhard, K., Andronache, V.: A real-time robotic model of human reference resolution using visual constraints. Connection Science Journal 16(3), 145–167 (2004)

    Article  Google Scholar 

  10. Brick, T., Scheutz, M.: Incremental natural language processing for HRI. In: Proceeding of the ACM/IEEE international conference on Human-Robot Interaction (HRI 2007), pp. 263–270 (2007)

    Google Scholar 

  11. Altmann, G., Steedman, M.: Interaction with context during human sentence processing. Cognition 30(3), 191–238 (1988)

    Article  Google Scholar 

  12. Altmann, G., Kamide, Y.: Now you see it, now you don’t: Mediating the mapping between language and the visual world. In: Henderson, J., Ferreira, F. (eds.) The Interface of Language, Vision, and Action: Eye Movements and The Visual World, pp. 347–386. Psychology Press, New York (2004)

    Google Scholar 

  13. Knoeferle, P., Crocker, M.: The coordinated interplay of scene, utterance, and world knowledge: evidence from eye tracking. Cognitive Science

    Google Scholar 

  14. Hadelich, K., Crocker, M.: Gaze alignment of interlocutors in conversational dialogues. In: Proc. 19th CUNY Conference on Human Sentence Processing, New York, USA (2006)

    Google Scholar 

  15. Steedman, M.: The Syntactic Process. The MIT Press, Cambridge (2000)

    Google Scholar 

  16. Baldridge, J., Kruijff, G.: Multi-modal combinatory categorial grammmar. In: Proceedings of EACL 2003, Budapest, Hungary (2003)

    Google Scholar 

  17. Blackburn, P.: Representation, reasoning, and relational structures: a hybrid logic manifesto. Logic Journal of the IGPL 8(3), 339–625 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  18. Kruijff, G.: A categorial-modal logical architecture of informativity: Dependency grammar logic & information structure, Ph.D. thesis, Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic (April 2001)

    Google Scholar 

  19. Areces, C.: Logic engineering. the case of description and hybrid logics, Ph.D. thesis, Institute for Logic, Language and Computation, University of Amsterdam, Amsterdam, The Netherlands (October 2000)

    Google Scholar 

  20. Baldridge, J., Kruijff, G.: Coupling CCG and hybrid logic dependency semantics. In: Proc. ACL 2002, Philadelphia, PA, pp. 319–326 (2002)

    Google Scholar 

  21. Kruijff, G., Kelleher, J., Hawes, N.: Information fusion for visual reference resolution in dynamic situated dialogue. In: André, E., Dybkjaer, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS (LNAI), vol. 4021, pp. 117–128. Springer, Heidelberg (2006), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff/etal:2006-PIT

    Chapter  Google Scholar 

  22. Jacobsson, H., Hawes, N., Kruijff, G., Wyatt, J.: Crossmodal content binding in information-processing architectures. In: Proceedings of the 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI), Amsterdam, The Netherlands (2008), http://cognitivesystems.org/cosybook/chap8.asp#jacobsson+hawes+kruijff+wyatt_2008

  23. Kruijff, G., Brenner, M.: Modelling spatio-temporal comprehension in situated human-robot dialogue as reasoning about intentions and plans. In: Proceedings of the Symposium on Intentions in Intelligent Systems, Stanford University, Palo Alto, CA. AAAI Spring Symposium Series (2007), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff/Brenner:2007

  24. Asher, N., Lascarides, A.: Logics of Conversation. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  25. Moens, M., Steedman, M.: Temporal ontology and temporal reference. Journal of Computational Linguistics 14, 15–28 (1988)

    Google Scholar 

  26. Kelleher, J., Kruijff, G., Costello, F.: Proximity in context: an empirically grounded computational model of proximity for processing topological spatial expressions. In: Proceedings of ACL/COLING 2006 (2006), http://cognitivesystems.org/cosybook/chap8.asp#Kelleher/etal:2006

  27. Zender, H., Kruijff, G.: Towards generating referring expressions in a mobile robot scenario. In: Language and Robots: Proceedings of the Symposium, Aveiro, Portugal, pp. 101–106 (2007), http://cognitivesystems.org/cosybook/chap8.asp#zender/kruijff:2007-gre

  28. Kruijff, G., Kelleher, J., Berginc, G., Leonardis, A.: Structural descriptions in human-assisted robot visual learning. In: Proc. 1st Annual Conference on Human-Robot Interaction, HRI 2006 (2006), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff/etal:2006-vision

  29. Bloom, P.: How children learn the meanings of words. The MIT Press, Cambridge (2000)

    Google Scholar 

  30. Stone, M.: Intention, interpretation and the computational structure of language. Cognitive Science 28(5), 781–809 (2004)

    Google Scholar 

  31. Van Berkum, J., Zwitserlood, P., Brown, C., Hagoort, P.: When and how do listeners relate a sentence to the wider discourse? evidence from the n400 effect. Cognitive Brain Research 17, 701–718 (2003)

    Article  Google Scholar 

  32. Crain, S., Steedman, M.: On not being led up the garden path: The use of context by the psychological syntax processor. In: Dowty, D.R., Karttunen, L., Zwicky, A.M. (eds.) Natural language parsing: Psychological, computational, and theoretical perspectives. Cambridge University Press, Cambridge (1985)

    Google Scholar 

  33. Tanenhaus, M., Spivey-Knowlton, M., Eberhard, K., Sedivy, J.: Integration of visual and linguistic information in spoken language comprehension. Science 268, 1632–1634 (1995)

    Article  Google Scholar 

  34. Liversedge, S., Findlay, J.: Saccadic eye movements and cognition. Trends in Cognitive Science 4(1), 6–14 (2000)

    Article  Google Scholar 

  35. Van Berkum, J.: Sentence comprehension in a wider discourse: Can we use erps to keep track of things? In: Carreiras, M., Clifton Jr., C. (eds.) The on-line study of sentence comprehension: Eyetracking, ERPs and beyond, pp. 229–270. Psychology Press, New York (2004)

    Google Scholar 

  36. Allopenna, P., Magnuson, J., Tanenhaus, M.: Tracking the time course of spoken word recognition using eye movements: Evidence for continuous mapping models. Journal of Memory and Language 38(4), 419–439 (1998)

    Article  Google Scholar 

  37. van Berkum, J., Hagoort, P., Brown, C.: Semantic integration in sentences and discourse: Evidence from the n400. Journal of Cognitive Neuroscience 11(6), 657–671 (1999)

    Article  Google Scholar 

  38. Van Petten, C., Coulson, S., Rubin, S., Plante, E., Parks, M.: Time course of word identification and semantic integration in spoken language. Journal of Experimental Psychology: Learning, Memory, and Cognition 25(2), 394–417 (1999)

    Article  Google Scholar 

  39. Altmann, G.M.: Ambiguity in sentence processing. Trends in Cognitive Sciences 2(4)

    Google Scholar 

  40. Spivey, M., Trueswell, J., Tanenhaus, M.: Context effects in syntactic ambiguity resolution: discourse and semantic influences in parsing reduced relative clauses. Canadian Journal of Experimental Psychology 47(2), 276–309 (1993)

    Google Scholar 

  41. Spivey, M., Tanenhaus, M.: Syntactic ambiguity resolution in discourse: Modeling the effects of referential context and lexical frequency. Journal of Experimental Psychology: Learning, Memory, and Cognition 24, 1521–1543 (1998)

    Article  Google Scholar 

  42. van Berkum, J., Brown, C., Hagoort, P.: Early referential context effects in sentence processing: Evidence from event-related brain potentials. Journal of Memory and Language 41, 147–182 (1999)

    Article  Google Scholar 

  43. Tanenhaus, M., Magnuson, J., Dahan, D., Chambers, G.: Eye movements and lexical access in spoken-language comprehension: Evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research 29(6), 557–580 (2000)

    Article  Google Scholar 

  44. Dahan, D., Tanenhaus, M.: Continuous mapping from sound to meaning in spoke-language comprehension: Immediate effects of verb-based thematic constraints. Journal of Experimental Psychology: Learning, Memory, and Cognition 30(2), 498–513 (2004)

    Article  Google Scholar 

  45. Van Berkum, J., Brown, C., Zwitserlood, P., Kooijman, V., Hagoort, P.: Anticipating upcoming words in discourse: Evidence from erps and reading times. Journal of Experimental Psychology: Learning, Memory, & Cognition 31(3), 443–467 (2005)

    Article  Google Scholar 

  46. Nieuwland, M., Van Berkum, J.: When peanuts fall in love: N400 evidence for the power of discourse. Journal of Cognitive Neuroscience 18(7), 1098–1111 (2006)

    Article  Google Scholar 

  47. Botvinick, M., Braver, T., Barch, D., Carter, C., Cohen, J.: Conflict monitoring and cognitive control. Psychological Review 108(3), 624–652 (2001)

    Article  Google Scholar 

  48. Hommel, B., Ridderinkhof, K., Theeuwes, J.: Cognitive control of attention and action: Issues and trends. Psychological Research 66, 215–219 (2002)

    Article  Google Scholar 

  49. Novick, J., Trueswell, J., Thompson-Schill, S.: Cognitive control and parsing: Reexamining the role of Broca’s area in sentence comprehension. Cognitive, Affective, and Behavioral Neuroscience 5(3), 263–281 (2005)

    Article  Google Scholar 

  50. Altmann, G., Kamide, Y.: Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73(3), 247–264 (1999)

    Article  Google Scholar 

  51. Chambers, C., Tanenhaus, M., Magnuson, J.: Actions and affordances in syntactic ambiguity resolution. Jnl. Experimental Psychology 30(3), 687–696 (2004)

    Google Scholar 

  52. Endsley, M.: Theoretical underpinnings of situation awareness: A critical review. In: Endsley, M.R., Garland, D.J. (eds.) Situation awareness analysis and measurement. Lawrence Erlbaum, Mahwah (2000)

    Google Scholar 

  53. Kamide, Y., Altmann, G., Haywood, S.: The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye-movements. Jnl. Memory and Language 49(1), 133–156 (2003)

    Article  Google Scholar 

  54. Glenberg, A., Kaschak, M.: Grounding language in action. Psychonomic Bulletin & Review 9(3), 558–565 (2002)

    Google Scholar 

  55. De Vega, M., Robertson, D., Glenberg, A., Kaschak, M., Rinck, M.: On doing two things at once: Temporal constraints on actions in language comprehension. Memory and Cognition 32(7), 1033–1043 (2004)

    Google Scholar 

  56. Glenberg, A.: What memory is for. Behavioral & Brain Sciences 20, 1–55 (1997)

    Google Scholar 

  57. Barsalou, L.: Perceptual symbol systems. Behavioral & Brain Sciences 22, 577–660 (1999)

    Google Scholar 

  58. Pickering, M., Garrod, S.: Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences 27, 169–225 (2004)

    Google Scholar 

  59. Oepen, S., Carroll, J.: Ambiguity packing in constraint-based parsing: Practical results. In: Proceedings of the 6th Applied Natural Language Processing Conference (ANLP 2000), pp. 162–169 (2000)

    Google Scholar 

  60. Carroll, J., Oepen, S.: High efficiency realization for a wide-coverage unification grammar. In: Proceedings of the International Joint Conference on Natural Language Processing (IJCNLP 2005), pp. 165–176 (2005)

    Google Scholar 

  61. Moore, R.K.: Spoken language processing: piecing together the puzzle. Speech Communication: Special Issue on Bridging the Gap Between Human and Automatic Speech Processing 49, 418–435 (2007)

    Google Scholar 

  62. Lison, P., Kruijff, G.: Salience-driven contextual priming of speech recognition for human-robot interaction. In: Proceedings of ECAI 2008, Athens, Greece (2008), http://cognitivesystems.org/cosybook/chap8.asp#Lison/Kruijff:2008

  63. Collins, M.: Parameter estimation for statistical parsing models: theory and practice of distribution-free methods. In: New developments in parsing technology, pp. 19–55. Kluwer Academic Publishers, Dordrecht (2004)

    Google Scholar 

  64. Kruijff, G., Brenner, M., Hawes, N.: Continual planning for cross-modal situated clarification in human-robot interaction. In: Proceedings of the 17th International Symposium on Robot and Human Interactive Communication (RO-MAN 2008), Munich, Germany (2008), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff/etal:2008

  65. Kruijff, G.: Context-sensitive utterance planning for CCG. In: Proceedings of the European Workshop on Natural Language Generation, Aberdeen, Scotland (2005), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff:2005

  66. Bateman, J.: Enabling technology for multilingual natural language generation: the kpml development environment. Journal of Natural Language Engineering 3(1), 15–55 (1997)

    Article  Google Scholar 

  67. Dale, R., Reiter, E.: Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science 19(2), 233–263 (1995)

    Article  Google Scholar 

  68. White, M., Baldridge, J.: Adapting chart realization to CCG. In: Proceedings of the Ninth European Workshop on Natural Language Generation, Budapest, Hungary (2003)

    Google Scholar 

  69. White, M.: Efficient realization of coordinate structures in combinatory categorial grammar. Research on Language and Computation 4(1), 39–75 (2006)

    Article  MathSciNet  Google Scholar 

  70. Schröder, M., Trouvain, J.: The german text-to-speech synthesis system mary: A tool for research, development and teaching. International Journal of Speech Technology 6, 365–377 (2003)

    Article  Google Scholar 

  71. Steedman, M., Kruijff-Korbayová, I.: Discourse and information structure. Journal of Logic, Language and Information 12, 249–259 (2003)

    Article  MATH  Google Scholar 

  72. Kelleher, J., Kruijff, G.: Incremental generation of spatial referring expressions in situated dialog. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pp. 1041–1048 (2006), http://cognitivesystems.org/cosybook/chap8.asp#Kelleher/Kruijff:2006

  73. Kuipers, B.: Representing knowledge of large-scale space, Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA (1977)

    Google Scholar 

  74. Kruijff, G., Zender, H., Jensfelt, P., Christensen, H.: Situated dialogue and spatial organization: What, where.. and why? International Journal of Advanced Robotic Systems 4(2), http://cognitivesystems.org/cosybook/chap8.asp#Kruijff/etal:2007-JARS

  75. Zender, H., Jensfelt, P., Mozos, O.M., Kruijff, G., Burgard, W.: An integrated robotic system for spatial understanding and situated interaction in indoor environments. In: Proc. of AAAI 2007, Vancouver, BC, Canada, pp. 1584–1589 (2007), http://cognitivesystems.org/cosybook/chap8.asp#Zender/etal:2007-AAAI

  76. Zender, H., Jensfelt, P., Mozos, O.M., Kruijff, G., Burgard, W.: Conceptual spatial representations for indoor mobile robots. Robotics and Autonomous Systems, special Issue From Sensors to Human Spatial Concepts 56(6), http://cognitivesystems.org/cosybook/chap8.asp#Zender/etal:2008

  77. Topp, E.A., Hüttenrauch, H., Christensen, H., Severinson Eklundh, K.: Bringing together human and robotic environment representations – a pilot study. In: Proc. of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Beijing, China (2006)

    Google Scholar 

  78. Shi, H., Tenbrink, T.: Telling rolland where to go: Hri dialogues on route navigation. In: Proceedings of the Workshop on Spatial Language and Dialogue (5th Workshop on Language and Space), Delmenhorst, Germany (2005)

    Google Scholar 

  79. Brown, R.: How shall a thing be called? Psychological Review 65(1), 14–21 (1958)

    Article  Google Scholar 

  80. Rosch, E.: Principles of categorization. In: Rosch, E., Lloyd, B. (eds.) Cognition and Categorization, pp. 27–48. Lawrence Erlbaum Associates, Hillsdale (1978)

    Google Scholar 

  81. Brenner, M., Hawes, N., Kelleher, J., Wyatt, J.: Mediating between qualitative and quantitative representations for task-orientated human-robot interaction. In: Proceedings of the Twentieth International Joint Conference on Artificial Intelligence, IJCAI 2007 (2007), http://cognitivesystems.org/cosybook/chap8.asp#Brenner/etal:2007

  82. Sidner, C.L., Lee, C., Kidd, C., Lesh, N., Rich, C.: Explorations in engagement for humans and robots. Artificial Intelligence 166(1-2), 140–164 (2005)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DFKI GmbH, Saarbrücken, Germany

    Geert-Jan M. Kruijff, Pierre Lison, Trevor Benjamin, Henrik Jacobsson, Hendrik Zender & Ivana Kruijff-Korbayová

  2. Intelligent Robotics Laboratory, School of Computer Science, University of Birmingham, Birmingham, UK

    Nick Hawes

Authors
  1. Geert-Jan M. Kruijff
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre Lison
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Trevor Benjamin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Henrik Jacobsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hendrik Zender
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ivana Kruijff-Korbayová
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nick Hawes
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Georgia Tech, RIM@GT, 30332, Atlanta, GA, USA

    Henrik Iskov Christensen

  2. DFKI GmbH, Stuhlsatzenhausweg 3, 66123, Saarbrücken, Germany

    Geert-Jan M. Kruijff

  3. School of Computer Science, University of Birmingham, B15 2TT, Birmingham, UK

    Jeremy L. Wyatt

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Kruijff, GJ.M. et al. (2010). Situated Dialogue Processing for Human-Robot Interaction. In: Christensen, H.I., Kruijff, GJ.M., Wyatt, J.L. (eds) Cognitive Systems. Cognitive Systems Monographs, vol 8. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11694-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-11694-0_8

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-11693-3

  • Online ISBN: 978-3-642-11694-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics