Skip to main content
Springer Nature Link
Log in

Incremental Unit Networks for Distributed, Symbolic Multimodal Processing and Representation

  • Conference paper
  • First Online:
Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13320))

Included in the following conference series:

  • 1432 Accesses

Abstract

Incremental dialogue processing has been an important topic in spoken dialogue systems research, but the broader research community that makes use of language interaction (e.g., chatbots, conversational AI, spoken interaction with robots) have not adopted incremental processing despite research showing that humans perceive incremental dialogue as more natural. In this paper, we extend prior work that identifies the requirements for making spoken interaction with a system natural with the goal that our framework will be generalizable to many domains where speech is the primary method of communication. The Incremental Unit framework offers a model of incremental processing that has been extended to be multimodal, temporally aligned, enables real-time information updates, and creates complex network of information as a fine-grained information state. One challenge is that multimodal dialogue systems often have computationally expensive modules, requiring computation to be distributive. Most importantly, when speech is the means of communication, it brings the added expectation that systems understand what they (humans) say, but also that systems understand and respond without delay. In this paper, we build on top of the Incremental Unit framework and make it amenable to a distributive architecture made up of a robot and spoken dialogue system modules. To enable fast communication between the modules and to maintain module state histories, we compared two different implementations of a distributed Incremental Unit architecture. We compare both implementations systematically then with real human users and show that the implementation that uses an external attribute-value database is preferred, but there is some flexibility in which variant to use depending on the circumstances. This work offers the Incremental Unit framework as an architecture for building powerful, complete, and natural dialogue systems, specifically applicable to robots and multimodal systems researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aist, G., et al.: Software architectures for incremental understanding of human speech. In: Proceedings of CSLP, pp. 1922–1925 (2006)

    Google Scholar 

  2. Asri, L.E., Laroche, R., Pietquin, O., Khouzaimi, H.: NASTIA: negotiating appointment setting interface. In: Proceedings of LREC, pp. 266–271 (2014)

    Google Scholar 

  3. Bartneck, C., Kulić, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots. Int. J. Soc. Robot. 1(1), 71–81 (2009)

    Article  Google Scholar 

  4. Baumann, T., Schlangen, D.: The InproTK 2012 release. In: NAACL-HLT Workshop on Future Directions and Needs in the Spoken Dialog Community: Tools and Data (SDCTD 2012), pp. 29–32 (2012)

    Google Scholar 

  5. Bocklisch, T., Faulkner, J., Pawlowski, N., Nichol, A.: Rasa: open source language understanding and dialogue management. In: Proceedings of the 31st Conference on Neural Information Processing Systems (2017). http://alborz-geramifard.com/workshops/nips17-Conversational-AI/Papers/17nipsw-cai-rasa.pdf, http://arxiv.org/abs/1712.05181

  6. Bohus, D., Andrist, S., Jalobeanu, M.: Rapid development of multimodal interactive systems: a demonstration of platform for situated intelligence. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 493–494 (2017)

    Google Scholar 

  7. Buß, O., Schlangen, D.: DIUM-an incremental dialogue manager that can produce self-corrections. In: Proceedings of SemDial 2011 (Los Angelogue) (2011)

    Google Scholar 

  8. Carlson, J.L.: Redis in Action. Manning Publications Co. (2013)

    Google Scholar 

  9. Edlund, J., Gustafson, J., Heldner, M., Hjalmarsson, A.: Towards human-like spoken dialogue systems. Speech Commun. 50(8–9), 630–645 (2008). https://doi.org/10.1016/j.specom.2008.04.002

    Article  Google Scholar 

  10. Fillmore, C.J.: Pragmatics and the description of discourse. Radical Pragma. 143–166 (1981)

    Google Scholar 

  11. Hough, J., Schlangen, D.: It’s not what you do, it’s how you do it: grounding uncertainty for a simple robot. In: Proceedings of the 2017 Conference on Human-Robot Interaction (HRI2017) (2017)

    Google Scholar 

  12. Jang, Y., Lee, J., Park, J., Lee, K.H., Lison, P., Kim, K.E.: PyOpenDial: a python-based domain-independent toolkit for developing spoken dialogue systems with probabilistic rules. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pp. 187–192. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/D19-3032, https://www.aclweb.org/anthology/D19-3032

  13. Kennington, C., Han, T., Schlangen, D.: Temporal alignment using the incremental unit framework. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, ICMI 2017, pp. 297–301. ACM, New York (2017). https://doi.org/10.1145/3136755.3136769

  14. Kennington, C., Kousidis, S., Schlangen, D.: InproTKs: a toolkit for incremental situated processing. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 84–88. Association for Computational Linguistics, Philadelphia (2014). http://www.aclweb.org/anthology/W14-4312

  15. Kennington, C., Moro, D., Marchand, L., Carns, J., McNeill, D.: rrSDS: towards a robot-ready spoken dialogue system. In: Proceedings of the 21st Annual SIGdial Meeting on Discourse and Dialogue. Association for Computational Linguistics, Virtual (2020)

    Google Scholar 

  16. Kennington, C., Schlangen, D.: Simple learning and compositional application of perceptually grounded word meanings for incremental reference resolution. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 292–301. Association for Computational Linguistics, Beijing, July 2015. https://doi.org/10.3115/v1/P15-1029, https://www.aclweb.org/anthology/P15-1029

  17. Kennington, C., Schlangen, D.: Supporting spoken assistant systems with a graphical user interface that signals incremental understanding and prediction state. In: Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 242–251. Association for Computational Linguistics, Los Angeles, September 2016. http://www.aclweb.org/anthology/W16-3631

  18. Kennington, C., Schlangen, D.: Incremental unit networks for multimodal, fine-grained information state representation. In: Proceedings of the 1st Workshop on Multimodal Semantic Representations (MMSR), pp. 89–94. Association for Computational Linguistics, Groningen, Netherlands, June 2021. https://aclanthology.org/2021.mmsr-1.8

  19. Lison, P., Kennington, C.: OpenDial: a toolkit for developing spoken dialogue systems with probabilistic rules. In: 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016 - System Demonstrations (2016)

    Google Scholar 

  20. Lison, P., Kennington, C.: Incremental processing for a neural conversational model. In: Proceedings of SemDial (2017)

    Google Scholar 

  21. Marge, M., Espy-Wilson, C., Ward, N.: Spoken language interaction with robots: research issues and recommendations, report from the NSF future directions workshop. arXiv preprint arXiv:2011.05533 (2020)

  22. Marge, M., et al.: A research platform for multi-robot dialogue with humans. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 132–137. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://aclanthology.org/N19-4023, https://doi.org/10.18653/v1/N19-4023

  23. Michael, T., Möller, S.: ReTiCo: an open-source framework for modeling real-time conversations in spoken dialogue systems. Studientexte Sprachkommun.: Elektron. Sprachsignalverarbeitung 2019, 134–140 (2019)

    Google Scholar 

  24. Novikova, J., Ren, G., Watts, L.: It’s not the way you look, it’s how you move: validating a general scheme for robot affective behaviour. In: Abascal, J., Barbosa, S., Fetter, M., Gross, T., Palanque, P., Winckler, M. (eds.) INTERACT 2015. LNCS, vol. 9298, pp. 239–258. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22698-9_16

    Chapter  Google Scholar 

  25. Peltason, J., Riether, N., Wrede, B., Lütkebohle, I.: Talking with robots about objects. In: Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction - HRI 2012, p. 479. 7th ACM/IEEE Conference on Human-Robot-Interaction (2012)

    Google Scholar 

  26. Peltason, J., Wrede, B.: Pamini: a framework for assembling mixed-initiative human-robot interaction from generic interaction patterns. In: Proceedings of the SIGDIAL 2010 Conference, pp. 229–232. Association for Computational Linguistics, Tokyo, September 2010. https://www.aclweb.org/anthology/W10-4341

  27. Plane, S., Marvasti, A., Egan, T., Kennington, C.: Predicting perceived age: both language ability and appearance are important. In: Proceedings of SigDial (2018)

    Google Scholar 

  28. Schlangen, D., Skantze, G.: A general, abstract model of incremental dialogue processing. Dialogue Discour. 2, 83–111 (2011). https://pub.uni-bielefeld.de/record/2095091

  29. Skantze, G., Schlangen, D.: Incremental dialogue processing in a micro-domain. In: Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics on EACL 2009 (April), pp. 745–753 (2009). https://doi.org/10.3115/1609067.1609150, http://portal.acm.org/citation.cfm?doid=1609067.1609150

  30. Sun, S., Gong, J., Zomaya, A.Y., Wu, A.: A distributed incremental information acquisition model for large-scale text data. Clust. Comput. 22(1), 2383–2394 (2017). https://doi.org/10.1007/s10586-017-1498-8

    Article  Google Scholar 

  31. Tanenhaus, M.K., Spivey-Knowlton, M.J.: Integration of visual and linguistic information in spoken language comprehension. Science 268(5217), 1632 (1995). https://doi.org/10.1126/science.7777863

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Boise State University, 1910 W University Dr., Boise, ID, 83725, USA

    Mir Tahsin Imtiaz & Casey Kennington

Authors
  1. Mir Tahsin Imtiaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Casey Kennington
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Casey Kennington .

Editor information

Editors and Affiliations

  1. School of Industrial Engineering, Purdue University, West Lafayette, IN, USA

    Vincent G. Duffy

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Imtiaz, M.T., Kennington, C. (2022). Incremental Unit Networks for Distributed, Symbolic Multimodal Processing and Representation. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. Health, Operations Management, and Design. HCII 2022. Lecture Notes in Computer Science, vol 13320. Springer, Cham. https://doi.org/10.1007/978-3-031-06018-2_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06018-2_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06017-5

  • Online ISBN: 978-3-031-06018-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics