Peter Poller
Florian Michahelles
ABSTRACT
Extended Reality (XR) devices have great potential to become the next wave in mobile interaction. They provide powerful, easy-to-use Augmented Reality (AR) and/or Mixed Reality (MR) in conjunction with multimodal interaction facilities using gaze, gesture, and speech. However, current implementations typically lack a coherent semantic representation for the virtual elements, backend-communication, and dialog capabilities. Existing devices are often restricted to mere command and control interactions. To improve these shortcomings and realize enhanced system capabilities and comprehensive interactivity, we have developed a flexible modular approach that integrates powerful back-end platforms using standard API interfaces. As a concrete example, we present our distributed implementation of a multimodal dialog system on the Microsoft Hololens®. It uses the SiAM-dp multimodal dialog platform as a back-end service and an Open Semantic Framework (OSF) back-end server to extract the semantic models for creating the dialog domain model.
- Simon Mayer, Jack Hodges, Dan Yu, Mareike Kritzler, and Florian Michahelles. 2017. An Open Semantic Framework for the Industrial Internet of Things. IEEE Intelligent Systems 32, 1 (1 1 2017), 96–101. https://doi.org/10.1109/MIS.2017.9Google Scholar
Digital Library
- Florian Michahelles, Andrei Ciortea, Kimberly García, and Markus Funk. 2017. Combining semantics and augmented reality to support the human mind. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. ACM, 697–703.Google ScholarDigital Library
- Robert Neßelrath. 2015. SiAM-dp : An open development platform for massively multimodal dialogue systems in cyber-physical environments. Ph.D. Dissertation. Universität des Saarlandes, Postfach 151141, 66041 Saarbrücken.Google Scholar
- Peter Poller, Margarita Chikobava, Jack Hodges, Mareike Kritzler, Michahelles Florian, and Tilman Becker. 2019. HoloLens Dialog with SiAM-dp. https://www.youtube.com/watch?v=0g-QyNvVJUI&feature=youtu.beGoogle Scholar
- Alexander Prange, Margarita Chikobava, Peter Poller, Michael Barz, and Daniel Sonntag. 2017. A Multimodal Dialogue System for Medical Decision Support inside Virtual Reality. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. 23–26.Google ScholarCross Ref
- Christopher Reardon, Kevin Lee, John G Rogers, and Jonathan Fink. 2019. Augmented Reality for Human-Robot Teaming in Field Environments. In International Conference on Human-Computer Interaction. Springer, 79–92.Google Scholar
- Peer Schütt, Max Schwarz, and Sven Behnke. 2019. Semantic Interaction in Augmented Reality Environments for Microsoft HoloLens. In 2019 European Conference on Mobile Robots, ECMR 2019, Prague, Czech Republic, September 4-6, 2019. 1–6. https://doi.org/10.1109/ECMR.2019.8870937Google ScholarCross Ref
Index Terms
- Back-end semantics for multimodal dialog on XR devices
Recommendations
Extended Reality (XR) Toward Building Immersive Solutions: The Key to Unlocking Industry 4.0
When developing XR applications for Industry 4.0, it is important to consider the integration of visual displays, hardware components, and multimodal interaction techniques that are compatible with the entire system. The potential use of multimodal ...
From vocal to multimodal dialogue management
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesMultimodal, speech-enabled systems pose different research problems when compared to unimodal, voice-only dialogue systems. One of the important issues is the question of how a multimodal interface should look like in order to make the multimodal ...
Human-robot collaborative tutoring using multiparty multimodal spoken dialogue
HRI '14: Proceedings of the 2014 ACM/IEEE international conference on Human-robot interactionIn this paper, we describe a project that explores a novel experimental setup towards building a spoken, multi-modally rich, and human-like multiparty tutoring robot. A human-robot interaction setup is designed, and a human-human dialogue corpus is ...
Comments