Abstract
Inspired by a Wizard of Oz (WOZ) simulation experiment, we developed a working prototype of a system that enables users to interact with a map display through synergistic combinations of pen and voice. To address many of the issues raised by multimodal fusion, our implementation employed a distributed multi-agent framework to coordinate parallel competition and cooperation among processing components. Since then, the agent-based infrastructure has been enhanced with a collaboration technology, creating a framework in which multiple humans and automated agents can naturally interact within the same graphical workspace.
Our current endeavour is the leveraging of this architecture to create a unified implementation framework for simultaneously developing both WOZ simulated systems and their fully-automated counterparts. Bootstrapping effects made possible by such an approach are illustrated by an experiment currently under way in our laboratory: as a naive subject draws, writes, and speaks requests to a (simulated) interactive map, a hidden Wizard responds as efficiently as possible using our best fully-automated system, through either standard graphical interface devices or multimodal combinations of pen and voice. The input choices made by both subject and Wizard are invaluable, and the data collected from each can be applied directly to evaluating and improving the automated part of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cheyer, A. (1998) MVIEWS: Multimodal tools for the video analyst. In: Proceedings ofIUI’98, San Francisco (USA), 55–62.
Cheyer A. and Julia L. (1995) Multimodal maps: An agent-based approach. In: Proceedings of the International COnference on Cooperative Multimodal Communication CMC’95, Eindhoven (The Netherlands), May 1995, 103–113.
Cheyer A. and Julia L. (1998) Multimodal maps: An agent-based approach. In: H. Bunt, R.J. Beun, and T. Borghuis (eds.), Multimodal Human-Computer Communication; Systems, Techniques and Experiments. Lecture Notes in Artificial Intelligence 1374, Berlin: Springer, 111–121.
Cohen, P., Cheyer, A., Wang, M., and Baeg, S. (1998) An Open Agent Architecture. In: M.N. Huhns and M.P. Singh (eds.), Readings in Agents, San Francisco: Morgan Kaufmann Publishers, 197–204.
Guzzoni, D., Cheyer, A., Julia, L., and Konolige, K. (1997) Many robots make short work. AI Magazine, Vol. 18, No. 1, Spring 1997, 55–64.
Julia, L. and Cheyer, A. (1997) Speech: a privileged modality. In: Proceedings of EuroSpeech’97, Rhodes (Greece), vol. 4, 1843–1846.
Julia, L., Cheyer, A., Neumeyer, L., Dowding, J., and Charafeddine, M. (1997) http://www.speech.sri.com/demos/atis.html. In: Proceedings of AAAI’97, Stanford (USA), 72–76.
Kehler A., Martin J.C., Cheyer A., Julia L., Hobbs J., and Bear J. (1998) On representing Salience and Reference in Multimodal Human-Computer Interaction. In: Proceedings of AAAI’98 (Representations for Multi-Modal Human-Computer Interaction), Madison (USA), 33–39.
Martin, D., Cheyer, A., and Moran, D. (1999) The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence: An International Journal. (13,1–2).
Martin, D., Cheyer, A., and Moran, D. (1998) Building Distributed Software Systems with the Open Agent Architecture. See http://www.ai.sri.com/~oaa and “Publications”, 1998.
Martin, D., Cheyer, A., and Lee, GL. (1996) Agent development tools for the Open Agent Architecture. In: Proceedings of the International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1996.
Martin, D., Oohama, H., Moran, D., and Cheyer, A. (1997) Information brokering in an agent architecture. In: Proceedings of the Second International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, London, April 1997.
Martin, J.C., Julia, L., and Cheyer, A. (1998) A theoretical framework for multimodal user studies. In: Proceedings of the Second International COnference on Cooperative Multimodal Communication CMC/98, Tilburg (The Netherlands), January 1998, 104–110.
Moore, R., Dowding, J., Bratt, H., Gawron, J.M., and Cheyer, A. (1997) Com-mandTalk: A spoken-language interface for battlefield simulations. In: Proceedings of Fifth Conference on Applied Natural Language Processing, Washington, D.C., April 1997.
Moran, D., Cheyer, A., Julia, L., and Park, S. (1997) Multimodal user interfaces in the Open Agent Architecture. In: Proceedings of IUI’97, Orlando (USA), 61–68.
Oviatt, S. (1996) Multimodal interfaces for dynamic interactive maps. In: Proceedings of CHI’96, April 13–18, 1996. 95–102.
Oviatt, S., De Angeli, A., and Kuhn, K. (1997) Integration and synchronization of input modes during multimodal human-computer interaction. In: Proceedings of the workshop “Referring Phenomena in a Multimedia Context and their Computational Treatment”, ACL/EACL’97 (Madrid), 1–13. Also: http://www.dfki.uni-sb.de/imedia/workshops/mm-references.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cheyer, A., Julia, L., Martin, JC. (2001). A Unified Framework for Constructing Multimodal Experiments and Applications. In: Bunt, H., Beun, R.J. (eds) Cooperative Multimodal Communication. CMC 1998. Lecture Notes in Computer Science(), vol 2155. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45520-5_14
Download citation
DOI: https://doi.org/10.1007/3-540-45520-5_14
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42806-0
Online ISBN: 978-3-540-45520-2
eBook Packages: Springer Book Archive