David R. McGee
Philip R. Cohen
ABSTRACT
In this paper we describe how we have enhanced our multimodal paper-based system, Rasa, with visual perceptual input. We briefly explain how Rasa improves upon current decision-support tools by augmenting, rather than replacing, the paper-based tools that people in command and control centers have come to rely upon. We note shortcomings in our initial approach, discuss how we have added computer-vision as another input modality in our multimodal fusion system, and characterize the advantages that it has to offer. We conclude by discussing our current limitations and the work we intend to pursue to overcome them in the future.
- Gorman, P., Ash, J., Lavelle, M., Lyman, J., Delcambre, L., and Maier, D. Bundles in the Wild: Managing Information to Solve Problems and Maintain Situation Awareness. Library Trends, 49(2 2000): 266--289.Google Scholar
- Heath, C. and Luff, P. Technology in Action. Learning in Doing: Social, cognitive and computational perspectives, R. Pea, J. S. Brown, and C. Heath (Eds.). Cambridge, UK: Cambridge University Press, 2000.Google Scholar
- Huber, M. J., Kumar, S., Cohen, P. R., and McGee, D. R. A formal semantics for proxy communicative acts, in the Proceedings of the Eighth International Workshop on Agent Theories, Architectures, and Languages (ATAL-2001) (Seattle, WA, Aug. 1-3 2001). Google ScholarDigital Library
- Ishii, H. and Ullmer, B. Tangible bits: towards seamless interfaces between people, bits and atoms, in the Proceedings of the Conference on Human Factors in Computing Systems (Atlanta, GA, March 1997), ACM Press, 234--241. Google ScholarDigital Library
- Johnston, M. Unification-based multimodal parsing, in the Proceedings of the International Joint Conference of the Association for Computational Linguistics and the International Committee on Computational Linguistics (Montreal, Canada, August 1998), Association for Computational Linguistics Press, 624--630. Google ScholarDigital Library
- Klemmer, S. R., Newman, M. W., Farrell, R., Bilezikjian, M., and Landay, J. A. The Designer's Outpost: A tangible interface for collaborative web site design, in the Proceedings of the Symposium on User Interface Software and Technology (UIST'01) (Orlando, FL, Nov. 11--14 2001), ACM Press. Google ScholarDigital Library
- Kumar, S., Cohen, P. R., and Levesque, H. J. The Adaptive Agent Architecture: Achieving Fault-Tolerance Using Persistent Broker Teams, in the Proceedings of the International Conference on Multi-Agent Systems (Boston, MA, July 7--12 2000. Google ScholarDigital Library
- Mackay, W. E. Is paper safer? The role of flight strips in air traffic control. ACM Transactions on Computer-Human Interaction, 6(4 1999): 311--340. Google ScholarDigital Library
- Mackay, W. E., Fayard, A.-L., Frobert, L., and Médini, L. Reinventing the familiar: Exploring an augmented reality design space for air traffic control, in the Proceedings of the Conference on Human Factors in Computing Systems (Los Angeles, CA, April 18--23 1998), ACM Press, 558--565. Google ScholarDigital Library
- McGee, D. R. and Cohen, P. R. Creating tangible interfaces by transforming physical objects with multimodal language, in the Proceedings of the International Conference on Intelligent User Interfaces (Santa Fe, NM, Jan. 14-17 2001), ACM Press, 113--119. Google ScholarDigital Library
- McGee, D. R., Cohen, P. R., Wesson, R. M., and Horman, S. Comparing paper and tangible, multimodal tools, inn submission.Google Scholar
- McGee, D. R., Cohen, P. R., and Wu, L. Something from nothing: Augmenting a paper-based work practice with multimodal interaction, in the Proceedings of the Conference on Designing Augmented Reality Environments (Helsingor, Denmark, April 12--14 2000), ACM Press, 71--80. Google ScholarDigital Library
- McGee, D. R., Pavel, M., Adami, A., Wang, G., and Cohen, P. R. A visual modality for the augmentation of paper, in the Proceedings of the Workshop on Perceptive User Interfaces (PUI'01) (Orlando, FL, Nov. 15--16 2001), ACM Press. Google ScholarDigital Library
- Moran, T. P., Saund, E., Melle, W. v., Bryll, R., Gujar, A. U., Fishkin, K. P., and Harrison, B. L. The ins and outs of collaborative walls: Demonstrating the Collaborage concept, in the Proceedings of the Conference on Human Factors in Computing Systems (Pittsburgh, PA, May 15--20 1999), ACM Press, CHI'99 Extended Abstracts, 192--193. Google ScholarDigital Library
- Oviatt, S. L. Multimodal interfaces for dynamic interactive maps, in the Proceedings of the Conference on Human Factors in Computing Systems (1996), ACM Press, 95--102. Google ScholarDigital Library
- Shepard, R. and Meltzer, J. Mental rotations of three-dimensional objects. Science, 17(1 1971): 701--3.Google Scholar
- Ullmer, B. and Ishii, H. The metaDESK: models and prototypes for tangible user Interfaces, in the Proceedings of the Symposium on User Interface Software and Technology (Banff, Alberta, Canada, October 1997), ACM Press, 223--232. Google ScholarDigital Library
- Underkoffler, J. and Ishii, H. Urp: a luminous-tangible workbench for urban planning and design, in the Proceedings of the Conference on Human Factors in Computing Systems (Pittsburgh, PA, May 1999), ACM Press, 386--393. Google ScholarDigital Library
- Wellner, P. The DigitalDesk calculator: tangible manipulation on a desktop display, in the Proceedings of the Symposium on User Interface Software and Technology (Hilton Head, SC, November 1991), ACM Press, 27--33. Google ScholarDigital Library
- Wu, L., Oviatt, S., and Cohen, P. Multimodal integration - A statistical view. IEEE Transactions on Multimedia, 1(4 1999): 334--341. Google ScholarDigital Library
Recommendations
Facilitating multiparty dialog with gaze, gesture, and speech
ICMI-MLMI '10: International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal InteractionWe study how synchronized gaze, gesture and speech rendered by an embodied conversational agent can influence the flow of conversations in multiparty settings. We begin by reviewing a computational framework for turn-taking that provides the foundation ...
From vocal to multimodal dialogue management
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesMultimodal, speech-enabled systems pose different research problems when compared to unimodal, voice-only dialogue systems. One of the important issues is the question of how a multimodal interface should look like in order to make the multimodal ...
Multimodal interaction systems: information and time features
Multimodal interaction systems combine visual information (involving images, text, sketches and so on) with voice, gestures and other modalities to provide flexible and powerful dialogue approaches, enabling users to choose one or more of the multiple ...
Comments