David R. McGee
Philip R. Cohen
ABSTRACT
In command posts, officers maintain situational awareness using paper maps, Post-it notes, and hand-written annotations. They do so because paper is robust to failure, it is portable, it offers a flexible means of capturing information, it has ultra-high resolution, and it readily supports face-to-face collaboration. We report herein on an evaluation comparing maps and Post-its with a tangible multimodal system called Rasa. Rasa augments these paper tools with sensors, enabling it to recognize the multimodal language (both written and spoken) that naturally occurs on them. In this study, we found that not only do users prefer Rasa to paper alone, they find it as easy or easier to use than paper tools. Moreover, Rasa introduces no discernible overhead in its operation other than error repair, yet grants the benefits inherent in digital systems. Finally, subjects confirmed that by combining physical and computational tools, Rasa is resistant to computational failure
- Bolt, R.A., Put-That-There: Voice and gesture at the graphics interface. Computer Graphics, 14(3) (1980): 262--270. Google Scholar
Digital Library
- Cohen, P.R., Johnston, M., McGee, D.R., Oviatt, S., Pittman, J., Smith, I., Chen, L., and Clow, J. QuickSet: multimodal interaction for distributed applications, in the Proceedings of the International Multimedia Conference, ACM Press, (Seattle, WA, November 1997), 31--40. Google ScholarDigital Library
- Cohen, P.R., McGee, D.R., and Clow, J. The efficiency of multimodal interaction, in the Proceedings of the Applied Natural Language Programming Conference, Association for Computational Linguistics Press, (Seattle, WA, April 2000). Google ScholarDigital Library
- Gorman, P., Ash, J., Lavelle, M., Lyman, J., Delcambre, L., and Maier, D., Bundles in the Wild: Managing Information to Solve Problems and Maintain Situation Awareness. Library Trends, 49(2) (2000): 266--289.Google Scholar
- Heath, C. and Luff, P., Technology in Action. Learning in Doing: Social, Cognitive and Computational Perspectives, R. Pea, J.S. Brown, and C. Heath (Eds.). Cambridge, UK: Cambridge University Press, (2000). Google ScholarDigital Library
- Johnston, M. Unification-based multimodal parsing, in the Proceedings of the International Joint Conference of the Association for Computational Linguistics and the International Committee on Computational Linguistics, Association for Computational Linguistics Press, (Montreal, Canada, August 1998), 624--630. Google ScholarDigital Library
- Mackay, W.E., Is paper safer? The role of flight strips in air traffic control. ACM Transactions on Computer-Human Interaction, 6(4) (1999): 311--340. Google ScholarDigital Library
- McGee, D.R. and Cohen, P.R. Creating tangible interfaces by transforming physical objects with multimodal language, in the Proceedings of the International Conference on Intelligent User Interfaces, ACM Press, (Santa Fe, NM, Jan. 14--17 2001), 113--119. Google ScholarDigital Library
- McGee, D.R., Cohen, P.R., and Wu, L. Something from nothing: Augmenting a paper-based work practice with multimodal interaction, in the Proceedings of the Conference on Designing Augmented Reality Environments, ACM Press, (Helsingor, Denmark, April 12--14 2000), 71--80. Google ScholarDigital Library
- McGee, D.R., Pavel, M., Adami, A., Wang, G., and Cohen, P.R. A visual modality for the augmentation of paper, in the Proceedings of the Workshop on Perceptive User Interfaces (PUI'01), ACM Press, (Orlando, FL, Nov. 15--16 2001). Google ScholarDigital Library
- Wellner, P. The DigitalDesk calculator: tangible manipulation on a desktop display, in the Proceedings of the Symposium on User Interface Software and Technology, ACM Press, (Hilton Head, SC, November 1991), 27--33. Google ScholarDigital Library
- Wu, L., Oviatt, S.L., and Cohen, P.R., From members to teams to committee -- a robust recognition approach to gestural and multimodal signal processing. IEEE Transactions on Neural Networks, (special issue on Intelligent Multimedia) (2002): in press. Google ScholarDigital Library
Index Terms
- Comparing paper and tangible, multimodal tools
Recommendations
Creating tangible interfaces by augmenting physical objects with multimodal language
IUI '01: Proceedings of the 6th international conference on Intelligent user interfacesRasa is a tangible augmented reality environment that digitally enhances the existing paper-based command and control capability in a military command post. By observing and understanding the users' speech, pen, and touch-based multimodal language, Rasa ...
Something from nothing: augmenting a paper-based work practice via multimodal interaction
DARE '00: Proceedings of DARE 2000 on Designing augmented reality environmentsIn this paper, we describe Rasa: an environment designed to augment, rather than replace, the work habits of its users. These work habits include drawing on Post-it™ notes using a symbolic language. Rasa observes and understands this language, assigning ...
Natural and tangible human-computer interfaces for augmented environments
SIGDOC '08: Proceedings of the 26th annual ACM international conference on Design of communicationIn this talk, the author will introduce Augmented Reality as a new media to improve human-computer communication and industrial processes. The author will make a special emphasis in introducing and demonstrating new human-computer interfaces appropriate ...
Comments