Dirk Schnelle-Walka
ABSTRACT
Touch- and voice-based input have emerged as the most popular and relevant interaction modes to enable a natural interaction with computer systems. However, until now, they have mostly been treated separately. In particular, explicit design knowledge on the effective combinations of these modes for an improved user experience is currently not available in a comprehensive form. In this paper, we address this shortage and introduce design patterns which support developers in exploiting the possibilities of combined voice and touch interaction for newly developed systems, so that interaction with these systems becomes more natural for the respective end users.
- Aitenbichler, E. and Schnelle-Walka, D. 2010. An extensible architecture for multitouch & pen interactive tables. In Proceedings of EICS 2010. ACM.Google Scholar
- Apple. Apple iPhone 4 Product Information.Google Scholar
- Asus. Eee PC product information.Google Scholar
- Baggia, P., Burnett, D. C., Carter, J., Dahl, D. A., McCobb, G., and Raggett, D. 2009. EMMA: Extensible MultiModal Annotation markup language.Google Scholar
- Ball, R. and North, C. 2005. Effects of tiled high-resolution display on basic visualization and navigation tasks. In CHI '05: Extended Abstracts of the SIGCHI conference on Human factors in computing systems. ACM, 1196--1199. Google ScholarDigital Library
- Ball, R., North, C., and Bowman, D. A. 2007. Move to improve: promoting physical navigation to increase user performance with large displays. In CHI '07: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 191--200. Google ScholarDigital Library
- Block, M., Link, J., and Phleps, A. 2010. Design patterns for multitouch-TUIs. In Engineering patterns for multi-touch interfaces 2010. ACM, 33--36.Google Scholar
- Bolt, R. A. 1980. "Put-that-there": Voice and gesture at the graphics interface. In SIGGRAPH '80: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques. ACM, 262--270. Google ScholarDigital Library
- Borchers, J. 2001. A Pattern Approach to Interaction Design. John Wiley & Sons, Inc., New York, NY, USA. Google ScholarDigital Library
- Brandl, P., Forlines, C., Wigdor, D., Haller, M., and Shen, C. 2008. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. In Proceedings of the working conference on Advanced visual interfaces. AVI '08. ACM, New York, NY, USA, 154--161. Google ScholarDigital Library
- Buxton, W. Multi-Touch Systems that I Have Known and Loved.Google Scholar
- Cohen, P. and Oviatt, S. 1995. The role of voice input for human-machine communication. Proceedings of the National Academy of Sciences of the United States of America 92, 22, 9921.Google ScholarCross Ref
- Dietz, P. and Leigh, D. 2001. Diamondtouch: a multi-user touch technology. In Proceedings of the 14th annual ACM symposium on User interface software and technology. UIST '01. ACM, New York, NY, USA, 219--226. Google ScholarDigital Library
- Everitt, K., Everitt, K., Forlines, C., Forlines, C., Ryall, K., Ryall, K., Shen, C., and Shen, C. 2004. Observations of a shared tabletop user study. In In Proc. CSCW 2005: Interactive Poster. ACM Press.Google Scholar
- Gamma, E., Helm, R., Johnson, R. E., and Vlissides, J. 1992. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley. Google ScholarDigital Library
- Google. Nexus S Product Information.Google Scholar
- Han, J. Y. 2005. Low-cost multi-touch sensing through frustrated total internal reflection. In UIST '05: Proceedings of the ACM symposium on User interface software and technology. ACM, 115--118. Google ScholarDigital Library
- Hancock, M. S., Shen, C., Forlines, C., and Ryall, K. 2005. Exploring non-speech auditory feedback at an interactive multi-user tabletop. In Proceedings of Graphics Interface 2005. GI '05. Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 41--50. Google ScholarDigital Library
- Hesselmann, T., Fl "oring, S., and Schmitt, M. 2009. Stacked half-pie menus: navigating nested menus on interactive tabletops. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces. ACM, 173--180. Google ScholarDigital Library
- Huggins-Daines, D. and Rudnicky, A. I. 2008. Interactive ASR error correction for touchscreen devices. In HLT '08: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies. Association for Computational Linguistics, Morristown, NJ, USA, 17--19. Google ScholarDigital Library
- Hunter, S. and Maes, P. 2008. WordPlay: A Table-Top Interface for Collaborative Brainstorming and Decision Making. In Proceedings of IEEE Tabletops and Interactive Surfaces. Springer Berlin/Heidelberg.Google Scholar
- Johnston, M. 2009. Building multimodal applications with emma. In Proceedings of the 2009 international conference on Multimodal interfaces. ICMI-MLMI '09. ACM, New York, NY, USA, 47--54. Google ScholarDigital Library
- Khan, A., Matejka, J., Fitzmaurice, G., Kurtenbach, G., Burtnyk, N., and Buxton, B. 2009. Toward the Digital Design Studio: Large Display Explorations. Human-Computer Interaction Volume 24, 1 & 2, 9--47.Google Scholar
- Lepinski, G. J., Grossman, T., and Fitzmaurice, G. 2010. The design and evaluation of multitouch marking menus. In CHI '10: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 2233--2242. Google ScholarDigital Library
- Mandryk, R. L., Scott, S. D., and Inkpen, K. M. 2005. Display Factors Influencing Co-located Collaboration. In CSCW '02: Proceedings of the ACM conference on Computer supported cooperative work.Google Scholar
- Microsoft. Windows Mousegrid.Google Scholar
- Morris, M. R., Morris, D., and Winograd, T. 2004. Individual audio channels with single display groupware: effects on communication and task strategy. In CSCW '04: Proceedings of the ACM conference on Computer supported cooperative work. ACM, 242--251. Google ScholarDigital Library
- Nilsson, E. G. 2009. Design patterns for user interface for mobile applications. Advances in Engineering Software 40, 12, 1318--1328. Google ScholarDigital Library
- Nuance. Dragon NaturallySpeaking Product Information.Google Scholar
- Oviatt, S. 2003. Multimodal interfaces 2nd Ed. Erlbaum, Chapter 21, 413--432.Google Scholar
- Pieraccini, R. and Lubensky, D. 2005. Spoken language communication with machines: the long and winding road from research to business. In Proceedings of the 18th international conference on Innovations in Applied Artificial Intelligence. IEA/AIE'2005. Springer-Verlag, London, UK, 6--15. Google ScholarDigital Library
- Pinelle, D., Gutwin, C., and Greenberg, S. 2003. Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Transactions on Computer-Human Interaction 10, 4, 281--311. Google ScholarDigital Library
- Rauschert, I., Agrawal, P., Sharma, R., Fuhrmann, S., Brewer, I., and MacEachren, A. M. 2002. Designing a human-centered, multimodal GIS interface to support emergency management. In GIS '02: Proceedings of the ACM international symposium on Advances in geographic information systems. ACM, 119--124. Google ScholarDigital Library
- Remy, C., Weiss, M., Ziefle, M., and Borchers, J. 2010. A pattern language for interactive tabletops in collaborative workspaces. In EuroPLoP '10: Proceedings of the 15th European Conference on Pattern Languages of Programs. Irsee Monastery, Germany. Google ScholarDigital Library
- Rossi, G., Schwabe, D., and Lyardet, F. 2000. User interface patterns for hypermedia applications. In AVI '00: Proceedings of the working conference on Advanced visual interfaces. AVI '00. ACM, 136--142. Google ScholarDigital Library
- Schnelle, D. 2008. Context Aware Voice User Interfaces for Workflow Support. Ph.D. thesis, Technische Universität Darmstadt.Google Scholar
- Schnelle, D. and Lyardet, F. 2006. Voice User Interface Designt Patterns. In EuroPLoP 2006 Conference Proceedings.Google Scholar
- Schnelle, D., Lyardet, F., and Wei, T. 2005. Audio navigation patterns. In Proceedings of 10th European Conference on Pattern Languages of Programs (EuroPlop 2005), U. Z. Andy Longshaw, Ed. UVK Universitätsverlag Konstanz, 237--260.Google Scholar
- Schnelle-Walka, D. 2010. A Pattern Language for Error Management in Voice User Interfaces. In EuroPLoP '10: Proceedings of the European Conference on Pattern Languages of Programs. 24. Google ScholarDigital Library
- Suhm, B., Myers, B., and Waibel, A. 2001. Multimodal error correction for speech user interfaces. ACM Trans. Comput.-Hum. Interact. 8, 60--98. Google ScholarDigital Library
- Sulaiman, A. N. and Olivier, P. 2008. Attribute gates. In UIST '08: Proceedings of the annual ACM symposium on User interface software and technology. ACM, 57. Google ScholarDigital Library
- Tan, D. S., Gergle, D., Scupelli, P., and Pausch, R. 2003. With similar visual angles, larger displays improve spatial performance. In CHI '03: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 217--224. Google ScholarDigital Library
- Tešanović, A. 2005. What is a pattern. In Dr.ing. course DT8100 (prev. 78901/45942/DIF8901) Object-oriented Systems. IDA Department of Computer and Information Science, Linköping, Sweden.Google Scholar
- Tse, E., Histon, J., Scott, S., and Greenberg, S. 2004. Avoiding Interference: How People Use Spatial Separation and Partitioning in SDG Workspaces. In CSCW '04: Proceedings of the 2004 ACM conference on Computer supported cooperative work. ACM, Chicago, Illinois, USA, 252--261. Google ScholarDigital Library
- Université Catholique de Louvain. User interface extensible markup language (usixml).Google Scholar
- Weinberg, G. and Harsham, B. 2010. Object-oriented multimodality for safer in-vehicle interfaces. In SiMPE: 5th workshop on speech in mobile and pervasive environments.Google Scholar
- Wobbrock, J. O., Morris, M. R., and Wilson, A. D. 2009. User-defined gestures for surface computing. In Proceedings of the 27th international conference on Human factors in computing systems. CHI '09. ACM, New York, NY, USA, 1083--1092. Google ScholarDigital Library
- Wu, M. and Balakrishnan, R. 2003. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proceedings of the 16th annual ACM symposium on User interface software and technology. UIST '03. ACM, New York, NY, USA, 193--202. Google ScholarDigital Library
- Zhang, X. and Takatsuka, M. 2007. Put That There NOW: Group Dynamics of Tabletop Interaction under Time Pressure. In TABLETOP '07: Proceedings of the IEEE International Workshop on Horizontal Interactive Human Computer Systems. IEEE, 37--43.Google Scholar
Index Terms
- Speech augmented multitouch interaction patterns
Recommendations
Towards standardized distributed multitouch interaction
MindTrek '09: Proceedings of the 13th International MindTrek Conference: Everyday Life in the Ubiquitous EraWe introduce an extension of an existing interaction framework that allows a grouping of input data in multitouch (MT) setups to "hands", "users" and "pens". The data of users and pens is analyzed by an interaction wrapper that informs registered ...
Extended multitouch: recovering touch posture and differentiating users using a depth camera
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technologyMultitouch surfaces are becoming prevalent, but most existing technologies are only capable of detecting the user's actual points of contact on the surface and not the identity, posture, and handedness of the user. In this paper, we define the concept ...
Multitouch Interaction with Parallel Coordinates on Large Vertical Displays
ISSThis paper presents a multitouch vocabulary for interacting with parallel coordinates plots on wall-sized displays. The gesture set relies on principles such as two-finger range definition, a functional distinction of background and foreground for ...
Comments