Jacob O. Wobbrock
Andrew D. Wilson
ABSTRACT
Although mobile, tablet, large display, and tabletop computers increasingly present opportunities for using pen, finger, and wand gestures in user interfaces, implementing gesture recognition largely has been the privilege of pattern matching experts, not user interface prototypers. Although some user interface libraries and toolkits offer gesture recognizers, such infrastructure is often unavailable in design-oriented environments like Flash, scripting environments like JavaScript, or brand new off-desktop prototyping environments. To enable novice programmers to incorporate gestures into their UI prototypes, we present a "$1 recognizer" that is easy, cheap, and usable almost anywhere in about 100 lines of code. In a study comparing our $1 recognizer, Dynamic Time Warping, and the Rubine classifier on user-supplied gestures, we found that $1 obtains over 97% accuracy with only 1 loaded template and 99% accuracy with 3+ loaded templates. These results were nearly identical to DTW and superior to Rubine. In addition, we found that medium-speed gestures, in which users balanced speed and accuracy, were recognized better than slow or fast gestures for all three recognizers. We also discuss the effect that the number of templates or training examples has on recognition, the score falloff along recognizers' N-best lists, and results for individual gestures. We include detailed pseudocode of the $1 recognizer to aid development, inspection, extension, and testing.
- Anderson, D., Bailey, C. and Skubic, M. (2004) Hidden Markov Model symbol recognition for sketch-based interfaces. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 15--21.Google Scholar
- Cao, X. and Balakrishnan, R. (2003) VisionWand: Interaction techniques for large displays using a passive wand tracked in 3D. Proc. UIST '03. New York: ACM Press, 173--182. Google ScholarDigital Library
- Cao, X. and Balakrishnan, R. (2005) Evaluation of an on-line adaptive gesture interface with command prediction. Proc. Graphics Interface '05. Waterloo, Ontario: CHCCS, 187--194. Google ScholarDigital Library
- Cho, M.G. (2006) A new gesture recognition algorithm and segmentation method of Korean scripts for gesture-allowed ink editor. Information Sciences 176 (9), 1290--1303. Google ScholarDigital Library
- Guimbretière, F., Stone, M. and Winograd, T. (2001) Fluid interaction with high-resolution wall-size displays. Proc. UIST '01. New York: ACM Press, 21--30. Google ScholarDigital Library
- Henry, T.R., Hudson, S.E. and Newell, G.L. (1990) Integrating gesture and snapping into a user interface toolkit. Proc. UIST '90. New York: ACM Press, 112--122. Google ScholarDigital Library
- Hinckley, K., Ramos, G., Guimbretiere, F., Baudisch, P. and Smith, M. (2004) Stitching: Pen gestures that span multiple displays. Proc. AVI '04. New York: ACM Press, 23--31. Google ScholarDigital Library
- Hong, J.I. and Landay, J.A. (2000) SATIN: A toolkit for informal ink-based applications. Proc. UIST '00. New York: ACM Press, 63--72. Google ScholarDigital Library
- Kara, L.B. and Stahovich, T.F. (2004) An image-based trainable symbol recognizer for sketch-based interfaces. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 99--105.Google Scholar
- Karlson, A.K., Bederson, B.B. and SanGiovanni, J. (2005) AppLens and LaunchTile: Two designs for one-handed thumb use on small devices. Proc. CHI '05. New York: ACM Press, 201--210. Google ScholarDigital Library
- Kristensson, P. and Zhai, S. (2004) SHARK2: A large vocabulary shorthand writing system for pen-based computers. Proc. UIST '04. New York: ACM Press, 43--52. Google ScholarDigital Library
- Landay, J. and Myers, B.A. (1993) Extending an existing user interface toolkit to support gesture recognition. Adjunct Proc. CHI '93. New York: ACM Press, 91--92. Google ScholarDigital Library
- Lin, J., Newman, M.W., Hong, J.I. and Landay, J.A. (2000) DENIM: Finding a tighter fit between tools and practice for web site design. Proc. CHI '00. New York: ACM Press, 510--517. Google ScholarDigital Library
- Long, A.C., Landay, J.A. and Rowe, L.A. (1999) Implications for a gesture design tool. Proc. CHI '99. New York: ACM Press, 40--47. Google ScholarDigital Library
- Mitchell, T.M. (1997) Machine Learning. New York: McGraw-Hill. Google ScholarDigital Library
- Morris, M.R., Huang, A., Paepcke, A. and Winograd, T. (2006) Cooperative gestures: Multi-user gestural interactions for co-located groupware. Proc. CHI '06. New York: ACM Press, 1201--1210. Google ScholarDigital Library
- Myers, B.A., McDaniel, R.G., Miller, R.C., Ferrency, A.S., Faulring, A., Kyle, B.D., Mickish, A., Klimovitski, A. and Doane, P. (1997) The Amulet environment: New models for effective user interface software development. IEEE Trans. Software Engineering 23 (6), 347--365. Google ScholarDigital Library
- Myers, C.S. and Rabiner, L.R. (1981) A comparative study of several dynamic time-warping algorithms for connected word recognition. The Bell System Technical J. 60 (7), 1389--1409.Google ScholarCross Ref
- Notowidigdo, M. and Miller, R.C. (2004) Off-line sketch interpretation. AAAI Fall Symposium. Menlo Park, CA: AAAI Press, 120--126.Google Scholar
- Pittman, J.A. (1991) Recognizing handwritten text. Proc. CHI '91. New York: ACM Press, 271--275. Google ScholarDigital Library
- Plamondon, R. and Srihari, S.N. (2000) On-line and off-line handwriting recognition: A comprehensive survey. IEEE Trans. Pattern Analysis & Machine Int. 22 (1), 63--84. Google ScholarDigital Library
- Press, W.H., Teukolsky, S.A., Vetterling, W.T. and Flannery, B.P. (1992) Numerical Recipes in C. Cambridge Univ. Press.Google Scholar
- Rubine, D. (1991) Specifying gestures by example. Proc. SIGGRAPH '91. New York: ACM Press, 329--337. Google ScholarDigital Library
- Salvador, S. and Chan, P. (2004) FastDTW: Toward accurate dynamic time warping in linear time and space. 3rd Wkshp. on Mining Temporal and Sequential Data, ACM KDD '04. Seattle, Washington (August 22--25, 2004).Google Scholar
- Sezgin, T.M. and Davis, R. (2005) HMM-based efficient sketch recognition. Proc. IUI '05. New York: ACM Press, 281--283. Google ScholarDigital Library
- Stojmenovi., M., Nayak, A. and Zunic, J. (2006) Measuring linearity of a finite set of points. Proc. CIS '06. Los Alamitos, CA: IEEE Press, 1--6.Google Scholar
- Swigart, S. (2005) Easily write custom gesture recognizers for your Tablet PC applications. Tablet PC Technical Articles.Google Scholar
- Tappert, C.C. (1982) Cursive script recognition by elastic matching. IBM J. of Research & Development 26 (6), 765--771.Google Scholar
- Tappert, C.C., Suen, C.Y. and Wakahara, T. (1990) The state of the art in online handwriting recognition. IEEE Trans. Pattern Analysis & Machine Int. 12 (8), 787--808. Google ScholarDigital Library
- Vermunt, J.K. (1997) Log-linear Models for Event Histories. Thousand Oaks, CA: Sage Publications.Google Scholar
- Wilson, A.D. and Shafer, S. (2003) XWand: UI for intelligent spaces. Proc. CHI '03. New York: ACM Press, 545--552. Google ScholarDigital Library
- Zhai, S. and Kristensson, P. (2003) Shorthand writing on stylus keyboard. Proc. CHI '03. New York: ACM Press, 97--104. Google ScholarDigital Library
Index Terms
- Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes
Recommendations
A lightweight multistroke recognizer for user interface prototypes
GI '10: Proceedings of Graphics Interface 2010With the expansion of pen- and touch-based computing, new user interface prototypes may incorporate stroke gestures. Many gestures comprise multiple strokes, but building state-of-the-art multistroke gesture recognizers is nontrivial and time-consuming. ...
Gestures à Go Go: Authoring Synthetic Human-Like Stroke Gestures Using the Kinematic Theory of Rapid Movements
Special Issue on Causal Discovery and InferenceTraining a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future data. However, recruiting participants, data collection, and labeling, etc., necessary for achieving this goal are ...
G3: bootstrapping stroke gestures design with synthetic samples and built-in recognizers
MobileHCI '16: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services AdjunctStroke gestures are becoming increasingly important with the ongoing success of touchscreen-capable devices. However, training a high-quality gesture recognizer requires providing a large number of examples to enable good performance on unseen, future ...
Comments