Chris Harrison
Robert Xiao
Julia Schwarz
Scott E. Hudson
ABSTRACT
The average person can skillfully manipulate a plethora of tools, from hammers to tweezers. However, despite this remarkable dexterity, gestures on today's touch devices are simplistic, relying primarily on the chording of fingers: one-finger pan, two-finger pinch, four-finger swipe and similar. We propose that touch gesture design be inspired by the manipulation of physical tools from the real world. In this way, we can leverage user familiarity and fluency with such tools to build a rich set of gestures for touch interaction. With only a few minutes of training on a proof-of-concept system, users were able to summon a variety of virtual tools by replicating their corresponding real-world grasps.
Supplemental Material
- Apple Computer, Inc. Multi-touch gesture dictionary (2008). US Patent 20070177803.Google Scholar
- Bartindale, T. Harrison, C., Olivier, P. L., and Hudson, S. SurfaceMouse: Supplementing Multi-Touch Interaction with a Virtual Mouse. In Proc. TEI '11. 293--296. Google ScholarDigital Library
- Baudel, T., and Beaudouin-Lafon, M. Charade: remote control of objects using free-hand gestures. Comm. ACM, 36(7). 28--35. Google ScholarDigital Library
- Cao, X., A. Wilson, R. Balakrishnan, K. Hinckley, S. Hudson. ShapeTouch: Leveraging contact shape on interactive surfaces. In Proc. ITS '08. 129--136.Google Scholar
- Epps, J., Lichman, S. and Wu, M. A study of hand shape use in tabletop gesture interaction. In CHI EA '06. 748--753. Google ScholarDigital Library
- Erol, A., Bebis, G., Nicolescu, M., Boyle, R., and Twombly, X. Vision-based hand pose estimation: A review. Computer Vision and Image Understanding, 108(1), 2007, 52--73. Google ScholarDigital Library
- Ewerling, P., Kulik, A., and Froehlich, B. Finger and hand detection for multi-touch interfaces based on maximally stable extremal regions. In Proc. ITS '12. 173--182. Google ScholarDigital Library
- Freeman, D., Benko, H., Morris, M. and Wigdor, D. ShadowGuides: Visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proc. ITS '09. 165--172. Google ScholarDigital Library
- Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, D., Wilson, A., Benko, H. and Buxton, B. Pen + touch = new tools. In Proc. UIST '10. 27--36. Google ScholarDigital Library
- Holz, C. and Baudisch, P. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proc. CHI '10. 581--590. Google ScholarDigital Library
- Kratz, S., Rohs, M., Guse, D., Müller, J., Bailly, G. and Nischt, M. PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices. In Proc. AVI '12. 181--188. Google ScholarDigital Library
- MacKenzie, C. L., and Iberall, T. (1994). The Grasping Hand. Advances in Psychology, Vol. 104. North Holland, Elsevier Science B.V. Amsterdam, The Netherlands.Google Scholar
- Marquardt, N., Kiemer, J., Ledo, D., Boring, S. and Greenberg, S. Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit. In Proc. ITS '11. 21--30. Google ScholarDigital Library
- Murugappan, S., Vinayak, Elmqvist, N. and Ramani, K. Extended multitouch: recovering touch posture and differentiating users using a depth camera. In Proc. UIST '12. 487--496. Google ScholarDigital Library
- Patkin, M. (1981). Ergonomics and the Surgeon. Clinical Science for Surgeons. Butterworth. London, UK.Google Scholar
- Poupyrev, I., Weghorst, S., Billinghurst, M., Ichikawa, T., A framework and testbed for studying manipulation techniques for immersive VR. In Proc. VRST '97. 21--28. Google ScholarDigital Library
- Rekimoto, J. SmartSkin: An infrastructure for free-hand manipulation on interactive surfaces. In Proc. CHI '02. 113--120. Google ScholarDigital Library
- Rogers, S., Williamson, J., Stewart, C., and Murray-Smith, R. FingerCloud: uncertainty and autonomy handover in capacitive sensing. In Proc. CHI '10. 577--580. Google ScholarDigital Library
- Schmidt, D., Chong, M.K. and Gellersen, H. HandsDown: hand-contour-based user identification for interactive surfaces. In Proc. NordiCHI '10. 432--441. Google ScholarDigital Library
- Weiss, M., Wagner, J., Jansen, Y., Jennings, R., Khoshabeh, R., Hollan, J.D. and Borchers, J. SLAP widgets: bridging the gap between virtual and physical controls on tabletops. In Proc. CHI '09. 481--490. Google ScholarDigital Library
- Wigdor, D., Benko, H., Pella, J., Lombardo, J. and Williams, S. Rock & rails: extending multi-touch interactions with shape gestures to enable precise spatial manipulations. In Proc. CHI '11. 1581--1590. Google ScholarDigital Library
- Wilson, A. Simulating grasping behavior on an imaging interactive surface. In Proc. ITS '09. 125--132. Google ScholarDigital Library
- Wilson, A., Izadi, S., Hilliges, O., Garcia-Mendoza, A. and Kirk, D. Bringing physics to the surface. In Proc. UIST '08. 67--76. Google ScholarDigital Library
- Wilson, F. (1998). The Hand: How Its Use Shapes the Brain, Language, and Human Culture. Pantheon Books, New York.Google Scholar
- Wobbrock, J., Morris, M., and Wilson, A. User-defined gestures for surface computing. In Proc. CHI '09. 1083--1092. Google ScholarDigital Library
- Wu, M. and Balakrishnan, R. Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In Proc. UIST '03. 193--202. Google ScholarDigital Library
- Wu, M., Shen C., Ryall, K., Forlines, C. and Balakrishna, R. Gesture Registration, Relaxation, and Reuse for Multi-Point Direct-Touch Surfaces, In Proc. Tabletop '06. 185--192. Google ScholarDigital Library
- Wynn-Parry, C. B. (1981). Rehabilitation of the Hand. Butterworths, London, UK.Google Scholar
Index Terms
- TouchTools: leveraging familiarity and skill with physical tools to augment touch interaction
Recommendations
Combining Touchscreens with Passive Rich-ID Building Blocks to Support Context Construction in Touchscreen Interactions
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsThis research investigates the design space of combining touchscreens with passive rich-ID building block systems to support the physical construction of contexts in touchscreen interactions. With two proof-of-concept systems, RFIPillars and RFITiles, ...
Comparing elicited gestures to designer-created gestures for selection above a multitouch surface
ITS '12: Proceedings of the 2012 ACM international conference on Interactive tabletops and surfacesMany new technologies are emerging that make it possible to extend interaction into the three-dimensional space directly above or in front of a multitouch surface. Such techniques allow people to control these devices by performing hand gestures in the ...
Multi-touch 3D positioning with the pantograph technique
I3D '19: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and GamesOne advantage of touch interaction is the sense of direct manipulation; there is perhaps no more-intuitive interface than just reaching out and touching virtual entities. However, direct manipulation is generally limited to objects located on the 2D ...
Comments