Tue Haste Andersen
Shumin Zhai
Abstract
We investigate the use of auditory feedback in pen-gesture interfaces in a series of informal and formal experiments. Initial iterative exploration showed that gaining performance advantage with auditory feedback was possible using absolute cues and state feedback after the gesture was produced and recognized. However, gaining learning or performance advantage from auditory feedback tightly coupled with the pen-gesture articulation and recognition process was more difficult. To establish a systematic baseline, Experiment 1 formally evaluated gesture production accuracy as a function of auditory and visual feedback. Size of gestures and the aperture of the closed gestures were influenced by the visual or auditory feedback, while other measures such as shape distance and directional difference were not, supporting the theory that feedback is too slow to strongly influence the production of pen stroke gestures. Experiment 2 focused on the subjective aspects of auditory feedback in pen-gesture interfaces. Participants' rating on the dimensions of being wonderful and stimulating was significantly higher with musical auditory feedback. Several lessons regarding pen gestures and auditory feedback are drawn from our exploration: a few simple functions such as indicating the pen-gesture recognition results can be achieved, gaining performance and learning advantage through tightly coupled process-based auditory feedback is difficult, pen-gesture sets and their recognizers can be designed to minimize visual dependence, and people's subjective experience of gesture interaction can be influenced using musical auditory feedback. These lessons may serve as references and stepping stones toward future research and development in pen-gesture interfaces with auditory feedback.
- Algazi, V. R., Duda, R. O., Thompson, D. M., and Avendano, C. 2001. The cipic hrtf database. In Proceedings of the Workshop on Applications of Signal Processing to Audio and Electroacoustics. IEEE, Los Alamitos, CA, 99--102.Google Scholar
- Blattner, M., Sumikawa, D., and Greenberg, R. 1990. Earcons and icons: Their structure and common design principles. Hum. Comput. Interact. 16, 523--531.Google Scholar
- Blythe, M., Monk, A., Overbeeke, C., and Wright, P., Eds. 2003. Funology: From Usability to User Enjoyment. Kluwer, Dordrecht, The Netherlands. Google ScholarDigital Library
- Brewster, S., Lumsden, J., and et.al., M. B. 2003. Multimodal 'eyes-free' interaction techniques for wearable devices. In Proceedings of the Conference on Human Factor in Computing Systems. ACM, New York, 473--480. Google ScholarDigital Library
- Broadbent, D. 1977. The hidden pre-attentive processes. Am. Psychol. 32, 2, 109--118.Google ScholarCross Ref
- Buxton, W. 1995. Readings in Human Computer Interaction: Towards the Year 2000. Morgan Kaufmann Publishers, San Francisco, 525.Google Scholar
- Chin, J., Diehl, V., and Norman, K. 1988. Development of an instrument measuring user satisfaction of the human-computer interface. In Proceedings of the Conference on Human Factor in Computing Systems. ACM, New York, 213--218. Google ScholarDigital Library
- Clarke, E. 1999. Rhythm and timing in music. In The Psychology of Music 2nd Ed. Deutsch, D. Ed., Academic Press, San Diego, CA, 473--500.Google Scholar
- Deatherage, B. H. 1972. Human Engineering Guide to Equipment Design. Revised Ed. John Wiley and Sons, New York, NJ, 123--160.Google Scholar
- Fletcher, H., Blackham, E., and Stratton, R. 1962. Quality of piano tones. J. Acoust. Soc. Am. 34, 6, 749--761.Google ScholarCross Ref
- Gaver, W. 1989. The SonicFinder: An interface that uses auditory icons. Hum. Comput. Interact. 4, 1, 67--94. Google ScholarDigital Library
- Ghez, C., Rikakis, T., DuBois, R. L., and Cook, P. 2000. An auditory display system for aiding interjoint coordination. In Proceedings of the International Conference on Auditory Display. ACM, New York.Google Scholar
- Goldberg, D. and Richardson, C. 1993. Touch-typing with a stylus. In Proceedings of the Conference on Human Factor in Computing Systems. ACM, New York, 80--87. Google ScholarDigital Library
- Järveläinen, H., Välimäki, V., and Karjalainen, M. 2001. Audibility of the timbral effects of inharmonicity in stringed instrument tones. Acoust. Res. Lett. Online 2, 79--84.Google ScholarCross Ref
- Kramer, G. 1994. Auditory Display: Sonification, Audification, and Auditory Interface. Addison-Wesley, Reading, MA, 1--77. Google ScholarDigital Library
- Kristensson, P.-O. and Zhai, S. 2004. Shark2: A large vocabulary shorthand writing system for pen-based computers. In Proceedings of the Symposium on User Interface Software and Technology. ACM, New York, 43--52. Google ScholarDigital Library
- Kurtenbach, G. and Buxton, W. 1994. User learning and performance with marking menus. In Proceedings of the Conference on Human Factor in Computing Systems. ACM, New York, 258--264. Google ScholarDigital Library
- Lacuaniti, F., Terzuolo, C., and Viviani, P. 1983. The law relating the kinematic and figural aspects of drawing movements. Acta Psychologica 54, 115--130.Google ScholarCross Ref
- Legge, D., Steinberg, H., and Summerfield, A. 1964. Simple measures of handwriting as indices of drug effects. Percept. Motor Skills 18, 549--558.Google ScholarCross Ref
- Loeb, R. and Fitch, W. 2002. A laboratory evaluation of an auditory display designed to enhance intraoperative monitoring. Anesthesia Analgesia 94, 362--368.Google Scholar
- Luschei, E., Saslow, C., and Glickstein, M. 1967. Muscle potentials in reaction time. Exp. Neurol. 18, 429--442.Google ScholarCross Ref
- MacKenzie, I. 2002. Kspc (keystrokes per character) as a characteristic of text entry techniques. In Proceedings of the Conference on Mobile Human Computer Interaction. Springer-Verlag, Berlin, 195--210. Google ScholarDigital Library
- MacMillan, K., Droettboom, M., and Fujinaga, I. 2001. Audio latency measurements of desktop operating systems. In Proceedings of the International Computer Music Conference. 259--262.Google Scholar
- Norman, D. 2004. Emotional Design: Why We Love (or Hate) Everyday Things. Basic Books, New York.Google ScholarDigital Library
- Olive, T. and Piolat, A. 2002. Suppressing visual feedback in written composition: Effects on processing demands and coordination of the writing processes. Int. J. Psych. 37, 4, 209--218.Google ScholarCross Ref
- Patterson, R. 1989. Guidelines for the design of auditory warning sounds. In Proceedings of the Institute of Acoustics Spring Conference. Vol. 2. Institute of Acoustics, 17--24.Google Scholar
- Pirhonen, A., Brewster, S., and Holguin, C. 2002. Gestural and audio metaphors as a means of control for mobile devices. In Proceedings of Conference on Human Factor in Computing Systems. ACM, New York, 291--298. Sears, A. and Arora, R. 2002. Data entry for mobile devices: An empirical comparison of novice performance with jot and graffiti. Interact. Comput. 14, 413--433. Google ScholarDigital Library
- Shea, C., Wulf, G., Park, J., and Gaunt, B. 2001. Effects of an auditory model on the learning of relative and absolute timing. J. Motor Behav. 33, 2, 127--138.Google ScholarCross Ref
- Singh, P. 1987. Perceptual organization of complex-tone sequences: A tradeoff between pitch and timbre? J. Acoust. Soc. Am. 82, 886--899.Google ScholarCross Ref
- Smyth, M. and Silvers, G. 1987. Functions of vision in the control of handwriting. Acta Psychologica 66, 47--64.Google ScholarCross Ref
- Teulings, H. L. and Schomaker, L. 1993. Invariant properties between stroke features in handwriting. Acta Psychologica 82, 69--88.Google ScholarCross Ref
- Tractinsky, N., Katz, A., and Ikar, D. 2000. What is beautiful is usable. Interact. Comput. 13, 127--145.Google ScholarCross Ref
- van Doorn, R. and Keuss, P. 1992. The role of vision in the temporal and spatial control of handwriting. Acta Psychologica 81, 269--286.Google ScholarCross Ref
- van Doorn, R. and Keuss, P. 1993. Does production of letter strokes in handwriting benefit from vision? Acta Psychologica 82, 275--290.Google Scholar
- Welch, R. 1999. Meaning, attention, and the “unity assumption” in the intersensory bias of spatial and temporal perceptions. In Cognitive Contributions to the Perception of Spatial and Temporal Events, Elsevier.Google Scholar
- Wobbrock, J. O., Myers, B. A., and Kembel, J. A. 2003. Edgewrite: A stylus-based text entry method designed for high accuracy and stability of motion. In Proceedings of the ACM Symposium on User Interface Software and Technology. ACM, New York, 61--70. Google ScholarDigital Library
- Woods, D. 1995. The alarm problem and directed attention in dynamic fault management. Ergonomics 38, 11, 2371--2394.Google ScholarCross Ref
- Wright, C. E. 1990. Generalized motor programs: Reexamining claims of effector independence in writing. In Attention and Performance XIII, Erlbaum, Hillsdale, NJ, 294--320.Google Scholar
- Yost, W. A. 1994. Fundamentals of Hearing: An Introduction 3rd Ed. Academic Press, New York.Google Scholar
- Zhai, S. and Kristensson, P.-O. 2003. Shorthand writing on stylus keyboard. InProceedings of the Conference on Human Factor in Computing Systems. ACM, New York, 97--104. Google ScholarDigital Library
Index Terms
- “Writing with music”: Exploring the use of auditory feedback in gesture interfaces
Recommendations
De-Mo: designing action-sound relationships with the mo interfaces
CHI EA '13: CHI '13 Extended Abstracts on Human Factors in Computing SystemsThe Modular Musical Objects (MO) are an ensemble of tangible interfaces and software modules for creating novel musical instruments or for augmenting objects with sound. In particular, the MOs allow for designing action-sound relationships and behaviors ...
Form Follows Sound: Designing Interactions from Sonic Memories
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsSonic interaction is the continuous relationship between user actions and sound, mediated by some technology. Because interaction with sound may be task oriented or experience-based it is important to understand the nature of action-sound relationships ...
GlanceWriter: Writing Text by Glancing Over Letters with Gaze
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing SystemsWriting text with eye gaze only is an appealing hands-free text entry method. However, existing gaze-based text entry methods introduce eye fatigue and are slow in typing speed because they often require users to dwell on letters of a word, or mark the ...
Comments