Keith Vertanen
Per Ola Kristensson
Skip Abstract Section
Abstract
A common methodology for evaluating text entry methods is to ask participants to transcribe a predefined set of memorable sentences or phrases. In this article, we explore if we can complement the conventional transcription task with a more externally valid composition task. In a series of large-scale crowdsourced experiments, we found that participants could consistently and rapidly invent high quality and creative compositions with only modest reductions in entry rates. Based on our series of experiments, we provide a best-practice procedure for using composition tasks in text entry evaluations. This includes a judging protocol which can be performed either by the experimenters or by crowdsourced workers on a microtask market. We evaluated our composition task procedure using a text entry method unfamiliar to participants. Our empirical results show that the composition task can serve as a valid complementary text entry evaluation method.
- Tamara Broderick and David J. C. MacKay. 2009. Fast and Flexible Selection with a Single Switch. PLoS ONE 4, 10 (10 2009), e7481.Google Scholar
- Steven J. Castellucci and Scott I. MacKenzie. 2008. Graffiti vs. unistrokes: an empirical comparison. In CHI’08: Proceedings of the SIGCHI Conference on Human factors in Computing Systems. ACM Press, 305--308. Google ScholarDigital Library
- Edward Clarkson, James Clawson, Kent Lyons, and Thad Starner. 2005. An empirical study of typing rates on mini-QWERTY keyboards. In CHI’05: Extended abstracts on Human Factors in Computing Systems. ACM Press, 1288--1291. Google ScholarDigital Library
- Alan Dix. 2010. Human-computer interaction: A stable discipline, a nascent science, and the growth of the long tail. Interacting with Computers 22 (January 2010), 13--27. Issue 1. Google ScholarDigital Library
- Mark D. Dunlop and Michelle Montgomery Masters. 2009. Pickup Usability Dominates: A Brief History of Mobile Text Entry Research and Adoption. International Journal of Mobile Human Computer Interaction 1, 1 (2009), 42--59.Google ScholarCross Ref
- S. G. Hart and L. E. Stavenland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Human Mental Workload, P. A. Hancock and N. Meshkati (Eds.). Elsevier, Chapter 7, 139--183.Google Scholar
- Jeffrey Heer and Michael Bostock. 2010. Crowdsourcing graphical perception: using mechanical turk to assess visualization design. In CHI’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 203--212. Google ScholarDigital Library
- Poika Isokoski and Timo Linden. 2004. Effect of foreign language on text transcription performance: Finns writing English. In NordiCHI’04: Proceedings of the Third Nordic Conference on Human-Computer Interaction. ACM Press, 109--112. Google ScholarDigital Library
- Poika Isokoski and Roope Raisamo. 2000. Device independent text input: a rationale and an example. In AVI’00: Proceedings of the Working Conference on Advanced Visual Interfaces. ACM Press, 76--83. Google ScholarDigital Library
- Akiyo Kano, Janet C. Read, and Alan Dix. 2006. Children’s phrase set for text input method evaluations. In NordiCHI’06: Proceedings of the Third Nordic Conference on Human-Computer Interaction. ACM Press, 449--452. Google ScholarDigital Library
- Clare-Marie Karat, Christine Halverson, Daniel Horn, and John Karat. 1999. Patterns of Entry and Correction in Large Vocabulary Continuous Speech Recognition Systems. In CHI’99: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 568--575. Google ScholarDigital Library
- Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In CHI’09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 453--456. Google ScholarDigital Library
- Bryan Klimt and Yiming Yang. 2004. The Enron Corpus: A New Dataset for Email Classification Research. In European Conference on Machine Learning. 217--226.Google Scholar
- Per Ola Kristensson. 2007. Discrete and Continuous Shape Writing for Text Entry and Control. Ph.D. Dissertation. Linköping University.Google Scholar
- Per Ola Kristensson. 2009. Five Challenges for Intelligent Text Entry Methods. AI Magazine 30, 4 (2009), 85--94.Google ScholarDigital Library
- Per Ola Kristensson and Leif C. Denby. 2009. Text entry performance of state of the art unconstrained handwriting recognition: a longitudinal user study. In CHI’09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 567--570. Google ScholarDigital Library
- Per Ola Kristensson and Keith Vertanen. 2012a. Performance Comparisons of Phrase Sets and Presentation Styles for Text Entry Evaluations. In IUI’12: Proceedings of the International Conference on Intelligent User Interfaces. ACM Press, 29--32. Google ScholarDigital Library
- Per Ola Kristensson and Keith Vertanen. 2012b. The Potential of Dwell-Free Eye-Typing for Fast Assistive Gaze Communication. In ETRA’12: Proceedings of the ACM Symposium on Eye-Tracking Research and Applications. 241--244. Google ScholarDigital Library
- Kent Lyons, Thad Starner, and Brian Gane. 2006. Experimental evaluations of the Twiddler one-handed chording mobile keyboard. Human-Computer Interaction 21 (November 2006), 343--392. Issue 4. Google ScholarDigital Library
- I. S. MacKenzie and R. W. Soukoreff. 2002. Text Entry for Mobile Computing: Models and Methods, Theory and Practice. Human-Computer Interaction 17 (2002), 147--198.Google ScholarCross Ref
- Ian Scott MacKenzie and William Soukoreff. 2003. Phrase sets for evaluating text entry techniques. In CHI’03: Extended Abstracts on Human Factors in Computing Systems. ACM Press, 754--755. Google ScholarDigital Library
- I. Scott MacKenzie and Shawn X. Zhang. 1999. The design and evaluation of a high-performance soft keyboard. In CHI’99: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, New York, NY, USA, 25--31. Google ScholarDigital Library
- Time Paek and Bo-June Hsu. 2011. Sampling representative phrase sets for text entry experiments: a procedure and public resource. In CHI’11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 2477--2480. Google ScholarDigital Library
- R. W. Soukoreff and I. S. MacKenzie. 2003. Metrics for text entry research: An evaluation of MSD and KSPC, and a new unified error metric. In CHI’03: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 113--120. Google ScholarDigital Library
- Keith Vertanen and Per Ola Kristensson. 2009. Parakeet: A Continuous Speech Recognition System for Mobile Touch-Screen Devices. In IUI’09: Proceedings of the 14th International Conference on Intelligent User Interfaces. ACM Press, 237--246. Google ScholarDigital Library
- Keith Vertanen and Per Ola Kristensson. 2011a. The imagination of crowds: conversational AAC language modeling using crowdsourcing and large data sources. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing. ACL, 700--711. Google ScholarDigital Library
- Keith Vertanen and Per Ola Kristensson. 2011b. A Versatile Dataset for Text Entry Evaluations Based on Genuine Mobile Emails. In MobileHCI’11: Proceedings of the International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM Press, 295--298. Google ScholarDigital Library
- D. J. Ward and D. J. C. MacKay. 2002. Fast Hands-free writing by Gaze Direction. Nature 418, 6900 (2002), 838.Google ScholarCross Ref
- Jacob O. Wobbrock, Duen Horng Chau, and Brad A. Myers. 2007. An Alternative to Push, Press, and Tap-tap-tap: Gesturing on an Isometric Joystick for Mobile Phone Text Entry. In CHI’07: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM Press, 667--676. Google ScholarDigital Library
- Shumin Zhai, Per Ola Kristensson, and Barton A. Smith. 2005. In search of effective text input interfaces for off the desktop computing. Interacting with Computers 17, 3 (2005), 229--250.Google ScholarCross Ref
Index Terms
- Complementing text entry evaluations with a composition task
Recommendations
Enhancing the Composition Task in Text Entry Studies: Eliciting Difficult Text and Improving Error Rate Calculation
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsParticipants in text entry studies usually copy phrases or compose novel messages. A composition task mimics actual user behavior and can allow researchers to better understand how a system might perform in reality. A problem with composition is that ...
A versatile dataset for text entry evaluations based on genuine mobile emails
MobileHCI '11: Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and ServicesMobile text entry methods are typically evaluated by having study participants copy phrases. However, currently there is no available phrase set that has been composed by mobile users. Instead researchers have resorted to using invented phrases that ...
The inviscid text entry rate and its application as a grand goal for mobile text entry
MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & servicesWe introduce the concept of the inviscid text entry rate: the point when the user's creativity is the bottleneck rather than the text entry method. We then apply the inviscid text entry rate to define a grand goal for mobile text entry. Via a proxy ...
Comments