skip to main content
10.1145/2207676.2208703acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Identifying usability issues via algorithmic detection of excessive visual search

Published: 05 May 2012 Publication History

Abstract

Automated detection of excessive visual search (ES) experienced by a user during software use presents the potential for substantial improvement in the efficiency of supervised usability analysis. This paper presents an objective evaluation of several methods for the automated segmentation and classification of ES intervals from an eye movement recording, a technique that can be utilized to aid in the identification of usability problems during software usability testing. Techniques considered for automated segmentation of the eye movement recording into unique intervals include mouse/keyboard events and eye movement scanpaths. ES is identified by a number of eye movement metrics, including: fixation count, saccade amplitude, convex hull area, scanpath inflections, scanpath length, and scanpath duration. The ES intervals identified by each algorithm are compared to those produced by manual classification to verify the accuracy, precision, and performance of each algorithm. The results indicate that automated classification can be successfully employed to substantially reduce the amount of recorded data reviewed by HCI experts during usability testing, with relatively little loss in accuracy.

Supplementary Material

WMV File (paperfile365-3.wmv)
Supplemental video for “Identifying usability issues via algorithmic detection of excessive visual search”

References

[1]
A. A. Witold, et al., "Consolidating the ISO Usability Models," presented at the 11th International Software Quality Management Conference and 8th Annual INSPIRE Conference, 2003.
[2]
J. Rubin and D. Chisnell, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2 ed. New York, NY: Wiley, 2008.
[3]
L. Vukelja, et al., "Are engineers condemned to design? a survey on software engineering and UI design in Switzerland," presented at the 11th IFIP TC 13 international conference on Human-computer interaction, Rio de Janeiro, Brazil, 2007.
[4]
J. S. Dumas and J. C. Redish, A Practical Guide to Usability Testing: Intellect Books, 1999.
[5]
R. J. Leigh and D. S. Zee, The Neurology of Eye Movements, 4 ed.: Oxford University Press, USA, 2006.
[6]
A. Duchowski, Eye Tracking Methodology: Theory and Practice, 2nd ed.: Springer, 2007.
[7]
A. Poole and L. J. Ball, "Eye tracking in humancomputer interaction and usability research: current status and future prospects," in Encyclopedia of Human-Computer Interaction, C. Ghaoui, Ed., ed: Idea Group, 2005, pp. 211--219.
[8]
L. J. Ball, et al., "Applying the Post-Experience EyeTracked Protocol (PEEP) Method in Usability Testing," Interfaces, vol. 67, pp. 15--19, 2006.
[9]
M. C. Russell, "Hotspots and hyperlinks: using eyetracking to supplement usability testing," Usability News, vol. 7, 2005.
[10]
J. M. Wolfe, "What Can 1 Million Trials Tell Us About Visual Search?," Psychological Science, vol. 9, pp. 33--39, 1998.
[11]
J. Shen, et al., "Distractor ratio influences patterns of eye movements during visual search," Perception, vol. 29, pp. 241--250, 2000.
[12]
I. D. Gilchrist and M. Harvey, "Refixation frequency and memory mechanisms in visual search," Current Biology, vol. 10, pp. 1209--1212, 2000.
[13]
A. J. Hornof and T. Halverson, "Cognitive strategies and eye movements for searching hierarchical computer displays," in SIGCHI conference on Human factors in computing systems, Ft. Lauderdale, Florida, USA, 2003, pp. 249--256.
[14]
O. V. Komogortsev, et al., "Eye movement driven usability evaluation via excessive search identification," in 14th International Conference on Human-Computer Interaction, 2011.
[15]
O. Komogortsev, et al., "EMA: Automated eyemovement-driven approach for identification of usability issues," in Design, user experience, and usability. Theory, methods, tools and practice. vol. 6770, A. Marcus, Ed., ed: Springer Berlin / Heidelberg, 2011, pp. 459--468.
[16]
O. Komogortsev, et al., "Aiding usability evaluation via detection of excessive visual search," presented at the 2011 ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), Vancouver, BC, Canada, 2011.
[17]
P. M. Fitts, et al., "Eye movements of aircraft pilots during instrument-landing approaches," Aeronautical Engineering Review, vol. 9, pp. 24--29, 1950.
[18]
R. Jacob and K. Karn, "Commentary on Section 4. Eye Tracking in Human-Computer Interaction and Usability Research: Ready to Deliver the Promises," in The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research, ed: Elsevier, 2003, pp. 573--607.
[19]
F. T. W. Au, et al., "Automated usability testing framework," presented at the Proceedings of the ninth conference on Australasian user interface Volume 76, Wollongong, Australia, 2008.
[20]
M. Ivory and A. Chevalier, "A Study of Automated Web Site Evaluation Tools," University of Washington, Department of Computer Science2002.
[21]
J. H. Goldberg and X. P. Kotval, "Computer interface evaluation using eye movements: methods and constructs," International Journal of Industrial Ergonomics, vol. 24, pp. 631--645, 1999.
[22]
O. V. Komogortsev, et al., "Standardization of Automated Analyses of Oculomotor Fixation and Saccadic Behaviors," IEEE Transactions on Biomedical Engineering, vol. 57, pp. 2635--2645, 2010.
[23]
W. Sewell and O. Komogortsev, "Real-time eye gaze tracking with an unmodified commodity webcam employing a neural network," in 28th of the international conference extended abstracts on Human factors in computing systems, Atlanta, Georgia, USA, 2010, pp. 3739--3744.
[24]
J. S. Agustin, et al., "Low-cost gaze interaction: ready to deliver the promises," presented at the Proceedings of the 27th international conference extended abstracts on Human factors in computing systems, Boston, MA, USA, 2009.

Cited By

View all
  • (2021)EMICS’21: Eye Movements as an Interface to Cognitive StateExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3441357(1-6)Online publication date: 8-May-2021
  • (2021)Writing Futures: InvestigationsWriting Futures: Collaborative, Algorithmic, Autonomous10.1007/978-3-030-70928-0_5(109-139)Online publication date: 19-Jun-2021
  • (2020)EMICS'20: Eye Movements as an Interface to Cognitive StateExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3381062(1-4)Online publication date: 25-Apr-2020
  • Show More Cited By

Index Terms

  1. Identifying usability issues via algorithmic detection of excessive visual search

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2012
    3276 pages
    ISBN:9781450310154
    DOI:10.1145/2207676
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 May 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. eye tracking
    2. human-computer interaction
    3. usability
    4. usability testing.
    5. visual search

    Qualifiers

    • Research-article

    Conference

    CHI '12
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)18
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 18 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)EMICS’21: Eye Movements as an Interface to Cognitive StateExtended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411763.3441357(1-6)Online publication date: 8-May-2021
    • (2021)Writing Futures: InvestigationsWriting Futures: Collaborative, Algorithmic, Autonomous10.1007/978-3-030-70928-0_5(109-139)Online publication date: 19-Jun-2021
    • (2020)EMICS'20: Eye Movements as an Interface to Cognitive StateExtended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3334480.3381062(1-4)Online publication date: 25-Apr-2020
    • (2013)Changing perspectives on evaluation in HCICHI '13 Extended Abstracts on Human Factors in Computing Systems10.1145/2468356.2468714(1969-1978)Online publication date: 27-Apr-2013
    • (2013)Remote Eye Tracking SystemsProceedings of the 2013 26th Conference on Graphics, Patterns and Images Tutorials10.1109/SIBGRAPI-T.2013.8(15-22)Online publication date: 5-Aug-2013

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media