skip to main content
10.1145/2207676.2208734acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

Published: 05 May 2012 Publication History

Abstract

We present a general-purpose implementation of a target aware pointing technique, functional across an entire desktop and independent of application implementations. Specifically, we implement Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature. Our implementation obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation. We discuss the most novel aspects of our implementation, including methods for interactive creation and correction of pixel-level prototypes of interface elements and methods for interactive annotation of how the cursor should select identified elements. We also report on limitations of the Bubble Cursor unearthed by examining our implementation in the complexity of real-world interfaces. We therefore contribute important progress toward real-world deployment of an important family of techniques and shed light on the gap between understanding techniques in controlled settings versus behavior with real-world interfaces.

Supplementary Material

MP4 File (paperfile657-3.mp4)
Supplemental video for “A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces”

References

[1]
Ahlström, D., Hitz, M. and Leitner, G. An Evaluation of Sticky and Force Enhanced Targets in Multi Target Situations. NordiCHI 2006. 58--67.
[2]
Bolin, M., Webber, M., Rha, P., Wilson, T., and Miller, R.C. Automation and Customization of Rendered Web Pages. UIST 2005. 163--172.
[3]
Baudisch, P., Cutrell, E., Robbins, D., Czerwinski, M., Tandler, P., Bederson, B. and Zierlinger, A. Drag-and-Pop and Drag-and-Pick: Techniques for Accessing Remote Screen Content on Touch- and Pen-Operated Systems. INTERACT 2003. 57--64.
[4]
Baudisch, P., Cutrell, E., Hinckley, K. and Gruen, R. Mouse Ether: Accelerating the Acquisition of Targets Across Multi-Monitor Displays. CHI 2004. Extended Abstracts. 1379--1382.
[5]
Blanch, R., Guiard, Y. and Beaudouin-Lafon, M. Semantic Pointing: Improving Target Acquisition with Control-Display Ratio Adaptation. CHI 2004. 519--526.
[6]
Casiez, G., Vogel, D., Balakrishnan, R. and Cockburn, A. The Impact of Control-Display Gain on User Performance in Pointing Tasks. Human-Computer Interaction 23 (3). 215--250.
[7]
Chang, T., Yeh, T., and Miller, M. Associating the Visual Representation of User Interfaces with their Internal Structures and Metadata. UIST 2011. To appear.
[8]
Chapuis, O. Labrune, J., and and Pietriga, E. DynaSpot: Speed-Dependent Area Cursor. CHI 2009. 1391--1400.
[9]
Chapuis, O. and Roussel, N. UIMarks: Quick Graphical Interaction with Specific Targets. UIST 2010. 173--182.
[10]
Cockburn, A. and Firth, A. Improving the Acquisition of Small Targets. HCI 2003. 181--196.
[11]
Dixon, M. and Fogarty, J. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. CHI 2010. 1525--1534.
[12]
Dixon, M., Leventhal, D., and Fogarty, J. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. CHI 2011. 969--978.
[13]
Edwards, W.K., Hudson, S.E., Marinacci, J., Rodenstein, R., Rodriguez, T. and Smith, I. Systematic Output Modification in a 2D User Interface Toolkit. UIST 1997. 151--158.
[14]
Eagan, J.R., Mackay, W.E., and Beaudouin-Lafon, M. Cracking the Cocoa Nut: User Interface Programming at Runtime. UIST 2011. To appear.
[15]
Findlater, L., Jansen, A., Shinohara, K., Dixon, M., Kamb, P., Rakita, J., and Wobbrock, J.O. Enhanced Area Cursors: Reducing Fine Pointing Demands for People with Motor Impairments. UIST 2010. 153--162.
[16]
Findlater, L., Moffatt, K., McGrenere, J. and Dawson, J. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. CHI 2009. 1655--1664.
[17]
Fujima, J., Lunzer, A., Hornbæk, K. and Tanaka, Y. Clip, Connect, Clone: Combining Applications Elements to Build Custom Interfaces for Information Access. UIST 2004. 175--184.
[18]
Grossman, T. and Balakrishnan, R. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. CHI 2005. 281--290.
[19]
Guiard, Y., Blanch, R. and Beaudouin-Lafon, M. Object Pointing: A Complement to Bitmap Pointing in GUIs. GI 2004. 9--16.
[20]
Hartmann, B., Wu, L., Collins, K. and Klemmer, S.R. Programming by a Sample: Rapidly Creating Web Applications with d.Mix. UIST 2007. 241--250.
[21]
Hourcade, J.P., Perry, K.B. and Sharma, A. PointAssist: Helping Four Year Olds Point with Ease. IDC 2008. 202--209.
[22]
Hurst,A.,Hudson,S.E.andMankoff,J.AutomaticallyIdentifying TargetsUsersInteractwithDuringRealWorld Tasks. IUI2010.11--20.
[23]
Hurst, A., Mankoff, J., Dey, A.K. and Hudson, S.E. Dirty Desktops: Using a Patina of Magnetic Mouse Dust to Make Common Interactor Targets Easier to Select. UIST 2007. 183--186.
[24]
Hwang, F., Keates, S., Langdon,P. and Clarkson, P.J.Multiple Haptic Targets for Motion-Impaired Computer Users. CHI 2003.41--48.
[25]
Jansen, A., Findlater, L. and Wobbrock, J.O. From the Lab to the World: Lessons from Extending a Pointing Technique for Real-World Use. CHI 2011. Extended Abstracts. 1867--1872.
[26]
Kabbash, P. and Buxton, W. The "Prince" Technique: Fitts' Law and Selection Using Area Cursors. CHI 1995. 273--279.
[27]
Kushmerick, N., Weld, D.S, and Doorenbos, R. Wrapper Induction for Information Extraction. IJCAI 1997.
[28]
Lin, J., Wong, J., Nichols, J., Cypher, A. and Lau, T.A. End-User Programming of Mashups with Vegemite. IUI 2009. 97--106.
[29]
Little, G., Lau, T.A., Cypher, A., Lin, J., Haber, E.M. and Kandogan, E. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI 2007. 943--946.
[30]
McGuffin, M. and Balakrishnan, R. Acquisition of Expanding Targets. CHI 2002. 57--64.
[31]
Nichols, J. and Lau, T.A. Mobilizing by Demonstration: Using Traces to Re-Author Existing Web Sites. IUI 2008. 149--158.
[32]
Olsen, D.R., Hudson, S.E., Verratti, T., Heiner, J.M. and Phelps, M. Implementing Interface Attachments Based on Surface Representations. CHI 1999. 191--198.
[33]
Olsen, D.R., Taufer, T. and Fails, J.A. ScreenCrayons: Annotating Anything. UIST 2004. 165--174.
[34]
Potter, R. Triggers: Guiding Automaton with Pixel to Achieve Data Access. A. Cypher, eds. MIT Press.
[35]
St. Amant, R., Lieberman, H., Potter, R. and Zettlemoyer, L.S. Visual Generalization in Programming by Example. Communications of the ACM 43(3). 107--114.
[36]
St. Amant, R., Riedl, M.O., Ritter, F.E. and Reifers, A. Image Processing in Cognitive Models with SegMan. HCII 2005.
[37]
Stuerzlinger, W., Chapuis, O., Phillips, D. and Roussel, N. User Interface Façades: Towards Fully Adaptable User Interfaces. UIST 2006. 309--318.
[38]
Takagi, H., Kawanaka, S., Kobayashi, M., Itoh, T., and Asakawa, C. Social Accessibility: Achieving Accessibility through Collaborative Metadata Authoring. Assets 2008. 193--200.
[39]
Tan, D.S., Meyers, B.R. and Czerwinski, M. Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. CHI 2004. 1525--1528.
[40]
Yeh, T., Chang, T.-H. and Miller, R.C. Sikuli: Using GUI Screenshots for Search and Automation. UIST 2009. 183--192.
[41]
Wobbrock, J.O., Fogarty, J., Liu, S., Kimuro, S., and Harada, S. The Angle Mouse: Target-Agnostic Dynamic Gain Adjustment Based on Angular Deviation. CHI 2009. 1401--1410.
[42]
Worden, A., Walker, N., Bharat, K. and Hudson, S.E. Making Computers Easier for Older Adults to Use: Area cursors and Sticky Icons. CHI 1997. 266--271.
[43]
Zellweger, P.T., Bouvin, N.O., Jehøj, H., and Mackinlay, J.D. Fluid Annotations in an Open World. Hypertext 2001. 9--18.
[44]
Zhai, S., Morimoto, C. and Ihde, S. Manual and Gaze Input Cascaded (MAGIC) Pointing. CHI 1999. 246--253.
[45]
Zettlemoyer, L.S. and St. Amant, R. A Visual Medium for Programmatic Control of Interactive Applications. CHI 1999. 199--206.
[46]
Zettlemoyer, L.S., St. Amant, R. and Dulberg, M.S. IBOTS: Agent Control Through the User Interface. IUI 1998. 31--37.

Cited By

View all
  • (2023)“I just thought it was me”: How Smartphones Fail Users with Mild-to-Moderate Dexterity DifferencesProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608396(1-12)Online publication date: 22-Oct-2023
  • (2022)Quantifying Touch: New Metrics for Characterizing What Happens During a TouchProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544804(1-13)Online publication date: 23-Oct-2022
  • (2021)Idyll Studio: A Structured Editor for Authoring Interactive & Data-Driven ArticlesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474731(1-12)Online publication date: 10-Oct-2021
  • Show More Cited By

Index Terms

  1. A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    May 2012
    3276 pages
    ISBN:9781450310154
    DOI:10.1145/2207676
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 May 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. bubble cursor
    2. pixel-based reverse engineering
    3. prefab
    4. real-world interfaces
    5. social annotation
    6. target-aware pointing

    Qualifiers

    • Research-article

    Conference

    CHI '12
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)12
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)“I just thought it was me”: How Smartphones Fail Users with Mild-to-Moderate Dexterity DifferencesProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3608396(1-12)Online publication date: 22-Oct-2023
    • (2022)Quantifying Touch: New Metrics for Characterizing What Happens During a TouchProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544804(1-13)Online publication date: 23-Oct-2022
    • (2021)Idyll Studio: A Structured Editor for Authoring Interactive & Data-Driven ArticlesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474731(1-12)Online publication date: 10-Oct-2021
    • (2021)Screen Recognition: Creating Accessibility Metadata for Mobile Applications from PixelsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445186(1-15)Online publication date: 6-May-2021
    • (2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
    • (2019)Improving Pointing in Graphical User Interfaces for People With Motor Impairments Through Ability-Based DesignHuman Performance Technology10.4018/978-1-5225-8356-1.ch059(1193-1243)Online publication date: 2019
    • (2019)Improving User Experience of Eye Tracking-Based InteractionACM Transactions on Computer-Human Interaction10.1145/333884426:6(1-46)Online publication date: 2-Nov-2019
    • (2019)Exploring the Opportunities for Technologies to Enhance Quality of Life with People who have Experienced Vision LossProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300421(1-8)Online publication date: 2-May-2019
    • (2019)Cluster TouchProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300257(1-14)Online publication date: 2-May-2019
    • (2019)Assistive TechnologiesWeb Accessibility10.1007/978-1-4471-7440-0_18(317-335)Online publication date: 4-Jun-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media