skip to main content
10.1145/1753326.1753554acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

Published: 10 April 2010 Publication History

Abstract

Current chasms between applications implemented with different user interface toolkits make it difficult to implement and explore potentially important interaction techniques in new and existing applications, limiting the progress and impact of human-computer interaction research. We examine an approach based in the single most common characteristic of all graphical user interface toolkits, that they ultimately paint pixels to a display. We present Prefab, a system for implementing advanced behaviors through the reverse engineering of the pixels in graphical interfaces. Informed by how user interface toolkits paint interfaces, Prefab features a separation of the modeling of widget layout from the recognition of widget appearance. We validate Prefab in implementations of three applications: target-aware pointing techniques, Phosphor transitions, and Side Views parameter spectrums. Working only from pixels, we demonstrate a single implementation of these enhancements in complex existing applications created in different user interface toolkits running on different windowing systems.

Supplementary Material

JPG File (p1525.jpg)
MOV File (p1525.mov)

References

[1]
Baudisch, P., Tan, D.S., Collomb, M., Robbins, D., Hinckley, K., Agrawala, M., Zhao, S. and Ramos, G. (2006). Phosphor: Explaining Transitions in the User Interface using Afterglow Effects. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2006), 169--178.
[2]
Edwards, W.K., Hudson, S.E., Marinacci, J., Rodenstein, R., Rodriguez, T. and Smith, I. (1997). Systematic Output Modification in a 2D User Interface Toolkit. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 1997), 151--158.
[3]
Grossman, T. and Balakrishnan, R. (2005). The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 2005), 281--290.
[4]
Hartmann, B., Wu, L., Collins, K. and Klemmer, S.R. (2007). Programming by a Sample: Rapidly Creating Web Applications with d.Mix. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2007), 241--250.
[5]
Hudson, S.E. (1991). Incremental Attribute Evaluation: A Flexible Algorithm for Lazy Update. ACM Transactions on Programming Languages and Systems (TOPLAS), 13(3). 315--341.
[6]
Hudson, S.E., Mankoff, J. and Smith, I. (2005). Extensible Input Handling in the subArctic Toolkit. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 2005), 381--390.
[7]
Hudson, S.E. and Smith, I. (1997). Supporting Dynamic Downloadable Appearances in an Extensible User Interface Toolkit. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 1997), 159--168.
[8]
Hudson, S.E. and Tanaka, K. (2000). Providing Visually Rich Resizable Images for User Interface Components. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2000), 227--235.
[9]
Hurst, A., Hudson, S.E. and Mankoff, J. (2010). Automatically Identifying Targets Users Interact with During Real World Tasks. Proc. of the International Conference on Intelligent User Interfaces (IUI 2010), To Appear.
[10]
Hurst, A., Mankoff, J., Dey, A.K. and Hudson, S.E. (2007). Dirty Desktops: Using a Patina of Magnetic Mouse Dust to Make Common Interactor Targets Easier to Select. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2007), 183--186.
[11]
Hutchings, D.R. and Stasko, J. (2005). mudibo: Multiple Dialog Boxes for Multiple Monitors. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI 2005), 1471--1474.
[12]
Myers, B.A., Hudson, S.E. and Pausch, R. (2000). Past, Present, and Future of User Interface Software Tools. ACM Transactions on Computer--Human Interaction (TOCHI), 7(1). 3--28.
[13]
Olsen, D.R., Hudson, S.E., Verratti, T., Heiner, J.M. and Phelps, M. (1999). Implementing Interface Attachments Based on Surface Representations. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 1999), 191--198.
[14]
Olsen, D.R., Taufer, T. and Fails, J.A. (2004). ScreenCrayons: Annotating Anything. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2004), 165--174.
[15]
Potter, R. (1993). Triggers: Guiding Automaton with Pixel to Achieve Data Access. A. Cypher, eds. MIT Press.
[16]
Rissanen, J. (1978). Modeling by Shortest Data Description. Automatica 14(5). 465--471.
[17]
Russell, S. and Norvig, P. (2002). Artificial Intelligence: A Modern Approach. Prentice Hall.
[18]
St. Amant, R., Lieberman, H., Potter, R. and Zettlemoyer, L.S. (2000). Visual Generalization in Programming by Example. Communications of the ACM 43(3). 107--114.
[19]
St. Amant, R., Riedl, M.O., Ritter, F.E. and Reifers, A. (2005). Image Processing in Cognitive Models with SegMan. Proc. of the International Conference on Human-Computer Interaction (HCII 2005).
[20]
Stuerzlinger, W., Chapuis, O., Phillips, D. and Roussel, N. (2006). User Interface Façades: Towards Fully Adaptable User Interfaces. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2006), 309--318.
[21]
Tan, D.S., Meyers, B.R. and Czerwinski, M. (2004). WinCuts: Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. Extended Abstracts of the ACM Conference on Human Factors in Computing Systems (CHI 2004), 1525--1528.
[22]
Terry, M. and Mynatt, E.D. (2002). Side Views: Persistent, On Demand Previews for Open-Ended Tasks. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2002), 71--80.
[23]
Wobbrock, J.O., Fogarty, J., Liu, S., Kimuro, S. and Harada, S. (2009). The Angle Mouse: Target-Agnostic Dynamic Gain Adjustment Based on Angular Deviation. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 2009), 1401--1410.
[24]
Worden, A., Walker, N., Bharat, K. and Hudson, S.E. (1997). Making Computers Easier for Older Adults to Use: Area Cursors and Sticky Icons. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 1997), 266--271.
[25]
Yeh, T., Change, T.-H. and Miller, R.C. (2009). Sikuli: Using GUI Screenshots for Search and Automation. Proc. of the ACM Symposium on User Interface Software and Technology (UIST 2009), 183--192.
[26]
Zettlemoyer, L.S. and St. Amant, R. (1999). A Visual Medium for Programmatic Control of Interactive Applications. Proc. of the ACM Conference on Human Factors in Computing Systems (CHI 1999), 199--206.
[27]
Zettlemoyer, L.S., St. Amant, R. and Dulberg, M.S. (1998). IBOTS: Agent Control Through the User Interface. Proc. of the International Conference on Intelligent User Interfaces (IUI 1998), 31--37.

Cited By

View all
  • (2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
  • (2024)Computational Representations for Graphical User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638191(1-6)Online publication date: 11-May-2024
  • (2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
    April 2010
    2690 pages
    ISBN:9781605589299
    DOI:10.1145/1753326
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 April 2010

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. pixel-based reverse engineering
    2. prefab
    3. user interface toolkits

    Qualifiers

    • Research-article

    Conference

    CHI '10
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)93
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Beyond Manual Modeling: Automating GUI Model Generation Using Design DocumentsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695032(91-103)Online publication date: 27-Oct-2024
    • (2024)Computational Representations for Graphical User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3638191(1-6)Online publication date: 11-May-2024
    • (2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
    • (2024)AXNav: Replaying Accessibility Tests from Natural LanguageProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642777(1-16)Online publication date: 11-May-2024
    • (2024)MUD: Towards a Large-Scale and Noise-Filtered UI Dataset for Modern Style UI ModelingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642350(1-14)Online publication date: 11-May-2024
    • (2024)NeuroUI: A Metamorphic Testing Strategy to Make UI Component Detection Models Robust2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00065(636-641)Online publication date: 6-Oct-2024
    • (2023)Relation-enhanced DETR for component detection in graphic design reverse engineeringProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence10.24963/ijcai.2023/532(4785-4793)Online publication date: 19-Aug-2023
    • (2023)A Unified Framework for Mini-game Testing: Experience on WeChatProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613868(1623-1634)Online publication date: 30-Nov-2023
    • (2023)SpaceX MagProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962537:2(1-36)Online publication date: 12-Jun-2023
    • (2023)The Future of Computational Approaches for Understanding and Adapting User InterfacesExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3573805(1-5)Online publication date: 19-Apr-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media