skip to main content
10.1145/2508468.2508469acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
extended-abstract

Pixel-based reverse engineering of graphical interfaces

Published: 08 October 2013 Publication History

Abstract

My dissertation proposes a vision in which anybody can modify any interface of any application. Realizing this vision is difficult because of the rigidity and fragmentation of current interfaces. Specifically, rigidity makes it difficult or impossible for a designer to modify or customize existing interfaces. Fragmentation results from the fact that people generally use many different applications built with a variety of toolkits. Each is implemented differently, so it is difficult to consistently add new functionality. As a result, researchers are often limited to demonstrating new ideas in small testbeds, and practitioners often find it difficult to adopt and deploy ideas from the literature. In my dissertation, I propose transcending the rigidity and fragmentation of modern interfaces by building upon their single largest commonality: that they ultimately consist of pixels painted to a display. Building from this universal representation, I propose pixel-based interpretation to enable modification of interfaces without their source code and independent of their underlying toolkit implementation.

References

[1]
Banovic, B., Grossman, T., Matejka, J., and Fitzmaurice, G. Waken: Reverse Engineering Usage Information and Interface Structure from Software Videos. UIST 2012. 83--92.
[2]
Bolin, M., Webber, M., Rha, P., Wilson, T. and Miller, R.C. Automation and Customization of Rendered Web Pages. UIST 2005. 163--172.
[3]
Dixon, M. and Fogarty, J. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. CHI 2010. 1525--1534.
[4]
Dixon, M., Leventhal, D., and Fogarty, J. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. CHI 201 969--978.
[5]
Dixon, M., Fogarty, J., and Wobbrock, W. 2012. A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces. CHI 2012. 3167--3176.
[6]
Eagan, J.R., Beaudouin-Lafon, M., and Mackay, W.E. Cracking the Cocoa Nut: User Interface Programming at Runtime. UIST 2011. 225--234.
[7]
Findlater, L., Moffatt, K., McGrenere, J. and Dawson, J. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. CHI 2009. 1655--1664.
[8]
Grossman, T. and Balakrishnan, R. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. CHI 2005. 281--290.
[9]
Grossman, T., Matejka, J., and Fitzmaurice, G. Chronicle: Capture, exploration, and playback of document workflow histories. UIST 2010. 143--152.
[10]
Hartmann, B., Wu, L., Collins, K., and Klemmer, S.R. Programming by a Sample: Rapidly Creating Web Applications with d.mix. UIST 2007. 241--250.
[11]
Kelleher, C. and Pausch, R. Stencils-Based Tutorials: Design and Evaluation. CHI 2005. 541--550.
[12]
Little, G., Lau, T.A., Cypher, A., Lin, J., Haber, E.M. and Kandogan, E. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI 2007. 943--946.
[13]
Moscovich, T. Contact Area Interaction with Sliding Widgets. UIST 2009. 13--22.
[14]
Stuerzlinger, W., Chapuis, O., Phillips, D. and Roussel, N. User Interface Façades: Towards Fully Adaptable User Interfaces. UIST 2006. 309--318.
[15]
Pongnumkul, S., Dontcheva, M, Li, W., Wang, J., Bourdev, L., Avidan, S., and Cohen, M.F. 2011. Pause-and-play: automatically linking screencast video tutorials with applications. UIST 2011. 135--144.
[16]
Yeh, T., Change, T.-H. and Miller, R.C. (2009). Sikuli: Using GUI Screenshots for Search and Automation. UIST 2009. 183--192.
[17]
Yeh, T., Chang, T., Xie, B., Walsh, G., Watkins, I., Wongsuphasawat, K., Huang, M., Davis, L.S., and Bederson. B.B. Creating contextual help for GUIs using screenshots. UIST 2011. 145--154.

Index Terms

  1. Pixel-based reverse engineering of graphical interfaces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '13 Adjunct: Adjunct Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology
    October 2013
    126 pages
    ISBN:9781450324069
    DOI:10.1145/2508468
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 08 October 2013

    Check for updates

    Author Tags

    1. pixel-based reverse engineering
    2. user interface toolkits

    Qualifiers

    • Extended-abstract

    Conference

    UIST'13
    UIST'13: The 26th Annual ACM Symposium on User Interface Software and Technology
    October 8 - 11, 2013
    St. Andrews, Scotland, United Kingdom

    Acceptance Rates

    UIST '13 Adjunct Paper Acceptance Rate 49 of 317 submissions, 15%;
    Overall Acceptance Rate 355 of 1,733 submissions, 20%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 199
      Total Downloads
    • Downloads (Last 12 months)4
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media