skip to main content
10.1145/1978942.1979086acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Content and hierarchy in pixel-based methods for reverse engineering interface structure

Published: 07 May 2011 Publication History

Abstract

The rigidity and fragmentation of GUI toolkits are fundamentally limiting the progress and impact of interaction research. Pixel-based methods offer unique potential for addressing these challenges independent of the implementation of any particular interface or toolkit. This work builds upon Prefab, which enables the modification of existing interfaces. We present new methods for hierarchical models of complex widgets, real-time interpretation of interface content, and real-time interpretation of content and hierarchy throughout an entire interface. We validate our new methods through implementations of four applications: stencil-based tutorials, ephemeral adaptation, interface translation, and end-user interface customization. We demonstrate these enhancements in complex existing applications created from different user interface toolkits running on different operating systems.

Supplementary Material

MP4 File (paper1877.mp4)

References

[1]
Baudisch, P., Tan, D.S., Collomb, M., Robbins, D., Hinckley, K., Agrawala, M., Zhao, S. and Ramos, G. Phosphor: Explaining Transitions in the User Interface using Afterglow Effects. UIST 2006. 169--178.
[2]
Bolin, M., Webber, M., Rha, P., Wilson, T. and Miller, R.C. Automation and Customization of Rendered Web Pages. UIST 2005. 163--172.
[3]
Chang, T.-H., Yeh, T. and Miller, R. C. GUI Testing User Computer Vision. CHI 2010. 1535--1544.
[4]
Dixon, M. and Fogarty, J. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. CHI 2010. 1525--1534.
[5]
Findlater, L., Moffatt, K., McGrenere, J. and Dawson, J. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. CHI 2009. 1655--1664.
[6]
Fujima, J., Lunzer, A., Hornbæk, K. and Tanaka, Y. Clip, Connect, Clone: Combining Applications Elements to Build Custom Interfaces for Information Access. UIST 2004. 175--184.
[7]
Gaeremynck, Y., Bergman, L. D. and Lau, T. A. MORE for Less: Model Recovery from Visual Interfaces for Multi-Device Application Design. IUI 2003. 69--76.
[8]
Greenberg, S. and Buxton, B. Usability Evaluation Considered Harmful (Some of the Time). CHI 2008. 111--120.
[9]
Grossman, T. and Balakrishnan, R. The Bubble Cursor: Enhancing Target Acquisition by Dynamic Resizing of the Cursor's Activation Area. CHI 2005. 281--290.
[10]
Hartmann, B., Wu, L., Collins, K. and Klemmer, S. R. Programming by a Sample: Rapidly Creating Web Applications with d.Mix. UIST 2007. 241--250.
[11]
Hudson, S. E. Incremental Attribute Evaluation: A Flexible Algorithm for Lazy Update. TOPLAS, 13(3). 315--341.
[12]
Hudson, S. E., Mankoff, J. and Smith, I. Extensible Input Handling in the subArctic Toolkit. CHI 2005. 381--390.
[13]
Hudson, S. E. and Smith, I. Supporting Dynamic Downloadable Appearances in an Extensible User Interface Toolkit. UIST 1997. 159--168.
[14]
Hudson, S. E. and Tanaka, K. Providing Visually Rich Resizable Images for User Interface Components. UIST 2000. 227--235.
[15]
Hurst, A., Hudson, S. E. and Mankoff, J. Automatically Identifying Targets Users Interact with During Real World Tasks. IUI 2010. 11--20.
[16]
Kelleher, C. and Pausch, R. Stencils-Based Tutorials: Design and Evaluation. CHI 2005. 541--550.
[17]
Lin, J., Wong, J., Nichols, J., Cypher, A. and Lau, T.A. End-User Programming of Mashups with Vegemite. IUI 2009. 97--106.
[18]
Little, G., Lau, T. A., Cypher, A., Lin, J., Haber, E. M. and Kandogan, E. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. CHI 2007. 943--946.
[19]
Myers, B. A., Hudson, S. E. and Pausch, R. Past, Present, and Future of User Interface Software Tools. TOCHI, 7(1). 3--28.
[20]
Nichols, J. and Lau, T. A. Mobilizing by Demonstration: Using Traces to Re-Author Existing Web Sites. IUI 2008. 149--158.
[21]
Olsen, D. R. Evaluating User Interface Systems Research. UIST 2007. 251--258.
[22]
Olsen, D. R., Taufer, T. and Fails, J. A. ScreenCrayons: Annotating Anything. UIST 2004. 165--174.
[23]
Potter, R. (1993). Triggers: Guiding Automaton with Pixel to Achieve Data Access. A. Cypher, eds. MIT Press.
[24]
Rissanen, J. Modeling by Shortest Data Description. Automatica, 14(5). 465--471.
[25]
St. Amant, R., Lieberman, H., Potter, R. and Zettlemoyer, L. S. Visual Generalization in Programming by Example. 43(3). 107--114.
[26]
St. Amant, R., Riedl, M. O., Ritter, F. E. and Reifers, A. Image Processing in Cognitive Models with SegMan. HCII 2005.
[27]
Stuerzlinger, W., Chapuis, O., Phillips, D. and Roussel, N. User Interface Façades: Towards Fully Adaptable User Interfaces. UIST 2006. 309--318.
[28]
Takagi, H., Kawanaka, S., Kobayashi, M., Itoh, T. and Asakawa, C. Social Accessibility: Achieving Accessibility through Collaborative Metadata Authoring. ASSETS 2008. 193--200.
[29]
Tan, D. S., Meyers, B. R. and Czerwinski, M. WinCuts: Manipulating Arbitrary Window Regions for More Effective Use of Screen Space. CHI 2004. 1525--1528.
[30]
Wachenfeld, S., Klein, H.-U. and Jiang, X. Recognition of Screen-Rendered Text. ICPR 2006. 1086--1089.
[31]
Yeh, T., Chang, T.-H. and Miller, R. C. Sikuli: Using GUI Screenshots for Search and Automation. UIST 2009. 183--192.
[32]
Zettlemoyer, L. S. and St. Amant, R. A Visual Medium for Programmatic Control of Interactive Applications. CHI 1999. 199--206.
[33]
Zettlemoyer, L. S., St. Amant, R. and Dulberg, M. S. IBOTS: Agent Control Through the User Interface. IUI 1998. 31--37.

Cited By

View all
  • (2024)GTLayout: Learning General Trees for Structured Grid Layout GenerationComputational Visual Media10.1007/978-981-97-2092-7_7(131-153)Online publication date: 30-Mar-2024
  • (2023)“Unfold and Go Touch”: A Portable Method for Making Existing Touchscreens Accessible to Blind and Low Vision People in Self-Service TerminalsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585819(1-7)Online publication date: 19-Apr-2023
  • (2022)Describing UI Screenshots in Natural LanguageACM Transactions on Intelligent Systems and Technology10.1145/356470214:1(1-28)Online publication date: 9-Nov-2022
  • Show More Cited By

Index Terms

  1. Content and hierarchy in pixel-based methods for reverse engineering interface structure
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
        May 2011
        3530 pages
        ISBN:9781450302289
        DOI:10.1145/1978942
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 May 2011

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. content
        2. hierarchy
        3. pixel-based reverse engineering
        4. prefab

        Qualifiers

        • Research-article

        Conference

        CHI '11
        Sponsor:

        Acceptance Rates

        CHI '11 Paper Acceptance Rate 410 of 1,532 submissions, 27%;
        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)19
        • Downloads (Last 6 weeks)0
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)GTLayout: Learning General Trees for Structured Grid Layout GenerationComputational Visual Media10.1007/978-981-97-2092-7_7(131-153)Online publication date: 30-Mar-2024
        • (2023)“Unfold and Go Touch”: A Portable Method for Making Existing Touchscreens Accessible to Blind and Low Vision People in Self-Service TerminalsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585819(1-7)Online publication date: 19-Apr-2023
        • (2022)Describing UI Screenshots in Natural LanguageACM Transactions on Intelligent Systems and Technology10.1145/356470214:1(1-28)Online publication date: 9-Nov-2022
        • (2022)A Survey on the Use of Computer Vision to Improve Software Engineering TasksIEEE Transactions on Software Engineering10.1109/TSE.2020.303298648:5(1722-1742)Online publication date: 1-May-2022
        • (2021)ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-ConstraintsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445043(1-18)Online publication date: 6-May-2021
        • (2021)It Takes Two to TangoProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00091(957-969)Online publication date: 22-May-2021
        • (2021)Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning AgentsArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_15(495-537)Online publication date: 5-Nov-2021
        • (2020)RataplanProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/33973294:2(1-23)Online publication date: 15-Jun-2020
        • (2020)Towards Multi-Wheel Input Device for Non-Visual InteractionAdjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology10.1145/3379350.3416168(7-9)Online publication date: 20-Oct-2020
        • (2020)Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile AppsIEEE Transactions on Software Engineering10.1109/TSE.2018.284478846:2(196-221)Online publication date: 1-Feb-2020
        • Show More Cited By

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media