skip to main content
10.1145/3173574.3174078acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

Rewire: Interface Design Assistance from Examples

Published: 21 April 2018 Publication History

Abstract

Interface designers often use screenshot images of example designs as building blocks for new designs. Since images are unstructured and hard to edit, designers typically reconstruct screenshots with vector graphics tools in order to reuse or edit parts of the design. Unfortunately, this reconstruction process is tedious and slow. We present Rewire, an interactive system that helps designers leverage example screenshots. Rewire automatically infers a vector representation of screenshots where each UI component is a separate object with editable shape and style properties. Based on this representation, the system provides three design assistance modes that help designers reuse or redraw components of the example design. The results from our quantitative and user evaluations demonstrate that Rewire can generate accurate vector representations of interface screenshots found in the wild and that design assistance enables users to reconstruct and edit example designs more efficiently compared to a baseline design tool.

Supplementary Material

ZIP File (pn4084-file4.zip)
suppl.mov (pn4084-file3.mp4)
Supplemental video
suppl.mov (pn4084-file5.mp4)
Supplemental video

References

[1]
Pablo Arbelaez. 2006. Boundary Extraction in Natural Images Using Ultrametric Contour Maps. In Conference on Computer Vision and Pattern Recognition Workshop (CVPRW '06). IEEE, 182--182.
[2]
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 12 (Dec 2017), 2481--2495.
[3]
Dana H Ballard. 1981. Generalizing the Hough transform to detect arbitrary shapes. Pattern recognition 13, 2 (1981), 111--122.
[4]
Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. CoRR abs/1705.07962 (2017). http://arxiv.org/abs/1705.07962
[5]
William Buxton. 2007. Sketching User Experiences: getting the design right and the right design. Elsevier/Morgan Kaufmann, San Francisco, CA; Amsterdam.
[6]
Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. Tools and Algorithms for the Construction and Analysis of Systems (2008), 337--340.
[7]
Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016. ERICA: Interaction Mining Mobile Apps. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 767--776.
[8]
Morgan Dixon and James Fogarty. 2010. Prefab: Implementing Advanced Behaviors Using Pixel-Based Reverse Engineering of Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1525--1534.
[9]
Morgan Dixon, Daniel Leventhal, and James Fogarty. 2011. Content and Hierarchy in Pixel-Based Methods for Reverse Engineering Interface Structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 969--978.
[10]
Morgan E. Dixon, Gierad Laput, and James A. Fogarty. 2014. Pixel-Based Methods for Widget State and Style in a Runtime Implementation of Sliding Widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, New York, USA, 2231--2240.
[11]
Steven P. Dow, Alana Glassco, Jonathan Kass, Melissa Schwarz, Daniel L. Schwartz, and Scott R. Klemmer. 2010. Parallel Prototyping Leads to Better Design Results, More Divergence, and Increased Self-efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4, Article 18 (Dec. 2010), 24 pages.
[12]
Álvaro González, Luis M. Bergasa, J. Javier Yebes, and Sebastián Bronte. 2012. Text Location in Complex Images. In Proceedings of the International Conference on Pattern Recognition (ICPR '12). IEEE, 617--620.
[13]
Scarlett R. Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P. Bailey. 2009. Getting Inspired!: Understanding How and Why Examples Are Used in Creative Design Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 87--96.
[14]
Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3083--3092.
[15]
Ranjitha Kumar, Jerry O. Talton, Salman Ahmad, and Scott R. Klemmer. 2011. Bricolage: Example-based Retargeting for Web Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2197--2206.
[16]
Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klemmer. 2010. Designing with Interactive Example Galleries. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2257--2266.
[17]
Y. Li and H. Lu. 2012. Scene Text Detection via Stroke Width. In Proceedings of the International Conference on Pattern Recognition (ICPR '12). IEEE, 681--684.
[18]
Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In IEEE/ACM International Conference on Automated Software Engineering (ASE '15). IEEE, 248--259.
[19]
Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning Layouts for Single-Page Graphic Designs. IEEE Transactions on Visualization and Computer Graphics 20, 8 (Aug 2014), 1200--1213.
[20]
Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1221--1224.
[21]
Gaurav Sharma, Wencheng Wu, and Edul N. Dalal. 2005. The CIEDE2000 Color-Difference Formula: Implementation notes, supplementary test data, and mathematical observations. Color Research and Application 30, 1 (2005), 21--30.
[22]
Ray Smith. 2007. An Overview of the Tesseract OCR Engine. In International Conference on Document Analysis and Recognition (ICDAR '07), Vol. 2. IEEE, 629--633.
[23]
Holm Sture. 1979. A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6, 2 (1979), 65--70. http://www.jstor.org/stable/4615733
[24]
Pengfei Xu, Hongbo Fu, Takeo Igarashi, and Chiew-Lan Tai. 2014. Global Beautification of Layouts with Interactive Ambiguity Resolution. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 243--252.
[25]
Tom Yeh, Tsung-Hsiang Chang, and Robert C. Miller. 2009. Sikuli: Using GUI Screenshots for Search and Automation. In Proceedings of the Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, New York, NY, USA, 183--192.

Cited By

View all
  • (2024)Towards Educator-Driven Tutor Authoring: Generative AI Approaches for Creating Intelligent Tutor InterfacesProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664694(305-309)Online publication date: 9-Jul-2024
  • (2024)DesignChecker: Visual Design Support for Blind and Low Vision Web DevelopersProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676369(1-19)Online publication date: 13-Oct-2024
  • (2024)GuiEvo: Automated Evolution of Mobile App UIsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644936(335-347)Online publication date: 15-Apr-2024
  • Show More Cited By

Index Terms

  1. Rewire: Interface Design Assistance from Examples

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
    April 2018
    8489 pages
    ISBN:9781450356206
    DOI:10.1145/3173574
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 April 2018

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. pixel-based reverse engineering
    2. user interface design
    3. wireframing

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    CHI '18
    Sponsor:

    Acceptance Rates

    CHI '18 Paper Acceptance Rate 666 of 2,590 submissions, 26%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)248
    • Downloads (Last 6 weeks)26
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Towards Educator-Driven Tutor Authoring: Generative AI Approaches for Creating Intelligent Tutor InterfacesProceedings of the Eleventh ACM Conference on Learning @ Scale10.1145/3657604.3664694(305-309)Online publication date: 9-Jul-2024
    • (2024)DesignChecker: Visual Design Support for Blind and Low Vision Web DevelopersProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676369(1-19)Online publication date: 13-Oct-2024
    • (2024)GuiEvo: Automated Evolution of Mobile App UIsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644936(335-347)Online publication date: 15-Apr-2024
    • (2024)When and How to Use AI in the Design Process? Implications for Human-AI Design CollaborationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.235345141:2(1569-1584)Online publication date: 22-May-2024
    • (2023)Evaluating a Large Language Model on Searching for GUI LayoutsProceedings of the ACM on Human-Computer Interaction10.1145/35932307:EICS(1-37)Online publication date: 19-Jun-2023
    • (2023)ActFloor-GAN: Activity-Guided Adversarial Networks for Human-Centric Floorplan DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.312647829:3(1610-1624)Online publication date: 1-Mar-2023
    • (2023)The methodology of studying fairness perceptions in Artificial IntelligenceInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102954170:COnline publication date: 8-Feb-2023
    • (2023)Using deep learning for selenium web UI functional testsEngineering Applications of Artificial Intelligence10.1016/j.engappai.2022.105446117:PAOnline publication date: 1-Jan-2023
    • (2022)GANSpiration: Balancing Targeted and Serendipitous Inspiration in User Interface Design with Style-Based Generative Adversarial NetworkProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517511(1-15)Online publication date: 29-Apr-2022
    • (2022)Automated Visual Testing for Mobile Apps in an Industrial Setting2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)10.1109/ICSE-SEIP55303.2022.9793948(55-64)Online publication date: May-2022
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media