Peitong Duan
Jeremy Warner
Bjoern Hartmann
ABSTRACT
Feedback on user interface (UI) mockups is crucial for the design process, and designers often seek and leverage feedback to improve their UIs. However, human feedback is not always readily available. Given the recent emergence of LLMs, which have been shown to be proficient in rule-based reasoning, we explore the potential of LLMs to provide feedback automatically. In particular, we investigate automating heuristic evaluation, which currently entails a human expert assessing how well a UI adheres to a given set of design guidelines. We build an LLM-based heuristic evaluation plugin for Figma, which designers can use to evaluate their UI mockups. The plugin queries the LLM with the guidelines and a JSON representation of the UI mockup and then renders the identified guideline violations as constructive suggestions for design improvements. Future work is needed to study what types of usability problems can be successfully identified by LLM-driven heuristic evaluation.
Supplemental Material
Available for Download
- Zoya Bylinskii, Nam Wook Kim, Peter O’Donovan, Sami Alsheikh, Spandan Madan, Hanspeter Pfister, Fredo Durand, Bryan Russell, and Aaron Hertzmann. 2017. Learning Visual Importance for Graphic Designs and Data Visualizations. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 57–69. https://doi.org/10.1145/3126594.3126653Google Scholar
Digital Library
- Tsung-Hsiang Chang, Tom Yeh, and Robert C. Miller. 2010. GUI Testing Using Computer Vision. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 1535–1544. https://doi.org/10.1145/1753326.1753555Google ScholarDigital Library
- Chun-Fu (Richard) Chen, Marco Pistoia, Conglei Shi, Paolo Girolami, Joseph W. Ligman, and Yong Wang. 2017. UI X-Ray: Interactive Mobile UI Testing Based on Computer Vision(IUI ’17). Association for Computing Machinery, New York, NY, USA, 245–255. https://doi.org/10.1145/3025171.3025190Google ScholarDigital Library
- Chunggi Lee, Sanghoon Kim, Dongyun Han, Hongjun Yang, Young-Woo Park, Bum Chul Kwon, and Sungahn Ko. 2020. GUIComp: A GUI Design Assistant with Real-Time, Multi-Faceted Feedback. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376327Google ScholarDigital Library
- Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 165–175. https://doi.org/10.1145/3180155.3180246Google ScholarDigital Library
- D Royce Sadler. 1989. Formative assessment and the design of instructional systems. Instructional science 18 (1989), 119–144.Google Scholar
- Eldon Schoop, Xin Zhou, Gang Li, Zhourong Chen, Bjoern Hartmann, and Yang Li. 2022. Predicting and Explaining Mobile UI Tappability with Vision Modeling and Saliency Analysis. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 36, 21 pages. https://doi.org/10.1145/3491102.3517497Google ScholarDigital Library
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arxiv:2201.11903 [cs.CL]Google Scholar
- Xiaoyi Zhang, Lilian de Greef, Amanda Swearngin, Samuel White, Kyle Murray, Lisa Yu, Qi Shan, Jeffrey Nichols, Jason Wu, Chris Fleizach, Aaron Everitt, and Jeffrey P Bigham. 2021. Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 275, 15 pages. https://doi.org/10.1145/3411764.3445186Google ScholarDigital Library
Index Terms
- Towards Generating UI Design Feedback with LLMs
Recommendations
Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User InterfacesDuring the User interface (UI) design process, designers use UI design patterns for conceptualizing different UI wireframes for an application. This paper introduces Akin, a UI wireframe generator that allows designers to chose a UI design pattern and ...
UI Fin: a process-oriented interface design tool
IUI '09: Proceedings of the 14th international conference on Intelligent user interfacesEven though over the years a multitude of user interface design tools have been created, designers in practice find themselves limited to a small set of realistic options. These options include interface builders that are attached to development ...
Scenario-based interactive UI design
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsClearly picturing user behavior is one of the key requirements when designing successful interactive software. However, covering all possible user behaviors with one UI is a complex challenge. The Scenario-based Interactive UI Design tool is designed to ...
Comments