Yuwen Lu
Chengzhi Zhang
ABSTRACT
User interface (UI) and user experience (UX) design have become an indispensable part of today’s tech industry. Recently, much progress has been made in machine-learning-enabled design support tools for UX designers. However, few of these tools have been adopted by practitioners. To learn the underlying reasons and understand user needs for bridging this gap, we conducted a retrospective analysis with 8 UX professionals to understand their practice and identify opportunities for future research. We found that the current AI-enabled systems to support UX work mainly work on graphical interface elements, while design activities that involve more ‘design thinking” such as user interviews and user testings are more helpful for designers. Many current systems were also designed for overly-simplistic and generic use scenarios. We identified 4 areas in the UX workflow that can benefit from additional AI-enabled assistance: design inspiration search, design alternative exploration, design system customization, and design guideline violation check.
Supplemental Material
- Mike Adams. 2022. Cure your Zoom amnesia with Grain 2.0. https://grain.co/blog/announcing-grain-2-0Google Scholar
- Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). Association for Computing Machinery, New York, NY, USA, 665–676. https://doi.org/10.1145/3180155.3180240Google ScholarDigital Library
- Michael Chromik, Florian Lachner, and Andreas Butz. 2020. ML for UX? - An Inventory and Predictions on the Use of Machine Learning Techniques for UX Research. In Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society. Association for Computing Machinery, New York, NY, USA, Article 57, 11 pages. https://doi.org/10.1145/3419249.3420163Google ScholarDigital Library
- Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. 2021. Wordcraft: a Human-AI Collaborative Editor for Story Writing. arXiv preprint arXiv:2107.07430(2021).Google Scholar
- Lucas Colusso, Cynthia L. Bennett, Gary Hsieh, and Sean A. Munson. 2017. Translational Resources: Reducing the Gap Between Academic Research and HCI Practice. In Proceedings of the 2017 Conference on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association for Computing Machinery, New York, NY, USA, 957–968. https://doi.org/10.1145/3064663.3064667Google ScholarDigital Library
- Scott Davidoff, Min Kyung Lee, Anind K Dey, and John Zimmerman. 2007. Rapidly exploring application design through speed dating. In International Conference on Ubiquitous Computing. Springer, 429–446. https://doi.org/10.1007/978-3-540-74853-3_25Google ScholarCross Ref
- Nicholas Davis, Chih-PIn Hsiao, Kunwar Yashraj Singh, Lisa Li, and Brian Magerko. 2016. Empirically Studying Participatory Sense-Making in Abstract Drawing with a Co-Creative Cognitive Agent. In Proceedings of the 21st International Conference on Intelligent User Interfaces (Sonoma, California, USA) (IUI ’16). Association for Computing Machinery, New York, NY, USA, 196–207. https://doi.org/10.1145/2856767.2856795Google ScholarDigital Library
- Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A Mobile App Dataset for Building Data-Driven Design Applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery, New York, NY, USA, 845–854. https://doi.org/10.1145/3126594.3126651Google ScholarDigital Library
- Zhen Feng, Jiaqi Fang, Bo Cai, and Yingtao Zhang. 2021. GUIS2Code: A Computer Vision Tool to Generate Code Automatically from Graphical User Interface Sketches. In International Conference on Artificial Neural Networks. Springer, 53–65.Google ScholarDigital Library
- Nishit Gajjar, Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Matthias Jarke. 2021. Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning. (2021), 40–42. https://doi.org/10.1145/3397482.3450727Google Scholar
- Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2015. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576(2015).Google Scholar
- Katy Ilonka Gero and Lydia B Chilton. 2019. Metaphoria: An algorithmic companion for metaphor creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.Google ScholarDigital Library
- Leo A Goodman. 1961. Snowball sampling. The annals of mathematical statistics(1961), 148–170.Google Scholar
- Colin M. Gray. 2014. Evolution of Design Competence in UX Practice. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1645–1654. https://doi.org/10.1145/2556288.2557264Google ScholarDigital Library
- David Ha and Douglas Eck. 2017. A neural representation of sketch drawings. arXiv preprint arXiv:1704.03477(2017).Google Scholar
- Eric Horvitz. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Pittsburgh, Pennsylvania, USA) (CHI ’99). Association for Computing Machinery, New York, NY, USA, 159–166. https://doi.org/10.1145/302979.303030Google ScholarDigital Library
- Forrest Huang, John F. Canny, and Jeffrey Nichols. 2019. Swire: Sketch-Based User Interface Retrieval. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3290605.3300334Google ScholarDigital Library
- Forrest Huang, Gang Li, Xin Zhou, John F Canny, and Yang Li. 2021. Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models. arXiv preprint arXiv:2110.07775(2021).Google Scholar
- Yue Jiang, Yuwen Lu, Jeffrey Nichols, Wolfgang Stuerzlinger, Chun Yu, Christof Lutteroth, Yang Li, Ranjitha Kumar, and Toby Jia-Jun Li. 2022. Computational Approaches for Understanding, Generating, and Adapting User Interfaces. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA. https://doi.org/10.1145/3491101.3504030Google ScholarDigital Library
- Pegah Karimi, Jeba Rezwana, Safat Siddiqui, Mary Lou Maher, and Nasrin Dehbozorgi. 2020. Creative sketching partner: an analysis of human-AI co-creativity. In Proceedings of the 25th International Conference on Intelligent User Interfaces. 221–230.Google ScholarDigital Library
- Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114(2013).Google Scholar
- Max Kreminski, Melanie Dickinson, Michael Mateas, and Noah Wardrip-Fruin. 2020. Why Are We Like This?: The AI Architecture of a Co-Creative Storytelling Game. In International Conference on the Foundations of Digital Games (Bugibba, Malta) (FDG ’20). Association for Computing Machinery, New York, NY, USA, Article 13, 4 pages. https://doi.org/10.1145/3402942.3402953Google ScholarDigital Library
- James A Landay. 1996. SILK: sketching interfaces like krazy. In Conference companion on Human factors in computing systems. 398–399.Google ScholarDigital Library
- Effie Lai-Chong Law, Paul Van Schaik, and Virpi Roto. 2014. Attitudes towards user experience (UX) measurement. International Journal of Human-Computer Studies 72, 6 (2014), 526–541.Google ScholarCross Ref
- Hsin-Ying Lee, Lu Jiang, Irfan Essa, Phuong B Le, Haifeng Gong, Ming-Hsuan Yang, and Weilong Yang. 2020. Neural Design Network: Graphic Layout Generation with Constraints. (2020). https://doi.org/10.1007/978-3-030-58580-8Google Scholar
- Jianan Li, Jimei Yang, Aaron Hertzmann, Jianming Zhang, and Tingfa Xu. 2019. LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators. 7th International Conference on Learning Representations, ICLR 2019 (jan 2019). arxiv:1901.06767Google Scholar
- Toby Jia-Jun Li, Lindsay Popowski, Tom Mitchell, and Brad A Myers. 2021. Screen2Vec: Semantic Embedding of GUI Screens and GUI Components. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.Google Scholar
- Q Vera Liao, Moninder Singh, Yunfeng Zhang, and Rachel Bellamy. 2021. Introduction to explainable ai. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–3.Google ScholarDigital Library
- Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J Cai. 2020. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Kevin Moran, Carlos Bernal-Cardenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2020. Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps. IEEE Transactions on Software Engineering 46, 2 (feb 2020), 196–221. https://doi.org/10.1109/TSE.2018.2844788Google ScholarCross Ref
- Michael J Muller and Sarah Kuhn. 1993. Participatory design. Commun. ACM 36, 6 (1993), 24–28.Google ScholarDigital Library
- Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, Present, and Future of User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 7, 1 (mar 2000), 3–28. https://doi.org/10.1145/344949.344959Google ScholarDigital Library
- Brad A. Myers. 1995. User Interface Software Tools. ACM Trans. Comput.-Hum. Interact. 2, 1 (mar 1995), 64–103. https://doi.org/10.1145/200968.200971Google ScholarDigital Library
- Brad A Myers, Dario A Giuse, Roger B Dannenberg, Brad Vander Zanden, David S Kosbie, Edward Pervin, Andrew Mickish, and Philippe Marchal. 1995. GARNET comprehensive support for graphical, highly interactive user interfaces. In Readings in Human–Computer Interaction. Elsevier, 357–371.Google Scholar
- Jakob Nielsen. 2017. A 100-Year View of User Experience (by Jakob Nielsen). https://www.nngroup.com/articles/100-years-ux/Google Scholar
- Donald A Norman. 2010. The research-Practice Gap: The need for translational developers. interactions 17, 4 (2010), 9–12.Google ScholarDigital Library
- Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174223Google ScholarDigital Library
- Kara Pernice. 2018. Affinity Diagramming: Collaboratively Sort UX Findings & Design Ideas. https://www.nngroup.com/articles/affinity-diagram/Google Scholar
- P Jonathon Phillips, Carina A Hahn, Peter C Fontana, David A Broniatowski, and Mark A Przybocki. 2020. Four principles of explainable artificial intelligence. Gaithersburg, Maryland(2020).Google Scholar
- Mitchel Resnick, Brad Myers, Kumiyo Nakakoji, Ben Shneiderman, Randy Pausch, Ted Selker, and Mike Eisenberg. 2005. Design principles for tools to support creative thinking. (2005).Google Scholar
- Joy Robinson, Candice Lanius, and Ryan Weber. 2018. The past, present, and future of UX empirical research. Communication Design Quarterly Review 5, 3 (2018), 10–23.Google ScholarDigital Library
- [42] Maria Rosala.2019. https://www.nngroup.com/articles/thematic-analysis/Google Scholar
- Emma Rose and Josh Tenenberg. 2017. Making practice-level struggles visible: researching UX practice to inform pedagogy. Communication Design Quarterly Review 5, 1 (2017), 89–97.Google ScholarDigital Library
- Daniel M Russell and Ed H Chi. 2014. Looking back: Retrospective study methods for HCI. In Ways of Knowing in HCI. Springer, 373–393.Google Scholar
- [45] Kim Salazar.2020. https://www.nngroup.com/articles/contextual-inquiry/Google Scholar
- Vinoth Pandian Sermuga Pandian, Sarah Suleri, and Prof. Dr. Matthias Jarke. 2021. UISketch: A Large-Scale Dataset of UI Element Sketches. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, Article 319, 14 pages. https://doi.org/10.1145/3411764.3445784Google ScholarDigital Library
- Ben Shneiderman, Michael Leavitt, 2006. Research-based web design & usability guidelines. Washington DC, Department of Health and Human Services (2006).Google Scholar
- Erik Stolterman. 2008. The nature of design practice and implications for interaction design research. International Journal of Design 2, 1 (2008).Google Scholar
- Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. ProtoAI: Model-Informed Prototyping for AI-Powered Interfaces. In 26th International Conference on Intelligent User Interfaces. Association for Computing Machinery, New York, NY, USA, 48–58. https://doi.org/10.1145/3397481.3450640Google Scholar
- Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J Ko. 2020. Scout: Rapid exploration of interface layout alternatives through high-level design constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.Google ScholarDigital Library
- Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. 2021. Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. 498–510.Google ScholarDigital Library
- Marcus Winter and Phil Jackson. 2020. Flatpack ML: How to support designers in creating a new generation of customizable machine learning applications. In International Conference on Human-Computer Interaction. Springer, 175–193.Google ScholarDigital Library
- Jacob O Wobbrock, Shaun K Kane, Krzysztof Z Gajos, Susumu Harada, and Jon Froehlich. 2011. Ability-based design: Concept, principles and examples. ACM Transactions on Accessible Computing (TACCESS) 3, 3 (2011), 1–27.Google ScholarDigital Library
- Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021. Don’t do that! hunting down visual design smells in complex uis against design guidelines. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 761–772.Google ScholarDigital Library
- Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference (Hong Kong, China) (DIS ’18). Association for Computing Machinery, New York, NY, USA, 585–596. https://doi.org/10.1145/3196709.3196730Google ScholarDigital Library
- Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining whether, why, and how human-AI interaction is uniquely difficult to design. In Proceedings of the 2020 chi conference on human factors in computing systems. 1–13.Google ScholarDigital Library
- ZiCheng Zhang, CongYing Han, and TianDe Guo. 2021. ExSinGAN: Learning an Explainable Generative Model from a Single Image. arXiv preprint arXiv:2105.07350(2021).Google Scholar
- Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. 2021. GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 748–760. https://doi.org/10.1109/ICSE43902.2021.00074Google ScholarDigital Library
Recommendations
Using human factors standards to support user experience and agile design
UAHCI'13: Proceedings of the 7th international conference on Universal Access in Human-Computer Interaction: design methods, tools, and interaction techniques for eInclusion - Volume Part IThe ISO 9241-210 standard provides a framework for human-centred design (HCD) activities comprising the four stages: context of use, specification of user and organisational requirements, design solutions, and evaluation against requirements. Other ...
Task Allocation Between UX Specialists and Developers in Agile Software Development Projects
Human-Computer Interaction – INTERACT 2015AbstractSynchronizing efforts between developers and user experience (UX) specialists is one of the major challenges in agile UX work. In this paper, we report results of a study conducted over a release cycle of six agile software development projects in ...
Usability and UX practices in small enterprises: lessons from a survey of the Brazilian context
SIGDOC '16: Proceedings of the 34th ACM International Conference on the Design of CommunicationThe role of small and medium enterprises (SME), especially small businesses, has significantly increased in the developing of interactive applications. Given the relevance of these companies in the market, the adoption of usability and user experience (...
Comments