skip to main content
10.1145/3613904.3642812acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Understanding Nonlinear Collaboration between Human and AI Agents: A Co-design Framework for Creative Design

Published: 11 May 2024 Publication History

Abstract

Creative design is a nonlinear process where designers generate diverse ideas in the pursuit of an open-ended goal and converge towards consensus through iterative remixing. In contrast, AI-powered design tools often employ a linear sequence of incremental and precise instructions to approximate design objectives. Such operations violate customary creative design practices and thus hinder AI agents’ ability to complete creative design tasks. To explore better human-AI co-design tools, we first summarize human designers’ practices through a formative study with 12 design experts. Taking graphic design as a representative scenario, we formulate a nonlinear human-AI co-design framework and develop a proof-of-concept prototype, OptiMuse. We evaluate OptiMuse and validate the nonlinear framework through a comparative study. We notice a subconscious change in people’s attitudes towards AI agents, shifting from perceiving them as mere executors to regarding them as opinionated colleagues. This shift effectively fostered the exploration and reflection processes of individual designers.

Supplemental Material

MP4 File - Video Presentation
Video Presentation
Transcript for: Video Presentation

References

[1]
Adobe. 2024. Adobe Firefly – Generative AI for Everyone. https://www.adobe.com/sensei/generative-ai/firefly.html
[2]
Niek Althuizen and Astrid Reichel. 2016. The Effects of IT-Enabled Cognitive Stimulation Tools on Creative Problem Solving: A Dual Pathway to Creativity. Journal of Management Information Systems 33, 1 (2016), 11–44. https://doi.org/10.1080/07421222.2016.1172439
[3]
Talat Ambreen, Naveed Ikram, Muhammad Usman, and Mahmood Niazi. 2018. Empirical research in requirements engineering: trends and opportunities. Requirements Engineering 23 (2018), 63–95.
[4]
Zahra Ashktorab, Q. Vera Liao, Casey Dugan, James Johnson, Qian Pan, Wei Zhang, Sadhana Kumaravel, and Murray Campbell. 2020. Human-AI Collaboration in a Cooperative Game Setting: Measuring Social Perception and Outcomes. Proc. ACM Hum.-Comput. Interact. 4, CSCW2, Article 96 (oct 2020), 20 pages. https://doi.org/10.1145/3415167
[5]
Amy Bruckman. 1998. Community support for constructionist learning. Computer Supported Cooperative Work (CSCW) 7 (1998), 47–86.
[6]
Daniel Buschek, Lukas Mecke, Florian Lehmann, and Hai Dang. 2021. Nine potential pitfalls when designing human-ai co-creative systems. (2021). https://doi.org/10.48550/arXiv.2104.00358 arXiv:arXiv:2104.00358
[7]
Giorgos Cheliotis, Nan Hu, Jude Yew, and Jianhui Huang. 2014. The Antecedents of Remix. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (Baltimore, Maryland, USA) (CSCW ’14). ACM, New York, NY, USA, 1011–1022. https://doi.org/10.1145/2531602.2531730
[8]
Alexander Chernev, Ulf Böckenholt, and Joseph Goodman. 2015. Choice overload: A conceptual review and meta-analysis. Journal of Consumer Psychology 25, 2 (2015), 333–358.
[9]
Elizabeth Clark, Anne Spencer Ross, Chenhao Tan, Yangfeng Ji, and Noah A. Smith. 2018. Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories. In 23rd International Conference on Intelligent User Interfaces (Tokyo, Japan) (IUI ’18). ACM, New York, NY, USA, 329–340. https://doi.org/10.1145/3172944.3172983
[10]
Design Council. 2024. Framework for Innovation - Design Council. https://www.designcouncil.org.uk/our-resources/framework-for-innovation/. Last accessed on 02.05.2024.
[11]
N. Dahlback, A. Jönsson, and L. Ahrenberg. 1993. Wizard of Oz studies–why and how. Knowledge-based systems 6, 4 (1993), 258–266. https://doi.org/10.1016/0950-7051(93)90017-N
[12]
Sayamindu Dasgupta, William Hale, Andrés Monroy-Hernández, and Benjamin Mako Hill. 2016. Remixing as a Pathway to Computational Thinking. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW ’16). ACM, New York, NY, USA, 1438–1449. https://doi.org/10.1145/2818048.2819984
[13]
Senay Tuna Demirel and Resul Das. 2018. Software requirement analysis: Research challenges and technical approaches. In 2018 6th International Symposium on Digital Forensic and Security (ISDFS). IEEE, Antalya, Turkey, 1–6. https://doi.org/10.1109/ISDFS.2018.8355322
[14]
Gerhard Fischer. 2004. Social creativity: turning barriers into opportunities for collaborative design. In Proceedings of the Eighth Conference on Participatory Design: Artful Integration: Interweaving Media, Materials and Practices - Volume 1 (Toronto, Ontario, Canada) (PDC 04). ACM, New York, NY, USA, 152–161. https://doi.org/10.1145/1011870.1011889
[15]
Linda Flower and John R. Hayes. 1981. A Cognitive Process Theory of Writing. College Composition and Communication 32, 4 (1981), 365–387. https://doi.org/10.2307/356600
[16]
Scratch Foundation. 2022. Scratch - Imagine, Program, Share. https://en.scratch-wiki.info/. Last accessed on 02.05.2024.
[17]
Norman M. Fraser and G.Nigel Gilbert. 1991. Simulating speech systems. Computer Speech & Language 5, 1 (1991), 81–99. https://doi.org/10.1016/0885-2308(91)90019-M
[18]
Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). ACM, New York, NY, USA, 1–18. https://doi.org/10.1145/3290605.3300619
[19]
John S Gero. 1990. Design prototypes: a knowledge representation schema for design. AI magazine 11, 4 (1990), 26–26.
[20]
John S Gero and Udo Kannengiesser. 2004. The situated function–behaviour–structure framework. Design studies 25, 4 (2004), 373–391.
[21]
Victor Girotto. 2016. Collective Creativity through a Micro-Tasks Crowdsourcing Approach. In Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion (San Francisco, California, USA) (CSCW ’16 Companion). ACM, New York, NY, USA, 143–146. https://doi.org/10.1145/2818052.2874356
[22]
Google. 2022. Imagen: Text-to-Image Diffusion Models. https://imagen.research.google/. Last accessed on 14.09.2023.
[23]
Google. 2022. Parti: Pathways Autoregressive Text-to-Image Model. https://parti.research.google/. Last accessed on 14.09.2023.
[24]
A. Green, H. Huttenrauch, and K.S. Eklundh. 2004. Applying the Wizard-of-Oz framework to cooperative service discovery and configuration. In RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No.04TH8759). IEEE, Kurashiki, Japan, 575–580. https://doi.org/10.1109/ROMAN.2004.1374824
[25]
The Dynamic Medium Group. n.d. Dynamicland. https://dynamicland.org/. Last accessed on 02.05.2024.
[26]
Matthew Guzdial and Mark Riedl. 2019. An interaction framework for studying co-creative ai. (2019). https://doi.org/10.48550/arXiv:1903.09709 arXiv:arXiv:1903.09709
[27]
Armand Hatchuel and Benoit Weil. 2003. A new approach of innovative Design: an introduction to CK theory. In DS 31: Proceedings of ICED 03, the 14th International Conference on Engineering Design. the Design Society, Stockholm, Sweden, 109–110.
[28]
T.J. Howard, S.J. Culley, and E. Dekoninck. 2008. Describing the creative design process by the integration of engineering design and cognitive psychology literature. Design Studies 29, 2 (2008), 160–180. https://doi.org/10.1016/j.destud.2008.01.001
[29]
Masa Inakage. 2007. Collective Creativity: Toward a New Paradigm for Creative Culture. In Proceedings of the 2nd International Conference on Digital Interactive Media in Entertainment and Arts (Perth, Australia) (DIMEA ’07). ACM, New York, NY, USA, 8. https://doi.org/10.1145/1306813.1306822
[30]
Anna Kantosalo and Hannu Toivonen. 2016. Modes for creative human-computer collaboration: Alternating and task-divided co-creativity. In Proceedings of the seventh international conference on computational creativity. 77–84.
[31]
Chinmay Kulkarni, Stefania Druga, Minsuk Chang, Alex Fiannaca, Carrie Cai, and Michael Terry. 2023. A Word is Worth a Thousand Pictures: Prompts as AI Design Material. (2023). https://doi.org/10.48550/arXiv.2303.12647 arXiv:arXiv:2303.12647
[32]
Haotian Li, Yun Wang, Q Vera Liao, and Huamin Qu. 2023. Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling. (2023). https://doi.org/10.48550/arXiv.2304.08366 arXiv:arXiv:2304.08366
[33]
Haotian Li, Yun Wang, and Huamin Qu. 2023. Where Are We So Far? Understanding Data Storytelling Tools from the Perspective of Human-AI Collaboration. (2023). https://doi.org/10.48550/arXiv.2309.15723 arXiv:arXiv:2309.15723
[34]
Daniel Lopes, João Correia, and Penousal Machado. 2022. EvoDesigner: Towards Aiding Creativity in Graphic Design. In Artificial Intelligence in Music, Sound, Art and Design. Springer, Cham, 162–178.
[35]
Ryan Louie, Andy Coenen, Cheng Zhi Huang, Michael Terry, and Carrie J. Cai. 2020. Novice-AI Music Co-Creation via AI-Steering Tools for Deep Generative Models. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). ACM, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376739
[36]
Todd Lubart. 2005. How can computers be partners in the creative process: classification and commentary on the special issue. International journal of human-computer studies 63, 4-5 (2005), 365–369.
[37]
Martin Maguire and Nigel Bevan. 2002. User requirements analysis: a review of supporting methods. In IFIP World Computer Congress, TC 13. Springer, Boston, MA, 133–148.
[38]
Mary Lou Maher. 2012. Computational and collective creativity: Who’s being creative?. In Proceedings of the International Conference on Computational Creativity. Association for Computational Creativity, Jönköping, Sweden, 67–71.
[39]
Jennifer Marlow and Laura Dabbish. 2014. From rookie to all-star: professional development in a graphic design social networking site. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 922–933.
[40]
Microsoft. 2023. Microsoft 365 Copilot. https://www.microsoft.com/en-us/microsoft-365/blog/2023/03/16/introducing-microsoft-365-copilot-a-whole-new-way-to-work/. Last accessed on 14.09.2023.
[41]
Midjourney. 2022. Midjourney. https://www.midjourney.com/. Last accessed on 14.09.2023.
[42]
Michael D. Mumford and Sigrid B. Gustafson. 1988. Creativity syndrome: Integration, application, and innovation. Psychological Bulletin 103, 1 (1988), 27–43. https://doi.org/10.1037/0033-2909.103.1.27
[43]
Felix Müller-Wienbergen, Oliver Müller, Stefan Seidel, and Jörg Becker. 2011. Leaving the beaten tracks in creative work–A design theory for systems that support convergent and divergent thinking. Journal of the Association for Information Systems 12, 11 (2011), 2.
[44]
SANTIAGO NEGRETE YANKELEVICH and NORA ANGELICA MORALES ZARAGOZA. 2014. The apprentice framework: planning and assessing creativity. In Proceedings of the International Conference on Computational Creativity. Association for Computational Creativity, Jönköping, Sweden, 280–283.
[45]
P. D. T. O’Connor. 1991. Total design: integrated methods for successful product engineering. Wiley Online Library.
[46]
Lora Oehlberg, Wesley Willett, and Wendy E. Mackay. 2015. Patterns of Physical Design Remixing in Online Maker Communities. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). ACM, New York, NY, USA, 639–648. https://doi.org/10.1145/2702123.2702175
[47]
Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. 2018. I Lead, You Help but Only with Enough Details: Understanding User Experience of Co-Creation with Artificial Intelligence. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). ACM, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3174223
[48]
OpenAI. 2022. DALL·E 2. https://openai.com/dall-e-2/. Last accessed on 14.09.2023.
[49]
OpenAI. 2022. DALL·E: Creating Images from Text. https://openai.com/blog/dall-e/. Last accessed on 31.08.2022.
[50]
Katherine O’Toole. 2023. Collaborative Creativity in TikTok Music Duets. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (, Hamburg, Germany, ) (CHI ’23). ACM, New York, NY, USA, Article 791, 16 pages. https://doi.org/10.1145/3544548.3581380
[51]
Lihang Pan, Chun Yu, Zhe He, and Yuanchun Shi. 2023. A Human-Computer Collaborative Editing Tool for Conceptual Diagrams. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 360, 29 pages. https://doi.org/10.1145/3544548.3580676
[52]
Brandon Reynante, Steven P. Dow, and Narges Mahyar. 2021. A Framework for Open Civic Design: Integrating Public Participation, Crowdsourcing, and Design Thinking. Digital Government: Research and Practice 2, 4 (Oct. 2021), 1–22. https://doi.org/10.1145/3487607
[53]
Jeba Rezwana and Mary Lou Maher. 2022. Understanding User Perceptions, Collaborative Experience and User Engagement in Different Human-AI Interaction Designs for Co-Creative Systems. In Proceedings of the 14th Conference on Creativity and Cognition (Venice, Italy) (C&C ’22). ACM, New York, NY, USA, 38–48. https://doi.org/10.1145/3527927.3532789
[54]
Laurel D. Riek. 2012. Wizard of Oz studies in HRI: a systematic review and new reporting guidelines. J. Hum.-Robot Interact. 1, 1 (jul 2012), 119–136. https://doi.org/10.5898/JHRI.1.1.Riek
[55]
Winston W Royce. 1987. Managing the development of large software systems: concepts and techniques. In Proceedings of the 9th international conference on Software Engineering. ACM, New York, NY, USA, 328–338.
[56]
Purvi Saraiya, Chris North, and Karen Duca. 2005. Visualizing biological pathways: requirements analysis, systems evaluation and research agenda. Information Visualization 4, 3 (2005), 191–205.
[57]
Yang Shi, Tian Gao, Xiaohan Jiao, and Nan Cao. 2023. Understanding Design Collaboration Between Designers and Artificial Intelligence: A Systematic Literature Review. Proc. ACM Hum.-Comput. Interact. 7, CSCW2, Article 368 (oct 2023), 35 pages. https://doi.org/10.1145/3610217
[58]
Juan Tan, Congcong Qi, Xiaohui Gao, Jianle Lu, and Qiong Tan. 2022. Conflict or Collaboration—The Impact of Knowledge Endowment Heterogeneity on Remix in Open Collaborative Communities. Frontiers in Psychology 13 (2022), 941448.
[59]
UIUX Trend. 2024. System Usability Scale (SUS) | Usability.gov. https://uiuxtrend.com/sus-calculator/. Last accessed on 11.02.24.
[60]
Dakuo Wang, Elizabeth Churchill, Pattie Maes, Xiangmin Fan, Ben Shneiderman, Yuanchun Shi, and Qianying Wang. 2020. From Human-Human Collaboration to Human-AI Collaboration: Designing AI Systems That Can Work Together with People. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 1–6. https://doi.org/10.1145/3334480.3381069
[61]
Kai Wang and Jeffrey V. Nickerson. 2017. A literature review on individual creativity support systems. Computers in Human Behavior 74 (2017), 139–151.
[62]
Koji Yatani. 2014. hcistats:wilcoxonsigned [Koji Yatani’s Course Webpage]. https://yatani.jp/teaching/doku.php?id=hcistats:wilcoxonsigned. Last accessed on 14.09.2023.
[63]
Feng Yue, Ma Qiao-ge, and Ren Nan. 2019. Improving makers’ remix contribution behavior in the open innovation communities——An example of Thingiverse. In 2019 4th International Conference on Social Sciences and Economic Development (ICSSED 2019). Atlantis Press, GX Dordrecht, The Netherlands, 224–229.
[64]
Shahid Yusuf. 2009. From creativity to innovation. Technology in society 31, 1 (2009), 1–8.

Cited By

View all
  • (2025)Understanding the Differences in an AI-Based Creativity Support Tool Between Creativity Types in Fashion DesignInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2448484(1-14)Online publication date: 14-Jan-2025
  • (2024)Design “Made by Human” in the World of AI: An Analysis of the Impact of Artificial Intelligence on Graphic Design and the Role of Humans in the Creative ProcessMedia & Marketing Identity10.34135/mmidentity-2024-34(332-338)Online publication date: 2024
  • (2024)Making the case for introducing generative artificial intelligence (AI) into design curriculaArt, Design & Communication in Higher Education10.1386/adch_00088_1Online publication date: 14-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Creative Design
  2. Creativity Support Tool
  3. Human-AI Co-creativity

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

CHI '24

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2,623
  • Downloads (Last 6 weeks)376
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Understanding the Differences in an AI-Based Creativity Support Tool Between Creativity Types in Fashion DesignInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2448484(1-14)Online publication date: 14-Jan-2025
  • (2024)Design “Made by Human” in the World of AI: An Analysis of the Impact of Artificial Intelligence on Graphic Design and the Role of Humans in the Creative ProcessMedia & Marketing Identity10.34135/mmidentity-2024-34(332-338)Online publication date: 2024
  • (2024)Making the case for introducing generative artificial intelligence (AI) into design curriculaArt, Design & Communication in Higher Education10.1386/adch_00088_1Online publication date: 14-Jun-2024
  • (2024)Exploring the Impact of AI-generated Image Tools on Professional and Non-professional Users in the Art and Design FieldsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681890(451-458)Online publication date: 11-Nov-2024
  • (2024)Exploring the Potential for Generative AI-based Conversational Cues for Real-Time Collaborative IdeationProceedings of the 16th Conference on Creativity & Cognition10.1145/3635636.3656184(117-131)Online publication date: 23-Jun-2024
  • (2024)ArtEyer: Enriching GPT-based agents with contextual data visualizations for fine art authenticationVisual Informatics10.1016/j.visinf.2024.11.0018:4(48-59)Online publication date: Dec-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media