skip to main content
10.1145/3544549.3577061acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
extended-abstract

Mixed-Initiative Interaction with Computational Generative Systems

Published: 19 April 2023 Publication History

Abstract

Machine learning models provide functions to transform and generate image and text data. This promises powerful applications but it remains unclear how users can interact with these models. With my research, I focus on designing, implementing, and evaluating functional prototypes for understanding human-AI interactions. Methodologically, I focus on web-based experiments with a mixed-methods approach. Furthermore, I use these prototypes and generative models as a material to understand fundamental concepts in human-AI interactions, such as initiative, intent, and control. In an already conducted study, for example, I showed that the levels of initiative and control afforded by the UI influence perceived authorship when writing text. For the future, I plan to carry out more studies on collaborative writing. With my dissertation, I contribute to how we will build human-AI interactions and how we will collaborate with computational generative systems in future.

References

[1]
Saleema Amershi, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, and Paul N. Bennett. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233
[2]
Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2018. Sentiment Bias in Predictive Text Recommendations Results in Biased Writing. In Proceedings of the 44th Graphics Interface Conference(GI ’18). Canadian Human-Computer Communications Society, Toronto, Canada, 42–49. https://doi.org/10.20380/GI2018.07
[3]
Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces(IUI ’20). Association for Computing Machinery, Cagliari, Italy, 128–138. https://doi.org/10.1145/3377325.3377523
[4]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] (July 2020). http://arxiv.org/abs/2005.14165 arXiv:2005.14165.
[5]
Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. arXiv:2101.09157 [cs] (Jan. 2021). https://doi.org/10.1145/3411764.3445372 arXiv:2101.09157.
[6]
John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–19. https://doi.org/10.1145/3491102.3501819
[7]
Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 278–288. https://doi.org/10.1145/3025453.3025739
[8]
Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing using Language Models. In Designing Interactive Systems Conference. ACM, Virtual Event Australia, 1002–1019. https://doi.org/10.1145/3532106.3533533
[9]
Matthew Guzdial and Mark Riedl. 2019. An Interaction Framework for Studying Co-Creative AI. arXiv:1903.09709 [cs] (March 2019). http://arxiv.org/abs/1903.09709 arXiv:1903.09709.
[10]
Felix Henninger, Yury Shevchenko, Ulf Kai Mertens, Pascal J. Kieslich, and Benjamin E. Hilbig. 2019. lab.js: A free, open, online study builder. preprint. PsyArXiv. https://doi.org/10.31234/osf.io/fqr49
[11]
Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems(CHI ’99). ACM Press, New York, NY, USA, 159–166. https://doi.org/10.1145/302979.303030
[12]
Anna Kantosalo and Hannu Toivonen. 2016. Modes for Creative Human-Computer Collaboration: Alternating and Task-Divided Co-Creativity. In Proceedings of the Seventh International Conference on Computational Creativity (ICCC 2016). Sony CSL, Paris, 77–84. http://www.computationalcreativity.net/iccc2016/wp-content/uploads/2016/01/Modes-for-Creative-Human-Computer-Collaboration.pdf
[13]
Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–19. https://doi.org/10.1145/3491102.3502030
[14]
Florian Lehmann and Daniel Buschek. 2021. Examining Autocompletion as a Basic Concept for Interaction with Generative AI. i-com 19, 3 (Jan. 2021), 251–264. https://doi.org/10.1515/icom-2020-0025
[15]
Florian Lehmann, Niklas Markert, Hai Dang, and Daniel Buschek. 2022. Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship. In Mensch und Computer 2022. ACM, Darmstadt Germany, 192–208. https://doi.org/10.1145/3543758.3543947
[16]
Sebastiaan Mathôt, Daniel Schreij, and Jan Theeuwes. 2012. OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods 44, 2 (June 2012), 314–324. https://doi.org/10.3758/s13428-011-0168-7
[17]
Santiago Negrete-Yankelevich and Nora Morales-Zaragoza. 2014. The apprentice framework: planning and assessing creativity. In Proceedings of the Seventh International Conference on Computational Creativity (ICCC 2014). computationalcreativity.net, Ljubljana, 280–283. http://computationalcreativity.net/iccc2014/wp-content/uploads/2014/06/13.4_NegreteYankelevich.pdf
[18]
Jonathan W. Peirce. 2007. PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods 162, 1-2 (May 2007), 8–13. https://doi.org/10.1016/j.jneumeth.2006.11.017
[19]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019), 24. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
[20]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. http://jmlr.org/papers/v21/20-074.html
[21]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8821–8831. https://proceedings.mlr.press/v139/ramesh21a.html
[22]
Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–7. https://doi.org/10.1145/3411763.3451760
[23]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.
[24]
Oliver Schmitt and Daniel Buschek. 2021. CharacterChat: Supporting the Creation of Fictional Characters through Conversation and Progressive Manifestation with a Chatbot. In Creativity and Cognition. ACM, Virtual Event Italy, 1–10. https://doi.org/10.1145/3450741.3465253
[25]
William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton, Mifflin and Company, Boston, MA, US. Pages: xxi, 623.
[26]
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. http://arxiv.org/abs/1909.08053 arXiv:1909.08053 [cs].
[27]
Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L. Glassman. 2022. Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence. ACM Transactions on Computer-Human Interaction (Feb. 2022), 3511599. https://doi.org/10.1145/3511599
[28]
Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2022. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. IEEE Transactions on Visualization and Computer Graphics (2022), 1–11. https://doi.org/10.1109/TVCG.2022.3209479
[29]
Kashyap Todi, Luis A. Leiva, Daniel Buschek, Pin Tian, and Antti Oulasvirta. 2021. Conversations with GUIs. In Designing Interactive Systems Conference 2021. ACM, Virtual Event USA, 1447–1457. https://doi.org/10.1145/3461778.3462124
[30]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS’17). Curran Associates Inc., Long Beach, California, USA, 6000–6010.
[31]
Tongshuang Wu, Michael Terry, and Carrie J. Cai. 2021. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. arXiv:2110.01691 [cs] (Oct. 2021). http://arxiv.org/abs/2110.01691 arXiv:2110.01691.
[32]
Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T. Iqbal, and Jaime Teevan. 2019. Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300415
[33]
Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference(DIS ’18). Association for Computing Machinery, Hong Kong, China, 585–596. https://doi.org/10.1145/3196709.3196730
[34]
Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). ACM Press, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376301
[35]
Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. ACM, Helsinki Finland, 841–852. https://doi.org/10.1145/3490099.3511105
[36]
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. http://arxiv.org/abs/2205.01068 arXiv:2205.01068 [cs].

Cited By

View all
  • (2024)Mapping the individual, social and biospheric impacts of Foundation ModelsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658939(776-796)Online publication date: 3-Jun-2024
  • (2024)ChatTL;DR – You Really Ought to Check What the LLM Said on Your BehalfExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3644062(1-7)Online publication date: 11-May-2024
  • (2024)A Design Space for Intelligent and Interactive Writing AssistantsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642697(1-35)Online publication date: 11-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems
April 2023
3914 pages
ISBN:9781450394222
DOI:10.1145/3544549
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Author Tags

  1. control
  2. functional prototypes
  3. generative systems
  4. human-ai interaction
  5. initiative
  6. intent
  7. language model
  8. mixed-initiative
  9. text generation
  10. typing
  11. writing

Qualifiers

  • Extended-abstract
  • Research
  • Refereed limited

Conference

CHI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)385
  • Downloads (Last 6 weeks)67
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Mapping the individual, social and biospheric impacts of Foundation ModelsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658939(776-796)Online publication date: 3-Jun-2024
  • (2024)ChatTL;DR – You Really Ought to Check What the LLM Said on Your BehalfExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3644062(1-7)Online publication date: 11-May-2024
  • (2024)A Design Space for Intelligent and Interactive Writing AssistantsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642697(1-35)Online publication date: 11-May-2024
  • (2024)User Experience Design Professionals’ Perceptions of Generative Artificial IntelligenceProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642114(1-18)Online publication date: 11-May-2024
  • (2023)Stepping Stones for Self-LearningGenerative AI in Teaching and Learning10.4018/979-8-3693-0074-9.ch005(85-142)Online publication date: 5-Dec-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media