Florian Lehmann
ABSTRACT
Machine learning models provide functions to transform and generate image and text data. This promises powerful applications but it remains unclear how users can interact with these models. With my research, I focus on designing, implementing, and evaluating functional prototypes for understanding human-AI interactions. Methodologically, I focus on web-based experiments with a mixed-methods approach. Furthermore, I use these prototypes and generative models as a material to understand fundamental concepts in human-AI interactions, such as initiative, intent, and control. In an already conducted study, for example, I showed that the levels of initiative and control afforded by the UI influence perceived authorship when writing text. For the future, I plan to carry out more studies on collaborative writing. With my dissertation, I contribute to how we will build human-AI interactions and how we will collaborate with computational generative systems in future.
- Saleema Amershi, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, Eric Horvitz, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, and Paul N. Bennett. 2019. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233Google Scholar
Digital Library
- Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2018. Sentiment Bias in Predictive Text Recommendations Results in Biased Writing. In Proceedings of the 44th Graphics Interface Conference(GI ’18). Canadian Human-Computer Communications Society, Toronto, Canada, 42–49. https://doi.org/10.20380/GI2018.07Google ScholarDigital Library
- Kenneth C. Arnold, Krysta Chauncey, and Krzysztof Z. Gajos. 2020. Predictive text encourages predictable writing. In Proceedings of the 25th International Conference on Intelligent User Interfaces(IUI ’20). Association for Computing Machinery, Cagliari, Italy, 128–138. https://doi.org/10.1145/3377325.3377523Google ScholarDigital Library
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] (July 2020). http://arxiv.org/abs/2005.14165 arXiv:2005.14165.Google Scholar
- Daniel Buschek, Martin Zürn, and Malin Eiband. 2021. The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers. arXiv:2101.09157 [cs] (Jan. 2021). https://doi.org/10.1145/3411764.3445372 arXiv:2101.09157.Google ScholarDigital Library
- John Joon Young Chung, Wooseok Kim, Kang Min Yoo, Hwaran Lee, Eytan Adar, and Minsuk Chang. 2022. TaleBrush: Sketching Stories with Generative Pretrained Language Models. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–19. https://doi.org/10.1145/3491102.3501819Google ScholarDigital Library
- Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. 2017. UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, USA, 278–288. https://doi.org/10.1145/3025453.3025739Google ScholarDigital Library
- Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing using Language Models. In Designing Interactive Systems Conference. ACM, Virtual Event Australia, 1002–1019. https://doi.org/10.1145/3532106.3533533Google ScholarDigital Library
- Matthew Guzdial and Mark Riedl. 2019. An Interaction Framework for Studying Co-Creative AI. arXiv:1903.09709 [cs] (March 2019). http://arxiv.org/abs/1903.09709 arXiv:1903.09709.Google Scholar
- Felix Henninger, Yury Shevchenko, Ulf Kai Mertens, Pascal J. Kieslich, and Benjamin E. Hilbig. 2019. lab.js: A free, open, online study builder. preprint. PsyArXiv. https://doi.org/10.31234/osf.io/fqr49Google ScholarCross Ref
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems(CHI ’99). ACM Press, New York, NY, USA, 159–166. https://doi.org/10.1145/302979.303030Google ScholarDigital Library
- Anna Kantosalo and Hannu Toivonen. 2016. Modes for Creative Human-Computer Collaboration: Alternating and Task-Divided Co-Creativity. In Proceedings of the Seventh International Conference on Computational Creativity (ICCC 2016). Sony CSL, Paris, 77–84. http://www.computationalcreativity.net/iccc2016/wp-content/uploads/2016/01/Modes-for-Creative-Human-Computer-Collaboration.pdfGoogle Scholar
- Mina Lee, Percy Liang, and Qian Yang. 2022. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–19. https://doi.org/10.1145/3491102.3502030Google ScholarDigital Library
- Florian Lehmann and Daniel Buschek. 2021. Examining Autocompletion as a Basic Concept for Interaction with Generative AI. i-com 19, 3 (Jan. 2021), 251–264. https://doi.org/10.1515/icom-2020-0025Google ScholarCross Ref
- Florian Lehmann, Niklas Markert, Hai Dang, and Daniel Buschek. 2022. Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship. In Mensch und Computer 2022. ACM, Darmstadt Germany, 192–208. https://doi.org/10.1145/3543758.3543947Google ScholarDigital Library
- Sebastiaan Mathôt, Daniel Schreij, and Jan Theeuwes. 2012. OpenSesame: An open-source, graphical experiment builder for the social sciences. Behavior Research Methods 44, 2 (June 2012), 314–324. https://doi.org/10.3758/s13428-011-0168-7Google ScholarCross Ref
- Santiago Negrete-Yankelevich and Nora Morales-Zaragoza. 2014. The apprentice framework: planning and assessing creativity. In Proceedings of the Seventh International Conference on Computational Creativity (ICCC 2014). computationalcreativity.net, Ljubljana, 280–283. http://computationalcreativity.net/iccc2014/wp-content/uploads/2014/06/13.4_NegreteYankelevich.pdfGoogle Scholar
- Jonathan W. Peirce. 2007. PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods 162, 1-2 (May 2007), 8–13. https://doi.org/10.1016/j.jneumeth.2006.11.017Google ScholarCross Ref
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. (2019), 24. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdfGoogle Scholar
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. http://jmlr.org/papers/v21/20-074.htmlGoogle Scholar
- Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-Shot Text-to-Image Generation. In Proceedings of the 38th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8821–8831. https://proceedings.mlr.press/v139/ramesh21a.htmlGoogle Scholar
- Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–7. https://doi.org/10.1145/3411763.3451760Google ScholarDigital Library
- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-Resolution Image Synthesis With Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10684–10695.Google ScholarCross Ref
- Oliver Schmitt and Daniel Buschek. 2021. CharacterChat: Supporting the Creation of Fictional Characters through Conversation and Progressive Manifestation with a Chatbot. In Creativity and Cognition. ACM, Virtual Event Italy, 1–10. https://doi.org/10.1145/3450741.3465253Google ScholarDigital Library
- William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Houghton, Mifflin and Company, Boston, MA, US. Pages: xxi, 623.Google Scholar
- Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2020. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. http://arxiv.org/abs/1909.08053 arXiv:1909.08053 [cs].Google Scholar
- Nikhil Singh, Guillermo Bernal, Daria Savchenko, and Elena L. Glassman. 2022. Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence. ACM Transactions on Computer-Human Interaction (Feb. 2022), 3511599. https://doi.org/10.1145/3511599Google ScholarDigital Library
- Hendrik Strobelt, Albert Webson, Victor Sanh, Benjamin Hoover, Johanna Beyer, Hanspeter Pfister, and Alexander M. Rush. 2022. Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. IEEE Transactions on Visualization and Computer Graphics (2022), 1–11. https://doi.org/10.1109/TVCG.2022.3209479Google ScholarCross Ref
- Kashyap Todi, Luis A. Leiva, Daniel Buschek, Pin Tian, and Antti Oulasvirta. 2021. Conversations with GUIs. In Designing Interactive Systems Conference 2021. ACM, Virtual Event USA, 1447–1457. https://doi.org/10.1145/3461778.3462124Google ScholarDigital Library
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS’17). Curran Associates Inc., Long Beach, California, USA, 6000–6010.Google ScholarDigital Library
- Tongshuang Wu, Michael Terry, and Carrie J. Cai. 2021. AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. arXiv:2110.01691 [cs] (Oct. 2021). http://arxiv.org/abs/2110.01691 arXiv:2110.01691.Google Scholar
- Qian Yang, Justin Cranshaw, Saleema Amershi, Shamsi T. Iqbal, and Jaime Teevan. 2019. Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19. ACM Press, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300415Google ScholarDigital Library
- Qian Yang, Alex Scuito, John Zimmerman, Jodi Forlizzi, and Aaron Steinfeld. 2018. Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference(DIS ’18). Association for Computing Machinery, Hong Kong, China, 585–596. https://doi.org/10.1145/3196709.3196730Google ScholarDigital Library
- Qian Yang, Aaron Steinfeld, Carolyn Rosé, and John Zimmerman. 2020. Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). ACM Press, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376301Google ScholarDigital Library
- Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces. ACM, Helsinki Finland, 841–852. https://doi.org/10.1145/3490099.3511105Google ScholarDigital Library
- Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trained Transformer Language Models. http://arxiv.org/abs/2205.01068 arXiv:2205.01068 [cs].Google Scholar
Index Terms
- Mixed-Initiative Interaction with Computational Generative Systems
Recommendations
An Evidential Model for Tracking Initiative in Collaborative Dialogue Interactions
In this paper, we argue for the need to distinguish between task initiative and dialogue initiative, and present an evidential model for tracking shifts in both types of initiatives in collaborative dialogue interactions. Our model predicts the task and ...
Mixed-initiative interaction = mixed computation
We show that partial evaluation can be usefully viewed as a programming model for realizing mixed-initiative functionality in interactive applications. Mixed-initiative interaction between two participants is one where the parties can take turns at any ...
Mixed-initiative interaction = mixed computation
PEPM '02: Proceedings of the 2002 ACM SIGPLAN workshop on Partial evaluation and semantics-based program manipulationWe show that partial evaluation can be usefully viewed as a programming model for realizing mixed-initiative functionality in interactive applications. Mixed-initiative interaction between two participants is one where the parties can take turns at any ...
Comments