skip to main content
10.1145/3648188.3675140acmconferencesArticle/Chapter ViewAbstractPublication PageshtConference Proceedingsconference-collections
research-article
Open access

Measuring Group Creativity of Dialogic Interaction Systems by Means of Remote Entailment Analysis

Published: 10 September 2024 Publication History

Abstract

We present a procedure for assessing group creativity that allows us to compare the contributions of human interlocutors and chatbots based on generative AI such as ChatGPT. We focus on everyday creativity in terms of dialogic communication and test four hypotheses about the difference between human and artificial communication. Our procedure is based on a test that requires interlocutors to cooperatively interpret a sequence of sentences for which we control for coherence gaps with reference to the notion of entailment. Using NLP methods, we automatically evaluate the spoken or written contributions of interlocutors (human or otherwise). The paper develops a routine for automatic transcription based on Whisper, for sampling texts based on their entailment relations, for analyzing dialogic contributions along their semantic embeddings, and for classifying interlocutors and interaction systems based on them. In this way, we highlight differences between human and artificial conversations under conditions that approximate free dialogic communication. We show that despite their obvious classificatory differences, it is difficult to see clear differences even in the domain of dialogic communication given the current instruments of NLP.

References

[1]
Oscar Ardaiz, Maria Luisa Sanz de Acedo, and Maria Teresa Sanz de Acedo. 2008. Wikideas and creativity connector: supporting group ideational creativity. In Proceedings of the 4th International Symposium on Wikis (Porto, Portugal) (WikiSym ’08). Association for Computing Machinery, New York, NY, USA, Article 31, 2 pages. https://doi.org/10.1145/1822258.1822299
[2]
Claus Atzenbeck, Dene Grigar, and Manolis Tzagarakis. 2023. Interdisciplinary Teaching Toward the Next Generation Hypertext Researchers. In Proceedings of the 34th ACM Conference on Hypertext and Social Media (Rome, Italy) (HT ’23). Association for Computing Machinery, New York, NY, USA, Article 43, 5 pages. https://doi.org/10.1145/3603163.3609055
[3]
Nic Beech, Robert MacIntosh, and Donald MacLean. 2010. Dialogues between academics and practitioners: The role of generative dialogic encounters. Organization Studies 31, 9-10 (2010), 1341–1367.
[4]
Mathias Benedek, Yoed N. Kenett, Konstantin Umdasch, David Anaki, Miriam Faust, and Aljoscha C Neubauer. 2017. How semantic memory structure and intelligence contribute to creative thought: a network science approach. Thinking & Reasoning 23, 2 (2017), 158–183.
[5]
Michael Mose Biskjaer, Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, and Peter Dalsgaard. 2019. How Time Constraints in a Creativity Support Tool Affect the Creative Writing Experience. In Proceedings of the 31st European Conference on Cognitive Ergonomics (BELFAST, United Kingdom) (ECCE ’19). Association for Computing Machinery, New York, NY, USA, 100–107. https://doi.org/10.1145/3335082.3335084
[6]
Margaret A Boden. 1994. What Is Creativity? (1994), 75–117.
[7]
Jay David Bolter and Michael Joyce. 1987. Hypertext and creative writing. In Proceedings of the ACM Conference on Hypertext (Chapel Hill, North Carolina, USA) (HYPERTEXT ’87). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/317426.317431
[8]
Ronald Carter. 2004. Language and creativity: The art of common talk. Routledge.
[9]
Nichol Castro and Cynthia S. Q. Siew. 2020. Contributions of modern network science to the cognitive sciences: Revisiting research spirals of representation and process. Proceedings of the Royal Society A 476, 2238 (2020), 20190825.
[10]
Honghua Chen and Nai Ding. 2023. Probing the “Creativity” of Large Language Models: Can models produce divergent semantic association?. In Findings of the Association for Computational Linguistics: EMNLP 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 12881–12888. https://doi.org/10.18653/v1/2023.findings-emnlp.858
[11]
Yueying Chu and Peng Liu. 2023. Public aversion against ChatGPT in creative fields?The Innovation 4, 4 (2023), 100449.
[12]
David Cropley. 2023. Is artificial intelligence more creative than humans?: ChatGPT and the divergent association task. Learning Letters 2 (2023), 13–13.
[13]
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Morgan & Claypool Publishers, San Rafael.
[14]
Joshua Davis, Liesbet Van Bulck, Brigitte N Durieux, and Charlotta Lindvall. 2024. The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research. JMIR Hum Factors 11 (8 Mar 2024), e53559. https://doi.org/10.2196/53559
[15]
María-Isabel de Vicente-Yagüe-Jara, Olivia López-Martínez, Verónica Navarro-Navarro, and Francisco Cuéllar-Santiago. 2023. Writing, Creativity, and Artificial Intelligence: ChatGPT in the University Context. Comunicar: Media Education Research Journal 31, 77 (2023), 45–54.
[16]
Giorgio Franceschelli and Mirco Musolesi. 2023. On the creativity of large language models. arXiv preprint arXiv:2304.00008 (2023).
[17]
Guillaume Fürst and François Grin. 2018. Multilingualism and creativity: A multivariate approach. Journal of Multilingual and Multicultural Development 39, 4 (2018), 341–355.
[18]
Hans-Georg Gadamer. 1990. Wahrheit und Methode: Grundzüge einer philosophischen Hermeneutik. Mohr Siebeck, Tübingen.
[19]
Harold Garfinkel. 1967. Studies in Ethnomethodology. Prentice Hall, Malden, MA.
[20]
Ken Gilhooly. 2024. AI vs humans in the AUT: Simulations to LLMs. Journal of Creativity 34, 1 (2024), 100071. https://doi.org/10.1016/j.yjoc.2023.100071
[21]
Toni Giorgino. 2009. Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package. Journal of Statistical Software 31, 7 (2009). https://doi.org/10.18637/jss.v031.i07
[22]
J Pv Guilford. 1964. Some new looks at the nature of creative processes. In Contributions to mathematical psychology, N. Fredrickson and H. Gilliksen (Eds.). Holt, Rinehart & Winston, New York.
[23]
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented Transformer for Speech Recognition. arxiv:2005.08100 [eess.AS]
[24]
Erik E. Guzik, Christian Byrge, and Christian Gilde. 2023. The originality of machines: AI takes the Torrance Test. Journal of Creativity 33, 3 (2023), 100065. https://doi.org/10.1016/j.yjoc.2023.100065
[25]
Jennifer Haase and Paul H.P. Hanel. 2023. Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity. Journal of Creativity 33, 3 (2023), 100066. https://doi.org/10.1016/j.yjoc.2023.100066
[26]
Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, Evelina Bakhturina, Vahid Noroozi, Sandeep Subramanian, Koluguri Nithin, Huang Jocelyn, Fei Jia, Jagadeesh Balam, Xuesong Yang, Micha Livne, Yi Dong, Sean Naren, and Boris Ginsburg. [n. d.]. NeMo: a toolkit for Conversational AI and Large Language Models. https://github.com/NVIDIA/NeMo
[27]
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. In The Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=sE7-XhLxHA
[28]
Leah Henrickson and Albert Meroño-Peñuela. 2023. Prompting meaning: a hermeneutic approach to optimising prompt engineering with ChatGPT. AI & SOCIETY (2023), 1–16.
[29]
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python. (2020). https://doi.org/10.5281/zenodo.1212303
[30]
Daphne Ippolito, Ann Yuan, Andy Coenen, and Sehmon Burnam. 2022. Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers. arxiv:2211.05030 [cs.HC]
[31]
Fei Jia, Somshubra Majumdar, and Boris Ginsburg. 2021. MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 6818–6822. https://doi.org/10.1109/ICASSP39728.2021.9414470
[32]
Rodney H Jones. 2010. Creativity and discourse. World Englishes 29, 4 (2010), 467–480.
[33]
Yoed N. Kenett, David Anaki, and Miriam Faust. 2014. Investigating the structure of semantic networks in low and high creative persons. Frontiers in human neuroscience 8 (2014), 407.
[34]
Yoed N. Kenett, Roger E Beaty, Paul J Silvia, David Anaki, and Miriam Faust. 2016. Structure and flexibility: Investigating the relation between the structure of the mental lexicon, fluid intelligence, and creative achievement. Psychology of Aesthetics, Creativity, and the Arts 10, 4 (2016), 377.
[35]
Andruid Kerne and Eunyee Koh. 2007. Creativity support: the mixed-initiative composition space. In Proceedings of the 7th ACM/IEEE-CS Joint Conference on Digital Libraries (Vancouver, BC, Canada) (JCDL ’07). Association for Computing Machinery, New York, NY, USA, 509. https://doi.org/10.1145/1255175.1255309
[36]
Walter Kintsch. 1998. Comprehension. A Paradigm for Cognition. Cambridge University Press, Cambridge.
[37]
Keith Kirkpatrick. 2023. Can AI demonstrate creativity?Commun. ACM 66, 2 (2023), 21–23.
[38]
Moritz Laurer, Wouter van Atteveldt, Andreu Salleras Casas, and Kasper Welbers. 2022. Less Annotating, More Classifying – Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT - NLI. Preprint (June 2022). https://osf.io/74b8k Publisher: Open Science Framework.
[39]
Min Liu. 1998. The effect of hypermedia authoring on elementary school students’ creative thinking. Journal of Educational Computing Research 19, 1 (1998), 27–51.
[40]
Jérôme Louradour. 2023. whisper-timestamped. https://github.com/linto-ai/whisper-timestamped.
[41]
Janet Maybin and Joan Swann. 2007. Everyday creativity in language: Textuality, contextuality, and critique. Applied linguistics 28, 4 (2007), 497–517.
[42]
Sarnoff Mednick. 1962. The associative basis of the creative process. Psychological review 69, 3 (1962), 220.
[43]
Reijo Miettinen. 2013. Creative encounters and collaborative agency in science, technology and innovation. In Handbook of research on creativity. Edward Elgar Publishing, 435–449.
[44]
Carey K Morewedge. 2022. Preference for human, not algorithm aversion. Trends in cognitive sciences 26, 10 (2022), 824–826.
[45]
Ulrich Oevermann. 1986. Kontroversen über sinnverstehende Soziologie. Einige wiederkehrende Probleme und Mißverständnisse in der Rezeption der “objektiven Hermeneutik”. In Handlung und Sinnstruktur: Bedeutung und Anwendung der objektiven Hermeneutik, S. Aufenanger and M. Lenssen (Eds.). Kindt München, 19–83.
[46]
Ulrich Oevermann, With Tilman Allert, Elisabeth Konau, and Jürgen Krambeck. 1987. 21. Structures of Meaning and Objective Hermeneutics. In Modern German Sociology, Volker Meja, Dieter Misgeld, and Nico Stehr (Eds.). Columbia University Press, New York Chichester, West Sussex, 436–448. https://doi.org/
[47]
Jay A. Olson, Johnny Nahas, Denis Chmoulevitch, Simon J. Cropper, and Margaret E. Webb. 2021. Naming unrelated words predicts creativity. Proceedings of the National Academy of Sciences 118, 25 (2021), e2022340118. https://doi.org/10.1073/pnas.2022340118 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2022340118
[48]
OpenAI. 2023. GPT-4 Technical Report. arxiv:2303.08774 [cs.CL]
[49]
OpenAI. 2024. Best practices for prompt engineering with the OpenAI API. https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-the-openai-api. [Accessed April 12, 2024].
[50]
OpenAI. 2024. OpenAI API. https://platform.openai.com/docs/api-reference/completions. Accessed: April 14, 2024.
[51]
Peter Organisciak, Selcuk Acar, Denis Dumas, and Kelly Berthiaume. 2023. Beyond semantic distance: Automated scoring of divergent thinking greatly improves with large language models. Thinking Skills and Creativity 49 (2023), 101356. https://doi.org/10.1016/j.tsc.2023.101356
[52]
Howard H. Pattee. 1989. Simulations, Realizations, and Theories of Life. In Artificial Life. SFI Studies in the Sciences of Complexity, Christopher G. Langton (Ed.). Addison-Wesley, Redwood, 63–77.
[53]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825–2830.
[54]
Martin J. Pickering and Simon Garrod. 2004. Toward a mechanistic psychology of dialogue. Behavioral and Brain Sciences 27 (2004), 169–226.
[55]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In Proceedings of the 40th International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 28492–28518. https://proceedings.mlr.press/v202/radford23a.html
[56]
Ralph Raiola. 2023. ChatGPT, Can You Tell Me a Story? An Exercise in Challenging the True Creativity of Generative AI. Commun. ACM 66, 5 (2023), 104–ff.
[57]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. https://arxiv.org/abs/1908.10084
[58]
Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974. A Simplest Systematics for the Organization of Turn Taking for Conversation. Language 50 (1974), 696–735.
[59]
Keith Sawyer. 2012. Explaining Creativity: The Science of Human Innovation. Oxford University Press, Oxford.
[60]
Keith Sawyer. 2016. Creativity and dialogue: The improvisational nature of conversational interaction. In The Routledge handbook of language and creativity. Routledge, 78–91.
[61]
R Keith Sawyer and Stacy DeZutter. 2009. Distributed creativity: How collective creations emerge from collaboration.Psychology of aesthetics, creativity, and the arts 3, 2 (2009), 81.
[62]
Verena D. Schmittmann, Angélique O.J. Cramer, Lourens J. Waldorp, Sacha Epskamp, Rogier A. Kievit, and Denny Borsboom. 2013. Deconstructing the construct: A network perspective on psychological phenomena. New Ideas in Psychology 31, 1 (2013), 43–53. https://doi.org/10.1016/j.newideapsych.2011.02.007
[63]
Dean Keith Simonton. 2013. What is a creative idea? Little-c versus Big-C creativity. In Handbook of research on creativity. Edward Elgar Publishing, 69–83.
[64]
Douglas Summers-Stay, Clare R Voss, and Stephanie M Lukin. 2023. Brainstorm, then select: a generative language model improves its creativity score. In The AAAI-23 Workshop on Creative AI Across Modalities.
[65]
Viriya Taecharungroj. 2023. “What can ChatGPT do?” Analyzing early reactions to the innovative AI chatbot on Twitter. Big Data and Cognitive Computing 7, 1 (2023), 35.
[66]
Ellis Paul Torrance. 2017. Torrance tests of creative thinking.
[67]
Jared Vasil, Paul B Badcock, Axel Constant, Karl Friston, and Maxwell JD Ramstead. 2020. A world unto itself: human communication as active inference. Frontiers in psychology 11 (2020), 480375.
[68]
Andreas Wernet. 2006. Einführung in die Interpretationstechnik der objektiven Hermeneutik. Springer.
[69]
Ching-Lin Wu, Shih-Yuan Huang, Pei-Zhen Chen, and Hsueh-Chih Chen. 2020. A Systematic Review of Creativity-Related Studies Applying the Remote Associates Test From 2000 to 2019. Frontiers in Psychology 11 (2020).
[70]
Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, Qi Guo, Ling Li, and Yunji Chen. 2024. Assessing and Understanding Creativity in Large Language Models. arxiv:2401.12491 [cs.CL]
[71]
Rolf A. Zwaan. 2016. Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic bulletin & review 23, 4 (2016), 1028–1034.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
HT '24: Proceedings of the 35th ACM Conference on Hypertext and Social Media
September 2024
415 pages
ISBN:9798400705953
DOI:10.1145/3648188
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 September 2024

Check for updates

Author Tags

  1. Creative AI
  2. Creativity
  3. Generative AI
  4. Hermeneutics
  5. NLP

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

Conference

HT '24
Sponsor:

Acceptance Rates

Overall Acceptance Rate 378 of 1,158 submissions, 33%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 179
    Total Downloads
  • Downloads (Last 12 months)179
  • Downloads (Last 6 weeks)31
Reflects downloads up to 10 Feb 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media