Abstract
Recent advances in the usability of generative AI platforms, such as ChatGPT, suggest that artificial intelligence can seemingly capture many of the rules and meanings underlying human language. As a result, AI can potentially automate many tasks requiring human-like understanding and generation of natural language. Qualitative data analysis tends to be a time-consuming task which is susceptible to human bias and mistakes. In this study, we explored the use of ChatGPT and the GPT-4 model to assist with analyzing qualitative data from a previous study by Rannastu-Avalos, Mäeots, and Siiman (2022). In that prior study, pairs of adults collaborated via a free-form, text-based chat interface to solve a computer simulation problem about balancing a seesaw. To re-analyze the data using AI assistance, both deductive and inductive approaches were applied and the results compared to human coding and human interpretation of the data. The results show that it is important to structure and phrase prompts so that AI responses best align with human interpretation. Deductive analysis performed better than inductive analysis, presumably because prompts with richer contextual information referring to specific theoretical perspectives could be crafted. Our results suggest that AI-assisted qualitative analysis has the potential to improve transparency in the coding of qualitative data by encouraging human analysts to report AI prompts that agree with their interpretations of the data, and in turn can be reused by other researchers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Weitzman, E., Miles, M.B.: Computer Programs for Qualitative Data Analysis. Sage, Thousand Oaks (1995)
Rose, J., Johnson, C.W.: Contextualizing reliability and validity in qualitative research: toward more rigorous and trustworthy qualitative social science in leisure research. J. Leisure Res. 51(4), 432–451 (2020)
Condor, A., Pardos, Z., Linn, M.: Representing scoring rubrics as graphs for automatic short answer grading. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds.) Artificial Intelligence in Education. LNCS, vol. 13355, pp. 354–365. Springer, Cham (2022)
Twining, P., Heller, R.S., Nussbaum, M., Tsai, C.C.: Some guidance on conducting and reporting qualitative studies. Comput. Educ. 106, A1–A9 (2017)
Braun, V., Clarke, V.: Using thematic analysis in psychology. Qual. Res. Psychol. 3(2), 77–101 (2006)
Scoular, C., Care, E., Hesse, F.W.: Designs for operationalizing collaborative problem solving for automated assessment. J. Educ. Measur. 54(1), 12–35 (2017)
Hesse, F., Care, E., Buder, J., Sassenberg, K., Griffin, P.: A framework for teachable collaborative problem solving skills. In: Griffin, P., Care, E. (eds.) Assessment and teaching of 21st century skills. EAIA, pp. 37–56. Springer, Dordrecht (2015). https://doi.org/10.1007/978-94-017-9395-7_2
Care, E., Kim, H.: Assessment of twenty-first century skills: the issue of authenticity. In: Care, E., Griffin, P., Wilson, M. (eds.) Assessment and teaching of 21st century skills. EAIA, pp. 21–39. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-65368-6_2
Care, E., Scoular, C., Griffin, P.: Assessment of collaborative problem solving in education environments. Appl. Measur. Educ. 29(4), 250–264 (2016)
Rannastu, M., Siiman, L.A., Mäeots, M., Pedaste, M., Leijen, Ä.: Does group size affect students’ inquiry and collaboration in using computer-based asymmetric collaborative simulations? In: Herzog, M., Kubincová, Z., Han, P., Temperini, M. (eds.) Advances in Web-Based Learning, LNCS, vol. 11841, pp. 143–154. Springer, Cham. (2019)
Siiman, L.A., Rannastu-Avalos, M., Mäeots, M.: Developing smart device friendly asymmetric simulations for teaching collaborative scientific inquiry. In: 2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT), pp. 130–131. IEEE (2020)
Rannastu-Avalos, M., Mäeots, M., Siiman, L.A.: How teacher education students collaborate when solving an asymmetric digital task. In: Wong, L.H., Hayashi, Y., Collazos, C.A., Alvarez, C., Zurita, G., Baloian, N. (eds.) Collaboration Technologies and Social Computing, LNCS, vol. 13632, pp. 158–174. Springer, Cham. (2022)
Abram, M.D., Mancini, K.T., Parker, R.D.: Methods to integrate natural language processing into qualitative research. Int. J. Qual. Methods 19 (2020)
Alzubaidi, L., et al.: Review of deep learning: concepts, CNN architectures, challenges, applications, future directions. J. Big Data 8, 53 (2021). https://doi.org/10.1186/s40537-021-00444-8
Rosé, C., et al.: Analyzing collaborative learning processes automatically: exploiting the advances of computational linguistics in computer-supported collaborative learning. Int. J. Comput.-Support. Collab. Learn. 3, 237–271 (2008). https://doi.org/10.1007/s11412-007-9034-0
De Araujo, A., Papadopoulos, P.M., McKenney, S., de Jong, T.: Automated coding of student chats, a trans-topic and language approach. Comput. Educ. Artif. Intell. 4, 100123 (2023)
Acknowledgements
Author L.A.S. is grateful to the JYU Visiting Fellow Programme at the University of Jyväskylä for partially supporting this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Siiman, L.A., Rannastu-Avalos, M., Pöysä-Tarhonen, J., Häkkinen, P., Pedaste, M. (2023). Opportunities and Challenges for AI-Assisted Qualitative Data Analysis: An Example from Collaborative Problem-Solving Discourse Data. In: Huang, YM., Rocha, T. (eds) Innovative Technologies and Learning. ICITL 2023. Lecture Notes in Computer Science, vol 14099. Springer, Cham. https://doi.org/10.1007/978-3-031-40113-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-40113-8_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40112-1
Online ISBN: 978-3-031-40113-8
eBook Packages: Computer ScienceComputer Science (R0)