Abstract
Collaborative learning offers numerous benefits to learners, largely due to the dialogue that is unfolding between them. However, there is still much to learn about the structure of collaborative dialogue, and especially little is known about co-creative dialogues during learning. This paper reports on a study with learners engaged in co-creative tasks where the learners wrote code to create a song and while engaging in textual dialogue as they did so. After gathering the textual dialogue and the actions within the interface, we learned a hidden Markov model (HMM) to reveal co-creative states. The seven-state model revealed four states primarily composed of coding actions that included browsing the curriculum documents, working in the code editor, compiling the code successfully, and receiving a compile error. The remaining three states are primarily composed of dialogue that can be characterized as social, aesthetic, and technical dialogue. Next, we analyzed the relationships between the co-creative states revealed by the HMM and students’ partner satisfaction scores from a post-survey. The results reveal the relative frequency of actions in certain states and some transitions between states were predictive of partner satisfaction. For example, partner satisfaction was negatively associated with the Compilation Error state and with the relative frequency of transitions from the Curriculum Browsing state to the Code Editing state. Partner satisfaction was also negatively associated with the relative frequency of transitions from the Aesthetic Dialogue state to the Technical Dialogue state and the Code Editing state. This line of investigation reveals how co-creative processes are associated with partner satisfaction, and holds the potential to inform scaffolding for collaborative learning.
Similar content being viewed by others
References
Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705.
Arroyo, I., Wixon, N., Allessio, D., Woolf, B., Muldner, K., & Burleson, W. (2017). Collaboration improves student interest in online tutoring. Artificial intelligence in education, pp. 28–39. https://doi.org/10.1007/978-3-319-61425-0_3.
Bales, R.F., & Strodtbeck, F.L. (1951). Phases in group problemsolving. The Journal of Abnormal and Social Psychology, 46(4), 485. https://doi.org/10.1037/h0059886.
Berlyne, D.E. (1978). Curiosity and learning. Motivation and emotion, 2(2), 97–175. https://doi.org/10.1007/BF00993037.
Boyer, K.E., Ha, E.Y., Wallis, M.D., Phillips, R., Vouk, M.A., & Lester, J.C. (2009). Discovering tutorial dialogue strategies with hidden markov models. https://doi.org/10.3233/978-1-60750-028-5-141.
Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M.D., Vouk, M.A., & Lester, J.C. (2011). Investigating the relationship between dialogue structure and tutoring effectiveness: a hidden markov modeling approach. International Journal of Artificial Intelligence in Education, 21(1), 65–81. https://doi.org/10.3233/JAI-2011-018.
Braught, G., Wahls, T., & Eby, L.M. (2011). The case for pair programming in the computer science classroom. ACM Transactions on Computing Education (TOCE), 11(1), 1–21. https://doi.org/10.1145/1921607.1921609.
Campe, S., Denner, J., Green, E., & Torres, D. (2020). Pair programming in middle school: variations in interactions and behaviors. Computer Science Education, 30(1), 22–46. https://doi.org/10.1080/08993408.2019.1648119.
Carpenter, D., Emerson, A., Mott, B.W., Saleh, A., Glazewski, K.D., Hmelo-Silver, C.E., & Lester, J.C. (2020). Detecting off-task behavior from student dialogue in game-based collaborative learning. Artificial intelligence in education, pp. 55–66. https://doi.org/10.1007/978-3-030-52237-7_5.
Chaparro, E.A., Yuksel, A., Romero, P., & Bryant, S. (2005). Factors affecting the perceived effectiveness of pair programming in higher education. In Proceedings of the 17th workshop of the psychology of programming interest group (pp. 5–18).
Chng, E., Seyam, M.R., Yao, W., & Schneider, B. (2020). Using motion sensors to understand collaborative interactions in digital fabrication labs. Artificial intelligence in education, pp. 118–128. https://doi.org/10.1007/978-3-030-52237-7_10.
Davidson, N., & Major, C.H. (2014). Boundary crossings: Cooperative learning, collaborative learning, and problem-based learning. Journal on Excellence in College Teaching, 25(3/4), 7–55.
Denner, J., Werner, L., Campe, S., & Ortiz, E. (2014). Pair programming: Under what conditions is it advantageous for middle school students? Journal of Research on Technology in Education, 46(3), 277–296. https://doi.org/10.1080/15391523.2014.888272.
Dich, Y., Reilly, J., & Schneider, B. (2018). Using physiological synchrony as an indicator of collaboration quality, task performance and learning. Artificial intelligence in education, pp. 98–110. https://doi.org/10.1007/978-3-319-93843-1_8.
Dillenbourg, P. (1999). What do you mean by’collaborative learning’?. In P. Dillenbourg (Ed.) Collaborative learning: Cognitive and computational approaches. Oxford: Elsevier.
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22 (2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001.
Dyke, G., Adamson, D., Howley, I., & Rose, C.P. (2013). Enhancing scientific reasoning and discussion with conversational agents. IEEE Transactions on Learning Technologies, 6(3), 240–247. https://doi.org/10.1109/TLT.2013.25.
Ferschke, O., Yang, D., Tomar, G., & Rosé, C.P. (2015). Positive impact of collaborative chat participation in an edx mooc. In C. Conati, N. Heffernan, A. Mitrovic, & M.F. Verdejo (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-319-19773-912 (pp. 115–124).
Fincher, S.A., & Robins, A.V. (2019). The cambridge handbook of computing education research. Cambridge University Press.
Freeman, J., Magerko, B., McKlin, T., Reilly, M., Permar, J., Summers, C., & Fruchter, E. (2014). Engaging underrepresented groups in high school introductory computing through computational remixing with earsketch. pp. 85–90. https://doi.org/10.1145/2538862.2538906.
Freeman, J., Magerko, B., & Verdin, R. (2015). Earsketch: a web-based environment for teaching introductory computer science through music remixing. In The 46th acm technical symposium on computer science education. https://doi.org/10.1145/2676723.2691869 (p. 5).
Fuller, D., & Magerko, B. (2010). Shared mental models in improvisational performance. In Proceedings of the intelligent narrative technologies iii workshop. https://doi.org/10.1145/1822309.1822324. New York, NY, USA: Association for Computing Machinery.
Fuller, D., & Magerko, B. (2011). Shared mental models in improvisational theatre. In Proceedings of the 8th acm conference on creativity and cognition. https://doi.org/10.1145/2069618.2069663 (pp. 269–278). New York, NY, USA: Association for Computing Machinery.
Glăveanu, V.-P. (2011). How are we creative together? comparing sociocognitive and sociocultural answers. Theory & Psychology, 21(4), 473–492. https://doi.org/10.1177/0959354310372152.
Gokhale, A.A. (1995). Collaborative learning enhances critical thinking. Journal of Technology Education, 7(1), 22–30. https://doi.org/10.21061/jte.v7i1.a.2.
Goodman, B.A., Linton, F.N., Gaimari, R.D., Hitzeman, J.M., Ross, H.J., & Zarrella, G. (2005). Using dialogue features to predict trouble during collaborative learning. User Modeling and User-Adapted Interaction, 15(1), 85–134. https://doi.org/10.1007/s11257-004-5269-x.
Gorson, J., LaGrassa, N., Hu, C.H., Lee, E., Robinson, A.M., & O’Rourke, E. (2021). An approach for detecting student perceptions of the programming experience from interaction log data. In I. Roll, D. McNamara, S. Sosnovsky, R. Luckin, & V. Dimitrova (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-030-78292-4_13 (pp. 150–164).
Graesser, A.C., Fiore, S.M., Greiff, S., Andrews-Todd, J., Foltz, P.W., & Hesse, F.W. (2018). Advancing the science of collaborative problem solving. Psychological Science in the Public Interest, 19(2), 59–92. https://doi.org/10.1177/1529100618808244.
Grover, S. (2020). Computer science in k-12: an az handbook on teaching programming (S. Grover, Ed.). Efinity.
Howard, C., Jordan, P., Di Eugenio, B., & Katz, S. (2017). Shifting the load: a peer dialogue agent that encourages its human collaborator to contribute more to problem solving. International Journal of Artificial Intelligence in Education, 27(1), 101–129. https://doi.org/10.1007/s40593-015-0071-y.
K–12 Computer Science Framework. (2016). K-12 computer science framework. Retrieved from https://k12cs.org/.
Kantosalo, A., Toivanen, J., Xiao, P., & Toivonen, H. (2014). From isolation to involvement: Adapting machine creativity software to support human-computer co-creation. The fifth international conference on computational creativity, 2014, 1–7.
Katuka, G.A., Bex, R.T., Celepkolu, M., Boyer, K.E., Wiebe, E., Mott, B., & Lester, J. (2021). My partner was a good partner: Investigating the relationship between dialogue acts and satisfaction among middle school computer science learners. In Proceedings of the 14th international conference on computer-supported collaborative learning-cscl 2021.
Katuka, G.A., Webber, A.R., Wiggins, J.B., Boyer, K.E., Magerko, B., McKlin, T., & Freeman, J. (2022). The relationship between co-creative dialogue and high school learners’ satisfaction with their collaborator in computational music remixing. Proc. ACM Hum.-Comput. Interact., 6(CSCW1). https://doi.org/10.1145/3512970.
Kinnunen, P., & Simon, B. (2010). Experiencing programming assignments in cs1: The emotional toll. In Proceedings of the sixth international workshop on computing education research. https://doi.org/10.1145/1839594.1839609 (pp. 77–86).
Kirschner, F., Paas, F., & Kirschner, P.A. (2011). Task complexity as a driver for collaborative learning efficiency: The collective working-memory effect. Applied Cognitive Psychology, 25(4), 615–624. https://doi.org/10.1002/acp.1730.
Kirschner, P.A., Sweller, J., Kirschner, F., & Zambrano, J. (2018). From cognitive load theory to collaborative cognitive load theory. International Journal of Computer-Supported Collaborative Learning, 13(2), 213–233. https://doi.org/10.1007/s11412-018-9277-y.
Knobelsdorf, M., & Romeike, R. (2008). Creativity as a pathway to computer science. SIGCSE Bull., 40(3), 286–290. https://doi.org/10.1145/1597849.1384347.
Lahtinen, E., Ala-Mutka, K., & Järvinen, H.-M. (2005). A study of the difficulties of novice programmers. In Proceedings of the 10th annual sigcse conference on innovation and technology in computer science education (pp. 14–18).
Landis, J.R., & Koch, G.G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310.
Lin, Y., Guo, J., Chen, Y., Yao, C., & Ying, F. (2020). It is your turn: Collaborative ideation with a co-creative robot through sketch. In Proceedings of the 2020 chi conference on human factors in computing systems. https://doi.org/10.1145/3313831.3376258 (pp. 1–14).
Magerko, B., Freeman, J., Mcklin, T., Reilly, M., Livingston, E., Mccoid, S., & Crews-Brown, A. (2016). Earsketch: A steam-based approach for underrepresented populations in high school computer science education. ACM Trans. Computers in Education, 16(4). https://doi.org/10.1145/2886418.
Matsumae, A., Raharja, F.T., Ehkirch, Q., & Nagai, Y. (2021). How the cocreative process affects concept formation. Proceedings of the Design Society, 1, 1775–1786. https://doi.org/10.1017/pds.2021.439.
McDowell, C., Werner, L., Bullock, H., & Fernald, J. (2002). The effects of pairprogramming on performance in an introductory programming course. In Proceedings of the 33rd sigcse technical symposium on computer science education. https://doi.org/10.1145/563340.563353 (pp. 38–42).
McLaren, B.M., & Isotani, S. (2011). When is it best to learn with all worked examples?. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-642-21869-9_30(pp. 222–229).
Moore, R., Helms, M., & Freeman, J. (2017). Steam-based interventions in computer science: Understanding feedback loops in the classroom. 2017 asee annual conference & exposition. https://doi.org/10.18260/1-2--28842.
Morales-Urrutia, E.K., Ocaña Ch., J.M., Pérez-Marín, D., & Pizarro-Romero, C. (2020). Promoting learning and satisfaction of children when interacting with an emotional companion to program. In I.I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.) Artificial intelligence in ed- ucation. https://doi.org/10.1007/978-3-030-52240-7_40 (pp. 220–223).
Murphy, L., Fitzgerald, S., Hanks, B., & McCauley, R. (2010). Pair debugging: a transactive discourse analysis. In Proceedings of the sixth international workshop on computing education research. https://doi.org/10.1145/1839594.1839604(pp. 51–58).
Neath, A.A., & Cavanaugh, J.E. (2012). The bayesian information criterion: background, derivation, and applications. Wiley Interdisciplinary Reviews: Computational Statistics, 4(2), 199–203.
Ogan, A., Finkelstein, S., Walker, E., Carlson, R., & Cassell, J. (2012). Rudeness and rapport: Insults and learning gains in peer tutoring. In S.A. Cerri, W.J. Clancey, G. Papadourakis, & K. Panourgia (Eds.) Intelligent tutoring systems. https://doi.org/10.1007/978-3-642-30950-2_2 (pp. 11–21).
Rabiner, L., & Juang, B. (1986). An introduction to hidden markov models. IEEE ASSP Magazine, 3(1), 4–16. https://doi.org/10.1109/MASSP.1986.1165342.
Radu, I., Tu, E., & Schneider, B. (2020). Relationships between body postures and collaborative learning states in an augmented reality study. In I.I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-030-52240-7_47 (pp. 257–262).
Rodríguez, F.J., & Boyer, K.E. (2015). Discovering individual and collaborative problem-solving modes with hidden markov models. In C. Conati, N. Heffernan, A. Mitrovic, & M.F. Verdejo (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-319-19773-9_41 (pp. 408–418).
Rodríguez, F.J., Price, K.M., & Boyer, K.E. (2017). Exploring the pair programming process: Characteristics of effective collaboration. In Proceedings of the 2017 acm sigcse technical symposium on computer science education. https://doi.org/10.1145/3017680.3017748 (pp. 507–512).
Rodríguez, F.J., Price, K.M., & Boyer, K.E. (2017). Expressing and addressing uncertainty: A study of collaborative problem-solving dialogues. Proceedings of the 12th international conference on computer supported collaborative learning (cscl). Philadelphia, PA: International Society of the Learning Sciences.
Roschelle, J. (1992). Learning by collaborating: Convergent conceptual change. Journal of the Learning Sciences, 2(3), 235–276. https://doi.org/10.1207/s15327809jls0203_1.
Roschelle, J., & Teasley, S.D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.) Computer supported collaborative learning. https://doi.org/10.1007/978-3-642-85098-1_5 (pp. 69–97).
Rosen, Y. (2015). Computer-based assessment of collaborative problem solving: Exploring the feasibility of human-to-agent approach. International Journal of Artificial Intelligence in Education, 25(3), 380–406. https://doi.org/10.1007/s40593-015-0042-3.
Samoilescu, R.-F., Dascalu, M., Sirbu, M.-D., Trausan-Matu, S., & Crossley, S.A. (2019). Modeling collaboration in online conversations using time series analysis and dialogism. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-030-23204-7_38 (pp. 458–468).
Sankaranarayanan, S., Kandimalla, S.R., Hasan, S., An, H., Bogart, C., Murray, R.C., & Rosé, C. (2020). Agent-in-the-loop: Conversational agent support in service of reflection for learning during collaborative programming. In I.I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-030-52240-750(pp. 273–278).
SAS Institute Inc. (2020). Jmp®15 fitting linear models [Computer software manual] Cary NC, SAS Institute Inc.
SAS Institute Inc. (2021). JMP®Pro 15. Retrieved from https://www.jmp.com/enus/software/predictive-analyticssoftware.html.
Schneider, B., & Pea, R. (2014). Toward collaboration sensing. International Journal of Computer-Supported Collaborative Learning, 9 (4), 371–395. https://doi.org/10.1007/s11412-014-9202-y.
Snyder, C., Hutchins, N.M., Biswas, G., Emara, M., Yett, B., & Mishra, S. (2020). Understanding collaborative question posing during computational modeling in science. In I.I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.) Artificial intelligence in education. https://doi.org/10.1007/978-3-030-52240-7_54 (pp. 296–300).
Soller, A. (2001). Supporting social interaction in an intelligent collaborative learning system. International Journal of Artificial Intelligence in Education, 12, 40–62.
Soller, A., & Lesgold, A. (2007). Modeling the process of collaborative learning. In H.U. Hoppe, H. Ogata, & A. Soller (Eds.) The role of technology in cscl: Studies in technology enhanced collaborative learning. https://doi.org/10.1007/978-0-387-71136-2_5 (pp. 63–86).
Swain, M. (2000). The output hypothesis and beyond: Mediating acquisition through collaborative dialogue. In Sociocultural theory and second language learning, (Vol. 78 pp. 97–114). Oxford University Press.
Teasley, S.D. (1997). Talking about reasoning: How important is the peer in peer collaboration?. In L.B. Resnick, R. Säljö, C. Pontecorvo, & B. Burge (Eds.) Discourse, tools and reasoning: Essays on situated cognition. https://doi.org/10.1007/978-3-662-03362-3_16 (pp. 361–384).
Tegos, S., Demetriadis, S., & Tsiatsos, T. (2014). A configurable conversational agent to trigger students’ productive dialogue: a pilot study in the call domain. International Journal of Artificial Intelligence in Education, 24(1), 62–91. https://doi.org/10.1007/s40593-013-0007-3.
Truesdell, E., Smith, J., Mathew, S., McKlin, T., Katuka, G.A., Griffith, A.E., & Boyer, K.E. (2021). Supporting computational music remixing with a co-creative learning companion. In Proceedings of the 12th international conference on computational creativity (pp. 113–121).
Tsan, J., Lynch, C.F., & Boyer, K.E. (2018). Alright, what do we need?: A study of young coders’ collaborative dialogue. International Journal of Child-Computer Interaction, 17, 61–71. https://doi.org/10.1016/j.ijcci.2018.03.001.
Tsan, J., Rodríguez, F.J., Boyer, K.E., & Lynch, C. (2018). i think we should...: Analyzing elementary students’ collaborative processes for giving and taking suggestions. https://doi.org/10.1145/3159450.3159507.
Tschan, F. (1995). Communication enhances small group performance if it conforms to task requirements: The concept of ideal communication cycles. Basic and Applied Social Psychology, 17 (3), 371–393. https://doi.org/10.1207/s15324834basp1703_6.
Waddock, S.A., & Bannister, B.D. (1991). Correlates of effectiveness and partner satisfaction in social partnerships. Journal of Organizational Change Management, 4(2), 64–79. (Publisher: MCB UP Ltd) https://doi.org/10.1108/EUM0000000001192.
Zhang, J., Scardamalia, M., Lamon, M., Messina, R., & Reeve, R. (2007). Sociocognitive dynamics of knowledge building in the work of 9- and 10-yearolds. Educational Technology Research and Development, 55(2), 117–145. https://doi.org/10.1007/s11423-006-9019-0.
Zhi, R., Price, T.W., Marwan, S., Milliken, A., Barnes, T., & Chi, M. (2019). Exploring the impact of worked examples in a novice programming environment. In: Proceedings of the 50th acm technical symposium on computer science education, pp. 98–104. https://doi.org/10.1145/3287324.3287385.
Zhong, B., Wang, Q., & Chen, J. (2016). The impact of social factors on pair programming in a primary school. Computers in Human Behavior, 64, 423–431. https://doi.org/10.1016/j.chb.2016.07.017.
Acknowledgements
This material is based upon work supported by the National Science Foundation under grants DRL-1813740 and DRL-1814083. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
Funding
This research was funded by the National Science Foundation, DRL-1813740 and DRL-1814083.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics approval
This work was approved by the University of Florida’s IRB prior to the conducted research.
Consent to participate
ThisEach student represented in this data assented and their parents consented for them to be part of this work.
Consent for Publication
All authors have approved the manuscript and agree with its submission to the International Journal of Artificial Intelligence in Education.
Conflict of Interests
There were no conflicts of interest to report for this research.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Transparency on Re-use of Material
The authors confirm that this manuscript has not been published elsewhere and is not under consideration by another journal. This manuscript extends a work that has been previously published at Artificial Intelligence in Education 2021 in Lecture Notes in Computer Science, vol 12748 which can be found at https://doi.org/10.1007/978-3-030-78292-4_14. The original work reported on the hidden Markov model states and their transitions. This manuscript includes additional analyses of post-survey partner satisfaction scores and their relationship with the hidden Markov model as well as additional figures and insights about the general flow of a co-creative session.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Griffith, A.E., Katuka, G.A., Wiggins, J.B. et al. Investigating the Relationship Between Dialogue States and Partner Satisfaction During Co-Creative Learning Tasks. Int J Artif Intell Educ 33, 543–582 (2023). https://doi.org/10.1007/s40593-022-00302-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-022-00302-5