Skip to main content
SpringerLink
Log in

Measuring and Analyzing Students’ Strategic Learning Behaviors in Open-Ended Learning Environments

  • ARTICLE
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

Strategies are an important component of self-regulated learning frameworks. However, the characterization of strategies in these frameworks is often incomplete: (1) they lack an operational definition of strategies; (2) there is limited understanding of how students develop and apply strategies; and (3) there is a dearth of systematic and generalizable approaches to measure and evaluate strategies when students’ work in open-ended learning environments (OELEs). This paper develops systematic methods for detecting, interpreting, and analyzing students’ use of strategies in OELEs, and demonstrates how students’ strategies evolve across tasks. We apply this framework in the context of tasks that students perform as they learn science topics by building conceptual and computational models in an OELE. Data from a classroom study, where sixth-grade students (N = 52) worked on science model-building activities in our Computational Thinking using Simulation and Modeling (CTSiM) environment demonstrates how we interpret students’ strategy use, and how strategy use relates to their learning performance. We also demonstrate how students’ strategies evolve as they work on multiple model-building tasks. The results demonstrate the effectiveness of our strategy framework in analyzing students’ behaviors and performance in CTSiM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

Data and materials created for this research are available upon request. Please direct all inquiries to the corresponding author. Access to the CTSiM software can be required at https://wp0.vanderbilt.edu/oele/software/.

Code Availability

The code created for this research is available upon request. Please direct all inquiries to the corresponding author.

Notes

  1. The students are not able to see the code for the expert model.

  2. These actions are labeled by combining the generic action types (IA, SC, and SA) and the observable data logged in the OELE (see also Fig. 2)

  3. Most of the computed p-values were < 0.0001. Therefore, the correction did not change the results

  4. The number of HG students is slightly larger because four students had the same learning gain as the median and were assigned to HG.

References

  • Agrawal, R., & Srikant, R. (1995). Mining sequential patterns. In Proceedings of the eleventh international conference on data engineering (pp. 3–14). https://doi.org/10.1109/ICDE.1995.380415.

  • Ahmad Uzir, N., Gašević, D., Matcha, W., Jovanović, J., & Pardo, A. (2020). Analytics of time management strategies in a flipped classroom. Journal of Computer Assisted Learning, 36(1), 70–88.

    Article  Google Scholar 

  • Alesandrini, K.L. (1981). Pictorial–verbal and analytic–holistic learning strategies in science learning. Journal of Educational Psychology, 73(3), 358.

    Article  Google Scholar 

  • Alexander, P.A., Graham, S., & Harris, K.R. (1998). A perspective on strategy research: Progress and prospects. Educational Psychology Review, 10 (2), 129–154.

    Article  Google Scholar 

  • Austin, P.C. (2009). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine, 28(25), 3083–3107.

    Article  MathSciNet  Google Scholar 

  • Azevedo, R., Johnson, A., Chauncey, A., & Burkett, C. (2010). Self-regulated learning with metatutor: Advancing the science of learning with metacognitive tools. In New science of learning (pp. 225–247). Springer.

  • Azevedo, R., Harley, J., Trevors, G., Duffy, M., Feyzi-Behnagh, R., Bouchet, F., & Landis, R. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems. In International handbook of metacognition and learning technologies (pp. 427–449). Springer.

  • Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and Learning, 9(2), 161–185.

    Article  Google Scholar 

  • Basu, S., Dukeman, A., Kinnebrew, J.S., Biswas, G., & Sengupta, P. (2014). Investigating student generated computational models of science. In International Conference on Learning Sciences Boulder, CO: International Society of the Learning Sciences.

  • Basu, S., Sengupta, P., & Biswas, G. (2015). A scaffolding framework to support learning of emergent phenomena using multi-agent-based simulation environments. Research in Science Education, 45(2), 293–324.

    Article  Google Scholar 

  • Basu, S., Biswas, G., Sengupta, P., Dickes, A., Kinnebrew, J.S., & Clark, D. (2016). Identifying middle school students’ challenges in computational thinking-based science learning. Research and Practice in Technology Enhanced Learning, 11(1), 13.

    Article  Google Scholar 

  • Basu, S., Biswas, G., & Kinnebrew, J.S. (2017). Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Modeling and User-Adapted Interaction, 27(1), 5–53.

    Article  Google Scholar 

  • Baxter, I.D., Yahin, A., Moura, L., Sant’Anna, M., & Bier, L. (1998). Clone detection using abstract syntax trees. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) (pp. 368–377). IEEE.

  • Beaudoin, L., & Winne, P. (2009). Nstudy: An internet tool to support learning, collaboration and researching learning strategies. In Canadian e-Learning conference.

  • Berland, M., Martin, T., Benton, T., Petrick Smith, C., & Davis, D. (2013). Using learning analytics to understand the learning pathways of novice programmers. Journal of the Learning Sciences, 22(4), 564–599.

    Article  Google Scholar 

  • Bernacki, M.L. (2017). Examining the cyclical, loosely sequenced, and contingent features of self-regulated learning: Trace data and their analysis. In Handbook of self-regulation of learning and performance (pp. 370–387). Routledge.

  • Bernacki, M.L., Byrnes, J.P., & Cromley, J.G. (2012). The effects of achievement goals and self-regulated learning behaviors on reading comprehension in technology-enhanced learning environments. Contemporary Educational Psychology, 37(2), 148–161.

    Article  Google Scholar 

  • Bers, M.U., Flannery, L., Kazakoff, E.R., & Sullivan, A. (2014). Computational thinking and tinkering: Exploration of an early childhood robotics curriculum. Computers & Education, 72, 145–157.

    Article  Google Scholar 

  • Bille, P. (2005). A survey on tree edit distance and related problems. Theoretical Computer Science, 337(1-3), 217–239.

    Article  MathSciNet  MATH  Google Scholar 

  • Biswas, G., Leelawong, K., Schwartz, D., Vye, N., & The teachable agents group at vanderbilt (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19(3-4), 363–392.

    Article  Google Scholar 

  • Biswas, G., Jeong, H., Kinnebrew, J.S., Sulcer, B., & ROSCOE, R. (2010). Measuring self-regulated learning skills through social interactions in a teachable agent environment. Research and Practice in Technology Enhanced Learning, 5(02), 123–152.

    Article  Google Scholar 

  • Biswas, G., Segedy, J.R., & Bunchongchit, K. (2016). From design to implementation to practice a learning by teaching system: Betty’s brain. International Journal of Artificial Intelligence in Education, 26(1), 350–364.

    Article  Google Scholar 

  • Boekaerts, M. (1996). Self-regulated learning at the junction of cognition and motivation. European Psychologist, 1(2), 100.

    Article  Google Scholar 

  • Boyer, K.E., Phillips, R., Ingram, A., Ha, E.Y., Wallis, M., Vouk, M., & Lester, J. (2011). Investigating the relationship between dialogue structure and tutoring effectiveness: a hidden markov modeling approach. International Journal of Artificial Intelligence in Education, 21(1-2), 65–81.

    Google Scholar 

  • Chao, P.Y. (2016). Exploring students’ computational practice, design and performance of problem-solving through a visual programming environment. Computers & Education, 95, 202–215.

    Article  Google Scholar 

  • Cheng, B., Ructtinger, L., Fujii, R., & Mislevy, R. (2010). Assessing systems thinking and complexity in science (large-scale assessment technical report 7). Menlo, Park, CA: SRI International Downloaded September 11:2010.

  • Chi, M.T. (2005). Commonsense conceptions of emergent processes: Why some misconceptions are robust. The Journal of the Learning Sciences, 14 (2), 161–199.

    Article  Google Scholar 

  • Chi, M.T., Roscoe, R.D., Slotta, J.D., Roy, M., & Chase, C.C. (2012). Misconceived causal explanations for emergent processes. Cognitive Science, 36(1), 1–61.

    Article  Google Scholar 

  • Cloude, E.B., Taub, M., & Azevedo, R. (2018). Investigating the role of goal orientation: metacognitive and cognitive strategy use and learning with intelligent tutoring systems. In International conference on intelligent tutoring systems (pp. 44–53). Springer.

  • Cornford, I.R. (2002). Learning-to-learn strategies as a basis for effective lifelong learning. International Journal of Lifelong Education, 21(4), 357–368.

    Article  Google Scholar 

  • Dent, E.B. (1999). Complexity science: a worldview shift. Emergence, 1(4), 5–19.

    Article  Google Scholar 

  • Derry, S.J. (1990). Learning strategies for acquiring useful knowledge. Dimensions of thinking and cognitive instruction. pp. 347–379.

  • Donker, A., de Boer, H., Kostons, D., van Ewijk, C.D., & van der Werf, M. (2014). Effectiveness of learning strategy instruction on academic performance: A meta-analysis, (Vol. 11 pp. 1–26). https://doi.org/10.1016/j.edurev.2013.11.002. http://www.sciencedirect.com/science/article/pii/S1747938X13000420.

  • Fincham, E., Gašević, D., Jovanović, J., & Pardo, A. (2018). From study tactics to learning strategies: an analytical method for extracting interpretable representations. IEEE Transactions on Learning Technologies, 12(1), 59–72.

    Article  Google Scholar 

  • Flavell, J.H. (1979). Metacognition and cognitive monitoring: a new area of cognitive–developmental inquiry. American Psychologist, 34(10), 906.

    Article  Google Scholar 

  • Garner, R. (1988). 5 - verbal-report data on cognitive and metacognitive strategies. In C.E. Weinstein, E.T. Goetz, & P.A. Alexander (Eds.) Learning and study strategies (pp. 63–76). San Diego: Academic Press. https://doi.org/10.1016/B978-0-12-742460-6.50011-6. http://www.sciencedirect.com/science/article/pii/B9780127424606500116.

  • Gasevic, D., Jovanovic, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128.

    Article  Google Scholar 

  • Gauch, B., & Biswas, G. (2016). Behavior changes across time and between populations in open-ended learning environments. In International Conference on Intelligent Tutoring Systems (pp. 187–196). Springer.

  • Gillies, R.M. (2003). The behaviors, interactions, and perceptions of junior high school students during small-group learning. Journal of Educational Psychology, 95(1), 137.

    Article  Google Scholar 

  • Glickman, M.E., Rao, S.R., & Schultz, M.R. (2014). False discovery rate control is a recommended alternative to bonferroni-type adjustments in health studies. Journal of Clinical Epidemiology, 67(8), 850–857.

    Article  Google Scholar 

  • Greene, J.A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45(4), 203–209.

    Article  Google Scholar 

  • Grosch, J., & Emmelmann, H. (1990). A tool box for compiler construction. In International Workshop on Compiler Construction (pp. 106–116). Springer.

  • Grover, S., Bienkowski, M., Niekrasz, J., & Hauswirth, M. (2016). Assessing problem-solving process at scale. In Proceedings of the third (2016) ACM conference on learning@ scale (pp. 245–248).

  • Hadwin, A.F., Nesbit, J.C., Jamieson-Noel, D., Code, J., & Winne, P.H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2-3), 107–124.

    Article  Google Scholar 

  • Hannafin, M., Land, S., & Oliver, K. (1999). Open learning environments: foundations, methods, and models. In Instructional-design theories and models: A new paradigm of instructional theory, (Vol. 2 pp. 115–140).

  • Hannafin, M.J., Hill, J.R., Land, S.M., & Lee, E. (2014). Student-centered, open learning environments: research, theory, and practice, (pp. 641–651). New York: Springer New York. https://doi.org/10.1007/978-1-4614-3185-5_51.

    Google Scholar 

  • Harris, A., Bonnett, V., Luckin, R., Yuill, N., & Avramides, K. (2009). Scaffolding effective help-seeking behaviour in mastery and performance oriented learners. In AIED (pp. 425–432).

  • Hestenes, D., Wells, M., & Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141–158.

    Article  Google Scholar 

  • Hmelo-Silver, C.E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.

    Article  Google Scholar 

  • Hopcroft, J.E., Motwani, R., & Ullman, J.D. (2001). Introduction to automata theory, languages, and computation. Acm Sigact News, 32(1), 60–65.

    Article  MATH  Google Scholar 

  • Hutchins, N.M., Biswas, G., Maróti, M., Lédeczi, Á., Grover, S., Wolf, R., Blair, K.P., Chin, D., Conlin, L., Basu, S., & et al. (2020a). C2stem: a system for synergistic learning of physics and computational thinking. Journal of Science Education and Technology, 29(1), 83–100.

    Article  Google Scholar 

  • Hutchins, N.M., Biswas, G., Zhang, N., Snyder, C., Lédeczi, Á., & Maróti, M. (2020b). Domain-specific modeling languages in computer-based learning environments: a systematic approach to support science learning through computational modeling. International Journal of Artificial Intelligence in Education, 30(4), 537–580.

    Article  Google Scholar 

  • Jacobson, M.J., & Wilensky, U. (2006). Complex systems in education: Scientific and educational importance and implications for the learning sciences. The Journal of the learning sciences, 15(1), 11–34.

    Article  Google Scholar 

  • Juhaňák, L., Zounek, J., & Rohlíková, L. (2019). Using process mining to analyze students’ quiz-taking behavior patterns in a learning management system. Computers in Human Behavior, 92, 496–506.

    Article  Google Scholar 

  • Käser, T., & Schwartz, D.L. (2020). Modeling and analyzing inquiry strategies in open-ended learning environments. International Journal of Artificial Intelligence in Education, 30(3), 504–535.

    Article  Google Scholar 

  • Kinnebrew, J.S., Loretz, K.M., & Biswas, G. (2013). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. JEDM— Journal of Educational Data Mining, 5(1), 190–219.

    Google Scholar 

  • Kinnebrew, J.S., Segedy, J.R., & Biswas, G. (2014). Analyzing the temporal evolution of students’ behaviors in open-ended learning environments. Metacognition and learning, 9(2), 187–215.

    Article  Google Scholar 

  • Kinnebrew, J.S., Segedy, J.R., & Biswas, G. (2017). Integrating model-driven and data-driven techniques for analyzing learning behaviors in open-ended learning environments. IEEE Transactions on Learning Technologies, 10(2), 140–153. https://doi.org/10.1109/TLT.2015.2513387.

    Article  Google Scholar 

  • Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89.

    Article  Google Scholar 

  • Land, S.M. (2000). Cognitive requirements for learning with open-ended learning environments. Educational Technology Research and Development, 48 (3), 61–78. https://doi.org/10.1007/BF02319858.

    Article  Google Scholar 

  • Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The betty’s brain system. International Journal of Artificial Intelligence in Education, 18(3), 181–208.

    Google Scholar 

  • Liu, R., Patel, R., & Koedinger, K.R. (2016). Modeling common misconceptions in learning process data. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (pp. 369–377).

  • Luckin, R., & du Boulay, B. (2016). Reflections on the ecolab and the zone of proximal development. International Journal of Artificial Intelligence in Education, 26(1), 416–430.

    Article  Google Scholar 

  • Luckin, R, & Hammerton, L (2002). Getting to know me: Helping learners understand their own learning needs through metacognitive scaffolding. In International Conference on Intelligent Tutoring Systems (pp. 759–771). Springer.

  • Maldonado-Mahauad, J., Pérez-sanagustín, M., Kizilcec, R.F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from big data: Identifying self-regulated learning strategies in massive open online courses. Computers in Human Behavior, 80, 179–196.

    Article  Google Scholar 

  • Matcha, W., Gašević, D., Jovanović, J., Pardo, A., Maldonado-Mahauad, J., & Pérez-Sanagustín, M. (2019). Detection of learning strategies: A comparison of process, sequence and network analytic approaches. In European conference on technology enhanced learning (pp. 525–540). Springer.

  • Mayer, RE (1988). 2 - learning strategies: An overview. In C.E. Weinstein, E.T. Goetz, & P.A. Alexander (Eds.) Learning and study strategies (pp. 11–22). San Diego: Academic Press. https://doi.org/10.1016/B978-0-12-742460-6.50008-6. http://www.sciencedirect.com/science/article/pii/B9780127424606500086.

  • McElhaney, K., Zhang, N., Basu, S., McBride, E., Biswas, G., & Chiu, J. (2020). Using computational modeling to integrate science and engineering curricular activities. In M. Gresalfi I.S. Horn (Eds.) The interdisciplinarity of the learning sciences, 14th international conference of the learning sciences (ICLS) 2020, Vol. 3.

  • Mendez, G.R., du Boulay, B., & Luckin, R. (2005). Be bold and take a challenge”: Could motivational strategies improve help-seeking. In Proceedings of the 2005 conference on artificial intelligence in education: supporting learning through intelligent and socially informed technology (pp. 459–466).

  • Mislevy, R.J., Almond, R.G., & Lukas, J.F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29.

    Article  Google Scholar 

  • Munshi, A., Rajendran, R., Ocumpaugh, J., Biswas, G., Baker, R.S., & Paquette, L. (2018). Modeling learners’ cognitive and affective states to scaffold srl in open-ended learning environments. In Proceedings of the 26th conference on user modeling, adaptation and personalization (pp. 131–138). ACM.

  • Neamtiu, I., Foster, J.S., & Hicks, M. (2005). Understanding source code evolution using abstract syntax tree matching. In Proceedings of the 2005 international workshop on Mining software repositories (pp. 1–5).

  • Newman, R.S. (1990). Children’s help-seeking in the classroom: The role of motivational factors and attitudes. Journal of Educational Psychology, 82(1), 71.

    Article  Google Scholar 

  • Nisbet, J., & Shucksmith, J. (2017). Learning strategies, Routledge, London.

  • NRC. (2000). How people learn, brain, mind, experience, and school: expanded edition. National Academies Press.

  • NRC. (2010). Report of a workshop on the scope and nature of computational thinking. National Academies Press.

  • Obergriesser, S., & Stoeger, H. (2020). Students’ emotions of enjoyment and boredom and their use of cognitive learning strategies–how do they affect one another? Learning and Instruction, 66, 101285.

    Article  Google Scholar 

  • Oxford, R.L. (2011). Strategies for learning a second or foreign language. Language Teaching, 44(2), 167.

    Article  Google Scholar 

  • Panadero, E., & Alonso Tapia, J. (2014). How do students self-regulate?: review of zimmerman’s cyclical model of self-regulated learning. Anales de psicologia.

  • Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: when measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735.

    Article  Google Scholar 

  • Parkinson, B., & Totterdell, P. (1999). Classifying affect-regulation strategies. Cognition & Emotion, 13(3), 277–303.

    Article  Google Scholar 

  • Pelánek, R. (2017). Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Modeling and User-Adapted Interaction, 27(3), 313–350.

    Article  Google Scholar 

  • Piech, C., Spencer, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L., & Sohl-Dickstein, J. (2015). Deep knowledge tracing. arXiv:1506.05908.

  • Pressley, M., Goodchild, F., Fleet, J., Zajchowski, R., & Evans, E.D. (1989). The challenges of classroom strategy instruction, (Vol. 89 pp. 301–342). http://www.jstor.org/stable/1001806

  • Rabinovich, M., Stern, M., & Klein, D. (2017). Abstract syntax networks for code generation and semantic parsing. arXiv:1704.07535.

  • Roscoe, R.D., Segedy, J.R., Sulcer, B., Jeong, H., & Biswas, G. (2013). Shallow strategy development in a teachable agent environment designed to support self-regulated learning. Computers & Education, 62, 286–297.

    Article  Google Scholar 

  • Roscoe, R.D., Allen, L.K., & McNamara, D.S. (2019). Contrasting writing practice formats in a writing strategy tutoring system. Journal of Educational Computing Research, 57(3), 723–754.

    Article  Google Scholar 

  • Rosenthal, R., Cooper, H., & Hedges, L. (1994). Parametric measures of effect size. The Handbook of Research Synthesis, 621, 231–244.

    Google Scholar 

  • Schmeck, R.R. (2013). Learning strategies and learning styles. Berlin: Springer Science & Business Media.

    Google Scholar 

  • Schraagen, J.M., Chipman, S.F., & Shalin, V.L. (2000). Cognitive task analysis. Psychology Press.

  • Schraw, G., Crippen, K.J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36(1-2), 111–139.

    Article  Google Scholar 

  • Schunk, D., & Greene, J. (2018). Handbook of self-regulation of learning and performance. https://doi.org/10.4324/9781315697048. London: Routledge.

  • Segedy, J.R., Kinnebrew, J.S., & Biswas, G. (2015a). Coherence over time: understanding day-to-day changes in students’ open-ended problem solving behaviors. In International Conference on Artificial Intelligence in Education (pp. 449–458). Springer.

  • Segedy, J.R., Kinnebrew, J.S., & Biswas, G. (2015b). Using coherence analysis to characterize self-regulated learning behaviours in open-ended learning environments. Journal of Learning Analytics, 2(1), 13–48.

    Article  Google Scholar 

  • Sengupta, P., Kinnebrew, J.S., Basu, S., Biswas, G., & Clark, D. (2013). Integrating computational thinking with k-12 science education using agent-based computation: a theoretical framework. Education and Information Technologies, 18(2), 351–380.

    Article  Google Scholar 

  • Shute, V.J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158.

    Article  Google Scholar 

  • Smit, K., de Brabander, C.J., Boekaerts, M., & Martens, R.L. (2017). The self-regulation of motivation: Motivational strategies as mediator between motivational beliefs and engagement for learning. International Journal of Educational Research, 82, 124–134.

    Article  Google Scholar 

  • Snyder, C., Biswas, G., Emara, M., Grover, S., & Conlin, L. (2019). Analyzing students’ synergistic learning processes in physics and ct by collaborative discourse analysis. Computer-supported collaborative learning.

  • Taub, M., Mudrick, N.V., Azevedo, R., Millar, G.C., Rowe, J., & Lester, J. (2017). Using multi-channel data with multi-level modeling to assess in-game performance during gameplay with crystal island. Computers in Human Behavior, 76, 641–655.

    Article  Google Scholar 

  • Taub, M., Mudrick, N.V., Rajendran, R., Dong, Y., Biswas, G., & Azevedo, R. (2018). How are students’ emotions associated with the accuracy of their note taking and summarizing during learning with itss?. In R. Nkambou, R. Azevedo, & J. Vassileva (Eds.) Intelligent tutoring systems (pp. 233–242). Cham: Springer International Publishing.

  • Taub, M., Azevedo, R., Rajendran, R., Cloude, E.B., Biswas, G., & Price, M.J. (2019). How are students’ emotions related to the accuracy of cognitive and metacognitive processes during learning with an intelligent tutoring system? Learning and Instruction.

  • Tsai, M.J., & Tsai, C.C. (2003). Information searching strategies in web-based science learning: The role of internet self-efficacy. Innovations in Education and Teaching International, 40(1), 43–50.

    Article  Google Scholar 

  • VanLehn, K. (2013). Model construction as a learning activity: a design space and review. Interactive Learning Environments, 21(4), 371–413.

    Article  Google Scholar 

  • Vermunt, J.D. (2020). Surveys and retrospective self-reports to measure strategies and strategic processing. In Handbook of Strategies and Strategic Processing (p. 118).

  • de Vries, E., Demetriadis, S., & Ainsworth, S. (2009). External representations for learning, (pp. 137–153). Berlin: Springer.

    Google Scholar 

  • Vygotsky, L.S. (1978). Mind in society: The development of higher mental process.

  • Wang, F., & Hannafin, M.J. (2005). Design-based research and technology-enhanced learning environments. Educational Technology Research and Development, 53(4), 5–23.

    Article  Google Scholar 

  • Weinstein, C., & Meyer, D. (1994). Learning strategies, teaching and testing. The International Encyclopedia of Education, 2, 3335–3340.

    Google Scholar 

  • Weinstein, C.E., Acee, T.W., & Jung, J. (2011). Self-regulation and learning strategies. New Directions for Teaching and Learning, 2011(126), 45–53.

    Article  Google Scholar 

  • Weintrop, D., Beheshti, E., Horn, M., Orton, K., Jona, K., Trouille, L., & Wilensky, U. (2016). Defining computational thinking for mathematics and science classrooms. Journal of Science Education and Technology, 25 (1), 127–147.

    Article  Google Scholar 

  • Whitelock-Wainwright, A., Laan, N., Wen, D., & Gašević, D. (2020). Exploring student information problem solving behaviour using fine-grained concept map and search tool data. Computers & Education, 145, 103731.

    Article  Google Scholar 

  • Wilensky, U., & Reisman, K. (2006). Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories—an embodied modeling approach. Cognition and instruction, 24(2), 171–209.

    Article  Google Scholar 

  • Wilensky, U., & Resnick, M. (1999). Thinking in levels: a dynamic systems approach to making sense of the world. Journal of Science Education and Technology, 8(1), 3–19.

    Article  Google Scholar 

  • Wing, J.M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35.

    Article  Google Scholar 

  • Winne, P.H. (1995). Inherent details in self-regulated learning. Educational Psychologist, 30(4), 173–187.

    Article  Google Scholar 

  • Winne, P.H., & Hadwin, A.F. (1998). Studying as self-regulated learning. Metacognition in Educational Theory and Practice, 93, 27–30.

    Google Scholar 

  • Winne, P.H., & Hadwin, A.F. (2013). nstudy: Tracing and supporting self-regulated learning in the internet. In International handbook of metacognition and learning technologies (pp. 293–308). Springer.

  • Winne, P.H., Jamieson-Noel, D., & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. Advances in motivation and achievement: New directions in measures and methods, 12, 121–155.

    Google Scholar 

  • Wittgenstein, L. (1968). Philosophical investigations. New York: Macmillan.

  • Xu, S., & Chee, Y.S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29(4), 360–384.

    Article  Google Scholar 

  • Yett, B., Snyder, C., Zhang, N., Hutchins, N. M., Mishra, S., & Biswas, G. (2020). Using log and discourse analysis to improve understanding of collaborative programming. In Proceedings of the International Conference on Computers in Education (ICCE).

  • Zaki, M.J. (2001). Spade: an efficient algorithm for mining frequent sequences. Machine Learning, 42(1-2), 31–60.

    Article  MATH  Google Scholar 

  • Zhang, L., VanLehn, K., Girard, S., Burleson, W., Chavez-Echeagaray, M.E., Gonzalez-Sanchez, J., & Hidalgo-Pontet, Y. (2014). Evaluation of a meta-tutor for constructing models of dynamic systems. Computers & Education, 75, 196–217.

    Article  Google Scholar 

  • Zhang, N. (2020). Supporting the integrated learning of science, engineering and computational thinking in an open-ended learning environment. PhD thesis, Vanderbilt University.

  • Zhang, N., & Biswas, G. (2018). Understanding students’ problem-solving strategies in a synergistic learning-by-modeling environment. In International conference on artificial intelligence in education (pp. 405–410). Springer.

  • Zhang, N., Biswas, G., & Dong, Y. (2017). Characterizing students’ learning behaviors using unsupervised learning methods. In International conference on artificial intelligence in education (pp. 430–441). Springer.

  • Zhang, N., Biswas, G., McElhaney, K.W., Basu, S., McBride, E., & Chiu, J.L. (2020). Studying the interactions between science, engineering, and computational thinking in a learning-by-modeling environment. In International conference on artificial intelligence in education (pp. 598–609). Springer.

  • Zhang, Y., Jin, R., & Zhou, Z.H. (2010). Understanding bag-of-words model: a statistical framework. International Journal of Machine Learning and Cybernetics, 1(1-4), 43–52.

    Article  Google Scholar 

  • Zimmerman, B.J. (2000). Chapter 2 - attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P.R. Pintrich, & M. Zeidner (Eds.) Handbook of self-regulation (pp. 13–39). San Diego: Academic Press. 10.1016/B978-012109890-2/50031-7. http://www.sciencedirect.com/science/article/pii/B9780121098902500317

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant Cyberlearning #1441542. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

The authors would like to thank Douglas Clark, Satabdi Basu, Ashlyn Pierson, Jenna Peet, and Naveeduddin Mohammed for their contribution in conceptualizing the research, adapting the pre-post assessment questions, providing educative materials, and developing the CTSiM learning environment. A different and abbreviated version of the present work using the same classroom study data has been published as a conference poster (Zhang and Biswas, 2018). We extended both the theoretical framework and analytical methods described in the poster paper, and have included a significant amount of new results in the present work.

Funding

This research was supported by a National Science Foundation Cyberlearning Grant #1441542

Author information

Authors and Affiliations

  1. Department of EECS, Institute for Software Integrated Systems, Vanderbilt University, Nashville, TN, USA

    Ningyu Zhang, Gautam Biswas & Nicole Hutchins

Authors
  1. Ningyu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gautam Biswas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicole Hutchins
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Gautam Biswas; Methodology: Gautam Biswas; Software: Ningyu Zhang; Investigation: Ningyu Zhang, Nicole Hutchins; Formal analysis: Ningyu Zhang, Gautam Biswas; Writing - Original Draft: Ningyu Zhang; Writing - Review & Editing: Gautam Biswas, Nicole Hutchins; Supervision: Gautam Biswas.

Corresponding author

Correspondence to Gautam Biswas.

Ethics declarations

Conflict of Interests

No potential conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Example Domain and CT question items used in the pre-post tests. Figure 5 shows a test item in the acceleration pre-post test. It was adapted from the Force Concept Inventory (Hestenes et al., 1992). Figure 6 is question 6 in the diffusion pre-post test adapted from Chi et al.’s work targeting the common misunderstanding of particle diffusion (Chi et al., 2012). Figures 7 and 8 target on the CT concepts of variable, sequence, Boolean logic, and conditionals. The items in the CT test come from Basu et al.’s work (Basu et al., 2017).

Fig. 5
figure 5

Question 1 in the acceleration unit pre-post test

Full size image
Fig. 6
figure 6

Question 2 in the diffusion unit pre-post test

Full size image
Fig. 7
figure 7

Question 2 in the CT pre-post test

Full size image
Fig. 8
figure 8

Question 3 in the CT pre-post test

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, N., Biswas, G. & Hutchins, N. Measuring and Analyzing Students’ Strategic Learning Behaviors in Open-Ended Learning Environments. Int J Artif Intell Educ 32, 931–970 (2022). https://doi.org/10.1007/s40593-021-00275-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-021-00275-x

Keywords

Search

Navigation