Abstract
Open Learner Models (OLMs) have great potential to support students’ Self-Regulated Learning (SRL) in Intelligent Tutoring Systems (ITSs). Yet few classroom experiments have been conducted to empirically evaluate whether and how an OLM can enhance students’ domain level learning outcomes through the scaffolding of SRL processes in an ITS. In two classroom experiments with a total of 302 7th- and 8th-grade students, we investigated the effect of (a) an OLM that supports students’ self-assessment of their equation-solving skills and (b) shared control over problem selection, on students’ equation-solving abilities, enjoyment of learning with the tutor, self-assessment accuracy, and problem selection decisions. In the first, smaller experiment, the hypothesized main effect of the OLM on students’ learning outcomes was confirmed; we found no main effect of shared control of problem selection, nor an interaction. In the second, larger experiment, the hypothesized main effects were not confirmed, but we found an interaction such that the students who had access to the OLM learned significantly better equation-solving skills than their counterparts when shared control over problem selection was offered in the system. Thus, the two experiments support the notion that an OLM can enhance students’ domain-level learning outcomes through scaffolding of SRL processes, and are among the first in-vivo classroom experiments to do so. They suggest that an OLM is especially effective if it is designed to support multiple SRL processes.








Similar content being viewed by others
Notes
The average scores of pre/post-tests are not normally distributed (even with Logarithmic and Square-Root transformations), so we used the learning gains as the dependent variables, which are normally distributed.
References
Aleven, V.: Rule-based cognitive modeling for intelligent tutoring systems. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 33–62. Springer, Berlin (2010)
Aleven, V., Koedinger, K.R.: Limitations of student control: do students know when they need help? In: Gauthier, G., Frasson, C., VanLehn, K. (eds.) Proceedings of the 5th International Conference on Intelligent Tutoring Systems, pp. 292–303. Springer, Berlin (2000)
Aleven, V., Koedinger, K.R.: Knowledge component approaches to learner modeling. In: Sottilare, R., Graesser, A., Hu, X., Holden, H. (eds.) Design Recommendations for Adaptive Intelligent Tutoring Systems, pp. 165–182. Orlando, US Army Research Laboratory (2013)
Aleven, V., McLaren, B.M., Sewall, J., Koedinger, K.R.: Example-tracing tutors: a new paradigm for intelligent tutoring systems. Int. J. Artif. Intell. Educ. 19(2), 105–154 (2009)
Aleven, V., McLaren, B.M., Sewall, J., van Velsen, M., Popescu, O., Demi, S., Koedinger, K.R.: Example-tracing tutors: intelligent tutor development for non-programmers. Int. J. Artif. Intell. Educ. 26(1), 224–269 (2016)
Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: lessons learned. J. Learn. Sci. 4(2), 167–207 (1995)
Atkinson, R.C.: Optimizing the learning of a second-language vocabulary. J. Exp. Psychol. 96(1), 124–129 (1972)
Arroyo, I., Ferguson, K., Johns, J., Dragon, T., Mehranian, H., Fisher, D., Barto, A., Mahadevan, S., Woolf, B.: Repairing disengagement with non invasive interventions. In: Proceedings of the International Conference on Artificial Intelligence in Education, pp. 195–202. Marina del Rey, CA (2007)
Azevedo, R., Aleven, V. (eds.): International Handbook on Metacognition in Computer-Based Learning Environments. Springer, Berlin, DE (2013)
Bandura, A.: Self-efficacy. In: Ramachaudran, V.S. (Ed.) Encyclopedia of Human Behaviour, vol. 4, pp. 71–81. Academic Press, New York (Reprinted in H. Friedman (Ed.). Encyclopedia of mental health, San Diego: Academic Press, 1998) (1994)
Basu, S., Biswas, G., Kinnebrew, J.S.: Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Model. User-Adapt. Interact. J. Personal. Res. 27 (2017) (Special Issue on Impact of Learner Modeling)
Bjork, R.A., Bjork, E.L.: Optimizing treatment and instruction: implications of a new theory of disuse. In: Nilsson, L.G., Ohta, N. (eds.) Memory and Society: Psychological Perspectives, pp. 109–133. Psychology Press, New York (2006)
Brusilovsky, P.: Adaptive navigation support: from adaptive hypermedia to the adaptive web and beyond. Psychol. J. 2(1), 7–23 (2004)
Brusilovsky, P., Schwarz, E., Weber, G.: ELM-ART: an intelligent tutoring system on World Wide Web. In: Frasson, C., Gauthier, G., Lesgold, A. (eds.) Proceedings of Third International Conference on Intelligent Tutoring Systems, ITS-96, pp. 261–269. Springer, Berlin (1996)
Brusilovsky, P., Sosnovsky, S., Shcherbinina, O.: QuizGuide: Increasing the educational value of individualized self-assessment quizzes with adaptive navigation support. In: Nall, J., Robson, R. (eds.) Proceedings of World Conference on E-Learning, pp. 1806–1813 (2004)
Brusilovsky, P., Somyürek, S., Guerra, J., Hosseini, R. Zadorozhny, V.: The value of social: comparing open student modeling and open social student modeling. In: Proceedings of the 23nd Conference on User Modeling, Adaptation and Personalization (UMAP 2015), June 29–July 3, 2015, Dublin, Ireland (2015)
Bull, S.: Supporting learning with open learner models. In: Proceedings of the 4th Hellenic Conference: Information and Communication Technologies in Education, Athens (2004)
Bull, S., Kay, J.: Metacognition and open learner models. In: Roll, I., Aleven, V. (eds.) Proceedings of Workshop on Metacognition and Self-Regulated Learning in Educational Technologies, International Conference on Intelligent Tutoring Systems, pp. 7–20 (2008)
Bull, S., Kay, J.: Open learner models. Advances in intelligent tutoring systems. In: Nkambou, R., Bourdeau, J., Mizoguchi, R. (eds.) Studies in Computational Intelligence, vol. 308, pp. 301–322. Springer, Berlin (2010)
Bull, S., Kay, J.: SMILI: a framework for interfaces to learning data in Open Learner Models (OLMs), learning analytics and related fields. Int. J. Artif. Intell. Educ. 26(1), 293–331 (2016)
Bull, S., Jackson, T., Lancaster, M.: Students’ interest in their misconceptions in first year electrical circuits and mathematics courses. Int. J. Electr. Eng. Educ. 47(3), 307–318 (2010)
Bull, S., Ginon, B., Boscolo, C., Johnson, M.D.: Introduction of learning visualisations and metacognitive support in a persuadable Open Learner Model. In: Gasevic, D., Lynch, G. (eds.) Proceeding of Learning Analytics and Knowledge 2016. ACM (2016)
Clark, C.R., Mayer, E.R.: E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Jossey-Bass, San Francisco (2011)
Cleary, T., Zimmerman, B.J.: Self-regulation differences during athletic practice by experts, nonexperts, and novices. J. Appl. Sport Psychol. 13, 61–82 (2000)
Corbalan, G., Kester, L., Van Merriënboer, J.J.G.: Selecting learning tasks: effects of adaptation and shared control on efficiency and task involvement. Contemp. Educ. Psychol. 33(4), 733–756 (2008)
Corbalan, G., Kester, L., van Merriënboer, J.J.G.: Combining shared control with variability over surface features: effects on transfer test performance and task involvement. Comput. Hum. Behav. 25(2), 290–298 (2009)
Corbett, A.: Cognitive mastery learning in the ACT Programming Tutor. AAAI Technical Report SS-00-01 (2000)
Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4(4), 253–278 (1995)
Corbett, A.T., Bhatnagar, A.: Student modeling in the ACT programming tutor: adjusting a procedural learning model with declarative knowledge. In: Jameson, A., Paris, C., Tasso, C. (eds.) User Modeling. Springer, New York (1997)
Cordova, D.I., Lepper, M.R.: Intrinsic motivation and the process of learning: beneficial effects of contextualization, personalization, and choice. J. Educ. Psychol. 88, 715–730 (1996)
Desmarais, M.C., Baker, R.S.: A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adapt. Interact. 22(1–2), 9–38 (2012)
Dimitrova, V., Brna, P.: From interactive open learner modeling to intelligent mentoring: STyLE-OLM and beyond. Int. J. Artif. Intell. Educ. 26(1), 332–349 (2016)
Dunlosky, J., Lipko, A.: Metacomprehension: a brief history and how to improve its accuracy. Curr. Dir. Psychol. Sci. 16, 228–232 (2007)
Flowerday, T., Schraw, G.: Effect of choice on cognitive and affective engagement. J. Educ. Res. 96(4), 207–215 (2003)
Gong, Y., Beck, J.E., Heffernan, N.T.: How to construct more accurate student models: comparing and optimizing knowledge tracing and performance factor analysis. Int. J. Artif. Intell. Educ. 21(1–2), 27–46 (2011)
Hartley, D., Mitrovic, A.: Supporting learning by opening the student model. In: Cerri, S.A., Gouardères, G., Paraguaçu, F. (eds.) Proceedings of the 6th International Conference on Intelligent Tutoring Systems, pp. 453–462. Springer, Berlin (2002)
Jameson, A., Schwarzkopf, E.: Pros and cons of controllability: an empirical study. In: De Bra, P., Brusilovsky, P., Conejo, R. (eds.) Adaptive Hypermedia and Adaptive Webbased Systems: Proceedings of AH 2002, pp. 193–202. Springer, Berlin (2002)
Kapur, M., Bielaczyc, K.: Designing for productive failure. J. Learn. Sci. 21(1), 45–83 (2012)
Kay, J.: Learner know thyself: student models to give learner control and responsibility. In: Halim, Z., Ottomann, T., Razak, Z. (eds.) Proceeding of the International Conference on Computers in Education. AACE (1997)
Kerly, A., Bull, S.: Children’s interactions with inspectable and negotiated learner models. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds.) Proceedings of the International Conference on Intelligent Tutoring Systems, pp. 132–141. Springer, Heidelberg (2008)
Koedinger, K.R., Corbett, A.T.: Cognitive tutors: technology bringing learning science to the classroom. In: Sawyer, K. (ed.) The Cambridge Handbook of the Learning Sciences. Cambridge University Press, Cambridge (2006)
Koedinger, K.R., Aleven, V.: Exploring the assistance dilemma in experiments with cognitive tutors. Educ. Psychol. Rev. 19(3), 239–264 (2007)
Koedinger, K.R., Corbett, A.T., Perfetti, C.: The Knowledge-Learning-Instruction (KLI) framework: toward bridging the science-practice chasm to enhance robust student learning. Technical report, Carnegie Mellon University, Human Computer Interaction Institute, Pittsburgh (2010)
Kostons, D., van Gog, T., Paas, F.: Self-assessment and task selection in learner-controlled instruction: differences between effective and ineffective learners. Comput. Educ. 54(4), 932–940 (2010)
Lee, S.J.H., Bull, S.: An open learner model to help parents help their children. Technol. Instr. Cognit. Learn. 6(1), 29–51 (2008)
Long, Y., Aleven, V.: Students’ understanding of their student model. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) Proceedings of the 15th International Conference on Artificial Intelligence in Education, pp. 179–186. Springer, Berlin (2011)
Long, Y., Aleven, V.: Active learners?: redesigning an intelligent tutoring system to support self-regulated learning. In: Proceedings of EC-TEL 2013: Scaling up Learning for Sustained Impact, pp. 490–495 (2013a)
Long, Y., Aleven, V.: Supporting students’ self-regulated learning with an open learner model in a linear equation tutor. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 219–228 (2013b)
Long, Y., Aleven, V.: Skill diaries: improve student learning in an intelligent tutoring system with periodic self-assessment. In: Proceedings of the 16th International Conference on Artificial Intelligence in Education, pp. 249–258 (2013c)
Long, Y., Aleven, V.: Gamification of joint student/system control over problem selection in a linear equation tutor. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panour-gia, K. (eds.) Proceedings of the 12th International Conference on Intelligent Tutoring Systems, pp. 378–387. Springer, New York (2014)
Long, Y., Aleven, V. (2016). Mastery-oriented shared student/system control over problem selection in a linear equation tutor. In: Proceedings of the 13th International Conference on Intelligent Tutoring System, pp. 90–100
Mabbott, A., Bull, S.: Comparing student-constructed open learner model presentations to the domain. In: Koedinger, K., Luckin, R., Greer, J. (eds.) Proceedings of the International Conference on Artificial Intelligence in Education. IOS Press, Amsterdam (2007)
Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., Clayphan, A.: The LATUX workflow: designing and deploying awareness tools in technology-enabled learning settings. In: Proceedings of the Fifth International Conference on Learning Analytics and Knowledge, pp. 1–10. ACM (2015)
Metcalfe, J.: Metacognitive judgments and control of study. Curr. Dir. Psychol. Sci. 18(3), 159–163 (2009)
Metcalfe, J., Kornell, N.: The dynamics of learning and allocation of study time to a region of proximal learning. J. Exp. Psychol. Gen. 132(4), 530 (2003)
Metcalfe, J., Kornell, N.: A region of proximal learning model of study time allocation. J. Mem. Lang. 52(4), 463–477 (2005)
Mitrovic, A., Martin, B.: Scaffolding and fading problem selection in SQL-Tutor. In: Hoppe, U., Verdejo, F., Kay, J. (eds.) Proceedings of the 11th International Conference on Artificial Intelligence in Education, pp. 479–481. Springer, Berlin (2003)
Mitrovic, A., Martin, B.: Evaluating the effect of open student models on self-assessment. Int. J. Artif. Intell. Educ. 17(2), 121–144 (2007)
Mitrovic, A., Mayo, M., Suraweera, P., Martin, B.: Constraint-based tutors: A success story. In: Monostori, L., Váncza, J., Ali, M. (eds.) Proceedings 14th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, pp. 931–940. Springer, Berlin (2001)
Niemiec, R.P., Sikorski, C., Walberg, H.J.: Learner-control effects: a review of reviews and a meta-analysis. J. Educ. Comput. Res. 15(2), 157–174 (1996)
Parra, D., Brusilovsky, P.: User-controllable personalization: a case study with SetFusion. Int. J. Hum.-Comput. Stud. 78, 43–67 (2015)
Pardos, Z. A., Heffernan, N. T.: Modeling individualization in a bayesian networks implementation of knowledge tracing. In: deBra, P., Kobsa, A., Chin, D. (eds.) Proceedings of the 18th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2010, PP. 255–266. Springer, Berlin (2010)
Pardos, Z.A., Bergner, Y., Seaton, D., Pritchard, D.E.: Adapting Bayesian knowledge tracing to a massive open online college course in edX. In: D’Mello, S.K., Calvo, R.A., Olney, A. (eds.) Proceedings of the 6th International Conference on Educational Data Mining (EDM), pp. 137–144. Memphis, TN (2013)
Perez-Marin, D., Alfonseca, E., Rodriguez, P., Pascual-Neito, I.: A study on the possibility of automatically estimating the confidence value of students’ knowledge in generated conceptual models. J. Comput. 2(5), 17–26 (2007)
Pintrich, P.R.: A conceptual framework for assessing motivation and self-regulated learning in college students. Educ. Psychol. Rev. 16, 385–407 (2004)
Pintrich, P.R., De Groot, E.V.: Motivational and self-regulated learning components of classroom academic performance. J. Educ. Psychol. 82, 33–40 (1990)
Schneider, W., Lockl, K.: The development of metacognitive knowledge in children and adolescents. In: Perfect, T.J., Schwartz, B.L. (eds.) Proceedings of Applied metacognition, pp. 224–257. Cambridge University Press, Cambridge (2002)
Schraw, G.A.: Conceptual analysis of five measures of metacognitive monitoring. Metacognit. Learn. 4(1), 33–45 (2009)
Schraw, G., Flowerday, T., Reisetter, M.: The role of choice in reader engagement. J. Educ. Psychol. 90, 705–714 (1998)
Thiede, K.W., Anderson, M.C.M., Therriault, D.: Accuracy of metacognitive monitoring affects learning of texts. J. Educ. Psychol. 95, 66–73 (2003)
Tongchai, N.: Impact of self-regulation and open learner model on learning achievement in blended learning environment. Int. J. Inf. Educ. Technol 6(5), 343–347 (2016)
University of Rochester: intrinsic motivation inventory (IMI) (1994). http://www.psych.rochester.edu/SDT/measures/IMI_description.php. Accessed January 2013
Vandewaetere, M., Clarebout, G.: Can instruction as such affect learning? The case of learner control. Comput. Educ. 57(4), 2322–2332 (2011)
VanLehn, K.: The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ. Psychol. 46(4), 197–221 (2011)
VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Wintersgill, M.: The Andes physics tutoring system: lessons learned. Int. J. Artif. Intell. Educ. 15(1), 147–204 (2005)
Waalkens, M., Aleven, V., Taatgen, N.: Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems. Comput. Educ. 60, 159–171 (2013)
Winne, P.H., Hadwin, A.E.: Studying as self-regulated learning. In: Hacker, D.J., Dunlosky, J., Graesser, A.C. (eds.) Metacognition in Educational Theory and Practice, pp. 277–304. Lawrence Erlbaum Associates, Mahwah (1998)
Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized bayesian knowledge tracing models. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) Proceedings of the 16th international conference on artificial intelligence in education, AIED 2013, pp. 171–180. Springer, Berlin (2013)
Zimmerman, B.J., Martinez-Pons, M.: Development of a structured interview for assessing students’ use of self-regulated learning strategies. Am. Educ. Res. J. 23, 614–628 (1986)
Zimmerman, B.J.: Attaining self-regulation: a social cognitive perspective. In: Boekaerts, M., Pintrich, P., Zeidner, M. (eds.) Handbook of Self-Regulation, pp. 1–39. Academic Press, San Diego (2000)
Acknowledgements
We thank Jonathan Sewall, Borg Lojasiewicz, Octav Popescu, Brett Leber, Gail Kusbit and Emily Zacchero for their kind help with this work. We would also like to thank the participating teachers and students. This work was funded by a National Science Foundation grant to the Pittsburgh Science of Learning Center (NSF Award SBE0354420).
Author information
Authors and Affiliations
Corresponding author
Additional information
The paper is based on work that was conducted while the first author was at the Human-Computer Interaction Institute of Carnegie Mellon University. A conference paper based on Classroom Experiment 1 was published in 2013 (Long and Aleven 2013b).
Rights and permissions
About this article
Cite this article
Long, Y., Aleven, V. Enhancing learning outcomes through self-regulated learning support with an Open Learner Model. User Model User-Adap Inter 27, 55–88 (2017). https://doi.org/10.1007/s11257-016-9186-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11257-016-9186-6