Skip to main content

Clustering Learner’s Metacognitive Judgment Accuracy and Bias to Explore Learning with AIEd Systems

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13355))

Included in the following conference series:

  • 4587 Accesses

Abstract

Metacognitive monitoring and regulation are key dynamic psychological processes in predicting learning, reasoning, and problem solving across AIEd systems. They are impacted by one’s metacognitive knowledge, skills, and experiences. Understanding the dynamical processes underlying metacognition are critical to design intelligent adaptive scaffolding. Metacognition may be assumed to function at hierarchical levels of abstraction including a local level (i.e., isolated metacognitive judgements) and global level (i.e., self-beliefs). However, metacognitive research for complex learning has traditionally disregarded the concept of a general metacognitive ability due to local level fluctuations in temporal metacognitive accuracy. In our study, we shift our analyses to reflect a global-level approach to study metacognitive judgment ability by measuring both accuracy and bias across a series of metacognitive judgments. Using hierarchical clustering on undergraduates’ (n = 58) metacognitive judgments’ accuracy and bias while learning about nine human biological systems with MetaTutor-IVH, a multimedia-based learning environment, we show that some learners show patterns of global-level metacognitive ability. Specifically, we find that learners who tended to have low metacognitive accuracy across all judgment types performed worse overall on learning outcomes. Examining the bias of these learners, we found they tended to be under-confident across all judgment types. Our work suggests that considering multiple metrics of local-level metacognitive judgments’ accuracy and bias, can be aggregated to depict a global-level metacognitive ability of learners that is correlated to learning outcomes. We further discuss the impact of our findings for the design of adaptive scaffolding of AIEd systems to foster metacognition.

The research presented in this paper is supported by funding from the National Science Foundation (DRL 1431532).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Azevedo, R., Wiedbusch, M.: Theories of metacognition and pedagogy applied in aied systems. In: du Boulay, B., Mitrovic, A., Yacef, K. (eds.) Handbook of Artificial Intelligence in Education. (In Press)

    Google Scholar 

  2. Flavell, J.H.: Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. Am. Psychol. 34(10), 906 (1979)

    Article  Google Scholar 

  3. Koriat, A.: When two heads are better than one and when they can be worse: the amplification hypothesis. J. Exp. Psychol. Gen. 144(5), 934 (2015)

    Article  Google Scholar 

  4. Tarricone, P.: The Taxonomy of Metacognition. Psychology Press, New York (2011)

    Google Scholar 

  5. Winne, P.H.: Cognition and metacognition within self-regulated learning. In: Schunk, D., Greene, J. (eds.) Handbook of Self-regulation of Learning and Performance, pp. 36–48. Routledge, New York (2018)

    Google Scholar 

  6. Nelson, T.O., Narens, L.: Metamemory: A theoretical framework and new findings. Psychology of Learning and Motivation, p. 125 (1990)

    Google Scholar 

  7. Veenman, M.V., Hout-Wolters, V., Bernadette, H., Afflerbach, P.: Metacognition and learning: conceptual and methodological considerations. Metacogn. Learn. 1(1), 3–14 (2006)

    Article  Google Scholar 

  8. Seow, T.X., Rouault, M., Gillan, C.M., Fleming, S.M.: How local and global metacognition shape mental health. Biol. Psychiat. 90(7), 436–446 (2021)

    Article  Google Scholar 

  9. Jemstedt, A., Kubik, V., Jönsson, F.U.: What moderates the accuracy of ease of learning judgments? Metacogn. Learn. 12(3), 337–355 (2017). https://doi.org/10.1007/s11409-017-9172-3

    Article  Google Scholar 

  10. Leonesio, R.J., Nelson, T.O.: Do different metamemory judgments tap the same underlying aspects of memory? J. Exp. Psychol. Learn. Mem. Cogn. 16(3), 464 (1990)

    Article  Google Scholar 

  11. Mazzoni, G., Cornoldi, C., Tomat, L., Vecchi, T.: Remembering the grocery shopping list: A study on metacognitive biases. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 11(3), 253–267 (1997)

    Article  Google Scholar 

  12. McCarley, J.S., Gosney, J.: Metacognitive judgments in a simulated luggage screening task. In: Proceedings of the human factors and ergonomics society annual meeting. vol. 49, pp. 1620–1624. SAGE Publications Sage CA: Los Angeles, CA (2005)

    Google Scholar 

  13. Dever, D.A., Wiedbusch, M., Azevedo, R.: Learners’ gaze behaviors and metacognitive judgments with an agent-based multimedia environment. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11626, pp. 58–61. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23207-8_11

    Chapter  Google Scholar 

  14. Brewer, W.F., Sampaio, C.: The metamemory approach to confidence: a test using semantic memory. J. Mem. Lang. 67(1), 59–77 (2012)

    Article  Google Scholar 

  15. Chua, E.F., Schacter, D.L., Rand-Giovannetti, E., Sperling, R.A.: Understanding metamemory: neural correlates of the cognitive process and subjective level of confidence in recognition memory. Neuroimage 229(4), 1150–1160 (2006)

    Article  Google Scholar 

  16. Dougherty, M.R., Scheck, P., Nelson, T.O., Narens, L.: Using the past to predict the future. Memory Cogn. 33(6), 1096–1115 (2005)

    Article  Google Scholar 

  17. Hines, J.C., Touron, D.R., Hertzog, C.: Metacognitive influences on study time allocation in an associative recognition task: an analysis of adult age differences. Psychol. Aging 24(2), 462 (2009)

    Article  Google Scholar 

  18. Ryals, A.J., Rogers, L.M., Gross, E.Z., Polnaszek, K.L., Voss, J.L.: Associative recognition memory awareness improved by theta-burst stimulation of frontopolar cortex. Cereb. Cortex 26(3), 1200–1210 (2016)

    Article  Google Scholar 

  19. Biswas, G., Segedy, J.R., Bunchongchit, K.: From design to implementation to practice a learning by teaching system: betty’s brain. Int. J. Artif. Intell. Educ. 26(1), 350–364 (2016)

    Article  Google Scholar 

  20. Britton, B.K., Van Dusen, L., Gülgöz, S., Glynn, S.M., Sharp, L.: Accuracy of learnability judgments for instructional texts. J. Educ. Psychol. 83(1), 43 (1991)

    Article  Google Scholar 

  21. Jönsson, F.U., Kerimi, N.: An investigation of students’ knowledge of the delayed judgements of learning effect. J. Cogn. Psychol. 23(3), 358–373 (2011)

    Article  Google Scholar 

  22. Koriat, A.: How do we know that we know? the accessibility model of the feeling of knowing. Psychol. Rev. 100(4), 609 (1993)

    Article  Google Scholar 

  23. Kelemen, W.L., Frost, P.J., Weaver, C.A.: Individual differences in metacognition: evidence against a general metacognitive ability. Memory Cogn. 28(1), 92–107 (2000)

    Article  Google Scholar 

  24. Schraw, G., Dunkle, M.E., Bendixen, L.D., Roedel, T.D.: Does a general monitoring skill exist? J. Educ. Psychol. 87(3), 433 (1995)

    Article  Google Scholar 

  25. Schraw, G., Nietfeld, J.: A further test of the general monitoring skill hypothesis. J. Educ. Psychol. 90(2), 236 (1998)

    Article  Google Scholar 

  26. Glaser, R., Schauble, L., Raghavan, K., Zeitz, C.: Scientific reasoning across different domains. In: De Corte, E., Linn, M.C., Mandl, H., Verschaffel, L. (eds.) Computer-Based Learning Environments and Problem Solving. NATO ASI Series, vol. 84, pp. 345–371. Springer, Heidelberg (1992). https://doi.org/10.1007/978-3-642-77228-3_16

    Chapter  Google Scholar 

  27. Veenman, M.V., Wilhelm, P., Beishuizen, J.J.: The relation between intellectual and metacognitive skills from a developmental perspective. Learn. Instr. 14(1), 89–109 (2004)

    Article  Google Scholar 

  28. Veenman, M.V., Prins, F.J., Verheij, J.: Learning styles: self-reports versus thinking-aloud measures. Br. J. Educ. Psychol. 73(3), 357–372 (2003)

    Article  Google Scholar 

  29. Azevedo, R., Taub, M., Mudrick, N.V., Martin, S., Grafsgaard, J.: Using multi-channel trace data to infer and foster self-regulated learning between humans and advanced learning technologies. In: Handbook of self-regulation of learning and performance, vol. 2. Routledge New York (2018)

    Google Scholar 

  30. Dever, D.A., Wortha, F., Wiedbusch, M.D., Azevedo, R.: Effectiveness of system-facilitated monitoring strategies on learning in an intelligent tutoring system. In: Zaphiris, P., Ioannou, A. (eds.) HCII 2021. LNCS, vol. 12784, pp. 250–263. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77889-7_17

    Chapter  Google Scholar 

  31. Taub, M., Azevedo, R.: How does prior knowledge influence eye fixations and sequences of cognitive and metacognitive SRL processes during learning with an intelligent tutoring system? Int. J. Artif. Intell. Educ. 29(1), 1–28 (2019)

    Article  Google Scholar 

  32. Wiedbusch, M., Dever, D., Wortha, F., Cloude, E.B., Azevedo, R.: Revealing data feature differences between system- and learner-initiated self-regulated learning processes within hypermedia. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2021. LNCS, vol. 12792, pp. 481–495. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77857-6_34

    Chapter  Google Scholar 

  33. Pekrun, R., Goetz, T., Frenzel, A.C., Barchfeld, P., Perry, R.P.: Measuring emotions in students’ learning and performance: the achievement emotions questionnaire (AEQ). Contemp. Educ. Psychol. 36(1), 36–48 (2011)

    Article  Google Scholar 

  34. Gross, J.J., John, O.P.: Individual differences in two emotion regulation processes: implications for affect, relationships, and well-being. J. Pers. Soc. Psychol. 85(2), 348 (2003)

    Article  Google Scholar 

  35. Chow, P.I., Berenbaum, H.: Perceived utility of emotion: the structure and construct validity of the perceived affect utility scale in a cross-ethnic sample. Cultur. Divers. Ethnic Minor. Psychol. 18(1), 55 (2012)

    Article  Google Scholar 

  36. Wiedbusch, M. D., Azevedo, R.: Modeling metacomprehension monitoring accuracy with eye gaze on informational content in a multimedia learning environment. In: ACM Symposium on Eye Tracking Research and Applications, pp. 1–9 (2020)

    Google Scholar 

  37. Schraw, G.: A conceptual analysis of five measures of metacognitive monitoring. Metacogn. Learn. 4(1), 33–45 (2009)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Megan D. Wiedbusch .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wiedbusch, M.D., Sonnenfeld, N., Dever, D., Azevedo, R. (2022). Clustering Learner’s Metacognitive Judgment Accuracy and Bias to Explore Learning with AIEd Systems. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11644-5_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11643-8

  • Online ISBN: 978-3-031-11644-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics