Jake Barrett
ABSTRACT
Modelling student performance is an increasingly popular goal in the learning analytics community. A common method for this task is Bayesian Knowledge Tracing (BKT), which predicts student performance and topic mastery using the student’s answer history. While BKT has strong qualities and good empirical performance, like many machine learning approaches it can be prone to bias. In this study we demonstrate an inherent bias in BKT with respect to students’ income support levels and gender, using publicly available data. We find that this bias is likely a result of the model’s ‘slip’ parameter disregarding answer speed when deciding if a student has lost mastery status. We propose a new BKT model variation that directly considers answer speed, resulting in a significant fairness increase without sacrificing model performance. We discuss the role of answer speed as a potential cause of BKT model bias, as well as a method to minimise bias in future implementations.
- Ghodai Abdelrahman, Qing Wang, and Bernardo Nunes. 2023. Knowledge Tracing: A Survey. ACM Comput. Surv. 55, 11, Article 224 (feb 2023), 37 pages. https://doi.org/10.1145/3569576Google Scholar
Digital Library
- Fangzhe Ai, Yishuai Chen, Yuchun Guo, Yongxiang Zhao, Zhenzhu Wang, Guowei Fu, and Guangyan Wang. 2019. Concept-aware deep knowledge tracing and exercise recommendation in an online learning system. In The 12th International Conference on Educational Data Mining. ERIC, 240–245.Google Scholar
- Anirudhan Badrinath, Frederic Wang, and Zachary Pardos. 2021. pybkt: An accessible python library of bayesian knowledge tracing models. arXiv preprint arXiv:2105.00385 (2021).Google Scholar
- Ryan S Baker and Aaron Hawn. 2021. Algorithmic bias in education. International Journal of Artificial Intelligence in Education (2021), 1–41. https://doi.org/10.35542/osf.io/pbmvzGoogle ScholarCross Ref
- George Clement Bond. 1981. Social Economic Status and Educational Achievement: A Review Article. https://doi.org/10.1525/aeq.1981.12.4.05x1811qGoogle ScholarCross Ref
- Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency(Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, 134–148. https://doi.org/10.1145/3555101Google ScholarDigital Library
- Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction 4 (1994), 253–278. https://doi.org/10.1007/BF01099821Google ScholarCross Ref
- Yossi Ben David, Avi Segal, and Ya’akov (Kobi) Gal. 2016. Sequencing Educational Content in Classrooms Using Bayesian Knowledge Tracing. In Proceedings of the Sixth International Conference on Learning Analytics & Knowledge (Edinburgh, United Kingdom) (LAK ’16). Association for Computing Machinery, New York, NY, USA, 354–363. https://doi.org/10.1145/2883851.2883885Google ScholarDigital Library
- Shayan Doroudi and Emma Brunskill. 2019. Fairer but Not Fair Enough On the Equitability of Knowledge Tracing. Proceedings of the 9th International Conference on Learning Analytics & Knowledge, 335–339. https://doi.org/10.1145/3303772.3303838Google ScholarDigital Library
- Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226. https://doi.org/10.1145/2090236.2090255Google ScholarDigital Library
- Matt Faus. 2014. Khan Academy Mastery Mechanics. https://mattfaus.com/2014/07/03/khan-academy-mastery-mechanics/Google Scholar
- Mingyu Feng, Neil Heffernan, and Kenneth Koedinger. 2009. Addressing the Assessment Challenge with an Online System That Tutors as it Assesses. User Modeling and User-Adapted Interaction 19 (2009), 243–266. https://doi.org/10.1007/s11257-009-9063-7Google ScholarDigital Library
- Dawn Forman, Lovemore Nyatanga, and Terry Rich. 2002. E-learning and educational diversity. Nurse Education Today 22, 1 (2002), 76–82. https://doi.org/10.1054/nedt.2001.0740Google ScholarCross Ref
- Josh Gardner, Christopher Brooks, and Ryan Baker. 2019. Evaluating the fairness of predictive student models through slicing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge. 225–234. https://doi.org/10.1145/3303772.3303791Google ScholarDigital Library
- Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. Proceedings of the 30th International Conference on Neural Information Processing Systems, 3323–3331. https://doi.org/10.5555/3157382.3157469Google ScholarDigital Library
- Kenneth Holstein and Shayan Doroudi. 2021. Equity and Artificial Intelligence in Education: Will "AIEd" Amplify or Alleviate Inequities in Education?arXiv preprint arXiv:2104.12920 (2021). https://doi.org/10.48550/arXiv.2104.12920Google ScholarCross Ref
- https://www.eedi.com. [n. d.]. NeurIPS 2020 Education Challenge. https://eedi.com/projects/neurips-education-challengeGoogle Scholar
- Qian Hu and Huzefa Rangwala. 2020. Towards Fair Educational Data Mining: A Case Study on Detecting At-Risk Students. International Educational Data Mining Society (2020).Google Scholar
- Stephen Hutt, Margo Gardner, Angela L Duckworth, and Sidney K D’Mello. 2019. Evaluating Fairness and Generalizability in Models Predicting On-Time Graduation from College Applications. International Educational Data Mining Society (2019).Google Scholar
- Weijie Jiang and Zachary A Pardos. 2021. Towards equity and algorithmic fairness in student grade prediction. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 608–617. https://doi.org/10.1145/3461702.3462623Google ScholarDigital Library
- Young J Juhn, Euijung Ryu, Chung-Il Wi, Katherine S King, Momin Malik, Santiago Romero-Brufau, Chunhua Weng, Sunghwan Sohn, Richard R Sharp, and John D Halamka. 2022. Assessing socioeconomic bias in machine learning algorithms in health care: a case study of the HOUSES index. Journal of the American Medical Informatics Association 29, 7 (04 2022), 1142–1151. https://doi.org/10.1093/jamia/ocac052Google ScholarCross Ref
- Slava Kalyuga. 2007. Enhancing instructional efficiency of interactive e-learning environments: A cognitive load perspective. Educational psychology review 19 (2007), 387–399. https://doi.org/10.1007/s10648-007-9051-6Google ScholarCross Ref
- Kim Kelly, Yan Wang, Tamisha Thompson, and Neil Heffernan. 2016. Defining mastery: Knowledge tracing versus n-consecutive correct responses. Student Modeling From Different Aspects, 39.Google Scholar
- Lynn Kettell. 2020. Young adult carers in higher education: the motivations, barriers and challenges involved – a UK study. Journal of Further and Higher Education 44, 1 (2020), 100–112. https://doi.org/10.1080/0309877X.2018.1515427Google ScholarCross Ref
- Mohammad Khajah, Robert V Lindsey, and Michael C Mozer. 2016. How deep is knowledge tracing?arXiv preprint arXiv:1604.02416 (2016).Google Scholar
- Michael Madaio, Su Lin Blodgett, Elijah Mayfield, and Ezekiel Dixon-Román. 2021. Confronting structural inequities in AI for education. arXiv preprint arXiv:2105.08847 (2021).Google Scholar
- Kate Malleson. 2018. Equality Law and the Protected Characteristics. Modern Law Review 81, 4 (2018), 598–621. https://doi.org/10.1111/1468-2230.12353Google ScholarCross Ref
- Chric Warwick-Evans. Michelle Chance. 2020. Social Class Discrimination – Time for a New Protected Characteristic for a Post-Covid Britain? (2020). https://www.rosenblatt-law.co.uk/insight/social-class-discrimination-time-for-a-new-protected-characteristic-for-a-post-covid-britain/Google Scholar
- Ayesha Nadeem, Olivera Marjanovic, Babak Abedin, 2022. Gender bias in AI-based decision-making systems: a systematic literature review. Australasian Journal of Information Systems 26 (2022). https://doi.org/10.3127/ajis.v26i0.3835Google ScholarCross Ref
- OECD. 2015. The ABC of Gender Equality in Education. 35–61 pages. https://doi.org/10.1787/9789264229945-enGoogle ScholarCross Ref
- Radek Pelánek. 2017. Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques. User Modeling and User-Adapted Interaction 27 (2017), 313–350. https://doi.org/10.1007/s11257-017-9193-2Google ScholarDigital Library
- Radek Pelánek. 2018. Conceptual Issues in Mastery Criteria: Differentiating Uncertainty and Degrees of Knowledge, Vol. 19. Artificial Intelligence in Education, 450–461. https://doi.org/10.1007/978-3-319-93843-1_33Google ScholarCross Ref
- Chris Piech, Jonathan Bassen, Jonathan Huang, Surya Ganguli, Mehran Sahami, Leonidas J Guibas, and Jascha Sohl-Dickstein. 2015. Deep knowledge tracing. Advances in neural information processing systems 28 (2015). https://doi.org/10.5555/2969239.2969296Google ScholarDigital Library
- Yumeng Qiu, Yingmei Qi, Hanyuan Lu, Zachary Pardos, and Neil Heffernan. 2011. Does Time Matter? Modeling the Effect of Time with Bayesian Knowledge Tracing. EDM 2011 - Proceedings of the 4th International Conference on Educational Data Mining, 139–148.Google Scholar
- Kit T Rodolfa, Hemank Lamba, and Rayid Ghani. 2021. Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policy. Nature Machine Intelligence 3, 10 (2021), 896–904. https://doi.org/10.1038/s42256-021-00396-xGoogle ScholarCross Ref
- Salisu Muhammad Sani, Abdullahi Baffa Bichi, and Shehu Ayuba. 2016. Artificial intelligence approaches in student modeling: Half decade review (2010-2015). IJCSN-International Journal of Computer Science and Network 5, 5.Google Scholar
- Lele Sha, Mladen Rakovic, Alexander Whitelock-Wainwright, David Carroll, Victoria M Yew, Dragan Gasevic, and Guanliang Chen. 2021. Assessing algorithmic fairness in automatic classifiers of educational forum posts. In Artificial Intelligence in Education: 22nd International Conference, AIED 2021, Utrecht, The Netherlands, June 14–18, 2021, Proceedings, Part I 22. Springer, 381–394. https://doi.org/10.1007/978-3-030-78292-4_31Google ScholarDigital Library
- Sunny Shrestha and Sanchari Das. 2022. Exploring gender biases in ML and AI academic research through systematic literature review. Frontiers in artificial intelligence 5 (2022), 976838. https://doi.org/10.3389/frai.2022.976838Google ScholarCross Ref
- Sebastian Tschiatschek, Maria Knobelsdorf, and Adish Singla. 2022. Equity and Fairness of Bayesian Knowledge Tracing. Proceedings of the 15th International Conference on Educational Data Mining, 578–582. https://doi.org/10.5281/zenodo.6853011Google ScholarCross Ref
- Jan Van Bavel, Christine R Schwartz, and Albert Esteve. 2018. The reversal of the gender gap in education and its consequences for family life. Annual review of sociology 44 (2018), 341–360. https://doi.org/10.1146/annurev-soc-073117-041215Google ScholarCross Ref
- Wim J Van Der Linden. 2009. Conceptual issues in response-time modeling. Journal of Educational Measurement 46, 3 (2009), 247–272. https://doi.org/10.1111/j.1745-3984.2009.00080.xGoogle ScholarCross Ref
- Yuyan Wang, Xuezhi Wang, Alex Beutel, Flavien Prost, Jilin Chen, and Ed H Chi. 2021. Understanding and improving fairness-accuracy trade-offs in multi-task learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 1748–1757. https://doi.org/10.1145/3447548.3467326Google ScholarDigital Library
- Sonja Bethune Wendy Conaway. 2015. Implicit Bias and First Name Stereotypes: What Are the Implications for Online Instruction?Journal of Asynchronous Learning Networks 19, 3. https://doi.org/10.24059/olj.v19i3.674Google ScholarCross Ref
- Renzhe Yu, Qiujie Li, Christian Fischer, Shayan Doroudi, and Di Xu. 2020. Towards Accurate and Fair Prediction of College Success: Evaluating Different Sources of Student Data. International Educational Data Mining Society (2020).Google Scholar
Index Terms
- Improving Model Fairness with Time-Augmented Bayesian Knowledge Tracing
Recommendations
Toward the Evaluation of Educational Videos using Bayesian Knowledge Tracing and Big Data
L@S '15: Proceedings of the Second (2015) ACM Conference on Learning @ ScaleAlong with the advent of MOOCs and other online learning platforms such as Khan Academy, the role of online education has continued to grow in relation to that of traditional on-campus instruction. Rather than tackle the problem of evaluating large ...
Evaluating Bayesian Knowledge Tracing for Estimating Learner Proficiency and Guiding Learner Behavior
L@S '20: Proceedings of the Seventh ACM Conference on Learning @ ScaleOpen navigation online learning systems allow learners to choose the next learning activity. These systems can be instrumented to provide learners with feedback to help them choose the next learning activity. One type of feedback is providing an ...
Modeling individualization in a bayesian networks implementation of knowledge tracing
UMAP'10: Proceedings of the 18th international conference on User Modeling, Adaptation, and PersonalizationThe field of intelligent tutoring systems has been using the well known knowledge tracing model, popularized by Corbett and Anderson (1995), to track student knowledge for over a decade Surprisingly, models currently in use do not allow for individual ...
Comments