skip to main content
10.1145/3636555.3636936acmotherconferencesArticle/Chapter ViewAbstractPublication PageslakConference Proceedingsconference-collections
research-article

Contexts Matter but How? Course-Level Correlates of Performance and Fairness Shift in Predictive Model Transfer

Published: 18 March 2024 Publication History

Abstract

Learning analytics research has highlighted that contexts matter for predictive models, but little research has explicated how contexts matter for models’ utility. Such insights are critical for real-world applications where predictive models are frequently deployed across instructional and institutional contexts. Building upon administrative records and behavioral traces from 37,089 students across 1,493 courses, we provide a comprehensive evaluation of performance and fairness shifts of predictive models when transferred across different course contexts. We specifically quantify how differences in various contextual factors moderate model portability. Our findings indicate an average decline in model performance and inconsistent directions in fairness shifts, without a direct trade-off, when models are transferred across different courses within the same institution. Among the course-to-course contextual differences we examined, differences in admin features account for the largest portion of both performance and fairness loss. Differences in student composition can simultaneously amplify drops in performance and fairness while differences in learning design have a greater impact on performance degradation. Given these complexities, our results highlight the importance of considering multiple dimensions of course contexts and evaluating fairness shifts in addition to performance loss when conducting transfer learning of predictive models in education.

References

[1]
Eyman Alyahyan and Dilek Düştegör. 2020. Predicting academic success in higher education: literature review and best practices. International Journal of Educational Technology in Higher Education 17 (2020), 1–21.
[2]
Jose Luis Arroyo-Barrigüete, Susana Carabias-López, Tomas Curto-González, and Adolfo Hernández. 2021. Portability of Predictive Academic Performance Models: An Empirical Sensitivity Analysis. Mathematics 9, 8 (2021), 870.
[3]
Jean A. Baker, Teresa P. Clark, Kimberly S. Maier, and Steve Viger. 2008. The differential influence of instructional context on the academic engagement of students with behavior problems. Teaching and Teacher Education 24, 7 (2008), 1876–1883. https://doi.org/10.1016/j.tate.2008.02.019
[4]
Ryan S. Baker. 2019. Challenges for the Future of Educational Data Mining: The Baker Learning Analytics Prizes. (6 2019).
[5]
Ryan S Baker and Aaron Hawn. 2021. Algorithmic bias in education. International Journal of Artificial Intelligence in Education (2021), 1–41.
[6]
Rebeca Cerezo, Miguel Sánchez-Santillán, M. Puerto Paule-Ruiz, and J. Carlos Núñez. 2016. Students’ LMS interaction patterns and their relationship with achievement: A case study in higher education. Computers & Education 96 (5 2016), 42–54. https://doi.org/10.1016/J.COMPEDU.2016.02.006
[7]
Rianne Conijn, Chris Snijders, Ad Kleingeld, and Uwe Matzat. 2016. Predicting student performance from LMS data: A comparison of 17 blended courses using Moodle LMS. IEEE Transactions on Learning Technologies 10, 1 (2016), 17–29.
[8]
Kimberlé Crenshaw. 2013. Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory and antiracist politics. In Feminist legal theories. Routledge, 23–51.
[9]
Shane Dawson, Erica Mcwilliam, and Jennifer Pei-Ling Tan. 2008. Teaching smarter: how mining ICT data can inform and improve learning and teaching practice. https://api.semanticscholar.org/CorpusID:61176352
[10]
Oscar Blessed Deho, Srecko Joksimovic, Lin Liu, Jiuyong Li, Chen Zhan, and Jixue Liu. 2023. Assessing the Fairness of Course Success Prediction Models in the Face of (Un) equal Demographic Group Distribution. In Proceedings of the Tenth ACM Conference on Learning@ Scale. 48–58.
[11]
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference. 214–226.
[12]
Nafsaniath Fathema and Mohammad H Akanda. 2020. Effects of instructors’ academic disciplines and prior experience with learning management systems: A study about the use of Canvas. Australasian Journal of Educational Technology 36, 4 (Jan. 2020), 113–125. https://doi.org/10.14742/ajet.5660
[13]
Josh Gardner, Christopher Brooks, and Ryan Baker. 2019. Evaluating the fairness of predictive student models through slicing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge. 225–234.
[14]
Joshua Gardner, Renzhe Yu, Quan Nguyen, Christopher Brooks, and Rene Kizilcec. 2023. Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1664–1684.
[15]
Dragan Gašević, Shane Dawson, Tim Rogers, and Danijela Gasevic. 2016. Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. Internet and Higher Education 28 (1 2016), 68–84.
[16]
Niki Gitinabard, Yiqiao Xu, Sarah Heckman, Tiffany Barnes, and Collin F Lynch. 2019. How widely can prediction models be generalized? Performance prediction in blended courses. IEEE Transactions on Learning Technologies 12, 2 (2019), 184–197.
[17]
Usman Gohar and Lu Cheng. 2023. A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges. arXiv preprint arXiv:2305.06969 (2023).
[18]
Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016).
[19]
Joon Heo, Hyoungjoon Lim, Sung Bum Yun, Sungha Ju, Sangyoon Park, Rebekah Lee, J Heo, H Lim, S B Yun, and S Ju. 2019. Descriptive and Predictive Modeling of Student Achievement, Satisfaction, and Mental Health for Data-Driven Smart Connected Campus Life Service. (2019).
[20]
Jelena Jovanovic, Dragan Gasevic, Shane Dawson, Abelardo Pardo, and Negin Mirriahi. 2017. Learning analytics to unveil learning strategies in a flipped classroom. Internet and Higher Education 33 (2017), 74–85.
[21]
Jelena Jovanović, Mohammed Saqr, Srećko Joksimović, and Dragan Gašević. 2021. Students matter the most in learning analytics: The effects of internal and instructional conditions in predicting academic success. Computers & Education 172 (2021), 104251.
[22]
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In International conference on machine learning. PMLR, 2564–2572.
[23]
René F Kizilcec and Hansol Lee. 2022. Algorithmic fairness in education. In The ethics of artificial intelligence in education. Routledge, 174–202.
[24]
Vitomir Kovanović, Dragan Gašević, Srećko Joksimović, Marek Hatala, and Olusola Adesope. 2015. Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education 27 (2015), 74–89.
[25]
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual fairness. Advances in neural information processing systems 30 (2017).
[26]
Jinsook Lee, Chris Brooks, Renzhe Yu, and Rene Kizilcec. 2023. Fairness Hub Technical Briefs: AUC Gap. arXiv e-prints (2023), arXiv–2309.
[27]
Liang Yi Li and Chin Chung Tsai. 2017. Accessing online learning material: Quantitative behavior patterns and their effects on motivation and learning performance. Computers and Education 114 (11 2017), 286–297. https://doi.org/10.1016/J.COMPEDU.2017.07.007
[28]
Martín Liz Domínguez, Manuel Caeiro Rodríguez, Martín Llamas Nistal, Fernando Ariel Mikic Fonte, 2019. Predictors and early warning systems in higher education: A systematic literature review. Learning Analytics Summer Institute Spain 2019 (LASI-SPAIN 2019), Vigo, España, 27-28 junio 2019 (2019).
[29]
Javier López-Zambrano, Juan A Lara, and Cristóbal Romero. 2022. Improving the portability of predicting students’ performance models by using ontologies. Journal of computing in higher education (2022), 1–19.
[30]
Javier López-Zambrano, Juan A. Lara, and Cristóbal Romero. 2020. Towards Portability of Models for Predicting Students’ Final Performance in University Courses Starting from Moodle Logs. Applied Sciences 2020, Vol. 10, Page 354 10 (1 2020), 354. Issue 1. https://doi.org/10.3390/APP10010354
[31]
Leah P. Macfadyen and Shane Dawson. 2010. Mining LMS data to develop an "early warning system" for educators: A proof of concept. Computers and Education 54 (2 2010), 588–599. Issue 2. https://doi.org/10.1016/J.COMPEDU.2009.09.008
[32]
Pedro Manuel Moreno-Marcos, Tinne De Laet, Pedro J Muñoz-Merino, Carolien Van Soom, Tom Broos, Katrien Verbert, and Carlos Delgado Kloos. 2019. Generalizing predictive models of admission test success based on online interactions. Sustainability 11, 18 (2019), 4940.
[33]
Hardt Moritz, Price Eric, Srebro Nati, DD Lee, M Sugiyama, UV Luxburg, I Guyon, and R Garnett. 2016. Equality of opportunity in supervised learning. Advances in neural information processing systems 29 (2016), 3315–3323.
[34]
Cedric Bheki Mpungose and Simon Bheki Khoza. 2022. Postgraduate Students’ Experiences on the Use of Moodle and Canvas Learning Management System. Technology, Knowledge and Learning 27, 1 (mar 2022), 1–16.
[35]
Abelardo Pardo, Feifei Han, and Robert A. Ellis. 2017. Combining University student self-regulated learning indicators and engagement with online learning events to Predict Academic Performance. IEEE Transactions on Learning Technologies 10 (1 2017), 82–92. Issue 1. https://doi.org/10.1109/TLT.2016.2639508
[36]
Bardh Prenkaj, Paola Velardi, Giovanni Stilo, Damiano Distante, and Stefano Faralli. 2020. A survey of machine learning approaches for student dropout prediction in online courses. ACM Computing Surveys (CSUR) 53, 3 (2020), 1–34.
[37]
J. Rhode, Stephanie Richter, Peter Gowen, T. Miller, and C. Wills. 2017. Understanding faculty use of the learning management system. Online Learning Journal 21 (01 2017), 68–86. https://doi.org/10.24059/olj.v%vi%i.1217
[38]
Tue Tjur. 2009. Coefficients of determination in logistic regression models—A new proposal: The coefficient of discrimination. The American Statistician 63, 4 (2009), 366–372.
[39]
U.S. Equal Employment Opportunity Commission. 2018. 29 CFR Part 1607 - Uniform Guidelines on Employee Selection Procedures (1978).
[40]
Jonathan Vasquez Verdugo, Xavier Gitiaux, Cesar Ortega, and Huzefa Rangwala. 2022. Faired: A systematic fairness analysis approach applied in a higher educational context. In LAK22: 12th International Learning Analytics and Knowledge Conference. 271–281.
[41]
Richard Joseph Waddington and Sungjin Nam. 2014. Practice Exams Make Perfect: Incorporating Course Resource Use into an Early Warning System. (2014). https://doi.org/10.1145/2567574.2567623
[42]
Feng Hsu Wang. 2017. An exploration of online behaviour engagement and achievement in flipped classroom supported by learning management system. Computers & Education 114 (11 2017), 79–91. https://doi.org/10.1016/J.COMPEDU.2017.06.012
[43]
William J Youden. 1950. Index for rating diagnostic tests. Cancer 3, 1 (1950), 32–35.
[44]
Renzhe Yu, Qiujie Li, Christian Fischer, Shayan Doroudi, and Di Xu. 2020. Towards Accurate and Fair Prediction of College Success: Evaluating Different Sources of Student Data. Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), 292–301. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_194.pdf
[45]
Nick Z. Zacharis. 2015. A multivariate approach to predicting student outcomes in web-enabled blended learning courses. The Internet and Higher Education 27 (10 2015), 44–53. https://doi.org/10.1016/J.IHEDUC.2015.05.002
[46]
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. 2013. Learning fair representations. In International conference on machine learning. PMLR, 325–333.

Cited By

View all
  • (2025)Towards Fair and Privacy-Aware Transfer Learning for Educational Predictive Modeling: A Case Study on Retention Prediction in Community CollegesProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706567(738-749)Online publication date: 3-Mar-2025
  • (2025)ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around InterpretationProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706498(837-843)Online publication date: 3-Mar-2025
  • (2025)The lack of generalisability in learning analytics research: why, how does it matter, and where to?Proceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706489(170-180)Online publication date: 3-Mar-2025

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
LAK '24: Proceedings of the 14th Learning Analytics and Knowledge Conference
March 2024
962 pages
ISBN:9798400716188
DOI:10.1145/3636555
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Algorithmic Fairness
  2. Higher Education
  3. Intersectionality
  4. Learning Management System
  5. Predictive Analytics
  6. Transfer Learning

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

LAK '24

Acceptance Rates

Overall Acceptance Rate 236 of 782 submissions, 30%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)209
  • Downloads (Last 6 weeks)14
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Towards Fair and Privacy-Aware Transfer Learning for Educational Predictive Modeling: A Case Study on Retention Prediction in Community CollegesProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706567(738-749)Online publication date: 3-Mar-2025
  • (2025)ABROCA Distributions For Algorithmic Bias Assessment: Considerations Around InterpretationProceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706498(837-843)Online publication date: 3-Mar-2025
  • (2025)The lack of generalisability in learning analytics research: why, how does it matter, and where to?Proceedings of the 15th International Learning Analytics and Knowledge Conference10.1145/3706468.3706489(170-180)Online publication date: 3-Mar-2025
  • (2024)Using Keystroke Behavior Patterns to Detect Nonauthentic Texts in Writing Assessments: Evaluating the Fairness of Predictive ModelsJournal of Educational Measurement10.1111/jedm.12416Online publication date: 18-Oct-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media