skip to main content
10.1145/3637528.3671684acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

AdaRD: An Adaptive Response Denoising Framework for Robust Learner Modeling

Published: 24 August 2024 Publication History

Abstract

Learner modeling is a crucial task in online learning environments, where Cognitive Diagnosis Models (CDMs) are employed to assess learners' knowledge mastery levels based on recorded response logs. However, the prevalence of noise in recorded response data poses significant challenges, including various behaviors such as guess and slip, casual answers, and system-induced errors. The existence of noise degrades the accuracy of diagnosis results and learner performance predictions. In this work, we propose a general framework, Adaptive Response Denoising (AdaRD), designed to salvage CDMs from the influence of noisy learner-exercise responses. AdaRD extends existing CDMs, incorporating primary training for denoised CDMs and auxiliary training for additional denoising support. The primary training employs binary Generalized Cross Entropy (GCE) loss to slow down the large update of learner knowledge states caused by noisy responses. Simultaneously, we utilize the variance of diagnosed knowledge mastery levels between primary and auxiliary diagnosis modules as a criterion to downweight high-variance responses that are likely to be noisy. In this manner, the proposed framework can prune noisy response learning during training, thereby enhancing the accuracy and robustness of CDMs. Extensive experiments on both real-world and synthetic datasets validate AdaRD's effectiveness in mitigating the impact of noisy learner-exercise responses.

Supplemental Material

MP4 File - AdaRD: An Adaptive Response Denoising Framework for Robust Learner Modeling
This video briefly introduces our work on "An Adaptive Response Denoising Framework for Robust Learner Modeling?. For more details, please refer to our paper.

References

[1]
Deepak Agarwal, Nishant Babel, and Ryan S Baker. 2018. Contextual Derivation of Stable BKT Parameters for Analysing Content Efficacy. In EDM.
[2]
Scott E Allen, Anna McLean Phillips, and Rene F Kizilcec. 2021. Student perceptions of pre-assessments: ?It's basically just guessing anyways.". In Physics Education Research Conference, 2021: Making Physics More Inclusive and Eliminating Exclusionary Practices in Physics. American Association of Physics Teachers (AAPT), 27--32.
[3]
Winfred Arthur Jr, Ellen Hagen, and Felix George Jr. 2021. The lazy or dishonest respondent: Detection and prevention. Annual Review of Organizational Psychology and Organizational Behavior, Vol. 8 (2021), 105--137.
[4]
Ryan SJ d Baker, Albert T Corbett, and Vincent Aleven. 2008. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. In Intelligent Tutoring Systems: 9th International Conference, ITS 2008, Montreal, Canada, June 23-27, 2008 Proceedings 9. Springer, 406--415.
[5]
Ryan SJ d Baker, Albert T Corbett, Sujith M Gowda, Angela Z Wagner, Benjamin A MacLaren, Linda R Kauffman, Aaron P Mitchell, and Stephen Giguere. 2010. Contextual slip and prediction of student performance after use of an intelligent tutor. In User Modeling, Adaptation, and Personalization: 18th International Conference, UMAP 2010, Big Island, HI, USA, June 20-24, 2010. Proceedings 18. Springer, 52--63.
[6]
Albert T Corbett and John R Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, Vol. 4, 4 (1994), 253--278.
[7]
Jimmy De La Torre. 2009. DINA model and parameter estimation: A didactic. Journal of educational and behavioral statistics, Vol. 34, 1 (2009), 115--130.
[8]
Rob Fergus, Yair Weiss, and Antonio Torralba. 2009. Semi-supervised learning in gigantic image collections. Advances in neural information processing systems, Vol. 22 (2009).
[9]
Weibo Gao, Qi Liu, Zhenya Huang, Yu Yin, Haoyang Bi, Mu-Chun Wang, Jianhui Ma, Shijin Wang, and Yu Su. 2021. Rcd: Relation map driven cognitive diagnosis for intelligent education systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 501--510.
[10]
Weibo Gao, Qi Liu, Hao Wang, Linan Yue, Haoyang Bi, Yin Gu, Fangzhou Yao, Zheng Zhang, Xin Li, and Yuanjing He. 2024. Zero-1-to-3: Domain-Level Zero-Shot Cognitive Diagnosis via One Batch of Early-Bird Students towards Three Diagnostic Objectives. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 8417--8426.
[11]
Weibo Gao, Hao Wang, Qi Liu, Fei Wang, Xin Lin, Linan Yue, Zheng Zhang, Rui Lv, and Shijin Wang. 2023. Leveraging transferable knowledge concept graph embedding for cold-start cognitive diagnosis. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 983--992.
[12]
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.
[13]
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. 2018. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings of the European conference on computer vision (ECCV). 135--150.
[14]
Yan Han, SOUMAVA ROY, Lars Petersson, and Mehrtash Harandi. 2020. Learning from noisy labels via discrepant collaborative training. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 3169--3178.
[15]
Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. 2022. Learning with neighbor consistency for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4672--4681.
[16]
Yogesh Jhamb, Travis Ebesu, and Yi Fang. 2018. Attentive contextual denoising autoencoder for recommendation. In Proceedings of the 2018 ACM SIGIR international conference on theory of information retrieval. 27--34.
[17]
Ömür Kaya Kalkan and. Ismail cCUHADAR. 2020. An evaluation of 4PL IRT and DINA models for estimating pseudo-guessing and slipping parameters. Journal of Measurement and Evaluation in Education and Psychology, Vol. 11, 2 (2020), 131--146.
[18]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[19]
Jiatong Li, Qi Liu, Fei Wang, Jiayu Liu, Zhenya Huang, Fangzhou Yao, Linbo Zhu, and Yu Su. 2024. Towards the Identifiability and Explainability for Personalized Learner Modeling: An Inductive Paradigm. In Proceedings of the ACM on Web Conference 2024. 3420--3431.
[20]
Fangfu Liu, Wenchang Ma, An Zhang, Xiang Wang, Yueqi Duan, and Tat-Seng Chua. 2023. Discovering Dynamic Causal Space for DAG Structure Learning. arXiv preprint arXiv:2306.02822 (2023).
[21]
Qi Liu. 2021. Towards a New Generation of Cognitive Diagnosis. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021. ijcai.org, 4961--4964. https://doi.org/10.24963/ijcai.2021/703
[22]
Qi Liu, Shiwei Tong, Chuanren Liu, Hongke Zhao, Enhong Chen, Haiping Ma, and Shijin Wang. 2019. Exploiting cognitive structure for adaptive learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 627--635.
[23]
Qi Liu, Runze Wu, Enhong Chen, Guandong Xu, Yu Su, Zhigang Chen, and Guoping Hu. 2018. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 9, 4 (2018), 1--26.
[24]
Frederic M Lord. 1980. Applications of item response theory to practical testing problems. Routledge.
[25]
Yueming Lyu and Ivor W Tsang. 2019. Curriculum loss: Robust learning and generalization against label corruption. arXiv preprint arXiv:1905.10045 (2019).
[26]
Eran Malach and Shai Shalev-Shwartz. 2017. Decoupling" when to update" from" how to update". Advances in neural information processing systems, Vol. 30 (2017).
[27]
D Joe Olmi, Robert C Sevier, and Deborah F Nastasi. 1997. Time-in/time-out as a response to noncompliance and inappropriate behavior with children with developmental disabilities: Two case studies. Psychology in the Schools, Vol. 34, 1 (1997), 31--39.
[28]
Zacharoula Papamitsiou and Anastasios A Economides. 2016. Process mining of interactions during computer-based testing for detecting and modelling guessing behavior. In Learning and Collaboration Technologies: Third International Conference, LCT 2016, Held as Part of HCI International 2016, Toronto, ON, Canada, July 17-22, 2016, Proceedings 3. Springer, 437--449.
[29]
Mark D Reckase. 2009. Multidimensional item response theory models. In Multidimensional item response theory. Springer, 79--112.
[30]
Paul R Rosenbaum. 1984. Testing the conditional independence and monotonicity assumptions of item response theory. Psychometrika, Vol. 49, 3 (1984), 425--435.
[31]
Bo Shen and Ang Chen. 2006. Examining the interrelations among knowledge, interests, and learning strategies. Journal of Teaching in Physical Education, Vol. 25, 2 (2006), 182--199.
[32]
Shiwei Tong, Jiayu Liu, Yuting Hong, Zhenya Huang, Le Wu, Qi Liu, Wei Huang, Enhong Chen, and Dan Zhang. 2022. Incremental Cognitive Diagnosis for Intelligent Education. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1760--1770.
[33]
Arash Vahdat. 2017. Toward robustness against label noise in training deep discriminative neural networks. Advances in neural information processing systems, Vol. 30 (2017).
[34]
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. 2017. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition. 839--847.
[35]
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yuying Chen, Yu Yin, Zai Huang, and Shijin Wang. 2020. Neural cognitive diagnosis for intelligent education systems. In Proceedings of the AAAI Conference on Artificial Intelligence. 6153--6161.
[36]
Fei Wang, Qi Liu, Enhong Chen, Zhenya Huang, Yu Yin, Shijin Wang, and Yu Su. 2022. NeuralCD: a general framework for cognitive diagnosis. IEEE Transactions on Knowledge and Data Engineering (2022).
[37]
Wenjie Wang, Fuli Feng, Xiangnan He, Liqiang Nie, and Tat-Seng Chua. 2021. Denoising implicit feedback for recommendation. In Proceedings of the 14th ACM international conference on web search and data mining. 373--381.
[38]
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. 2018. Iterative learning with open-set noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8688--8696.
[39]
Bihan Xu, Zhenya Huang, Jiayu Liu, Shuanghong Shen, Qi Liu, Enhong Chen, Jinze Wu, and Shijin Wang. 2023. Learning behavior-oriented knowledge tracing. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2789--2800.
[40]
Fangzhou Yao, Qi Liu, Min Hou, Shiwei Tong, Zhenya Huang, Enhong Chen, Jing Sha, and Shijin Wang. 2023. Exploiting non-interactive exercises in cognitive diagnosis. Interaction, Vol. 100, 200 (2023), 300.
[41]
Chun-Kit Yeung and Dit-Yan Yeung. 2018. Addressing two problems in deep knowledge tracing via prediction-consistent regularization. In Proceedings of the fifth annual ACM conference on learning at scale. 1--10.
[42]
Linan Yue, Qi Liu, Yichao Du, Yanqing An, Li Wang, and Enhong Chen. 2022. DARE: disentanglement-augmented rationale extraction. Advances in Neural Information Processing Systems, Vol. 35 (2022), 26603--26617.
[43]
Linan Yue, Qi Liu, Ye Liu, Weibo Gao, Fangzhou Yao, and Wenfeng Li. 2024. Cooperative classification and rationalization for graph generalization. In Proceedings of the ACM on Web Conference 2024. 344--352.
[44]
Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, Vol. 31 (2018).
[45]
Zhedong Zheng and Yi Yang. 2021. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision, Vol. 129, 4 (2021), 1106--1120.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 2024
6901 pages
ISBN:9798400704901
DOI:10.1145/3637528
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptive denoising
  2. cognitive diagnosis
  3. learner modeling
  4. noisy responses

Qualifiers

  • Research-article

Funding Sources

Conference

KDD '24
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 150
    Total Downloads
  • Downloads (Last 12 months)150
  • Downloads (Last 6 weeks)18
Reflects downloads up to 20 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media