Skip to main content
SpringerLink
Log in

A Probabilistic Framework for Temporal Cognitive Diagnosis in Online Learning Systems

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Cognitive diagnosis is an important issue of intelligent education systems, which aims to estimate students’ proficiency on specific knowledge concepts. Most existing studies rely on the assumption of static student states and ignore the dynamics of proficiency in the learning process, which makes them unsuitable for online learning scenarios. In this paper, we propose a unified temporal item response theory (UTIRT) framework, incorporating temporality and randomness of proficiency evolving to get both accurate and interpretable diagnosis results. Specifically, we hypothesize that students’ proficiency varies as a Wiener process and describe a probabilistic graphical model in UTIRT to consider temporality and randomness factors. Furthermore, based on the relationship between student states and exercising answers, we hypothesize that the answering result at time k contributes most to inferring a student's proficiency at time k, which also reflects the temporality aspect and enables us to get analytical maximization (M-step) in the expectation maximization (EM) algorithm when estimating model parameters. Our UTIRT is a framework containing unified training and inferencing methods, and is general to cover several typical traditional models such as Item Response Theory (IRT), multidimensional IRT (MIRT), and temporal IRT (TIRT). Extensive experimental results on real-world datasets show the effectiveness of UTIRT and prove its superiority in leveraging temporality theoretically and practically over TIRT.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Guo X, Li R, Yu Q, Haake A R. Modeling physicians’ utterances to explore diagnostic decision-making. In Proc. the 26th International Joint Conference on Artificial Intelligence, Aug. 2017, pp.3700–3706. DOI: 10.24963/ijcai.2017/517.

  2. Yao C L, Qu Y, Jin B, Guo L, Li C, Cui W J, Feng L. A convolutional neural network model for online medical guidance. IEEE Access, 2016, 4: 4094–4103. DOI: https://doi.org/10.1109/ACCESS.2016.2594839.

    Article  Google Scholar 

  3. Chen S, Joachims T. Predicting matchups and preferences in context. In Proc. the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Aug. 2016, pp.775–784. DOI: 10.1145/2939672.2939764.

  4. Wang F, Liu Q, Chen E H, Huang Z Y, Chen Y Y, Yin Y, Huang Z, Wang S J. Neural cognitive diagnosis for intelligent education systems. In Proc. the 34th AAAI Conference on Artificial Intelligence, Feb. 2020, pp.6153–6161. DOI: 10.1609/aaai.v34i04.6080.

  5. Kuh G D, Kinzie J, Buckley J, Bridges B K, Hayek J. Piecing together the student success puzzle: Research, propositions, and recommendations. ASHE Higher Education Report, 2007, 32(5): 1–182. DOI: https://doi.org/10.1002/aehe.3205.

    Article  Google Scholar 

  6. de la Torre J. Dina model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 2009, 34(1): 115–130. DOI: https://doi.org/10.3102/1076998607309474.

    Article  Google Scholar 

  7. Embretson S E, Reise S P. Item Response Theory. Psychology Press, 2013.

  8. Adams R J, Wilson M, Wang W C. The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 1997, 21(1): 1–23. DOI: https://doi.org/10.1177/0146621697211001.

    Article  Google Scholar 

  9. Wilson K H, Karklin Y, Han B J, Ekanadham C. Back to the basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation. arXiv: 1604.02336, 2016. https://arxiv.org/abs/1604.02336, Nov. 2023.

  10. Tatsuoka K K, Tatsuoka M M. Computerized cognitive diagnostic adaptive testing: Effect on remedial instruction as empirical validation. Journal of Educational Measurement, 1997, 34(1): 3–20. DOI: https://doi.org/10.1111/j.1745-3984.1997.tb00504.x.

    Article  Google Scholar 

  11. Leighton J P, Gierl M J, Hunka S M. The attribute hierarchy method for cognitive assessment: A variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 2004, 41(3): 205–237. DOI: https://doi.org/10.1111/j.1745-3984.2004.tb01163.x.

    Article  Google Scholar 

  12. Thai-Nghe N, Horváth T, Schmidt-Thieme L. Factorization models for forecasting student performance. In Proc. the 3rd International Conference on Educational Data Mining, June 2010, pp.11–20.

  13. Wang X J, Berger J O, Burdick D S. Bayesian analysis of dynamic item response models in educational testing. The Annals of Applied Statistics, 2013, 7(1): 126–153. DOI: https://doi.org/10.1214/12-AOAS608.

    Article  MathSciNet  Google Scholar 

  14. Anzanello M J, Fogliatto F S. Learning curve models and applications: Literature review and research directions. International Journal of Industrial Ergonomics, 2011, 41(5): 573–583. DOI: https://doi.org/10.1016/j.ergon.2011.05.001.

    Article  Google Scholar 

  15. Averell L, Heathcote A. The form of the forgetting curve and the fate of memories. Journal of Mathematical Psychology, 2011, 55(1): 25–35. DOI: https://doi.org/10.1016/j.jmp.2010.08.009.

    Article  MathSciNet  Google Scholar 

  16. Ebbinghaus H. Memory: A contribution to experimental psychology. Annals of Neurosciences, 2013, 20(4): 155–156. DOI: https://doi.org/10.5214/ans.0972.7531.200408.

    Article  Google Scholar 

  17. Malliaris A G. Wiener process. In Time Series and Statistics, Eatwell J, Milgate M, Newman P (eds.), Springer, 1990, pp.316–318. DOI: https://doi.org/10.1007/978-1-349-20865-4_43.

  18. Liu B B, Dong W, Liu J X, Zhang Y T, Wang D Y. ProSy: API-based synthesis with probabilistic model. Journal of Computer Science and Technology, 2020, 35(6): 1234–1257. DOI: https://doi.org/10.1007/s11390-020-0520-4.

    Article  Google Scholar 

  19. Qiang Y T, Fu Y W, Yu X, Guo Y W, Zhou Z H, Sigal L. Learning to generate posters of scientific papers by probabilistic graphical models. Journal of Computer Science and Technology, 2019, 34(1): 155–169. DOI: https://doi.org/10.1007/s11390-019-1904-1.

    Article  Google Scholar 

  20. Zhang Q. Dynamic uncertain causality graph for knowledge representation and reasoning: Discrete dag cases. Journal of Computer Science and Technology, 2012, 27(1): 1–23. DOI: https://doi.org/10.1007/s11390-012-1202-7.

    Article  MathSciNet  Google Scholar 

  21. Leighton J P, Gierl M J. Cognitive Diagnostic Assessment for Education: Theory and Applications. Cambridge University Press, 2007.

  22. Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. MESA Press, 1993.

  23. Khajah M, Wing R M, Lindsey R V, Mozer M C. Integrating latent-factor and knowledge-tracing models to predict individual differences in learning. In Proc. the 7th International Conference on Educational Data Mining, Jul. 2014, pp.99–106.

  24. Ekanadham C, Karklin Y. T-SKIRT: Online estimation of student proficiency in an adaptive learning system. arXiv: 1702.04282, 2017. https://arxiv.org/abs/1702.04282, Nov. 2023.

  25. Huang Z Y, Liu Q, Chen Y Y et al. Learning or forgetting? A dynamic approach for tracking the knowledge proficiency of students. ACM Trans. Information Systems, 2020, 38(2): 1–33. DOI: https://doi.org/10.1145/3379507.

    Article  Google Scholar 

  26. Cen H, Koedinger K, Junker B. Learning factors analysis—A general method for cognitive model evaluation and improvement. In Proc. the 8th International Conference on Intelligent Tutoring Systems, Jun. 2006, pp.164–175. DOI: 10.1007/11774303_17.

  27. Pavlik P I, Cen H, Koedinger K R. Performance factors analysis—A new alternative to knowledge tracing. In Proc. the 14th International Conference on Artificial Intelligence in Education, Jul. 2009. DOI: 10.3233/978-1-60750-028-5-531.

  28. Elo A E. The Rating of Chess Players, Past and Present. Arco Pub, 1978.

  29. Pelánek R. Application of time decay functions and the elo system in student modeling. In Proc. the 7th International Conference on Educational Data Mining, Jul. 2014, pp.21–27.

  30. Nižnan J, Pelánek R, Rihák J. Student models for prior knowledge estimation. In Proc. the 8th International Conference on Educational Data Mining, Jun. 2015, pp.109–116.

  31. Pelánek R, Papoušek J, Řihák J, Stanislav V, Nižnan J. Elo-based learner modeling for the adaptive practice of facts. User Modeling and User-Adapted Interaction, 2017, 27(1): 89–118. DOI: https://doi.org/10.1007/s11257-016-9185-7.

    Article  Google Scholar 

  32. Yudelson M. Individualization of Bayesian knowledge tracing through Elo-infusion. In Proc. the 22nd International Conference on Artificial Intelligence in Education, Jun. 2021, pp.412–416. DOI: 10.1007/978-3-030-78270-2_73.

  33. Kaya Y, Leite W L. Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 2017, 77(3): 369–388. DOI: https://doi.org/10.1177/0013164416659314.

    Article  Google Scholar 

  34. Zhan P D, Jiao H, Liao D D, Li F M. A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 2019, 44(3): 251–281. DOI: https://doi.org/10.3102/1076998619827593.

    Article  Google Scholar 

  35. Pan Q Q, Qin L, Kingston N. Growth modeling in a diagnostic classification model (DCM) framework—A multivariate longitudinal diagnostic classification model. Frontiers in Psychology, 2020, 11: 1714. DOI: https://doi.org/10.3389/fpsyg.2020.01714.

    Article  Google Scholar 

  36. Zhan P D, He K R. A longitudinal diagnostic model with hierarchical learning trajectories. Educational Measurement: Issues and Practice, 2021, 40(3): 18–30. DOI: https://doi.org/10.1111/emip.12422.

    Article  Google Scholar 

  37. Zhan P D. Longitudinal learning diagnosis: Minireview and future research directions. Frontiers in Psychology, 2020, 11: 1185. DOI: https://doi.org/10.3389/fpsyg.2020.01185.

    Article  Google Scholar 

  38. Corbett A T, Anderson J R. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 1994, 4(4): 253–278. DOI: https://doi.org/10.1007/BF01099821.

    Article  Google Scholar 

  39. González-Brenes J, Huang Y, Brusilovsky P. General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge. In Proc. the 7th International Conference on Educational Data Mining, Jul. 2014, pp.84–91.

  40. Käser T, Klingler S, Schwing A G, Gross M. Dynamic Bayesian networks for student modeling. IEEE Trans. Learning Technologies, 2017, 10(4): 450–462. DOI: https://doi.org/10.1109/TLT.2017.2689017.

    Article  Google Scholar 

  41. Pardos Z A, Heffernan N T. KT-IDEM: Introducing item difficulty to the knowledge tracing model. In Proc. the 19th International Conference on User Modeling, Adaptation, and Personalization, Jul. 2011, pp.243–254. DOI: 10.1007/978-3-642-22362-4_21.

  42. Thaker K, Huang Y, Brusilovsky P, He D Q. Dynamic knowledge modeling with heterogeneous activities for adaptive textbooks. In Proc. the 11th International Conference on Educational Data Mining, Jul. 2018, pp.592–595.

  43. Yudelson M V, Koedinger K R, Gordon G J. Individualized Bayesian knowledge tracing models. In Proc. the 16th International Conference on Artificial Intelligence in Education, Jul. 2013, pp.171–180. DOI: 10.1007/978-3-642-39112-5_18.

  44. Liu Q, Huang Z Y, Yin Y, Chen E H, Xiong H, Su Y, Hu G P. EKT: Exercise-aware knowledge tracing for student performance prediction. IEEE Trans. Knowledge and Data Engineering, 2019, 33(1): 100–115. DOI: https://doi.org/10.1109/TKDE.2019.2924374.

    Article  Google Scholar 

  45. Pardos Z A, Heffernan N T. Modeling individualization in a Bayesian networks implementation of knowledge tracing. In Proc. the 18th International conference on User Modeling, Adaptation, and Personalization, Jun. 2010, pp.255–266. DOI: 10.1007/978-3-642-13470-8_24.

  46. Pardos Z A, Heffernan N T. Using HMMs and bagged decision trees to leverage rich features of user and skill from an intelligent tutoring system dataset. Journal of Machine Learning Research W&CP, 201040. https://people.csail.mit.edu/zp/papers/pardos_JMLR_in_press.pdf, Nov. 2023.

  47. Piech C, Spencer J, Huang J, Ganguli S, Sahami M, Guibas L, Sohl-Dickstein J. Deep knowledge tracing. arXiv: 1506.05908, 2015. https://arxiv.org/abs/1506.05908, Nov. 2023.

  48. Zhang J N, Shi X J, King I, Yeung D Y. Dynamic keyvalue memory networks for knowledge tracing. In Proc. the 26th International Conference on World Wide Web, Apr. 2017, pp.765–774. DOI: 10.1145/3038912.3052580.

  49. Shen S H, Liu Q, Chen E H, Huang Z Y, Huang W, Yin Y, Su Y, Wang S J. Learning process-consistent knowledge tracing. In Proc. the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, Aug. 2021, pp.1452–1460. DOI: 10.1145/3447548.3467237.

  50. Huang T, Yang H L, Li Z, Xie H K, Geng J, Zhang H. A dynamic knowledge diagnosis approach integrating cognitive features. IEEE Access, 2021, 9: 116814–116829. DOI: https://doi.org/10.1109/ACCESS.2021.3105830.

    Article  Google Scholar 

  51. Lu Y, Wang D L, Meng Q G, Chen P H. Towards interpretable deep learning models for knowledge tracing. In Proc. the 21st International Conference on Artificial Intelligence in Education, Jul. 2020, pp.185–190. DOI: 10.1007/978-3-030-52240-7_34.

  52. Pardos Z A, Bergner Y, Seaton D T, Pritchard D E. Adapting Bayesian knowledge tracing to a massive open online course in edX. In Proc. the 6th International Conference on Educational Data Mining, Jul. 2013, pp.137–144.

  53. Johnson M J. Scaling cognitive modeling to massive open environments. In Proc. the ICML Workshop on Machine Learning in Education, Jul. 2015. http://ml4ed.cc/attachments/XuY.pdf, Nov. 2023.

  54. Ruder S. An overview of gradient descent optimization algorithms. arXiv: 1609.04747, 2016. https://arxiv.org/abs/1609.04747, Nov. 2023.

  55. Bock R D, Aitkin M. Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 1981, 46(4): 443–459. DOI: https://doi.org/10.1007/BF02293801.

    Article  MathSciNet  Google Scholar 

  56. Segall D O. Multidimensional adaptive testing. Psychometrika, 1996, 61(2): 331–354. DOI: https://doi.org/10.1007/BF02294343.

    Article  Google Scholar 

  57. Feng M Y, Heffernan N, Koedinger K. Addressing the assessment challenge with an online system that tutors as it assesses. User Modeling and User-Adapted Interaction, 2009, 19(3): 243–266. DOI: https://doi.org/10.1007/s11257-009-9063-7.

    Article  Google Scholar 

  58. Chang H S, Hsu H J, Chen K T. Modeling exercise relationships in E-learning: A unified approach. In Proc. the 8th International Conference on Educational Data Mining, Jun. 2015, pp.532–535.

  59. Yang H Q, Cheung L P. Implicit heterogeneous features embedding in deep knowledge tracing. Cognitive Computation, 2018, 10(1): 3–14. DOI: https://doi.org/10.1007/s12559-017-9522-0.

    Article  Google Scholar 

  60. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In Proc. the 13th International Conference on Artificial Intelligence and Statistics, May 2010, pp.249–256.

  61. Liu Q, Wu R Z, Chen E H, Xu G D, Su Y, Chen Z G, Hu G P. Fuzzy cognitive diagnosis for modelling examinee performance. ACM Trans. Intelligent Systems and Technology, 2018, 9(4): 1–26. DOI: https://doi.org/10.1145/3168361.

    Article  Google Scholar 

  62. Jang E E. A *validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL [Ph. D. Thesis]. University of Illinois at Urbana-Champaign, Champagne, 2005.

  63. Gervet T, Koedinger K, Schneider J, Mitchell T. When is deep learning the best approach to knowledge tracing?. Journal of Educational Data Mining, 2020, 12(3): 31–54. DOI: https://doi.org/10.5281/zenodo.4143614.

    Article  Google Scholar 

  64. Hodges J L. The significance probability of the Smirnov two-sample test. Arkiv för Matematik, 1958, 3(5): 469–486. DOI: https://doi.org/10.1007/BF02589501.

    Article  MathSciNet  Google Scholar 

  65. Wu R Z, Xu G D, Chen E H, Liu Q, Ng W. Knowledge or gaming?: Cognitive modelling based on multiple-attempt response. In Proc. the 26th International Conference on World Wide Web Companion, Apr. 2017, pp.321–329. DOI: 10.1145/3041021.3054156.

  66. Zhao X, Zhang J J, Li W S, Kahn K, Lu Y, Winters N. Learners’ non-cognitive skills and behavioral patterns of programming: A sequential analysis. In Proc. the 21st International Conference on Advanced Learning Technologies, Jul. 2021, pp.168–172. DOI: 10.1109/ICALT52272.2021.00058.

  67. Jiang L, Wang P Y, Cheng K, Liu K P, Yin M H, Jin B, Fu Y J. EduHawkes: A neural Hawkes process approach for online study behavior modeling. In Proc. the 2021 SIAM International Conference on Data Mining, Apr. 2021, pp.567–575. DOI: 10.1137/1.9781611976700.64.

  68. Zhang H, Huang T, Liu S Y, Yin H, Li J, Yang H L, Xia Y. A learning style classification approach based on deep belief network for large-scale online education. Journal of Cloud Computing, 2020, 9(1): 1–17. DOI: https://doi.org/10.1186/s13677-020-00165-y.

    Article  Google Scholar 

  69. Chen Y X, Li X O, Liu J C, Ying Z L. Recommendation system for adaptive learning. Applied Psychological Measurement, 2018, 42(1): 24–41. DOI: https://doi.org/10.1177/0146621617697959.

    Article  Google Scholar 

  70. Dang F R, Tang J T, Pang K Y, Wang T, Li S S, Li X. Constructing an educational knowledge graph with concepts linked to Wikipedia. Journal of Computer Science and Technology, 2021, 36(5): 1200–1211. DOI: https://doi.org/10.1007/s11390-020-0328-2.

    Article  Google Scholar 

  71. Zhu J Z, Jia Y T, Xu J, Qiao J Z, Cheng X Q. Modeling the correlations of relations for knowledge graph embedding. Journal of Computer Science and Technology, 2018, 33(2): 323–334. DOI: https://doi.org/10.1007/s11390-018-1821-8.

    Article  MathSciNet  Google Scholar 

  72. Nakagawa H, Iwasawa Y, Matsuo Y. Graph-based knowledge tracing: Modeling student proficiency using graph neural network. In Proc. the 2019 IEEE/WIC/ACM International Conference on Web Intelligence, Oct. 2019, pp.156–163. DOI: 10.1145/3350546.3352513.

  73. Chen C H, Liu G Z, Hwang G J. Interaction between gaming and multistage guiding strategies on students’ field trip mobile learning performance and motivation. British Journal of Educational Technology, 2016, 47(6): 1032–1050. DOI: https://doi.org/10.1111/bjet.12270.

    Article  Google Scholar 

  74. Hwang G J, Wang S Y. Single loop or double loop learning: English vocabulary learning performance and behavior of students in situated computer games with different guiding strategies. Computers & Education, 2016, 102: 188–201. DOI: https://doi.org/10.1016/j.compedu.2016.07.005.

    Article  Google Scholar 

  75. Chen S Y, Yeh C C. The effects of cognitive styles on the use of hints in academic English: A learning analytics approach. Educational Technology & Society, 2017, 20(2): 251–264.

    Google Scholar 

  76. Muir M, Conati C. Understanding student attention to adaptive hints with eye-tracking. In Proc. the 19th International Conference on Advances in User Modeling, Jul. 2011, pp.148–160. DOI: 10.1007/978-3-642-28509-7_15.

  77. Wang Y T, Heffernan N T. The “assistance” model: Leveraging how many hints and attempts a student needs. In Proc. the 24th International Florida Artificial Intelligence Research Society Conference, May 2011.

Download references

Author information

Authors and Affiliations

  1. School of Data Science, University of Science and Technology of China, Hefei, 230026, China

    Jia-Yu Liu

  2. State Key Laboratory of Cognitive Intelligence, Hefei, 230088, China

    Jia-Yu Liu, Fei Wang, Zhen-Ya Huang, Qi Liu & En-Hong Chen

  3. Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China, Hefei, 230026, China

    Jia-Yu Liu, Fei Wang, Zhen-Ya Huang, Qi Liu & En-Hong Chen

  4. School of Computer Science and Technology, University of Science and Technology of China, Hefei, 230026, China

    Fei Wang, Zhen-Ya Huang & Qi Liu

  5. Institutes of Physical Science and Information Technology, Anhui University, Hefei, 230601, China

    Hai-Ping Ma

  6. School of Computer Science and Artificial Intelligence, Hefei Normal University, Hefei, 230061, China

    Yu Su

Authors
  1. Jia-Yu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hai-Ping Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhen-Ya Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. En-Hong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yu Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hai-Ping Ma.

Supplementary Information

ESM 1

(PDF 209 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, JY., Wang, F., Ma, HP. et al. A Probabilistic Framework for Temporal Cognitive Diagnosis in Online Learning Systems. J. Comput. Sci. Technol. 38, 1203–1222 (2023). https://doi.org/10.1007/s11390-022-1332-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-022-1332-5

Keywords

Search

Navigation