Skip to main content
Springer Nature Link
Log in

Option Tracing: Beyond Correctness Analysis in Knowledge Tracing

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12748))

Included in the following conference series:

  • 3724 Accesses

Abstract

Knowledge tracing refers to a family of methods that estimate each student’s knowledge component/skill mastery level from their past responses to questions. One key limitation of most existing knowledge tracing methods is that they can only estimate an overall knowledge level of a student per knowledge component/skill since they analyze only the (usually binary-valued) correctness of student responses. Therefore, it is hard to use them to diagnose specific student errors. In this paper, we extend existing knowledge tracing methods beyond correctness prediction to the task of predicting the exact option students select in multiple choice questions. We quantitatively evaluate the performance of our option tracing methods on two large-scale student response datasets. We also qualitatively evaluate their ability in identifying common student errors in the form of clusters of incorrect options across different questions that correspond to the same error.

This work is supported by the National Science Foundation under grant IIS-1917713. We also thank the reviewers for their constructive feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://eedi.com/projects/neurips-education-challenge.

  2. 2.

    https://github.com/riiid/ednet.

References

  1. Adams, D.M., et al.: Using erroneous examples to improve mathematics learning with a web-based tutoring system. Comput. Hum. Behav. 36, 401–411 (2014)

    Article  Google Scholar 

  2. Anderson, J.R., Jeffries, R.: Novice LISP errors: undetected losses of information from working memory. Hum.-Comput. Interact. 1(2), 107–131 (1985)

    Article  Google Scholar 

  3. Berg, R., Kipf, T.N., Welling, M.: Graph convolutional matrix completion. arXiv preprint arXiv:1706.02263 (2017)

  4. Brown, J.S., Burton, R.R.: Diagnostic models for procedural bugs in basic mathematical skills. Cogn. Sci. 2(2), 155–192 (1978)

    Article  Google Scholar 

  5. Cen, H., Koedinger, K., Junker, B.: Learning factors analysis – a general method for cognitive model evaluation and improvement. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 164–175. Springer, Heidelberg (2006). https://doi.org/10.1007/11774303_17

    Chapter  Google Scholar 

  6. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: SMOTE: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    Article  Google Scholar 

  7. Choffin, B., Popineau, F., Bourda, Y., Vie, J.J.: DAS3H: modeling student learning and forgetting for optimally scheduling distributed practice of skills. In: Proceedings of the International Conference on Educational Data Mining, pp. 29–38 (2019)

    Google Scholar 

  8. Choi, Y., et al.: EdNet: a large-scale hierarchical dataset in education. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12164, pp. 69–73. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52240-7_13

    Chapter  Google Scholar 

  9. Corbett, A., Anderson, J.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4(4), 253–278 (1994). https://doi.org/10.1007/BF01099821

    Article  Google Scholar 

  10. Doroudi, S., Aleven, V., Brunskill, E.: Where’s the reward? Int. J. Artif. Intell. Educ. 29(4), 568–620 (2019). https://doi.org/10.1007/s40593-019-00187-x

    Article  Google Scholar 

  11. Erickson, J.A., Botelho, A.F., McAteer, S., Varatharaj, A., Heffernan, N.T.: The automated grading of student open responses in mathematics. In: Proceedings of the International Conference on Learning Analytics & Knowledge, pp. 615–624 (2020)

    Google Scholar 

  12. Feldman, M.Q., Cho, J.Y., Ong, M., Gulwani, S., Popović, Z., Andersen, E.: Automatic diagnosis of students’ misconceptions in K-8 mathematics. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2018)

    Google Scholar 

  13. Feng, J., Zhang, B., Li, Y., Xu, Q.: Bayesian diagnosis tracing: application of procedural misconceptions in knowledge tracing. In: Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., Luckin, R. (eds.) AIED 2019. LNCS (LNAI), vol. 11626, pp. 84–88. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-23207-8_16

    Chapter  Google Scholar 

  14. Fowlkes, E.B., Mallows, C.L.: A method for comparing two hierarchical clusterings. J. Am. Stat. Assoc. 78(383), 553–569 (1983)

    Article  Google Scholar 

  15. Ghosh, A., Heffernan, N., Lan, A.S.: Context-aware attentive knowledge tracing. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2330–2339 (2020)

    Google Scholar 

  16. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    MATH  Google Scholar 

  17. Graves, A., Mohamed, A.R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)

    Google Scholar 

  18. Gu, J., Lu, Z., Li, H., Li, V.O.: Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393 (2016)

  19. Gusukuma, L., Bart, A.C., Kafura, D., Ernst, J.: Misconception-driven feedback: results from an experimental study. In: Proceedings of the ACM Conference on International Computing Education Research, pp. 160–168 (2018)

    Google Scholar 

  20. He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: Proceedings of the International Conference on World Wide Web, pp. 173–182 (2017)

    Google Scholar 

  21. Khajah, M., Huang, Y., González-Brenes, J., Mozer, M., Brusilovsky, P.: Integrating knowledge tracing and item response theory: a tale of two frameworks. In: Proceedings of the International Workshop on Personalization Approaches in Learning Environments, vol. 1181, pp. 7–15 (2014)

    Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations (2015)

    Google Scholar 

  23. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  24. Koedinger, K.R., Corbett, A., et al.: Cognitive tutors: technology bringing learning sciences to the classroom. In: The Cambridge Handbook of the Learning Sciences, pp. 61–77 (2006)

    Google Scholar 

  25. Lan, A.S., Studer, C., Baraniuk, R.G.: Matrix recovery from quantized and corrupted measurements. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4973–4977, May 2014

    Google Scholar 

  26. Lan, A.S., Studer, C., Waters, A.E., Baraniuk, R.G.: Tag-aware ordinal sparse factor analysis for learning and content analytics. In: Proceedings of the 6th International Conference on Educational Data Mining, pp. 90–97, July 2013

    Google Scholar 

  27. Lan, A.S., Vats, D., Waters, A.E., Baraniuk, R.G.: Mathematical language processing: automatic grading and feedback for open response mathematical questions. In: Proceedings of the ACM Conference on Learning at Scale, pp. 167–176 (2015)

    Google Scholar 

  28. Lindsey, R., Shroyer, J., Pashler, H., Mozer, M.: Improving students’ long-term knowledge retention through personalized review. Psychol. Sci. 25(3), 639–647 (2014)

    Article  Google Scholar 

  29. Lloyd, S.: Least squares quantization in PCM. IEEE Trans. Inf. Theory 28(2), 129–137 (1982)

    Article  MathSciNet  Google Scholar 

  30. Ostini, R., Nering, M.L.: Polytomous Item Response Theory Models, No. 144, Sage (2006)

    Google Scholar 

  31. Pandey, S., Karypis, G.: A self attentive model for knowledge tracing. In: Proceedings of the International Conference on Educational Data Mining, pp. 384–389, July 2019

    Google Scholar 

  32. Pandey, S., Srivastava, J.: RKT: relation-aware self-attention for knowledge tracing. arXiv preprint arXiv:2008.12736 (2020)

  33. Pardos, Z.A., Heffernan, N.T.: Modeling individualization in a Bayesian networks implementation of knowledge tracing. In: De Bra, P., Kobsa, A., Chin, D. (eds.) UMAP 2010. LNCS, vol. 6075, pp. 255–266. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13470-8_24

    Chapter  Google Scholar 

  34. Pavlik Jr., P., Cen, H., Koedinger, K.: Performance factors analysis-a new alternative to knowledge tracing. In: Proceedings of the International Conference on Artificial Intelligence in Education (2009)

    Google Scholar 

  35. Piech, C., et al.: Deep knowledge tracing. In: Proceedings of the Conference on Advances in Neural Information Processing Systems, pp. 505–513 (2015)

    Google Scholar 

  36. Ritter, S., Anderson, J.R., Koedinger, K.R., Corbett, A.: Cognitive tutor: applied research in mathematics education. Psychon. Bull. Rev. 14(2), 249–255 (2007). https://doi.org/10.3758/BF03194060

    Article  Google Scholar 

  37. Selent, D.A.: Creating systems and applying large-scale methods to improve student remediation in online tutoring systems in real-time and at scale. Ph.D. thesis, Worcester Polytechnic Institute (2017)

    Google Scholar 

  38. Smith III, J.P., DiSessa, A.A., Roschelle, J.: Misconceptions reconceived: a constructivist analysis of knowledge in transition. J. Learn. Sci. 3(2), 115–163 (1994)

    Article  Google Scholar 

  39. Steinley, D.: Properties of the Hubert-arable adjusted rand index. Psychol. Methods 9(3), 386 (2004)

    Article  Google Scholar 

  40. Thissen, D., Steinberg, L.: A taxonomy of item response models. Psychometrika 51(4), 567–577 (1986)

    Article  Google Scholar 

  41. VanLehn, K.: Bugs are not enough: empirical studies of bugs, impasses and repairs in procedural skills. J. Math. Behav. (1982)

    Google Scholar 

  42. Wang, F., et al.: Neural cognitive diagnosis for intelligent education systems. Pro. AAAI Conf. Artif. Intell. 34, 6153–6161 (2020)

    Google Scholar 

  43. Wang, Z., et al.: Diagnostic questions: the NeurIPS 2020 education challenge. arXiv preprint arXiv:2007.12061 (2020)

  44. Woolf, B.P.: Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing E-learning. Morgan Kaufmann (2010)

    Google Scholar 

  45. Yang, Y., et al.: GIKT: a graph-based interaction model for knowledge tracing. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds.) ECML PKDD 2020. LNCS (LNAI), vol. 12457, pp. 299–315. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67658-2_18

    Chapter  Google Scholar 

  46. Yudelson, M.V., Koedinger, K.R., Gordon, G.J.: Individualized Bayesian knowledge tracing models. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS (LNAI), vol. 7926, pp. 171–180. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39112-5_18

    Chapter  Google Scholar 

  47. Zhang, J., Shi, X., King, I., Yeung, D.Y.: Dynamic key-value memory networks for knowledge tracing. In: Proceedings of the International Conference on World Wide Web, pp. 765–774, April 2017

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Massachusetts Amherst, Amherst, USA

    Aritra Ghosh & Andrew Lan

  2. Pittsburgh, PA, USA

    Jay Raspat

Authors
  1. Aritra Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jay Raspat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew Lan .

Editor information

Editors and Affiliations

  1. Technion – Israel Institute of Technology, Haifa, Israel

    Ido Roll

  2. Arizona State University, Tempe, AZ, USA

    Danielle McNamara

  3. Utrecht University, Utrecht, The Netherlands

    Sergey Sosnovsky

  4. London Knowledge Lab, London, UK

    Rose Luckin

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ghosh, A., Raspat, J., Lan, A. (2021). Option Tracing: Beyond Correctness Analysis in Knowledge Tracing. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78292-4_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78291-7

  • Online ISBN: 978-3-030-78292-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics