Abstract
Adaptive learning systems (ALS) tailor educational material to the level of the users. In ALS ability should be continuously estimated based on the users’ responses to adaptively selected practice items. However, the large-scale, adaptive, and dynamic nature of ALS poses challenges for traditional estimation methods. The Urnings algorithm [1] has been recently proposed to address these challenges. However, the original algorithm does not address the cold-start problem which ALS suffer from: Initially, it is difficult to adapt item selection to the users’ abilities based on limited available information. We develop a modification of the Urnings algorithm aiming to alleviate the cold-start problem by increasing the step size of the algorithm when a systematic change in the ratings is detected, and decreasing it when the ratings are relatively stable. The results of our simulation studies showed that the modified algorithm moves away from the initial values faster, responds to sudden changes in ability better, and results in overall higher accuracy than the original algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Note, that in this paper we focus only on adapting step size for the users, as there are typically much more responses available per item and item difficulty is less prone to sudden changes which makes the cold-start problem and the problem of following change less important on the item side.
- 2.
i.e., the two-sided p-value is computed
- 3.
Note that \(n_{min}\) and \(n_{max}\) should be chosen in such a way that the n min can be reached by dividing the \(n_{max}\) by a power of 2
- 4.
The analysis script is hosted at https://github.com/mrpogge/Urnings_AIED.git
References
Bolsinova, M., Maris, G., Hofman, A.D., van der Maas, H.L., Brinkhuis, M.J.: Urnings: a new method for tracking dynamically changing parameters in paired comparison systems. J. Roy. Stat. Soc. Ser. C (Appl. Stat.) (2022)
Brinkhuis, M.J., Maris, G.: Dynamic parameter estimation in student monitoring systems. Measurement and Research Department Reports (Rep. No. 2009–1). Arnhem: Cito 146 (2009)
Brinkhuis, M.J., Savi, A.O., Hofman, A.D., Coomans, F., van Der Maas, H.L., Maris, G.: Learning as it happens: a decade of analyzing and shaping a large-scale online learning system. J. Learn. Anal. 5(2), 29–46 (2018)
Elo, A.E.: The Rating of Chessplayers, Past and Present. Arco Publications (1978)
Glickman, M.E.: Dynamic paired comparison models with stochastic variances. J. Appl. Stat. 28(6), 673–689 (2001)
Hofman, A.D., Brinkhuis, M.J., Bolsinova, M., Klaiber, J., Maris, G., van der Maas, H.L.: Tracking with (un) certainty. J. Intell. 8(1), 10 (2020)
Klinkenberg, S., Straatemeier, M., van der Maas, H.L.: Computer adaptive practice of maths ability using a new item response model for on the fly ability and difficulty estimation. Comput. Educ. 57(2), 1813–1824 (2011)
Ostrow, K.: Motivating learning in the age of the adaptive tutor. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M.F. (eds.) AIED 2015. LNCS (LNAI), vol. 9112, pp. 852–855. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19773-9_131
Pankiewicz, M.: Assessing the cold start problem in adaptive systems. In: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education, vol. 2, pp. 650–650 (2021)
Pliakos, K., Joo, S.H., Park, J.Y., Cornillie, F., Vens, C., Van den Noortgate, W.: Integrating machine learning into item response theory for addressing the cold start problem in adaptive learning systems. Comput. Educ. 137, 91–103 (2019)
Rasch, G.: Studies in mathematical psychology: I. Probabilistic models for some intelligence and attainment tests (1960)
Shemshack, A., Spector, J.M.: A systematic literature review of personalized learning terms. Smart Learn. Environ. 7(1), 1–20 (2020). https://doi.org/10.1186/s40561-020-00140-9
Tritchler, D.: On Inverting Permutation Tests. J. Am. Stat. Assoc. 79(385), 200–207 (1984)
Wauters, K., Desmet, P., Van Den Noortgate, W.: Adaptive item-based learning environments based on the item response theory: possibilities and challenges. J. Comput. Assist. Learn. 26(6), 549–562 (2010)
Wauters, K., Desmet, P., Van Noortgate, W.: Monitoring learners’ proficiency: weight adaptation in the elo rating system. In: Educational Data Mining 2011 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gergely, B., der Maas, H.L.J.v., Maris, G.K.J., Bolsinova, M. (2023). Warming up the Cold Start: Adaptive Step Size Method for the Urnings Algorithm. In: Wang, N., Rebolledo-Mendez, G., Dimitrova, V., Matsuda, N., Santos, O.C. (eds) Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky. AIED 2023. Communications in Computer and Information Science, vol 1831. Springer, Cham. https://doi.org/10.1007/978-3-031-36336-8_64
Download citation
DOI: https://doi.org/10.1007/978-3-031-36336-8_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36335-1
Online ISBN: 978-3-031-36336-8
eBook Packages: Computer ScienceComputer Science (R0)