Assessing the Cold Start Problem in Adaptive Systems

This study presents a comparison of the following methods for estimating task difficulty in terms of the so-called "cold start" problem - during the initial phase of the introduction of the adaptive educational system to the public. The data originates from the item-based online programming course made available on the RunCode online learning platform. The dataset contains 50055 submissions on 76 tasks uploaded by 299 learners. The reference task difficulty values have been estimated with the Item Response Theory Graded Response Model on the full dataset. Results of this study show, that for the smallest size of the sample n = 5 learners, the Elo rating algorithm achieves a reasonably high correlation of 0.702. For small sizes of the sample n = 5, 10, and 20 learners, the Elo algorithm delivers slightly better estimates than the Glicko rating algorithm and outperforms the proportion correct (PC) method. For the size of the sample n = 50, the Glicko algorithm delivers better results than the Elo, and the difference between algorithms and the proportion correct method decreases.