Higher Education Programming Competencies: A Novel Dataset

Abstract

Students’ challenges in introductory programming courses have long been subject to research. In fact, learners are faced with cognitively complex tasks, such as modeling and writing programs. At the same time, educators are known to experience challenges with the classification of a competency’s cognitive complexity. In this paper, we present a text dataset with competency goals expected in basic programming courses. We then apply a deep learning approach to the dataset to classify the competency-based learning objectives as a use case. A manually annotated dataset of 35 German universities and their learning objectives in 129 introductory programming courses was processed into a machine-readable format to achieve these goals. It contains 1015 competency goals (both in German and English) and their classification into dimensions of complexity. Different state-of-the-art machine learning (ML) models, e.g., BERT, along with Natural Language Processing techniques, i.e., parts-of-speech-tagging, were combined to train a deep learning model in a supervised manner for the classification of competencies. The proof-of-concept shows that knowledge can be derived from the dataset. In the presented use case, the ML classification achieved a maximum accuracy of 81.4%. This work has several implications for educators, as it is the foundation for an application that classifies competency goals according to their cognitive complexity. The dataset can further be used to test language models as a baseline performance task. Moreover, the dataset can be extended, e.g., with data from other countries and languages. The dataset is available online under a Creative Commons license (https://github.com/nkiesler-cs/HEPComp-Dataset).