Revisiting Annotations in Online Student Engagement

Assessing the engagement of students in online classroom is crucial to meet their learning objectives. Many machine learning and deep learning models have been proposed to handle this problem using a variety of sensors, with videos cameras being the most prominent. However, most of these approaches are not interoperable because different datasets use different labeling protocols. As a result, the classification models range from binary, multi-class to regression problems. Another problem is the lack of rigor and definition of engagement to annotate the data. In this paper, firstly we showed inconsistencies in the labeling of a popular student engagement DAiSEE dataset. Then, we re-labeled more than 7000 videos of this dataset using a methodical engagement annotation protocol, HELP, to convert it from four class to binary classification problem. Further analysis highlights issues in DAiSEE annotation in comparison to the HELP protocol. Lastly, we tested three state-of-the-art deep learning and feature-based methods and discussed their performance. Data imbalance in the newly and previously annotated data was found to be the main issue in developing predictive models.