Combining empirical and machine learning techniques to predict math expertise using pen signal features

Multimodal learning analytics aims to automatically analyze students' natural communication patterns based on speech, writing, and other modalities during learning activities. This research used the Math Data Corpus, which contains time-synchronized multimodal data from collaborating students as they jointly solved problems varying in difficulty. The aim was to investigate how reliably pen signal features, which were extracted as students wrote with digital pens and paper, could identify which student in a group was the dominant domain expert. An additional aim was to improve prediction of expertise based on joint bootstrapping of empirical science and machine learning techniques. To accomplish this, empirical analyses first identified which data partitioning and pen signal features were most reliably associated with expertise. Then alternative machine learning techniques compared classification accuracies based on all pen features, versus empirically selected ones. The best unguided classification accuracy was 70.8%, which improved to 83.3% with empirical guidance. These results demonstrate that handwriting signal features can predict domain expertise in math with high reliability. Hybrid methods also can outperform black-box machine learning in both accuracy and transparency.