Learner’s Mental State Estimation with PC Built-in Camera

Abstract

The purpose of this research is to estimate learners’ mental states such as difficulty, interest, fatigue, and concentration that change with the time series between learners and their learning tasks. Nowadays, we have many opportunities to learn specific topics in the individual learning process, such as active learning and self-directed learning. In such situations, it is challenging to grasp learners’ progress and engagement in their learning process. Several studies have estimated learners’ engagement from facial images/videos in the learning process. However, there is no extensive benchmark dataset except for the video watching process. Therefore, we gathered learners’ videos with facial expression and retrospective self-report from 19 participants through the CAB test process using a PC built-in camera. In this research, we applied an existing face image recognition library Face++ to extract the data such as estimated emotion, eye gaze, face orientation, face position (percentage on the screen) by each frame of the videos. Then, we built a couple of machine learning models, including deep learning methods, to estimate their mental states from the facial expressions and compared them with the average accuracy of prediction. The results demonstrated the potential of the proposed method to the estimation and provided the improvement plan from the accuracy point of view.