Naturalistic Emotion Recognition Using EEG and Eye Movements

Abstract

Emotion recognition in affective brain-computer interfaces (aBCI) has emerged as a prominent research area. However, existing experimental paradigms for collecting emotional data often rely on stimuli-based elicitation, which may not accurately reflect emotions experienced in everyday life. Moreover, these paradigms are limited in terms of stimulus types and lack investigation into decoding naturalistic emotional states. To address these limitations, we propose a novel experimental paradigm that enables the recording of physiological signals in a more natural way. In our approach, emotions are allowed to arise spontaneously, unrestricted by specific experimental activities. Participants have the autonomy to determine the start and end of each recording session and provide corresponding emotion label. Over a period of three months, we recruited six subjects and collected data through multiple recording sessions per subject. We utilized electroencephalogram (EEG) and eye movement signals in both subject-dependent and cross-subject settings. In the subject-dependent unimodal condition, our attentive simple graph convolutional network (ASGC) achieved the highest accuracy of 76.32% for emotion recognition based on EEG data. For the cross-subject unimodal condition, our domain adversarial neural network (DANN) outperformed other models, achieving an average accuracy of 71.90% based on EEG data. These experimental results demonstrate the feasibility of recognizing emotions in naturalistic settings. The proposed experimental paradigm holds significant potential for advancing emotion recognition in various practical applications. By allowing emotions to unfold naturally, our approach enables the future emergence of more robust and applicable emotion recognition models in the field of aBCI.