Abstract
The emergence of domain adaptation has brought remarkable advancement to EEG-based emotion recognition by reducing subject variability thus increasing the accuracy of cross-subject tasks. A wide variety of materials have been employed to elicit emotions in experiments, however, artistic works that aim to evoke emotional resonance of observers are relatively less frequently utilized. Previous research has shown promising results in electroencephalogram(EEG)-based emotion recognition on static oil paintings. As video clips are widely recognized as the most commonly used and effective stimuli, we adopted animated live oil paintings, a novel set of emotional stimuli in the live form which are essentially a type of video clip while possessing fewer potential influencing factors for EEG signals compared to traditional video clips, such as abrupt switches on background sound, contrast, and color tones. Moreover, previous studies on static oil paintings focused primarily on the subject-dependent task, and further research involving cross-subject analysis remains to be investigated. In this paper, we proposed a novel DAformer model which combines the advantages of Transformer and adversarial learning. In order to enhance the evocative performance of oil paintings, we introduced a type of innovative emotional stimuli by transforming static oil paintings into animated live forms. We developed a new emotion dataset SEED-LOP (SJTU EEG Emotion Dataset-Live Oil Painting) and constructed DAformer to verify the effectiveness of SEED-LOP. The results demonstrated higher accuracies in three-class emotion recognition tasks when watching live oil paintings, with a subject-dependent accuracy achieving 61.73% and a cross-subject accuracy reaching 54.12%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alarcao, S.M., Fonseca, M.J.: Emotions recognition using EEG signals: a survey. IEEE Trans. Affect. Comput. 10(3), 374–393 (2017)
Zheng, W.-L., Zhu, J.-Y., Lu, B.-L.: Identifying stable patterns over time for emotion recognition from EEG. IEEE Trans. Affect. Comput. 10(3), 417–429 (2019)
Schaaff, K., Schultz, T.: Towards emotion recognition from electroencephalographic signals. In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, pp. 1-6. IEEE (2009)
Koelstra, S., Muhl, C., Soleymani, M.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)
Lin, Y.P., et al.: EEG-based emotion recognition in music listening. IEEE Trans. Biomed. Eng. 57(7), 1798–1806 (2010)
Luo, S., Lan, Y.T., Peng, D., Li, Z., Zheng, W.L., Lu, B.L.: Multimodal emotion recognition in response to oil paintings. In: 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4167-4170. IEEE (2022)
Lan, Y.T., Li, Z.C., Peng, D., Zheng, W.L., Lu, B.L.: Identifying artistic expertise difference in emotion recognition in response to oil paintings. In: 11th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 1–4. IEEE (2023)
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Mach. Learn. 79, 151–175 (2010)
Li, H., Jin, Y.-M., Zheng, W.-L., Lu, B.-L.: Cross-subject emotion recognition using deep adaptation networks. In: Cheng, L., Leung, A.C.S., Ozawa, S. (eds.) ICONIP 2018. LNCS, vol. 11305, pp. 403–413. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04221-9_36
Zhu, Y., et al.: Deep subdomain adaptation network for image classification. IEEE Trans. Neural Networks Learn. Syst. 32(4), 1713–1722 (2021)
Wang, Y., Jiang, W. B., Li, R., Lu, B.L.: Emotion transformer fusion: complementary representation properties of EEG and eye movements on recognizing anger and surprise. In: 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 1575–1578. IEEE (2021)
Li, R., Wang, Y., Lu, B.L.: Measuring decision confidence levels from EEG using a spectral-spatial-temporal adaptive graph convolutional neural network. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds.) ICONIP 2022. LNCS, pp. 395–406. Springer, Cham (2022). https://doi.org/10.1007/978-981-99-1642-9_34
Schaefer, A., et al.: Assessing the effectiveness of a large database of emotion-eliciting films: a new tool for emotion researchers. Cogn. Emot. 24(7), 1153–1172 (2010)
Zheng, W.L., Lu, B.L.: Personalizing EEG-based affective models with transfer learning. In: International Joint Conference on Artificial Intelligence, pp. 2732–2738. AAAI Press, New York (2016)
Batt, R., Palmiero, M., Nakatani, C., van Leeuwen, C.: Style and spectral power: processing of abstract and representational art in artists and non-artists. Perception 39(12), 1659–1671 (2010)
Chatterjee, A., Widick, P., Sternschein, R., Smith, W.B., Bromberger, B.: The assessment of art attributes. Empir. Stud. Arts 28(2), 207–222 (2010)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Sun, B., Saenko, K.: Deep CORAL: correlation alignment for deep domain adaptation. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 443–450. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49409-8_35
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2030–2096 (2016)
Wang, J., Feng, W., Chen, Y., Yu, H., Huang, M., Yu, P.S.: Visual domain adaptation with manifold embedded distribution alignment. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 402-410 (2018)
Acknowledgements
This work was supported in part by grants from National Natural Science Foundation of China (Grant No. 61976135), STI 2030-Major Projects+2022ZD0208500, Shanghai Municipal Science and Technology Major Project (Grant No. 2021SHZD ZX), Shanghai Pujiang Program (Grant No. 22PJ1408600), Medical-Engineering Interdisciplinary Research Foundation of Shanghai Jiao Tong University “Jiao Tong Star” Program (YG2023ZD25), and GuangCi Professorship Program of RuiJin Hospital Shanghai Jiao Tong University School of Medicine.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jin, ZW., Liu, JW., Zheng, WL., Lu, BL. (2024). DAformer: Transformer with Domain Adversarial Adaptation for EEG-Based Emotion Recognition with Live-Oil Paintings. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1963. Springer, Singapore. https://doi.org/10.1007/978-981-99-8138-0_32
Download citation
DOI: https://doi.org/10.1007/978-981-99-8138-0_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8137-3
Online ISBN: 978-981-99-8138-0
eBook Packages: Computer ScienceComputer Science (R0)