Skip to main content
SpringerLink
Log in

Learning CNN features from DE features for EEG-based emotion recognition

  • Theoretical advances
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Recently, deep neural networks (DNNs) have shown the remarkable success of feature representations in computer vision, audio analysis, and natural language processing. Furthermore, DNNs have been used for electroencephalography (EEG) signal classification in recent studies on brain–computer interface. However, most works use one-dimensional EEG features to learn DNNs that ignores the local information within multichannel or multiple frequency bands in the EEG signals. In this paper, we propose a novel emotion recognition method using a convolutional neural network (CNN) while preventing the loss of local information. The proposed method consists of two parts. The first part generates topology-preserving differential entropy features while keeping the distance from the center electrode to other electrodes. The second part learns the proposed CNN to estimate three-class emotional states (positive, neutral, negative). We evaluate our work on SEED dataset, including 62-channel EEG signals recorded from 15 subjects. Our experimental results demonstrate that the proposed method achieved superior performance on SEED dataset with an average accuracy of 90.41% with the visualization of extracted features from the proposed CNN using t-SNE to show our representation outperforms the other representations based on standard features for EEG analysis. Besides, with the additional experiment on VIG dataset to estimate the vigilance of EEG dataset, we show the off-the-shelf availability of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Walter S, Wendt C, Böhnke J, Crawcour S, Tan J-W, Chan A, Limbrecht K, Gruss S, Traue HC (2014) Similarities and differences of emotions in human-machine and human-human interactions: what kind of emotions are relevant for future companion systems? Ergonomics 57(3):374–386. https://doi.org/10.1080/00140139.2013.822566

    Article  Google Scholar 

  2. Saxen F, Werner P, Al-Hamadi A (2017) Real vs. fake emotion challenge: Learning to rank authenticity from facial activity descriptors, In: 2017 IEEE international conference on computer vision workshops (ICCVW), pp 3073–3078. https://doi.org/10.1109/ICCVW.2017.363

  3. Rejer I, Cieszyński Ł (2019) Independent component analysis for a low-channel ssvep-bci. Pattern Anal Appl 22(1):47–62. https://doi.org/10.1007/s10044-018-0758-4

    Article  MathSciNet  Google Scholar 

  4. Mert A, Akan A (2018) Emotion recognition from eeg signals by using multivariate empirical mode decomposition. Pattern Anal Appl 21(1):81–89. https://doi.org/10.1007/s10044-016-0567-6

    Article  MathSciNet  Google Scholar 

  5. Alarcao SM, Fonseca MJ (2017) Emotions recognition using eeg signals: a survey. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2017.2714671

    Article  Google Scholar 

  6. Bashivan P, Rish I, Yeasin M, Codella N (2016) Learning representations from EEG with deep recurrent‐convolutional neural networks. International conference on learning representations

  7. Yik M, Russell JA, Steiger JH (2011) A 12-point circumplex structure of core affect. Emotion 11(4):705–31. https://doi.org/10.1037/a0023980

    Article  Google Scholar 

  8. Ekman P (1992) An argument for basic emotions. Cognit Emot 6(3–4):169–200. https://doi.org/10.1080/02699939208411068

    Article  Google Scholar 

  9. Plutchik R (1982) A psychoevolutionary theory of emotions. Soc Sci Inf 21(4–5):529–553. https://doi.org/10.1177/053901882021004003

    Article  Google Scholar 

  10. Zheng W-L, Lu B-L (2015) Investigating critical frequency bands and channels for eeg-based emotion recognition with deep neural networks. IEEE Trans Auton Ment Dev 7(3):162–175. https://doi.org/10.1109/TAMD.2015.2431497

    Article  Google Scholar 

  11. Jirayucharoensak S, Pan-Ngum S, Israsena P (2014) EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014:627892. https://doi.org/10.1155/2014/627892

    Article  Google Scholar 

  12. Duan R-N, Zhu J-Y, Lu B-L (2013) Differential entropy feature for eeg-based emotion classification. In: 2013 6th international IEEE/EMBS conference on neural engineering (NER), IEEE, pp 81–84. https://doi.org/10.1109/NER.2013.6695876

  13. Lin Y-P, Wang C-H, Jung T-P, Wu T-L, Jeng S-K, Duann J-R, Chen J-H (2010) Eeg-based emotion recognition in music listening. IEEE Trans Biomed Eng 57(7):1798–1806

    Article  Google Scholar 

  14. Brown L, Grundlehner B, Penders J (2011) Towards wireless emotional valence detection from eeg. In: Engineering in medicine and biology society, EMBC, 2011 annual international conference of the IEEE, IEEE, pp 2188–2191

  15. Petrantonakis PC, Hadjileontiadis LJ (2011) A novel emotion elicitation index using frontal brain asymmetry for enhanced eeg-based emotion recognition. IEEE Trans Inf Technol Biomed 15(5):737–746

    Article  Google Scholar 

  16. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) Deap: a database for emotion analysis; using physiological signals. IEEE Trans Affect Comput 3(1):18–31

    Article  Google Scholar 

  17. Hadjidimitriou SK, Hadjileontiadis LJ (2012) Toward an eeg-based recognition of music liking using time-frequency analysis. IEEE Trans Biomed Eng 59(12):3498–3510

    Article  Google Scholar 

  18. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3(1):42–55

    Article  Google Scholar 

  19. Wang X-W, Nie D, Lu B-L (2014) Emotional state classification from eeg data using machine learning approach. Neurocomputing 129:94–106

    Article  Google Scholar 

  20. Murugappan M, Ramachandran N, Sazali Y (2010) Classification of human emotion from eeg using discrete wavelet transform. J Biomed Sci Eng 3(04):390

    Article  Google Scholar 

  21. Zheng W-L, Zhu J-Y, Lu B-L (2017) Identifying stable patterns over time for emotion recognition from eeg. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2017.2712143

    Article  Google Scholar 

  22. Bhavsar R, Sun Y, Helian N, Davey N, Mayor D, Steffert T (2018) The correlation between eeg signals as measured in different positions on scalp varying with distance. Procedia Comput Sci 123:92–97. https://doi.org/10.1016/j.procs.2018.01.015

    Article  Google Scholar 

  23. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  24. Huo Xue-Qin, Zheng W, Lu B (2016) Driving fatigue detection with fusion of eeg and forehead eog. In: 2016 international joint conference on neural networks (IJCNN), pp 897–904. https://doi.org/10.1109/IJCNN.2016.7727294

  25. Schleif F, Chen H, Tino P (2015) Incremental probabilistic classification vector machine with linear costs. In: 2015 international joint conference on neural networks (IJCNN), pp 1–8. https://doi.org/10.1109/IJCNN.2015.7280377

  26. Qin AK, Suganthan PN (2004) A novel kernel prototype-based learning algorithm. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004., vol 4, pp 621–624. https://doi.org/10.1109/ICPR.2004.1333849

  27. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemometr Intell Lab Syst 2(1–3):37–52

    Article  Google Scholar 

  28. McLachlan G (2004) Discriminant analysis and statistical pattern recognition, vol 544. Wiley, Hoboken

    MATH  Google Scholar 

  29. Balakrishnama S, Ganapathiraju A (1998) Linear discriminant analysis-a brief tutorial. Inst Signal Inf Process 18:1–8

    Google Scholar 

  30. Peterson LE (2009) K-nearest neighbor. Scholarpedia 4(2):1883

    Article  Google Scholar 

  31. Mahalanobis PC (1936) On the generalized distance in statistics. National Institute of Science of India, Berhampur

    MATH  Google Scholar 

  32. Pal M, Maxwell AE, Warner TA (2013) Kernel-based extreme learning machine for remote-sensing image classification. Remote Sens Lett 4(9):853–862. https://doi.org/10.1080/2150704X.2013.805279

    Article  Google Scholar 

  33. Wang F, Zhong S-h, Peng J, Jiang J, Liu Y (2018) Data augmentation for eeg-based emotion recognition with deep convolutional neural networks. In: International conference on multimedia modeling, Springer, pp 82–93

  34. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans Cybern PP(99):1–9. https://doi.org/10.1109/TCYB.2017.2788081

    Article  Google Scholar 

  35. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  36. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  37. Spampinato C, Palazzo S, Kavasidis I, Giordano D, Souly N, Shah M (2017) Deep learning human mind for automated visual classification. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 4503–4511. https://doi.org/10.1109/CVPR.2017.479

  38. Grill-Spector K, Henson R, Martin A (2006) Repetition and the brain: neural models of stimulus-specific effects. Trends Cognit Sci 10(1):14–23. https://doi.org/10.1016/j.tics.2005.11.006

    Article  Google Scholar 

  39. Zheng W-L, Lu B-L (2017) A multimodal approach to estimating vigilance using EEG and forehead EOG. J Neural Eng 14(2):026017. https://doi.org/10.1088/1741-2552/aa5a98

    Article  Google Scholar 

  40. Zhang N, Zheng W-L, Liu W, Lu B-L (2016) Continuous vigilance estimation using lstm neural networks. In: Proceedings of the 23rd international conference on neural information processing—Volume 9948. Springer, Berlin, pp 530–537. https://doi.org/10.1007/978-3-319-46672-9_59

  41. Shi L-C, Lu B-L (2010) Off-line and on-line vigilance estimation based on linear dynamical system and manifold learning. In: Engineering in medicine and biology society (EMBC), 2010 Annual International Conference of the IEEE, IEEE, pp 6587–6590. https://doi.org/10.1109/IEMBS.2010.5627125

  42. Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison Z, Antiga L, Lerer A (2017) Automatic differentiation in pytorch. In: NIPS-W

  43. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12(Oct):2825–2830

    MathSciNet  MATH  Google Scholar 

  44. Maaten Lvd, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(Nov):2579–2605

    MATH  Google Scholar 

  45. Ma J-X, Shi L-C, Lu B-L (2010) Vigilance estimation by using electrooculographic features. In: 2010 annual international conference of the ieee engineering in medicine and biology, IEEE, pp 6591–6594

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Yonsei University, 50 Yonsei-ro, Seodaemun-Gu, Seoul, 03722, Republic of Korea

    Sunhee Hwang, Kibeom Hong & Hyeran Byun

  2. The Graduate Program in Cognitive Science, Yonsei University, 50 Yonsei-ro, Seodaemun-Gu, Seoul, 03722, Republic of Korea

    Guiyoung Son & Hyeran Byun

Authors
  1. Sunhee Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kibeom Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guiyoung Son
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hyeran Byun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hyeran Byun.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hwang, S., Hong, K., Son, G. et al. Learning CNN features from DE features for EEG-based emotion recognition. Pattern Anal Applic 23, 1323–1335 (2020). https://doi.org/10.1007/s10044-019-00860-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-019-00860-w

Keywords

Search

Navigation