Skip to main content
SpringerLink
Log in

Empirical analysis of deep learning networks for affective video tagging

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper presents, a thorough evaluation of popular deep learning models to analyze and classify electroencephalogram (EEG) data for characterizing human affective states for video content tagging and retrieval. We use two pre-trained convolutional neural network (CNN) models AlexNet and GoogLeNet, and a Long Short Term Memory (LSTM) model to classify EEG data into appropriate affect categories using trans-domain learning. The purpose behind the use of pre-trained networks or trans-domain learning is twofold – to establish the versatility of pre-trained networks by testing their ability to classify EEG data for emotion recognition and the other is to reduce over cost of computation while training the networks. Our work tries to establish the answer of a simple question: Are pre-trained deep models versatile enough for classifying not only similar type of problems but are also effective for classifying problems pertaining to completely different domains? Also, using pre-trained models saves considerable computation time required for training a new model from scratch and fine tuning it. We use DEAP dataset for training and evaluation of these networks over a single modality ‘valence’ to simplify the comparison among these networks. Experiments are carried out by training the networks on EEG recordings obtained from single as well as multiple subjects to show the effects of subject-specific and generalized data on classification accuracy. Experimental results suggest the superiority of GoogLeNet for individual subject data while AlexNet outperforms other networks and has shown its capability of generalizing well. We compare the performance of these networks with state-of-art classifiers handcrafted by other authors for classifying EEG data and find that the performance of pre-trained CNNs used in our work are comparable or even better than the other handcrafted classifiers used by many authors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig.7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Arnau-González P, Arevalillo-Herráez M, Ramzan N (2017) Fusing highly dimensional energy and connectivity features to identify affective states from EEG signals. Neurocomputing 244(2017):81–89, ISSN 0925-2312. https://doi.org/10.1016/j.neucom.2017.03.027

    Article  Google Scholar 

  2. Atkinson J, Campos D (2015) Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers. Expert Syst Appl 47(2016):35–41, ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2015.10.049

    Article  Google Scholar 

  3. Gupta R, ur Rehman Laghari K, Falk TH (2016) Relevance vector classifier decision fusion and EEG graph-theoretic features for automatic affective state characterization. Neurocomputing 174(Part B):875–884, ISSN 0925-2312. https://doi.org/10.1016/j.neucom.2015.09.085

    Article  Google Scholar 

  4. Jirayucharoensak S, Pan-ngum S, Israsena P (2014) EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. TheScientificWorldJournal. 2014:627892. https://doi.org/10.1155/2014/627892

    Article  Google Scholar 

  5. Kierkels JJM, Soleymani M, Pun T (2009) Queries and tags in affect-based multimedia retrieval. in Proc. Int. Conf. Multimedia and Expo. New York, NY, USA: IEEE Press, pp. 1436–1439

  6. Koelstra S et al. (2010) Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos. In: Yao Y., Sun R., Poggio T., Liu J., Zhong N., Huang J. (eds) Brain Informatics. BI 2010. Lecture Notes in Computer Science, vol 6334. Springer, Berlin, Heidelberg https://doi.org/10.1007/978-3-642-15314-3_9

  7. Koelstra S, Muhl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2011) Deap: A database for emotion analysis using physiological signals. IEEE transaction on affective computing. Spec Issue Natural Affect Resourc Syst Build Eval 3(1):18–31

  8. Li M, Xu H, Liu X, Lu S (2018) Emotion recognition from multichannel EEG signals using K-nearest neighbor classification. Technol Health Care 26(S1):509–519

    Article  Google Scholar 

  9. Lin W, Li C, Sun S (2017) Deep Convolutional Neural Network for Emotion Recognition Using EEG and Peripheral Physiological Signal. In: Zhao Y., Kong X., Taubman D. (eds) Image and Graphics. ICIG 2017. Lecture Notes in Computer Science, vol 10667. Springer, Cham https://doi.org/10.1007/978-3-319-71589-6_33

  10. Liu J, Meng H, Nandi A, Li M (2016) Emotion detection from EEG recordings. 2016 12th International Conference on Natural Computation. Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, pp. 1722-1727. https://doi.org/10.1109/FSKD.2016.7603437

  11. Liu W, Zheng W, Lu B (2016) Multimodal emotion recognition using multimodal deep learning. ArXiv, abs/1602.08225.

  12. Mert A, Akan A (2016) Emotion recognition from EEG signals by using multivariate empirical mode decomposition. Pattern Anal Applic 2016:1–9

    Google Scholar 

  13. Mohammadpour M, Khaliliardali H, Hashemi SMR, AlyanNezhadi MM (2017) Facial emotion recognition using deep convolutional networks. 2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI), Tehran, pp. 0017–0021. https://doi.org/10.1109/KBEI.2017.8324974

  14. Russakovsky O, Deng J, Su H et al (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis (IJCV) 115(3):211–252

    Article  MathSciNet  Google Scholar 

  15. Sang DV, Van Dat N, Thuan DP (2017) Facial expression recognition using deep convolutional neural networks. 2017 9th International Conference on Knowledge and Systems Engineering (KSE), Hue, pp. 130–135. https://doi.org/10.1109/KSE.2017.8119447

  16. Shan MK, Kuo FF, Chiang MF, Lee SY (2009) Emotion-based music recommendation by affinity discovery from film music. Expert Syst Appl 36(4):7666–7674

    Article  Google Scholar 

  17. Sharma N, Jain V, Mishra A (2018) An analysis of convolutional neural networks for image classification. Proc Comput Sci 132:377–384

    Article  Google Scholar 

  18. Tkalčič M, Burnik U, Košir A (2010) Using affective parameters in a content-based recommender system for images. User Modeling and User-Adapted Interaction, pp. 1–33–33

  19. Wichakam I, Vateekul P (2014) An evaluation of feature extraction in EEG-based emotion prediction with support vector machines. In International Joint Conference on Computer Science and Software Engineering 2014

  20. Yin Z, Zhao M, Wang Y, Yang J, Zhang J (2017) Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Prog Biomed 140:93–110, ISSN 0169-2607. https://doi.org/10.1016/j.cmpb.2016.12.005

    Article  Google Scholar 

  21. Zhai J, Zhao H-g, Ji Q, Xie X-d (2018) Computational Resource Constrained Deep Learning Based Target Recognition from Visible Optical Images. J Info Hiding Multimed Signal Process 9(3):558–566

    Google Scholar 

  22. Zhang S, Zhang S, Huang T, Gao W (2018) Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching. IEEE Transact Multimed 20(6):1576–1590. https://doi.org/10.1109/TMM.2017.2766843

    Article  Google Scholar 

  23. Zhang F-Q, Mao Z-J, Huang Y-F, Xu L, Ding G (2018) Deep Learning Models for EEG-based Rapid Serial Visual Presentation Event Classification. J Info Hiding Multimed Signal Process 9(1):177–187

    Google Scholar 

  24. Zheng W, Lu B (2015) Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks. IEEE Trans Auton Ment Dev 7(3):162–175. https://doi.org/10.1109/TAMD.2015.2431497

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Amity School of Engineering & Technology, Amity University Uttar Pradesh, Noida, India

    Anju Mishra

  2. Department of Electronics & Electrical Engineering, Amity School of Engineering & Technology, Amity University Uttar Pradesh, Noida, India

    Priya Ranjan

  3. School of Vocational Studies and Applied Sciences, Gautam Buddha University, Greater Noida, India

    Amit Ujlayan

Authors
  1. Anju Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Priya Ranjan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amit Ujlayan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anju Mishra.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mishra, A., Ranjan, P. & Ujlayan, A. Empirical analysis of deep learning networks for affective video tagging. Multimed Tools Appl 79, 18611–18626 (2020). https://doi.org/10.1007/s11042-020-08714-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08714-y

Keywords

Search

Navigation