An-Qi Bi
Yu-Dong Zhang
Skip Abstract Section
Abstract
In this article, we focus on the dynamic facial emotion recognition from online video. We combine deep neural networks with transfer learning theory and propose a novel model named DT-EFER. In detail, DT-EFER uses GoogLeNet to extract the deep features of key images from video clips. Then to solve the dynamic facial emotion recognition scenario, the framework introduces transfer learning theory. Thus, to improve the recognition performance, model DT-EFER focuses on the differences between key images instead of those images themselves. Moreover, the time complexity of this model is not high, even if previous exemplars are introduced here. In contrast to other exemplar-based models, experiments based on two datasets, namely, BAUM-1s and Extended Cohn–Kanade, have shown the efficiency of the proposed DT-EFER model.
- [1] . 2016. Incremental enhanced \( \alpha \) expansion move for large data: A probability regularization perspective. International Journal of Machine Learning and Cybernetics 8, 5 (2016), 1–17.Google Scholar
- [2] . 2016. Bayesian enhanced \( \alpha \) expansion move clustering with loose link constraints. Neurocomputing 194 (2016), 288–300.
DOI: Google ScholarDigital Library
- [3] . 2019. EEG-based multi-modal emotion recognition using bag of deep features: An optimal feature selection approach. Sensors 19, 23 (2019).Google ScholarCross Ref
- [4] . 2019. Deep features-based speech emotion recognition for smart affective services. Multimedia Tools and Applications 78, 5 (2019), 5571–5589.Google ScholarDigital Library
- [5] , Wenhao Ying, and Lu Zhao. 2020. Fast Enhanced Exemplar-Based Clustering for Incomplete EEG Signals. Computational and Mathematical Methods in Medicine 2020, Article ID 4147807 (2020), 11 pages. Google ScholarCross Ref
- [6] . 2020. Mimamo net: Integrating micro-and macro-motion for video emotion recognition. In Proceedings of the AAAI Conference on Artificial Intelligence. 2621–2628.Google ScholarCross Ref
- [7] . 2021. A convolution bidirectional long short-term memory neural network for driver emotion recognition. IEEE Transactions on Intelligent Transportation Systems 22, 7 (2021), 4570–4578.
DOI: Google ScholarDigital Library - [8] . 2016. Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. 445–450.Google ScholarDigital Library
- [9] . 2021. Privacy protection for medical data sharing in smart healthcare. ACM Transactions on Multimedia Computing Communications and Applications 16, 3s (2021), 1–18.Google ScholarDigital Library
- [10] . 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972–976.
DOI: Google ScholarCross Ref - [11] . 2019. Eyemotion: Classifying facial expressions in VR using eye-tracking cameras. In Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision. 1626–1635.
DOI: Google ScholarCross Ref - [12] . 2020. Facial expression recognition under partial occlusion from virtual reality headsets based on transfer learning. In Proceedings of the 2020 IEEE 6th International Conference on Multimedia Big Data. 70–75.
DOI: Google ScholarCross Ref - [13] . 2012. Transfer spectral clustering. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 789–803.Google ScholarCross Ref
- [14] . 2016. Emonets: Multimodal deep learning approaches for emotion recognition in video. Journal on Multimodal User Interfaces 10, 2 (2016), 99–111.Google ScholarCross Ref
- [15] . 2017. Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Transactions on Affective Computing 10, 2 (2017), 223–236.Google ScholarCross Ref
- [16] . 2018. A brief review of facial emotion recognition based on visual information. Sensors 18, 2 (2018), 401.Google ScholarCross Ref
- [17] . 2020. Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing (2020), 1–1.
DOI: Google ScholarCross Ref - [18] . 2010. The extended cohn-kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. 94–101.Google ScholarCross Ref
- [19] . 2018. Facial emotion recognition: A survey and real-world user experiences in mixed reality. Sensors 18, 2 (2018), 416–440.
DOI: Google ScholarCross Ref - [20] . 2019. Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks. Journal of Visual Communication and Image Representation 59 (2019), 176–185. Google ScholarDigital Library
- [21] . 2015. A survey on data stream clustering and classification. Knowledge and Information Systems 45, 3 (2015), 535–569.Google ScholarDigital Library
- [22] . 2021. Survey on emotional body gesture recognition. IEEE Transactions on Affective Computing 12, 2 (2021), 505–523.
DOI: Google ScholarDigital Library - [23] . 2017. Audio-visual emotion recognition in video clips. IEEE Transactions on Affective Computing 10, 1 (2017), 60–75.Google ScholarDigital Library
- [24] . 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1–9.Google ScholarCross Ref
- [25] . 2021. Pinball loss twin support vector clustering. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 2s (2021).Google ScholarDigital Library
- [26] . 2021. Classification of alzheimer’s disease using ensemble of deep neural networks trained through transfer learning. IEEE Journal of Biomedical and Health Informatics 26, 4 (2021), 1453–1463.
DOI: Google ScholarCross Ref - [27] . 2021. Neural style palette: A multimodal and interactive style transfer from a single style image. IEEE Transactions on Multimedia 23 (2021), 2245–2258.Google ScholarDigital Library
- [28] . 2018. Intelligent facial emotion recognition based on stationary wavelet entropy and jaya algorithm. Neurocomputing 272 (2018), 668–676.Google ScholarDigital Library
- [29] . 2022. Two-stage fuzzy fusion based-convolution neural network for dynamic emotion recognition. IEEE Transactions on Affective Computing 13, 2 (2022), 805–817.Google Scholar
- [30] . 2021. Physiological signals based affective computing: A systematic review. Acta Automatica Sinica 47, x (2021).
DOI: Google ScholarCross Ref - [31] . 2013. Clustering based on enhanced expansion move. IEEE Transaction on Knowledge and Data Engineering 25, 10 (2013), 2206–2216.
DOI: 109/TKDE.2012.202Google Scholar - [32] . 2017. Seizure classification from EEG signals using transfer learning, semi-supervised learning and TSK fuzzy system. IEEE Transactions on Neural Systems and Rehabilitation Engine 25, 12 (2017), 2270–2284.
DOI: Google ScholarCross Ref - [33] . 2019. Exemplar-based data stream clustering toward internet of things. The Journal of Supercomputing 76, 7 (2019).Google Scholar
- [34] . 2020. Constructing a personalized cross-day EEG-based emotion-classification model using transfer learning. IEEE Journal of Biomedical and Health Informatics 24, 5 (2020), 1255–1264.
DOI: Google ScholarCross Ref - [35] . 2018. Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273 (2018), 643–649. Google ScholarCross Ref
- [36] . 2020. Driver emotion recognition for intelligent vehicles: A survey. ACM Computing Surveys 53, 3 (2020), 1–30.
DOI: Google ScholarDigital Library - [37] . 2017. BAUM-1: A spontaneous audio-visual face database of affective and mental states. IEEE Transactions on Affective Computing 8, 3 (2017), 300–313.Google ScholarDigital Library
- [38] . 2020. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Information Fusion 59 (2020), 103–126.Google ScholarCross Ref
- [39] . 2016. Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation. IEEE Access 4 (2016), 8375–8385.Google ScholarCross Ref
- [40] . 2019. A novel double-index-constrained, multi-view, fuzzy-clustering algorithm and its application for detecting epilepsy electroencephalogram signals. IEEE Access 7 (2019), 103823–103832.Google ScholarCross Ref
Index Terms
- Dynamic Transfer Exemplar based Facial Emotion Recognition Model Toward Online Video
Recommendations
Facial Emotion Recognition with Varying Poses and/or Partial Occlusion Using Multi-stage Progressive Transfer Learning
Image AnalysisAbstractThis paper describes the use of multi-stage Progressive Transfer Learning (MSPTL) to improve the performance of automated Facial Emotion Recognition (FER). Our proposed FER solution is designed to work with 2D images, and is able to classify ...
Toward Personalized Emotion Recognition: A Face Recognition Based Attention Method for Facial Emotion Recognition
2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021)This paper aims to address the subject-dependent challenge of the facial emotion recognition (FER) task. To accomplish this, we propose a novel face recognition based attention FER (FRA-FER) framework which propagates subtle face recognition (FR) features ...
Facial emotion recognition based on deep transfer learning approach
AbstractFacial expressions play a major role in the communication of emotions through nonverbal channels. In recent years, the topic of automatic facial expression recognition (FER) has become very popular. Researchers are looking at how it may be used in ...
Comments