Abstract
Facial emotion recognition (FER) from videos is now considered a significant role in HCI (Human-Computer Interaction). The dynamic variations shown by various facial movements need to be realized quickly without degrading the recognition performance. Therefore, the procedure of classifying the facial emotions from videos is now demanded as a challenging and interesting issue. This work proposes a practical methodology for identifying emotions through facial expressions from videos. At first, the Lucas–Kanade (LK) based optical flow scheme is used for motion detection from the input videos. After finding the set of LK frames, the pre-processing scheme is applied. In this process, the Viola-Jones algorithm is utilized for face detection, and then the gray scale conversion is involved. Moreover, the FAST corner detection approach is used to detect the facial landmark points over the gray scale frame. The Neighborhood Difference Features (NDF) are extracted in feature extraction (FE). The optimal set of features is selected from the mined features using the Modified Plant Genetics-Inspired Evolutionary Optimization (MPGEO) algorithm in the feature selection (FS). Finally, the chosen features are fed into the Deep Attention-based Bidirectional LSTM with Equilibrium Optimizer (DABLEO) classifier for the emotion classification. The proposed scheme is performed in the Python software using four standard datasets like FAMED, CK+, AFEW, and MMI, and it delivers the classification accuracy of 96.5%, 99.2%, 90%, and 92% individually. As related to other schemes, the proposed scheme is better for all performances of emotion recognition.
Similar content being viewed by others
Data availability
Data sharing not applicable to this article as no datasets were generated or analysed during the current study.
References
Abdallah BT, Guermazi R, Hammami M (2020) Using Normal/abnormal video sequence categorization to efficient facial expression recognition in the wild. In: International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, Cham, 504–516
Abdulsalam WH, Alhamdani RS, Abdullah MN (2019) Facial emotion recognition from videos using deep convolutional neural networks. Int J Mach Learn Comput 9(1):14–19
Alreshidi A, Ullah M (2020) Facial emotion recognition using hybrid features. Informatics, Multidisciplinary Digital Publishing Institute 7(1): 6
Al-Tuwaijari JM, Shaker SA (2020) Face detection system based Viola-Jones algorithm. In: 2020 6th international engineering conference sustainable technology and development(IEC), IEEE, 211-215
Basbrain A, Gan JQ (2020) One-shot only real-time video classification: a case study in facial emotion recognition. In: International conference on intelligent data engineering and automated learning. Springer, Cham, pp 197–208
Demochkina P, Savchenko AV (2021) MobileEmotiFace: efficient facial image representations in video-based emotion recognition on mobile devices. In: International conference on pattern recognition. Springer, Cham, pp 266–274
Dey T, Deb T (2015) Facial landmark detection using FAST corner detector of UGC-DDMC face database of Tripura tribes. In: Proceedings of the 2015 third international conference on computer, communication, control and information technology (C3IT), IEEE, pp 1-4
Dhall A, Goecke R, Lucey S, Gedeon T (2011) Acted facial expressions in the wild database. Australian National University, Canberra, Australia, technical report TR-CS-11: 2 1
Du Z, Wu S, Huang D, Li W, Wang Y (2019) Spatio-temporal encoder-decoder fully convolutional network for video-based dimensional emotion recognition. IEEE Trans Affect Comput 12(3):565–578
Fan Y, Lam JCK, Li VO (2018) Video-based emotion recognition using deeply-supervised neural networks. In: Proceedings of the 20th ACM international conference on multimodal interaction, pp 584-588
Faramarzi A, Heidarinejad M, Stephens B, Mirjalili S (2020) Equilibrium optimizer: a novel optimization algorithm. Knowl-Based Syst 191:105190
Gautam KS, Thangavel SK (2021) Video analytics-based facial emotion recognition system for smart buildings. Int J Comput Appl 43(9):858–867
Gupta R, Vishwamitra LK (2021) Facial expression recognition from videos using CNN and feature aggregation. Mater Today Proc
Gupta N, Khosravy M, Patel N, Mahela OP, Varshney G (2020) Plant genetics-inspired evolutionary optimization: a descriptive tutorial. In: Frontier applications of nature inspired computation. Springer, Singapore, pp 53–77
Haddad J, Lézoray O, Hamel P (2020) 3d-cnn for facial emotion recognition in videos. In: International symposium on visual computing. Springer, Cham, pp 298–309
Hajarolasvadi N, Bashirov E, Demirel H (2021) Video-based person-dependent and person-independent facial emotion recognition. SIViP 15(5):1049–1056
Hossain SM, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Inf Fusion 49:69–78
Hu M, Wang H, Wang X, Yang J, Ronggui Wang R (2019) Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks. J Vis Commun Image Represent 59:176–185
Huang J, Li Y, Tao J, Lian Z, Yi J (2018) End-to-end continuous emotion recognition from video using 3D ConvLSTM networks. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 6837–6841
Knyazev B, Shvetsov R, Efremova N, Kuharenko A (2018) Leveraging large face recognition data for emotion classification. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, pp 692-696
Li Y, Tao J, Schuller B, Shan S, Jiang D, Jia J (2018) Mec 2017: Multimodal emotion recognition challenge. In: 2018 First Asian conference on affective computing and intelligent interaction (ACII Asia), IEEE, pp 1–5
Liu X, Ge Y, Yang C, Jia P (2018) Adaptive metric learning with deep neural networks for video-based facial expression recognition. J Electron Imaging 27(1):013022
Longmore CA, Tree JJ (2013) Motion as a cue to face recognition: evidence from congenital prosopagnosia. Neuropsychologia 51(5):864–875
Lou L, Liang S, Zhang Y (2019) Application research of moving target detection based on optical flow algorithms. In: Journal of physics: conference series, IOP Publishing, 1237(2): 022073
Lu C, Zheng WW, Li C, Tang C, Liu S, Yan S, Zong Y (2018) Multiple spatio-temporal feature learning for video-based emotion recognition in the wild. In: Proceedings of the 20th ACM international conference on multimodal interaction, pp 646-652
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94-101
Meng D, Peng X, Wang K, Qiao Y (2019) Frame attention networks for facial expression recognition in videos. In: 2019 IEEE international conference on image processing (ICIP), IEEE, pp 3866-3870
Mo S, Niu J, Su Y, Das (2018) A novel feature set for video emotion recognition. Neurocomputing 291: 11–20
Ngoc TQ, Lee SS, Song BC (2020) Facial landmark-based emotion recognition via directed graph neural network. Electronics 9(5):764
Pan X, Ying G, Chen G, Li H, Li W (2019) A deep spatial and temporal aggregation framework for video-based facial expression recognition. IEEE Access 7:48807–48815
Pan X, Zhang S, Guo W, Zhao X, Chuang Y, Chen Y, Zhang H (2020) Video-based facial expression recognition using deep temporal–spatial networks. IETE Tech Rev 37(4):402–409
Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and expo, IEEE, 5
Priya RV (2019) Emotion recognition from geometric fuzzy membership functions. Multimed Tools Appl 78(13):17847–17878
Rajan S, Chenniappan P, Devaraj S, Madian N (2020) Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process 14(7):1373–1381
Rocktäschel T, Grefenstette E, Hermann KM, Kočiský T, Blunsom P (2015) Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664
Samadiani N, Huang G, Luo W, Chi CH, Shu Y, Wang R, Kocaturk T (2022) A multiple feature fusion framework for video emotion recognition in the wild. Concurr Comput Pract Exp 34(8):e5764
Sepas-Moghaddam A, Etemad A, Pereira F, Correia PL (2020) Facial emotion recognition using light field images with deep attention-based bidirectional LSTM. In: ICASSP 2020–2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 3367–3371
Smith KE, Leitzke BT, Pollak SD (2020) Youths’ processing of emotion information: responses to chronic and video-based laboratory stress. Psychoneuroendocrinology 122:104873
Sreenivas V, Namdeo V, Kumar EV (2020) Group based emotion recognition from video sequence with hybrid optimization based recurrent fuzzy neural network. J Big Data 7(1):1–21
Sun M-C, Hsu S-H, Yang M-C, Chien J-H (2018) Context-aware cascade attention-based RNN for video emotion recognition. In: 2018 First Asian conference on affective computing and intelligent interaction (ACII Asia), IEEE, pp 1–6
Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimed Tools Appl 79:21487–21512
Xing B, Zhang H, Zhang K, Zhang L, Wu X, Shi X, Yu S, Zhang S (2019) Exploiting EEG signals and audiovisual feature fusion for video emotion recognition. IEEE Access 7:59844–59861
Zhang S, Pan X, Cui Y, Zhao X, Liu L (2019 Mar 4) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We declare that there is no conflict of interest.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Vedantham, R., Reddy, E.S. Facial emotion recognition on video using deep attention based bidirectional LSTM with equilibrium optimizer. Multimed Tools Appl 82, 28681–28711 (2023). https://doi.org/10.1007/s11042-023-14491-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-14491-1