Skip to main content
Log in

Facial emotion recognition on video using deep attention based bidirectional LSTM with equilibrium optimizer

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial emotion recognition (FER) from videos is now considered a significant role in HCI (Human-Computer Interaction). The dynamic variations shown by various facial movements need to be realized quickly without degrading the recognition performance. Therefore, the procedure of classifying the facial emotions from videos is now demanded as a challenging and interesting issue. This work proposes a practical methodology for identifying emotions through facial expressions from videos. At first, the Lucas–Kanade (LK) based optical flow scheme is used for motion detection from the input videos. After finding the set of LK frames, the pre-processing scheme is applied. In this process, the Viola-Jones algorithm is utilized for face detection, and then the gray scale conversion is involved. Moreover, the FAST corner detection approach is used to detect the facial landmark points over the gray scale frame. The Neighborhood Difference Features (NDF) are extracted in feature extraction (FE). The optimal set of features is selected from the mined features using the Modified Plant Genetics-Inspired Evolutionary Optimization (MPGEO) algorithm in the feature selection (FS). Finally, the chosen features are fed into the Deep Attention-based Bidirectional LSTM with Equilibrium Optimizer (DABLEO) classifier for the emotion classification. The proposed scheme is performed in the Python software using four standard datasets like FAMED, CK+, AFEW, and MMI, and it delivers the classification accuracy of 96.5%, 99.2%, 90%, and 92% individually. As related to other schemes, the proposed scheme is better for all performances of emotion recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

References

  1. Abdallah BT, Guermazi R, Hammami M (2020) Using Normal/abnormal video sequence categorization to efficient facial expression recognition in the wild. In: International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, Cham, 504–516

  2. Abdulsalam WH, Alhamdani RS, Abdullah MN (2019) Facial emotion recognition from videos using deep convolutional neural networks. Int J Mach Learn Comput 9(1):14–19

    Article  Google Scholar 

  3. Alreshidi A, Ullah M (2020) Facial emotion recognition using hybrid features. Informatics, Multidisciplinary Digital Publishing Institute 7(1): 6

  4. Al-Tuwaijari JM, Shaker SA (2020) Face detection system based Viola-Jones algorithm. In: 2020 6th international engineering conference sustainable technology and development(IEC), IEEE, 211-215

  5. Basbrain A, Gan JQ (2020) One-shot only real-time video classification: a case study in facial emotion recognition. In: International conference on intelligent data engineering and automated learning. Springer, Cham, pp 197–208

  6. Demochkina P, Savchenko AV (2021) MobileEmotiFace: efficient facial image representations in video-based emotion recognition on mobile devices. In: International conference on pattern recognition. Springer, Cham, pp 266–274

  7. Dey T, Deb T (2015) Facial landmark detection using FAST corner detector of UGC-DDMC face database of Tripura tribes. In: Proceedings of the 2015 third international conference on computer, communication, control and information technology (C3IT), IEEE, pp 1-4

  8. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Acted facial expressions in the wild database. Australian National University, Canberra, Australia, technical report TR-CS-11: 2 1

  9. Du Z, Wu S, Huang D, Li W, Wang Y (2019) Spatio-temporal encoder-decoder fully convolutional network for video-based dimensional emotion recognition. IEEE Trans Affect Comput 12(3):565–578

  10. Fan Y, Lam JCK, Li VO (2018) Video-based emotion recognition using deeply-supervised neural networks. In: Proceedings of the 20th ACM international conference on multimodal interaction, pp 584-588

  11. Faramarzi A, Heidarinejad M, Stephens B, Mirjalili S (2020) Equilibrium optimizer: a novel optimization algorithm. Knowl-Based Syst 191:105190

    Article  Google Scholar 

  12. Gautam KS, Thangavel SK (2021) Video analytics-based facial emotion recognition system for smart buildings. Int J Comput Appl 43(9):858–867

  13. Gupta R, Vishwamitra LK (2021) Facial expression recognition from videos using CNN and feature aggregation. Mater Today Proc

  14. Gupta N, Khosravy M, Patel N, Mahela OP, Varshney G (2020) Plant genetics-inspired evolutionary optimization: a descriptive tutorial. In: Frontier applications of nature inspired computation. Springer, Singapore, pp 53–77

  15. Haddad J, Lézoray O, Hamel P (2020) 3d-cnn for facial emotion recognition in videos. In: International symposium on visual computing. Springer, Cham, pp 298–309

  16. Hajarolasvadi N, Bashirov E, Demirel H (2021) Video-based person-dependent and person-independent facial emotion recognition. SIViP 15(5):1049–1056

    Article  Google Scholar 

  17. Hossain SM, Muhammad G (2019) Emotion recognition using deep learning approach from audio–visual emotional big data. Inf Fusion 49:69–78

    Article  Google Scholar 

  18. Hu M, Wang H, Wang X, Yang J, Ronggui Wang R (2019) Video facial emotion recognition based on local enhanced motion history image and CNN-CTSLSTM networks. J Vis Commun Image Represent 59:176–185

    Article  Google Scholar 

  19. Huang J, Li Y, Tao J, Lian Z, Yi J (2018) End-to-end continuous emotion recognition from video using 3D ConvLSTM networks. In: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 6837–6841

  20. Knyazev B, Shvetsov R, Efremova N, Kuharenko A (2018) Leveraging large face recognition data for emotion classification. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), IEEE, pp 692-696

  21. Li Y, Tao J, Schuller B, Shan S, Jiang D, Jia J (2018) Mec 2017: Multimodal emotion recognition challenge. In: 2018 First Asian conference on affective computing and intelligent interaction (ACII Asia), IEEE, pp 1–5

  22. Liu X, Ge Y, Yang C, Jia P (2018) Adaptive metric learning with deep neural networks for video-based facial expression recognition. J Electron Imaging 27(1):013022

    Article  Google Scholar 

  23. Longmore CA, Tree JJ (2013) Motion as a cue to face recognition: evidence from congenital prosopagnosia. Neuropsychologia 51(5):864–875

    Article  Google Scholar 

  24. Lou L, Liang S, Zhang Y (2019) Application research of moving target detection based on optical flow algorithms. In: Journal of physics: conference series, IOP Publishing, 1237(2): 022073

  25. Lu C, Zheng WW, Li C, Tang C, Liu S, Yan S, Zong Y (2018) Multiple spatio-temporal feature learning for video-based emotion recognition in the wild. In: Proceedings of the 20th ACM international conference on multimodal interaction, pp 646-652

  26. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94-101

  27. Meng D, Peng X, Wang K, Qiao Y (2019) Frame attention networks for facial expression recognition in videos. In: 2019 IEEE international conference on image processing (ICIP), IEEE, pp 3866-3870

  28. Mo S, Niu J, Su Y, Das (2018) A novel feature set for video emotion recognition. Neurocomputing 291: 11–20

  29. Ngoc TQ, Lee SS, Song BC (2020) Facial landmark-based emotion recognition via directed graph neural network. Electronics 9(5):764

    Article  Google Scholar 

  30. Pan X, Ying G, Chen G, Li H, Li W (2019) A deep spatial and temporal aggregation framework for video-based facial expression recognition. IEEE Access 7:48807–48815

    Article  Google Scholar 

  31. Pan X, Zhang S, Guo W, Zhao X, Chuang Y, Chen Y, Zhang H (2020) Video-based facial expression recognition using deep temporal–spatial networks. IETE Tech Rev 37(4):402–409

  32. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and expo, IEEE, 5

  33. Priya RV (2019) Emotion recognition from geometric fuzzy membership functions. Multimed Tools Appl 78(13):17847–17878

    Article  Google Scholar 

  34. Rajan S, Chenniappan P, Devaraj S, Madian N (2020) Novel deep learning model for facial expression recognition based on maximum boosted CNN and LSTM. IET Image Process 14(7):1373–1381

    Article  Google Scholar 

  35. Rocktäschel T, Grefenstette E, Hermann KM, Kočiský T, Blunsom P (2015) Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664

  36. Samadiani N, Huang G, Luo W, Chi CH, Shu Y, Wang R, Kocaturk T (2022) A multiple feature fusion framework for video emotion recognition in the wild. Concurr Comput Pract Exp 34(8):e5764

  37. Sepas-Moghaddam A, Etemad A, Pereira F, Correia PL (2020) Facial emotion recognition using light field images with deep attention-based bidirectional LSTM. In: ICASSP 2020–2020 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 3367–3371

  38. Smith KE, Leitzke BT, Pollak SD (2020) Youths’ processing of emotion information: responses to chronic and video-based laboratory stress. Psychoneuroendocrinology 122:104873

    Article  Google Scholar 

  39. Sreenivas V, Namdeo V, Kumar EV (2020) Group based emotion recognition from video sequence with hybrid optimization based recurrent fuzzy neural network. J Big Data 7(1):1–21

    Article  Google Scholar 

  40. Sun M-C, Hsu S-H, Yang M-C, Chien J-H (2018) Context-aware cascade attention-based RNN for video emotion recognition. In: 2018 First Asian conference on affective computing and intelligent interaction (ACII Asia), IEEE, pp 1–6

  41. Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimed Tools Appl 79:21487–21512

  42. Xing B, Zhang H, Zhang K, Zhang L, Wu X, Shi X, Yu S, Zhang S (2019) Exploiting EEG signals and audiovisual feature fusion for video emotion recognition. IEEE Access 7:59844–59861

    Article  Google Scholar 

  43. Zhang S, Pan X, Cui Y, Zhao X, Liu L (2019 Mar 4) Learning affective video features for facial expression recognition via hybrid deep learning. IEEE Access 7:32297–32304

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ramachandran Vedantham.

Ethics declarations

Conflict of interest

We declare that there is no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vedantham, R., Reddy, E.S. Facial emotion recognition on video using deep attention based bidirectional LSTM with equilibrium optimizer. Multimed Tools Appl 82, 28681–28711 (2023). https://doi.org/10.1007/s11042-023-14491-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14491-1

Keywords

Navigation