Skip to main content

Advertisement

SpringerLink
Log in

Optimized face-emotion learning using convolutional neural network and binary whale optimization

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Human emotion detection using facial expressions might be easy for humans, but computing technology to accomplish the same task is more challenging. We can recognize emotions from images using the latest computer vision and machine learning (ML) advancements. This research proposes a novel optimized face emotion learning method with binary whale optimization (OFELBW). The OFELBW is implemented in three phases, the first phase with a convolutional neural network (CNN) in which from the image the background noise is removed in the initial phase, and the facial feature extraction is performed in the second phase. Finally, the binary whale optimization algorithm is used for the feature selection to obtain the most relevant feature subset. The proposed OFELBW method was examined with more than 750 K images using SFEW, CK+, JAFFE, and FERG datasets. We have compared our proposed OFELBW model with other existing techniques to examine the accuracy of our models with the above-mentioned datasets and received an accuracy of 98.35% with the CK+ dataset, 99.42% with the FERG dataset, 96.6% with the JAFFE dataset and 64.98% with the SFEW with 80% training, 10% testing, and 10% validation set. This technique will be useful in various applications such as human social/physiological interaction systems, mental disease diagnosis and military environment, etc.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The dataset used in this work can be accessed from [5, 10, 29, 32].

References

  1. Abadi, M (2016) TensorFlow: learning functions at scale. In proceedings of the 21st ACM SIGPLAN international conference on functional programming (pp. 1-1). https://doi.org/10.1145/3022670.2976746

  2. Ali, MF, Khatun, M, Turzo, NA (2020) Facial emotion detection using neural network. Int J Sci Eng Res

  3. Ambert-Dahan E, Giraud AL, Mecheri H, Sterkers O, Mosnier I, Samson S (2017) Emotional recognition of dynamic facial expressions before and after cochlear implantation in adults with progressive deafness. Hear Res 354:64–72

    Article  Google Scholar 

  4. Bairaju, SPR, Ari, S, Garimella, RM (2019) Emotion detection using visual information with deep auto-encoders. In 2019 IEEE 5th international conference for convergence in technology (I2CT) (pp. 1-5). IEEE. https://doi.org/10.1109/i2ct45611.2019.9033902

  5. Burns EJ, Martin J, Chan AH, Xu H (2017) Impaired processing of facial happiness, with or without awareness, in developmental prosopagnosia. Neuropsychologia 102:217–228. https://doi.org/10.1016/j.neuropsychologia.2017.06.020

  6. Ch S (2021) An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimed Tools Appl 80(12):17543–17568. https://doi.org/10.1007/s11042-021-10547-2

  7. Dantas, AC, Do Nascimento, MZ (2022) Recognition of emotions for people with autism: An approach to improve skills. Int J Comput Games Technol, 2022. https://doi.org/10.1155/2022/6738068

  8. Dantas, AC, do Nascimento, MZ (2022) Face emotions: improving emotional skills in individuals with autism. Multimed Tools Appl, 1–23

  9. Demochkina, P, Savchenko, AV (2021) Neural network model for video-based facial expression recognition in-the-wild on mobile devices. In 2021 international conference on information technology and nanotechnology (ITNT) (pp. 1-5). https://doi.org/10.1109/itnt52450.2021.9649076

  10. Dhall, A, Goecke, R, Lucey, S, Gedeon, T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In 2011 IEEE international conference on computer vision workshops (ICCV workshops) (pp. 2106-2112). IEEE

  11. Feutry, C, Piantanida, P, Bengio, Y, Duhamel, P (2018) Learning anonymized representations with adversarial neural networks. arXiv preprint arXiv:1802.09386. (pp. 1-20)

  12. Georgescu MI, Ionescu RT, Popescu M (2019) Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7:64827–64836

    Article  Google Scholar 

  13. Giannopoulos, P, Perikos, I, Hatzilygeroudis, I (2018) Deep learning approaches for facial emotion recognition: a case study on FER-2013. In advances in hybridization of intelligent methods (pp. 1–16). Springer, Cham

  14. Gogić I, Manhart M, Pandžić IS, Ahlberg J (2020) Fast facial expression recognition using local binary features and shallow neural networks. Vis Comput 36(1):97–112. https://doi.org/10.1007/s00371-018-1585-8

  15. Greco, A, Strisciuglio, N, Vento, M, Vigilante, V (2022) Benchmarking deep networks for facial emotion recognition in the wild. Multimed Tools Appl, 1–32

  16. Han, S, Meng, Z, Khan, AS, Tong, Y (2016) Incremental boosting convolutional neural network for facial action unit recognition. Adv Neural Inf Proces Syst, 29

  17. Hasani, B, Mahoor, MH (2017) Facial expression recognition using enhanced deep 3D convolutional neural networks. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 30-40)

  18. Hong SW, Yoon KL (2018) Intensity dependence in high-level facial expression adaptation aftereffect. Psychon Bull Rev 25(3):1035–1042

    Article  Google Scholar 

  19. Hossain S, Umer S, Asari V, Rout RK (2021) A unified framework of deep learning-based facial expression recognition system for diversified applications. Appl Sci 11(19):9174. https://doi.org/10.3390/app11199174

  20. Hu, J, Shen, L, Sun, G (2018) Squeeze-and-excitation networks. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141)

  21. Ismail, HA, Hashim, IA, Abd, BH (2019) A survey on linguistic interpretation of facial expressions and technologies. In 2019 2nd international conference on engineering technology and its applications (IICETA) (pp. 161-166). IEEE

  22. Izen SC, Ciaramitaro VM (2020) A crowd of emotional voices influences the perception of emotional faces: using adaptation, stimulus salience, and attention to probe audio-visual interactions for emotional stimuli. Attention Percept Psycho 82(8):3973–3992

    Article  Google Scholar 

  23. Kazemi, V, Sullivan, J (2014) One millisecond face alignment with an ensemble of regression trees. In proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1867-1874). https://doi.org/10.1109/cvpr.2014.241

  24. Kowalski, M, Naruniec, J, Trzcinski, T (2017) Deep alignment network: a convolutional neural network for robust face alignment. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 88-97)

  25. Kumar A, Jaiswal A, Garg S, Verma S, Kumar S (2019) Sentiment analysis using cuckoo search for optimized feature selection on Kaggle tweets. Int J Inf Retriev Res (IJIRR) 9(1):1–15

    Google Scholar 

  26. Kumar M, Jindal MK, Kumar M (2021) A novel attack on monochrome and greyscale Devanagari CAPTCHAs. Trans Asian Low-Resource Language Inf Process 20(4):1–30

    Article  Google Scholar 

  27. Kumar M, Jindal MK, Kumar M (2022) Distortion, rotation and scale invariant recognition of hollow Hindi characters. Sādhanā 47(2):1–6. https://doi.org/10.1145/3439798

  28. Li S, Deng W (2019) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE transactions on image processing, image processing, IEEE transactions on. IEEE Trans Image Process 28(1):356–370

    Article  MathSciNet  Google Scholar 

  29. Lyons, MJ, Kamachi, M, Gyoba, J (2014) Japanese female facial expressions (JAFFE), Database Digit Images 2007

  30. Mehendale N (2020) Facial emotion recognition using convolutional neural networks (FERC). SN Appl Sci 2(3):1–8. https://doi.org/10.1007/s42452-020-2234-1

  31. Meng, Z, Liu, P, Cai, J, Han, S, Tong, Y (2017) Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE international conference on Automatic Face & Gesture Recognition (FG 2017) (pp. 558-565)

  32. Minaee S, Minaei M, Abdolrashidi A (2021) Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9) 3046:1–16

    Article  Google Scholar 

  33. Mohammed AR, Kosonogov V, Lyusin D (2021) Expressive suppression versus cognitive reappraisal: effects on self-report and peripheral psychophysiology. Int J Psychophysiol 167:30–37. https://doi.org/10.1016/j.ijpsycho.2021.06.007

  34. Mollahosseini A, Hasani B, Mahoor MH (2017) Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31. https://doi.org/10.1109/taffc.2017.2740923

  35. Müller T, Schäfer R, Hahn S, Franz M (2019) Adults' facial reaction to affective facial expressions of children and adults. Int J Psychophysiol 139:33–39. https://doi.org/10.1016/j.ijpsycho.2019.01.001

  36. Ning, GY, Cao, DQ (2021) Improved Whale Optimization Algorithm for Solving Constrained Optimization Problems Discrete Dyn Nature Soc, 2021. https://doi.org/10.1155/2021/8832251

  37. Otberdout N, Kacem A, Daoudi M, Ballihi L, Berretti S (2019) Automatic analysis of facial expressions based on deep covariance trajectories. IEEE Trans Neural Netw Learn Syst 31(10):3892–3905. https://doi.org/10.1109/tnnls.2019.2947244

  38. Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41(7):3383–3390. https://doi.org/10.1016/j.eswa.2013.11.041

  39. Qi L, Binu D, Rajakumar BR, Mohammed Ismail B (2022) 2-D canonical correlation analysis-based image super-resolution scheme for facial emotion recognition. Multimed Tools Appl 81(10):13911–13934. https://doi.org/10.1007/s11042-022-11922-3

  40. Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimed Tools Appl 80(16):25241–25253

    Article  Google Scholar 

  41. Shetty AB, Rebeiro J (2021) Facial recognition using Haar cascade and LBP classifiers. Global Trans Proceed 2(2):330–335

    Article  Google Scholar 

  42. Shima, Y, Omori, Y (2018) Image augmentation for classifying facial expression images by using deep neural network pre-trained with object image database. In proceedings of the 3rd international conference on robotics, control and automation (pp. 140-146). https://doi.org/10.1145/3265639.3265664

  43. Sun W, Zhao H, Jin Z (2017) An efficient unconstrained facial expression recognition algorithm based on stack binarized auto-encoders and binarized neural networks. Neurocomputing 267:385–395

    Article  Google Scholar 

  44. Sun W, Zhao H, Jin Z (2018) A visual attention based ROI detection method for facial expression recognition. Neurocomputing 296:12–22

    Article  Google Scholar 

  45. Tautkute, I, Trzcinski, T, Bielski, A (2018) I know how you feel: emotion recognition with facial landmarks. In proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 1878-1880)

  46. Tejada, J, Freitag, RMK, Pinheiro, BFM, Cardoso, PB, Souza, VRA, Silva, LS (2021) Building and validation of a set of facial expression images to detect emotions: a transcultural study. Psychol Res, 1–11

  47. Teufel C, Westlake MF, Fletcher PC, von dem Hagen E (2019) A hierarchical model of social perception: psychophysical evidence suggests late rather than early integration of visual information from facial expression and body posture. Cognition 185:131–143. https://doi.org/10.1016/j.cognition.2018.12.012

  48. Varcin KJ, Nangle MR, Henry JD, Bailey PE, Richmond JL (2019) Intact spontaneous emotional expressivity to non-facial but not facial stimuli in schizophrenia: an electromyographic study. Schizophr Res 206:37–42. https://doi.org/10.1016/j.schres.2018.12.019

  49. Verma B, Choudhary A (2021) Affective state recognition from hand gestures and facial expressions using Grassmann manifolds. Multimed Tools Appl 80(9):14019–14040

    Article  Google Scholar 

  50. Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069. https://doi.org/10.1109/tip.2019.2956143

  51. Wong SF, Trespalacios F, Ellenbogen MA (2020) Poor inhibition of personally-relevant facial expressions of sadness and anger predicts an elevated cortisol response following awakening six months later. Int J Psychophysiol 150:73–82

    Article  Google Scholar 

  52. Wu Y, Ji Q (2019) Facial landmark detection: a literature survey. Int J Comput Vis 127(2):115–142

    Article  Google Scholar 

  53. Yu, J, Yu, L (2018) Synthesizing photo-realistic 3D talking head: learning lip synchronicity and emotion from audio and video. In 2018 25th IEEE international conference on image processing (ICIP) (pp. 1448-1452)

  54. Zhang, Z, Luo, P, Loy, CC, Tang, X (2014) Facial landmark detection by deep multi-task learning. In European conference on computer vision (pp. 94-108). Springer, Cham. https://doi.org/10.1007/978-3-319-10599-4_7

  55. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 99:1–9

    Google Scholar 

  56. Zhao, H, Liu, Q, Yang, Y (2018) Transfer learning with ensemble of multiple feature representations. In 2018 IEEE 16th international conference on software engineering research, management and applications (SERA) (pp. 54-61)

  57. Zhao Y, Oveneke MC, Jiang D, Sahli H (2019) A video prediction approach for animating single face image. Multimed Tools Appl 78(12):16389–16410

    Article  Google Scholar 

Download references

Funding

The author declares that they do not have any funding or grant for the manuscript.

Author information

Authors and Affiliations

  1. School of Information Technology and Engineering, VIT University, Vellore, India

    T. Muthamilselvan & K. Brindha

  2. School of Computer Science and Engineering in Vellore Institute of Technology, Vellore, India

    Sudha Senthilkumar

  3. Tata Consultancy Services (TCS), Pune, India

    Saransh

  4. Department of IT, Lord Buddha Education Foundation, Kathmandu, Nepal

    Jyotir Moy Chatterjee

  5. Department of Computer Science and Information Management, Providence University, Taipei, Taiwan, China

    Yu-Chen Hu

Authors
  1. T. Muthamilselvan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. K. Brindha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sudha Senthilkumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saransh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jyotir Moy Chatterjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yu-Chen Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jyotir Moy Chatterjee.

Ethics declarations

Ethics approval

No animals were involved in this study. All applicable international, national, and/or institutional guidelines for the care and use of animals were followed.

Conflict of interest

The authors declare that they do not have any conflicts of interest that influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Muthamilselvan, T., Brindha, K., Senthilkumar, S. et al. Optimized face-emotion learning using convolutional neural network and binary whale optimization. Multimed Tools Appl 82, 19945–19968 (2023). https://doi.org/10.1007/s11042-022-14124-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14124-z

Keywords

Search

Navigation