Skip to main content
SpringerLink
Log in

The facial expression recognition technology under image processing and neural network

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

A facial expression recognition (FER) algorithm is built on the advanced convolutional neural network (CNN) to improve the current FER algorithms’ recognition rate. The advanced CNN model (the ExpressionNet model), containing two continuous convolutional (Conv) layers and one maximum cache layer, is obtained through the AlexNet CNN. The proposed algorithm is compared with the SingleNet model, CNN with three Conv layers, and the AlexNet model through simulation experiments. The experiment results show that the ExpressionNet model takes the longest training time and test time, followed by the AlexNet and the SingleNet. In terms of recognition rate, ExpressionNet (77%) is superior to AlexNet (72.5%) and SingleNet (69.5%); however, its convergence rate is slightly slower than the other two models. The ExpressionNet model has only one layer more than that of the AlexNet model. Therefore, the advanced CNN-based FER algorithm is of great significance to theoretical research and practical application of the FER technology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Wang Y, Yan J, Yang Z, Zhao Y, Liu T (2020) GIS partial discharge pattern recognition via lightweight convolutional neural network in the ubiquitous power internet of things context. IET Sci Meas Technol 14(8):864–871

    Article  Google Scholar 

  2. Minaee S, Abdolrashidi A (2019) Deep-emotion: facial expression recognition using attentional convolutional network, vol 15, pp 114–123. https://arxiv.org/pdf/1902.01019.pdf

  3. Zeng N, Zhang H, Baoye L, Weibo L (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 15:15–23

    Google Scholar 

  4. Mittal U, Sharma M (2021) Artificial intelligence and its application in different areas of indian economy. Int J Adv Res Sci Commun Technol 125:125–131

    Google Scholar 

  5. Lyons MJ, Budynek J (2019) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 26:352–363

    Google Scholar 

  6. Lopes AT, Aguiar ED, Souza A, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  7. Bahreini K, Wim V, Westera W (2019) A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimed Tools Appl 78(14):18943–18966

    Article  Google Scholar 

  8. Vardhana M, Arunkumar N, Lasrado S, Abdulhay E, Ramirez-Gonzalez G (2018) Convolutional neural network for bio-medical image segmentation with hardware acceleration. Cogn Syst Res 50:10–14

    Article  Google Scholar 

  9. Kvasic I, Miskovic N, Vukic Z (2019) Convolutional neural network architectures for sonar-based diver detection and tracking. In: OCEANS 2019 - Marseille, vol 195. IEEE, pp 25–31

  10. An S, Ji LJ, Michael M, Zhang Z (2017) Two sides of emotion: exploring positivity and negativity in six basic emotions across cultures. Front Psychol 8:257–263

    Google Scholar 

  11. Kemnitz J, Eckstein F, Culvenor AG, Ruhdorfer A, Wirth W (2017) Validation of an active shape model-based semi-automated segmentation algorithm for the analysis of thigh muscle and adipose tissue cross-sectional areas. Magn Reson Mater Phys Biol Med 30(28):489–503

    Article  Google Scholar 

  12. Cohen RF, Tubiana PA, Kahn JP (2015) French validation of the “reading the mind in the eyes test”: relation with subclinical psychotic positive symptoms in general population. Eur Psychiatry 30:1226–1226

    Article  Google Scholar 

  13. Baspinar E, Sarti A, Citti G (2020) A sub-Riemannian model of the visual cortex with frequency and phase. J Math Neurosci 10(1):114–121

    Article  MathSciNet  MATH  Google Scholar 

  14. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355(25):82–92

    Article  Google Scholar 

  15. Garcia M, Ramirez S (2020) Deep neural network architecture: application for facial expression recognition. IEEE Lat Am Trans 8(7):1311–1319

    Article  Google Scholar 

  16. Krithika LB, Priya G (2020) Graph based feature extraction and hybrid classification approach for facial expression recognition. J Ambient Intell Humaniz Comput 2:1–17

    Google Scholar 

  17. Shah JH, Sharif M, Yasmin M, Fernandes SL (2017) Facial expressions classification and false label reduction using LDA and threefold SVM. Pattern Recogn Lett 234:167865517302271–167865517302283

    Google Scholar 

  18. Sadeghi H, Raie AA (2019) Human vision inspired feature extraction for facial expression recognition. Multimed Tools Appl 78(21):30335–30353

    Article  Google Scholar 

  19. Hassan MM, Alam M, Uddin MZ, Huda S, Almogren A, Fortino G (2018) Human emotion recognition using deep belief network architecture. Inf Fusion 51:10–18

    Article  Google Scholar 

  20. Zangeneh E, Rahmati M, Mohsenzadeh Y (2017) Low resolution face recognition using a two-branch deep convolutional neural network architecture. Expert Syst Appl 124:12–16

    Google Scholar 

  21. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proc. Intern. Workshop on Emotion Corpora for Research on Emotion & Affect vol 25, pp 316–323

  22. Kas M, Merabet YE, Messoussi R, Ruichek Y (2020) New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach. Inf Sci 6:549–554

    Google Scholar 

  23. Xiao S, Man L, Quan C, Ren F (2017) Improved facial expression recognition method based on ROI deep convolutional neutral network. In: Seventh International Conference on Affective Computing & Intelligent Interaction, vol 25. IEEE Computer Society, pp 142–153

  24. Chen Y, Ming D, Lv X (2019) Superpixel based land cover classification of VHR satellite image combining multi-scale CNN and scale parameter estimation. Earth Sci Inf 23:266–278

    Google Scholar 

  25. Li S, Deng W, Du JP (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol 152. IEEE, pp 1142–1153

  26. Meng H, Bianchi BN, Deng Y, Cheng J, Cosmas JP (2017) Time-delay neural network for continuous emotional dimension prediction from facial expression sequences. IEEE Trans Cybern 46(4):916–929

    Article  Google Scholar 

  27. Ding H, Zhou SK, Chellappa R (2016) Facenet2expnet: regularizing a deep face recognition net for expression recognition, vol 152. IEEE, pp 1136–1141

  28. Liu X, Ge Y, Yang C, Jia P (2018) Adaptive metric learning with deep neural networks for video-based facial expression recognition. J Electron Imaging 27(1):406–414

    Article  Google Scholar 

  29. Kim BK, Dong SY, Roh J, Kim G, Lee SY (2016) Fusing aligned and non-aligned face information for automatic affect recognition in the wild: a deep learning approach. In: Computer Vision & Pattern Recognition Workshops, vol 23. IEEE, pp 115–123

  30. Kim B-K, Lee H, Roh J, Lee S-Y (2015) Hierarchical committee of deep CNNs with exponentially-eeighted decision fusion for static facial expression recognition. In: ACM on International Conference on Multimodal Interaction, vol 3. ACM, pp 142–153

  31. Pons G, Masip D (2018) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9(3):343–350

    Article  Google Scholar 

  32. Connie T, Al-Shabi M, Cheah WP, Goh M (2016) Facial expression recognition using a hybrid CNN-SIFT aggregator, vol 26. Springer, pp 114–121

    Google Scholar 

  33. Lei X, Fei M, Zhou W, Yang A (2018) Face expression recognition based on convolutional neural network*. In: 2018 Australian & New Zealand Control Conference (ANZCC), vol 63, pp 147–151

  34. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process Publ IEEE Signal Process Soc 2:1–1

    MATH  Google Scholar 

  35. Wen G, Zhi H, Li H, Li D, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput 9(4):1–14

    Google Scholar 

  36. Li Y, Wang G, Nie L, Wang Q, Tan W (2018) Distance metric optimization driven convolutional neural network for age invariant face recognition. Pattern Recogn J Pattern Recogn Soc 23:152–163

    Google Scholar 

  37. Zhu Y, Jiang Y (2020) Optimization of face recognition algorithm based on deep learning multi feature fusion driven by big data—ScienceDirect. Image Vis Comput 104:114–121

    Article  Google Scholar 

  38. Hu L, Cui J (2019) Digital image recognition based on fractional-order-PCA-SVM coupling algorithm. Measurement 145:150–159

    Article  Google Scholar 

  39. Yayilgan SY, Arifaj B, Rahimpour M, Hardeberg JY, Ahmedi L (2020) Pre-trained CNN based deep features with hand-crafted features and patient data for skin lesion classification. Lect Notes Comput Sci 5805:58–63

    Google Scholar 

  40. Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411:115–121

    Google Scholar 

  41. Luh GC, Wu HB, Yong YT, Lai YJ, Chen YH (2020) Facial expression based emotion recognition employing YOLOv3 deep neural networks. IEEE 4:25–36

    Google Scholar 

  42. ALiang TN (2018) Contentious North Korean disarmament prospects. In: Security, economics and nuclear non-proliferation morality, pp 36–43

  43. Mollahosseini A, Hasani B, Mahoor MH (1949) Affectnet: a database for facial expression, valence, and arousal computing in the wild. In: IEEE Transactions on Affective Computing, vol 34, pp 59–63

  44. Zeng J, Zhao X, Qin C, Lin Z (2018) Single sample per person face recognition based on deep convolutional neural network. In: 2017 3rd IEEE International Conference on Computer and Communications (ICCC), vol 58. IEEE, pp 114–121

  45. Long B, Yu K, Qin J (2017) Data augmentation for unbalanced face recognition training sets. Neurocomputing 235(26):10–14

    Article  Google Scholar 

  46. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), vol 2, pp 442–451

  47. Wang B, Tian R (2019) Judgement of critical state of water film rupture on corrugated plate wall based on SIFT feature selection algorithm and SVM classification method. Nucl Eng Des 347:132–139

    Article  Google Scholar 

  48. Wei L, Min L, Zhong S, Zhu Z (2015) A deep-learning approach to facial expression recognition with candid images. In: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), vol 23. IEEE, pp 134–142

  49. Wang F, Lv J, Ying G, Chen S, Zhang C (2019) Facial expression recognition from image based on hybrid features understanding. J Vis Commun Image Represent 59:84–88

    Article  Google Scholar 

  50. Islam B, Mahmud F, Hossain A (2019) High performance facial expression recognition system using facial region segmentation, fusion of HOG & LBP features and multiclass SVM. In: 2018 10th International Conference on Electrical and Computer Engineering (ICECE), vol 25, pp 114–121

  51. Chen M (2018) The research of human individual’s conformity behavior in emergency situations. Libr Hi Tech 38(3):593–609

    Article  Google Scholar 

  52. Shen CW, Min C, Wang C (2019) Analyzing the trend of O2O commerce by bilingual text mining on social media. Comput Hum Behav 101:474–483

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Scientific research project of Inner Mongolia College and University (No. NJZY17266); Scientific research fund of Baotou Medical College (No. BYJJ-QM201777).

Author information

Authors and Affiliations

  1. Ticketing Center, Shenzhen Metro Group Co., Ltd., Shenzhen, China

    Dezhu Zhao

  2. School of Science, Hubei University of Technology, Wuhan, China

    Yufeng Qian

  3. Qingdao University of Science & Technology, Gaomi Campus, Weifang, Shandong, China

    Jun Liu

  4. Department of Computer Science and Technology, Baotou Medical College, Inner Mongolia University of Science and Technology, Baotou, 014040, China

    Min Yang

Authors
  1. Dezhu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yufeng Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Min Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, D., Qian, Y., Liu, J. et al. The facial expression recognition technology under image processing and neural network. J Supercomput 78, 4681–4708 (2022). https://doi.org/10.1007/s11227-021-04058-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-04058-y

Keywords

Search

Navigation