Skip to main content
SpringerLink
Log in

An accurate recognition of facial expression by extended wavelet deep convolutional neural network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expressions are essential in community based interactions and in the analysis of emotions behaviour. The automatic identification of face is a motivating topic for the researchers because of its numerous applications like health care, video conferencing, cognitive science etc. In the computer vision with the facial images, the automatic detection of facial expression is a very challenging issue to be resolved. An innovative methodology is introduced in the presented work for the recognition of facial expressions. The presented methodology is described in subsequent stages. At first, input image is taken from the facial expression database and pre-processed with high frequency emphasis (HFE) filtering and modified histogram equalization (MHE). After the process of image enhancement, Viola Jones (VJ) framework is utilized to detect the face in the images and also the face region is cropped by finding the face coordinates. Afterwards, different effective features such as shape information is extracted from enhanced histogram of gradient (EHOG feature), intensity variation is extracted with mean, standard deviation and skewness, facial movement variation is extracted with facial action coding (FAC),texture is extracted using weighted patch based local binary pattern (WLBP) and spatial information is extracted byentropy based Spatial feature. Subsequently, dimensionality of the features are reduced by attaining the most relevant features using Residual Network (ResNet). Finally, extended wavelet deep convolutional neural network (EWDCNN) classifier uses the extracted features and accurately detects the face expressions as sad, happy, anger, fear disgust, surprise and neutral classes. The implementation platform used in the work is PYTHON. The presented technique is tested with the three datasets such as JAFFE, CK+ and Oulu-CASIA.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

References

  1. Al-Dabagh MZN, Alhabib MHM, Al-Mukhtar FH (2018) Face recognition system based on kernel discriminant analysis, k-nearest neighbor and support vector machine. International Journal of Research and Engineering 5(3):335–338

    Article  Google Scholar 

  2. Alenazy WM and Alqahtani AS (2020) Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. Journal of ambient intelligence and humanized computing 1-16.

  3. Amani N, Shahbahrami A, Nahvi M (2013) A new approach for face image enhancement and recognition. International Journal of Advanced Science and Technology 52(01):1–10

    Google Scholar 

  4. Arora M, Kumar M (2021) AutoFER: PCA and PSO based automatic facial emotion recognition. Multimed Tools Appl 80(2):3039–3049

    Article  Google Scholar 

  5. Ashir AM, Eleyan A, Akdemir B (2020) Facial expression recognition with dynamic cascaded classifier. Neural Comput & Applic 32(10):6295–6309

    Article  Google Scholar 

  6. Bansal M, Kumar M, Kumar M (2021) 2D object recognition: a comparative analysis of SIFT, SURF and ORB feature descriptors. Multimed Tools Appl 80(12):18839–18857

    Article  Google Scholar 

  7. Bargshady G, Zhou X, Deo RC, Soar J, Whittaker F, Wang H (2020) Enhanced deep learning algorithm development to detect pain intensity from facial expression images. Expert Syst Appl 149:113305

    Article  Google Scholar 

  8. Chen J, Chen Z, Chi Z, Fu H (2016) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput 9(1):38–50

    Article  Google Scholar 

  9. Chhabra P, Garg NK, Kumar M (2020) Content-based image retrieval system using. ORB and SIFT features Neural Computing and Applications 32(7):2725–2733

    Article  Google Scholar 

  10. Dino HI and Abdulrazzaq MB (2019) Facial expression classification based on SVM, KNN and MLP classifiers. In 2019 international conference on advanced science and engineering (ICOASE), IEEE 70-75.

  11. Du L, Hu H (2019) Weighted patch-based manifold regularization dictionary pair learning model for facial expression recognition using iterative optimization classification strategy. Comput Vis Image Underst 186:13–24

    Article  Google Scholar 

  12. Dubey AK and Jain V (2019) Comparative study of convolution neural network’s relu and leaky-relu activation functions. In Applications of computing, Automation and Wireless Systems in Electrical Engineering, Springer, Singapore, 873–880.

  13. Dubey AK, Jain V (2019) A review of face recognition methods using deep learning network. J Inf Optim Sci 40(2):547–558

    Google Scholar 

  14. Dubey AK, Jain V (2020) Automatic facial recognition using VGG16 based transfer learning model. J Inf Optim Sci 41(7):1589–1596

    Google Scholar 

  15. Dubey AK and Jain V (2020) Automatic facial expression recognition based on deep layered representation of convolution neural networks. In proceedings of 3rd international conference on computing informatics and networks: ICCIN, springer nature 65

  16. Gautam G, Choudhary K, Chatterjee S, and Kolekar MH (2017) Facial expression recognition using Krawtchouk moments and support vector machine classifier. In 2017 fourth international conference on image information processing (ICIIP), IEEE 1-6.

  17. Georgescu MI, Ionescu RT, Popescu M (2019) Local learning with deep and handcrafted features for facial expression recognition. IEEE Access 7:64827–64836

    Article  Google Scholar 

  18. González-Hernández F, Zatarain-Cabada R, Barrón-Estrada ML, Rodríguez-Rangel H (2018) Recognition of learning-centered emotions using a convolutional neural network. Journal of Intelligent & Fuzzy Systems 34(5):3325–3336

    Article  Google Scholar 

  19. Han B, Yun WH, Yoo JH, Kim WH (2020) Toward unbiased facial expression recognition in the wild via cross-dataset adaptation. IEEE Access 8:159172–159181

    Article  Google Scholar 

  20. Happy SL, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  21. Kalsum T, Anwar SM, Majid M, Khan B, Ali SM (2018) Emotion recognition from facial expressions using hybrid feature descriptors. IET Image Process 12(6):1004–1012

    Article  Google Scholar 

  22. Kamarol SKA, Jaward MH, Parkkinen J, Parthiban R (2016) Spatiotemporal feature extraction for facial expression recognition. IET Image Process 10(7):534–541

    Article  Google Scholar 

  23. Kas M, Ruichek Y, Messoussi R (2021) New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach. Inf Sci 549:200–220

    Article  MathSciNet  Google Scholar 

  24. Kawakami T, Murahira K and Taguchi A (2009) Modified histogram equalization with variable enhancement degree for image contrast enhancement. Intelligent signal processing and communication systems, 570-573

  25. Kola DGR, Samayamantula SK (2020) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl:1–20

  26. Kumar M, Kumar M (2021) XGBoost: 2D-object recognition using shape descriptors and extreme gradient boosting classifier. Computational Methods and Data Engineering:207–222

  27. Kumar M, Chhabra P, Garg NK (2018) An efficient content-based image retrieval system using BayesNet and K-NN. Multimed Tools Appl 77(16):21557–21570

    Article  Google Scholar 

  28. Kumar A, Kaur A, Kumar M (2019) Face detection techniques: a review. Artif Intell Rev 52(2):927–948

    Article  Google Scholar 

  29. Kumar A, Kumar M, Kaur A (2021) Face detection in still images under occlusion and non-uniform illumination. Multimed Tools Appl 80(10):14565–14590

    Article  Google Scholar 

  30. Lalitha SD and Thyagharajan KK (2020) Micro-facial expression recognition based on deep-rooted learning algorithm.arXiv preprint arXiv: 2009.05778.

  31. Li J, Jin K, Zhou D, Kubota N, Ju Z (2020) Attention mechanism-based CNN for facial expression recognition. Neurocomputing 411:340–350

    Article  Google Scholar 

  32. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  33. Mahmood MR, Abdulrazzaq MB, Zeebaree SR, Ibrahim AK, Zebari RR, Dino HI (2021) Classification techniques’ performance evaluation for facial expression recognition. Indonesian Journal of Electrical Engineering and Computer Science 21(2):176–1184

    Google Scholar 

  34. Makhmudkhujaev F, Abdullah-Al-Wadud M, Iqbal MTB, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12

    Article  Google Scholar 

  35. Meng Z, Liu P, Cai J, Han S and Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In 2017 12th IEEE international conference on Automatic Face & Gesture Recognition (FG 2017), IEEE 558-565.

  36. Mistry K, Zhang L, Neoh SC, Lim CP, Fielding B (2016) A micro-GA embedded PSO feature selection approach to intelligent facial emotion recognition. IEEE transactions on cybernetics 47(6):1496–1509

    Article  Google Scholar 

  37. Mohan K, Seal A, Krejcar O, Yazidi A (2020) Facial expression recognition using local gravitational force descriptor-based deep convolution neural networks. IEEE Trans Instrum Meas 70:1–12

    Article  Google Scholar 

  38. Mollahosseini A, Chan D and Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In 2016 IEEE winter conference on applications of computer vision (WACV), IEEE 1–10.

  39. Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features. arXiv preprint arXiv:1408.3750.

  40. Owayjan M, Achkar R and Iskandar M (2016) Face detection with expression recognition using artificial neural networks. Middle East Conference on Biomedical Engineering, IEEE, 15–119.

  41. Singh S, Ahuja U, Kumar M, Kumar K, Sachdeva M (2021) Face mask detection using YOLOv3 and faster R-CNN models: COVID-19 environment. Multimed Tools Appl 80(13):19753–19768

    Article  Google Scholar 

  42. Sun A, Li Y, Huang YM, Li Q, Lu G (2018) Facial expression recognition using optimized active regions. Human-centric Computing and Information Sciences 8(1):1–24

    Article  Google Scholar 

  43. Turan C, Lam K-M (2018) Histogram-based local descriptors for facial expression recognition (fer): a comprehensive study. J Vis Commun Image Represent 55:331–341

    Article  Google Scholar 

  44. Uçar A, Demir Y, Güzeliş C (2016) A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering. Neural Comput & Applic 27(1):131–142

    Article  Google Scholar 

  45. Wu B-F, Lin C-H (2018) Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE access 6:12451–12461

    Article  Google Scholar 

  46. Zeng N, Zhang H, Song B, Liu W, Li Y, Dobaie AM (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273:643–649

    Article  Google Scholar 

  47. Zhang Z, Luo P, Loy CC, Tang X (2018) From facial expression recognition to interpersonal relation prediction. Int J Comput Vis 126(5):550–569

    Article  MathSciNet  Google Scholar 

  48. Zhang F, Zhang T, Mao Q, Xu C (2020) Geometry guided pose-invariant facial expression recognition. IEEE Trans Image Process 29:4445–4460

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Research Scholar, University School of Information, Communication and Technology, GGSIPU, Dwarka, Delhi, India

    Arun Kumar Dubey

  2. Bharati Vidyapeeth’s College of Engineering, New Delhi, India

    Vanita Jain

Authors
  1. Arun Kumar Dubey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vanita Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arun Kumar Dubey.

Ethics declarations

Conflict of interest

Authors Arun Kumar Dubey and Vanita Jain declared that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dubey, A.K., Jain, V. An accurate recognition of facial expression by extended wavelet deep convolutional neural network. Multimed Tools Appl 81, 28295–28325 (2022). https://doi.org/10.1007/s11042-022-12871-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12871-7

Keywords

Search

Navigation