Skip to main content

Advertisement

SpringerLink
Log in

Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Facial expression recognition (FER) is an essential part of effective human–computer interaction and serves as a helpful medium for children and patients who have problems with communication. However, most of the previous studies focus on building a FER model based on supervised and unsupervised approaches. This paper is focused on a semi-supervised deep belief network (DBN) approach to predict the facial expressions from the CK+, Oulu CASIA, MMI, and JAFFE datasets. To achieve accurate classification of the facial expressions, a gravitational search algorithm (GSA) is applied to optimize some parameters in the DBN network. The Histogram oriented gradients (HOG) and 2D-Discrete Wavelet Transform (2D-DWT) are used for feature extraction from the lip, cheek, brow, eye, and furrow patches. The unwanted information present in the image is eliminated using a feature selection approach. The feature extraction is done by the Kernel-principal component analysis to obtain higher-order correlations between input variables and detect non-linear components. The HOG features extracted from the lip patch provides the best performance for accurate facial expression classification. Finally, a comparative analysis to compare the proposed model with different machine learning techniques based on the evaluation criteria. The results demonstrate that DBN-GSA based classifier is more accurate than the rest of the classifiers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Abbreviations

DBN:

Deep belief network

FER:

Facial expression recognition

GSA:

Gravitational search algorithm

HOG:

Histogram oriented gradients

2D-DWT:

2D-Discrete Wavelet transform

DR:

Dimensionality reduction

SVM:

Support-vector machines

DBTR-BR:

Deep convolutional neural network, discrete Beamlet transform regularization-blind restoration

MSCNN-PHRNN:

Multi-signal convolutional neural network part-based hierarchical bidirectional neural network

CK+:

Cohn Kanade

MDSTFN:

Multi-channel deep spatial–temporal feature neural network

MMI:

Man–machine interaction

PHRNN:

Part-based hierarchical bidirectional neural network

MSCNN:

Multi-signal convolutional neural network

Oulu-CASIA:

Oulu-Chinese Academy of Science Institute of Automation

SMO:

Spider monkey optimization

DCT:

Discrete Cosine transform

DISFA+:

Extended denver intensity of spontaneous facial actions

EEG:

Electroencephalography

RBM:

Restricted Boltzmann machine

GO:

Gradient and orientation

LPF:

Low pass filter

HPF:

High pass filter

PCA:

Principal component analysis

(DR):

Dimensionality reduction

(CD):

Hinton contrast divergence

(LLE):

Locally linear embedding

(t-SNE):

t-Distributed stochastic neighbor embedding

(ROC):

Receiver operating characteristic curve

(AUC):

Area under the ROC curve

References

  • Cheng F, Yu J, Xiong H (2010) Facial expression recognition in JAFFE dataset based on Gaussian process classification. IEEE Trans Neural Netw 21(10):1685–1690

    Article  Google Scholar 

  • Dahmouni A, Aharrane N, Moutaouakil KE, Satori K (2016) A new hybrid face recognition system via local gradient probabilistic pattern (LGPP) and 2D-DWT. In: Advances in intelligent systems and computing Europe and MENA cooperation advances in information and communication technologies, pp 269–278

  • Fan X, Tjahjadi T (2019) Fusing dynamic deep learned features and handcrafted features for facial expression recognition. J Vis Commun Image Represent 65:102659

    Article  Google Scholar 

  • Guha T, Yang Z, Grossman RB, Narayanan SS (2018) A computational study of expressive facial dynamics in children with autism. IEEE Trans Affect Comput 9(1):14–20

    Article  Google Scholar 

  • Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  • Hinton GE, Osindero S, Teh YW (2006) A Fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  Google Scholar 

  • Kurup AR, Ajith M, Ramón MM (2019) Semi-supervised facial expression recognition using reduced spatial features and deep belief networks. Neurocomputing 367:188–197

    Article  Google Scholar 

  • Lanihun O, Tiddeman B, Tuci E, Shaw P (2015) Improving active vision system categorization capability through histogram of oriented gradients. In: Dixon C, Tuyls K (eds) Towards autonomous robotic systems, TAROS 2015, vol 9287. Lecture Notes in Computer Science. Springer, Cham

    Google Scholar 

  • Liu Q, Liu H (2020) Criminal psychological emotion recognition based on deep learning and EEG signals. Neural Comput Appl. https://doi.org/10.1007/s00521-020-05024-0

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, pp 94–101. IEEE

  • Lv Y, Feng Z, Xu C (2014) Facial expression recognition via deep learning. In: 2014 international conference on smart computing

  • Ly TS, Do NT, Kim SH, Yang HJ, Lee GS (2019) A novel 2D and 3D multimodal approach for in-the-wild facial expression recognition. Image Vis Comput 92:103817

    Article  Google Scholar 

  • McDuff D, Kaliouby RE, Picard RW (2012) Crowdsourcing facial responses to online videos. IEEE Trans Affect Comput 3(4):456–468

    Article  Google Scholar 

  • McDuff D, Kaliouby RE, Cohn JF, Picard RW (2015) Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. IEEE Trans Affect Comput 6(3):223–235

    Article  Google Scholar 

  • Minaee S, Abdolrashidi A (2019) Deep-emotion: Facial expression recognition using attentional convolutional network. arXiv preprint arXiv:1902.01019

  • Palm G, Glodek M (2013) Towards emotion recognition in human computer interaction. In: Neural nets and surroundings. Springer, Berlin, Heidelberg, pp 323–336

    Chapter  Google Scholar 

  • Pantic M, Rothkrantz LJM (2003) Toward an affect-sensitive multimodal human-computer interaction. Proc IEEE 91(9):1370–1390

    Article  Google Scholar 

  • Rashedi E, Nezamabadi-Pour H, Saryazdi S (2009) GSA: a gravitational search algorithm. Inf Sci 179(13):2232–2248

    Article  Google Scholar 

  • Reddy GV, Savarni CD, Mukherjee S (2020) Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features. Cognit Syst Res 62:23–34

    Article  Google Scholar 

  • Rejeesh MR (2019) Interest point based face recognition using adaptive neuro fuzzy inference system. Multimed Tools Appl 78(16):22691–22710

    Article  Google Scholar 

  • Schölkopf B, Smola A, Müller KR (1997) Kernel principal component analysis In: Lecture notes in computer science artificial neural networks—ICANN97, pp 583–588

  • Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  • Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recognit Lett 119:49–61

    Article  Google Scholar 

  • Sundararaj V (2016) An efficient threshold prediction scheme for wavelet based ECG signal noise reduction using variable step size firefly algorithm. Int J Intell Eng Syst 9(3):117–126

    Google Scholar 

  • Sundararaj V (2019) Optimised denoising scheme via opposition-based self-adaptive learning PSO algorithm for wavelet-based ECG signal noise reduction. Int J Biomed Eng Technol 31(4):325

    Article  Google Scholar 

  • Sundararaj V, Muthukumar S, Kumar RS (2018) An optimal cluster formation based energy efficient dynamic scheduling hybrid MAC protocol for heavy traffic load in wireless sensor networks. Comput Secur 77:277–288

    Article  Google Scholar 

  • Tian Y, Chen S (2012) Understanding effects of image resolution for facial expression analysis. J Comput Vis Image Process

  • Tieleman T, Hinton G (2009) Using fast weights to improve persistent contrastive divergence. In: Proceedings of the 26th annual international conference on machine learning—ICML 09

  • Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: Proceedings of the 3rd international workshop on EMOTION (satellite of LREC): corpora for research on emotion and affect

  • Vedantham R, Reddy ES (2020) A robust feature extraction with optimized DBN-SMO for facial expression recognition. Multimedia Tools Appl. https://doi.org/10.1007/s11042-020-08901-x

    Article  Google Scholar 

  • Vinu S (2019) Optimal task assignment in mobile cloud computing by queue based ant-bee algorithm. Wirel Pers Commun 104(1):173–197

    Article  Google Scholar 

  • Wu H, Liu Y, Liu Y, Liu S (2019) Efficient facial expression recognition via convolution neural network and infrared imaging technology. Infrared Phys Technol 102:103031

    Article  Google Scholar 

  • Xie W, Jia X, Shen L, Yang M (2019) Sparse deep feature learning for facial expression recognition. Pattern Recognit 96:106966

    Article  Google Scholar 

  • Yang S, Kafai M, An L, Bhanu B (2014) Zapping index: using smile to measure advertisement zapping likelihood. IEEE Trans Affect Comput 5(4):432–444

    Article  Google Scholar 

  • Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  • Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

    Article  MathSciNet  Google Scholar 

  • Zhao Y, Ju YF (2018) A family of efficient appearance models based on histogram of oriented gradients (HOG), color histogram and their fusion for human pose estimation. In: Proceedings of the fifth Euro-China conference on intelligent data analysis and applications advances in intelligent systems and computing, pp 842–850

  • Zhao XM, Shi XG, Zhang SQ (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):347–355

    Article  Google Scholar 

Download references

Acknowledgment

The Authors extend their warm thanks to Deanship of Scientific Research (DSR), King Saud University, Riyadh, Saudi Arabia for permitting us to carry out the research and also for acknowledging the financial aid from DSR through the Project Group No. RG-1441-343.

Author information

Authors and Affiliations

  1. Department of Self Development Skills, Common First Year Deanship (Computer Skills), King Saud University, Riyadh, Saudi Arabia

    Wael Mohammad Alenazy & Abdullah Saleh Alqahtani

Authors
  1. Wael Mohammad Alenazy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdullah Saleh Alqahtani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wael Mohammad Alenazy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alenazy, W.M., Alqahtani, A.S. Gravitational search algorithm based optimized deep learning model with diverse set of features for facial expression recognition. J Ambient Intell Human Comput 12, 1631–1646 (2021). https://doi.org/10.1007/s12652-020-02235-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02235-0

Keywords

Search

Navigation