Skip to main content

Advertisement

Springer Nature Link
Log in

A deep-learning-based facial expression recognition method using textural features

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Human facial expression and emotion play pivotal roles in our day-to-day communication, and detecting them are one of the formidable tasks in the field of human–computer interfaces (HCI). This paper presents a new facial expressions detection method by exploiting textural image features such as local binary patterns (LBP), local ternary patterns (LTP) and completed local binary pattern (CLBP). This paper utilizes the advantages of textural features which are highly correlated with the facial expression changes and thereby trains a convolution neural network (CNN) model to detect facial expressions. The CNN model is trained on the images from the extended Cohn-Kanade (CK +), JAFEE and FER2013 datasets that are converted into LBP, LTP and CLBP image features. The performance of our facial expression recognition system is validated on modified CK+, JAFEE and FER2013 dataset. The results reported here illustrates that the CNN model yields better efficiency when we train the model with textural images. Moreover, we have shown that the CNN model trained with CLBP outperforms than that of with LBP and LTP images. In case of CLBP images, accuracies are 91.0%, 82.2% and 64.5% for CK+, JAFFE and FER2013 dataset, respectively. In case of LBP, accuracies are 79.5%, 75% and 58.45% and in case of LTP images accuracies are 89.2%, 77.3% and 62.79% for the datasets CK+, JAFFE and FER2013, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

All data analysed are included in this paper.

References

  1. Dzedzickis A, Kaklauskas A, Bucinskas V (2020) Human emotion recognition: review of sensors and methods. Sensors (Basel) 20(3):1–40

    Article  Google Scholar 

  2. Mukhopadhyay M, Pal S, Nayyar A, Dutta Pramanik PK, Dasgupta N, Choudhury P, Facial emotion detection to assess learner’s state of mind in an online learning system. In: Proceedings of the 2020 5th international conference on intelligent information technology, Hanoi (2020)

  3. Verma A, Singh P, Rani Alex JS (2019) Modified convolutional neural network architecture analysis for facial emotion recognition. In: 2019 international conference on systems, signals and image processing (IWSSIP), Osijek

  4. Ozdemir MA, Elagoz B, Alaybeyoglu A, Sadighzadeh R, Akan A (2019) Real time emotion recognition from facial expressions using CNN architecture. In: 2019 medical technologies congress (TIPTEKNO), pp 1–4, https://doi.org/10.1109/TIPTEKNO.2019.8895215., Izmir

  5. Krithika LB, Lakshmi Priya GG (2016) Student emotion recognition system (SERS) for e-learning improvement based on learner concentration metric. In: International conference on computational modeling and security (CMS 2016), Bangalore

  6. Suk M, Prabhakaran B (2015) Real-time facial expression recognition on smartphones. In: 2015 IEEE winter conference on applications of computer vision, Waikoloa, HI

  7. Perveen N, Ahmad N, Bilal Khan MAQD, Khalid R, Qadri S (2016) Facial expression recognition through machine learning. Int J Sci Technol Res 5(3):91–97

    Google Scholar 

  8. Ashir AM, Eleyan A, Akdemir B (2019) Facial expression recognition with dynamic cascaded classifier. Neural Comput Appl 32:6295–6309

    Article  Google Scholar 

  9. Murugappan M, Mutawa A (2021) Facial geometric feature extraction based emotional expression classification using machine learning algorithms. PLoS ONE. https://doi.org/10.1371/journal.pone.0247131

    Article  Google Scholar 

  10. Adholiya A (2021) Facial expression recognition using CNN with Keras. Biosc Biotech Res Commun Special Issue 14(5):47–50

    Article  Google Scholar 

  11. Praveen R, Benjula Anbu Malar B (2020) Emotion recognition using convolutional neural network. Int J Recent Technol Eng (IJRTE) 8(6):1748–1765

    Article  Google Scholar 

  12. KumarJain D, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recogn Lett 120:69–74

    Article  Google Scholar 

  13. Chowdary MK, Nguyen TN, Hemanth DJ (2021) Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput Appl. https://doi.org/10.1007/s00521-021-06012-8

    Article  Google Scholar 

  14. Choi I-K, Ahn H-E, Yoo J (2018) Facial expression classification using deep convolutional neural network. J Electr Eng Technol 13(1):485–492

    Google Scholar 

  15. Li S, Deng W (2020) Deep facial expression recognition: a survey. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2022.3205170

    Article  Google Scholar 

  16. Jaiswal A, Raju AK, Deb S (2020) Facial emotion detection using deep learning. In: International conference for emerging technology (INCET), Belgaum

  17. Minaee S, Mehdi M, Abdolrashidi A (2021) Deep-emotion: facial expression recognition using attentional convolutional network. MDPI 21:1–16

    Google Scholar 

  18. Fan Y, Lu X, Li D, Liu Y (2016) Video-based emotion recognition using CNN-RNN and C3D hybrid networks. In: Proceedings of the 18th ACM international conference on multimodal interaction, Tokyo

  19. Mohan K, Seal A, Krejcar O, Yazidi A (2021) FER-net: facial expression recognition using deep neural net. Neural Comput Appl 33:9125–9136

    Article  Google Scholar 

  20. Ojala T, Ainen MP, Aenp TM (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  21. Mukhopadhyay M, Dey A, Shaw RN, Ghosh A (2021) Facial emotion recognition based on textural pattern and convolutional neural network. In: IEEE 4th international conference on computing, power and communication technologies (GUCON), Kuala Lumpur

  22. Lakshmi D, Ponnusamy R (2021) Facial emotion recognition using modified HOG and LBP features with deep stacked autoencoders. Microprocess Microsyst 82:1–9

    Article  Google Scholar 

  23. Yu L, Li Q, Wang H, Shi C (2021) Improved MB-LBP feature extraction algorithm based on reduced-dimensional HOG. In: IEEE international conference on mechatronics and automation (ICMA), Takamatsu

  24. Wei X, Yu X, Liu B, Zhi L (2019) Convolutional neural networks and local binary patterns for hyperspectral image classification. Eur J Remote Sens 52(1):448–462

    Article  Google Scholar 

  25. Verma A, Tyagi D, Sharma S (2016) Recent advancement of LBP techniques: a survey. In: International conference on computing, communication and automation (ICCCA), Greater Noida

  26. Slimani K, Kas M, El Merabet Y, Messoussi R, Ruichek Y (2018) Facial emotion recognition: a comparative analysis using 22 LBP variants. In: MedPRAI '18: Proceedings of the 2nd Mediterranean conference on pattern recognition and artificial intelligence, Rabat Morocco

  27. VenkataRamiReddy C, Kishore KK, Bhattacharyya D, Kim TH (2014) Multi-feature fusion based facial expression classification using DLBP and DCT. Int J Software Eng Appl 8(9):55–68

    Google Scholar 

  28. Jun H, Feng C J, Zhi FL, Wen HZ (2015) A method of facial expression recognition based on LBP fusion of key expressions areas. In: The 27th Chinese control and decision conference (2015 CCDC), Qingdao

  29. Vikas, Kaur A (2014) Face recognition using local ternary pattern for low resolution image. Int J Sci Res 3(12):1494–1497

    Google Scholar 

  30. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 9(6):1635–1650

    MathSciNet  MATH  Google Scholar 

  31. Slimani K, Kas M, Merabet YE, Messoussi R, Ruichek Y (2018) Facial emotion recognition: a comparative analysis using 22 LBP variants. In: MedPRAI'18: proceedings of the 2nd Mediterranean conference on pattern recognition and artificial intelligence, Kenitra

  32. Yee SY, Rassem TH, Mohammed MF, Makbol NM (2019) Performance evaluation of completed local ternary pattern (CLTP) for face image recognition. Int J Adv Comput Sci Appl 10(4):379–387

    Google Scholar 

  33. Ravi R, Yadhukrishna SV, Prithviraj R (2020) A face expression recognition using CNN & LBP. In: Fourth international conference on computing methodologies and communication (ICCMC), Erode

  34. Rani PE (2017) Face recognition based on LBP & LTP features. Int J Pure Appl Math 117(22):85–89

    Google Scholar 

  35. Chen J, Chen Z, Chi Z, Fu H (2018) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput 9(1):38–50

    Article  Google Scholar 

  36. Alizadeh S, Fazel A (2017) Convolutional neural networks for facial expression recognition. Comput Vis Pattern Recogn 1–8

  37. Zhao X, Shi X, Zhang S (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):1–9

    Article  Google Scholar 

  38. Kim DH, Baddar WJ, Jang J, Ro YM (2019) Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput 10:223–236

    Article  Google Scholar 

  39. Pons G, Masip D (2018) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9(3):343–350

    Article  Google Scholar 

  40. Villanueva MG, Zavala SR (2020) Deep neural network architecture: application for facial expression recognition. IEEE Latin Am Trans 8(7):1311–1319

    Article  Google Scholar 

  41. Liu X, Kumar BV, You J, Jia P (2017) Adaptive deep metric learning for identity-aware facial expression recognition. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), Honolulu, HI

  42. Zhang T, Liu Z, Wang X-H, Xing X-F, Chen CLP, Chen E (2018) Facial expression recognition via broad learning system. In: IEEE international conference on systems, man, and cybernetics (SMC), Miyazaki

  43. Orozco D, Lee CT, Arabadzhi Y, Gupta D (2018) Transfer learning for facial expression recognition. Technical report. Florida State University, USA, pp 1–7

  44. Sawardekar S, Naik PSR (2018) Facial expression recognition using efficient LBP and CNN. Int Res J Eng Technol (IRJET) 5(6):2273–2277

    Google Scholar 

  45. Ravi R, Yadhukrishna S, Prithviraj R (2020) A face expression recognition using CNN & LBP. In: Proceedings of the fourth international conference on computing methodologies and communication (ICCMC 2020), Erode

  46. Jumani SZ, Ali F, Kandhro IA, Lakhan QA, Ali U, Haroon MW, Ahmed S (2019) Facial emotion identification based on local binary pattern feature detector. Indian J Sci Technol 12(28):1–6

    Article  Google Scholar 

  47. Yang X, Li M, Zhao S (2017) Facial expression recognition algorithm based on CNN and LBP feature fusion,” in Proceedings of the 2017 international conference on robotics and artificial intelligence, Shanghai

  48. Sharma M, Jalal AS, Khan A (2019) Emotion recognition using facial expression by fusing key points descriptor and texture features. Multimedia Tools Appl 78:16195–16219

    Article  Google Scholar 

  49. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  50. Xi Z, Niu Y, Chen J, Kan X, Liu H (2021) Facial expression recognition of industrial internet of things by parallel neural networks combining texture features”. IEEE Trans Ind Inform 17(4):2784–2793

    Article  Google Scholar 

  51. Guo Z, Zhang L, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663

    Article  MathSciNet  MATH  Google Scholar 

  52. Wang Y, Ai H, Wu B, Huang C (2004) Real time facial expression recognition with AdaBoost. In: Proceedings of the 17th international conference on pattern recognition, 2004. ICPR 2004, Cambridge

  53. Ou J, Bai XB, Pei Y, Ma L, Liu W (2010) Automatic facial expression recognition using gabor filter and expression analysis. In: Second international conference on computer modeling and simulation, Sanya

  54. Reddy KS, Kumar VV, Reddy BE (2015) Face recognition based on texture features using local ternary patterns. IJ Image Graph Signal Process 7(10):37–46

    Google Scholar 

  55. Tripathi M (2021) Facial emotion recognition using convolution neural network. ICTACT J Image Video Process 12(1):2531–2536

    Google Scholar 

  56. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The Extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE computer society conference on computer vision and pattern recognition-workshops, San Francisco

  57. Benitez-Quiroz CF, Srinivasan R, Martinez AM (2019) Discriminant functional learning of color features for the recognition of facial action units and their intensities. IEEE Trans Pattern Anal Mach Intell 41(12):2835–2845

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science Engineering, Bengal Institute of Technology, Bantala, Kolkata, India

    Moutan Mukhopadhyay

  2. Department of Information Technology, MAKAUT, Kolkata, India

    Aniruddha Dey

  3. TCS-Research and Innovation, Kolkata, India

    Sayan Kahali

Authors
  1. Moutan Mukhopadhyay
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Aniruddha Dey
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Sayan Kahali
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Aniruddha Dey.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mukhopadhyay, M., Dey, A. & Kahali, S. A deep-learning-based facial expression recognition method using textural features. Neural Comput & Applic 35, 6499–6514 (2023). https://doi.org/10.1007/s00521-022-08005-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-08005-7

Keywords