Skip to main content

Advertisement

Springer Nature Link
Log in

Improved convolutional neural network-based approach using hand-crafted features for facial expression recognition

  • 1226: Deep-Patterns Emotion Recognition in the Wild
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition is still one of the most attractive and challenging problems. This study designed a facial expression recognition approach based on the feature fusion strategy. In this proposed approach, two types of features are used to classify the facial expressions. The first type is deep learned features obtained from the CNN layers, and the other is hand-crafted features in which a geometric approach called DAISY is used to have a more discriminative model. The DAISY descriptor is used to extract the features because of its efficiency and performance in many problems like object detection, image classification, etc. Besides, the Convolutional Neural Network (CNN) layers are used in both standard and custom structures. A robust and highly distinguishing feature vector is conducted when these two types of features are concatenated. This feature vector helps CNN s work in an enhanced manner. The extra information provided by DAISY made it easy for the resulting model to make decisions because this feature descriptor does not require much data to work precisely. Finally, we used the Random Forest classifier for the classification task to make the proposed pipeline complete. To validate the efficiency of the proposed approach, two well-known facial expression datasets, CK+ and FER2013 are used. The proposed feature fusion-based method’s accuracy is 98.48% in the CK+ dataset and 70% in FER2013. The results are compared with some newly proposed approaches in this field to validate our strategy. Since this performance is in the range of state-of-the-art systems, the proposed strategy that enhances the CNN features by hand-crafted techniques can be presented as a suitable FER method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Agrawal A, Mittal N (2020) Using CNN for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput 36:405–412

    Article  Google Scholar 

  2. Barman A, Dutta P (2019) Facial expression recognition using distance and texture signature relevant features. Appl Soft Comput 77:88–105

    Article  Google Scholar 

  3. Benitez-Quiroz CF, Srinivasan R, Martinez AM (2018) Discriminant functional learning of color features for the recognition of facial action units and their intensities. IEEE Trans Pattern Anal Mach Intell 41:2835–2845

    Article  Google Scholar 

  4. Boughrara H, Chtourou M, Ben AC, Chen L (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75:709–731

    Article  Google Scholar 

  5. Bridge J, Harding SP, Zheng Y (2020) DAISY descriptors combined with deep learning to diagnose retinal disease from high resolution 2D OCT images. In: Communications in Computer and Information Science

  6. Cao X, Zhang H, Deng C et al (2014) Action recognition using 3D DAISY descriptor. Mach Vis Appl 25:159–171. https://doi.org/10.1007/s00138-013-0545-6

  7. Chakraborty N, Chatterjee A, Singh PK et al (2021) Application of daisy descriptor for language identification in the wild. Multimed Tools Appl 80:323–344

    Article  Google Scholar 

  8. Chatterjee A, Malakar S, Sarkar R, Nasipuri M (2018) Handwritten digit recognition using DAISY descriptor: a study. In: 2018 fifth international conference on emerging applications of information technology (EAIT). IEEE, pp 1–4

  9. Chen J, Chen Z, Chi Z, Fu H (2016) Facial expression recognition in video with multiple feature fusion. IEEE Trans Affect Comput 9:38–50

    Article  Google Scholar 

  10. Dash M, Liu H (1997) Feature selection for classification. Intel Data Anal 1:131–156

    Article  Google Scholar 

  11. Deng H-B, Jin L-W, Zhen L-X, Huang J-C (2005) A new facial expression recognition method based on local Gabor filter bank and PCA plus LDA. Int J Inf Technol 11:86–96

    Google Scholar 

  12. Fei Z, Yang E, Li DD-U et al (2020) Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388:212–227

  13. Goodfellow IJ, Erhan D, Carrier PL et al (2013) Challenges in representation learning: a report on three machine learning contests. International conference on neural information processing. Springer, In, pp 117–124

    Google Scholar 

  14. Guo J-M, Liu Y-F, Wu Z-J (2013) Duplication forgery detection using improved DAISY descriptor. Expert Syst Appl 40:707–714

    Article  Google Scholar 

  15. Happy SL, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6:1–12

    Article  Google Scholar 

  16. He Y, Chen S (2020) Person-independent facial expression recognition based on improved local binary pattern and higher-order singular value decomposition. IEEE Access 8:190184–190193

    Article  Google Scholar 

  17. Hosseini S, Cho NI (2019) GF-CapsNet: using gabor jet and capsule networks for facial age, gender, and expression recognition. In: Proceedings - 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019

  18. Hosseini S, Lee SH, Cho NI (2018) Feeding hand-crafted features for enhancing the performance of convolutional neural networks. arXiv preprint arXiv:1801.07848

  19. Hua W, Dai F, Huang L et al (2019) HERO: human emotions recognition for realizing intelligent internet of things. IEEE Access 7:24321–24332

  20. Khanbebin SN, Mehrdad V (2020) Local improvement approach and linear discriminant analysis-based local binary pattern for face recognition. Neural Comput Appl. 1–17

  21. Khanbebin SN, Mehrdad V (2020) Genetic-based feature fusion in face recognition using arithmetic coded local binary patterns. IET Image Process 14:3742–3750

    Article  Google Scholar 

  22. Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE Trans Image Process 28:2439–2450

    Article  MathSciNet  Google Scholar 

  23. Liu M, Li S, Shan S, Chen X (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing 159:126–136

    Article  Google Scholar 

  24. Liu J, Chen Y, Sun S (2019) A novel local texture feature extraction method called multi-direction local binary pattern. Multimed Tools Appl 78:18735–18750. https://doi.org/10.1007/s11042-018-7095-x

    Article  Google Scholar 

  25. Liu P, Lin Y, Meng Z et al (2021) Point adversarial self-mining: A simple method for facial expression recognition. IEEE Transactions on Cybernetics

  26. Liu C, Hirota K, Ma J et al (2021) Facial expression recognition using hybrid features of pixel and geometry. IEEE Access 9:18876–18889

  27. Lucey P, Cohn JF, Kanade T et al (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101.

  28. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. Proceedings Third IEEE international conference on automatic face and gesture recognition. IEEE, In, pp 200–205

  29. Malakar S, Ghosh M, Chaterjee A et al (2020) Offline music symbol recognition using daisy feature and quantum Grey wolf optimization based feature selection. Multimed Tools Appl 79:32011–32036

  30. Meena HK, Sharma KK, Joshi SD (2020) Effective curvelet-based facial expression recognition using graph signal processing. SIViP 14:241–247

    Article  Google Scholar 

  31. Noshad Z, Javaid N, Saba T et al (2019) Fault detection in wireless sensor networks through the random forest classifier. Sensors 19:1568

  32. Ortac G, Ozcan G (2021) Comparative study of hyperspectral image classification by multidimensional convolutional neural network approaches to improve accuracy. Expert Syst Appl 115280:115280

    Article  Google Scholar 

  33. Pedregosa F, Varoquaux G, Gramfort A et al (2011) Scikit-learn: machine learning in python. The. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  34. Pons G, Masip D (2017) Supervised committee of convolutional neural networks in automated facial expression analysis. IEEE Trans Affect Comput 9:343–350

    Article  Google Scholar 

  35. Reddy KS, Kumar VV, Reddy BE (2015) Face recognition based on texture features using local ternary patterns. International Journal of Image, Graphics and Signal Processing, 7(10), 37

  36. Reddy GV, Savarni CVRD, Mukherjee S (2020) Facial expression recognition in the wild, by fusion of deep learnt and hand-crafted features. Cogn Syst Res 62:23–34

    Article  Google Scholar 

  37. Ruan D, Yan Y, Lai S et al (2021) Feature decomposition and reconstruction learning for effective facial expression recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, In, pp 7660–7669

    Google Scholar 

  38. Sheykhmousa M, Mahdianpari M, Ghanbari H et al (2020) Support vector machine versus random forest for remote sensing image classification: A meta-analysis and systematic review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13:6308–6325

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 14091556

  40. Su R, Liu T, Sun C et al (2020) Fusing convolutional neural network features with hand-crafted features for osteoporosis diagnoses. Neurocomputing 385:300–309

  41. Sun X, Lv M (2019) Facial expression recognition based on a hybrid model combining deep and shallow features. Cogn Comput 11:587–597

    Article  Google Scholar 

  42. Sun N, Li Q, Huan R et al (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn Lett 119:49–61

  43. Tola E, Lepetit V, Fua P (2009) Daisy: an efficient dense descriptor applied to wide-baseline stereo. IEEE Trans Pattern Anal Mach Intell 32:815–830

    Article  Google Scholar 

  44. Vicnesh J, Wei JKE, Ciaccio EJ et al UR (2019) Automated diagnosis of celiac disease by video capsule endoscopy using DAISY descriptors. Automated diagnosis of celiac disease by video capsule endoscopy using DAISY Descriptors. J Med Syst 43:43–49. https://doi.org/10.1007/s10916-019-1285-6

  45. Villanueva MG, Zavala SR (2020) Deep neural network architecture: application for facial expression recognition. IEEE Lat Am Trans 18:1311–1319

    Article  Google Scholar 

  46. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154

    Article  Google Scholar 

  47. Wang B, Gao L, Juan Z (2017) Travel mode detection using GPS data and socioeconomic attributes based on a random forest classifier. IEEE Trans Intell Transp Syst 19:1547–1558

    Article  Google Scholar 

  48. Wang F, Lv J, Ying G et al (2019) Facial expression recognition from image based on hybrid features understanding. J Vis Commun Image Represent 59:84–88

  49. Wang Z, Zeng F, Liu S, Zeng B (2021) OAENet: oriented attention ensemble for accurate facial expression recognition. Pattern Recogn 112:107694

    Article  Google Scholar 

  50. Zhang H, Su W, Wang Z (2020) Weakly supervised local-global attention network for facial expression recognition. IEEE Access 8:37976–37987

    Article  Google Scholar 

  51. Zhu C, Bichot C-E, Chen L (2011) Visual object recognition using daisy descriptor. In: 2011 IEEE International Conference on Multimedia and Expo. IEEE, pp 1–6.

  52. Zhu X, He Z, Zhao L et al (2022) A Cascade Attention Based Facial Expression Recognition Network by Fusing Multi-Scale Spatio-Temporal Features Sensors. 22:22. https://doi.org/10.3390/s22041350

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Electronics Engineering, Faculty of Engineering, Lorestan University, Khorramabad, Iran

    Saeed Najafi Khanbebin & Vahid Mehrdad

Authors
  1. Saeed Najafi Khanbebin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Vahid Mehrdad
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Vahid Mehrdad.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest related to this paper. We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khanbebin, S.N., Mehrdad, V. Improved convolutional neural network-based approach using hand-crafted features for facial expression recognition. Multimed Tools Appl 82, 11489–11505 (2023). https://doi.org/10.1007/s11042-022-14122-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14122-1

Keywords