Skip to main content
SpringerLink
Log in

Facial expression recognition based on multi-channel fusion and lightweight neural network

  • Data analytics and machine learning
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

In the process of facial expression recognition, face detection is the prerequisite, image preprocessing is the foundation, facial expression feature extraction is the key, and facial expression classification is the target. Effective feature extraction in this process can improve the accuracy of facial expression classifications. On the other hand, traditional facial expression recognition methods are not only complicated in the feature extraction process, but also unable to obtain more in-depth high-semantic features and deep features from the original image. To solve the above problems, this paper proposes a facial expression recognition method based on multi-channel fusion and lightweight neural network. First, a cascade classifier based on Haar features is used to detect the face region of the facial expression image. Second, the local binary pattern (LBP) is used to extract the local texture features from the face region. Third, face edge features are simultaneously obtained by performing edge detection in the face region based on the Canny edge detection algorithm. Fourth, the obtained face image, LBP texture feature image, and edge detection Canny image are fused, and the fused image is input into the constructed lightweight neural network for training and recognition. Experiments are carried out on the public image databases Facial Expression Recognition 2013 (Fer2013) and extended Cohn–Kanade (CK +) using the hold-out cross-validation method. The experimental results show that the proposed method effectively extracts more complete image features by combining traditional feature extraction algorithms with deep learning feature extraction algorithms, improves the accuracy and robustness of facial expression recognition, and has better recognition rate and generalization ability compared to other mainstream methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

The image databases used in this paper are publicly available for scientific research.

References

  • Abasi S, Tehran MA, Fairchild MD (2020) Colour metrics for image edge detection. Color Res Appl 45(4):632–643

    Article  Google Scholar 

  • Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1800–1807

  • Cootes T, Edwards G, Taylor C (1999) Comparing active shape models with active appearance models. 1:173–182. https://doi.org/10.5244/C.13.18.Nottingham

  • D’Angelo G, Palmieri F (2021) Enhancing covid-19 tracking apps with human activity recognition using a deep convolutional neural network and har-images. Neural Comput Appl. https://doi.org/10.1007/s00521-021-05913-y

    Article  Google Scholar 

  • D’Angelo G, Palmieri F, Robustelli A (2022) A federated approach to android malware classification through perm-maps. Clust Comput 25(4):2487–2500. https://doi.org/10.1007/s10586-021-03490-2

    Article  Google Scholar 

  • Ding Y, Zhao Q, Li B, Yuan X (2017) Facial expression recognition from image sequence based on lbp and taylor expansion. IEEE Access 5:1–2. https://doi.org/10.1109/ACCESS.2017.2737821

    Article  Google Scholar 

  • Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Personal Soc Psychol 17(2):124–129

    Article  Google Scholar 

  • Goodfellow IJ, Erhan D, Luc Carrier P et al (2015) Challenges in representation learning: a report on three machine learning contests. Neural Netw 64:59–63

    Article  Google Scholar 

  • Han S, Hu J, Li W, Zhao S, Chen M, Xu P, Luo Y (2020) From structure to concepts: the two stages of facial expression recognition. Neuropsychologia 150:8

    Google Scholar 

  • Howard AG, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: efficient convolutional neural networks for mobile vision applications. CoRR. abs/ 1704.04861

  • Hsieh C-C, Hsih M-H, Jiang M-K, Cheng Y-M, Liang E-H (2016) Effective semantic features for facial expressions recognition using svm. Multimed Tools Appl 75(11):6663–6682

    Article  Google Scholar 

  • Huang D, Shan C, Mohsen A, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews: A publication of the IEEE Systems, Man, and Cybernetics Society) 41(6):765–781. https://doi.org/10.1109/TSMCC.2011.2118750

  • Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recogn Lett 120:69–74

    Article  Google Scholar 

  • Jia Q, Gao X, Guo H, Luo Z, Wang Y (2015) Multi-layer sparse representation for weighted lbp-patches based facial expression recognition. Sensors 3:6719–6739. https://doi.org/10.3390/s150306719

    Article  Google Scholar 

  • Ko B (2018) A brief review of facial emotion recognition based on visual information. Sensors 18(2):401

    Article  Google Scholar 

  • Kong D, Zhu M, Yu J (2019) Research on the application and method of facial expression recognition in assistive medical care. Life Sci Instr 2:43–48

    Google Scholar 

  • Li Y, Li L (2013) Application of affective computing in web-based distance education system:functions, current research and key problems. Modern Distance Educ Res 2:100–106

    Google Scholar 

  • Li Y, Zeng J, Shan S, Chen X (2019) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450. https://doi.org/10.1109/TIP.2018.2886767

    Article  MathSciNet  Google Scholar 

  • Lin M, Chen Q, Yan S (2013) Network in network. Comput Sci

  • Liu Z, Zhu F, Zhang K et al (2023) Manifold transfer subspace learning based on double relaxed discriminative regression. Artif Intell Rev. https://doi.org/10.1007/s10462-023-10547-8

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specifified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1680–1687. https://doi.org/10.1109/CVPRW.2010.5543262

  • Ma L, Chen W, Fu X, Wang T (2018) Emotional expression and micro-expression recognition in depressive patients. Chin Sci Bull 63(20):2048–2056

    Article  Google Scholar 

  • McIlhagga W (2011) The canny edge detector revisited. Int J Comput vis 91(3):251–261

    Article  MathSciNet  MATH  Google Scholar 

  • Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56

    Google Scholar 

  • Meng Q, Hu X, Kang J, Wu Y (2020) On the effectiveness of facial expression recognition for evaluation of urban sound perception. Sci Total Environ 710:2

    Article  Google Scholar 

  • Mohammed S, Karim A (2020) A novel facial emotion recognition scheme based on graph mining. Defence Technol 16(5):1062–1072. https://doi.org/10.1016/j.dt.2019.12.006

    Article  Google Scholar 

  • Navabifar F, Yusof R, Emadi M (2013) Using rotated asymmetric haar-like features for non-frontal face detection. Adv Sci Lett 12:3520–3524

    Article  Google Scholar 

  • Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  • Opschoor JAA, Petersen PC, Schwab C (2020) Deep relu networks and high-order finite element methods. Anal Appl 18(5):715–770. https://doi.org/10.1142/S0219530519410136

    Article  MathSciNet  MATH  Google Scholar 

  • Petersen P, Voigtlaender F (2018) Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Netw 108:296–330

    Article  MATH  Google Scholar 

  • Rizzo L, Longo L (2020) Self-reported data for mental workload modelling in human–computer interaction and third-level education. Data Brief. https://doi.org/10.1016/j.dib.2020.105433

    Article  Google Scholar 

  • Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. Mobile networks for classification, detection and segmentation. abs/ 1801.04381

  • Sharifara A, Mohd Rahim MS, Anisi Y (2014) A general review of human face detection including a study of neural networks and haar feature-based cascade classifier in face detection. In: 2014 International Symposium on Biometrics and Security Technologies (ISBAST), Kuala Lumpur, Malaysia, pp. 73–78. https://doi.org/10.1109/ISBAST.2014.7013097. Institute of Electrical and Electronics Engineers

  • Szegedy C, Ioffffe S, Vanhoucke V, Alemi A (2016) Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv

  • Tang Y (2013) Deep learning using linear support vector machines. Springer Briefs Comput Sci 4:41–56

    Google Scholar 

  • Tong Z, Li M (2019) The method of cutting image of vehicle face based on haar feature and improved cascade classifier. In: Proceedings of 2019 3rd International Conference on Computer Graphics and Digital Image Processing (CGDIP 2019). Journal of Physics Conference Series, vol. 1335, pp. 12–18. https://doi.org/10.1088/1742-6596/1335/1/012018

  • Turgay C, Hui M (2019) Fer-net: facial expression recognition using densely connected convolutional network. Electron Lett 55(4):184–186. https://doi.org/10.1049/el.2018.7871

    Article  Google Scholar 

  • Wang X-H, Liu A (2015) Zhang S-Q (2015) New facial expression recognition based on fsvm and knn. Optik 126(21):3132–3134

    Article  Google Scholar 

  • Wang Z, Jia C, Cai B, Fan L, Tao C, Zhang Z, Wang Y, Zhang M, Lyu G (2018) A novel tracking-by-detection method with local binary pattern and kalman filter. J Harbin Inst Technol (new Ser) 25(3):74–87

    Google Scholar 

  • Wang F, Lv J, Ying G, Chen S, Zhang C (2019) Facial expression recognition from image based on hybrid features understanding. J vis Commun Image Represent 59:84–88

    Article  Google Scholar 

  • Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. ICMI ’15, pp. 435–442. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2818346.2830595

  • Zeng Y (2018) Facial expression recognition based on deep learning. PhD thesis, University of Science and Technology of China, Anhui

  • Zhang G, Su G, Chen J, Wang J (2013) Local information enhanced lbp. J Central South Univ 20(11):3150–3155

    Article  Google Scholar 

  • Zhou J, Zhang S, Mei H, Wang D (2016) A method of facial expression recognition based on gabor and nmf. Pattern Recogn Image Anal 26(1):119–124

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by the National Natural Science Foundation of China (61672210), the National Key Research and Development Program of China (2017YFB0306403), and the Research Program of Foundation and Advanced Technology of Henan in China (162300410183).

Funding

This research is supported by the National Natural Science Foundation of China (61672210), the National Key Research and Development Program of China (2017YFB0306403), and the Research Program of Foundation and Advanced Technology of Henan in China (162300410183).

Author information

Author notes

  1. Hua Huo and Junqiang Liu contributed equally to this work.

Authors and Affiliations

  1. Engineering Technology Research Center of Big Data and Computational Intelligence, Henan University of Science and Technology, Kaiyuan Avenue, Luoyang, 471003, Henan, China

    Yali Yu, Hua Huo & Junqiang Liu

  2. School of Computing, Guangdong University of Science and Technology, Dongyuan Avenue, Dongguan, 523083, Guangdong, China

    Yali Yu

Authors
  1. Yali Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hua Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junqiang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by YY, HH, and JL. The first draft of the manuscript was written by YY and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Hua Huo.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this article.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Y., Huo, H. & Liu, J. Facial expression recognition based on multi-channel fusion and lightweight neural network. Soft Comput 27, 18549–18563 (2023). https://doi.org/10.1007/s00500-023-09199-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-023-09199-1

Keywords

Search

Navigation