Skip to main content
SpringerLink
Log in

Hybrid neural networks based facial expression recognition for smart city

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

With the development of science and technology and the progress of human beings, intelligence is gradually integrated into human daily life. The smart city uses innovative technology to manage and operate cities intelligently. Through the research of facial expression recognition technology, this paper explores the application of facial expression recognition in smart city construction. In this paper, a hybrid neural network structure is proposed, which includes Sparse Autoencoder and Convolutional Neural Network (SCNN). The network reconstructs the input data by Sparse Autoencoder, so as to learn the approximate value between the original data and the reconstructed data, and obtain more high-dimensional abstract features. Then, combined with the Convolutional Neural Network, the features are further extracted and dimensionally reduced. The model can effectively solve the problem that the shallow network structure can not fully extract image features and train the model with a small number of samples. In this paper, CK+, FER2013 and Oulu-CASIA databases are used for Cross-Validation of the model. The experimental results show that the model has achieved good results in both databases. Compared with other methods, the accuracy of this model has been greatly improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Cugu I, Sener E, Akbas E (2019) Microexpnet: An extremely small and fast model for expression recognition from face images. In 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA) pp 1–6

  2. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) 1:886–893

  3. Ding H, Zhou SK, Chellappa R (2017) Facenet2expnet: Regularizing a deep face recognition net for expression recognition. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition pp 118–126

  4. Gan Y, Chen J, Yang Z, Xu L (2020) Multiple attention network for facial expression recognition. IEEE Access 8:1

  5. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H et al (2013) Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing. Springer pp 117–124

  6. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In Proc IEEE Conf Comput Vis Pattern Recognit pp 770–778

  7. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580

  8. Hu X, Wang X, Meng F, Hua X, Yan Y, Li Y, Huang J, Jiang X (2020) Gabor-cnn for object detection based on small samples. Def Technol 16(6):1116–1129

    Article  Google Scholar 

  9. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167

  10. Jain DK, Zhang Z, Huang K (2017) Multi angle optimal pattern-based deep learning for automatic facial expression recognition. Pattern Recogn Lett 139:1–9

    Google Scholar 

  11. Jung H, Lee S, Yim J, Park S, Kim J (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In Proceedings of the IEEE International Conference on Computer Vision pp 2983–2991

  12. Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang PTP (2016) On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836

  13. King DE (2009) Dlib-ml: A machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  14. Kuo C-M, Lai S-H, Sarkis M (2018) A compact deep learning model for robust facial expression recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops pp 2121–2129

  15. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  16. Leng X, Yang J (2014) Research on improved sift algorithm. J Chem Pharm Res pp 2589–2595

  17. Li C, Ma N, Deng Y (2018) Multi-network fusion based on cnn for facial expression recognition. In 2018 International Conference on Computer Science, Electronics and Communication Engineering. Atlantis Press

  18. Li S, Deng W (2020) Deep facial expression recognition: A survey. IEEE Transactions on Affective Computing p 1

  19. Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    Article  MathSciNet  Google Scholar 

  20. Liu Y, Li Y, Ma X, Song R (2017) Facial expression recognition with fusion features extracted from salient facial areas. Sensors 17(4):712

    Article  Google Scholar 

  21. Mehrabian A (2008) Communication without words. Commun Theory 6:193–200

    Google Scholar 

  22. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) pp 1–10

  23. Pramerdorfer C, Kampel M (2016) Facial expression recognition using convolutional neural networks: state of the art. arXiv preprint arXiv:1612.02903

  24. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  25. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  26. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  27. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In Proc IEEE Conf Comput Vis Pattern Recognit pp 1–9

  28. Tumen V, Soylemez OF, Ergen B (2017) Facial emotion recognition on a dataset using convolutional neural network. In 2017 International Artificial Intelligence and Data Processing Symposium (IDAP). IEEE pp 1–5

  29. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  30. Wang L, Li R-F, Wang K, Chen J (2014) Feature representation for facial expression recognition based on facs and lbp. Int J Autom Comput 11(5):459–468

    Article  Google Scholar 

  31. Yan L, Lu H, Wang C, Ye Z, Chen H, Ling H (2018) Deep linear discriminant analysis hashing for image retrieval. Multimed Tools Appl 1:1

  32. Yang H, Ciftci U, Yin L (2018) Facial expression recognition by de-expression residue learning. In Proc IEEE Conf Comput Vis Pattern Recognit pp 2168–2177

  33. Zeng N, Zhang H, Song B, Liu W, Li Y, Dobaie AM (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273:643–649

    Article  Google Scholar 

  34. Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  35. Zhang Y, Yuan Y, Feng D et al (2020) Improving restore performance for in-line backup system combining deduplication and delta compression. IEEE Trans Parallel Distrib Syst 99:1

Download references

Acknowledgements

This work is funded by the National Natural Science Foundation of China under Grant No.61772180, the Key R&D plan of Hubei Province(2020BHB004, 2020BAB012) and Natural Science Foundation of Hubei Province No.2020CFB798. No other author has reported a potential conflict of interest relevant to this article.

Author information

Authors and Affiliations

  1. School of Computer Science, Hubei University of Technology, Wuhan, China

    Lingyu Yan, Menghan Sheng, Chunzhi Wang & Rong Gao

  2. Wuhan Fiberhome Technical Services Co_,Ltd, Fiberhome Telecommunication Technologies Co_,Ltd, Wuhan, China

    Han Yu

Authors
  1. Lingyu Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Menghan Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chunzhi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rong Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Han Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lingyu Yan.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, L., Sheng, M., Wang, C. et al. Hybrid neural networks based facial expression recognition for smart city. Multimed Tools Appl 81, 319–342 (2022). https://doi.org/10.1007/s11042-021-11530-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11530-7

Keywords

Search

Navigation