Skip to main content
SpringerLink
Log in

Identifying emotions from facial expressions using a deep convolutional neural network-based approach

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Sentiment identification on facial expression is an interesting study domain with applications in various disciplines, including security, health, and human-machine interfaces. The main goal of sentiment analysis is to decide an individual’s perspective on a topic or the document’s overall contextual polarity. In nonverbal communication, sentiment analysis plays a vital role in an individual’s feelings, reflecting on the faces. Researchers in this area are interested in improving models and methods and extracting various characteristics to provide a better computer prediction of sentiments. Sentiment polarities are mainly classified as positive, negative, and neutral. Many sentiment analysis approaches exist, but deep learning architectures can handle extensive data and provide better performances. We presented a solution based on the CNN (Convolutional Neural Network) model for handling this problem. This work uses the extended Cohn Kanade (CK+) and FER-2013 datasets for facial expression recognition study. Several existing architectures are used to evaluate the efficiency of the proposed model. Extensive experiments are carried out on both CK+ and FER-2013 data sets, and our framework outperforms state-of-the-art techniques. According to obtained results, the CNN3 model gives 79% and 95% accuracy for FER-2013 and CK+ datasets, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://towardsdatascience.com/face-detection-for-beginners-e58e8f21aad9

  2. https://www.kaggle.com/msambare/fer2013

  3. https://www.kaggle.com/shawon10/ckplus

References

  1. Huang Y, Xu H (2021) Fully convolutional network with attention modules for semantic segmentation. Signal, Image and Video Processing 15:1031–1039

    Article  Google Scholar 

  2. You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29 (2015)

  3. Islam, J., Zhang, Y.: Visual sentiment analysis for social images using transfer learning approach. In: 2016 IEEE International Conferences on Big Data and Cloud Computing (BDCloud), Social Computing and Networking (SocialCom), Sustainable Computing and Communications (SustainCom)(BDCloud-SocialCom-SustainCom), pp. 124–130 (2016). IEEE

  4. Campos V, Jou B, Giro-i-Nieto X (2017) From pixels to sentiment: Fine-tuning cnns for visual sentiment prediction. Image and Vision Computing 65:15–22

    Article  Google Scholar 

  5. Tsytsarau M, Palpanas T (2012) Survey on mining subjective data on the web. Data Mining and Knowledge Discovery 24:478–514

    Article  Google Scholar 

  6. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Challenges in representation learning: A report on three machine learning contests. In: Neural Information Processing: 20th International Conference, ICONIP 2013, Daegu, Korea, November 3-7, 2013. Proceedings, Part III 20, pp. 117–124 (2013). Springer

  7. Montoyo A, Martínez-Barco P, Balahur A (2012) Subjectivity and sentiment analysis: An overview of the current state of the area and envisaged developments. Decision Support Systems 53(4):675–679

    Article  Google Scholar 

  8. Maynard, D., Funk, A.: Automatic detection of political opinions in tweets. In: The Semantic Web: ESWC 2011 Workshops: ESWC 2011 Workshops, Heraklion, Greece, May 29-30, 2011, Revised Selected Papers 8, pp. 88–99 (2012). Springer

  9. Bengio Y, Goodfellow I, Courville A (2017) Deep learning. MIT press Cambridge, MA, USA

    Google Scholar 

  10. Dalai, R., Senapati, K.K.: Comparison of various rcnn techniques for classification of object from image. International Research Journal of Engineering and Technology (IRJET) 4(07) (2017)

  11. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 Ieee Computer Society Conference on Computer Vision and Pattern Recognition-workshops, pp. 94–101 (2010). IEEE

  12. Patel K, Mehta D, Mistry C, Gupta R, Tanwar S, Kumar N, Alazab M (2020) Facial sentiment analysis using ai techniques: state-of-the-art, taxonomies, and challenges. IEEE Access 8:90495–90519

    Article  Google Scholar 

  13. Song K, Yao T, Ling Q, Mei T (2018) Boosting image sentiment analysis with visual attention. Neurocomputing 312:218–228

    Article  Google Scholar 

  14. Rashid, T.A.: Convolutional neural networks based method for improving facial expression recognition. In: Intelligent Systems Technologies and Applications 2016, pp. 73–84 (2016). Springer

  15. Torres, A.D., Yan, H., Aboutalebi, A.H., Das, A., Duan, L., Rad, P.: Patient facial emotion recognition and sentiment analysis using secure cloud with hardware acceleration. In: Computational Intelligence for Multimedia Big Data on the Cloud with Engineering Applications, pp. 61–89. Elsevier, (2018)

  16. Wang, J., Fu, J., Xu, Y., Mei, T.: Beyond object recognition: Visual sentiment analysis with deep coupled adjective and noun neural networks. In: IJCAI, pp. 3484–3490 (2016). Citeseer

  17. Ain, Q.T., Ali, M., Riaz, A., Noureen, A., Kamran, M., Hayat, B., Rehman, A.: Sentiment analysis using deep learning techniques: a review. International Journal of Advanced Computer Science and Applications 8(6) (2017)

  18. Chen, T., Borth, D., Darrell, T., Chang, S.-F.: Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv preprint http://arxiv.org/abs/1410.8586arXiv:1410.8586 (2014)

  19. Chen, T., Borth, D., Darrell, T., Chang, S.-F.: Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv preprint http://arxiv.org/abs/1410.8586arXiv:1410.8586 (2014)

  20. Jindal, S., Singh, S.: Image sentiment analysis using deep convolutional neural networks with domain specific fine tuning. In: 2015 International Conference on Information Processing (ICIP), pp. 447–451 (2015). IEEE

  21. Girshick R, Donahue J, Darrell T, Malik J (2015) Region-based convolutional networks for accurate object detection and segmentation. IEEE transactions on pattern analysis and machine intelligence 38(1):142–158

    Article  Google Scholar 

  22. Cai, G., Xia, B.: Convolutional neural networks for multimedia sentiment analysis. In: Natural Language Processing and Chinese Computing: 4th CCF Conference, NLPCC 2015, Nanchang, China, October 9-13, 2015, Proceedings 4, pp. 159–167 (2015). Springer

  23. Jokhio, F.A., Jokhio, A.: Image classification using alexnet with svm classifier and transfer learning. Journal of Information Communication Technologies and Robotic Applications, 44–51 (2019)

  24. Gajarla, V., Gupta, A.: Emotion detection and sentiment analysis of images. Georgia Institute of Technology, 1–4 (2015)

  25. Mandhyani, J., Khatri, L., Ludhrani, V., Nagdev, R., Sahu, S.: Image sentiment analysis. International Journal of Engineering Science 4566 (2017)

  26. Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)

  27. Yang J, She D, Sun M, Cheng M-M, Rosin PL, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Transactions on Multimedia 20(9):2513–2525

    Article  Google Scholar 

  28. Salunke, V., Panicker, S.S.: Image sentiment analysis using deep learning. In: Inventive Communication and Computational Technologies: Proceedings of ICICCT 2020, pp. 143–153 (2021). Springer

  29. Onita, D., Dinu, L.P., Birlutiu, A.: From image to text in sentiment analysis via regression and deep learning. In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pp. 862–868 (2019)

  30. Gudi, A., Tasli, H.E., Den Uyl, T.M., Maroulis, A.: Deep learning based facs action unit occurrence and intensity estimation. In: 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), vol. 6, pp. 1–5 (2015). IEEE

  31. Moran, J.L.: Classifying emotion using convolutional neural networks. UC Merced Undergraduate Research Journal 11(1) (2019)

  32. Sadr H, Pedram MM, Teshnehlab M (2021) Convolutional neural network equipped with attention mechanism and transfer learning for enhancing performance of sentiment analysis. Journal of AI and data mining 9(2):141–151

    Google Scholar 

  33. Parimala, M., Swarna Priya, R., Praveen Kumar Reddy, M., Lal Chowdhary, C., Kumar Poluru, R., Khan, S.: Spatiotemporal-based sentiment analysis on tweets for risk assessment of event using deep learning approach. Software: Practice and Experience 51(3), 550–570 (2021)

  34. Gan Y, Chen J, Xu L (2019) Facial expression recognition boosted by soft label with a diverse ensemble. Pattern Recognition Letters 125:105–112

    Article  Google Scholar 

  35. Renda A, Barsacchi M, Bechini A, Marcelloni F (2019) Comparing ensemble strategies for deep learning: An application to facial expression recognition. Expert Systems with Applications 136:1–11

    Article  Google Scholar 

  36. Babajee, P., Suddul, G., Armoogum, S., Foogooa, R.: Identifying human emotions from facial expressions with deep learning. In: 2020 Zooming Innovation in Consumer Technologies Conference (ZINC), pp. 36–39 (2020). IEEE

  37. Tai, Y., Tan, Y., Gong, W., Huang, H.: Bayesian convolutional neural networks for seven basic facial expression classifications. arXiv preprint http://arxiv.org/abs/2107.04834arXiv:2107.04834 (2021)

  38. Benamara NK, Val-Calvo M, Alvarez-Sanchez JR, Diaz-Morcillo A, Ferrandez-Vicente JM, Fernandez-Jover E, Stambouli TB (2021) Real-time facial expression recognition using smoothed deep neural network ensemble. Integrated Computer-Aided Engineering 28(1):97–111

    Article  Google Scholar 

  39. Yang J, She D, Sun M, Cheng M-M, Rosin PL, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Transactions on Multimedia 20(9):2513–2525

    Article  Google Scholar 

  40. Yu, J.X., Lim, K.M., Lee, C.P.: Move-cnns: Model averaging ensemble of convolutional neural networks for facial expression recognition. IAENG International Journal of Computer Science 48(3) (2021)

  41. Agrawal A, Mittal N (2020) Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. The Visual Computer 36(2):405–412

    Article  Google Scholar 

  42. Kim JH, Poulose A, Han DS (2021) The extensive usage of the facial image threshing machine for facial emotion recognition performance. Sensors 21(6):2026

    Article  Google Scholar 

  43. Benmohamed A, Neji M, Ramdani M, Wali A, Alimi AM (2015) Feast: face and emotion analysis system for smart tablets. Multimedia Tools and Applications 74:9297–9322

    Article  Google Scholar 

  44. Said Y, Barr M (2021) Human emotion recognition based on facial expressions via deep learning on high-resolution images. Multimedia Tools and Applications 80(16):25241–25253

    Article  Google Scholar 

  45. Gupta, S., Kumar, P., Tekchandani, R.K.: Facial emotion recognition based real-time learner engagement detection system in online learning context using deep learning models. Multimedia Tools and Applications, 1–30 (2022)

  46. Castellano, G., De Carolis, B., Macchiarulo, N.: Automatic facial emotion recognition at the covid-19 pandemic time. Multimedia Tools and Applications, 1–19 (2022)

  47. Kumar A, Tripathi AR, Satapathy SC, Zhang Y-D (2022) Sars-net: Covid-19 detection from chest x-rays by combining graph convolutional network and convolutional neural network. Pattern Recognition 122:108255

    Article  Google Scholar 

  48. Ng, A.: Deep learning specialization. Internet: https://www.coursera.org/specializations/deep-learning (2017)

  49. Haykin, S.: Neural networks and learning machines, 3/E. Pearson Education India (2009)

  50. Meena G, Mohbey KK, Indian A (2022) Categorizing sentiment polarities in social networks data using convolutional neural network. SN Computer Science 3(2):116

    Article  Google Scholar 

  51. Pandey, A., Shukla, S., Mohbey, K.K.: Comparative analysis of a deep learning approach with various classification techniques for credit score computation. Recent Advances in Computer Science and Communications (Formerly: Recent Patents on Computer Science) 14(9), 2785–2799 (2021)

  52. Qin, Z., Wu, J.: Visual saliency maps can apply to facial expression recognition. arXiv preprint http://arxiv.org/abs/1811.04544arXiv:1811.04544 (2018)

  53. Riaz MN, Shen Y, Sohail M, Guo M (2020) Exnet: An efficient approach for emotion recognition in the wild. Sensors 20(4):1087

    Article  Google Scholar 

  54. Jiang P, Wan B, Wang Q, Wu J (2020) Fast and efficient facial expression recognition using a gabor convolutional network. IEEE Signal Processing Letters 27:1954–1958

    Article  Google Scholar 

  55. Zang H, Foo SY, Bernadin S, Meyer-Baese A (2021) Facial emotion recognition using asymmetric pyramidal networks with gradient centralization. IEEE Access 9:64487–64498

    Article  Google Scholar 

  56. Alsharekh MF (2022) Facial emotion recognition in verbal communication based on deep learning. Sensors 22(16):6105

    Article  Google Scholar 

  57. Borgalli, M.R.A., Surve, S.: Deep learning for facial emotion recognition using custom cnn architecture. In: Journal of Physics: Conference Series, vol. 2236, p. 012004 (2022). IOP Publishing

  58. Ul Haq I, Ullah A, Muhammad K, Lee MY, Baik SW (2019) Personalized movie summarization using deep cnn-assisted facial expression recognition. Complexity 2019:1–10

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Central University of Rajasthan, Ajmer, 305817, India

    Gaurav Meena, Krishna Kumar Mohbey & Ajay Indian

  2. Department of Computer Science, Taibah University Faculty of Engineering, Yanbu, Saudi Arabia

    Mohammad Zubair Khan

  3. School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, India

    Sunil Kumar

Authors
  1. Gaurav Meena
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Krishna Kumar Mohbey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ajay Indian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohammad Zubair Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sunil Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gaurav Meena.

Ethics declarations

Conflicts of interest

The authors confirm that this article’s content has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Meena, G., Mohbey, K.K., Indian, A. et al. Identifying emotions from facial expressions using a deep convolutional neural network-based approach. Multimed Tools Appl 83, 15711–15732 (2024). https://doi.org/10.1007/s11042-023-16174-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16174-3

Keywords

Search

Navigation