Skip to main content
Log in

Expert System for Smart Virtual Facial Emotion Detection Using Convolutional Neural Network

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Detecting facial emotions among people is a crucial task in social communication, as it reflects their internal character. In the future, virtual face emotion detection will play a vital role in various fields, such as virtual human detection, security systems, online games, human psychology analysis, virtual classrooms, and monitoring abnormalities in patients. Integrating facial emotion detection into virtual human detection enhances the entire virtual experience, infusing interactions with authenticity, emotional intelligence, and customization for individual users. Human emotions, depicted on the face represent the brain's reactions that can be captured in the form of video or image for accurate diagnosis. This paper introduces a technology-aided face emotion detection system using convolutional neural networks (CNN). The CNN model performs the emotion detection function by executing image pre-processing, feature extraction, and image classification. Computational modules within the neural network extract features from images to enhance prediction. The proposed CNN model uses data augmentation, max pooling, and batch normalization techniques to expand facial emotion classification and improve performance and generalization. Additionally, ResNet50 architecture used with CNN improves accuracy and reduces error rate with identity mapping. Comparing performance metrics, accuracy, loss and complexity to existing models, the proposed models outperform them. The proposed CNN achieves a maximum of 15.53% higher accuracy and 25.22% lower loss in face emotion detection than the lowest-performing existing model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig.6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of Data and Materials

Not applicable.

References

  1. Yang, K., Wang, C., Sarsenbayeva, Z., Tag, B., Dingler, T., Wadley, G., & Goncalves, J. (2021). Benchmarking commercial emotion detection systems using realistic distortions of facial image datasets. The Visual Compute, 37, 1447–1466. https://doi.org/10.1007/s00371-020-01881-x

    Article  Google Scholar 

  2. Hansheng, F., & Jian, Z. (2018). Representations of face images and collaborative representation classification for face recognition. Journal of Circuits, Systems and Computers, 27(01), 1850017. https://doi.org/10.1142/S0218126618500172

    Article  Google Scholar 

  3. Jiang, Z., Harati, S., Crowell, A., Mayberg, H. S., Nemati, S., & Clifford, G. D. (2021). Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions. IEEE Transactions on Biomedical Engineering, 68(2), 664–672. https://doi.org/10.1109/TBME.2020.3010472

    Article  PubMed  Google Scholar 

  4. Li, J., Kale Edmiston, E., Tang, Y., et al. (2018). Shared facial emotion processing functional network findings in medication-naïve major depressive disorder and healthy individuals: Detection by sICA. BMC Psychiatry, 18, 96. https://doi.org/10.1186/s12888-018-1631-0

    Article  PubMed  PubMed Central  Google Scholar 

  5. Maglogiannis, I., Vouyioukas, D., & Aggelopoulos, C. (2009). Face detection and recognition of natural human emotion using Markov random fields. Personal and Ubiquitous Computing, 13, 95–101. https://doi.org/10.1007/s00779-007-0165-0

    Article  Google Scholar 

  6. Jain, N., Gupta, V., Shubham, S., et al. (2021). Understanding cartoon emotion using integrated deep neural network on large dataset. Neural Computing and Applications. https://doi.org/10.1007/s00521-021-06003-9

    Article  PubMed  Google Scholar 

  7. Nasir, M., Dutta, P., & Nandi, A. (2021). Fuzzy triangulation signature for detection of change in human emotion from face video image sequence. Multimedia Tools and Applications, 80, 31993–32022. https://doi.org/10.1007/s11042-021-11196-1

    Article  Google Scholar 

  8. Salama, E. S., El-Khoribi, R. A., Shoman, M. E., & Shalaby, M. A. W. (2021). A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egyptian Informatics Journal, 22(2), 167–176. https://doi.org/10.1016/j.eij.2020.07.005

    Article  Google Scholar 

  9. Qiuyu, L., Jun, Y., Toru, K., Haiyan, Z., & Shu, Z. (2020). Deep convolutional neural network with optical flow for facial micro-expression recognition. Journal of Circuits, Systems and Computers., 29(01), 2050006. https://doi.org/10.1142/S0218126620500061

    Article  Google Scholar 

  10. Vaidya, K. S., Patil, P. M., & Alagirisamy, M. (2023). Hybrid CNN-SVM classifier for human emotion recognition using ROI extraction and feature fusion. Wireless Personal Communications, 132, 1099–1135. https://doi.org/10.1007/s11277-023-10650-7

    Article  Google Scholar 

  11. Ashwin, T. S., & Guddeti, R. M. R. (2020). Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks. Education and Information Technologies, 25, 1387–1415. https://doi.org/10.1007/s10639-019-10004-6

    Article  Google Scholar 

  12. Wang, Y., Li, Y., Song, Y., & Rong, X. (2019). Facial expression recognition based on random forest and convolutional neural network. Information, 10(12), 375. https://doi.org/10.3390/info10120375

    Article  CAS  Google Scholar 

  13. Jaiswal, S., & Nandi, G. C. (2020). Robust real-time emotion detection system using CNN architecture. Neural Computing and Applications, 32, 11253–11262. https://doi.org/10.1007/s00521-019-04564-4

    Article  Google Scholar 

  14. Yao, L., & Zhao, H. (2023). Deep learning method of facial expression recognition based on gabor filter bank combined with PCNN. Wireless Personal Communications, 131, 955–971. https://doi.org/10.1007/s11277-023-10463-8

    Article  Google Scholar 

  15. Wijayasingha, L., & Stankovic, J. A. (2021). Robustness to noise for speech emotion classification using CNNs and attention mechanisms. Smart Health, 19, 100165. https://doi.org/10.1016/j.smhl.2020.100165

    Article  Google Scholar 

  16. Hannane, R., Elboushaki, A., & Afdel, K. (2020). A divide-and-conquer strategy for facial landmark detection using dual-task CNN architecture. Pattern Recognition, 107, 107504. https://doi.org/10.1016/j.patcog.2020.107504

    Article  Google Scholar 

  17. Farhoudi, Z., & Setayeshi, S. (2021). Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition. Speech Communication, 127, 92–103. https://doi.org/10.1016/j.specom.2020.12.001

    Article  Google Scholar 

  18. Bejani, M., Gharavian, D., & Charkari, N. M. (2014). Audiovisual emotion recognition using ANOVA feature selection method and multi-classifier neural networks. Neural Computing and Applications, 24, 399–412. https://doi.org/10.1007/s00521-012-1228-3

    Article  Google Scholar 

  19. Dino, H. I., & Abdulrazzaq, M. B. (2019). Facial expression classification based on SVM, KNN and MLP classifiers. International Conference on Advanced Science and Engineering (ICOASE), 2019, 70–75. https://doi.org/10.1109/ICOASE.2019.8723728

    Article  Google Scholar 

  20. Sohail, A. S. M., & Bhattacharya, P. (2007). Classification of facial expressions using K-nearest neighbor classifier. In A. Gagalowicz & W. Philips (Eds.), Computer vision/computer graphics collaboration techniques. MIRAGE 2007. Lecture Notes in Computer Science. (Vol. 4418). Berlin: Springer.

    Google Scholar 

  21. Aamir, M., Ali, T., Shaf, A., et al. (2020). ML-DCNNet: Multi-level deep convolutional neural network for facial expression recognition and intensity estimation. Arabian Journal for Science and Engineering, 45, 10605–10620. https://doi.org/10.1007/s13369-020-04811-0

    Article  Google Scholar 

  22. Al Chanti, D., & Caplier, A. (2021). Deep learning for spatio-temporal modeling of dynamic spontaneous emotions. IEEE Transactions on Affective Computing, 12(2), 363–376. https://doi.org/10.1109/TAFFC.2018.2873600

    Article  Google Scholar 

  23. Devi, D. A. S., & Satyanarayana, C. (2021). An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimedia Tools and Applications, 80, 17543–17568. https://doi.org/10.1007/s11042-021-10547-2

    Article  Google Scholar 

  24. Atila, O., & Şengür, A. (2021). Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition. Applied Acoustics, 182, 108260. https://doi.org/10.1016/j.apacoust.2021.108260

    Article  Google Scholar 

  25. Chirra, V. R. R., Uyyala, S. R., & Kolli, V. K. K. (2021). Virtual facial expression recognition using deep CNN with ensemble learning. Journal of Ambient Intelligence and Humanized Computing, 12, 10581–10599. https://doi.org/10.1007/s12652-020-02866-3

    Article  Google Scholar 

  26. Mookkaiah, S. S., Thangavelu, G., Hebbar, R., et al. (2022). Design and development of smart Internet of Things—based solid waste management system using computer vision. Environmental Science and Pollution Research. https://doi.org/10.1007/s11356-022-20428-2

    Article  PubMed  Google Scholar 

  27. Zhang, H., Jolfaei, A., & Alazab, M. (2019). A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access, 7, 159081–159089. https://doi.org/10.1109/ACCESS.2019.2949741

    Article  Google Scholar 

  28. Zhou, N., Liang, R., & Shi, W. (2021). A lightweight convolutional neural network for real-time facial expression detection. IEEE Access, 9, 5573–5584. https://doi.org/10.1109/ACCESS.2020.3046715

    Article  Google Scholar 

  29. Do, L. N., Yang, H. J., Nguyen, H. D., et al. (2021). Deep neural network-based fusion model for emotion recognition using visual data. The Journal of Supercomputing, 77, 10773–10790. https://doi.org/10.1007/s11227-021-03690-y

    Article  Google Scholar 

  30. Pitaloka, D. A., Wulandari, A., Basaruddin, T., & Liliana, D. Y. (2017). Enhancing CNN with preprocessing stage in automatic emotion recognition. Procedia Computer Science, 116, 523–529. https://doi.org/10.1016/j.procs.2017.10.038

    Article  Google Scholar 

  31. Zheng, K., Yang, D., Liu, J., & Cui, J. (2020). Recognition of teachers’ facial expression intensity based on convolutional neural network and attention mechanism. IEEE Access, 8, 226437–226444. https://doi.org/10.1109/ACCESS.2020.3046225

    Article  Google Scholar 

  32. Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019). Recognizing facial expressions using a shallow convolutional neural network. IEEE Access, 7, 78000–78011. https://doi.org/10.1109/ACCESS.2019.2921220

    Article  Google Scholar 

  33. Nayak, S., Nagesh, B., Routray, A., & Sarma, M. (2021). A human–computer Interaction framework for emotion recognition through time-series thermal video sequences. Computers & Electrical Engineering, 93, 107280. https://doi.org/10.1016/j.compeleceng.2021.107280

    Article  Google Scholar 

  34. Lu, C.-T., Su, C.-W., Jiang, H.-L., & Lu, Y.-Y. (2022). An interactive greeting system using convolutional neural networks for emotion recognition. Entertainment Computing, 40, 100452. https://doi.org/10.1016/j.entcom.2021.100452

    Article  Google Scholar 

  35. Sarvakar, K., Senkamalavalli, R., Raghavendra, S., Santosh-Kumar, J., Manjunath, R., & Jaiswal, S. (2021). Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings. https://doi.org/10.1016/j.matpr.2021.07.297

    Article  Google Scholar 

  36. Arun, A. N., Maheswaravenkatesh, P., & Jayasankar, T. (2023). Facial micro emotion detection and classification using swarm intelligence based modified convolutional network. Expert Systems with Applications, 233, 120947. https://doi.org/10.1016/j.eswa.2023.120947

    Article  Google Scholar 

  37. Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2, 446. https://doi.org/10.1007/s42452-020-2234-1

    Article  Google Scholar 

  38. Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021). Driver stress detection via multimodal fusion using attention-based CNN-LSTM. Expert Systems with Applications, 173, 114693. https://doi.org/10.1016/j.eswa.2021.114693

    Article  Google Scholar 

  39. Orrite, C., Gañán, A., & Rogez, G. (2009). HOG-based decision tree for facial expression classification. In H. Araujo, A. M. Mendonça, A. J. Pinho, & M. I. Torres (Eds.), pattern recognition and image analysis. IbPRIA 2009. Lecture Notes in Computer Science. (Vol. 5524). Berlin: Springer. https://doi.org/10.1007/978-3-642-02172-5_24

    Chapter  Google Scholar 

  40. Maturana, D., Mery, D., & Soto, Á. (2011). Face recognition with decision tree-based local binary patterns. In R. Kimmel, R. Klette, & A. Sugimoto (Eds.), Computer vision—ACCV 2010. ACCV 2010. Lecture Notes in Computer Science. (Vol. 6495). Berlin: Springer. https://doi.org/10.1007/978-3-642-19282-1_49

    Chapter  Google Scholar 

  41. Salmam, F. Z., Madani, A., & Kissi, M. (2016). Facial expression recognition using decision trees. In 2016 13th international conference on computer graphics, imaging and visualization (CGiV), 2016 (pp. 125–130). https://doi.org/10.1109/CGiV.2016.33.

Download references

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the study's conception and design. MSS has been developed an idea and administrated the project work as a supervisor. Four different models of the research work were implemented by MSS and TG. RTP, and LML have implemented three models of the work as per the direction given by MSS. The final version of the manuscript has been developed, analyzed, and concluded by MSS. All authors read and approved the final manuscript.

Corresponding author

Correspondence to M. Senthil Sivakumar.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Ethical Approval

Not applicable.

Consent to Participate

Not applicable.

Consent to Publish

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Senthil Sivakumar, M., Gurumekala, T., Megalan Leo, L. et al. Expert System for Smart Virtual Facial Emotion Detection Using Convolutional Neural Network. Wireless Pers Commun 133, 2297–2319 (2023). https://doi.org/10.1007/s11277-024-10867-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-024-10867-0

Keywords

Navigation