Abstract
Detecting facial emotions among people is a crucial task in social communication, as it reflects their internal character. In the future, virtual face emotion detection will play a vital role in various fields, such as virtual human detection, security systems, online games, human psychology analysis, virtual classrooms, and monitoring abnormalities in patients. Integrating facial emotion detection into virtual human detection enhances the entire virtual experience, infusing interactions with authenticity, emotional intelligence, and customization for individual users. Human emotions, depicted on the face represent the brain's reactions that can be captured in the form of video or image for accurate diagnosis. This paper introduces a technology-aided face emotion detection system using convolutional neural networks (CNN). The CNN model performs the emotion detection function by executing image pre-processing, feature extraction, and image classification. Computational modules within the neural network extract features from images to enhance prediction. The proposed CNN model uses data augmentation, max pooling, and batch normalization techniques to expand facial emotion classification and improve performance and generalization. Additionally, ResNet50 architecture used with CNN improves accuracy and reduces error rate with identity mapping. Comparing performance metrics, accuracy, loss and complexity to existing models, the proposed models outperform them. The proposed CNN achieves a maximum of 15.53% higher accuracy and 25.22% lower loss in face emotion detection than the lowest-performing existing model.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of Data and Materials
Not applicable.
References
Yang, K., Wang, C., Sarsenbayeva, Z., Tag, B., Dingler, T., Wadley, G., & Goncalves, J. (2021). Benchmarking commercial emotion detection systems using realistic distortions of facial image datasets. The Visual Compute, 37, 1447–1466. https://doi.org/10.1007/s00371-020-01881-x
Hansheng, F., & Jian, Z. (2018). Representations of face images and collaborative representation classification for face recognition. Journal of Circuits, Systems and Computers, 27(01), 1850017. https://doi.org/10.1142/S0218126618500172
Jiang, Z., Harati, S., Crowell, A., Mayberg, H. S., Nemati, S., & Clifford, G. D. (2021). Classifying major depressive disorder and response to deep brain stimulation over time by analyzing facial expressions. IEEE Transactions on Biomedical Engineering, 68(2), 664–672. https://doi.org/10.1109/TBME.2020.3010472
Li, J., Kale Edmiston, E., Tang, Y., et al. (2018). Shared facial emotion processing functional network findings in medication-naïve major depressive disorder and healthy individuals: Detection by sICA. BMC Psychiatry, 18, 96. https://doi.org/10.1186/s12888-018-1631-0
Maglogiannis, I., Vouyioukas, D., & Aggelopoulos, C. (2009). Face detection and recognition of natural human emotion using Markov random fields. Personal and Ubiquitous Computing, 13, 95–101. https://doi.org/10.1007/s00779-007-0165-0
Jain, N., Gupta, V., Shubham, S., et al. (2021). Understanding cartoon emotion using integrated deep neural network on large dataset. Neural Computing and Applications. https://doi.org/10.1007/s00521-021-06003-9
Nasir, M., Dutta, P., & Nandi, A. (2021). Fuzzy triangulation signature for detection of change in human emotion from face video image sequence. Multimedia Tools and Applications, 80, 31993–32022. https://doi.org/10.1007/s11042-021-11196-1
Salama, E. S., El-Khoribi, R. A., Shoman, M. E., & Shalaby, M. A. W. (2021). A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. Egyptian Informatics Journal, 22(2), 167–176. https://doi.org/10.1016/j.eij.2020.07.005
Qiuyu, L., Jun, Y., Toru, K., Haiyan, Z., & Shu, Z. (2020). Deep convolutional neural network with optical flow for facial micro-expression recognition. Journal of Circuits, Systems and Computers., 29(01), 2050006. https://doi.org/10.1142/S0218126620500061
Vaidya, K. S., Patil, P. M., & Alagirisamy, M. (2023). Hybrid CNN-SVM classifier for human emotion recognition using ROI extraction and feature fusion. Wireless Personal Communications, 132, 1099–1135. https://doi.org/10.1007/s11277-023-10650-7
Ashwin, T. S., & Guddeti, R. M. R. (2020). Automatic detection of students’ affective states in classroom environment using hybrid convolutional neural networks. Education and Information Technologies, 25, 1387–1415. https://doi.org/10.1007/s10639-019-10004-6
Wang, Y., Li, Y., Song, Y., & Rong, X. (2019). Facial expression recognition based on random forest and convolutional neural network. Information, 10(12), 375. https://doi.org/10.3390/info10120375
Jaiswal, S., & Nandi, G. C. (2020). Robust real-time emotion detection system using CNN architecture. Neural Computing and Applications, 32, 11253–11262. https://doi.org/10.1007/s00521-019-04564-4
Yao, L., & Zhao, H. (2023). Deep learning method of facial expression recognition based on gabor filter bank combined with PCNN. Wireless Personal Communications, 131, 955–971. https://doi.org/10.1007/s11277-023-10463-8
Wijayasingha, L., & Stankovic, J. A. (2021). Robustness to noise for speech emotion classification using CNNs and attention mechanisms. Smart Health, 19, 100165. https://doi.org/10.1016/j.smhl.2020.100165
Hannane, R., Elboushaki, A., & Afdel, K. (2020). A divide-and-conquer strategy for facial landmark detection using dual-task CNN architecture. Pattern Recognition, 107, 107504. https://doi.org/10.1016/j.patcog.2020.107504
Farhoudi, Z., & Setayeshi, S. (2021). Fusion of deep learning features with mixture of brain emotional learning for audio-visual emotion recognition. Speech Communication, 127, 92–103. https://doi.org/10.1016/j.specom.2020.12.001
Bejani, M., Gharavian, D., & Charkari, N. M. (2014). Audiovisual emotion recognition using ANOVA feature selection method and multi-classifier neural networks. Neural Computing and Applications, 24, 399–412. https://doi.org/10.1007/s00521-012-1228-3
Dino, H. I., & Abdulrazzaq, M. B. (2019). Facial expression classification based on SVM, KNN and MLP classifiers. International Conference on Advanced Science and Engineering (ICOASE), 2019, 70–75. https://doi.org/10.1109/ICOASE.2019.8723728
Sohail, A. S. M., & Bhattacharya, P. (2007). Classification of facial expressions using K-nearest neighbor classifier. In A. Gagalowicz & W. Philips (Eds.), Computer vision/computer graphics collaboration techniques. MIRAGE 2007. Lecture Notes in Computer Science. (Vol. 4418). Berlin: Springer.
Aamir, M., Ali, T., Shaf, A., et al. (2020). ML-DCNNet: Multi-level deep convolutional neural network for facial expression recognition and intensity estimation. Arabian Journal for Science and Engineering, 45, 10605–10620. https://doi.org/10.1007/s13369-020-04811-0
Al Chanti, D., & Caplier, A. (2021). Deep learning for spatio-temporal modeling of dynamic spontaneous emotions. IEEE Transactions on Affective Computing, 12(2), 363–376. https://doi.org/10.1109/TAFFC.2018.2873600
Devi, D. A. S., & Satyanarayana, C. (2021). An efficient facial emotion recognition system using novel deep learning neural network-regression activation classifier. Multimedia Tools and Applications, 80, 17543–17568. https://doi.org/10.1007/s11042-021-10547-2
Atila, O., & Şengür, A. (2021). Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition. Applied Acoustics, 182, 108260. https://doi.org/10.1016/j.apacoust.2021.108260
Chirra, V. R. R., Uyyala, S. R., & Kolli, V. K. K. (2021). Virtual facial expression recognition using deep CNN with ensemble learning. Journal of Ambient Intelligence and Humanized Computing, 12, 10581–10599. https://doi.org/10.1007/s12652-020-02866-3
Mookkaiah, S. S., Thangavelu, G., Hebbar, R., et al. (2022). Design and development of smart Internet of Things—based solid waste management system using computer vision. Environmental Science and Pollution Research. https://doi.org/10.1007/s11356-022-20428-2
Zhang, H., Jolfaei, A., & Alazab, M. (2019). A face emotion recognition method using convolutional neural network and image edge computing. IEEE Access, 7, 159081–159089. https://doi.org/10.1109/ACCESS.2019.2949741
Zhou, N., Liang, R., & Shi, W. (2021). A lightweight convolutional neural network for real-time facial expression detection. IEEE Access, 9, 5573–5584. https://doi.org/10.1109/ACCESS.2020.3046715
Do, L. N., Yang, H. J., Nguyen, H. D., et al. (2021). Deep neural network-based fusion model for emotion recognition using visual data. The Journal of Supercomputing, 77, 10773–10790. https://doi.org/10.1007/s11227-021-03690-y
Pitaloka, D. A., Wulandari, A., Basaruddin, T., & Liliana, D. Y. (2017). Enhancing CNN with preprocessing stage in automatic emotion recognition. Procedia Computer Science, 116, 523–529. https://doi.org/10.1016/j.procs.2017.10.038
Zheng, K., Yang, D., Liu, J., & Cui, J. (2020). Recognition of teachers’ facial expression intensity based on convolutional neural network and attention mechanism. IEEE Access, 8, 226437–226444. https://doi.org/10.1109/ACCESS.2020.3046225
Miao, S., Xu, H., Han, Z., & Zhu, Y. (2019). Recognizing facial expressions using a shallow convolutional neural network. IEEE Access, 7, 78000–78011. https://doi.org/10.1109/ACCESS.2019.2921220
Nayak, S., Nagesh, B., Routray, A., & Sarma, M. (2021). A human–computer Interaction framework for emotion recognition through time-series thermal video sequences. Computers & Electrical Engineering, 93, 107280. https://doi.org/10.1016/j.compeleceng.2021.107280
Lu, C.-T., Su, C.-W., Jiang, H.-L., & Lu, Y.-Y. (2022). An interactive greeting system using convolutional neural networks for emotion recognition. Entertainment Computing, 40, 100452. https://doi.org/10.1016/j.entcom.2021.100452
Sarvakar, K., Senkamalavalli, R., Raghavendra, S., Santosh-Kumar, J., Manjunath, R., & Jaiswal, S. (2021). Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings. https://doi.org/10.1016/j.matpr.2021.07.297
Arun, A. N., Maheswaravenkatesh, P., & Jayasankar, T. (2023). Facial micro emotion detection and classification using swarm intelligence based modified convolutional network. Expert Systems with Applications, 233, 120947. https://doi.org/10.1016/j.eswa.2023.120947
Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC). SN Applied Sciences, 2, 446. https://doi.org/10.1007/s42452-020-2234-1
Mou, L., Zhou, C., Zhao, P., Nakisa, B., Rastgoo, M. N., Jain, R., & Gao, W. (2021). Driver stress detection via multimodal fusion using attention-based CNN-LSTM. Expert Systems with Applications, 173, 114693. https://doi.org/10.1016/j.eswa.2021.114693
Orrite, C., Gañán, A., & Rogez, G. (2009). HOG-based decision tree for facial expression classification. In H. Araujo, A. M. Mendonça, A. J. Pinho, & M. I. Torres (Eds.), pattern recognition and image analysis. IbPRIA 2009. Lecture Notes in Computer Science. (Vol. 5524). Berlin: Springer. https://doi.org/10.1007/978-3-642-02172-5_24
Maturana, D., Mery, D., & Soto, Á. (2011). Face recognition with decision tree-based local binary patterns. In R. Kimmel, R. Klette, & A. Sugimoto (Eds.), Computer vision—ACCV 2010. ACCV 2010. Lecture Notes in Computer Science. (Vol. 6495). Berlin: Springer. https://doi.org/10.1007/978-3-642-19282-1_49
Salmam, F. Z., Madani, A., & Kissi, M. (2016). Facial expression recognition using decision trees. In 2016 13th international conference on computer graphics, imaging and visualization (CGiV), 2016 (pp. 125–130). https://doi.org/10.1109/CGiV.2016.33.
Funding
No funding was received to assist with the preparation of this manuscript.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study's conception and design. MSS has been developed an idea and administrated the project work as a supervisor. Four different models of the research work were implemented by MSS and TG. RTP, and LML have implemented three models of the work as per the direction given by MSS. The final version of the manuscript has been developed, analyzed, and concluded by MSS. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Ethical Approval
Not applicable.
Consent to Participate
Not applicable.
Consent to Publish
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Senthil Sivakumar, M., Gurumekala, T., Megalan Leo, L. et al. Expert System for Smart Virtual Facial Emotion Detection Using Convolutional Neural Network. Wireless Pers Commun 133, 2297–2319 (2023). https://doi.org/10.1007/s11277-024-10867-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11277-024-10867-0