Skip to main content
SpringerLink
Log in

Smart-ESP System for Emotion Strength Prediction for Static Facial Images

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

The detection of a person present emotional state using facial expressions is a useful study subject in facial emotion identification. The difficult problem of automatically recognizing facial emotions is currently a topic of intense research for scholars. One of the main challenges for the face emotion detection system is predicting the intensity of the emotion, and deep learning requires large amounts of data for prediction. The transfer learning approach is mostly employed in prediction systems when there is insufficient data available for training. Our suggestion was to employ this method for determining the intensity of facial emotions.The shared parameters and priors are learned in this study using the parameter transfer learning technique. ResNet-50 The foundational design is built on Convolutional Neural Networks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The CK + and JAFFE dataset are accessed with prior permission from the publisher for the research purpose.

References

  1. Ahonen, T., Hadid, A., & & Pietikainen, M. (2006). Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(12), 2037–2041.

    Article  Google Scholar 

  2. Akhand, M. A. H., Roy, S., Siddique, N., Kamal, M. A. S., & Shimamura, T. (2021). Facial emotion recognition using transfer learning in the deep CNN, Electronics (Switzerland), 10(9).

  3. Aljoloud, A. S., Ullah, H., & Alanazi, A. A. (2020). Facial emotion recognition using neighborhood features. International Journal Advanced Computing Science Application, 11(1), 299–306.

    Article  Google Scholar 

  4. Andalibi, N., & Buss, J. (2020). The Human in Emotion Recognition on Social Media: Attitudes, Outcomes, Risks, Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3313831.3376680.

  5. Arabian, H., Wagner-Hartl, V., & Moeller, K. (2021). Traditional versus neural network classification methods for facial emotion recognition. Current Directions in Biomedical Engineering, 7(2), 203–206.

    Article  Google Scholar 

  6. Aruna, Chakraborty, & Amit Konar,Uday Kumar Chakraborty & Amita Chatterjee. (2009). Emotion Recognition From Facial Expressions and Its Control Using Fuzzy Logic, IEEE Transactions on Systems, Man and Cybernetics-Part A:Systems and Humans, 39(4), 726–743.

  7. BartlettMS, Littlewort, G., Fasel, I., & MovellanJR (2003). Real Time Face Detection and facial expression recognition: Development and applications to Human Computer Interaction. Proceeding IEEE Computing Social Conference Computing Visual Pattern Recognition Work, 2003, 10.1109/ CVPRW.2003.10057.

  8. Bendjillali, R. I., Beladgham, M., Merit, K., & Taleb-Ahmed, A. (2019). Improved facial expression recognition based on DWT feature for deep CNN, Electronics (Switzerland), 8(3).

  9. Bhatti, Y. K., Jamil, A., Nida, N., Yousaf, M. H., Viriri, S., & Velastin, S. A. (2021). Facial expression recognition of instructor using deep features and Extreme Learning Machine. Computational Intelligence and Neuroscience, 2021, https://doi.org/10.1155/2021/5570870.

  10. Byeon, Y. H., & Kwak, K. C. (2014). Facial expression recognition using 3d convolutional neural network. International Journal of Advanced Computer Science and Applications (IJACSA), 5.

  11. Chao, W. L., Ding, J. J., & Liu, J. Z. (2015). Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Processing, 117, 1–10.

    Article  Google Scholar 

  12. Chen, L., ‘ Yen, Y. (2007). Taiwanese Facial Expression Image Database; Brain Mapping Laboratory. Institute of Brain Science, National Yang-Ming University: Taipei, Taiwan.

  13. Cowie, E., Douglas-Cowie, N., Tsapatsoulis, G., Votsis, S., Kollias, W., Fel- lenz, & Taylor (2007). JG 2001, Emotion recognition in human-computer interaction, IEEE Signal processing magazine, 18(1), 32–80.

  14. Deepak Kumar Jain, Pourya Shamsolmoali & Paramjit Sehdev. (2019) Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120,. 69–74.

  15. Farhoumandi, N., Mollaey, S., Heysieattalab, S., Zarean, M., & & Eyvazpour, R. (2021). Facial emotion Recognition predicts Alexithymia using machine learning. Computational Intelligence and Neuroscience. https://doi.org/10.1155/2021/2053795.

    Article  Google Scholar 

  16. Gautam, C., & Seeja, K. R. (2022). Facial emotion recognition using handcrafted features and CNN. Procedia Computer Science, 218, 1295–1303.

    Article  Google Scholar 

  17. Georgescu, M. I., Ionescu, R. T., & Popescu, M. (2019). Local learning with deep and hand-crafted features for facial expression recognition. Ieee Access: Practical Innovations, Open Solutions. https://doi.org/10.1109/ACCESS.2019.2917266.

    Article  Google Scholar 

  18. Ghosh, A., Rab, R., & Rahman, A. (2022). Transfer learning based Facial Emotion Detection for Animated Characters, Proceedings of 2022 25th International Conference on Computer and Information Technology, ICCIT, pp. 876–881.

  19. Goyani, M., & Patel, N. (2017). Multi-level Haar Wavelet based Facial Expression Recognition using logistic regression. Indian Journal of Science and Technology, 10(9), 1–9.

    Article  Google Scholar 

  20. Gu, W., Xiang, C., Venkatesh, Y. V., Huang, D., & Lin, H. (2012). Facial expression recognition using radial encoding of local Gabor features and classifier synthesis, Pattern Recognition, 45 (1), 80–91.

  21. Hablani, R. (2020). Facial expression Recognition using transfer learning on deep Convolutional Network, Bioscience Biotechnology Research Communications, 13, (14), 185–188.

  22. Happy, S. L., & Routray, A. (2015). Automatic facial expression recognition using features of salient facial patches. IEEE Transaction Affecting Computing, 6(1), 1–12.

    Article  Google Scholar 

  23. Happy, S. L., Dantcheva, A., & Bremond, F. (2021). Expression recognition with deep features extracted from holistic and part-based models, Image and Vision Computing, 105.

  24. Jain, D. K., Zhang, Z., & Huang, K. (2016). Hybrid patch based diagonal pattern geometric appearance model for facial expression recognition, in: Conference on Intelligent Visual Surveillance, pp. 107–113.

  25. Jain, D. K., Zhang, Z., & Huang, K. (2017). Multi angle optimal pattern-based deep learn- ing for automatic facial expression recognition. Pattern Recognition Letters. https://doi.org/10.1016/j.patrec.2017.06.025.

    Article  Google Scholar 

  26. Jain, N., Kumar, S., Kumar, A., Shamsolmoali, P., & & Zareapoor, M. (2018). Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters, 115, 101–106.

    Article  Google Scholar 

  27. Jayapradha Soumya Sharma & Yash Dugar. (2018). Detection and Recognition of Human Emotion using neural network. International Journal of Applied Engineering Research ISSN, 13(8), 0973–4562.

    Google Scholar 

  28. Joseph, A., & Geetha, P. (2020). Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow, Visual Computer, 36 (3), 529–539.

  29. Keogh, E., & Mueen, A. (2011). Curse of dimensionality. Encyclopedia of machine learning (pp. 257–258). Springer.

  30. Khan, R. A., Meyer, A., Konik, H., & Bouakaz, S. (2012). Exploring human visual system: study to aid the development of automatic facial expression recognition framework, in 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. IEEE, pp. 49–54.

  31. Khandait, S. P., Thool, R. C., Khandait, P. D. Automatic Facial Feature Extraction and Expression Recognition based on Neural Network, International Journal of Advanced Computer Science and Applications, 2(1), 752-758.

  32. Kumar, R. J. R., Sundaram, M., & Arumugam, N. (2021). Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine, Visual Computer, 37 (8), 2315–2329.

  33. Lee, C. C., Shih, C. Y., Lai, W. P., & Lin, P. C. (2012). An improved boosting algorithm and its application to facial emotion recognition. J Ambient Intelligent Humanization Computing, 3, 11–17.

    Article  Google Scholar 

  34. Leh Luoh, C. C., & Huang (2010). & Hsueh-Yen Liu Image Processing based emotion recognition, International Conference on System Science and Engineering, IEEE, pp. 491–494.

  35. Li, J., Zhang, D., Zhang, J., Zhang, J., Li, T., Xia, Y., Yan, Q., & Xun, L. (2017). Facial expression recognition with faster R-CNN. Procedia Computer Science, 107, 135–140.

    Article  Google Scholar 

  36. Liu, P., Han, S., Meng, Z., & Tong, Y. (2014). Facial expression recognition via a boosted deep belief network. Procedia IEEE Computing Social Conference Computing Visual Pattern Recognition, 2014, https://doi.org/10.1109/CVPR.2014.233.

  37. Lopes, A. T., de Agular, E., De Souza, A. F., & Thiago, O. S. (2017). Facial expression recognition with convolutional neural networks: Coping with few data and the training sample order, Pattern Recognition, 61, pp. 610–628.

  38. Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010). The extended Cohn-Kanade dataset (CK++): A complete dataset for action unit and emotion-specified expression, 2010 IEEE Computing Social Conference Computing Visual Pattern Recognition Work. CVPRW, pp. 94–101.

  39. Lyons, M. J., Akamatsu, S., Kamachi, M., Gyoba, J., & Budynek, J. (1998). The japanese female facial expression (jaffe) database.

  40. Mayya, V., Pai, R. M., & Manohara Pai, M. M. (2016). Automatic facial expression Recognition using DCNN. Procedia Computer Science, 93, 453–461.

    Article  Google Scholar 

  41. Mehendale, N. (2020). Facial emotion recognition using convolutional neural networks (FERC), SN Applied Sciences, 2(3), 446.

  42. Moore, S., & Bowden, R. (2011). Local Binary Patterns for Multi-view facial expression Recognition, Computer Vision and Image Understanding, 115, 541–558.

  43. Nanda, A., Im, W., Choi, K. S., & Yang, H. S. (2021). Combined center dispersion loss function for deep facial expression recognition, Pattern Recognition Letters, 141, 8–15.

  44. Nithya, S. (2019). Emotion recognition from facial expression using deep learning. International Journal of Engineering and Advanced Technology (IJEAT) ISSN: 2249–8958, 8(Issue-6S), 91–95.

    Google Scholar 

  45. Panagiotics Tzirakis, G., Trigeorgis, M. I. A. N., Bjorn, W., & Schuller, S. Z. (2017). End-to-end multimodal emotion Recognition using deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 11, 8.

    Google Scholar 

  46. Pandey, A., Gupta, A., & Shyam, R. (2022). Facial emotion detection and recognition. International Journal of Engineering Applied Sciences and Technology, 7(1), 176–179.

    Article  Google Scholar 

  47. Qazi, A. S., Farooq, M. S., Rustam, F., Villar, M. G., Rodríguez, C. L., & Ashraf, I. (2022). Emotion Detection Using Facial Expression Involving Occlusions and Tilt, Applied Sciences (Switzerland), 12(22).

  48. Raqim Raheem, K., & Hadi Ali, I. (2020). Facial expression recognition using hybrid CNN-SVM technique. International Journal of Advanced Science and Technology, 29(4), 5528–5534.

    Google Scholar 

  49. Rescigno, M., Spezialetti, M., & Rossi, S. (2020). Personalized models for facial emotion recognition through transfer learning. Multimedia Tools and Applications, 79, 47–48.

    Article  Google Scholar 

  50. Salmam, F. Z., Madani, A., & Kissi, M. (2018). Emotion recognition from facial expression based on fiducial points detection and using neural network. International Journal of Electrical and Computer Engineering, 8(1), 52–59.

    Google Scholar 

  51. Sarvakar, K., Senkamalavalli, R., Raghavendra, S., Santosh Kumar, J., Manjunath, R., & Jaiswal, S. (2023). Facial emotion recognition using convolutional neural networks. Materials Today: Proceedings, 80, 3560–3564.

    Google Scholar 

  52. Shaees, S., Naeem, H., Arslan, M., Naeem, M. R., Ali, S. H., & Aldabbas, H. (2020). Facial Emotion Recognition Using Transfer Learning, 2020.

  53. Shah, J. H., Sharif, M., Raza, M., & Murtaza, M. & Saeed-Ur-rehman 2015, Robust face recognition technique under varying illumination. Journal of Applied Research Technology, 13, 1, pp. 97–105.

  54. Shan, C., Gong, S., & Mc Owan, P. W. (2009). Facial expression recognition based on local binary patterns: A comprehensive study, Image Visual Computing, 27(6), 803–816.

  55. Shih, F. Y., Chuang, C. F., & Wang, P. S. P. (2008). Performance comparisons of facial expression recognition in JAFFE database. Int J Pattern Recognition Artificial Intelligence, 22, 445–459.

    Article  Google Scholar 

  56. Song, I., Kim, H. J., & Jeon, P. B. (2014). Deep learning for real-time robust facial expression recognition on a smartphone, in: International Conference on Consumer Electronics (ICCE), Institute of Electrical & Electronics Engineers (IEEE), Las Vegas, NV, USA, 2014.

  57. Sujanaa, J. (2020). Real-time video based emotion recognition using convolutional neural network and transfer learning. Indian Journal of Science and Technology, 13(31), 3222–3229.

    Article  Google Scholar 

  58. Sultana, A., Dey, S. K., & Rahman, M. A. (2023). Facial emotion recognition based on deep transfer learning approach. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-023-15570-z.

    Article  Google Scholar 

  59. Sun, A., Li, Y., Huang, Y. M., Li, Q., & Lu, G. (2018). Facial expression recognition using optimized active regions, Human-centric Computing Information Science, 8(1), 1–24.

  60. Teja Chavali, S., Tej Kandavalli, C., Sugash, T. M., & Subramani, R. (2022). Smart Facial Emotion Recognition with Gender and Age Factor Estimation, Procedia Computer Science, 218, 113–123.

  61. Vasanth, P. C., & Nataraj, K. R. (2015). Facial expression Recognition using SVM Classifier. Indonesian Journal of Electrical Engineering and Informatics (IJEEI), 3(1), 16–20.

    Google Scholar 

  62. Veena Mayya, Radhika, M., Pai, & Manohara Pai, M. M. (2016). Automatic Facial Expression Recognition Using DCNN, ICACC 2016, Precedia Computer Science, 93, 453–461.

  63. Vinay, A., Rao, A. S., Shekhar, V. S., Akshay Kumar, C., Murthy, K. N. B., & Natarajan, S. (2015). Feature extractionusing ORB-RANSAC for Face Recognition. Procedia Computer Science, 70, 174–184.

    Article  Google Scholar 

  64. Wang, X., Chen, X., & Cao, C. (2020). Human emotion recognition by optimally fusing facial expression and speech feature (Vol. 84). Image Communication.

  65. Wang, X., Wang, X., & Ni, Y. (2018). Unsupervised domain adaptation for facial expression recognition using generative adversarial networks, Computing Intelligent Neuroscience, 2018, 7208794.

  66. Wang, X., Wang, Y., & Zhang, D. (2023). Complex Emotion Recognition via Facial Expressions with Label Noises Self-Cure Relation Networks, Computational Intelligence and Neuroscience, 2023, 7850140.

  67. Yang, D., Alsadoon, A., Prasad, P. W. C., Singh, A. K., & Elchouemi, A. (2018). An Emotion Recognition Model Based on Facial Recognition in virtual learning environment. Procedia Computer Science, 125, 2–10.

    Article  Google Scholar 

  68. Yao, L., Wan, Y., Ni, H., & Xu, B. (2021). Action unit classification for facial expression recognition using active learning and SVM. Multimedia Tools and Applications, 80(16), 24287–24301.

    Article  Google Scholar 

  69. Yasmin, S., Pathan, R. K., Biswas, M., Khandaker, M. U., & Faruque, M. R. I. (2020). Development of a robust multi-scale featured local binary pattern for improved facial expression recognition. Sensors (Switzerland), 20(18), 1–17.

    Article  Google Scholar 

  70. Zhao, K., Zhang, H., Dong, M., Guo, J., Yonggang Qi, & Yi-Zhe, S. (2013). A Multi-label classification Approach for Facial expression Recognition, IEEE 2013 Visual communications and Image Processing (VCIP), https://doi.org/10.1109/VCIP.2013.6706330.

Download references

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, SSN College of Engineering: Sri Sivasubramaniya Nadar College of Engineering, Chennai, India

    S. Benisha & T. T. Mirnalinee

Authors
  1. S. Benisha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. T. Mirnalinee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Both authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by S.Benisha and TT.Mirnalinee. The first draft of the manuscript was written by both authors and commented on previous versions of the manuscript.

Corresponding authors

Correspondence to S. Benisha or T. T. Mirnalinee.

Ethics declarations

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Benisha, S., Mirnalinee, T. Smart-ESP System for Emotion Strength Prediction for Static Facial Images. Wireless Pers Commun 134, 1059–1074 (2024). https://doi.org/10.1007/s11277-024-10993-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-024-10993-9

Keywords

Search

Navigation