Skip to main content
SpringerLink
Log in

Hybrid CNN-SVM Classifier for Human Emotion Recognition Using ROI Extraction and Feature Fusion

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Emotions expressed on a human face have a significant impact on decisions and arguments on a variety of topics. According to psychological theory, a person’s emotional states can be categorized as the afraid, disgusted, angry, sad, happy, neutral face and surprised. The automatic extraction of these emotions from images of human faces can help in human–computer interaction, among other things. Convolution Neural Network (CNN), Deep Belief Network (DBN), Bi-directional Long Short Term Memory (Bi-LSTM) are some of the existing techniques used to recognize the emotions of a human. This technique has some impacts like low accuracy and high error. To achieve better accuracy, hybrid CNN-SVM (Support Vector Machine) model is designed for classifying emotional state of humans. Initially, preprocessing is used to remove unwanted things from the image dataset. Resizing, Gaussian filter, Median filter, Histogram Equalization and Wiener filters are used in the preprocessing stage. After that, Region of Interest of the preprocessed image is extracted. Then features of the images are extracted based on Local Binary Pattern and Gabor feature technique. These obtained features are fused using the feature fusion process. The fused image data is fed to a hybrid CNN-SVM classifier. The hybrid CNN-SVM classifies the different emotional states of humans. The proposed method achieves an accuracy of 94% for CK_Plus, 86% FER_2013, 78% for KDEF, 96% for KMU_FED and 85% for the TFEID dataset. Thus the proposed human emotion recognition using the CNN-SVM approach produced optimal solutions compared to the existing systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Data Availability

Not applicable.

References

  1. Padhmashree, V., & Bhattacharyya, A. (2022). Human emotion recognition based on time–frequency analysis of multivariate EEG signal. Knowledge-Based Systems, 238, 107867.

    Article  Google Scholar 

  2. Jiang, D., Wu, K., Chen, D., Tu, G., Zhou, T., Garg, A., & Gao, L. (2020). A probability and integrated learning based classification algorithm for high-level human emotion recognition problems. Measurement, 150, 107049.

    Article  Google Scholar 

  3. Jain, D. K., Shamsolmoali, P., & Sehdev, P. (2019). Extended deep neural network for facial emotion recognition. Pattern Recognition Letters, 120, 69–74.

    Article  Google Scholar 

  4. Karthick, S., & Maniraj, S. (2019). Different medical image registration techniques: A comparative analysis. Current Medical Imaging Formerly Current Medical Imaging Reviews, 15(10), 911–921. https://doi.org/10.2174/1573405614666180905094032

    Article  Google Scholar 

  5. Chowdary, M. K., Nguyen, T. N., & Hemanth, D. J. (2021). Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Computing and Applications, 1–18. https://doi.org/10.1007/s00521-021-06012-8

  6. Cimtay, Y., Ekmekcioglu, E., & Caglar-Ozhan, S. (2020). Cross-subject multimodal emotion recognition based on hybrid fusion. IEEE Access, 8, 168865–168878.

    Article  Google Scholar 

  7. Pal, S., Mukhopadhyay, S., & Suryadevara, N. (2021). Development and progress in sensors and technologies for human emotion recognition. Sensors, 21(16), 5554.

    Article  Google Scholar 

  8. Arunnehru, J., & Kalaiselvi Geetha, M. (2017). Automatic human emotion recognition in surveillance video. Springer, Cham: In Intelligent Techniques in Signal Processing for Multimedia Security.

    Book  Google Scholar 

  9. Zhang, Y., Du, J., Wang, Z., Zhang, J., & Tu, Y. (2018). Attention based fully convolutional network for speech emotion recognition. In 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) (pp. 1771–1775). IEEE.

  10. Gupta, V., Chopda, M. D., & Pachori, R. B. (2018). Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals. IEEE Sensors Journal, 19(6), 2266–2274.

    Article  Google Scholar 

  11. Egger, M., Ley, M., & Hanke, S. (2019). Emotion recognition from physiological signal analysis: A review. Electronic Notes in Theoretical Computer Science, 343, 35–55.

    Article  Google Scholar 

  12. Bhattacharyya, A., Tripathy, R. K., Garg, L., & Pachori, R. B. (2020). A novel multivariate-multiscale approach for computing EEG spectral and temporal complexity for human emotion recognition. IEEE Sensors Journal, 21(3), 3579–3591.

    Article  Google Scholar 

  13. Liang, Z., Oba, S., & Ishii, S. (2019). An unsupervised EEG decoding system for human emotion recognition. Neural Networks, 116, 257–268.

    Article  Google Scholar 

  14. Liu, Y., & Fu, G. (2021). Emotion recognition by deeply learned multi-channel textual and EEG features. Future Generation Computer Systems, 119, 1–6.

    Article  Google Scholar 

  15. Jerritta, S., Murugappan, M., Nagarajan, R., & Wan, K. (2011). Physiological signals based human emotion recognition: a review. In 2011 IEEE 7th international colloquium on signal processing and its applications (pp. 410–415). IEEE.

  16. Batbaatar, E., Li, M., & Ryu, K. H. (2019). Semantic-emotion neural network for emotion recognition from text. IEEE Access, 7, 111866–111878.

    Article  Google Scholar 

  17. Hassan, M. M., Alam, M. G. R., Uddin, M. Z., Huda, S., Almogren, A., & Fortino, G. (2019). Human emotion recognition using deep belief network architecture. Information Fusion, 51, 10–18.

    Article  Google Scholar 

  18. Hossain, M. S., & Muhammad, G. (2019). Emotion recognition using deep learning approach from audio–visual emotional big data. Information Fusion, 49, 69–78.

    Article  Google Scholar 

  19. Meng, H., Yan, T., Yuan, F., & Wei, H. (2019). Speech emotion recognition from 3D log-mel spectrograms with deep learning network. IEEE access, 7, 125868–125881.

    Article  Google Scholar 

  20. Bhatti, A. M., Majid, M., Anwar, S. M., & Khan, B. (2016). Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior, 65, 267–275.

    Article  Google Scholar 

  21. Rahman, Z., Pu, Y. F., Aamir, M., & Ullah, F. (2019). A framework for fast automatic image cropping based on deep saliency map detection and Gaussian filter. International Journal of Computers and Applications, 41(3), 207–217.

    Article  Google Scholar 

  22. Shah, A., Bangash, J. I., Khan, A. W., Ahmed, I., Khan, A., Khan, A., & Khan, A. (2020). Comparative analysis of median filter and its variants for removal of impulse noise from gray scale images. Journal of King Saud University-Computer and Information Sciences, 34(3), 505.

    Article  Google Scholar 

  23. Rao, B. S. (2020). Dynamic histogram equalization for contrast enhancement for digital images. Applied Soft Computing, 89, 106114.

    Article  Google Scholar 

  24. Manju, B. R., & Sneha, M. R. (2020). ECG denoising using wiener filter and Kalman filter. Procedia Computer Science, 171, 273–281.

    Article  Google Scholar 

  25. Pattnaik, G., & Parvathi, K. (2021). Automatic detection and classification of tomato pests using support vector machine based on HOG and LBP feature extraction technique. Singapore: In Progress in Advanced Computing and Intelligent Engineering Springer.

    Book  Google Scholar 

  26. Hassaballah, M., Kenk, M. A., & El-Henawy, I. M. (2020). Local binary pattern-based on-road vehicle detection in urban traffic scene. Pattern Analysis and Applications, 23(4), 1505–1521.

    Article  Google Scholar 

  27. Muthukumar, A., & Kavipriya, A. (2019). A biometric system based on Gabor feature extraction with SVM classifier for finger-Knuckle-print. Pattern Recognition Letters, 125, 150–156.

    Article  Google Scholar 

  28. Hussain, M., Bird, J.J., & Faria, D.R. (2018). A study on cnn transfer learning for image classification. In UK Workshop on computational Intelligence (pp. 191–202). Springer, Cham.

  29. Ahlawat, S., & Choudhary, A. (2020). Hybrid CNN-SVM classifier for handwritten digit recognition. Procedia Computer Science, 167, 2554–2560.

    Article  Google Scholar 

Download references

Funding

The authors declare that no funds, grants, or other support were received during the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Lincoln University College, Petaling Jaya, Malaysia

    Kanchan S. Vaidya & Mukil Alagirisamy

  2. Adsul Technical Campus, Ahmednagar, Maharashtra, India

    Pradeep M. Patil

Authors
  1. Kanchan S. Vaidya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pradeep M. Patil
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mukil Alagirisamy
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by KS. V, Dr. PM P and Dr. MA. The first draft of the manuscript was written by KS. V and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kanchan S. Vaidya.

Ethics declarations

Competing interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vaidya, K.S., Patil, P.M. & Alagirisamy, M. Hybrid CNN-SVM Classifier for Human Emotion Recognition Using ROI Extraction and Feature Fusion. Wireless Pers Commun 132, 1099–1135 (2023). https://doi.org/10.1007/s11277-023-10650-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-023-10650-7

Keywords

Search

Navigation