Skip to main content
SpringerLink
Log in

Emotion-wise feature interaction analysis-based visual emotion distribution learning

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Image emotion is not exclusive, which makes emotion distribution learning more meaningful than emotion classification for visual emotion recognition. Consider the emotion correlations implicit in complex images do not strictly follow the universal psychological laws, which is essential for image sentiment analysis. We propose a novel emotion-wise feature interaction analysis (EFIA) method to study the emotion correlations for emotion distribution learning. It facilitates the interaction of specific features categories to learn complicated and specific inter-class relationships from the emotion feature perspective. In addition, we propose a distribution-oriented multi-task learning method to obtain a specialized distribution learning model. Experiments on public emotion datasets illustrate that our proposed method can achieve excellent performance on image emotion distribution learning and outperform most state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Panda, M.R., Kar, S.S., Nanda, A.K., et al.: Feedback through emotion extraction using logistic regression and CNN. Vis. Comput. 38(6), 1975–1987 (2022)

    Article  Google Scholar 

  2. Kumar, R., Sundaram, M., Arumugam, N.: Facial emotion recognition using subband selective multilevel stationary wavelet gradient transform and fuzzy support vector machine. Vis. Comput. 37(8), 2315–2329 (2021)

    Article  Google Scholar 

  3. Chen, S., Wang, J., Chen, Y., et al.: Label distribution learning on auxiliary label space graphs for facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 13984–13993 (2020)

  4. Toisoul, A., Kossaifi, J., Bulat, A., et al.: Estimation of continuous valence and arousal levels from faces in naturalistic conditions. Nat. Mach. Intell. 3(1), 42–50 (2021)

    Article  Google Scholar 

  5. Liu, T., Wang, J., Yang, B., et al.: Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom. Infr. Phys. Technol. 112, 103594 (2021)

    Article  Google Scholar 

  6. Sun, M., Yang, J., Wang, K., et al.: Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. In: 2016 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2016)

  7. Zhang, J., Sun, H., Wang, Z., et al.: Another dimension: Towards multi-subnet neural network for image sentiment analysis. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 1126–1131. IEEE (2019)

  8. Ruan, S., Zhang, K., Wang, Y., et al.: Context-aware generation-based net for multi-label visual emotion recognition. In: 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE Computer Society (2020)

  9. Geng, X., Yin, C., Zhou, Z.H.: Facial age estimation by learning from label distributions. IEEE Trans. Pattern Anal. Mach. Intell. 35(10), 2401–2412 (2013)

    Article  PubMed  Google Scholar 

  10. Peng, K.C., Chen, T., Sadovnik, A., et al.: A mixed bag of emotions: model, predict, and transfer emotion distributions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 860–868 (2015)

  11. Yang, J., She, D., Sun, M.: Joint image emotion classification and distribution learning via deep convolutional neural network. In: IJCAI, pp. 3266–3272 (2017)

  12. Yang, H., Fan, Y., Lv, G., et al.: Exploiting emotional concepts for image emotion recognition. Vis. Comput., 1–14 (2022)

  13. Fan, S., Shen, Z., Jiang, M., et al.: Emotional attention: a study of image sentiment and visual attention. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7521–7531 (2018)

  14. Cordel, M.O., Fan, S., Shen, Z., et al.: Emotion-aware human attention prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4026–4035 (2019)

  15. Ali, A.R., Shahid, U., Ali, M., et al.: High-level concepts for affective understanding of images. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 679–687. IEEE (2017)

  16. Zhu, X., Li, L., Zhang, W., et al.: Dependency exploitation: a unified CNN-RNN approach for visual emotion recognition. In: IJCAI, pp. 3595–3601 (2017)

  17. Rao, T., Li, X., Zhang, H., et al.: Multi-level region-based convolutional neural network for image emotion classification. Neurocomputing 333, 429–439 (2019)

    Article  Google Scholar 

  18. Xu, Z., Wang, S.: Emotional attention detection and correlation exploration for image emotion distribution learning. IEEE Trans. Affect. Comput. (2021)

  19. Zhou, Y., Xue, H., Geng, X.: Emotion distribution recognition from facial expressions. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1247–1250 (2015)

  20. Geng, X.: Label distribution learning. IEEE Trans. Knowl. Data Eng. 28(7), 1734–1748 (2016)

    Article  Google Scholar 

  21. Gao, B.B., Xing, C., Xie, C.W., et al.: Deep label distribution learning with label ambiguity. IEEE Trans. Image Process. 26(6), 2825–2838 (2017)

    Article  MathSciNet  PubMed  ADS  Google Scholar 

  22. Yang, J., She, D., Sun, M.: Joint image emotion classification and distribution learning via deep convolutional neural network. In: IJCAI, pp. 3266–3272 (2017)

  23. He, T., Jin, X.: Image emotion distribution learning with graph convolutional networks. In: Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 382–390 (2019)

  24. Xiong, H., Liu, H., Zhong, B., et al.: Structured and sparse annotations for image emotion distribution learning. Proceedings of the AAAI Conference on Artificial Intelligence 33(01), 363–370 (2019)

    Article  Google Scholar 

  25. Fan, Y., Yang, H., Li, Z., et al.: Predicting image emotion distribution by emotional region. In: 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1–9. IEEE (2018)

  26. Fan, Y., Yang, H., Li, Z., et al.: Predicting image emotion distribution by learning labels’ correlation. IEEE Access 7, 129997–130007 (2019)

    Article  Google Scholar 

  27. Zhang, J., Sun, H., Wang, Z., et al.: Another dimension: Towards multi-subnet neural network for image sentiment analysis. In: 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 1126–1131. IEEE (2019)

  28. Mikels, J.A., Fredrickson, B.L., Larkin, G.R., et al.: Emotional category data on images from the international affective picture system. Behav. Res. Methods 37(4), 626–630 (2005)

    Article  PubMed  PubMed Central  Google Scholar 

  29. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  30. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

  31. Niu, L., Cai, J., Veeraraghavan, A.: Zero-shot learning via category-specific visual-semantic mapping. arXiv preprint arXiv:1711.06167 (2017)

  32. Miller, A., Fisch, A., Dodge, J., et al.: Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126 (2016)

  33. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30, (2017)

  34. Kullback, S.: Information Theory and Statistics. Courier Corporationa, Chelmsford (1997)

    Google Scholar 

  35. Levina E., Bickel P.: The earth mover’s distance is the mallows distance: Some insights from statistics. In: Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 251–256. IEEE (2001)

  36. Chapelle, O., Zien, A.: Semi-supervised classification by low density separation. In: International Workshop on Artificial Intelligence and Statistics, PMLR, pp. 57–64 (2005)

  37. Yang, J., Sun, M., Sun, X.: Learning visual sentiment distributions via augmented conditional probability neural network. In: 31st AAAI Conference on Artificial Intelligence (2017)

Download references

Funding

This study was funded by the Nature Science Foundation of Shanghai (Grant Number 22ZR1418400).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, China

    Jing Zhang, Qiuge Qin, Xinyu Liu & Qi Ye

  2. DS Information Technology Co. Ltd, Shanghai, China

    Wen Du

Authors
  1. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiuge Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qi Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wen Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jing Zhang or Qi Ye.

Ethics declarations

Conflict of interest

Author Jing Zhang declares that she has no conflict of interest. Author Qiuge Qin declares that she has no conflict of interest. Author Xinyu Liu declares that he has no conflict of interest. Author Qi Ye declares that she has no conflict of interest. Author Wen Du declares that he has no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Qin, Q., Liu, X. et al. Emotion-wise feature interaction analysis-based visual emotion distribution learning. Vis Comput 40, 1359–1368 (2024). https://doi.org/10.1007/s00371-023-02854-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02854-6

Keywords

Search

Navigation