Skip to main content
SpringerLink
Log in

Opinion convergence-based sentiment prediction of image advertisement

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

In this study, a novel approach was proposed for the sentiment prediction of image advertisements. Unlike general multilabel problems with correct answers, image sentiment prediction is a multilabel problem that involves converging the opinions of various labelers. Therefore, an opinion convergence-based image sentiment prediction methodology was proposed to model the decision-making process of image sentiment prediction. Hypothetical labelers were generated by recombining training datasets to maximize the characteristic distance, and each prediction model was trained with each combination to represent each hypothetical labeler with distinct characteristics. The results of the experiment revealed that the proposed image sentiment prediction method outperformed other existing models with advanced architectures or considered various factors for improving the accuracy of image sentiment prediction tasks. Moreover, the effectiveness of the proposed method was verified through qualitative experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The Pitt Image Ads Dataset is available for download from the repository at https://people.cs.pitt.edu/~nhonarvar/data_analysis/interface.html [7].

References

  1. Achlioptas P, Ovsjanikov M, Haydarov K, Elhoseiny M, Guibas LJ (2021) Artemis: affective language for visual art. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 11569–11579

  2. Adaval R, Saluja G, Jiang Y (2019) Seeing and thinking in pictures: a review of visual information processing. Consum Psychol Rev 2(1):50–69

    Article  Google Scholar 

  3. Asakawa T, Aono M (2021) Multi-label prediction for visual sentiment analysis using eight different emotions based on psychology. In: Proceedings of the 4th international conference on control and computer vision, pp 142–146

  4. Corchs S, Fersini E, Gasparini F (2019) Ensemble learning on visual and textual data for social image emotion classification. Int J Mach Learn Cybern 10:2057–2070

    Article  Google Scholar 

  5. Ghani B, Malik MAR (2022) Social media and employee voice: a comprehensive literature review. Behav Inf Technol 42(14):2407–2427

    Article  Google Scholar 

  6. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  7. Hussain Z, Zhang M, Zhang X, Ye K, Thomas C, Agha Z, Ong N, Kovashka A (2017) Automatic understanding of image and video advertisements. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1705–1715

  8. Jabreel M, Moreno A (2019) A deep learning-based approach for multi-label emotion classification in tweets. Appl Sci 9(6):1123

    Article  Google Scholar 

  9. Jia Z, Narayana P, Akula AR, Pruthi G, Su H, Basu S, Jampani V (2023) Kafa: rethinking image ad understanding with knowledge-augmented feature adaptation of vision-language models. arXiv preprint arXiv:2305.18373

  10. Joshi D, Datta R, Fedorovskaya E, Luong QT, Wang JZ, Li J, Luo J (2011) Aesthetics and emotions in images. IEEE Signal Process Mag 28(5):94–115

    Article  Google Scholar 

  11. Kim BK, Choi J, Wakslak CJ (2019) The image realism effect: the effect of unrealistic product images in advertising. J Advert 48(3):251–270

    Article  Google Scholar 

  12. Kingma D, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference for learning representations (iclr’15), San Diego, vol 500

  13. Kong S, Huang Z, Scott N, Zhang Z, Shen Z (2019) Web advertisement effectiveness evaluation: attention and memory. J Vacat Mark 25(1):130–146

    Article  Google Scholar 

  14. Kujur F, Singh S (2020) Visual communication and consumer-brand relationship on social networking sites-uses & gratifications theory perspective. J Theor Appl Electron Commer Res 15(1):30–47

  15. Kumar Y, Jha R, Gupta A, Aggarwal M, Garg A, Malyan T, Bhardwaj A, Shah RR, Krishnamurthy B, Chen C (2023) Persuasion strategies in advertisements. Proceedings of the AAAI conference on artificial intelligence 37:57–66

    Article  Google Scholar 

  16. Li L, Zhu X, Hao Y, Wang S, Gao X, Huang Q (2019) A hierarchical CNN-RNN approach for visual emotion classification. ACM Trans Multimedia Comput, Commun, Appl (TOMM) 15(3s):1-17

  17. Lin C, Zhao S, Meng L, Chua TS (2020) Multi-source domain adaptation for visual sentiment classification. Proceedings of the AAAI conference on artificial intelligence 34:2661–2668

    Article  Google Scholar 

  18. Pilli S, Patwardhan M, Pedanekar N, Karande S (2020) Predicting sentiments in image advertisements using semantic relations among sentiment labels. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp 408–409

  19. Poels K, Dewitte S (2019) The role of emotions in advertising: a call to action. J Advert 48(1):81–90

    Article  Google Scholar 

  20. Ruan S, Zhang K, Wang Y, Tao H, He W, Lv G, Chen E (2020) Context-aware generation-based net for multi-label visual emotion recognition. In: 2020 IEEE international conference on multimedia and expo (ICME). IEEE Computer Society, pp 1–6

  21. Sanggar N (2022) The effectiveness of interactive stickers used on instagram stories in influencing generation z’s perception towards brands: a study on the persuasiveness of visual communication on social media. PhD thesis, Tunku Abdul Rahman University College

  22. Smith RA (1991) The effects of visual and verbal advertising information on consumers’ inferences. J Advert 20(4):13–24

    Article  Google Scholar 

  23. Song K, Yao T, Ling Q, Mei T (2018) Boosting image sentiment analysis with visual attention. Neurocomputing 312:218–228

    Article  Google Scholar 

  24. Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. PMLR, pp 6105–6114

  25. Yang J, Sun M, Sun X (2017) Learning visual sentiment distributions via augmented conditional probability neural network. In: Proceedings of the 31st AAAI conference on artificial intelligence, AAAI’17. AAAI Press, p 224–230

  26. Zhang H, Luo Y, Ai Q, Wen Y, Hu H (2020) Look, read and feel: Benchmarking ads understanding with multimodal multitask learning. In: Proceedings of the 28th ACM international conference on multimedia, pp 430–438

  27. Zhang H, Xu M (2022) Multiscale emotion representation learning for affective image recognition. IEEE Trans Multimedia 25:2203–2212

    Article  Google Scholar 

  28. Zhao S, Yao X, Yang J, Jia G, Ding G, Chua TS, Schuller BW, Keutzer K (2021) Affective image content analysis: two decades review and new perspectives. IEEE Trans Pattern Anal Mach Intell 44(10):6729–6751

    Article  Google Scholar 

Download references

Acknowledgements

This study was supported by a research program funded by SeoulTech.

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, Seoul National University of Science and Technology, 232, Gongneung-ro, Nowon-gu, Seoul, 01811, Republic of Korea

    Younghoon Lee

Authors
  1. Younghoon Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Younghoon Lee made contributions to the conception of the work and analysis, and drafted the work.

Corresponding author

Correspondence to Younghoon Lee.

Ethics declarations

Conflict of interest

The authors declare that the authors have no competing interests as defined by Springer.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, Y. Opinion convergence-based sentiment prediction of image advertisement. Int J Multimed Info Retr 13, 6 (2024). https://doi.org/10.1007/s13735-023-00314-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13735-023-00314-4

Keywords

Search

Navigation