Skip to main content
SpringerLink
Log in

A Novel Visual-Textual Sentiment Analysis Framework for Social Media Data

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Background

Sentiment analysis (SA) has turned out to be a new pattern in social networking, avidly helping people to realize views expressed in user-generated content and conventional platforms of social media. For performing numerous social media analytics tasks, SA of online user-produced content is vital. The performance of the sentiment classifiers utilizing a single modality, i.e., visual or textual, is still not matured because of the wide variety of data platforms.

Methods

In this paper, we propose a new framework called VIsual-TExtual SA (VITESA) that carries out visual analysis and textual analysis for polarity classification. In the VITESA framework, Brownian Movement-based Meerkat Clan Algorithm-centered DenseNet (BMMCA-DenseNet) is proposed that integrates textual and visual information for robust SA. In the visual phase, the images that are in the Flickr dataset are taken as input, and the operations: (1) preprocessing (2) feature extraction, and (3) feature selection utilizing Improved Coyote Optimization Algorithm (ICOA) are executed. In the textual phase, the user comments as of the Twitter dataset are taken as input, and the operations: (1) preprocessing (2) word embedding using adaptive Embedding for Language Models (ELMo), (3) emoticon and non-emoticon feature extraction, and SentiWordNet polarity assignment is carried out. The final stages of both phases are given as input to the proposed BMMCA-DenseNet classifier and intended to categorize the data into positive and negative polarity. The performance of BMMCA-DenseNet is compared with certain existing algorithms, and various performance metrics are evaluated.

Results

The proposed BMMCA-DenseNet classifier performs the polarity classification of the visual-textual data into two classes: positive or negative. The classifier categorizes the polarity of visual-textual data comprising 97% of accuracy, 94.44% of precision, 94.41% of recall, 94.41% of F-measure, 91.75% of Matthew’s Correlation Coefficient, 94.43% of sensitivity, 97.13% of specificity, and also minimal error.

Conclusions

The experiment is performed to evaluate the performance of the proposed method. The outcomes exhibit that BMMCA-DenseNet attains remarkable performance over other existing techniques. The result enhances the textual-visual communication systematically to improve sentiment prediction utilizing both information sources.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Chakraborty K, Bhatia S, Bhattacharyya S, Platos J, Bag R, Hassanien AE. Sentiment analysis of covid-19 tweets by deep learning classifiers–a study to show how popularity is affecting accuracy in social media. Appl Soft Comput. 2020;97:106754. https://doi.org/10.1016/j.asoc.2020.106754.

    Article  Google Scholar 

  2. Huang F, Zhang X, Zhao Z, Xu J, Li Z. Image-text sentiment analysis via deep multimodal attentive fusion. Knowl-Based Syst. 2019;167:26–37. https://doi.org/10.1016/j.knosys.2019.01.019.

    Article  Google Scholar 

  3. Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y. Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput. 2019;80:387–99. https://doi.org/10.1016/j.asoc.2019.04.010.

    Article  Google Scholar 

  4. Kumar A, Srinivasan K, Cheng WH, Zomaya AY. Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data. Inf Process Manag. 2020;57(1):102141. https://doi.org/10.1016/j.ipm.2019.102141.

    Article  Google Scholar 

  5. Poria S, Cambria E, Howard N, Huang GB, Hussain A. Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing. 2016;174:50–9. https://doi.org/10.1016/j.neucom.2015.01.095.

    Article  Google Scholar 

  6. Vashishtha S, Susan S. Fuzzy rule based unsupervised sentiment analysis from social media posts. Expert Syst Appl. 2019;138:112834. https://doi.org/10.1016/j.eswa.2019.112834.

    Article  Google Scholar 

  7. Zhao Z, Zhu H, Xue Z, Liu Z, Tian J, Chua MCH, Liu M. An image-text consistency driven multimodal sentiment analysis approach for social media. Inf Process Manag. 2019;56(6):102097. https://doi.org/10.1016/j.ipm.2019.102097.

    Article  Google Scholar 

  8. Yang C, Zhang H, Jiang B, Li K. Aspect-based sentiment analysis with alternating coattention networks. Inf Process Manag. 2019;56(3):463–78. https://doi.org/10.1016/j.ipm.2018.12.004.

    Article  Google Scholar 

  9. Asif M, Ishtiaq A, Ahmad H, Aljuaid H, Shah J. Sentiment analysis of extremism in social media from textual information. Telematics Inform. 2020;48:101345. https://doi.org/10.1016/j.tele.2020.101345.

    Article  Google Scholar 

  10. Wu C, Wu F, Wu S, Yuan Z, Liu J, Huang Y. Semi-supervised dimensional sentiment analysis with variational autoencoder. Knowl-Based Syst. 2019;165:30–9. https://doi.org/10.1016/j.knosys.2018.11.018.

    Article  Google Scholar 

  11. Yadav A, Vishwakarma DK. Sentiment analysis using deep learning architectures: a review. Artif Intell Rev. 2020;53(6):4335–85. https://doi.org/10.1007/s10462-019-09794-5.

    Article  Google Scholar 

  12. Onan A, Korukoğlu S, Bulut H. A multiobjective weighted voting ensemble classifier based on differential evolution algorithm for text sentiment classification. Expert Syst Appl. 2016;62:1–16. https://doi.org/10.1016/j.eswa.2016.06.005.

    Article  Google Scholar 

  13. Paolanti M, Kaiser C, Schallner R, Frontoni E, Zingaretti P. Visual and textual sentiment analysis of brand-related social media pictures using deep convolutional neural networks. In: International conference on image analysis and processing, pp. 402–413. Springer (2017). https://doi.org/10.1007/978-3-319-68560-1_36

  14. Ullah MA, Marium SM, Begum SA, Dipa NS. An algorithm and method for sentiment analysis using the text and emoticon. ICT Express. 2020;6(4):357–60. https://doi.org/10.1016/j.icte.2020.07.003.

    Article  Google Scholar 

  15. Xu J, Li Z, Huang F, Li C, Philip SY. Social image sentiment analysis by exploiting multimodal content and heterogeneous relations. IEEE Trans Ind Inf. 2020;17(4):2974–82. https://doi.org/10.1109/TII.2020.3005405.

    Article  Google Scholar 

  16. Li Z, Fan Y, Liu W, Wang F. Image sentiment prediction based on textual descriptions with adjective noun pairs. Multimedia Tools Appl. 2018;77(1):1115–32. https://doi.org/10.1007/s11042-016-4310-5.

    Article  Google Scholar 

  17. Song K, Yao T, Ling Q, Mei T. Boosting image sentiment analysis with visual attention. Neurocomputing. 2018;312:218–28. https://doi.org/10.1016/j.neucom.2018.05.104.

    Article  Google Scholar 

  18. Abid F, Li C, Alam M. Multi-source social media data sentiment analysis using bidirectional recurrent convolutional neural networks. Comput Commun. 2020;157:102–15. https://doi.org/10.1016/j.comcom.2020.04.002.

    Article  Google Scholar 

  19. Dashtipour K, Gogate M, Li J, Jiang F, Kong B, Hussain A. A hybrid persian sentiment analysis framework: integrating dependency grammar based rules and deep neural networks. Neurocomputing. 2020;380:1–10. https://doi.org/10.1016/j.neucom.2019.10.009.

    Article  Google Scholar 

  20. Jindal K, Aron R. A systematic study of sentiment analysis for social media data. In: Materials today: proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.048.

  21. Basiri ME, Nemati S, Abdar M, Cambria E, Acharya UR. Abcdm: An attention-based bidirectional cnn-rnn deep model for sentiment analysis. Futur Gener Comput Syst. 2021;115:279–94. https://doi.org/10.1016/j.future.2020.08.005.

    Article  Google Scholar 

  22. Hajek P, Barushka A, Munk M. Opinion mining of consumer reviews using deep neural networks with word-sentiment associations. In: IFIP international conference on artificial intelligence applications and innovations, pp. 419–429. Springer (2020). https://doi.org/10.1007/978-3-030-49161-1_35.

  23. Han Y, Liu M, Jing W. Aspect-level drug reviews sentiment analysis based on double bigru and knowledge transfer. IEEE Access. 2020;8:21314–25. https://doi.org/10.1109/ACCESS.2020.2969473.

    Article  Google Scholar 

  24. Kumar A, Narapareddy VT, Srikanth VA, Malapati A, Neti LBM. Sarcasm detection using multi-head attention based bidirectional lstm. IEEE Access. 2020;8:6388–97. https://doi.org/10.1109/ACCESS.2019.2963630.

    Article  Google Scholar 

  25. Xiong H, Liu Q, Song S, Cai Y. Region-based convolutional neural network using group sparse regularization for image sentiment classification. EURASIP J Image Video Process. 2019;2019(1):1–9. https://doi.org/10.1186/s13640-019-0433-8.

    Article  Google Scholar 

  26. Susanto Y, Cambria E, Ng BC, Hussain A. Ten years of sentic computing. Cognit Comput. 2021. https://doi.org/10.1007/s12559-021-09824-x.

    Article  Google Scholar 

  27. Cambria E, Speer R, Havasi C, Hussain A. Senticnet: A publicly available semantic resource for opinion mining. In: AAAI fall symposium: commonsense knowledge, vol. 10. Citeseer (2010).

  28. Cambria E, Havasi C, Hussain A. Senticnet 2: A semantic and affective resource for opinion mining and sentiment analysis. In: Twenty-Fifth international FLAIRS conference (2012).

  29. Cambria E, Olsher D, Rajagopal D. Senticnet 3: a common and common-sense knowledge base for cognition-driven sentiment analysis. In: Proceedings of the AAAI conference on artificial intelligence, vol. 28 (2014).

  30. Cambria E, Poria S, Bajpai R, Schuller B. Senticnet 4: A semantic resource for sentiment analysis based on conceptual primitives. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers, pp. 2666–2677 (2016).

  31. Cambria E, Poria S, Hazarika D, Kwok K. Senticnet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In: Proceedings of the AAAI conference on artificial intelligence, vol. 32 (2018).

  32. Vilares D, Peng H, Satapathy R, Cambria E. Babelsenticnet: a commonsense reasoning framework for multilingual sentiment analysis. In: 2018 IEEE symposium series on computational intelligence (SSCI), pp. 1292–1298. IEEE (2018). https://doi.org/10.1109/SSCI.2018.8628718.

  33. Dragoni M, Donadello I, Cambria E. Ontosenticnet 2: enhancing reasoning within sentiment analysis. IEEE Intell Syst. 2021;36:5.

    Google Scholar 

  34. Cambria E, Li Y, Xing FZ, Poria S, Kwok K. Senticnet 6: Ensemble application of symbolic and subsymbolic AI for sentiment analysis. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp. 105–114 (2020). https://doi.org/10.1145/3340531.3412003.

  35. Akhtar S, Ghosal D, Ekbal A, Bhattacharyya P, Kurohashi S. All-in-one: Emotion, sentiment and intensity prediction using a multi-task ensemble framework. IEEE Trans Affect Comput. 2019. https://doi.org/10.1109/TAFFC.2019.2926724.

    Article  Google Scholar 

  36. Akhtar MS, Ekbal A, Cambria E. How intense are you? predicting intensities of emotions and sentiments using stacked ensemble [application notes]. IEEE Comput Intell Mag. 2020;15(1):64–75. https://doi.org/10.1109/MCI.2019.2954667.

    Article  Google Scholar 

  37. Stappen L, Baird A, Cambria E, Schuller BW. Sentiment analysis and topic recognition in video transcriptions. IEEE Intell Syst. 2021;36(2):88–95. https://doi.org/10.1109/MIS.2021.3062200.

    Article  Google Scholar 

  38. Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S. Multimodal sentiment analysis using hierarchical fusion with context modeling. Knowl-Based Syst. 2018;161:124–33. https://doi.org/10.1016/j.knosys.2018.07.041.

    Article  Google Scholar 

  39. Wöllmer M, Weninger F, Knaup T, Schuller B, Sun C, Sagae K, Morency LP. Youtube movie reviews: sentiment analysis in an audio-visual context. IEEE Intell Syst. 2013;28(3):46–53. https://doi.org/10.1109/MIS.2013.34.

    Article  Google Scholar 

  40. Poria S, Peng H, Hussain A, Howard N, Cambria E. Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing. 2017;261:217–30. https://doi.org/10.1016/j.neucom.2016.09.117.

    Article  Google Scholar 

  41. Xu G, Li W, Liu J. A social emotion classification approach using multi-model fusion. Futur Gener Comput Syst. 2020;102:347–56. https://doi.org/10.1016/j.future.2019.07.007.

    Article  Google Scholar 

  42. Dashtipour K, Gogate M, Cambria E, Hussain A. A novel context-aware multimodal framework for persian sentiment analysis. Neurocomputing. 2021. https://doi.org/10.1016/j.neucom.2021.02.020.

  43. SIFT Image Features. https://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/MURRAY/SIFT.html.

  44. Pierezan J, dos Santos Coelho L, Mariani VC, de Vasconcelos Segundo EH, Prayogo D. Chaotic coyote algorithm applied to truss optimization problems. Comput Struct. 2021;242:106353. https://doi.org/10.1016/j.compstruc.2020.106353.

    Article  Google Scholar 

  45. Liu F, Zheng L, Zheng J. Hienn-dwe: a hierarchical neural network with dynamic word embeddings for document level sentiment classification. Neurocomputing. 2020;403:21–32. https://doi.org/10.1016/j.neucom.2020.04.084.

    Article  Google Scholar 

  46. Hamouda A, Rohaim M. Reviews classification using sentiwordnet lexicon. In: World congress on computer science and information technology, vol. 23, pp. 104–105. sn (2011).

  47. https://www.baeldung.com/cs/sentiment-analysis-dictionaries.

  48. Ruiz P. Understanding and visualizing DenseNets (2018). https://towardsdatascience.com/understanding-and-visualizing-densenets-7f688092391a.

  49. Al-Obaidi ATS, Abdullah HS, Ahmed ZO. Meerkat clan algorithm: a new swarm intelligence algorithm. Indones J Electr Eng Comput Sci. 2018;10(1):354–60. https://doi.org/10.11591/ijeecs.v10.i1.pp354-360.

    Article  Google Scholar 

  50. Fake News. https://kaggle.com/c/fake-news.

  51. Flickr8K. https://kaggle.com/shadabhussain/flickr8k.

Download references

Funding

This work has no funding resource.

Author information

Authors and Affiliations

  1. Lovely Professional University, Jalandhar, Punjab, India

    Kanika Jindal

  2. SVKM’s Narsee Monjee Institute of Management Studies (NMIMS) University, Mumbai, Maharashtra, India

    Rajni Aron

Authors
  1. Kanika Jindal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rajni Aron
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kanika Jindal.

Ethics declarations

Ethics approval and consent to participate

This article does not contain any studies with human participants or animals performed by any of the authors.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jindal, K., Aron, R. A Novel Visual-Textual Sentiment Analysis Framework for Social Media Data. Cogn Comput 13, 1433–1450 (2021). https://doi.org/10.1007/s12559-021-09929-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-021-09929-3

Keywords

Search

Navigation