Skip to main content
SpringerLink
Log in

A hybrid fusion-based machine learning framework to improve sentiment prediction of assamese in low resource setting

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In recent times, sentiment analysis works are dedicated to unimodal data and less effort has been paid to multimodal data. Due to the growth of multimedia, multimodal sentiment analysis is growing as one of the forefront research areas in natural language processing. However, the presence of multimodal information helps us to get a clearer understanding of the sentiment. However, multimodal sentiment analysis in a low-resource setting is yet to be explored for several resource-poor languages like Assamese. In this paper, we propose a hybrid fusion-based multimodal sentiment analysis framework for the Assamese news domain. We concentrate on the lexical features and the specific image objects to develop two individual semantic and visual models and predict the sentiment separately. Next, we combine the image and the text features by employing feature-level fusion to introduce a multimodal for joint sentiment classification. Finally, a decision/late fusion scheme is applied to three models, i.e., textual, visual, and multimodal systems, for final sentiment prediction. The hybrid fusion multimodal framework improves sentiment prediction performance over a single modality and feature-level multimodality of the Assamese news domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study. Data available on request from the authors.

Notes

  1. http://learn101.org

  2. http://ganaadhikar.com

  3. https://niyomiyabarta.org/home/

  4. https://www.asomiyapratidin.in/

References

  1. Baroi S J, Singh N, Das R, Singh T D (2020) NITS-Hinglish-sentimix at SemEval-2020 task 9: sentiment analysis for code-mixed social media text using an ensemble model

  2. Borth D, Ji R, Chen T, Breuel T, Chang S -F (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM international conference on multimedia, pp 223–232

  3. Cambria E, Hazarika D, Poria S, Hussain A, Subramanyam R (2017) Benchmarking multimodal sentiment analysis. In: International conference on computational linguistics and intelligent text processing. Springer, pp 166–179

  4. Campos V, Jou B, Giro-i-Nieto X (2017) From pixels to sentiment: fine-tuning cnns for visual sentiment prediction. Image Vis Comput 65:15–22

    Article  Google Scholar 

  5. Cao D, Ji R, Lin D, Li S (2016) Visual sentiment topic model based microblog image sentiment analysis. Multimed Tools Appl 75(15):8955–8968

    Article  Google Scholar 

  6. Chen T, Guestrin C (2016) Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd Acm Sigkdd international conference on knowledge discovery and data mining, pp 785–794

  7. Chen X, Wang Y, Liu Q (2017) Visual and textual sentiment analysis using deep fusion convolutional neural networks. In: 2017 IEEE International conference on image processing (ICIP). IEEE, pp 1557–1561

  8. Das A, Bandyopadhyay S (2010) Opinion-polarity identification in bengali. In: International conference on computer processing of oriental languages, pp 169–182

  9. Das A, Bandyopadhyay S (2010) Phrase-level polarity identification for bangla. Int J Comput Linguist Appl (IJCLA) 1(1–2):169–182

    Google Scholar 

  10. Das R, Singh T D (2021) A step towards sentiment analysis of assamese news articles using lexical features. In: Proceedings of the international conference on computing and communication systems: I3CS 2020, NEHU, Shillong, India, vol 170. Springer, p 15

  11. Das R, Singh T D (2022) Assamese news image caption generation using attention mechanism. Multimed Tools Appl 81(7):10051–10069

    Article  Google Scholar 

  12. Das R, Singh T D (2022) A multi-stage multimodal framework for sentiment analysis of assamese in low resource setting. Expert Syst Appl 117575

  13. Ghosal D, Akhtar M S, Chauhan D, Poria S, Ekbal A, Bhattacharyya P (2018) Contextual inter-modal attention for multi-modal sentiment analysis. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 3454–3466

  14. Han W, Chen H, Gelbukh A, Zadeh A, Morency L-P, Poria S (2021) Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis. In: Proceedings of the 2021 international conference on multimodal interaction, pp 6–15

  15. Hazarika D, Zimmermann R, Poria S (2020) Misa: modality-invariant and-specific representations for multimodal sentiment analysis. In: Proceedings of the 28th ACM international conference on multimedia, pp 1122–1131

  16. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  CAS  PubMed  Google Scholar 

  17. Jindal S, Singh S (2015) Image sentiment analysis using deep convolutional neural networks with domain specific fine tuning. In: 2015 International conference on information processing (ICIP). IEEE, pp 447–451

  18. Kumar A, Jaiswal A (2017) Image sentiment analysis using convolutional neural network. In: International conference on intelligent systems design and applications. Springer, pp 464–473

  19. LeCun Y, Haffner P, Bottou L, Bengio Y, Bottou L, Haffner P, Howard P, Simard P, Bengio Y, LeCun Y (1988) Object recognition with gradient-based learning. Feature Grouping 66:233–240

    Google Scholar 

  20. Meetei L S, Singh T D, Borgohain S K, Bandyopadhyay S (2021) Low resource language specific pre-processing and features for sentiment analysis task. Lang Resour Eval 1–23

  21. Ortis A, Farinella G M, Torrisi G, Battiato S (2020) Exploiting objective text description of images for visual sentiment analysis. Multimed Tools Appl 1–24

  22. Pang B, Lee L (2004) A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the 42nd annual meeting on association for computational linguistics. Association for Computational Linguistics, p 271

  23. Pang B, Lee L, Vaithyanathan S (2002) Thumbs up?: sentiment classification using machine learning techniques. In: Proceedings of the ACL-02 conference on empirical methods in natural language processing, vol 10. Association for Computational Linguistics, pp 79–86

  24. Pereira M H R, Pádua F L C, Pereira A C M, Benevenuto F, Dalip D H (2016) Fusing audio, textual, and visual features for sentiment analysis of news videos. In: Tenth international AAAI conference on web and social media

  25. Poria S, Gelbukh A, Cambria E, Yang P, Hussain A, Durrani T (2012) Merging senticnet and wordnet-affect emotion lists for sentiment analysis. In: 2012 IEEE 11th international conference on signal processing, vol 2. IEEE, pp 1251–1255

  26. Poria S, Cambria E, Gelbukh A (2015) Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp 2539–2544

  27. Poria S, Cambria E, Hazarika D, Mazumder N, Zadeh A, Morency L -P (2017) Multi-level multiple attentions for contextual multimodal sentiment analysis. In: 2017 IEEE International conference on data mining (ICDM). IEEE, pp 1033–1038

  28. Poria S, Cambria E, Hazarika D, Majumder N, Zadeh A, Morency L -P (2017) Context-dependent sentiment analysis in user-generated videos. In: Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), pp 873–883

  29. Sarkar K, Bhowmick M (2017) Sentiment polarity detection in bengali tweets using multinomial na2017 IEEE Calcutta Conference (CALCON). IEEE, pp 31–36

  30. Sharma A, Dey S (2012) A comparative study of feature selection and machine learning techniques for sentiment analysis. In: Proceedings of the 2012 ACM research in applied computation symposium, pp 1–7

  31. Sherstinsky A (2020) Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena 404:132306

    Article  MathSciNet  Google Scholar 

  32. Siersdorfer S, Minack E, Deng F, Hare J (2010) Analyzing and predicting sentiment of images on the social web. In: Proceedings of the 18th ACM international conference on multimedia, pp 715–718

  33. Singh T D, Singh T J, Shadang M, Thokchom S (2021) Review comments of manipuri online video: Good, bad or ugly. In: Proceedings of the international conference on computing and communication systems: I3CS 2020, NEHU, Shillong, India, vol 170. Springer, p 45

  34. Soleymani M, Garcia D, Jou B, Schuller B, Chang S -F, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14

    Article  Google Scholar 

  35. Song K, Yao T, Ling Q, Mei T (2018) Boosting image sentiment analysis with visual attention. Neurocomputing 312:218–228

    Article  Google Scholar 

  36. Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3156–3164

  37. Wang J, Fu J, Xu Y, Mei T (2016) Beyond object recognition: visual sentiment analysis with deep coupled adjective and noun neural networks. In: IJCAI, pp 3484–3490

  38. You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Twenty-ninth AAAI conference on artificial intelligence

  39. You Q, Luo J, Jin H, Yang J (2015) Joint visual-textual sentiment analysis with deep neural networks. In: Proceedings of the 23rd ACM international conference on multimedia, pp 1071–1074

  40. You Q, Cao L, Jin H, Luo J (2016) Robust visual-textual sentiment analysis: when attention meets tree-structured recursive neural networks. In: Proceedings of the 24th ACM international conference on multimedia, pp 1008–1017

  41. Yuan J, Mcdonough S, You Q, Luo J (2013) Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the second international workshop on issues of sentiment discovery and opinion mining, pp 1–8

  42. Zhang Y, Shang L, Jia X (2015) Sentiment analysis on microblogging by integrating text and image features. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, pp 52–63

  43. Zhao Z, Zhu H, Xue Z, Liu Z, Tian J, Chua MCH, Liu M (2019) An image-text consistency driven multimodal sentiment analysis approach for social media. Inf Process 56(6):102097

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Engineering, National Institute of Technology Silchar, Silchar, 788010, Assam, India

    Ringki Das & Thoudam Doren Singh

Authors
  1. Ringki Das
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thoudam Doren Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ringki Das.

Ethics declarations

Conflict of Interests

There is no conflict of interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Das, R., Singh, T.D. A hybrid fusion-based machine learning framework to improve sentiment prediction of assamese in low resource setting. Multimed Tools Appl 83, 22153–22172 (2024). https://doi.org/10.1007/s11042-023-15356-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15356-3

Keywords

Search

Navigation