Abstract
Over the years there has been a lot of speculation with respect to single modal sentiment analysis of twitter (which is one of the world’s largest micro blogging platforms) data i.e. either text or image mining. But unfortunately most of the researchers didn’t use the non-trivial elements such as memes (i.e. combination of image and text data) and GIFs (i.e. combination of video and audio data) which dominate the twitter world today. Hence looking at the limitations of the existing systems we proposes a novel framework called “Multimodal Twitter Sentiment Analysis using Feature Learning” which defines the polarity of the tweets by considering all types of data such as text, image and GIFs. The framework consist of three main modules i.e. Data Collection for gathering real-time tweets using twitter streaming API, Data Processing module which has context-aware hybrid algorithm used for text sentiment analysis and ‘Fast R-CNN’ for image sentiment analysis. GIFs are handled using an optical character recognizer which separate texts from images for defining the polarity and finally multimodal sentiment scoring is done by aggregating polarity scores of images are texts. Evaluation results of proposed framework shows accuracy of 96.7% against SVM and Naïve Bayes which outperforms the single modal sentiment analysis models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Antonakaki, D., Fragopoulou, P., Ioannidis, S.: A survey of Twitter research: data model, graph structure, sentiment analysis and attacks. Expert Syst. Appl. 164, 114006 (2021)
Aquino, P.A., López, V.F., Moreno, M.N., Muñoz, M.D., Rodríguez, S.: Opinion mining system for Twitter sentiment analysis. In: Antonio, E., de la Cal, J., Flecha, R.V., Quintián, H., Corchado, E. (eds.) Hybrid Artificial Intelligent Systems: 15th International Conference, HAIS 2020, Gijón, Spain, November 11–13, 2020, Proceedings, pp. 465–476. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-61705-9_38
Mehta, R.P., Sanghvi, M.A., Shah, D.K., Singh, A.: Sentiment analysis of tweets using supervised learning algorithms. In: Luhach, A.K., Kosa, J.A., Poonia, R.C., Gao, X.-Z., Singh, D. (eds.) First International Conference on Sustainable Technologies for Computational Intelligence. AISC, vol. 1045, pp. 323–338. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-0029-9_26
Ortis, A., Farinella, G.M., Battiato, S.: Survey on visual sentiment analysis. IET Image Proc. 14(8), 1440–1456 (2020)
Jianqiang, Z., Xiaolin, G.: Comparison research on text pre-processing methods on twitter sentiment analysis. IEEE Access 5, 2870–2879 (2017)
Zimbra, D., Abbasi, A., Zeng, D., Chen, H.: The state-of-the-art in Twitter sentiment analysis: a review and benchmark evaluation. ACM Trans. Manage. Inf. Syst. 9(2), 1–29 (2018)
Symeonidis, S., Effrosynidis, D., Arampatzis, A.: A comparative evaluation of pre-processing techniques and their interactions for twitter sentiment analysis. Expert Syst. Appl. 110, 298–310 (2018)
Chaturvedi, I., Cambria, E., Welsch, R.E., Herrera, F.: Distinguishing between facts and opinions for sentiment analysis: survey and challenges. Inf. Fusion 44, 65–77 (2018)
Giachanou, A., Crestani, F.: Like it or not: a survey of twitter sentiment analysis methods. ACM Comput. Surveys 49(2), 1–41 (2016)
Zou, P., Yang, S.: Multimodal tweet sentiment classification algorithm based on attention mechanism. In: Monreale, A., Alzate, C., Kamp, M., Krishnamurthy, Y., Paurat, D., Sayed-Mouchaweh, M., Bifet, A., Gama, J., Ribeiro, R.P. (eds.) ECML PKDD 2018 Workshops: DMLE 2018 and IoTStream 2018, Dublin, Ireland, September 10–14, 2018, Revised Selected Papers, pp. 68–79. Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-14880-5_6
Mittal, N., Sharma, D., Joshi, M.L.: Image sentiment analysis using deep learning. In: 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 684–687. IEEE (2018)
Ahsan, U., De Choudhury, M., Essa, I.: Towards using visual attributes to infer image sentiment of social events. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 1372–1379. IEEE (2017)
Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis: a survey. Wiley Interdiscipl. Rev. Data Mining Knowl. Discov. 8(4), e1253 (2018)
Soleymani, M., Garcia, D., Jou, B., Schuller, B., Chang, S.F., Pantic, M.: A survey of multimodal sentiment analysis. Image Vis. Comput. 65, 3–14 (2017)
Yue, L., Chen, W., Li, X., Zuo, W., Yin, M.: A survey of sentiment analysis in social media. Knowl. Inf. Syst. 60(2), 617–663 (2018). https://doi.org/10.1007/s10115-018-1236-4
Saleena, N.: An ensemble classification system for twitter sentiment analysis. Procedia Comput. Sci. 132, 937–946 (2018)
Jianqiang, Z., Xiaolin, G., Xuejun, Z.: Deep convolution neural networks for twitter sentiment analysis. IEEE Access 6, 23253–23260 (2018)
Naseem, U., Razzak, I., Musial, K., Imran, M.: Transformer based deep intelligent contextual embedding for twitter sentiment analysis. Futur. Gener. Comput. Syst. 113, 58–69 (2020)
Nagamanjula, R., Pethalakshmi, A.: A novel framework based on bi-objective optimization and LAN 2 FIS for Twitter sentiment analysis. Soc. Netw. Anal. Min. 10, 1–16 (2020)
Alharbi, A.S.M., de Doncker, E.: Twitter sentiment analysis with a deep neural network: an enhanced approach using user behavioral information. Cogn. Syst. Res. 54, 50–61 (2019)
Yadav, A., Vishwakarma, D.K.: A deep learning architecture of RA-DLNet for visual sentiment analysis. Multimedia Syst. 26(4), 431–451 (2020). https://doi.org/10.1007/s00530-020-00656-7
Kumar, A., Srinivasan, K., Cheng, W.H., Zomaya, A.Y.: Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data. Inf. Process. Manage. 57(1), 102141 (2020)
Huang, F., Zhang, X., Zhao, Z., Xu, J., Li, Z.: Image–text sentiment analysis via deep multimodal attentive fusion. Knowl. Based Syst. 167, 26–37 (2019)
Zhao, Z., Zhu, H., Xue, Z., Liu, Z., Tian, J., Chua, M.C.H., et al.: An image-text consistency driven multimodal sentiment analysis approach for social media. Inf. Process. Manage. 56(6), 102097 (2019)
Huang, F., Wei, K., Weng, J., Li, Z.: Attention-based modality-gated networks for image-text sentiment analysis. ACM Trans. Multimedia Comput. Commun. Appl. 16(3), 1–19 (2020)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Murthy, J.S., Shekar, A.C., Bhattacharya, D., Namratha, R., Sripriya, D. (2021). A Novel Framework for Multimodal Twitter Sentiment Analysis Using Feature Learning. In: Singh, M., Tyagi, V., Gupta, P.K., Flusser, J., Ören, T., Sonawane, V.R. (eds) Advances in Computing and Data Sciences. ICACDS 2021. Communications in Computer and Information Science, vol 1441. Springer, Cham. https://doi.org/10.1007/978-3-030-88244-0_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-88244-0_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88243-3
Online ISBN: 978-3-030-88244-0
eBook Packages: Computer ScienceComputer Science (R0)