E2SAM: Evolutionary ensemble of sentiment analysis methods for domain adaptation
Introduction
Sentiment Analysis (SA) is the field of Natural Language Processing (NLP) whose aim is to analyse automatically subjective information. People often express their opinions or sentiments towards events, topics, proposals, companies or products [23]. This area of study has grown over last few years due to the large amount of text stored in the Web 2.0 such as social networks, blogs and discussion platforms. Consequently, the interest in developing Sentiment Analysis Methods (SAMs) capable of detecting the polarity of a text has also risen [35], and nowadays there is a great variety of different tools trained for extracting and classifying opinions. The task of polarity classification can be defined as a binary or multi-class classification problem. In this paper, we consider polarity detection as a three class (Positive, Neutral and Negative) classification problem.
The performance of a SAM strongly depends on the learning approach followed [20]. Supervised based SAMs are mainly determined by the domain (financial [5], restaurant [22] or health [3]) and the genre (news, microblogging or reviews) of the data used in their training, whereas the unsupervised models depend on the language coverage of the linguistic resources used in their development. Consequently, there are two problems: (1) the generalisation capacity of machine learning (ML) algorithms, and (2) the lexical coverage of linguistic resources. These problems are widely known in the literature of SA as the domain adaptation problem for polarity detection and classification. For instance, the word unpredictable is negative in the domain of car reviews, “my car has an unpredictable steering”. However, unpredictable is positive in the domain of film reviews, “the plot of the last film that I watched is unpredictable” [23].
Due to the high industrial demand of SA methods, several off-the-shelf SAMs have been released in the last few years. Each one of those SAMs has its own characteristics, i.e. training data, learning approach, features used for representing the input text and so on. In summary, each of them have their advantages and drawbacks. Specifically, their main drawback is related to the domain adaptation problem, which means that they only perform well when they classify domain text which is similar to that of the training set.
In this paper, we argue that the domain adaptation problem can be diminished by the right combination of a set of off-the-shelf SAMs. Accordingly, we propose a new methodology called Evolutionary Ensemble of SAMs (E2SAM) for learning the most suitable combination of a set of off-the-shelf SAMs depending on the domain of the input text. E2SAM is built upon an evolutionary algorithm (EA), which is able to optimise the contribution of each base SAM [12]. Since E2SAM is based on a EA, we assessed the performance of three EAs in our specific scenario, specifically the implementation of a Memetic Algorithm [30], and the algorithms L-SHADE [42] and jSO [8], which reached strong results in CEC competitions.1
We select 7 off-the-shelf SAMs from the state-of-the-art, and we evaluate E2SAM on 13 corpora of reviews from different domains and text genre. We compare our proposal with two baselines ensemble methods described in [45] in order to demonstrate its effectiveness. The results show that E2SAM substantially outperforms the best SAM and the two baselines in 11 of the 13 corpora, which confirms the validity of E2SAM.
The rest of this work is organized as follows: Section 2 describes some related studies; Section 3 presents our proposal; Section 4 shows and analyses the results, and Section 5 details the conclusions and future work.
Section snippets
Sentiment analysis and related works
We briefly describe the SA task in Section 2.1, the use of evolutionary algorithms in SA in Section 2.2, the use of ensemble methods in SA in Section 2.3 and the challenge of domain adaptation in Section 2.4.
Evolutionary ensemble optimisation method
The performance of an off-the-shelf SAM depends on its learning approach and the relation between the domain and genre of the training and test data. We know that ensemble methods are able to overcome the results of individual classifiers, because they join the search space of the base systems and, consequently, they are able to find a better solution for specific expert domains. However, the challenge in the development of ensemble methods is how to learn the contribution of each base system
Experimental setup
In this section we describe the set up of our evaluation. First, we depict the corpora employed (Section 4.1), subsequently we define the evaluation framework (Section 4.2), then how the output of each of the base off-the-shelf SAMs is homogenised (Section 4.3), and the results reached by the baselines and our proposal and their analysis are in Section 4.5.
Conclusions
It is well-known that SA algorithms lack of versatility, i.e., their performance is soundly inefficient in domains which differ from their training domain. In order to address this problem, we propose to ensemble different base SAMs. We present a method built upon an evolutionary ensemble approach, E2SAM, which learns the right combination of base SAMs according to the domain of the input data. We compare E2SAM with the base SAMs and two ensembles as baselines.
The polarity detection results of
Acknowledgements
We want to take this opportunity to thank Matheus Araújo for providing us all their labelled datasets. We also thank Mike Thelwall for sharing the code of SentiStrength with us.
This research work is partially supported by the Spanish Government project TIN2017-89517-P, and a grant from the Fondo Europeo de Desarrollo Regional (FEDER). Eugenio Martínez Cámara was supported by the Spanish Government Programme Juan de la Cierva Formación (FJCI-2016-28353).
Miguel López studied Data Science’s M.Sc. at University of Granada and received a B.Sc. degree in Computer Science from the same university. Currently, he works as researcher at the University of Granada and he is specially interested on Natural Language Processing and Deep Learning.
References (49)
- et al.
Numerical solution of systems of second-order boundary value problems using continuous genetic algorithm
Inf. Sci.
(2014) - et al.
Opinion mining and information fusion: a survey
Inf. Fusion
(2016) - et al.
Distinguishing between facts and opinions for sentiment analysis: Survey and challenges
Inf. Fusion
(2018) - et al.
Real-coded genetic algorithms and interval-schemata
- et al.
Sentiment analysis: bayesian ensemble learning
Decis. Support Syst.
(2014) - et al.
Tweet sentiment analysis with classifier ensembles
Decis. Support Syst.
(2014) - et al.
A review of natural language processing techniques for opinion mining systems
Inf. Fusion
(2017) - et al.
Consensus vote models for detecting and filtering neutrality in sentiment analysis
Inf. Fusion
(2018) - et al.
A survey of multiple classifier systems as hybrid systems
Inf. Fusion
(2014) - et al.
Sentiment analysis in multiple languages: feature selection for opinion classification in web forums
ACM Trans. Inf. Syst.
(2008)
A consensus approach to the sentiment analysis problem driven by support-based iowa majority
Int. J. Intell. Syst.
COPOS: corpus of patient opinions in spanish. application of sentiment analysis techniques
Procesamiento del Lenguaje Natural
Using frame-based resources for sentiment analysis within the financial domain
Progr. Artif. Intell.
Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification
Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics
Single objective real-parameter optimization: algorithm jso
2017 IEEE Congress on Evolutionary Computation (CEC)
Sentic Computing
Subjectivity detection using genetic algorithm
Proceedings of the 1st Workshop on Computational Approaches to Subjectivity and Sentiment Analysis
Introduction to Evolutionary Computing
A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the cec’2005 special session on real parameter optimization
J. Heurist.
Generating, maintaining, and exploiting diversity in a memetic algorithm for protein structure prediction
Evol. Comput.
Mining and summarizing customer reviews
Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining
Vader: a parsimonious rule-based model for sentiment analysis of social media text
Eighth International Conference on Weblogs and Social Media (ICWSM-14)
Instance weighting for domain adaptation in nlp
Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics
On negative results when using sentiment analysis tools for software engineering research
Empir. Software Eng.
Cited by (32)
Unsupervised domain adaptation via re-weighted transfer subspace learning with inter-class sparsity
2023, Knowledge-Based SystemsAugmented language model with deep learning adaptation on sentiment analysis for E-learning recommendation
2022, Cognitive Systems ResearchCitation Excerpt :Hung, Wu, and Chou (2021) constructed a revised lexicon by re-assigning a new sentiment score to the objective words in SenticNet 3 to sort out the problem of the lack of sentiments of objective words. Set of the researchers focused on leveraging ensemble models to enhance the accuracy of the sentiment analysis models, for domain adaption (El Mekki, El Mahdaouy, Berrada, & Khoumsi, 2021; López, Valdivia, Martínez-Cámara, Luzón, & Herrera, 2019), towards sentiment analysis for Aspect-based (Gong, Yu, & Xia, 2020) and other for analyzing sentiment of the review (Tiwari & Nagpal, 2021). While Cambria, Li, Xing, Poria, and Kwok (2020) reported a combination of top-down and bottom-up based on an ensemble of symbolic and subsymbolic AI tools, wherein the knowledge base for sentiment analysis is created by integrating logical reasoning and deep learning to infer the polarity of text.
Ensemble learning-based approach for improving generalization capability of machine reading comprehension systems
2021, NeurocomputingCitation Excerpt :Obviously, this model is a complex one with many parameters which is in contrary with our purpose. López et al. [43] proposed an evolutionary-based ensemble technique to aggregate multiple models’ outputs according to the input text domain in the sentiment analysis task. The ensemble models were trained on new datasets; however, in real-world settings, there may be no or little labeled data from the new domain.
Cross-domain sentiment classification via parameter transferring and attention sharing mechanism
2021, Information SciencesSentiment analysis with genetic programming
2021, Information SciencesCitation Excerpt :Thus, there is great interest from industry, e-commerce, celebrities, and political agents, among other entities, in applying SA to these texts. This demand has motivated research in SA [2–4]. SA is performed by sentiment classifiers, which are text classifiers developed to identify the polarity of pieces of text according to the prevalent sentiment they transmit.
Fuzzy logic applied to opinion mining: A review
2021, Knowledge-Based SystemsCitation Excerpt :A vast number of new studies in the field of opinion mining are published every year. Different new contributions can be found regarding to the use of generation, adaptation or use of new sentimental lexicons [2–6], the application of classical machine learning algorithms such as support vector machines [7], naive Bayes [8], regression methods [9], decision trees [10], clustering [11] or even, ensemble classifiers and genetic algorithms [12] or transfer learning [13]. And particularly, the application of deep learning is gaining a lot of interest [14].
Miguel López studied Data Science’s M.Sc. at University of Granada and received a B.Sc. degree in Computer Science from the same university. Currently, he works as researcher at the University of Granada and he is specially interested on Natural Language Processing and Deep Learning.
Ana Valdivia obtained her B.Sc. in Mathematics from Polytechnic University of Catalonia (UPC) in 2014. She then worked as a research assistant at IESE Business School of Barcelona in 2015. She received her M.Sc. in Data Science and Computer Engineering from University of Granada (UGR) in 2016. Now, she is currently pursuing the Ph.D. degree in Computer Science and Artificial Intelligence in UGR. Her research is focused on Natural Language Processing, Sentiment Analysis and Deep Learning. She is also a strong advocate for Data Science for Social Good.
Eugenio Martínez-Cámara is a postdoctoral researcher at University of Granada, Spain. He received a B.Sc. degree in Computer Science and Management and M.Sc. degree in Computer Science from the University of Jaén, Spain, in 2008 and 2010, respectively. He received his Ph.D. in Computer Science in 2015 at the University of Jaén. Dr. Martínez-Cámara also worked as postdoctoral researcher at Technische Universität Darmstadt, Germany. His current research interest are sentiment analysis, information extraction and the use of deep learning in natural language processing.
M. Victoria Luzón is an associate professor in the Software Engineering Department at University of Granada. Her research interests include sentiment analysis, artificial intelligence, computer graphics and cultural heritage. Luzón has a Ph.D. in Industrial Engineering from the University of Vigo, Spain. Contact her at [email protected].
Francisco Herrera received his M.Sc. in Mathematics in 1988 and Ph.D. in Mathematics in 1991 (University of Granada, Spain). He is currently a professor in the Department of Computer Science and Artificial Intelligence at the University of Granada. Prof. Herrera has supervised 42 Ph.D. thesis and published more than 380 journal papers (h-index = 121, Scholar Google). He currently acts as Editor in Chief of the international journals “Information Fusion” (Elsevier) and “Progress in Artificial Intelligence” (Springer). He has been selected as a Highly Cited Researcher http://highlycited.com/ (fields of Computer Science and Engineering, respectively, 2014 to present, Clarivate Analytics). His current research interests include soft computing (fuzzy modeling, evolutionary algorithms and deep learning), computing with words, information fusion, and data science (data preprocessing, prediction and big data).