Elsevier

Information Sciences

Volume 480, April 2019, Pages 273-286
Information Sciences

E2SAM: Evolutionary ensemble of sentiment analysis methods for domain adaptation

https://doi.org/10.1016/j.ins.2018.12.038Get rights and content

Highlights

  • Domain adaptation is one of the main challenges of sentiment analysis.

  • Most polarity detection methods are focused on a specific domain.

  • Ensemble methods may improve the performance of a set of polarity detection methods.

  • Evolutionary algorithms may find out the right combination of polarity classification methods in an ensemble classifier.

  • The results show that our claim holds on 13 corpora.

Abstract

Currently, a plethora of industrial and academic sentiment analysis methods for classifying the opinion polarity of a text are available and ready to use. However, each of those methods have their strengths and weaknesses, due mainly to the approach followed in their design (supervised/unsupervised) or the domain of text used in their development. The weaknesses are usually related to the capacity of generalisation of machine learning algorithms, and the lexical coverage of linguistic resources. Those issues are two of the main causes of one of the challenges of Sentiment Analysis, namely the domain adaptation problem. We argue that the right ensemble of a set of heterogeneous Sentiment Analysis Methods will lessen the domain adaptation problem. Thus, we propose a new methodology for optimising the contribution of a set of off-the-shelf Sentiment Analysis Methods in an ensemble classifier depending on the domain of the input text. The results clearly show that our claim holds.

Introduction

Sentiment Analysis (SA) is the field of Natural Language Processing (NLP) whose aim is to analyse automatically subjective information. People often express their opinions or sentiments towards events, topics, proposals, companies or products [23]. This area of study has grown over last few years due to the large amount of text stored in the Web 2.0 such as social networks, blogs and discussion platforms. Consequently, the interest in developing Sentiment Analysis Methods (SAMs) capable of detecting the polarity of a text has also risen [35], and nowadays there is a great variety of different tools trained for extracting and classifying opinions. The task of polarity classification can be defined as a binary or multi-class classification problem. In this paper, we consider polarity detection as a three class (Positive, Neutral and Negative) classification problem.

The performance of a SAM strongly depends on the learning approach followed [20]. Supervised based SAMs are mainly determined by the domain (financial [5], restaurant [22] or health [3]) and the genre (news, microblogging or reviews) of the data used in their training, whereas the unsupervised models depend on the language coverage of the linguistic resources used in their development. Consequently, there are two problems: (1) the generalisation capacity of machine learning (ML) algorithms, and (2) the lexical coverage of linguistic resources. These problems are widely known in the literature of SA as the domain adaptation problem for polarity detection and classification. For instance, the word unpredictable is negative in the domain of car reviews, “my car has an unpredictable steering”. However, unpredictable is positive in the domain of film reviews, “the plot of the last film that I watched is unpredictable” [23].

Due to the high industrial demand of SA methods, several off-the-shelf SAMs have been released in the last few years. Each one of those SAMs has its own characteristics, i.e. training data, learning approach, features used for representing the input text and so on. In summary, each of them have their advantages and drawbacks. Specifically, their main drawback is related to the domain adaptation problem, which means that they only perform well when they classify domain text which is similar to that of the training set.

In this paper, we argue that the domain adaptation problem can be diminished by the right combination of a set of off-the-shelf SAMs. Accordingly, we propose a new methodology called Evolutionary Ensemble of SAMs (E2SAM) for learning the most suitable combination of a set of off-the-shelf SAMs depending on the domain of the input text. E2SAM is built upon an evolutionary algorithm (EA), which is able to optimise the contribution of each base SAM [12]. Since E2SAM is based on a EA, we assessed the performance of three EAs in our specific scenario, specifically the implementation of a Memetic Algorithm [30], and the algorithms L-SHADE [42] and jSO [8], which reached strong results in CEC competitions.1

We select 7 off-the-shelf SAMs from the state-of-the-art, and we evaluate E2SAM on 13 corpora of reviews from different domains and text genre. We compare our proposal with two baselines ensemble methods described in [45] in order to demonstrate its effectiveness. The results show that E2SAM substantially outperforms the best SAM and the two baselines in 11 of the 13 corpora, which confirms the validity of E2SAM.

The rest of this work is organized as follows: Section 2 describes some related studies; Section 3 presents our proposal; Section 4 shows and analyses the results, and Section 5 details the conclusions and future work.

Section snippets

Sentiment analysis and related works

We briefly describe the SA task in Section 2.1, the use of evolutionary algorithms in SA in Section 2.2, the use of ensemble methods in SA in Section 2.3 and the challenge of domain adaptation in Section 2.4.

Evolutionary ensemble optimisation method

The performance of an off-the-shelf SAM depends on its learning approach and the relation between the domain and genre of the training and test data. We know that ensemble methods are able to overcome the results of individual classifiers, because they join the search space of the base systems and, consequently, they are able to find a better solution for specific expert domains. However, the challenge in the development of ensemble methods is how to learn the contribution of each base system

Experimental setup

In this section we describe the set up of our evaluation. First, we depict the corpora employed (Section 4.1), subsequently we define the evaluation framework (Section 4.2), then how the output of each of the base off-the-shelf SAMs is homogenised (Section 4.3), and the results reached by the baselines and our proposal and their analysis are in Section 4.5.

Conclusions

It is well-known that SA algorithms lack of versatility, i.e., their performance is soundly inefficient in domains which differ from their training domain. In order to address this problem, we propose to ensemble different base SAMs. We present a method built upon an evolutionary ensemble approach, E2SAM, which learns the right combination of base SAMs according to the domain of the input data. We compare E2SAM with the base SAMs and two ensembles as baselines.

The polarity detection results of

Acknowledgements

We want to take this opportunity to thank Matheus Araújo for providing us all their labelled datasets. We also thank Mike Thelwall for sharing the code of SentiStrength with us.

This research work is partially supported by the Spanish Government project TIN2017-89517-P, and a grant from the Fondo Europeo de Desarrollo Regional (FEDER). Eugenio Martínez Cámara was supported by the Spanish Government Programme Juan de la Cierva Formación (FJCI-2016-28353).

Miguel López studied Data Science’s M.Sc. at University of Granada and received a B.Sc. degree in Computer Science from the same university. Currently, he works as researcher at the University of Granada and he is specially interested on Natural Language Processing and Deep Learning.

References (49)

  • O. Appel et al.

    A consensus approach to the sentiment analysis problem driven by support-based iowa majority

    Int. J. Intell. Syst.

    (2017)
  • F.M. Plaza del Arco et al.

    COPOS: corpus of patient opinions in spanish. application of sentiment analysis techniques

    Procesamiento del Lenguaje Natural

    (2016)
  • M. Atzeni et al.

    Using frame-based resources for sentiment analysis within the financial domain

    Progr. Artif. Intell.

    (2018)
  • J. Blitzer et al.

    Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification

    Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics

    (2007)
  • J. Brest et al.

    Single objective real-parameter optimization: algorithm jso

    2017 IEEE Congress on Evolutionary Computation (CEC)

    (2017)
  • E. Cambria et al.

    Sentic Computing

    (2012)
  • A. Das et al.

    Subjectivity detection using genetic algorithm

    Proceedings of the 1st Workshop on Computational Approaches to Subjectivity and Sentiment Analysis

    (2010)
  • A.E. Eiben et al.

    Introduction to Evolutionary Computing

    (2015)
  • S. García et al.

    A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the cec’2005 special session on real parameter optimization

    J. Heurist.

    (2008)
  • M. Garza-Fabre et al.

    Generating, maintaining, and exploiting diversity in a memetic algorithm for protein structure prediction

    Evol. Comput.

    (2016)
  • M. Hu et al.

    Mining and summarizing customer reviews

    Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

    (2004)
  • C.J. Hutto et al.

    Vader: a parsimonious rule-based model for sentiment analysis of social media text

    Eighth International Conference on Weblogs and Social Media (ICWSM-14)

    (2014)
  • J. Jiang et al.

    Instance weighting for domain adaptation in nlp

    Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics

    (2007)
  • R. Jongeling et al.

    On negative results when using sentiment analysis tools for software engineering research

    Empir. Software Eng.

    (2017)
  • Cited by (32)

    • Augmented language model with deep learning adaptation on sentiment analysis for E-learning recommendation

      2022, Cognitive Systems Research
      Citation Excerpt :

      Hung, Wu, and Chou (2021) constructed a revised lexicon by re-assigning a new sentiment score to the objective words in SenticNet 3 to sort out the problem of the lack of sentiments of objective words. Set of the researchers focused on leveraging ensemble models to enhance the accuracy of the sentiment analysis models, for domain adaption (El Mekki, El Mahdaouy, Berrada, & Khoumsi, 2021; López, Valdivia, Martínez-Cámara, Luzón, & Herrera, 2019), towards sentiment analysis for Aspect-based (Gong, Yu, & Xia, 2020) and other for analyzing sentiment of the review (Tiwari & Nagpal, 2021). While Cambria, Li, Xing, Poria, and Kwok (2020) reported a combination of top-down and bottom-up based on an ensemble of symbolic and subsymbolic AI tools, wherein the knowledge base for sentiment analysis is created by integrating logical reasoning and deep learning to infer the polarity of text.

    • Ensemble learning-based approach for improving generalization capability of machine reading comprehension systems

      2021, Neurocomputing
      Citation Excerpt :

      Obviously, this model is a complex one with many parameters which is in contrary with our purpose. López et al. [43] proposed an evolutionary-based ensemble technique to aggregate multiple models’ outputs according to the input text domain in the sentiment analysis task. The ensemble models were trained on new datasets; however, in real-world settings, there may be no or little labeled data from the new domain.

    • Sentiment analysis with genetic programming

      2021, Information Sciences
      Citation Excerpt :

      Thus, there is great interest from industry, e-commerce, celebrities, and political agents, among other entities, in applying SA to these texts. This demand has motivated research in SA [2–4]. SA is performed by sentiment classifiers, which are text classifiers developed to identify the polarity of pieces of text according to the prevalent sentiment they transmit.

    • Fuzzy logic applied to opinion mining: A review

      2021, Knowledge-Based Systems
      Citation Excerpt :

      A vast number of new studies in the field of opinion mining are published every year. Different new contributions can be found regarding to the use of generation, adaptation or use of new sentimental lexicons [2–6], the application of classical machine learning algorithms such as support vector machines [7], naive Bayes [8], regression methods [9], decision trees [10], clustering [11] or even, ensemble classifiers and genetic algorithms [12] or transfer learning [13]. And particularly, the application of deep learning is gaining a lot of interest [14].

    View all citing articles on Scopus

    Miguel López studied Data Science’s M.Sc. at University of Granada and received a B.Sc. degree in Computer Science from the same university. Currently, he works as researcher at the University of Granada and he is specially interested on Natural Language Processing and Deep Learning.

    Ana Valdivia obtained her B.Sc. in Mathematics from Polytechnic University of Catalonia (UPC) in 2014. She then worked as a research assistant at IESE Business School of Barcelona in 2015. She received her M.Sc. in Data Science and Computer Engineering from University of Granada (UGR) in 2016. Now, she is currently pursuing the Ph.D. degree in Computer Science and Artificial Intelligence in UGR. Her research is focused on Natural Language Processing, Sentiment Analysis and Deep Learning. She is also a strong advocate for Data Science for Social Good.

    Eugenio Martínez-Cámara is a postdoctoral researcher at University of Granada, Spain. He received a B.Sc. degree in Computer Science and Management and M.Sc. degree in Computer Science from the University of Jaén, Spain, in 2008 and 2010, respectively. He received his Ph.D. in Computer Science in 2015 at the University of Jaén. Dr. Martínez-Cámara also worked as postdoctoral researcher at Technische Universität Darmstadt, Germany. His current research interest are sentiment analysis, information extraction and the use of deep learning in natural language processing.

    M. Victoria Luzón is an associate professor in the Software Engineering Department at University of Granada. Her research interests include sentiment analysis, artificial intelligence, computer graphics and cultural heritage. Luzón has a Ph.D. in Industrial Engineering from the University of Vigo, Spain. Contact her at [email protected].

    Francisco Herrera received his M.Sc. in Mathematics in 1988 and Ph.D. in Mathematics in 1991 (University of Granada, Spain). He is currently a professor in the Department of Computer Science and Artificial Intelligence at the University of Granada. Prof. Herrera has supervised 42 Ph.D. thesis and published more than 380 journal papers (h-index = 121, Scholar Google). He currently acts as Editor in Chief of the international journals “Information Fusion” (Elsevier) and “Progress in Artificial Intelligence” (Springer). He has been selected as a Highly Cited Researcher http://highlycited.com/ (fields of Computer Science and Engineering, respectively, 2014 to present, Clarivate Analytics). His current research interests include soft computing (fuzzy modeling, evolutionary algorithms and deep learning), computing with words, information fusion, and data science (data preprocessing, prediction and big data).

    View full text