Elsevier

Expert Systems with Applications

Volume 38, Issue 12, November–December 2011, Pages 14799-14804
Expert Systems with Applications

Experiments with SVM to classify opinions in different domains

https://doi.org/10.1016/j.eswa.2011.05.070Get rights and content

Abstract

Recently, opinion mining is receiving more attention due to the abundance of forums, blogs, e-commerce web sites, news reports and additional web sources where people tend to express their opinions. Opinion mining is the task of identifying whether the opinion expressed in a document is positive or negative about a given topic. In this paper we explore this new research area applying Support Vector Machines (SVM) for testing different domains of data sets and using several weighting schemes. We have accomplished experiments with different features on three corpora. Two of them have already been used in several works. The last one has been built from Amazon.com specifically for this paper in order to prove the feasibility of the SVM for different domains.

Highlights

► A opinion mining corpus has been built for research purposes. ► Several experiments are carried out using SVM. ► Different weighting models, n-gram schemes and domains have been tested.

Introduction

Nowadays, the web is the most important place for expressing sentiments, evaluations, and reviews. Lots of people are tending to give their opinions in forums, blogs or wikis. However, with the rapid growth of e-commerce activity, the number of reviews and opinions about products has increased exponentially and this source of information is becoming unworkable. A customer who wants to buy a product usually searches information on the Internet trying to find analyses of other customers. In fact, web sites such as Amazon,1 Epinions2 or IMDb3 can affect the customer decision.

Nevertheless, it is becoming an impossible task to read all of these reviews and opinions in different forums or blogs. On the other hand, it is also very difficult for the companies to track this amount of evaluations about their products or services. Therefore, it is necessary to develop new methods that can improve the access to this kind of information.

The automatic processing of documents to detect opinion expressed therein, as a unitary body of research, has been denominated opinion mining. Most work on this area has been carried out on highly subjective text types such as articles in blogs or product reviews. Authors of such type of documents mostly express their opinions quite freely. In general, an opinion is a message expressing a belief about something, the expression of a belief that is held with confidence but not substantiated by positive knowledge or proof.

Opinion mining is a recent research area in the field of the Text Mining (TM) that has been designated by different terms like subjectivity analysis, sentiment analysis or sentiment orientation. There are lots of definitions for each one. Pang and Lee (2008) captured different definitions about these terms based on applications done in this field. For example, Subjectivity Analysis is defined as the recognition of opinion-oriented language in order to distinguish it from objective language. Sentiment Analysis classifies reviews according to their polarity (positive or negative). Henceforth, all these terms refer to the same field of study.

Some tasks in opinion mining try to classify the detected opinion using different scales. In a number of cases, the purpose is to identify opinions in a text and classify them into positive, negative or neutral classes. In other occasions, the goal is to assign different rates, such as “very bad”, “bad”, “satisfactory”, “good”, “very good”, or “excellent”. The sentiments can be ranked into a range of one to five stars. Other systems use the “thumb up” or “thumb down” notation.

Although sentiment analysis may seem an easier task than text categorization, opinion mining includes several challenges which make researchers focus on this stimulating topic. These challenges can be found in review texts, since it is not a quality text such as the content found in newspapers or scientific journals. This text can contain many orthographic mistakes, abbreviations, colloquial expressions, idiomatic expressions or ironic sentences. Another important issue is the time influence. The opinions may change over time due to product improvement (Balog, Mishne, & de Rijke, 2006). In addition, sentiment analysis systems are highly domain dependant. The extraction of features for a corpus about movie reviews is different from one about electronic products. The results can vary significantly from a domain to another. All of these factors make the opinion mining a very interesting and challenging task.

Different approaches have been applied in the field of sentiment analysis but mainly we can distinguish two main methodologies used in opinion mining. On the one hand, there is a lot of work based on the symbolic approach, which applies manually crafted rules and lexicons. The document in this approach is represented as a collection of words. Then, the sentiment of each word can be determined by different methods, for example, using a web search (Hatzivassiloglou & Wiebe, 2000) or consulting a dictionary like WordNet4 (Kamps, Marx, Mokken, & Rijke, 2004). On the other hand, machine learning techniques are very extended in order to attack the classification of reviews according to their orientation (positive or negative). In this approach the document is represented by different features for classification task. Then, a machine learning algorithm is applied. These features may include the use of n-grams or defined grammatical roles like, for instance, adjectives. Machine learning algorithms commonly used are Support Vector Machines, Maximum Entropy or Naïve Bayes. Of course, there are several researches that combine both approaches (symbolic and machine learning).

In this work, we have applied a supervised machine learning method in order to classify reviews. Specifically, we have used Support Vector Machines (SVM) on three datasets with different sizes and domains. One is the Pang and Lee (2004) one about movie reviews; the second corpus is the one prepared by Taboada and Grieve (2004) about several topics like computers, hotels or music; and finally, we have generated one last corpus by crawling opinions about digital cameras from the Amazon website.

We chase several goals. First, we compare the results obtained with these corpora in order to characterize our own corpus feasibility. Secondly, we test the behavior of the system when we use n-grams. Finally, we check our model applied over several corpora with different sizes, domains and number of positive and negatives samples.

The paper is organized as follows. Next section comments some related work and approaches in sentiment analysis. The datasets used in our experiments are described in Section 3. Section 4 presents the method applied and the experiments carried out. Results obtained are discussed in Section 5. Finally, in Section 6, the main conclusions and proposals for further work are expounded.

Section snippets

Related work

In the recent years, relevant research has been developed in the area of opinion mining. Hatzivassiloglou and McKeown (1997) used adjectives as good clues to determinate the text orientation. They also studied phrases where adjectives are connected with conjunction words like “and” or “but”. Then, they applied log-linear regression to check whether the two adjectives had the same polarity. They performed clustering in order to separate the adjectives in two classes and they assumed the class

Corpora description

One of our goals in this paper is to apply SVM on several datasets with different sizes and domains. For this reason we have used three different corpora: the corpus used by Pang and Lee (2004), the corpus prepared by Taboada and Grieve (2004) and a new corpus that we have generated by crawling from Amazon.com. A detailed description of the three corpora is given below.

Support Vector Machines

In this work, Support Vector Machines have been applied in order to classify a set of opinions as positives or negatives. SVM is a product of applied complexity theory developed by Vapnik (1995). Some years ago, Joachims (1998) proposed SVM for text categorization tasks, to profit from its robustness in high dimensional spaces. The name of the algorithm taken from the idea behind it: find those samples (support vectors) that delimit the widest frontier between positive and negative samples in

Results and discussion

In order to test our system, we have applied 3-fold and 10-fold cross validation for each corpus. Analyzing the results according to the n-grams scheme, we can notice that, in general, the trigram model slightly overcomes unigram and bigram models. For example, in 10-fold cross validation, only for the SINAI corpus with TFIDF and BO and the Pang corpus with TO the results are vaguely lower. However, the differences are insignificant and the results obtained for the different corpora and

Conclusions

The main goal of this paper is to compare different corpora available for scientific research in opinion mining. In addition, we have introduced a new corpus that includes reviews about digital cameras. This corpus constitutes a valuable resource to test opinion mining systems. We have also applied a machine learning algorithm (SVM) with different features in order to test how the sentiment classification is affected. We have used different weighting schemes (TFIDF, BO, TO) and several n-grams

Acknowledgments

This paper has been partially supported by a Grant from the Spanish Government, project TEXT-COOL 2.0 (TIN2009-13391-C04-02), a Grant from the Andalusian Government, project GeOasis (P08-TIC-41999), and a Grant from the Fondo Europeo de Desarrollo Regional (FEDER). Also another part of this Project was funded by Agencia Española de Cooperación Internacional para el Desarrollo MAEC-AECID.

References (22)

  • R. Prabowo et al.

    Sentiment analysis: A combined approach

    Journal of Informetrics

    (2009)
  • K. Balog et al.

    Why are they excited?: Identifying and explaining spikes in blog mood levels

  • Chang, C. C., & Lin, C. J. (2001). LIBSVM: A library for support vector machines. Software available at...
  • Ding, X., & Liu, B. (2007). The utility of linguistic rules in opinion mining. In SIGIR ’07: Proceedings of the 30th...
  • Esuli, A., & Sebastiani, F. (2005). Determining the semantic orientation of terms through gloss classification. In CIKM...
  • Esuli, A., & Sebastiani, F. (2006). Sentiwordnet: A publicly available lexical resource for opinion mining. In...
  • Hatzivassiloglou, V., McKeown, K. R. (1997). Predicting the semantic orientation of adjectives. In Proceedings of the...
  • Hatzivassiloglou, V., & Wiebe, J. (2000). Effects of adjective orientation and gradability on sentence subjectivity. In...
  • Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In: KDD ’04: Proceedings of the 10th ACM SIGKDD...
  • T. Joachims

    Text categorization with support vector machines: Learning with many relevant features

  • Kamps, J., Marx, M., Mokken, R.J., & Rijke, M.D. (2004). Using wordnet to measure semantic orientation of adjectives....
  • Cited by (162)

    View all citing articles on Scopus
    View full text