Using information retrieval for sentiment polarity prediction

Highlights

• We propose a method for polarity prediction of Tweets.

• The novelty lies on the proposed features.

• Features are derived from the ranking generated by an Information Retrieval System.

• Results comparable to the state-of-the-art can be achieved with only 24 features.

Abstract

Social networks such as Twitter are used by millions of people who express their opinions on a variety of topics. Consequently, these media are constantly being examined by sentiment analysis systems which aim at classifying the posts as positive or negative. Given the variety of topics discussed and the short length of the posts, the standard approach of using the words as features for machine learning algorithms results in sparse vectors. In this work, we propose using features derived from the ranking generated by an Information Retrieval System in response to a query consisting of the post that needs to be classified. Our system can be fully automatic, has only 24 features, and does not depend on expensive resources. Experiments on real datasets have shown that a classifier that relies solely on these features outperforms established baselines and can reach accuracies comparable to the state-of-the-art approaches which are more costly.

Introduction

With over 500 million posts a day¹, Twitter² has consolidated itself as a major forum for expressing personal opinions on a variety of topics. Because of its popularity, this microblogging service has been the target of numerous studies from a broad range of research areas including Psychology, Sociology, Marketing, and Computer Science. For example, in Mostafa (2013), the analysis of tweets is used to determine the sentiment towards sixteen global brands.

Sentiment analysis, also called Opinion Mining, is dedicated to the computational study of opinions and sentiments expressed in text (Pang & Lee, 2008). This topic has been attracting increasing attention from the research community. Out of the different aspects of opinions that can be studied, the polarity of sentiments is the most well investigated. It consists in predicting whether the opinion expressed in the text is positive or negative.

While most of the research focuses on product reviews, recently, a number of studies on Twitter posts (or simply tweets) have emerged. Sentiment Analysis on Twitter can be done at three different levels: (i) entity, (ii) tweet, or (iii) expression. Entity-level analysis deals with discovering the overall opinion about an entity or topic, tweet-level analysis identifies the polarity of individual tweets, and expression level analysis deals with specific phrases within a tweet. Our focus is on the second – tweet-level analysis. The added challenge of analysing tweets (compared to product reviews) is their shorter length – at most 140 characters – which results in very sparse vector representations. In addition, the variety of topics, and the informal vocabulary, characterised by slangs, abbreviations, and misspellings, pose added difficulties to its computational treatment.

Successful approaches for polarity classification on tweets use one or more of the following: resources such as lexicons (which are sometimes manually created) (Fersini, Messina, Pozzi, 2016, Speriosu, Sudan, Upadhyay, Baldridge, 2011, Zhang, Ghosh, Dekhil, Hsu, Liu, 2011), costly preprocessing such as part-of-speech tagging (Fersini, Messina, Pozzi, 2016, Go, Bhayani, Huang, 2009, Hu, Tang, Tang, Liu, 2013, Saif, He, Alani, 2012), numerous features (Fersini, Messina, Pozzi, 2016, Go, Bhayani, Huang, 2009, Saif, He, Alani, 2012, Speriosu, Sudan, Upadhyay, Baldridge, 2011, Zhang, Ghosh, Dekhil, Hsu, Liu, 2011) large amounts of training data (Bakliwal et al., 2012), and elaborated machine learning methods such as classifier ensembles (Coletta, da Silva, Hruschka, Hruschka, 2014, Martìn-Valdivia, Martìnez-Cámara, Perea-Ortega, Ureña López, 2013, da Silva, Hruschka, Hruschka, , 2014). In this work, we propose a method called Sentiment Analysis Based on Information Retrieval (SABIR) which uses none of the above. We show that classification accuracy comparable to the state-of-the-art can be achieved with a single classification algorithm using only 24 features. Unlike existing approaches, we do not use the words of the tweets as features. Our features are derived from the ranking generated by an Information Retrieval System in response to a query q which consists of the tweet that we wish to classify. The ranking has the n most similar tweets for which we already know the class in decreasing order of similarity to the unlabelled tweet q. The rationale is to leverage information of the class of the similar posts to classify q.

We have carried out experiments with four datasets of tweets which have been used in similar studies. Since the training data for the classification system can be generated without manual annotation (Barbosa, Feng, 2010, Go, Bhayani, Huang, 2009), SABIR can be fully automatic. Our results have shown that there is no significant difference between SABIR and the best baseline classifier we implemented using over one thousand features.