Multi-modality weakly labeled sentiment learning based on Explicit Emotion Signal for Chinese microblog

Abstract

Understanding the sentiments of users from cross media contents which contain texts and images is an important task for many social network applications. However, due to the semantic gap between cross media features and sentiments, machine learning methods need a lot of human labeled samples. Furthermore, for each kind of media content, it is necessary to constantly add a lot of new human labeled samples because of new expressions of sentiments. Fortunately, there are some emotion signals, like emoticons, which denote users’ emotions in cross media contents. In order to use these weakly labels to build a unified multi-modality sentiment learning framework, we propose an Explicit Emotion Signal (EES) based multi-modality sentiment learning approach which uses huge number of weakly labeled samples in sentiment learning. There are three advantages in our approach. Firstly, only a few human labeled samples are needed to reach the same performance which can be obtained by the traditional machine learning based sentiment prediction approaches. Secondly, this approach is flexible and can easily combine text and vision based sentiment learning through deep neural networks. Thirdly, because a lot of weakly labeled samples can be used in EES, trained model is more robust in different domain transfer. In this paper, firstly, we investigate the correlation between sentiments and emoticons and choose emoticons as the Explicit Emotion Signals in our approach; secondly, we build a two stages multi-modality sentiment learning framework based on Explicit Emotion Signals. Our experiment results show that our approach not only achieves the best performance but also only needs 3% and 43% training samples to obtain the same performance of Visual Geometry Group (VGG) model and Long Short-Term Memory (LSTM) model in images and texts, respectively.

Introduction

With the development of web social network applications, many people express their opinions by using cross media contents which contains texts, images and videos in Internet. In this situation, understanding the sentiment of users’ opinions is becoming more and more important. Generally speaking, sentiment analysis aims to determine the attitude of a speaker or a writer with respect to some topics or the overall contextual polarity of a document, such as positive, negative and neutral attitude. Recently, how to fully understand the sentiment from cross media yields an interesting research topic, cross media sentiment learning and analysis. One of the important issue for cross media sentiment learning in social network applications is that huge number of labeled samples are needed. There are three reasons which yield this problem. Firstly, unlike the common classification problems, sentiment is a kind of high level semantic information which is hard to achieve high performance directly from low level features, such as pixels and colors in images, words and phrases in texts. Therefore, many labeled samples are needed to learn the semantic space from raw data. Secondly, the problem becomes worse when we face the cross media content in social network applications. Because of the complementation of cross media content, many posts contain different sentiments in different media contents. To automatically learn sentiments from cross media contents, annotations for each kind of media content are needed. Thirdly, the way of expressing sentiment is changing over time which means that we need to constantly add a lot of new labeled samples to learn the change of sentiment expression.

Our goal is understanding the sentiments of users’ opinions. However, it is very difficult to obtain a large number of human labeled sentiment data. Fortunately, users often use emoticons in Chinese microblog, and those emoticons show users’ emotions. Therefore, we can collect a lot of weakly labeled samples which contain emotions. Here, weakly labeled sample means that the data only contains some emotion related symbols, such as emoticons, but without any sentiment labels which are the learning target. We think that an effective way of using weakly labeled samples will greatly alleviate the multi-modality sentiment learning problem. Although Hu et al. [1] used weakly labeled samples in the text based sentiment analysis, this approach is not suitable for cross media sentiment learning. Firstly, it used the orthogonal nonnegative matrix tri-factorization model which cannot handle the huge number of weakly labeled data to learn the latent sentiment space. Secondly, image content cannot be processed in the word content matrix. Therefore, we propose an Explicit Emotion Signal (EES) based approach which learns the EES from weakly labeled cross media samples from multi-modality view. Compared with state-of-the-art approaches, huge number of weakly labeled cross media contents can be used in our multi-modality sentiment learning framework and more robust features can be learned, and sentiment predication performance can be improved by using a few labeled samples.

In particular, we make the following contributions:

• We propose an Explicit Emotion Signal based cross media sentiment learning approach to use huge number of weakly labeled samples in social network applications, and it learns robust features which are not sensitive in different domain transfer.

• We implement a two stages multi-modality sentiment learning framework for both images and texts. This framework is flexible and can be transformed into a single-modality sentiment learning framework. The single-modality of the framework outperforms two state-of-the-art approaches, Long Short-Term Memory (LSTM) [2] and Visual Sentiment Ontology (VSO)[3], in text and image sentiment predication, respectively.

The remainder of this paper is organized as follows. Section 2 briefly describes the background and related work in sentiment analysis. Section 3 presents the concept of Explicit Emotion Signal and shows the statistical relationship between EES and sentiment. The algorithm and framework of EES based approach is detailed in Section 4. Experimental results on Sina Weibo datasets are given in Section 5. Section 6 draws conclusions and gives directions for future work.