Microblogs data management: a survey

Abstract

Microblogs data is the microlength user-generated data that is posted on the web, e.g., tweets, online reviews, comments on news and social media. It has gained considerable attention in recent years due to its widespread popularity, rich content, and value in several societal applications. Nowadays, microblogs applications span a wide spectrum of interests including targeted advertising, market reports, news delivery, political campaigns, rescue services, and public health. Consequently, major research efforts have been spent to manage, analyze, and visualize microblogs to support different applications. This paper gives a comprehensive review of major research and system work in microblogs data management. The paper reviews core components that enable large-scale querying and indexing for microblogs data. A dedicated part gives particular focus for discussing system-level issues and on-going effort on supporting microblogs through the rising wave of big data systems. In addition, we review the major research topics that exploit these core data management components to provide innovative and effective analysis and visualization for microblogs, such as event detection, recommendations, automatic geotagging, and user queries. Throughout the different parts, we highlight the challenges, innovations, and future opportunities in microblogs data research.

Appendices

Appendix

Orthogonal research directions

This appendix gives an overview about sentiment and semantic analysis in microblogs as an example of an orthogonal research direction, from the natural language processing literature, that does not exploit much of the data management infrastructures. The appendix highlights the differences of new techniques on microlength data with the corresponding techniques on traditional long data. For detailed surveys about these topics, the reader can refer to [80, 117].

1.1 Sentiment analysis

Sentiment analysis automatically discovers the polarity of feelings expressed in a chunk of text, e.g., a citizen posts positive or negative opinions about certain election candidate. Traditional sentiment analysis techniques make use of the microblogs brevity to enhance the classification accuracy of user sentiment. As reported in [46], using a traditional sentiment classification technique on microblogs boosts the accuracy up to 10% higher for binary sentiment. This boost is an absolute advantage of the content brevity that makes it less confusing and more decisive to catch positive and negative feelings in user-generated content. However, microblogs brevity introduces both challenges and differences compared with traditional data. For example, feature extraction is more challenging due to lots of abbreviations and noise, e.g., extracting meaningful keywords is harder. In addition, compared with traditional data where sentiment is analyzed on three different level, document level, sentence level, and entity level, microblogs short content mostly limits the sentiment scope to a single sentence or a single entity that represents the whole microdocument. Moreover, microblogs come with additional advantageous features that were not available in traditional data, such as links, user information, and their interactions with different topics. Thus, the sentiment analysis research on microblogs has addressed a wider variety of challenges compared with traditional sentiment analysis. In this section, we give an overview about this rich literature.

Fig. 15

An overview of microblogs sentiment analysis literature

Figure 15 depicts an overview of the microblogs sentiment analysis literature. The major techniques can be categorized into four main categories, namely, machine learning techniques, lexical techniques, hybrid techniques, and miscellaneous techniques. The machine learning techniques represent the majority of techniques in the literature. It could be further categorized into four sub-categories as depicted in Fig. 15, namely, supervised, classifier ensemble, deep learning, and semi-supervised. The first sub-category of techniques use supervised machine learning, i.e., traditional classifiers [4, 36, 67, 87, 202, 220, 254, 255, 276, 281, 318, 351, 393]. The differences among these techniques are the classifier type, stages, and features used to distinguish sentiment. The major used classifiers are support vector machines (SVM) [4, 5, 31, 35, 39, 83, 87, 120, 131, 160, 164, 172, 180, 248, 262, 275, 306], (multinomial) naive Bayes (MNB and NB) [31, 35, 120, 131, 247, 262], k-nearest neighbor (kNN) [31, 82], MaxEnt [92, 120, 180], random forest (RF) [89, 319], logistic regression [36, 202, 393], and AdaBoost [185]. The used features include different types of language-based features such as unigrams [5, 35, 120, 164, 247, 262], bigrams [35, 120, 262], trigrams [262], n-grams [82, 180, 185, 248], and POS tags [5, 39, 120, 180, 185, 247, 248, 262], microblog-specific features [185, 248] such as retweets [39], hashtags [35, 39, 164], emotions [35, 39, 164, 180], links [35, 39], and other features such as punctuation-based [5, 82, 180, 248], pattern-based [5, 35, 82, 164, 180, 248] and semantic-based [180, 247].

To enhance the classification accuracy, techniques of the second sub-category ensemble multiple classifiers [61, 70, 75, 79, 86, 136, 172, 185, 194, 208, 244, 266, 317, 363]. The set of used features is almost identical to the single classifier techniques, while the used classification algorithms are overlapping but not identical. In specific, SVM [79, 136, 266], NB [70, 136], MNB [79], logistic regression [79, 136, 208], and AdaBoost [185] are still used, while new classifiers are also introduced such as neural models [86, 136, 363] and Bayes network [136]. A third sub-category is deep learning, which is an emerging field in machine learning. In the past few years, deep learning is getting increasing popularity and many learning problems migrated to deep learning frameworks. Deep learning offers a black box of neural networks that are trained with huge amounts of data that offer better accuracy over traditional classifiers. In the case of microblog platforms, huge amount of data is generated daily, which has motivated the use of deep learning techniques for sentiment analysis on microblogs. Existing deep learning techniques is exploited in short textual contexts in two-step fashion [37, 62, 71, 86, 90, 134, 147, 148, 165, 265, 280, 288, 321, 322, 337, 342, 359]. It first learns word embeddings, and then, it applies them to produce representations for the text sentiment.

The main limitation of all supervised techniques, either with single classifier, multiple classifiers, or deep learning models, is the sensitivity to dataset size. For increasing their performance, there is a high reliance on the manually annotated labels which is extremely expensive. To alleviate this problem, distance supervision has been employed where the labels are generated based on the emoticons and hashtags [258, 310]. However, this approach did not perform well. This encouraged a fourth sub-category of semi-supervised techniques to rise. The semi-supervised techniques rely on both a small set of manually annotated data as well as unlabeled data to train the model. They can be further divided into three main types as depicted in Fig. 15: graph-based, wrapper-based, and topic-based techniques. The graph-based techniques [77, 313, 320, 344] use label propagation to label the unlabeled training data based on the similarity metric between two nodes in the graph. Then, a classifier is trained and used as previous techniques. The wrapper-based techniques [44, 45, 214, 215, 380] rely either on self-training [44, 45, 380] or co-training [214, 215]. In both types, the classification process is an iterative process, starting with the initial labeled data, classify the other unlabeled data, and use the high confident ones in the next iteration of the classification till all data is labeled or it hits the maximum number of iterations. The difference between the self-training and the co-training is that in self-training only one classifier is used, whereas in the co-training two classifiers with different feature sets are used to provide two different views for the data. The more confident classification within the two classifiers is chosen to be within the labeled data in the next iteration. The last semi-supervised types are topic-based techniques [11, 123, 139, 166, 241, 301, 355], where topic information is extracted with sentiment analysis simultaneously under the observation that the context of the content affects the sentiment.

The second major category in Fig. 15 is lexical techniques [30, 146, 152, 207, 260, 278, 299, 323, 340], where a predefined list of positive and negative words is employed to classify the sentiment of the new microblog. There are two main sub-categories in lexical techniques, namely dictionary-based and corpus-based. The dictionary-based techniques [30, 146, 152, 207, 299, 340] use dictionaries as lexical resources and approximate lexical matching techniques are used to account for microblogs noise and abbreviations. The corpus-based technique [278] uses statistical or semantic methods to match incoming data with existing lexical resources. The third major category in Fig. 15 is hybrid techniques [107, 115, 175, 177, 182, 189, 376] that combine both machine learning and lexical methods to detect microblogs sentiment. These techniques use lexical terms either to train a machine learning model or to filter data in a first stage that is fed to a classifier for further processing on a second stage.

Other miscellaneous techniques are proposed for microblogs sentiment analysis. ConSent [183] uses concept analysis to determine sentiment based on associated topic. AppSent [184] uses appraisal terms to outperform supervised techniques. SocioSent [150] uses sociological information in the supervised learning process to improve the performance. ChineSentiment [365] proposes a rule-based model for analyzing sentiment features of different linguistic components, and a corresponding methodology for calculating sentiment using emoticon elements as auxiliary affective factors.

Fig. 16

An overview of microblogs semantic analysis literature

1.2 Semantic analysis

Semantic analysis is a popular analysis task that is widely used in microblogs literature for different applications, such as topic modeling [339], knowledge extraction [309], community detection [190, 311], stance detection [390], sentiment analysis [182], event analysis [48, 261], effective microblog retrieval and ranking [379], and user recommendations [106]. This task automates discovering the meanings of a chunk of text by discovering semantic relationships that relate to real-world entities, such as places, persons, and organizations. For example, a text like Trump to campaign for Cindy Hyde-Smith in Mississippi can be related to two persons, Trump and Cindy Hyde-Smith, and one place, Mississippi. This relatedness connects the input text to a predefined set of semantic concepts or categories that are commonly extracted from human-contributed content, such as Wikipedia, or professionally maintained ontologies, such as FOAF and DBpedia ontologies. Such type of analysis used to be performed on long chunks of text, e.g., news articles, blog posts, or web documents. However, in microblogs, the textual content is very short and contains a plenty of abbreviations, informality, and noisy terms. Such brevity hurts the performance of traditional semantic analysis techniques, as shown in [242], that depend on lexical matching and search-based retrieval in, for example, Wikipedia concepts.

To overcome the brevity problem, a general theme of semantic analysis research on microblogs is exploring different ways to enrich the microblogs short textual content to enable accurate semantic relations discovery. Existing techniques in the literature can be categorized into four categories, as depicted in Fig. 16, based on the source of enrichment through: external documents-based techniques, machine learning-based techniques, hashtag-based techniques, and lexical techniques. Techniques of the first category [57, 113, 339, 350] depend on linking the microblog short document to external long documents, e.g., news articles or web documents, which allow traditional semantic analysis techniques to be applied with high precision. ToSem [339] performs semantic enrichment based on explicit web links that are included in the microblog to associate the linked web document. Then, it extracts both named entities and top-k terms from the web document to be appended to the microblog as auxiliary terms. NwSem [57] identifies online news articles that are related to the microblog post in order to extract named entities and include them in the user profile as semantic tags. UsrSem [350] explores semantics of user interactions, specifically retweets and links that are embedded in tweets, and their role in inferring notions such as quality of user relationships, trust, and other attributes of user relationships. This could be applied to re-ranking microblogs based on importance, user interest, quality, etc. DiSem [113] maps microblog posts to Wikipedia articles, then use the Wikipedia ontology for semantic categorization.

The second category is machine learning-based techniques [110, 149, 216, 217, 242, 287, 311, 370] that use either: (1) clustering to group different related microblogs and use their collective content to semantically label the whole cluster, or (2) classification that exploits annotated training data as an external source of information to learn different semantic classes of new microblogs. TrSem [370] introduces a novel transfer learning approach, namely transfer latent semantic learning, that utilizes a large number of tagged documents with rich information from domain-specific sources to discover latent semantics of the abbreviated text. AccSem [149] clusters related microblogs and use the collective content of each cluster to automatically assign semantically meaningful labels. The semantic labels are solicited from external knowledge sources, such as Wikipedia and WordNet, based on informative fragments parsed from microblogs contents. AdSem [242] uses SVM and naive Bayes classifiers to enhance the precision of mapping tweets to Wikipedia-based concepts. For this, it obtains an initial ranked list of candidate concepts through lexical matching, language modeling, and traditional techniques. Then, annotated training data is used to train classifiers that further classify microblogs based on different feature vectors to the correct semantic category, which significantly boosts both precision and recall. NomSem [216] uses SVM classifiers to identify nominal predicates in tweets. Then, a factor graph for each nominal predicate is constructed and joined with graphs of other predicates so their semantic arguments are jointly resolved. ComSem [311] clusters related microblogs to detect user groups within sub-communities. Then, a probabilistic model is employed to measure the semantic, or topical, coherence of the user group and filter out non-coherent groups. GeoSem [314] clusters microblogs based on spatial, temporal, and semantic features, including LDA topics, to evaluate the performance of combining different features in retrieving insights from microblogs data. ST-SRL [217] proposes a semi-supervised self-training approach that utilizes a small training dataset to label unlabeled tweets in an iterative way to increase the training dataset size. Labeled data records with highest confidence from two different labelers are used to enhance the classification accuracy in the following iterations. VecSem [110] has performed a unique study that explores the effect of changing microblog-specific semantic representation features on the performance of semantic prediction. It studies a set of 13 microblog-specific prediction tasks to understand both textual and social aspects of different representations.

The third category is hashtag-based techniques [38, 264, 345]. Hashtags are user-defined tags included in microblog posts, which indicate the discussed topics and enable posts related to the same topics to be searched easily. These hashtags are used in different ways to discover latent semantic content in microblogs. SMOB [264] uses hashtags as seeds to generate potential related links to web documents and ontology entries from both FOAF and DBpedia ontologies. Then, relevant semantic relations to the discovered entities are appended to the microblog. EntSem [38] enriches semantics through retrieving a ranked list of the top-k hashtags that are relevant to a user’s query and segments them into relevant individual words. Then, it retrieves a set of Wikipedia articles that are related to tweet text, hashtags, and segmented hashtags. HGTM [345] introduces a new topic model through using hashtags to determine semantic relatedness to each other through a graph structure. A graph of hashtag relatedness is constructed using probabilistic models; then, related hashtags are grouped in coherent topics.

The fourth category is lexical techniques [9, 10, 48, 81, 106, 179, 261, 273, 309, 392] that improve traditional techniques that are used for long text to be effective for short textual microblog content. InducSem [273] induces semantic entities using lexical pattern-based approach that match microblog text with seed keywords of each semantic category, e.g., food, sports, or vehicles. KnoSem [309] uses lexical resources that include corpus and POS-tagged terms to label tweets with semantic frames for knowledge extraction purposes. EveSem [261] analyzes word co-occurrences to discover relationships among word pairs. Then, such features are used to calculate the pairwise similarity of tweets for event detection purposes. HIVSem [10] uses lexical matching techniques to analysis the presence of an HIV prevention drug on Twitter. PlcSem [179] extracts place semantics through LDA topic modeling from a collection of microblogs to abstract their content through probabilistic models into a set of coherent topics. Then, the extracted place semantics is analyzed for temporal changes, e.g., a sports arena could evolve over time to be a place for concerts and exhibitions. MonSem [48] uses lexical matching to match microblog content with semantic knowledge bases to monitor unexpected events on social media. LikSem [9] uses semantic user attributes to enhance link prediction among social media users. RetSem [392] uses lexical semantic features to enhance microblogs retrieval performance. TriSem [81] uses semantic relevance to filter tweets based on Wikipedia concepts and trigrams. RecSem [106] uses semantic relatedness to recommend users to follow. It links users to Wikipedia through lexical and disambiguation algorithms; then, similar users are recommended.