Semantic multi-grain mixture topic model for text analysis

Abstract

Granular topic extraction and modeling are fundament tasks in text analysis. Hierarchical topic clustering algorithms and hierarchical topic models are usually employed for these purposes. However, it is difficult to make a clear distinguish between each pair of hierarchical topics from the semantic granularity point of view. STG (semantic topic granularity) is proposed to indicate the details degree of topic description, and aim at providing discrimination for topics from semantic aspect. A new model, mgMTM (multi-grain mixture topic model) based on STG is then proposed to model grain topics. DCT (discrete cosine transform) is employed to provide a mechanism for computing STG, extracting grain topics and learning mgMTM. Experiments on real world datasets show that the proposed model has lower perplexity score than that of LDA model and thus has better generalization performance in describing text. Experiments also show that the description of the extracted grain topics can be well explained with respect to a dataset including topics about recent global financial crisis.

Introduction

Enormous text has been produced every day on the Internet. Hence, the computer automatic analysis of text is strongly required to provide recognizable topics for kinds of topic-based analysis tasks, such as opinions extraction, topic propagation and topic evolution (Mei et al., 2007, Zeng et al., 2009). For these purposes, many topic extraction algorithms and topic models have been devised to help find out topics that hide in text corpus (Zeng and Zhang, 2007, Zeng and Zhang, 2009, Zhong and Ghosh, 2005). However, topics are inherently hierarchical or granular, as a result, finding the hierarchical topic structures becomes an important task in topic extraction.

A typical method is to utilize hierarchical clustering algorithms (Gil-García1, Badía-Contelles, & Pons-Porrata, 2006). By considering each document as a point in high dimension space composed of words, a merge operation on two most similar points then is performed. By iteration the merge step, a hierarchical structure can finally be generated (Duan et al., 2005; Gil-García1 et al., 2006). Several improvements on the basic algorithms are done to utilize similarity measurement (Wang & Imad, 2007), merge and split strategy (Rodrigues, Gama, & Pedroso, 2008), etc. Considering a document as a probability distribution is another choice. In this case, document similarity is usually measured by probability distance, such as KL-divergence (Zhong, 2003). Zhong utilized model based k-means clustering algorithm by considering a document as multinomial model, and a general document hierarchical clustering algorithm was proposed (Zhong, 2003, Zhong and Ghosh, 2005). Several efficient algorithms introduce hashing technique to produce a hierarchical decomposition of topic space based on the asymmetric relationships between terms (Gollapudi & Panigraphy, 2006).

As another approach, hierarchical topic models like hLDA (Blei, Griths, & Jordan, 2004) and Pachinko allocation (Li & McCallum, 2006), models topics in a hierarchical structure. These kinds of topic models describe semantic interactions between topics that are typically at the document level. In hLDA model, each document is assigned to a path through a hierarchical topic tree, and each word in a given document is assigned to a topic at one of the levels of that path. MGLDA is proposed to model opinion review at document or sentence levels based on LDA. However, it produces topics of global and local granularity (Titov & McDonald, 2008).

Obviously, the two approaches provide natural ways to generate hierarchical topic structure. However, as far as the hierarchical topics clustering algorithms are concerned, the clusters are generated in the sense of topics similarity in measurement space, which consists of only feature words selected from original documents. Note that some valuable words in expressing granularity are usually omitted in this process. As a result, general semantic topics need to be induced by users. Hierarchical topic models describe the word distribution between each neighbor hierarchical levels, and the granularity, that is, the hierarchical level, is lack of comparable indication. Hence, from the semantic granularity point of view, it is difficult to make a clear distinguish between each pair of hierarchical topics via the two approaches, which prevents the improvement on the accuracy of topic modeling.

We first explore the topic granularity which is closely related to semantic characteristic, and STG (semantic topic granularity) is proposed to provide a semantic discrimination indicator for topics. By considering the topics in corpus as a mixture of STG topics, a new model, mgMTM (multi-grain mixture topic model) is proposed to model multi-grain topics. DCT (discrete cosine transform) is employed to provide a mechanism for computing STG and extracting grain topics in learning mgMTM.

The main contributions of the paper are as follows:

• Semantic topic granularity is proposed to provide a mechanism for describing the semantic detail degree of selected topics. Being inspired by the granularity expression in image compression, DCT is employed to figure out the value of STG.

• A mixture STG topic model is proposed to describe the multi-grain topics. A quantization method to determine feature words for a grain topic is proposed, based on which, STG topic extraction algorithm is proposed to construct the mixture model.

• Extensive experiments are performed on real world datasets. The validity of the proposed model and algorithms is verified by the means of perplexity and the explanation on the generated topic description.

The organization of the paper is as follows: In the next section, we present some basic term definitions and induce the definition of STG based on DCT. In Section 3, we describe multi-grain mixture topic model, and the model learning algorithm. In the fourth section, we describe the experiments on real world datasets and analyze the results, including perplexity score and topic description. In Section 5, conclusion and future work are pointed out.