Mining consumer product data via latent semantic indexing

Abstract

One important focus of data mining research is in the development of algorithms for extracting valuable information from large databases in order to facilitate business decisions. This study explores a new technique for data mining – latent semantic indexing (LSI). LSI is an efficient information retrieval method for textual documents. By determining the singular value decomposition (SVD) of a large sparse term-by-document matrix, LSI constructs an approximate vector space model which represents important associative relationships between terms and documents that are not evident in individual documents. This paper explores the applicability of the LSI model to numerical databases, namely consumer product data. By properly choosing attributes of data records as terms or documents, a term-by-document frequency matrix is built from which a distribution-based indexing scheme is employed to construct a correlated distribution matrix (CDM). An LSI-like vector space model is then used to detect useful or hidden patterns in the numerical data. The extracted information can then be validated using statistical hypotheses testing or resampling. LSI is an automatic yet intelligent indexing method. Its application to numerical data introduces a promising way to discover knowledge in important commercial application areas such as retail and consumer banking.

Introduction

Large amounts of data have been collected in daily operations of organizations due to inexpensive storage and high computing power, but many companies have been unable to extract useful information from the data and utilize the information to benefit their business. Data mining involves the application of algorithms for extracting valid and useful information from large databases in order to make critical business decisions. The fact that data is being accumulated at a faster rate than it can be analyzed creates a significant demand for efficient data mining systems [7], [11]. Techniques used for data mining include decision trees and rule induction [22], association rules [1], nonlinear regression and classification [5], [13], genetic algorithms [19], [23], and neural networks [6], [14]. This study explores a new technique, latent semantic indexing (LSI), for data mining.

LSI is an efficient information retrieval technique which has been commonly used for textual documents [4], [8]. Traditional lexical-matching methods try to match words of queries with words of documents, which may fail to retrieve related documents or may return unrelated documents to users. This kind of failure to retrieve relevant documents or the retrieval of irrelevant documents is called the word-matching problem. LSI addresses the word-matching problem through the use of statistically derived conceptual indices instead of individual words [4]. Using the singular value decomposition (SVD) [15] of a large sparse term-by-document matrix, LSI constructs a conceptual vector space in which each term or document is represented as a vector in the space. The positioning of term and document vectors within the vector space reveals the underlying semantic structure of association between terms and documents in the data.

This paper explores the applicability of the LSI vector-space model to numerical databases. By properly choosing attributes of data records as terms or documents, a term-by-document frequency matrix is built. A distribution-based indexing scheme is then employed to construct a correlated distribution matrix (CDM) which reflects relationships between attributes of data records. Hence, the LSI-like vector space model is generated so that the encoding of attributes in the space can be analyzed for the detection of useful or hidden patterns. The extracted information can then be validated using statistical hypotheses testing or resampling.

Applications for data mining extract information from the data to make important business decisions, predict business trends, and develop new products. A common application for data mining is to analyze customer purchases (see Fig. 1) to discover patterns among existing customer preferences and then use those patterns for forecasting sales and optimizing marketing strategies. In this work, the LSI model is presented as an automated yet scalable (i.e., practical for large data collections) approach to extract underlying patterns from consumer product data.

The remaining sections of this paper outline the development and application of LSI to numerical databases. Section 2 is a brief overview of the LSI vector space model. Section 3 illustrates how an LSI-like model for numerical databases can be designed and implemented for mining a consumer product database. Finally, a summary and discussion of future work are provided in Section 4.