Incompleteness in Data Mining

Abstract

Database technology, as well as the bulk of data mining technology, is founded upon logic, with absolute notions of truth and falsehood, at least with respect to the data set. Patterns are discovered exhaustively, with carefully engineered algorithms devised to determine all patterns in a data set that belong to a certain class. For large data sets, many such data mining techniques are extremely expensive, leading to considerable research towards solving these problems more cheaply.

We argue that the central goal of data mining is to find SOME interesting patterns, and not necessarily ALL of them. As such, techniques that can find most of the answers cheaply are clearly more valuable than computationally much more expensive techniques that can guarantee completeness. In fact, it is probably the case that patterns that can be found cheaply are indeed the most important ones.

Furthermore, knowledge discovery can be the most effective with the human analyst heavily involved in the endeavor. To engage a human analyst, it is important that data mining techniques be interactive, hopefully delivering (close to) real time responses and feedback. Clearly then, extreme accuracy and completeness (i.e., finding all patterns satisfying some specified criteria) would almost always be a luxury. Instead, incompleteness (i.e., finding only some patterns) and approximation would be essential.

We exemplify this discussion through the notion of fascicles. Often many records in a database share similar values for several attributes. If one is able to identify and group together records that share similar values for some - even if not all - attributes, one can both obtain a more parsimonious representation of the data, and gain useful insight into the data from a mining perspective. Such groupings are called fascicles. We explore the relationship of fascicle-finding to association rule mining, and experimentally demonstrate the benefit of incomplete but inexpensive algorithms. We also present analytical results demonstrating both the limits and the benefits of such incomplete algorithms.