Mining patterns at each scale in massive data

Abstract

An important but neglected aspect of data analysis is discovering phenomena at different scale in the same data. Scale plays the role analogous to error. It can be used to focus data exploration on differences that exceed the given scale (error) and to disregard those smaller. We introduce a discovery mechanism that applies to bi-variate patterns, in particular to time series. It combines search for maxima and minima with search for regularities in the form of equations. If it cannot find a regularity for all data, it uses other discovered patterns to divide data into subsets, and explores recursively each subset. Detected patterns are subtracted from data and the search continues in the residuals. Our mechanism does not skip patterns at any scale. Applied at many scales and to many data sets, it seems explosive, but it terminates surprisingly fast because of data reduction and the requirements of pattern stability and significance. We show application of our method on a time series of a half million datapoints. Our example shows that even simple data can reveal many surprising phenomena, and our method leads to fine conclusions about the environment in which they have been gathered.