TWStream: Finding Correlated Data Streams Under Time Warping

Abstract

Consider the problem of monitoring multiple data streams and finding all correlated pairs in real time. Such correlations are of special interest for many applications, e.g., the price of two stocks may demonstrate quite similar rise/fall patterns, which provides the market trader with an opportunity of arbitrage. However, the correlated patterns may occur on any unknown scale, with arbitrary lag or even out of phase, which blinds most traditional methods. In this paper, we propose TWStream, a method that can detect pairs of streams, of which subsequences are correlated with elastic shift and arbitrary lag in the time axis. Specifically, (1) to accommodate varying scale and arbitrary lag, we propose to use the geometric time frame in conjunction with a piecewise smoothing approach; (2) to detect unsynchronized correlation, we extend the cross correlation to support time warping, which is proved much more robust than Euclidian based metrics. Our method has a sound theoretical foundation, and is efficient in terms of both time and space complexity. Experiments on both synthetic and real data are done to show its effectiveness and efficiency.