Estimating Dominance Norms of Multiple Data Streams

Abstract

There is much focus in the algorithms and database communities on designing tools to manage and mine data streams. Typically, data streams consist of multiple signals. Formally, a stream of multiple signals is (i,a _i,j ) where i’s correspond to the domain, j’s index the different signals and a _i,j  ≥ 0 give the value of the jth signal at point i. We study the problem of finding norms that are cumulative of the multiple signals in the data stream.

For example, consider the max-dominance norm, defined as ∑  _i max _j {a _i,j }. It may be thought as estimating the norm of the “upper envelope” of the multiple signals, or alternatively, as estimating the norm of the “marginal” distribution of tabular data streams. It is used in applications to estimate the “worst case influence” of multiple processes, for example in IP traffic analysis, electrical grid monitoring and financial domain. In addition, it is a natural measure, generalizing the union of data streams or counting distinct elements in data streams.

We present the first known data stream algorithms for estimating max-dominance of multiple signals. In particular, we use workspace and time-per-item that are both sublinear (in fact, poly-logarithmic) in the input size. In contrast other notions of dominance on streams a, b — min-dominance (∑  _i min _j {a _i,j }), count-dominance (|{i|a _i  > b _i }|) or relative-dominance (∑  _i a _i /max{1,b _i }) — are all impossible to estimate accurately with sublinear space.