Article

Flow algorithms for two pipelined filter ordering problems

Authors:

Amol Deshpande,

Lisa Hellerstein,

Ning WuAuthors Info & Claims

PODS '06: Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

Pages 193 - 202

https://doi.org/10.1145/1142351.1142379

Published: 26 June 2006 Publication History

Abstract

Pipelined filter ordering is a central problem in database query optimization, and has received renewed attention recently in the context of environments such as the web, continuous high-speed data streams and sensor networks. We present algorithms for two natural extensions of the classical pipelined filter ordering problem: (1) a distributional type problem where the filters run in parallel and the goal is to maximize throughput, and (2) an adversarial type problem where the goal is to minimize the expected value of multiplicative regret. We show that both problems can be solved using similar flow algorithms, which find an optimal ordering scheme in time O(n²), where n is the number of filters. Our algorithm for (1) improves on an earlier O(n³ log n) algorithm of Kodialam.

References

[1]

R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pages 261--272. ACM Press, 2000.]]

Digital Library

[2]

B. Awerbuch and F. T. Leighton. A simple local-control approximation algorithm for multicommodity flow. In Proceedings of the 34th IEEE Symposium on Foundations of Computer Science, pages 459--468. IEEE Computer Society, 1993.]]

Digital Library

[3]

S. Babu, R. Motwani, K. Munagala, I. Nishizawa, and J. Widom. Adaptive ordering of pipelined stream filters. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data, pages 407--418. ACM Press, 2004.]]

Digital Library

[4]

A. Bar-Noy, M. Bellare, M.M. Halldórsson, H. Shachnai, and T. Tamir. On chromatic sums and distributed resource allocation. Inform. and Comput., 140(2):183--202, 1998.]]

Digital Library

[5]

S. Chaudhuri, U. Dayal, and T. W. Yan. Join queries with external text sources: Execution and optimization techniques. In Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, pages 410--422. ACM Press, 1995.]]

Digital Library

[6]

E. Coffman and I. Mitrani. A characterization of waiting time performance realizable by single server queues. Operations Research, 20:810--821, 1980.]]

[7]

E. Cohen, A. Fiat, and H. Kaplan. Efficient sequences of trials. In Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms, pages 737--746. ACM Press, 2003.]]

Digital Library

[8]

A. Deshpande, C. Guestrin, S. Madden, and W. Hong. Exploiting correlated attributes in acquisitional query processing. In Proceedings of the 21st International Conference on Data Engineering (ICDE), pages 143--154. IEEE Computer Society, 2005.]]

Digital Library

[9]

O. Etzioni, S. Hanks, T. Jiang, R. M. Karp, O. Madani, and O. Waarts. Efficient information gathering on the internet. In Proceedings of the 37th IEEE Symposium on Foundations of Computer Science, pages 234--243. IEEE Computer Society, 1996.]]

Digital Library

[10]

U. Feige, L. Lovász, and P. Tetali. Approximating min-sum set cover. Algorithmica, 40 (4):219--234, 2004.]]

Digital Library

[11]

L. Fleischer and K. Wayne. Fast and simple approximation schemes for generalized flow. Mathematical Programming, 91(2):215--238, 2002.]]

[12]

M. Garey. Optimal task scheduling with precedence constraints. Discrete Mathematics, 4:37--56, 1973.]]

Digital Library

[13]

E. Gelenbe and I. Mitrani. Analysis and synthesis of computer systems. Academic Press, 1980.]]

[14]

R. Goldman and J. Widom. Wsq/dsq: A practical approach for combined querying of databases and the web. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, pages 285--296. ACM, 2000.]]

Digital Library

[15]

H. Kaplan, E. Kushilevitz, and Y. Mansour. Learning with attribute costs. In The Thirty-Seventh Annual ACM Symposium on Theory of Computing (STOC), pages 356--365. ACM Press, 2005.]]

Digital Library

[16]

S. Karlin. Mathematical Methods and Theory in Games, Programming, and Economics. Dover, 2003.]]

[17]

M.S. Kodialam. The throughput of sequential testing. In Integer Programming and Combinatorial Optimization (IPCO) LNCS 2081, pages 280--292. Springer-Verlag Berlin Heidelberg, 2001.]]

Digital Library

[18]

Ravi Krishnamurthy, Haran Boral, and Carlo Zaniolo. Optimization of nonrecursive queries. In Wesley W. Chu, Georges Gardarin, Setsuo Ohsuga, and Yahiko Kambayashi, editors, VLDB'86 Twelfth International Conference on Very Large Data Bases, August 25--28, 1986, Kyoto, Japan, Proceedings, pages 128--137. Morgan Kaufmann, 1986.]]

Digital Library

[19]

K. Munagala, S. Babu, R. Motwani, and J. Widom. The pipelined set cover problem. In Tenth Intl. Conf. on Database Theory, pages 83--98, 2005.]]

Digital Library

[20]

M.A. Shayman and E. Fernandez-Gaucherand. Risk-sensitive decision-theoretic diagnosis. IEEE Transactions on Automatic Control, 46:1166--1171, 2001.]]

[21]

H. Simon and J. Kadane. Optimal problem-solving search: All-or-none solutions. Artificial Intelligence, 6:235--247, 1975.]]

Cited By

Shaukat KMasood NKhushi M(2019)A Novel Approach to Data Extraction on Hyperlinked WebpagesApplied Sciences10.3390/app92351029:23(5102)Online publication date: 25-Nov-2019
https://doi.org/10.3390/app9235102
Ducasse RTekin Cvan der Schaar M(2018)Finding It Now: Networked Classifiers in Real-Time Stream Mining SystemsHandbook of Signal Processing Systems10.1007/978-3-319-91734-4_3(87-131)Online publication date: 14-Oct-2018
https://doi.org/10.1007/978-3-319-91734-4_3
Hellerstein LÖzkan ÖSellie L(2017)Max-Throughput for (Conservative) k-of-n TestingAlgorithmica10.1007/s00453-015-0089-477:2(595-618)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s00453-015-0089-4
Show More Cited By

Index Terms

Flow algorithms for two pipelined filter ordering problems

Recommendations

Algorithms for distributional and adversarial pipelined filter ordering problems

Pipelined filter ordering is a central problem in database query optimization. The problem is to determine the optimal order in which to apply a given set of commutative filters (predicates) to a set of elements (the tuples of a relation), so as to find,...
Parallel pipelined filter ordering with precedence constraints

In the parallel pipelined filter ordering problem, we are given a set of n filters that run in parallel. The filters need to be applied to a stream of elements, to determine which elements pass all filters. Each filter has a rate limit r_i on the number ...
A generic flow algorithm for shared filter ordering problems
PODS '08: Proceedings of the twenty-seventh ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

We consider a fundamental flow maximization problem that arises during the evaluation of multiple overlapping queries defined on a data stream, in a heterogenous parallel environment. Each query is a conjunction of boolean filters, and each filter could ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PODS '06: Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems

June 2006

382 pages

ISBN:1595933182

DOI:10.1145/1142351

General Chair:
Georg Gottlob
Vienna University of Technology
,
Program Chair:
Jan Van den Bussche
Hasselt University

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGMOD/PODS06

Sponsor:

SIGMOD/PODS06: International Conference on Management of Data and Symposium on Principles Database and Systems

June 26 - 28, 2006

IL, Chicago, USA

Acceptance Rates

PODS '06 Paper Acceptance Rate 35 of 185 submissions, 19%;

Overall Acceptance Rate 642 of 2,707 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
306
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shaukat KMasood NKhushi M(2019)A Novel Approach to Data Extraction on Hyperlinked WebpagesApplied Sciences10.3390/app92351029:23(5102)Online publication date: 25-Nov-2019
https://doi.org/10.3390/app9235102
Ducasse RTekin Cvan der Schaar M(2018)Finding It Now: Networked Classifiers in Real-Time Stream Mining SystemsHandbook of Signal Processing Systems10.1007/978-3-319-91734-4_3(87-131)Online publication date: 14-Oct-2018
https://doi.org/10.1007/978-3-319-91734-4_3
Hellerstein LÖzkan ÖSellie L(2017)Max-Throughput for (Conservative) k-of-n TestingAlgorithmica10.1007/s00453-015-0089-477:2(595-618)Online publication date: 1-Feb-2017
https://dl.acm.org/doi/10.1007/s00453-015-0089-4
Xie QZhang XLi ZZhou X(2016)Optimizing Cost of Continuous Overlapping Queries over Data Streams by Filter AdaptionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2016.251654128:5(1258-1271)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1109/TKDE.2016.2516541
Deshpande AHellerstein L(2012)Parallel pipelined filter ordering with precedence constraintsACM Transactions on Algorithms10.1145/2344422.23444318:4(1-38)Online publication date: 4-Oct-2012
https://dl.acm.org/doi/10.1145/2344422.2344431
Das Sarma AFang LGupta NHalevy ALee HWu FXin RYu CCandan KChen YSnodgrass RGravano LFuxman A(2012)Finding related tablesProceedings of the 2012 ACM SIGMOD International Conference on Management of Data10.1145/2213836.2213962(817-828)Online publication date: 20-May-2012
https://dl.acm.org/doi/10.1145/2213836.2213962
Hellerstein LÖzkan ÖSellie L(2011)Max-throughput for (conservative) k-of-n testingProceedings of the 22nd international conference on Algorithms and Computation10.1007/978-3-642-25591-5_72(703-713)Online publication date: 5-Dec-2011
https://dl.acm.org/doi/10.1007/978-3-642-25591-5_72
Foo Bvan der Schaar M(2010)A distributed approach for optimizing cascaded classifier topologies in real-time stream mining systemsIEEE Transactions on Image Processing10.1109/TIP.2010.205186619:11(3035-3048)Online publication date: 1-Nov-2010
https://dl.acm.org/doi/10.1109/TIP.2010.2051866
Li JZhang PTan J(2010)An Adaptive Ordering Framework for Filtering Multimedia StreamsProceedings of the 2010 Second International Conference on MultiMedia and Information Technology - Volume 0210.1109/MMIT.2010.91(196-199)Online publication date: 24-Apr-2010
https://dl.acm.org/doi/10.1109/MMIT.2010.91
Ducasse RTuraga Dvan der Schaar M(2010)Adaptive Topologic Optimization for Large-Scale Stream MiningIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2009.20391804:3(620-636)Online publication date: Jun-2010
https://doi.org/10.1109/JSTSP.2009.2039180
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten