skip to main content
10.1145/2463676.2465333acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
research-article

Generalized scale independence through incremental precomputation

Published: 22 June 2013 Publication History

Abstract

Developers of rapidly growing applications must be able to anticipate potential scalability problems before they cause performance issues in production environments. A new type of data independence, called scale independence, seeks to address this challenge by guaranteeing a bounded amount of work is required to execute all queries in an application, independent of the size of the underlying data. While optimization strategies have been developed to provide these guarantees for the class of queries that are scale-independent when executed using simple indexes, there are important queries for which such techniques are insufficient.
Executing these more complex queries scale-independently requires precomputation using incrementally-maintained materialized views. However, since this precomputation effectively shifts some of the query processing burden from execution time to insertion time, a scale-independent system must be careful to ensure that storage and maintenance costs do not threaten scalability. In this paper, we describe a scale-independent view selection and maintenance system, which uses novel static analysis techniques that ensure that created views do not themselves become scaling bottlenecks. Finally, we present an empirical analysis that includes all the queries from the TPC-W benchmark and validates our implementation's ability to maintain nearly constant high-quantile query and update latency even as an application scales to hundreds of machines.

References

[1]
P. Agrawal et al. Asynchronous view maintenance for vlsd databases. In SIGMOD, 2009.
[2]
S. Agrawal, S. Chaudhuri, and V. R. Narasayya. Automated selection of materialized views and indexes in sql databases. In VLDB, 2000.
[3]
Y. Ahmad, O. Kennedy, et al. Dbtoaster: higher-order delta processing for dynamic, frequently fresh views. Proc. VLDB Endow., 5(10), 2012.
[4]
M. Armbrust, K. Curtis, T. Kraska, A. Fox, M. J. Franklin, and D. A. Patterson. PIQL: Success-tolerant query processing in the cloud. PVLDB, 5(3), 2011.
[5]
M. Armbrust et al. Scads: Scale-independent storage for social computing applications. In CIDR, 2009.
[6]
J. A. Blakeley, P.-Å. Larson, and F. W. Tompa. Efficiently updating materialized views. In SIGMOD, 1986.
[7]
S. Ceri and J. Widom. Deriving production rules for incremental view maintenance. In VLDB, 1991.
[8]
M. Chaabouni et al. The point-range tree: a data structure for indexing intervals. In Proc. of ACM CSC, 1993.
[9]
L. S. Colby et al. Algorithms for deferred view maintenance. SIGMOD Rec., 25(2), 1996.
[10]
E. Cunha et al. Analyzing the dynamic evolution of hashtags on twitter: a language-based approach. In Workshop on Languages in Social Media, 2011.
[11]
G. DeCandia et al. Dynamo: amazon's highly available key-value store. SIGOPS, 41, 2007.
[12]
J. Gray, A. Bosworth, A. Layman, D. Reichart, and H. Pirahesh. Data cube: A relational aggregation operator generalizing group-by, cross-tab, and sub-totals. 1996.
[13]
A. Gupta, D. Katiyar, and I. S. Mumick. Counting solutions to the view maintenance problem. In Workshop on Deductive Databases, JICSLP, 1992.
[14]
H. Gupta and I. Mumick. Selection of views to materialize in a data warehouse. Knowledge and Data Engineering, IEEE Transactions on, 17(1), 2005.
[15]
J. Kincaid. Zuckerberg: Online sharing is growing at an exponential rate. http://tinyurl.com/cskurl3.
[16]
C. Koch. Incremental query evaluation in a ring of databases. In PODS, 2010.
[17]
Y. Kotidis et al. Dynamat: a dynamic view management system for data warehouses. SIGMOD Rec., 28(2), 1999.
[18]
W. Labio et al. Performance issues in incremental warehouse maintenance. In VLDB, 2000.
[19]
X. Long and T. Suel. Three-level caching for efficient query processing in large web search engines. In WWW, 2005.
[20]
G. Luo et al. Locking protocols for materialized aggregate join views. In VLDB, 2003.
[21]
H. Mistry, P. Roy, S. Sudarshan, and K. Ramamritham. Materialized view selection and maintenance using multi-query optimization. SIGMOD Rec., 30(2), 2001.
[22]
M. E. J. Newman. Power laws, pareto distributions and zipf's law. Contemporary Physics, 46, 2005.
[23]
M. T. Özsu and P. Valduriez. Principles of distributed database systems (2nd ed.). 1999.
[24]
D. Quass and J. Widom. On-line warehouse view maintenance. In SIGMOD, 1997.
[25]
K. Salem et al. How to roll a join: asynchronous incremental view maintenance. SIGMOD Rec., 29(2), 2000.
[26]
B. Trushkowsky et al. The scads director: scaling a distributed storage system under stringent performance requirements. In FAST, 2011.
[27]
P. Valduriez. Join indices. ACM Trans. Database Syst., 12(2), 1987.
[28]
K. Weil. Measuring tweets. http://blog.twitter.com/2010/02/measuring-tweets.html.

Cited By

View all

Index Terms

  1. Generalized scale independence through incremental precomputation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGMOD '13: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data
    June 2013
    1322 pages
    ISBN:9781450320375
    DOI:10.1145/2463676
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 June 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. materialized view selection
    2. scalability
    3. scale independence

    Qualifiers

    • Research-article

    Conference

    SIGMOD/PODS'13
    Sponsor:

    Acceptance Rates

    SIGMOD '13 Paper Acceptance Rate 76 of 372 submissions, 20%;
    Overall Acceptance Rate 785 of 4,003 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Applications of Information Inequalities to Database Theory Problems2023 38th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)10.1109/LICS56636.2023.10175769(1-30)Online publication date: 26-Jun-2023
    • (2023)S/C: Speeding up Data Materialization with Bounded Memory2023 IEEE 39th International Conference on Data Engineering (ICDE)10.1109/ICDE55515.2023.00393(1981-1994)Online publication date: Apr-2023
    • (2020)On performance stability in LSM-based storage systemsProceedings of the VLDB Endowment10.14778/3372716.337271913:4(449-462)Online publication date: 6-Jan-2020
    • (2020)Uncovering Performance Interference of Multi-Tenants in Big Data Environments2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378158(2773-2778)Online publication date: 10-Dec-2020
    • (2020)Bounded Pattern Matching Using ViewsDatabase and Expert Systems Applications10.1007/978-3-030-59051-2_7(93-110)Online publication date: 13-Sep-2020
    • (2020)Bounded Pattern Matching Using ViewsDatabase and Expert Systems Applications10.1007/978-3-030-59003-1_19(285-303)Online publication date: 14-Sep-2020
    • (2019)Intermittent query processingProceedings of the VLDB Endowment10.14778/3342263.334227812:11(1427-1441)Online publication date: 1-Jul-2019
    • (2018)Bounded Query Rewriting Using ViewsACM Transactions on Database Systems10.1145/318367343:1(1-46)Online publication date: 23-Mar-2018
    • (2018)How not to structure your database-backed web applicationsProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180194(800-810)Online publication date: 27-May-2018
    • (2017)A Top-Down Approach to Achieving Performance Predictability in Database SystemsProceedings of the 2017 ACM International Conference on Management of Data10.1145/3035918.3064016(745-758)Online publication date: 9-May-2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media