Skip to main content
SpringerLink
Log in

Resource bricolage and resource selection for parallel database systems

  • Special Issue Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Running parallel database systems in an environment with heterogeneous resources has become increasingly common, due to cluster evolution and increasing interest in moving applications into public clouds. Performance differences among machines in the same cluster pose new challenges for parallel database systems. First, for database systems running in a heterogeneous cluster, the default uniform data partitioning strategy may overload some of the slow machines, while at the same time it may underutilize the more powerful machines. Since the processing time of a parallel query is determined by the slowest machine, such an allocation strategy may result in a significant query performance degradation. Second, since machines might have varying resources or performance, different choices of machines may lead to different costs or performance for executing the same workload. By carefully selecting the most suitable machines for running a workload, we may achieve better performance with the same budget, or we may meet the same performance requirements with a lower cost. We address these challenges by introducing techniques we call resource bricolage and resource selection that improve database performance in heterogeneous environments. Our approaches quantify the performance differences among machines with various resources as they process workloads with diverse resource requirements. For the purpose of better resource utilization, we formalize the problem of minimizing workload execution time and view it as an optimization problem, and then, we employ linear programming to obtain a recommended data partitioning scheme. For the purpose of better resource selection, we formalize two problems: One minimizes the total workload execution time with a given budget, and the other minimizes the total budget with a given performance target. We then employ different mixed-integer programs to search for the optimal resource selection decisions. We verify the effectiveness of both resource bricolage and resource selection techniques with an extensive experimental study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Notes

  1. We omit replicated tables in our problem. Since a full copy of a replicated table will be kept on a machine, there is no need to worry about partitioning.

  2. We assume that the I/Os for generating the sorted runs are done by a scan operator, and we omit the cost here.

References

  1. Agrawal, S., Narasayya, V., Yang, B.: Integrating vertical and horizontal partitioning into automated physical database design. In: SIGMOD (2004)

  2. Ahmad, F., Chakradhar, S.T., Raghunathan, A., Vijaykumar. T.N.: Tarazu: Optimizing mapreduce on heterogeneous clusters. In: ASPLOS (2012)

  3. Ahmad, M., Aboulnaga, A., Babu, S., Munagala, K.: Modeling and exploiting query interactions in database systems. In: CIKM (2008)

  4. Akdere, M., Çetintemel, U., Riondato, M., Upfal, E., Zdonik, S.B.: Learning-based query performance modeling and prediction. In: ICDE (2012)

  5. Chaudhuri, S., Kaushik, R., Ramamurthy, R.: When can we trust progress estimators for SQL queries? In: SIGMOD (2005)

  6. Chaudhuri, S., Narasayya, V., Ramamurthy, R.: Estimating progress of execution for SQL queries. In: SIGMOD (2004)

  7. Copeland, G., Alexander, W., Boughter, E., Keller, T.: Data placement in Bubba. In: SIGMOD Record (1988)

  8. Curino, C., Jones, E., Zhang, Y., Madden, S.: Schism: a workload-driven approach to database replication and partitioning. In: PVLDB (2010)

  9. Dantzig, G.B., Thapa, M.N.: Linear Programming 1: Introduction. Springer, Berlin (1997)

    MATH  Google Scholar 

  10. Delimitrou, C., and Kozyrakis, C.: Paragon: Qos-aware scheduling for heterogeneous datacenters. In: ASPLOS (2013)

  11. DeWitt, D.J., Naughton, J.F., Schneider, D.A., Seshadri, S.: Practical skew handling in parallel joins. In: VLDB (1992)

  12. Duggan, J., Cetintemel, U., Papaemmanouil, O., and Upfal, E.: Performance prediction for concurrent database workloads. In: SIGMOD (2011)

  13. Farley, B., Juels, A., Varadarajan, V., Ristenpart, T., Bowers, K.D., Swift, M.M.: More for your money: exploiting performance heterogeneity in public clouds. In: SoCC (2012)

  14. Ganapathi, A., Kuno, H., Dayal, U., Wiener, J.L., Fox, A., Jordan, M., Patterson, D.: Predicting multiple metrics for queries: better decisions enabled by machine learning. In: ICDE (2009)

  15. Ghandeharizadeh, S., DeWitt, D.J., Qureshi, W.: A performance analysis of alternative multi-attribute declustering strategies. In: SIGMOD (1992)

  16. Johnson, D.S.: The NP-completeness column: An ongoing guide. J. Algorithms. 6(3), 434–451 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  17. König, A.C., Ding, B., Chaudhuri, S., Narasayya, V.: A statistical approach towards robust progress estimation. In: PVLDB (2012)

  18. Li, J., Nehme, R.V., Naughton, J.F.: GSLPI: A cost-based query progress indicator. In ICDE (2012)

  19. Li, J., Nehme, R.V., Naughton, J.F., Toward progress indicators on steroids for big data systems. In CIDR (2013)

  20. Luo, G., Naughton, J.F., Ellmann, C.J., Watzke, M.W.: Toward a progress indicator for database queries. In: SIGMOD (2004)

  21. Luo, G., Naughton, J.F., Ellmann, C.J., Watzke, M.W.: Increasing the accuracy and coverage of SQL progress indicators. In: ICDE (2005)

  22. Mangot, D.: EC2 Variability: The Numbers Revealed. http://tech.mangot.com/roller/dave/entry/ec2_variability_the_numbers_revealed (2009)

  23. Mehta, M., DeWitt, D.J.: Data placement in shared-nothing parallel database systems. VLDB J. 6(1), 53–72 (1997)

    Article  Google Scholar 

  24. Mishra, C., Koudas, N.: A lightweight online framework for query progress indicators. In: ICDE (2007)

  25. Morton, K., Balazinska, M., Grossman, D.: Paratimer: a progress indicator for MapReduce DAGs. In: SIGMOD (2010)

  26. Morton, K., Friesen, A., Balazinska, M., Grossman, D.: Estimating the progress of MapReduce pipelines. In: ICDE (2010)

  27. Nehme, R., Bruno, N.: Automated partitioning design in parallel database systems. In: SIGMOD (2011)

  28. Ou, Z., Zhuang, H., Lukyanenko, A., Nurminen, J.K., Hui, P., Mazalov, V., Yla-Jaaski, A.: Is the same instance type created equal? Exploiting heterogeneity of public clouds. IEEE Trans. Cloud Comput. 1(2), 201–214 (2013)

    Article  Google Scholar 

  29. Ou, Z., Zhuang, H., Nurminen, J.K., Ylä-Jääski, A., Hui, P.: Exploiting hardware heterogeneity within the same instance type of Amazon EC2. In: HotCloud (2012)

  30. Pavlo, A., Curino, C., Zdonik, S.: Skew-aware automatic database partitioning in shared-nothing, parallel OLTP systems. In: SIGMOD (2012)

  31. Rao, J., Zhang, C., Megiddo, N., Lohman, G.: Automating physical database design in a parallel database. In: SIGMOD (2002)

  32. Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., Kozuch, M. A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In: SoCC (2012)

  33. Schad, J., Dittrich, J., Quiané-Ruiz, J.-A., Runtime measurements in the cloud: observing, analyzing, and reducing variance. In: PVLDB (2010)

  34. SQL Server 2012 Parallel Data Warehouse. http://www.microsoft.com/en-ca/server-cloud/products/analytics-platform-system/

  35. Wang, G., Ng, T.S.E.: The impact of virtualization on network performance of amazon EC2 data center. In: INFOCOM (2010)

  36. Wu, W., Chi, Y., Hacígümüş, H., Naughton, J.F.: Towards predicting query execution time for concurrent and dynamic database workloads. In: PVLDB (2013)

  37. Wu, W., Chi, Y., Zhu, S., Tatemura, J., Hacigümüs, H., Naughton, J.F.: Predicting query execution time: are optimizer cost models really unusable? In: ICDE (2013)

  38. Xu, Y., Kostamaa, P.: Efficient outer join data skew handling in parallel DBMS. In: PVLDB (2009)

  39. Xu, Y., Kostamaa, P., Zhou, X., Chen, L.: Handling data skew in parallel joins in shared-nothing systems. In: SIGMOD (2008)

  40. Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I. Improving mapreduce performance in heterogeneous environments. In: OSDI (2008)

  41. Zhuang, H., Liu, X., Ou, Z., Aberer, K.: Impact of instance seeking strategies on resource allocation in cloud data centers. In: CLOUD (2013)

  42. Zou, T., Bras, R.L., Salles, M.V., Demers, A., Gehrke, J.: ClouDiA: a deployment advisor for public clouds. In: PVLDB (2013)

  43. Zou, T., Wang, G., Salles, M.V., Bindel, D., Demers, A., Gehrke, J., White, W.: Making time-stepped applications tick in the cloud. In: SOCC (2011)

Download references

Acknowledgments

This research was supported by a grant from Microsoft Jim Gray Systems Lab, Madison, WI. We would like to thank everyone in the laboratory for valuable suggestions on this project

Author information

Authors and Affiliations

  1. Google Inc, Mountain View, CA, USA

    Jiexing Li

  2. Department of Computer Sciences, University of Wisconsin, Madison, Madison, WI, USA

    Jeffrey F. Naughton

  3. Microsoft Jim Gray Systems Lab, Madison, WI, USA

    Rimma V. Nehme

Authors
  1. Jiexing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeffrey F. Naughton
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rimma V. Nehme
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiexing Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, J., Naughton, J.F. & Nehme, R.V. Resource bricolage and resource selection for parallel database systems. The VLDB Journal 26, 31–54 (2017). https://doi.org/10.1007/s00778-016-0435-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-016-0435-4

Keywords

Search

Navigation