Abstract
Benchmarks that focus on running queries on a well-tuned database system ignore a long-standing problem: adverse runtime conditions can cause database system performance to vary widely and unexpectedly. When the query execution engine does not exhibit resilience to these adverse conditions, addressing the resultant performance problems can contribute significantly to the total cost of ownership for a database system in over-provisioning, lost efficiency, and increased human administrative costs. For example, focused human effort may be needed to manually invoke workload management actions or fine-tune the optimization of specific queries.
We believe a benchmark is needed to measure query execution robustness, that is, how adverse or unexpected conditions impact the performance of a database system. We offer a preliminary analysis of barriers to query execution robustness and propose some metrics for quantifying the impact of those barriers. We present and analyze results from preliminary tests on four real database systems and discuss how these results could be used to increase the robustness of query processing in each case. Finally, we outline how our efforts could be expanded into a benchmark to quantify query execution robustness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Babu, S., Bizarro, P., DeWitt, D.: Proactive re-optimization with rio. In: SIGMOD 2005: Proceedings of the 2005 ACM SIGMOD international conference on Management of data, pp. 936–938. ACM, New York (2005)
Bizarro, P., Babu, S., DeWitt, D., Widom, J.: Content-based routing: different plans for different data. In: VLDB 2005: Proceedings of the 31st international conference on Very large data bases, pp. 757–768. VLDB Endowment (2005)
Cole, R.L., Graefe, G.: Optimization of dynamic query evaluation plans. In: SIGMOD 1994: Proceedings of the 1994 ACM SIGMOD international conference on Management of data, pp. 150–160. ACM, New York (1994)
El Gebaly, K., Aboulnaga, A.: Robustness in automatic physical database design. In: EDBT (2008)
Graefe, G.: Query evaluation techniques for large databases. ACM Comput. Surv. 25(2), 73–169 (1993)
Graefe, G., Kuno, H.A., Wiener, J.L.: Visualizing the robustness of query execution. In: CIDR (2009)
Gupta, A., Davis, K.C., Grommon-Litton, J.: Performance comparison of property map and bitmap indexing. In: DOLAP 2002: Proceedings of the 5th ACM international workshop on Data Warehousing and OLAP, pp. 65–71. ACM, New York (2002)
Harish, D., Darera, P.N., Haritsa, J.R.: On the production of anorexic plan diagrams. In: VLDB 2007: Proceedings of the 33rd international conference on Very large data bases, pp. 1081–1092. VLDB Endowment (2007)
Harish, D., Darera, P.N., Haritsa, J.R.: Identifying robust plans through plan diagram reduction. In: VLDB, pp. 1124–1140 (2008)
Ioannidis, Y.E., Ng, R.T., Shim, K., Sellis, T.K.: Parametric query optimization. VLDB Journal 6(2), 132–151 (1997)
Krompass, S., Kuno, H., Dayal, U., Kemper, A.: Dynamic workload management for very large data warehouses: Juggling feathers and bowling balls. In: VLDB (2007)
Markl, V., Lohman, G.: Learning table access cardinalities with LEO. In: SIGMOD 2002: Proceedings of the 2002 ACM SIGMOD international conference on Management of data, pp. 613–613. ACM, New York (2002)
Markl, V., Raman, V., Simmen, D., Lohman, G., Pirahesh, H., Cilimdzic, M.: Robust query processing through progressive optimization. In: SIGMOD 2004: Proceedings of the 2004 ACM SIGMOD international conference on Management of data, pp. 659–670. ACM, New York (2004)
Othayoth, R., Poess, M.: The Making of TPC-DS. In: Proc. of the 32nd Intl. Conf. on Very Large Data Bases (VLDB), pp. 1049–1058 (2006)
Poess, M., Floyd, C.: New tpc benchmarks for decision support and web commerce. SIGMOD Rec. 29(4) (2000)
Reddy, N., Haritsa, J.R.: Analyzing plan diagrams of database query optimizers. In: VLDB 2005: Proceedings of the 31st international conference on Very large data bases, pp. 1228–1239. VLDB Endowment (2005)
Sarda, P., Haritsa, J.R.: Green query optimization: taming query optimization overheads through plan recycling. In: VLDB 2004: Proceedings of the Thirtieth international conference on Very large data bases, pp. 1333–1336. VLDB Endowment (2004)
Schneider, D.A., DeWitt, D.J.: A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment. SIGMOD Rec. 18(2), 110–121 (1989)
Stillger, M., Lohman, G.M., Markl, V., Kandil, M.: LEO - db2’s learning optimizer. In: VLDB 2001: Proceedings of the 27th International Conference on Very Large Data Bases, pp. 19–28. Morgan Kaufmann Publishers Inc., San Francisco (2001)
Zhang, N., Haas, P.J., Josifovski, V., Lohman, G.M., Zhang, C.: Statistical learning techniques for costing XML queries. In: VLDB 2005: Proceedings of the 31st international conference on Very large data bases, pp. 289–300. VLDB Endowment (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wiener, J.L., Kuno, H., Graefe, G. (2009). Benchmarking Query Execution Robustness. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2009. Lecture Notes in Computer Science, vol 5895. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10424-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-10424-4_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-10423-7
Online ISBN: 978-3-642-10424-4
eBook Packages: Computer ScienceComputer Science (R0)