Abstract
In 1986, Fleming and Wallace presented a case advocating the use of geomean in benchmark results. 23 years later in 2009, Alain Crolotte followed up on that proposal at TPCTC using TPC-D as a reference. Now 11 years later it is time to present another perspective on the age-old argument regarding the best metric for summarizing benchmark data. The aim of this paper is two-fold: (1) summarize the definition and interpretation of the current benchmark metrics for the OLAP family of the TPC benchmarks, including TPC-H, TPC-DS, and TPCx-BB. (2) illustrate the impact and tradeoffs of different statistical measures on the overall benchmark metric score, using both conceptual and data-driven arguments. Our hope is that the paper reinvigorates interest in the benchmark community to re-evaluate the design of benchmark metrics and offer insights that can influence the future direction of benchmark metrics design.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
ML Benchmark Design Challenges - Hot Chips. https://www.hotchips.org/hc31/HC31_1.9_MethodologyAndMLSystem-MLPerf-rev-b.pdf. Accessed 14 Sept 2020
TPC-H. http://www.tpc.org/tpch/default5.asp. Accessed 14 Sept 2020
TPC-DS. http://www.tpc.org/tpcds/default5.asp. Accessed 14 Sept 2020
TPCx-BB. http://www.tpc.org/tpcx-bb/default5.asp. Accessed 14 Sept 2020
Crolotte, A.: Issues in benchmark metric selection. In: Nambiar, R., Poess, M. (eds.) TPCTC 2009. LNCS, vol. 5895, pp. 146–152. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10424-4_11
Fleming, P.J., Wallace, J.J.: How not to lie with statistics: the correct way to summarize benchmark results. Commun. ACM 29(3), 218–221 (1986)
Vogel, R.M.: The geometric mean?. Commun. Stat. Theor. Methods 1–13 (2020)
Nambiar, R.O., Poess, M.: The Making of TPC-DS. In: VLDB, vol. 6, pp. 1049–1058 (2006)
Mashey, J.R.: War of the benchmark means: time for a truce. ACM SIGARCH Comput. Archit. News 32(4), 1–14 (2004)
John, LK.: More on finding a single number to indicate overall performance of a benchmark suite. ACM SIGARCH Comput. Archit. News 32(1), 3–8 (2004)
Iqbal, M.F., John, L.K.: Confusion by all means. In: Proceedings of the 6th International Workshop on Unique Chips and Systems (UCAS-6). (2010)
Citron, D., Hurani, A., Gnadrey, A.: The harmonic or geometric mean: does it really matter? ACM SIGARCH Comput. Archit. News 34(4), 18–25 (2006)
Three simple statistics for your data visualizations. https://breakforsense.net/three-statistics/. Accessed 14 Sept 2020
TPC-H Google Scholar Search Results. https://scholar.google.com/scholar?as_vis=1&q=tpc-h+&hl=en&as_sdt=1,48. Accessed 14 Sept 2020
TPC-H Results. http://www.tpc.org/tpch/results/tpch_advanced_sort5.asp?PRINTVER=false&FLTCOL1=ALL&ADDFILTERROW=&filterRowCount=1&SRTCOL1=h_sponsor&SRTDIR1=ASC&ADDSORTROW=&sortRowCount=1&DISPRES=100+PERCENT&include_withdrawn_results=none&include_historic_results=yes. Accessed 14 Sept 2020
TPC-H Publication. http://www.tpc.org/tpch/results/tpch_result_detail5.asp?id=119040201. Accessed 14 Sept 2020
TPC-DS Publication. http://www.tpc.org/tpcds/results/tpcds_result_detail5.asp?id=120061701. Accessed 14 Sept 2020
TPCx-BB Publication. http://www.tpc.org/tpcx-bb/results/tpcxbb_result_detail5.asp?id=119101101. Accessed 14 Sept 2020
Cisco UCS Integrated Infrastructure for Big Data. http://www.tpc.org/tpcds/results/tpcds_result_detail5.asp?id=118030501. Accessed 14 Sept 2020
Alibaba Cloud AnalyticDB. http://www.tpc.org/tpcds/results/tpcds_result_detail5.asp?id=120061701. Accessed 14 Sept 2020
TPC. http://www.tpc.org. Accessed 15 Sept 2020
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Elford, C., Aggarwal, D., Shekhar, S. (2021). Revisiting Issues in Benchmark Metric Selection. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2020. Lecture Notes in Computer Science(), vol 12752. Springer, Cham. https://doi.org/10.1007/978-3-030-84924-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-84924-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-84923-8
Online ISBN: 978-3-030-84924-5
eBook Packages: Computer ScienceComputer Science (R0)