Abstract
It is true that a metric can influence a benchmark but will esoteric metrics create more problems than they will solve? We answer this question affirmatively by examining the case of the TPC-D metric which used the much debated geometric mean for the single-stream test. We will show how a simple choice influenced the benchmark and its conduct and, to some extent, DBMS development. After examining other alternatives our conclusion is that the “real” measure for a decision-support benchmark is the arithmetic mean.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
TPC BENCHMARK D (Decision Support), Transaction Processing Council, http://www.tpc.org
Fleming, P., Wallace, J.: How Not to Lie With Statistics: The Correct Way to Summarize Benchmarks. Comm. ACM 29(3), 218–221 (1986)
Licea-Kane, B.: http://www.spec.org/gwpg/gpc.static/geometric.html
Crolotte, A.: Issues in Metric Selection and the TPC-D Single Stream Power, http://www.tpc.org
Smith, J.: Characterizing Computer Performance with a Single Number. Communications of of the ACM 31(10) (October 1988)
Kurian John, L.: More on Finding a Single Number to indicate Overall Performance of a Benchmark Suite. ACM SIGARCH Computer Architecture News 32 (March 2004)
Citron, D., Hurani, D., Gnadrey, A.: The Harmonic or Geometric Mean: Does it Really Matter? ACM SIGARCH Computer Architecture News 34(4) (September 2006)
Mashey, J.: War of the benchmark Means: Time for a Truce. ACM SIGARCH Computer Architecture News 32(4) (September 2004)
Calot, G.: Cours de Statistique Descriptive, Dunod, Paris (1964)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Crolotte, A. (2009). Issues in Benchmark Metric Selection. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2009. Lecture Notes in Computer Science, vol 5895. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10424-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-10424-4_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-10423-7
Online ISBN: 978-3-642-10424-4
eBook Packages: Computer ScienceComputer Science (R0)