Skip to main content
SpringerLink
Log in

Metrics for Big Data Benchmarks

  • Living reference work entry
  • First Online:
Encyclopedia of Big Data Technologies

  • 513 Accesses

Synonyms

Figure of merit; Gauge; Measure; Yardstick

Definition

A big data (BD) benchmark metric is a standard measure indicating the adequacy and cost-effectiveness of the system under test (SUT) to perform a particular big data task or set of tasks.

Overview

The following section provides a survey of the big data benchmark and associated metrics evolution over the years until present day. In the foundations section we list all the main metrics while the following sections present the definitions and properties for each metrics category namely, response time and throughput, availability and reliability, price-performance and system-level metrics. We then go over the various methods to aggregate these individual metrics while the criticisms section reviews the metrics surveyed. We then list the key applications of these benchmarks. Finally cross-references and references are provided.

Historical Background

The need for computer performance metrics was identified as early as 1985 (Anon 1985...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  • Anon et al (1985) A measure of transaction processing power. Datamation, 31 pp. 112–118, 1 April 1985

    Google Scholar 

  • Crolotte A (2009) Issues in benchmark metric selection. In: TPCTC, Lyon, pp 146–152

    Google Scholar 

  • Fleming P, Wallace J (1986) How to not lie with statistics: the correct way to summarize benchmark results. Commun ACM 29:218–221

    Article  Google Scholar 

  • Ghazal A, Rabl T, Hu M, Raab F, Poess M, Crolotte A, Jacobsen HA (2013) BigBench: towards an industry standard benchmark for big data analytics. In: SIGMOD

    Google Scholar 

  • Ghazal, A., Ivanov, T., Kostamaa, P., Crolotte, A., Voong, R., Al-Kateb, M., Ghazal, W., Zicari, R. (2017) BigBench V2 – the new and improved BigBench. In: SIGMOD

    Google Scholar 

  • Han R, Kurian L, Zhan J (2017) Benchmarking big data systems: a review. IEEE Trans Serv Comput, (99):1–18

    Google Scholar 

  • Huang S, Huang J, Dai J, Xie T and Huang B (2010) The HiBench benchmark suite: characterization of the mapreduce-based data analysis. In: ICDEW, Mar 2010

    Google Scholar 

  • Huppler K (2009) The art of building a good benchmark. In: Performance evaluation and benchmarking, vol 5895. Springer, Berlin/Heidellberg, pp 18–30

    Google Scholar 

  • Ivanov T, Rabl T, Poess M, Queralt A, Poelman J, Poggi N, Buell J (2015) Big data benchmark compendium. In: 7th TPC technology conference. (TPCTC)

    Google Scholar 

  • Laney D (2012) Deja VVVu: others claiming Gartner’s construct for big data. Gartner Blog Netw. 14 Jan 2012. https://blogs.gartner.com/doug-laney/deja-vvvue-others-claiming-gartners-volume-velocity-variety-construct-for-big-data/

Download references

Author information

Authors and Affiliations

  1. Teradata Corporation, El Segundo, CA, USA

    Alain Crolotte

Authors
  1. Alain Crolotte
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alain Crolotte .

Editor information

Editors and Affiliations

  1. School of Comp. Sci. and Engineering, University of New South Wales School of Comp. Sci. and Engineering, Eveleigh, New South Wales, Australia

    Sherif Sakr

  2. Sch of Info Techno, Building J12, University of Sydney Sch of Info Techno, Building J12, Sydney, Australia

    Albert Zomaya

Section Editor information

  1. Server Technologies, Oracle, Redwood Shores, California, United States

    Meikel Poess

  2. Database Systems and Information Management Group, Technische Universität Berlin, Berlin, Germany

    Tilmann Rabl

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Crolotte, A. (2018). Metrics for Big Data Benchmarks. In: Sakr, S., Zomaya, A. (eds) Encyclopedia of Big Data Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-63962-8_122-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-63962-8_122-1

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-63962-8

  • Online ISBN: 978-3-319-63962-8

  • eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation