Abstract
Benchmarking as yardsticks for system design and evaluation, has developed a long period and plays a pivotal role in many domains, such as database systems and high performance computing. Through prolonged and unremitting efforts, benchmarks on these domains have been reaching their maturity gradually. However, in terms of emerging scenarios of big data, its different properties in data volume, data types, data processing requirements and techniques, make that existing benchmarks are rarely appropriate for big data systems and further make us wonder how to define a good big data benchmark. In this paper, we revisit successful benchmarks in other domains from two perspectives: benchmarking principles which define fundamental rules, and methodologies which guide the benchmark constructions. Further, we conclude the benchmarking principle and methodology on big data benchmarking from a recent open-source effort – BigDataBench.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Spec. https://www.spec.org/
Tpc. http://www.tpc.org
Angles, R.: Benchmark principles and methods. In: Linked Data Benchmark Council (LDBC). Project No 317548, European Community’s Seventh Framework Programme FP7 (2012–2014)
Bienia, C., Li, K.: Benchmarking Modern Multiprocessors. Princeton University, New York (2011)
Chen, Y., Raab, F., Katz, R.: From TPC-C to big data benchmarks: a functional workload model. In: Rabl, T., Poess, M., Baru, C., Jacobsen, H.-A. (eds.) WBDB 2012. LNCS, vol. 8163, pp. 28–43. Springer, Heidelberg (2014)
Gao, W., Luo, C., Zhan, J., Ye, H., He, X., Wang, L., Zhu, Y., Tian, X.: Identifying dwarfs workloads in big data analytics (2015). arXiv preprint arXiv:1505.06872
Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., San Francisco (1992)
Levine, C.: TPC benchmarks. In: SIGMOD International Conference on Managementof Data - Industrial Session (1997)
Luo, C., Gao, W., Jia, Z., Han, R., Li, J., Lin, X., Wang, L., Zhu, Y., Zhan, J.: Handbook of bigdatabench (version 3.1) - a big data benchmark suite
Ming, Z., Luo, C., Gao, W., Han, R., Yang, Q., Wang, L., Zhan, J.: BDGS: A scalable big data generator suite in big data benchmarking. In: Rabl, T., Raghunath, N., Poess, M., Bhandarkar, M., Jacobsen, H.A., Baru, C. (eds.) Advancing Big Data Benchmarks. LNCS, vol. 8585. Springer, Heidelberg (2014)
Seltzer, M., Krinsky, D., Smith, K., Zhang, X.: The case for application-specific benchmarking. In: Proceedings of the Seventh Workshop on Hot Topics in Operating Systems, pp. 102–107. IEEE (1999)
Wang, L., Zhan, J., Luo, C., Zhu, Y., Yang, Q., He, Y., Gao, W., Jia, Z., Shi, Y., Zhang, S., et al.: BigDataBench: A big data benchmark suite from internet services. In: 2014 IEEE 20th International Symposium on High Performance Computer Architecture (HPCA), pp. 488–499. IEEE (2014)
Zhu, Y., Zhan, J., Weng, C., Nambiar, R., Zhang, J., Chen, X., Wang, L.: BigOP: generating comprehensive big data workloads as a benchmarking framework. In: Bhowmick, S.S., Dyreson, C.E., Jensen, C.S., Lee, M.L., Muliantara, A., Thalheim, B. (eds.) DASFAA 2014, Part II. LNCS, vol. 8422, pp. 483–492. Springer, Heidelberg (2014)
Acknowledgements
This work is supported by the National High Technology Research and Development Program of China (Grant No. 2015AA015308), the Major Program of National Natural Science Foundation of China (Grant No. 61432006), and the Key Technology Research and Development Programs of Guangdong Province, China (Grant No. 2015B010108006).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Zhao, L., Gao, W., Jin, Y. (2016). Revisiting Benchmarking Principles and Methodologies for Big Data Benchmarking. In: Zhan, J., Han, R., Zicari, R. (eds) Big Data Benchmarks, Performance Optimization, and Emerging Hardware. BPOE 2015. Lecture Notes in Computer Science(), vol 9495. Springer, Cham. https://doi.org/10.1007/978-3-319-29006-5_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-29006-5_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-29005-8
Online ISBN: 978-3-319-29006-5
eBook Packages: Computer ScienceComputer Science (R0)