Abstract
This paper discusses some problems the benchmark researches should pay attention to from the perspective of Metrology. Metrology is about the science of measurement, and it is considered as the foundation of industry development, for you have to measure it before you know what level it reaches. Metrology has a series of mechanisms to ensure the attributes of the measurement results, including Accuracy, Traceability, consistency and legality. Benchmark is widely used to evaluate the information technology products and helps the users to choose the products they need, and if absorbing the ideas of Metrology research during the designing, developing and application procedures of the benchmark, the quality of the measurement result of the benchmark will be improved greatly and become more authoritative.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Zhan, J.-F., Gao, W.-L., et al.: BigDataBench: an open-source big data benchmark suite. Chin. J. Comput. 39(1), 196–211 (2016)
Gray, J.: Benchmark Handbook for Database and Transaction System, 2nd edn. Morgan Kaufmann Publishers, San Francisco (1933)
Ghazal, A., Rabl, T., Hu, M., et al.: BigBench: towards an industry standard benchmark for big data analytics. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, USA, pp. 1197–1208 (2013)
Ferdman, M., Adileh, A., Kocberber, O., et al.: Clearing the clouds: a study of emerging scale-out workload on modern hardware. ACM SIGPLAN Not. 47(4), 37–48 (2012)
Huang, S., Huang, J., Dai, J.Q., et al.: The Hibench benchmark suite: characterization of the MapReduce-based data analysis. In: Proceedings of the ICDE Workshops on Information & Software as Services, LongBeach, USA, pp. 41–51 (2010)
Pavlo, A., Paulson, E., Rasin, A., et al.: A comparison of approaches to large-scale data analysis. In: Proceedings of the 2009 ACM SIGMOD International Conference on Management of Data, Providence, USA, pp. 165–178 (2009)
Coper, B., Silberstein, A., Tam, E., et al.: Benchmarking cloud serving systems with YCSB. In: Proceeding of the 1st ACM Symposium on Cloud Computing, Indianapolis, USA, pp. 143–154 (2010)
Armstrong, T.G., Ponnekanti, V., Borthakur, D., Callaghan, M.: LinkBench: a database benchmark based on the Facebook social graph. In: Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, New York, USA, pp. 1185–1196 (2013)
Xia, F., Li, Y., Yu, C., et al.: BSMA: a benchmark for analytical queries over social media data. Proc. VLDB Endow. 7(13), 1573–1576 (2014)
Bing-Bing, L., Xiao-Feng, M., Ying-Jie, S.: CloudBM: a benchmark for cloud data management systems. J. Front. Comput. Sci. Technol. 6(6), 504–512 (2012). (in Chinese)
Adolf, R., Rama, S., Reagen, B., Wei, G.-Y., Brooks, D.: Fathom: reference workloads for modern deep learning methods. In: Workload Characterization (IISWC), pp. 1–10. IEEE (2016)
Deepbench. https://svail.github.io/DeepBench/
Chen, T., et al.: BenchNN: an the broad potential application scope of hardware neural network accelerators. In: 2012 IEEE International Symposium on Workload Characterization (IISWC), pp. 36–45. IEEE (2012)
Dong, S., Kaeli, D.: DNNMark: a deep neural network benchmark suite for GPUs. In: Proceedings of the General Purpose GPUs, pp. 63–72. ACM (2017)
Coleman, C., et al.: DAWNBench: an end-to-end deep learning benchmark and competition. Training 100(101), 102 (2017)
Gao, W., Zhang, J., et al.: BigDataBench: a scalable and unified big data and AI benchmark suite. arXiv:1802.08254v2 (22 November 2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Yang, K., Wu, T., Shen, Q., Cui, W., Zhang, G. (2020). Benchmark Researches from the Perspective of Metrology. In: Gao, W., Zhan, J., Fox, G., Lu, X., Stanzione, D. (eds) Benchmarking, Measuring, and Optimizing. Bench 2019. Lecture Notes in Computer Science(), vol 12093. Springer, Cham. https://doi.org/10.1007/978-3-030-49556-5_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-49556-5_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49555-8
Online ISBN: 978-3-030-49556-5
eBook Packages: Computer ScienceComputer Science (R0)