Skip to main content
SpringerLink
Log in

Hmem: A Holistic Memory Performance Metric for Cloud Computing

  • Conference paper
  • First Online:
Benchmarking, Measuring, and Optimizing (Bench 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14521))

Included in the following conference series:

  • 90 Accesses

Abstract

With the proliferation of cloud computing, cloud service providers offer users a variety of choices in terms of pricing and computing performance. A critical factor impacting computing performance is main memory, often evaluated using bandwidth and access latency metrics. For two evaluations with the same workload while under different system configurations, it is hard to determine which system delivers better memory performance for the particular workload if neither evaluation data achieves higher bandwidth and lower latency simultaneously. This dilemma is further exacerbated under different memory access patterns. We recognize that state-of-the-art memory performance metrics cannot well address the dilemma. To address this challenge, we define a holistic memory performance metric, named Hmem, which is calculated from a fusion of bandwidth and latency metrics across different access patterns. To reflect the overall performance of a given workload, we calculate the correlation between our proposed metric and the workload’s throughput. Experimental results show that Hmem exhibits an average improvement of 70% on correlation coefficients compared to state-of-the-art memory performance metrics. A large cloud service provider has adopted Hmem to improve the efficiency of their memory performance evaluation and cloud server selection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://jihulab.com/solecnu/hmem.

References

  1. Garg, S.K., Versteeg, S., Buyya, R.: Smicloud: a framework for comparing and ranking cloud services. In: IEEE International Conference on Utility and Cloud Computing (UCC), pp. 210–218. IEEE (2011)

    Google Scholar 

  2. Gottscho, M., Govindan, S., Sharma, B., Shoaib, M., Gupta, P.: X-Mem: a cross-platform and extensible memory characterization tool for the cloud. In: IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 263–273. IEEE (2016)

    Google Scholar 

  3. Vish, V., Karthik, K., Thomas, W., Sri, S.: Intel Memory Latency Checker. https://www.intel.com/content/www/us/en/developer/articles/tool/intelr-memory-latency-checker.html

  4. John, L.K.: More on finding a single number to indicate overall performance of a benchmark suite. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(1), 3–8 (2004)

    Article  Google Scholar 

  5. Iqbal, M.F., John, L.K.: Confusion by all means. In: Proceedings of the 6th International Workshop on Unique Chips and Systems (UCAS) (2010)

    Google Scholar 

  6. Sun, X.H., Wang, D.: APC: a performance metric of memory systems. ACM SIGMETRICS Perf. Eval. Rev. (SIGMETRICS) 40(2), 125–130 (2012)

    Article  MathSciNet  Google Scholar 

  7. Chou, Y., Fahs, B., Abraham, S.: Microarchitecture optimizations for exploiting memory-level parallelism. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(2), 76–87 (2004)

    Article  Google Scholar 

  8. Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley, New York (1991)

    Google Scholar 

  9. Liu, J., Jaiyen, B., Veras, R., Mutlu, O.: RAIDR: retention-aware intelligent DRAM refresh. ACM SIGARCH Comput. Arch. News (SIGARCH) 40, 1–12 (2012)

    Article  Google Scholar 

  10. Hassan, H., Pekhimenko, G., Vijaykumar, N., Seshadri, V., Mutlu, O.: ChargeCache: reducing DRAM latency by exploiting row access locality. In: IEEE International Symposium on High Performance Computer Architecture (HPCA), pp. 581–593 (2016)

    Google Scholar 

  11. Hennessy, J.L., Patterson, D.A.: Computer Architecture: A Quantitative Approach. Elsevier, Amsterdam (2011)

    Google Scholar 

  12. Yi, L., Li, C., Guo, J.: CPI for runtime performance measurement: the good, the bad, and the ugly. In: IEEE International Symposium on Workload Characterization (IISWC), pp. 106–113 (2020)

    Google Scholar 

  13. Wang, L., Gao, W., Yang, K., Jiang, Z.: BOPS, a new computation-centric metric for datacenter computing. In: Benchmarking, Measuring, and Optimizing: Second BenchCouncil International Symposium (Bench), pp. 262–277 (2020)

    Google Scholar 

  14. Williams, S., Waterman, A., Patterson, D.: Roofline: an insightful visual performance model for multicore architectures. Commun. ACM 52(4), 65–76 (2009)

    Article  Google Scholar 

  15. Lee, J., Kim, C., Lin, K., Cheng, L., Govindaraju, R., Kim, J.: WSMeter: a performance evaluation methodology for google’s production warehouse-scale computers. In: Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), pp. 549–563 (2018)

    Google Scholar 

  16. Sun, L., Dong, H., Hussain, F.K., Hussain, O.K., Chang, E.: Cloud service selection: state-of-the-art and future research directions. J. Netw. Comput. Appl. 45, 134–150 (2014)

    Article  Google Scholar 

  17. Behzadian, M., Otaghsara, S.K., Yazdani, M., Ignatius, J.: A state-of-the-art survey of TOPSIS applications. Expert Syst. Appl. 39(17), 13051–13069 (2012)

    Article  Google Scholar 

  18. Sotoudeh-Anvari, A., Sadjadi, S., Molana, S., Sadi-Nezhad, S.: A new MCDM-based approach using BWM and SAW for optimal search model. Decis. Sci. Lett. 7(4), 395–404 (2018)

    Article  Google Scholar 

  19. de FSM Russo, R., Camanho, R.: Criteria in AHP: a systematic review of literature. Procedia Comput. Sci. 55, 1123–1132 (2015)

    Google Scholar 

  20. Mardani, A., Zavadskas, E.K., Govindan, K., Amat Senin, A., Jusoh, A.: VIKOR technique: a systematic review of the state of the art literature on methodologies and applications. Sustainability 8(1), 37–75 (2016)

    Article  Google Scholar 

  21. McCalpin, J.D.: Stream benchmark (1995). https://www.cs.virginia.edu/stream/ref.html

  22. McCalpin, J.D.: Stream2 benchmark (1999). https://cs.virginia.edu/stream/stream2

  23. Staelin, C.: Lmbench: an extensible micro-benchmark suite. Softw. Pract. Exp. 35(11), 1079–1105 (2005)

    Article  Google Scholar 

  24. Pase, D.: pChase benchmark (2013). https://github.com/maleadt/pChase

  25. Radulovi, M.: Memory bandwidth and latency in HPC: system requirements and performance impact. Universitat Politècnica de Catalunya, pp. 14–20 (2019)

    Google Scholar 

  26. Barenblatt, G.I.: Dimensional Analysis. CRC Press, Boca Raton (1987)

    Google Scholar 

  27. Little, J.D., Graves, S.C.: Little’s law, Building intuition: insights from basic operations management models and principles, pp. 81–100 (2008)

    Google Scholar 

  28. Dh, B.: Little’s law and high performance computing, NAS applications and tools group, NASA Ames Research Center (1977). https://crd.lbl.gov/~dhbailey/dhbpapers/little.pdf

  29. Mehta, S.: Performance analysis and optimization with little’s law. In: IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 12–23. IEEE (2022)

    Google Scholar 

  30. Islam, M.A., Arafath, M.Y., Hasan, M.J.: Design of DDR4 SDRAM controller. In: International Conference on Electrical and Computer Engineering (ICECE), pp. 148–151. IEEE (2014)

    Google Scholar 

  31. Citron, D., Hurani, A., Gnadrey, A.: The harmonic or geometric mean: does it really matter? ACM SIGARCH Comput. Arch. News (SIGARCH) 34(4), 18–25 (2006)

    Article  Google Scholar 

  32. Mashey, J.R.: War of the benchmark means: time for a truce. ACM SIGARCH Comput. Arch. News (SIGARCH) 32(4), 1–14 (2004)

    Article  Google Scholar 

  33. Marino, M.D.: Walter: wide I/O scaling of number of memory controllers versus frequency and voltage. IEEE Access 8, 193874–193889 (2020)

    Article  Google Scholar 

  34. Standard Performance Evaluation Corporation: SPEC CPU 2017. https://www.spec.org/cpu2017/

  35. Singh, S., Awasthi, M.: Memory centric characterization and analysis of SPEC CPU2017 suite. In: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (ICPE), pp. 285–292 (2019)

    Google Scholar 

Download references

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62272167). We want to thank our lab colleagues and anonymous reviewers for their valuable comments and suggestions. We would also like to thank Chengdong Li from Code Title Poetry (Hangzhou) Technology for his valuable insights during the research process.

Author information

Authors and Affiliations

  1. School of Data Science and Engineering, East China Normal University, Shanghai, 200062, China

    Yuyang Li, Ning Li, Yilei Zhang, Jianmei Guo & Bo Huang

  2. Tencent Cloud, Shenzhen, 518000, China

    Mengbang Xing & Wenxin Huang

Authors
  1. Yuyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ning Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yilei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianmei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mengbang Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wenxin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianmei Guo .

Editor information

Editors and Affiliations

  1. TU Wien, Vienna, Austria

    Sascha Hunold

  2. Chinese Academy of Sciences, Beijing, China

    Biwei Xie

  3. Illinois Institute of Technology, Chicago, IL, USA

    Kai Shu

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, Y. et al. (2024). Hmem: A Holistic Memory Performance Metric for Cloud Computing. In: Hunold, S., Xie, B., Shu, K. (eds) Benchmarking, Measuring, and Optimizing. Bench 2023. Lecture Notes in Computer Science, vol 14521. Springer, Singapore. https://doi.org/10.1007/978-981-97-0316-6_11

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-0316-6_11

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-0315-9

  • Online ISBN: 978-981-97-0316-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation