The Next 700 Benchmarking Frameworks for Concurrent Data Structures
Pages 1 - 9
Abstract
Concurrent data structures are essential building blocks for applications in almost every domain. However, differences among domains make it difficult to share results. Some testing frameworks model specific workloads, while others emphasize stress tests, but there is no easy way to evaluate if a new data structure will accelerate an application based only on such benchmarks. A confounding factor is the huge space of configuration options.
We address these issues with a new benchmarking tool for concurrent data structures in Java and C++. Our tool emphasizes a declarative description of workloads, a modular approach to defining components, and heterogeneity throughout.
As preliminary evidence of the effectiveness of our tool, we show that it can implement important and realistic workloads. We present six different workloads that lead to all six possibilities of relevant performance of the three most popular binary search trees written in Java.
References
[1]
TPC [n. d.]. TPC benchmarks. https://www.tpc.org/.
[2]
Yehuda Afek, Haim Kaplan, Boris Korenfeld, Adam Morrison, and Robert E Tarjan. 2014. The CB tree: a practical concurrent self-adjusting search tree. Distributed computing 27, 6 (2014), 393--417.
[3]
Vitaly Aksenov, Dan Alistarh, Alexandra Drozdova, and Amirkeivan Mohtashami. 2023. The splay-list: A distribution-adaptive concurrent skip-list. Distributed Computing (2023), 1--24.
[4]
Vitaly Aksenov, Vincent Gramoli, Petr Kuznetsov, Anna Malova, and Srivatsan Ravi. 2017. A concurrency-optimal binary search tree. In Euro-Par 2017: Parallel Processing. Springer, 580--593.
[5]
Nathan G Bronson, Jared Casper, Hassan Chafi, and Kunle Olukotun. 2010. A practical concurrent binary search tree. ACM Sigplan Notices 45, 5 (2010), 257--268.
[6]
Trevor Brown, Aleksandar Prokopec, and Dan Alistarh. 2020. Non-blocking interpolation search trees with doubly-logarithmic running time. In Proceedings of the 25th PPoPP. 276--291.
[7]
Brian F Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM symposium on Cloud computing. 143--154.
[8]
Tyler Crain, Vincent Gramoli, and Michel Raynal. 2013. A contention-friendly binary search tree. In Euro-Par 2013 Parallel Processing. Springer, 229--240.
[9]
Vincent Gramoli. 2015. More than you ever wanted to know about synchronization: Synchrobench, measuring the impact of the synchronization on concurrent algorithms. In Proceedings of the 20th Symposium on Principles and Practice of Parallel Programming. 1--10.
[10]
Andreas Haas, Thomas Hütter, Christoph M Kirsch, Michael Lippautz, Mario Preishuber, and Ana Sokolova. 2015. Scal: A benchmarking suite for concurrent data structures. In Networked Systems: Third International Conference, NETYS 2015, Agadir, Morocco, May 13-15, 2015, Revised Selected Papers 3. Springer, 1--14.
[11]
Nikolaos D. Kallimanis. 2021. Synch: A framework for concurrent data-structures and benchmarks. Journal of Open Source Software 6, 64 (2021), 3143.
[12]
Rosina Kharal and Trevor Brown. 2022. Performance Anomalies in Concurrent Data Structure Microbenchmarks. arXiv preprint arXiv:2208.08469 ( 2022).
[13]
David MW Powers. 1998. Applications and explanations of Zipf's law. In New methods in language processing and computational natural language learning.
Index Terms
- The Next 700 Benchmarking Frameworks for Concurrent Data Structures
Recommendations
Transactional Acceleration of Concurrent Data Structures
SPAA '15: Proceedings of the 27th ACM symposium on Parallelism in Algorithms and ArchitecturesConcurrent data structures are a fundamental building block for scalable multi-threaded programs. While Transactional Memory (TM) was originally conceived as a mechanism for simplifying the creation of concurrent data structures, modern hardware TM ...
On high performance distributed transactional data structures
MiddlewareDPT '13: Proceedings Demo & Poster Track of ACM/IFIP/USENIX International Middleware ConferenceWe present three protocols for developing high performance distributed transactional data structures. Our first protocol, QR-ON, incorporates the open nesting transactional model into QR, a quorum-based protocol for managing concurrency on distributed ...
Transactional data structure libraries
PLDI '16: Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and ImplementationWe introduce transactions into libraries of concurrent data structures; such transactions can be used to ensure atomicity of sequences of data structure operations. By focusing on transactional access to a well-defined set of data structure operations, ...
Comments
Information & Contributors
Information
Published In
June 2024
95 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 3 of 4 submissions, 75%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 72Total Downloads
- Downloads (Last 12 months)72
- Downloads (Last 6 weeks)7
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in