Abstract
Blockchain benchmarking systems are actively discussed in the literature, focusing on increasing the number of blockchains that can be supported. However, the constant inception of new blockchains into the market and their vast implementation differences make it a massive engineering challenge. We provide a general discussion on the main aspects of benchmarking blockchains, highlighting the necessary contributions from the developers and users of blockchains and benchmarking systems. We identify problem statements across four benchmarking factors by investigating five popular permissioned blockchains. Further, we define a broad methodology to tackle these problems. We conduct a case study of five existing blockchain benchmarking systems for our evaluation and identify their limitations, clarifying the need for our methodology.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amsden, Z.: The libra blockchain (2020). https://diem-developers-components.netlify.app/papers/the-diem-blockchain/2020-05-26.pdf
Androulaki, E., et al.: Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the Thirteenth EuroSys Conference, pp. 1–15 (2018)
Bano, S., et al.: State machine replication in the libra blockchain (2020). https://developers.libra.org/docs/state-machine-replication-paper. Accessed 19 Dec 2020)
Bermbach, D., Wittern, E., Tai, S.: Cloud Service Benchmarking. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-55483-9
Blockbench (2020). https://github.com/ooibc88/blockbench. Accessed 14 Aug 2023
Brent, L., Fekete, A.: A versatile framework for painless benchmarking of database management systems. In: Chang, L., Gan, J., Cao, X. (eds.) ADC 2019. LNCS, vol. 11393, pp. 45–56. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12079-5_4
Brown, R.G., Carlyle, J., Grigg, I., Hearn, M.: Corda: an introduction. R3 CEV 1(15), 14 (2016)
Btcmark (2020). https://gitlab.inria.fr/dsaingre/bctmark. Accessed 14 Aug 2023
Chacko, J.A., Mayer, R., Jacobsen, H.A.: Why do my blockchain transactions fail? a study of hyperledger fabric. In: Proceedings of the 2021 International Conference on Management of Data, SIGMOD/PODS 2021, pp. 221–234. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3448016.3452823
Chacko, J.A., Mayer, R., Jacobsen, H.A.: How to optimize my blockchain? a multi-level recommendation approach. Proc. ACM Manag. Data 1(1) (2023). https://doi.org/10.1145/3588704
Chainhammer (2020). https://github.com/drandreaskrueger/chainhammer. Accessed 14 Aug 2023
Chung, G., et al.: Performance tuning and scaling enterprise blockchain applications. arXiv preprint arXiv:1912.11456 (2019)
Clique proof-of-authority consensus protocol (2020). https://eips.ethereum.org/EIPS/eip-225. Accessed 14 Aug 2023
Configure bootnodes (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/bootnodes/. Accessed 14 Aug 2023
Configure static nodes (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/static-nodes/. Accessed 14 Aug 2023
Consensus protocols (2020). https://docs.goquorum.consensys.net/concepts/consensus. Accessed 14 Aug 2023
Corda configurations (2020). https://docs.r3.com/en/platform/corda/4.8/enterprise/node/setup/corda-configuration-fields.html. Accessed 14 Aug 2023
Corda key concepts (2020). https://docs.r3.com/en/platform/corda/4.10/community/key-concepts.html. Accessed 14 Aug 2023
Corda notaries (2020). https://docs.r3.com/en/platform/corda/4.10/community/key-concepts-notaries.html. Accessed 14 Aug 2023
Corda use case directory (2023). https://r3.com/products/use-case-directory-all/. Accessed 14 Aug 2023
Crain, T., Natoli, C., Gramoli, V.: Red belly: a secure, fair and scalable open blockchain. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 466–483. IEEE (2021)
Diablo blockchain benchmark suite (2020). https://diablobench.github.io/. Accessed 14 Aug 2023
Diablo blockchain benchmark suite (2020). https://diablobench.github.io/blockchain-howto. Accessed 14 Aug 2023
20 Diem consensus configurations (2020). https://github.com/diem/diem/blob/latest/config/src/config/consensus_config.rs. Accessed 14 Aug 2023
Diem mempool configurations (2020). https://github.com/diem/diem/blob/latest/config/src/config/mempool_config.rs. Accessed 14 Aug 2023
Diem validator nodes (2020). https://developers.diem.com/docs/basics/basics-validator-nodes. Accessed 14 Aug 2023
Diem white paper (2020). https://developers.diem.com/docs/technical-papers/the-diem-blockchain-paper/. Accessed 14 Aug 2023
Dinh, T.T.A., Wang, J., Chen, G., Liu, R., Ooi, B.C., Tan, K.L.: Blockbench: a framework for analyzing private blockchains. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1085–1100 (2017)
Gilad, Y., Hemo, R., Micali, S., Vlachos, G., Zeldovich, N.: Algorand: Scaling byzantine agreements for cryptocurrencies. In: Proceedings of the 26th Symposium on Operating Systems Principles, SOSP 2017, pp. 51–68. Association for Computing Machinery, New York (2017).https://doi.org/10.1145/3132747.3132757
Go ethereum (2020). https://geth.ethereum.org/docs/interface/command-line-options. Accessed 14 Aug 2023
Goquorum configuration file (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/use-configuration-file/. Accessed 14 Aug 2023
Goquorum qlight (2020). https://consensys.net/docs/goquorum/en/latest/concepts/qlight-node/. Accessed 14 Aug 2023
Gramoli, V., Guerraoui, R., Lebedev, A., Natoli, C., Voron, G.: Diablo: a benchmark suite for blockchains. In: 18th ACM European Conference on Computer Systems (EuroSys) (2023)
Gray, J.: Benchmark Handbook: for Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., Burlington (1992)
Greenspan, G., et al.: Multichain private blockchain-white paper, pp. 57–60 (2015). http://www.multichain.com/download/MultiChain-White-Paper.pdf
Gromit blockchain benchmarking tool (2020). https://github.com/grimadas/gromit. Accessed 14 Aug 2023
Hearn, M., Brown, R.G.: Corda: A distributed ledger. Corda Technical White Paper 2016 (2016)
How fabric networks are structured (2020). https://hyperledger-fabric.readthedocs.io/en/latest/network/network.html. Accessed 14 Aug 2023
Hyperledger besu (2020). https://www.hyperledger.org/use/besu. Accessed 14 Aug 2023
https://hyperledger.github.io/caliper/ (2020), [Online; accessed 14-August-2023]
Hyperledger caliper benchmarks (2020). https://github.com/hyperledger/caliper-benchmarks. Accessed 14 Aug 2023
Hyperledger foundation case studies (2023). https://www.hyperledger.org/learn/case-studies. Accessed 14 Aug 2023
Hyperledgerlab ii (2020). https://github.com/MSRG/HyperLedgerLab-2.0. Accessed 14 Aug 2023
Klenik, A., Kocsis, I.: Porting a benchmark with a classic workload to blockchain: TPC-c on hyperledger fabric. In: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp. 290–298 (2022)
Kokoris-Kogias, E., Jovanovic, P., Gailly, N., Khoffi, I., Gasser, L., Ford, B.: Enhancing bitcoin security and performance with strong consistency via collective signing. In: Proceedings of the 25th USENIX Conference on Security Symposium, SEC 2016, pp. 279–296, USENIX Association, USA (2016)
Li, M., et al.: Auto-tuning with reinforcement learning for permissioned blockchain systems. Proc. VLDB Endow. 16(5), 1000–1012 (2023). https://doi.org/10.14778/3579075.3579076
Liu, M., Yu, F.R., Teng, Y., Leung, V.C.M., Song, M.: Performance optimization for blockchain-enabled industrial internet of things (iiot) systems: a deep reinforcement learning approach. IEEE Trans. Ind. Inf. 15(6), 3559–3570 (2019). https://doi.org/10.1109/TII.2019.2897805
Lokhava, M., et al.: Fast and secure global payments with stellar. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 2019, pp. 80–96. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3341301.3359636
Mazzoni, M., Corradi, A., Di Nicola, V.: Performance evaluation of permissioned blockchains for financial applications: the consensys quorum case study. Blockchain: Res. Appl. 3(1), 100026 (2022). https://doi.org/10.1016/j.bcra.2021.100026
Morgan, J.: Quorum Whitepaper. JP Morgan Chase, New York (2016)
Multichain configurations (2020). https://www.multichain.com/developers/blockchain-parameters/. Accessed 14 Aug 2023
Multichain data streams (2020). https://www.multichain.com/developers/data-streams/. Accessed 14 August 2023
Multichain performance optimization (2020). https://www.multichain.com/developers/ performance-optimization/. Accessed 14 Aug 2023
Multichain product partners (2023). https://www.multichain.com/product-partners/. Accessed 14 Aug 2023
Multichain white paper (2020). https://www.multichain.com/download/MultiChain-White-Paper.pdf. Accessed 14 Aug 2023
Nasirifard, P., Mayer, R., Jacobsen, H.A.: Fabriccrdt: a conflict-free replicated datatypes approach to permissioned blockchains. In: Proceedings of the 20th International Middleware Conference, Middleware 2019, pp. 110–122. Association for Computing Machinery, New York (2019).https://doi.org/10.1145/3361525.3361540
Nasrulin, B., De Vos, M., Ishmaev, G., Pouwelse, J.: Gromit: benchmarking the performance and scalability of blockchain systems. In: 2022 IEEE International Conference on Decentralized Applications and Infrastructures (DAPPS), pp. 56–63. IEEE (2022)
Parity: Blockchain infrastructure for the decentralised web (2020). https://www.parity.io/. Accessed 14 Aug 2023
Quorum blockchain in action (2023). https://consensys.net/quorum/enterprise/. Accessed 14 Aug 2023
Quorum white paper (2020). https://github.com/ConsenSys/quorum/blob/master/docs/Quorum%20Whitepaper%20v0.2.pdf. Accessed 14 Aug 2023
Raasveldt, M., Holanda, P., Gubner, T., Mühleisen, H.: Fair benchmarking considered difficult: common pitfalls in database performance testing. In: Proceedings of the Workshop on Testing Database Systems, DBTest 2018. Association for Computing Machinery, New York (2018).https://doi.org/10.1145/3209950.3209955
Rauchs, M., Blandin, A., Bear, K., McKeon, S.B.: 2nd global enterprise blockchain benchmarking study. SSRN 3461765 (2019)
Rocket, T., Yin, M., Sekniqi, K., van Renesse, R., Sirer, E.G.: Scalable and probabilistic leaderless BFT consensus through metastability (2020)
Ruan, P., et al.: Blockchains vs. distributed databases: dichotomy and fusion. In: Proceedings of the 2021 International Conference on Management of Data, SIGMOD 2021, pp. 1504–1517. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3448016.3452789
Saingre, D., Ledoux, T., Menaud, J.M.: Bctmark: a framework for benchmarking blockchain technologies. In: 2020 IEEE/ACS 17th International Conference on Computer Systems and Applications (AICCSA), pp. 1–8. IEEE (2020)
Schuh, F., Larimer, D.: Bitshares 2.0: general overview (2017)
Smaalders, B.: Performance anti-patterns: want your apps to run faster? here’s what not to do. Queue 4(1) (2006). https://doi.org/10.1145/1117389.1117403
Solana: a new architecture for a high performance blockchain v0.8.13 (2020). https://solana.com/solana-whitepaper.pdf. Accessed 14 Aug 2023
Tennakoon, D., Gramoli, V.: Smart red belly blockchain: enhanced transaction management for decentralized applications. arXiv preprint arXiv:2207.05971 (2022)
Thakkar, P., Nathan, S., Viswanathan, B.: Performance benchmarking and optimizing hyperledger fabric blockchain platform. In: 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 264–276 (2018). https://doi.org/10.1109/MASCOTS.2018.00034
Tran, N.K., Babar, M.A., Walters, A.: A framework for automating deployment and evaluation of blockchain networks. J. Netw. Comput. Appl. 206, 103460 (2022). https://doi.org/10.1016/j.jnca.2022.103460
Types of transactions (2020). https://developers.diem.com/docs/transactions/txns-types/. Accessed 14 Aug 2023
Valenta, M., Sandner, P.G.: Comparison of ethereum, hyperledger fabric and corda (2017)
Wood, D.D.: Ethereum: a secure decentralised generalised transaction ledger (2014)
Writing connectors (2020). https://hyperledger.github.io/caliper/v0.5.0/writing-connectors/. Accessed 14 Aug 2023
Zhang, J., et al.: Performance analysis of the libra blockchain: an experimental study. In: 2019 2nd International Conference on Hot Information-Centric Networking (HotICN), pp. 77–83. IEEE (2019)
Acknowledgement
This work is funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 392214008, and by the Bavarian Cooperative Research Program of the Free State of Bavaria - DIK-2002-0013//DIK0114/02.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chacko, J.A., Mayer, R., Fekete, A., Gramoli, V., Jacobsen, HA. (2024). A Comprehensive Study on Benchmarking Permissioned Blockchains. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2023. Lecture Notes in Computer Science, vol 14247. Springer, Cham. https://doi.org/10.1007/978-3-031-68031-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-68031-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-68030-4
Online ISBN: 978-3-031-68031-1
eBook Packages: Computer ScienceComputer Science (R0)