Skip to main content

A Comprehensive Study on Benchmarking Permissioned Blockchains

  • Conference paper
  • First Online:
Performance Evaluation and Benchmarking (TPCTC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14247))

Included in the following conference series:

  • 185 Accesses

Abstract

Blockchain benchmarking systems are actively discussed in the literature, focusing on increasing the number of blockchains that can be supported. However, the constant inception of new blockchains into the market and their vast implementation differences make it a massive engineering challenge. We provide a general discussion on the main aspects of benchmarking blockchains, highlighting the necessary contributions from the developers and users of blockchains and benchmarking systems. We identify problem statements across four benchmarking factors by investigating five popular permissioned blockchains. Further, we define a broad methodology to tackle these problems. We conduct a case study of five existing blockchain benchmarking systems for our evaluation and identify their limitations, clarifying the need for our methodology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amsden, Z.: The libra blockchain (2020). https://diem-developers-components.netlify.app/papers/the-diem-blockchain/2020-05-26.pdf

  2. Androulaki, E., et al.: Hyperledger fabric: a distributed operating system for permissioned blockchains. In: Proceedings of the Thirteenth EuroSys Conference, pp. 1–15 (2018)

    Google Scholar 

  3. Bano, S., et al.: State machine replication in the libra blockchain (2020). https://developers.libra.org/docs/state-machine-replication-paper. Accessed 19 Dec 2020)

  4. Bermbach, D., Wittern, E., Tai, S.: Cloud Service Benchmarking. Springer, Heidelberg (2017). https://doi.org/10.1007/978-3-319-55483-9

    Book  Google Scholar 

  5. Blockbench (2020). https://github.com/ooibc88/blockbench. Accessed 14 Aug 2023

  6. Brent, L., Fekete, A.: A versatile framework for painless benchmarking of database management systems. In: Chang, L., Gan, J., Cao, X. (eds.) ADC 2019. LNCS, vol. 11393, pp. 45–56. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-12079-5_4

    Chapter  Google Scholar 

  7. Brown, R.G., Carlyle, J., Grigg, I., Hearn, M.: Corda: an introduction. R3 CEV 1(15), 14 (2016)

    Google Scholar 

  8. Btcmark (2020). https://gitlab.inria.fr/dsaingre/bctmark. Accessed 14 Aug 2023

  9. Chacko, J.A., Mayer, R., Jacobsen, H.A.: Why do my blockchain transactions fail? a study of hyperledger fabric. In: Proceedings of the 2021 International Conference on Management of Data, SIGMOD/PODS 2021, pp. 221–234. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3448016.3452823

  10. Chacko, J.A., Mayer, R., Jacobsen, H.A.: How to optimize my blockchain? a multi-level recommendation approach. Proc. ACM Manag. Data 1(1) (2023). https://doi.org/10.1145/3588704

  11. Chainhammer (2020). https://github.com/drandreaskrueger/chainhammer. Accessed 14 Aug 2023

  12. Chung, G., et al.: Performance tuning and scaling enterprise blockchain applications. arXiv preprint arXiv:1912.11456 (2019)

  13. Clique proof-of-authority consensus protocol (2020). https://eips.ethereum.org/EIPS/eip-225. Accessed 14 Aug 2023

  14. Configure bootnodes (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/bootnodes/. Accessed 14 Aug 2023

  15. Configure static nodes (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/static-nodes/. Accessed 14 Aug 2023

  16. Consensus protocols (2020). https://docs.goquorum.consensys.net/concepts/consensus. Accessed 14 Aug 2023

  17. Corda configurations (2020). https://docs.r3.com/en/platform/corda/4.8/enterprise/node/setup/corda-configuration-fields.html. Accessed 14 Aug 2023

  18. Corda key concepts (2020). https://docs.r3.com/en/platform/corda/4.10/community/key-concepts.html. Accessed 14 Aug 2023

  19. Corda notaries (2020). https://docs.r3.com/en/platform/corda/4.10/community/key-concepts-notaries.html. Accessed 14 Aug 2023

  20. Corda use case directory (2023). https://r3.com/products/use-case-directory-all/. Accessed 14 Aug 2023

  21. Crain, T., Natoli, C., Gramoli, V.: Red belly: a secure, fair and scalable open blockchain. In: 2021 IEEE Symposium on Security and Privacy (SP), pp. 466–483. IEEE (2021)

    Google Scholar 

  22. Diablo blockchain benchmark suite (2020). https://diablobench.github.io/. Accessed 14 Aug 2023

  23. Diablo blockchain benchmark suite (2020). https://diablobench.github.io/blockchain-howto. Accessed 14 Aug 2023

  24. 20 Diem consensus configurations (2020). https://github.com/diem/diem/blob/latest/config/src/config/consensus_config.rs. Accessed 14 Aug 2023

  25. Diem mempool configurations (2020). https://github.com/diem/diem/blob/latest/config/src/config/mempool_config.rs. Accessed 14 Aug 2023

  26. Diem validator nodes (2020). https://developers.diem.com/docs/basics/basics-validator-nodes. Accessed 14 Aug 2023

  27. Diem white paper (2020). https://developers.diem.com/docs/technical-papers/the-diem-blockchain-paper/. Accessed 14 Aug 2023

  28. Dinh, T.T.A., Wang, J., Chen, G., Liu, R., Ooi, B.C., Tan, K.L.: Blockbench: a framework for analyzing private blockchains. In: Proceedings of the 2017 ACM International Conference on Management of Data, pp. 1085–1100 (2017)

    Google Scholar 

  29. Gilad, Y., Hemo, R., Micali, S., Vlachos, G., Zeldovich, N.: Algorand: Scaling byzantine agreements for cryptocurrencies. In: Proceedings of the 26th Symposium on Operating Systems Principles, SOSP 2017, pp. 51–68. Association for Computing Machinery, New York (2017).https://doi.org/10.1145/3132747.3132757

  30. Go ethereum (2020). https://geth.ethereum.org/docs/interface/command-line-options. Accessed 14 Aug 2023

  31. Goquorum configuration file (2020). https://consensys.net/docs/goquorum/en/latest/configure-and-manage/configure/use-configuration-file/. Accessed 14 Aug 2023

  32. Goquorum qlight (2020). https://consensys.net/docs/goquorum/en/latest/concepts/qlight-node/. Accessed 14 Aug 2023

  33. Gramoli, V., Guerraoui, R., Lebedev, A., Natoli, C., Voron, G.: Diablo: a benchmark suite for blockchains. In: 18th ACM European Conference on Computer Systems (EuroSys) (2023)

    Google Scholar 

  34. Gray, J.: Benchmark Handbook: for Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc., Burlington (1992)

    Google Scholar 

  35. Greenspan, G., et al.: Multichain private blockchain-white paper, pp. 57–60 (2015). http://www.multichain.com/download/MultiChain-White-Paper.pdf

  36. Gromit blockchain benchmarking tool (2020). https://github.com/grimadas/gromit. Accessed 14 Aug 2023

  37. Hearn, M., Brown, R.G.: Corda: A distributed ledger. Corda Technical White Paper 2016 (2016)

    Google Scholar 

  38. How fabric networks are structured (2020). https://hyperledger-fabric.readthedocs.io/en/latest/network/network.html. Accessed 14 Aug 2023

  39. Hyperledger besu (2020). https://www.hyperledger.org/use/besu. Accessed 14 Aug 2023

  40. https://hyperledger.github.io/caliper/ (2020), [Online; accessed 14-August-2023]

  41. Hyperledger caliper benchmarks (2020). https://github.com/hyperledger/caliper-benchmarks. Accessed 14 Aug 2023

  42. Hyperledger foundation case studies (2023). https://www.hyperledger.org/learn/case-studies. Accessed 14 Aug 2023

  43. Hyperledgerlab ii (2020). https://github.com/MSRG/HyperLedgerLab-2.0. Accessed 14 Aug 2023

  44. Klenik, A., Kocsis, I.: Porting a benchmark with a classic workload to blockchain: TPC-c on hyperledger fabric. In: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, pp. 290–298 (2022)

    Google Scholar 

  45. Kokoris-Kogias, E., Jovanovic, P., Gailly, N., Khoffi, I., Gasser, L., Ford, B.: Enhancing bitcoin security and performance with strong consistency via collective signing. In: Proceedings of the 25th USENIX Conference on Security Symposium, SEC 2016, pp. 279–296, USENIX Association, USA (2016)

    Google Scholar 

  46. Li, M., et al.: Auto-tuning with reinforcement learning for permissioned blockchain systems. Proc. VLDB Endow. 16(5), 1000–1012 (2023). https://doi.org/10.14778/3579075.3579076

  47. Liu, M., Yu, F.R., Teng, Y., Leung, V.C.M., Song, M.: Performance optimization for blockchain-enabled industrial internet of things (iiot) systems: a deep reinforcement learning approach. IEEE Trans. Ind. Inf. 15(6), 3559–3570 (2019). https://doi.org/10.1109/TII.2019.2897805

    Article  Google Scholar 

  48. Lokhava, M., et al.: Fast and secure global payments with stellar. In: Proceedings of the 27th ACM Symposium on Operating Systems Principles, SOSP 2019, pp. 80–96. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3341301.3359636

  49. Mazzoni, M., Corradi, A., Di Nicola, V.: Performance evaluation of permissioned blockchains for financial applications: the consensys quorum case study. Blockchain: Res. Appl. 3(1), 100026 (2022). https://doi.org/10.1016/j.bcra.2021.100026

  50. Morgan, J.: Quorum Whitepaper. JP Morgan Chase, New York (2016)

    Google Scholar 

  51. Multichain configurations (2020). https://www.multichain.com/developers/blockchain-parameters/. Accessed 14 Aug 2023

  52. Multichain data streams (2020). https://www.multichain.com/developers/data-streams/. Accessed 14 August 2023

  53. Multichain performance optimization (2020). https://www.multichain.com/developers/ performance-optimization/. Accessed 14 Aug 2023

  54. Multichain product partners (2023). https://www.multichain.com/product-partners/. Accessed 14 Aug 2023

  55. Multichain white paper (2020). https://www.multichain.com/download/MultiChain-White-Paper.pdf. Accessed 14 Aug 2023

  56. Nasirifard, P., Mayer, R., Jacobsen, H.A.: Fabriccrdt: a conflict-free replicated datatypes approach to permissioned blockchains. In: Proceedings of the 20th International Middleware Conference, Middleware 2019, pp. 110–122. Association for Computing Machinery, New York (2019).https://doi.org/10.1145/3361525.3361540

  57. Nasrulin, B., De Vos, M., Ishmaev, G., Pouwelse, J.: Gromit: benchmarking the performance and scalability of blockchain systems. In: 2022 IEEE International Conference on Decentralized Applications and Infrastructures (DAPPS), pp. 56–63. IEEE (2022)

    Google Scholar 

  58. Parity: Blockchain infrastructure for the decentralised web (2020). https://www.parity.io/. Accessed 14 Aug 2023

  59. Quorum blockchain in action (2023). https://consensys.net/quorum/enterprise/. Accessed 14 Aug 2023

  60. Quorum white paper (2020). https://github.com/ConsenSys/quorum/blob/master/docs/Quorum%20Whitepaper%20v0.2.pdf. Accessed 14 Aug 2023

  61. Raasveldt, M., Holanda, P., Gubner, T., Mühleisen, H.: Fair benchmarking considered difficult: common pitfalls in database performance testing. In: Proceedings of the Workshop on Testing Database Systems, DBTest 2018. Association for Computing Machinery, New York (2018).https://doi.org/10.1145/3209950.3209955

  62. Rauchs, M., Blandin, A., Bear, K., McKeon, S.B.: 2nd global enterprise blockchain benchmarking study. SSRN 3461765 (2019)

    Google Scholar 

  63. Rocket, T., Yin, M., Sekniqi, K., van Renesse, R., Sirer, E.G.: Scalable and probabilistic leaderless BFT consensus through metastability (2020)

    Google Scholar 

  64. Ruan, P., et al.: Blockchains vs. distributed databases: dichotomy and fusion. In: Proceedings of the 2021 International Conference on Management of Data, SIGMOD 2021, pp. 1504–1517. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3448016.3452789

  65. Saingre, D., Ledoux, T., Menaud, J.M.: Bctmark: a framework for benchmarking blockchain technologies. In: 2020 IEEE/ACS 17th International Conference on Computer Systems and Applications (AICCSA), pp. 1–8. IEEE (2020)

    Google Scholar 

  66. Schuh, F., Larimer, D.: Bitshares 2.0: general overview (2017)

    Google Scholar 

  67. Smaalders, B.: Performance anti-patterns: want your apps to run faster? here’s what not to do. Queue 4(1) (2006). https://doi.org/10.1145/1117389.1117403

  68. Solana: a new architecture for a high performance blockchain v0.8.13 (2020). https://solana.com/solana-whitepaper.pdf. Accessed 14 Aug 2023

  69. Tennakoon, D., Gramoli, V.: Smart red belly blockchain: enhanced transaction management for decentralized applications. arXiv preprint arXiv:2207.05971 (2022)

  70. Thakkar, P., Nathan, S., Viswanathan, B.: Performance benchmarking and optimizing hyperledger fabric blockchain platform. In: 2018 IEEE 26th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 264–276 (2018). https://doi.org/10.1109/MASCOTS.2018.00034

  71. Tran, N.K., Babar, M.A., Walters, A.: A framework for automating deployment and evaluation of blockchain networks. J. Netw. Comput. Appl. 206, 103460 (2022). https://doi.org/10.1016/j.jnca.2022.103460

    Article  Google Scholar 

  72. Types of transactions (2020). https://developers.diem.com/docs/transactions/txns-types/. Accessed 14 Aug 2023

  73. Valenta, M., Sandner, P.G.: Comparison of ethereum, hyperledger fabric and corda (2017)

    Google Scholar 

  74. Wood, D.D.: Ethereum: a secure decentralised generalised transaction ledger (2014)

    Google Scholar 

  75. Writing connectors (2020). https://hyperledger.github.io/caliper/v0.5.0/writing-connectors/. Accessed 14 Aug 2023

  76. Zhang, J., et al.: Performance analysis of the libra blockchain: an experimental study. In: 2019 2nd International Conference on Hot Information-Centric Networking (HotICN), pp. 77–83. IEEE (2019)

    Google Scholar 

Download references

Acknowledgement

This work is funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - 392214008, and by the Bavarian Cooperative Research Program of the Free State of Bavaria - DIK-2002-0013//DIK0114/02.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jeeta Ann Chacko .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chacko, J.A., Mayer, R., Fekete, A., Gramoli, V., Jacobsen, HA. (2024). A Comprehensive Study on Benchmarking Permissioned Blockchains. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2023. Lecture Notes in Computer Science, vol 14247. Springer, Cham. https://doi.org/10.1007/978-3-031-68031-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-68031-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-68030-4

  • Online ISBN: 978-3-031-68031-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics