Skip to main content
Springer Nature Link
Log in

Consolidation of Ground Truth Sets for Weakness Detection in Smart Contracts

  • Conference paper
  • First Online:
Financial Cryptography and Data Security. FC 2023 International Workshops (FC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13953))

Included in the following conference series:

Abstract

Smart contracts are small programs on the blockchain that often handle valuable assets. Vulnerabilities in smart contracts can be costly, as time has shown over and over again. Countermeasures are high in demand and include best practice recommendations as well as tools supporting development, program verification, and post-deployment analysis. Many tools focus on detecting the absence or presence of a subset of the known vulnerabilities, delivering results of varying quality. Most comparative tool evaluations resort to selecting a handful of tools and testing them against each other. In the best case, the evaluation is based on a smallish ground truth. For Ethereum, there are commendable efforts by several author groups to manually classify contracts. However, a comprehensive ground truth is still lacking.

In this work, we construct a ground truth based on publicly available benchmark sets for Ethereum smart contracts with manually checked ground truth data. We develop a method to unify these sets. Additionally, we devise strategies for matching entries that pertain to the same contract, such that we can determine overlaps and disagreements between the sets and consolidate the disagreements. Finally, we assess the quality of the included ground truth sets. Our work reduces inconsistencies, redundancies, and incompleteness while increasing the number of data points and their heterogeneity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://swcregistry.io.

  2. 2.

    https://dasp.co/.

  3. 3.

    The distinction between crafted and wild sets is not strict. Crafted sets may contain some contracts from public chains in modified or unmodified form.

  4. 4.

    Addresses by themselves are not sufficient to identify a contract. Apart from information about the chain, we also need the deployment time if the contract or an ancestor is the result of a create2 operation. However, as the data in the repositories mostly predates the introduction of this operation, we encountered no contract of this type. Hence, for our purposes knowing the address and chain is sufficient. We use the block numbers of deployments only for analyzing changes over time.

  5. 5.

    An important opcode change occurred at block 7.28 M with the introduction of the shift operations, which now appear in most contracts, and create2. At block 9.069 M, selfbalance and chainid got introduced, and at block 12.9 M basefee.

  6. 6.

    https://swcregistry.io.

References

  1. di Angelo, M., Salzer, G.: Consolidation of ground truth sets for weakness detection in smart contracts. arXiv preprint 2304.11624 (2023). https://doi.org/10.48550/arXiv.2304.11624

  2. Bosu, M.F., MacDonell, S.G.: A taxonomy of data quality challenges in empirical software engineering. In: 2013 22nd Australian Software Engineering Conference, pp. 97–106. IEEE (2013). https://doi.org/10.1109/ASWEC.2013.21

  3. Chen, J., Xia, X., Lo, D., Grundy, J., Luo, X., Chen, T.: Defining smart contract defects on ethereum. IEEE Trans. Softw. Eng. (2020). https://doi.org/10.1109/TSE.2020.2989002

  4. Durieux, T., Ferreira, J.F., Abreu, R., Cruz, P.: Empirical review of automated analysis tools on 47,587 Ethereum smart contracts. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 530–541. ACM, New York, NY, USA (2020). https://doi.org/10.1145/3377811.3380364

  5. Ferreira, J.F., Cruz, P., Durieux, T., Abreu, R.: SmartBugs: a framework to analyze solidity smart contracts. In: 35th IEEE/ACM International Conference on Automated Software Engineering (ASE 2020), pp. 1349–1352. ACM (2020). https://doi.org/10.1145/3324884.3415298

  6. Ghaleb, A., Pattabiraman, K.: How effective are smart contract analysis tools? evaluating smart contract static analysis tools using bug injection. In: Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 415–427. ISSTA 2020, Association for Computing Machinery (2020). https://doi.org/10.1145/3395363.3397385

  7. Grech, N., Brent, L., Scholz, B., Smaragdakis, Y.: Gigahorse: thorough, declarative decompilation of smart contracts. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 1176–1186. IEEE (2019). https://doi.org/10.1109/ICSE.2019.00120

  8. Jiang, B., Liu, Y., Chan, W.K.: Contractfuzzer: fuzzing smart contracts for vulnerability detection. In: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pp. 259–269. ASE 2018, Association for Computing Machinery (2018). https://doi.org/10.1145/3238147.3238177

  9. Kalra, S., Goel, S., Dhawan, M., Sharma, S.: ZEUS: analyzing safety of smart contracts. In: NDSS Symposion. NDSS, Internet Society (2018). https://doi.org/10.14722/ndss.2018.23082

  10. Kolluri, A., Nikolic, I., Sergey, I., Hobor, A., Saxena, P.: Exploiting the laws of order in smart contracts. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 363–373. ISSTA 2019, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3293882.3330560

  11. Rameder, H., Angelo, M.D., Salzer, G.: Review of automated vulnerability analysis of smart contracts on ethereum. Front. Blockchain - Smart Contracts (2022). https://doi.org/10.3389/fbloc.2022.814977

    Article  Google Scholar 

  12. Ren, M., et al.: Empirical evaluation of smart contract testing: what is the best choice? In: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 566–579 (2021). https://doi.org/10.1145/3460319.3464837

  13. Schneidewind, C., Grishchenko, I., Scherer, M., Maffei, M.: EThor: practical and provably sound static analysis of ethereum smart contracts. In: Proceedings of the ACM Conference on Computer and Communications Security, pp. 621–640 (2020). https://doi.org/10.1145/3372297.3417250

  14. Soud, M., Qasse, I., Liebel, G., Hamdaqa, M.: Automesc: automatic framework for mining and classifying ethereum smart contract vulnerabilities and their fixes. arXiv preprint arXiv:2212.10660 (2022). https://doi.org/10.48550/arXiv.2212.10660

  15. Wang, S., Zhang, C., Su, Z.: Detecting nondeterministic payment bugs in ethereum smart contracts. Proc. ACM Program. Lang. (PACMPL) 3(189), 1–29 (2019). https://doi.org/10.1145/3360615

    Article  Google Scholar 

  16. Xue, Y., et al.: Doublade: unknown vulnerability detection in smart contracts via abstract signature matching and refined detection rules. arXiv preprint arXiv:1912.04466 (2019). https://doi.org/10.48550/arXiv.1912.04466

  17. Yashavant, C.S., Kumar, S., Karkare, A.: Scrawld: a dataset of real world ethereum smart contracts labelled with vulnerabilities. arXiv preprint arXiv:2202.11409 (2022). https://doi.org/10.48550/arXiv.2202.11409

  18. Zhang, P., Xiao, F., Luo, X.: A framework and dataset for bugs in ethereum smart contracts. In: IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 139–150. ICSME 2020, IEEE (2020). https://doi.org/10.1109/icsme46990.2020.00023

  19. Zhou, S., Yang, Z., Xiang, J., Cao, Y., Yang, Z., Zhang, Y.: An ever-evolving game: evaluation of real-world attacks and defenses in ethereum ecosystem. In: 29th USENIX Security Symposium (USENIX Security 20), pp. 2793–2810. USENIX Security 2020, USENIX Association (2020). https://www.usenix.org/conference/usenixsecurity20/presentation/zhou-shunfan

Download references

Author information

Authors and Affiliations

  1. TU Wien, Vienna, Austria

    Monika di Angelo & Gernot Salzer

Authors
  1. Monika di Angelo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gernot Salzer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Monika di Angelo .

Editor information

Editors and Affiliations

  1. Western University, London, ON, Canada

    Aleksander Essex

  2. Georgetown University, Washington, DC, USA

    Shin'ichiro Matsuo

  3. IT University of Copenhagen, Copenhagen, Denmark

    Oksana Kulyk

  4. Imperial College London, London, UK

    Lewis Gudgeon

  5. Imperial College London, London, UK

    Ariah Klages-Mundt

  6. Imperial College London, London, UK

    Daniel Perez

  7. Imperial College London, London, UK

    Sam Werner

  8. Stirling University, Stirling, UK

    Andrea Bracciali

  9. University College London, London, UK

    Geoff Goodell

Rights and permissions

Reprints and permissions

Copyright information

© 2024 International Financial Cryptography Association

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

di Angelo, M., Salzer, G. (2024). Consolidation of Ground Truth Sets for Weakness Detection in Smart Contracts. In: Essex, A., et al. Financial Cryptography and Data Security. FC 2023 International Workshops. FC 2023. Lecture Notes in Computer Science, vol 13953. Springer, Cham. https://doi.org/10.1007/978-3-031-48806-1_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-48806-1_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-48805-4

  • Online ISBN: 978-3-031-48806-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation