skip to main content
10.1145/3609437.3609455acmotherconferencesArticle/Chapter ViewAbstractPublication PagesinternetwareConference Proceedingsconference-collections
research-article

Can Neural Networks Help Smart Contract Testing? An Empirical Study

Published: 05 October 2023 Publication History

Abstract

Smart contracts are one of the most successful applications of blockchain technology. In order to guarantee the security of smart contracts, researchers have successively introduced various testing methodologies, including static analysis, symbolic execution, and fuzzing, which contribute to a more rigorous and precise evaluation of smart contract vulnerabilities. Deep learning techniques have been widely applied in traditional software vulnerability detection, while the opposite is true in the field of smart contract testing. Consequently, we anticipate that deep learning can be similarly applied to enhance traditional smart contract vulnerability detection tools. However, there is a lack of empirical study on the performance of deep learning applied to smart contract testing. In order to explore how deep neural networks can help with testing tools on smart contracts, we construct a test framework based on SMARTEST. We manage to train deep learning language models using various neural networks including Transformer, GRU, RNN and test the symbolic execution tool SMARTEST framework with the application of these models on the CVE dataset. Upon analyzing the experimental results, we find that deep neural networks did not surpass traditional language models in enhancing smart contract testing. In terms of accuracy, the SMARTEST tool, which utilizes a statistical 3-gram language model, succeeded in detecting the greatest number of vulnerabilities. Specifically, the 3-gram model was able to identify 69.8% of vulnerabilities in the benchmark set within the first 5 seconds. Based on our experimental findings and thorough analysis, we outline the challenges faced in DNN-assisted smart contract testing and suggest potential directions for improvement.

References

[1]
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. J. Mach. Learn. Res. 3, null (mar 2003), 1137–1155.
[2]
J Chen, X Xia, David Lo, J Grundy, X Luo, and T Chen. 2020. Defining smart contract defects on ethereum. TSE 48, 1 (2020), 327–345.
[3]
Jiachi Chen, Xin Xia, David Lo, John Grundy, Xiapu Luo, and Ting Chen. 2021. Defectchecker: Automated smart contract defect detection by analyzing evm bytecode. TSE 48, 7 (2021), 2189–2207.
[4]
K Cho, B van Merriënboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, and Y Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In EMNLP 14. ACL, Doha, Qatar, 1724–1734.
[5]
J Choi, D Kim, S Kim, G Grieco, A Groce, and S K Cha. 2022. SMARTIAN: Enhancing Smart Contract Fuzzing with Static and Dynamic Data-Flow Analyses. In ASE 21(ASE ’21). IEEE Press, Melbourne, Australia, 227–239.
[6]
ConsenSys. 2018. Mythril: a security analysis tool for EVM bytecode.https://github.com/ConsenSys/mythril-classic
[7]
Phil Daian. 2016. Analysis of the DAO exploit.https://hackingdistributed.com/2016/06/18/analysis-of-the-dao-exploit/
[8]
T Durieux, J F. Ferreira, R Abreu, and P Cruz. 2020. Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. In ICSE (Seoul, South Korea) (ICSE ’20). ACM, New York, NY, USA, 530–541.
[9]
Torres C F, Schütte J, and R State. 2018. Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts. In ACSAC ’18 (San Juan, PR, USA). ACM, New York, NY, USA, 664–676.
[10]
J Feist, G Greico, and A Groce. 2019. Slither: A Static Analysis Framework for Smart Contracts. In WETSEB 19(WETSEB ’19). IEEE Press, Montreal, Quebec, Canada, 8–15.
[11]
Z Feng, D Guo, D Tang, N Duan, X Feng, M Gong, L Shou, B Qin, T Liu, D Jiang, and M Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In EMNLP 2020. ACL, Online, 1536–1547.
[12]
João F. Ferreira, P Cruz, T Durieux, and R Abreu. 2021. SmartBugs: A Framework to Analyze Solidity Smart Contracts. In ASE (Virtual Event, Australia) (ASE ’20). ACM, New York, NY, USA, 1349–1352.
[13]
N Grech, M Kong, A Jurisevic, L Brent, B Scholz, and Y Smaragdakis. 2020. MadMax: Analyzing the out-of-Gas World of Smart Contracts. Commun. ACM 63, 10 (sep 2020), 87–95.
[14]
J He, M Balunović, N Ambroladze, P Tsankov, and M Vechev. 2019. Learning to Fuzz from Symbolic Execution with Application to Smart Contracts. In CCS 19 (London, United Kingdom) (CCS ’19). ACM, New York, NY, USA, 531–548.
[15]
B Jiang, Y Liu, and W. K. Chan. 2018. ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection. In ASE 18 (Montpellier, France) (ASE ’18). ACM, New York, NY, USA, 259–269.
[16]
Johannes K and Christian R. 2018. teEther: Gnawing at Ethereum to Automatically Exploit Smart Contracts. In USENIX Security 18. USENIX Association, Baltimore, MD, 1317–1333.
[17]
Sukrit Kalra, Seep Goel, Mohan Dhawan, and Subodh Sharma. 2018. Zeus: analyzing safety of smart contracts. In Ndss. NDSS ’18, San Diego, California, USA, 1–12.
[18]
L Luu, D Chu, H Olickel, P Saxena, and A Hobor. 2016. Making Smart Contracts Smarter. In CCS 16 (Vienna, Austria) (CCS ’16). ACM, New York, NY, USA, 254–269.
[19]
M Mossberg, F Manzano, E Hennenfent, A Groce, G Grieco, J Feist, T Brunson, and A Dinaburg. 2019. Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts. In ASE ’19. IEEE, San Diego, CA, USA, 1186–1189.
[20]
T D Nguyen, Lon H Pham, J Sun, Y Lin, and Quang T M. 2020. SFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts. In ICSE (Seoul, South Korea) (ICSE ’20). ACM, New York, NY, USA, 778–788.
[21]
I Nikolić, A Kolluri, I Sergey, P Saxena, and A Hobor. 2018. Finding The Greedy, Prodigal, and Suicidal Contracts at Scale. In ACSAC ’18 (San Juan, PR, USA). ACM, New York, NY, USA, 653–663.
[22]
Tsankov P, Dan A, Drachsler-Cohen D, Gervais A, Bünzli F, and Vechev M. 2018. Securify: Practical Security Analysis of Smart Contracts. In CCS’ 18 (Toronto, Canada) (CCS ’18). ACM, New York, NY, USA, 67–82.
[23]
Reza M. Parizi, A Dehghantanha, K R Choo, and A Singh. 2018. Empirical Vulnerability Analysis of Automated Smart Contracts Security Testing on Blockchains. In CASCON (Markham, Ontario, Canada) (CASCON ’18). IBM Corp., USA, 103–113.
[24]
A Permenev, imitar Dimitrov, P Tsankov, D Drachsler-Cohen, and M Vechev. 2020. VerX: Safety Verification of Smart Contracts. In SP ’20. IEEE, San Francisco, CA, USA, 1661–1677.
[25]
Peng Qian, Zhenguang Liu, Qinming He, Roger Zimmermann, and Xun Wang. 2020. Towards automated reentrancy detection for smart contracts based on sequential models. IEEE Access 8 (2020), 19685–19695.
[26]
M Ren, Z Yin, F Ma, Zhenyan Xu, Y Jiang, C Sun, H Li, and Y Cai. 2021. Empirical Evaluation of Smart Contract Testing: What is the Best Choice?. In ISSTA 21 (Virtual, Denmark) (ISSTA 2021). ACM, New York, NY, USA, 566–579.
[27]
Sunbeom S, Seongjoon H, and Hakjoo O. 2021. SmarTest: Effectively Hunting Vulnerable Transaction Sequences in Smart Contracts through Language Model-Guided Symbolic Execution. In USENIX Security 21. USENIX Association, none, 1361–1378.
[28]
Sunbeom So, Myungho Lee, Jisu Park, Heejo Lee, and Hakjoo Oh. 2020. VERISMART: A Highly Precise Safety Verifier for Ethereum Smart Contracts. In SP ’20. IEEE, San Francisco, CA, USA, 1678–1694.
[29]
Jianzhong Su, Hong-Ning Dai, Lingjun Zhao, Zibin Zheng, and Xiapu Luo. 2023. Effectively Generating Vulnerable Transaction Sequences in Smart Contracts with Reinforcement Learning-Guided Fuzzing. In ASE ’22 (Rochester, MI, USA) (ASE ’22). ACM, New York, NY, USA, Article 36, 12 pages.
[30]
Wesley Joon-Wie Tann, Xing Jie Han, Sourav Sen Gupta, and Yew-Soon Ong. 2018. Towards safer smart contracts: A sequence learning approach to detecting security threats. arXiv preprint arXiv:1811.06632 1, 1 (2018).
[31]
S Tikhomirov, E Voskresenskaya, I Ivanitskiy, R Takhaviev, E Marchenko, and Y Alexandrov. 2018. SmartCheck: Static Analysis of Ethereum Smart Contracts. In WETSEB 18 (Gothenburg, Sweden) (WETSEB ’18). ACM, New York, NY, USA, 9–16.
[32]
Wei Wang, Jingjing Song, Guangquan Xu, Yidong Li, Hao Wang, and Chunhua Su. 2020. Contractward: Automated vulnerability detection models for ethereum smart contracts. IEEE Transactions on Network Science and Engineering 8, 2 (2020), 1133–1144.
[33]
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In EMNLP 21. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8696–8708.
[34]
A Waswani, N Shazeer, N Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and I Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
[35]
Gavin Wood 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper 151, 2014 (2014), 1–32.
[36]
Y Zhuang, Z Liu, P Qian, Q Liu, X Wang, and Q He. 2020. Smart Contract Vulnerability Detection using Graph Neural Network. In IJCAI-20, Christian Bessiere (Ed.). IJCAI 20, Yokohama, Yokohama, Japan, 3283–3290. Main track.

Cited By

View all
  • (2025)A comprehensive survey of smart contracts vulnerability detection tools: Techniques and methodologiesJournal of Network and Computer Applications10.1016/j.jnca.2025.104142(104142)Online publication date: Feb-2025

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
Internetware '23: Proceedings of the 14th Asia-Pacific Symposium on Internetware
August 2023
332 pages
ISBN:9798400708947
DOI:10.1145/3609437
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 October 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. neural networks
  2. smart contract
  3. vulnerabilities detection

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

Internetware 2023

Acceptance Rates

Overall Acceptance Rate 55 of 111 submissions, 50%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)70
  • Downloads (Last 6 weeks)7
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)A comprehensive survey of smart contracts vulnerability detection tools: Techniques and methodologiesJournal of Network and Computer Applications10.1016/j.jnca.2025.104142(104142)Online publication date: Feb-2025

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media