Skip to main content
Springer Nature Link
Log in

Conformance Testing of Formal Semantics Using Grammar-Based Fuzzing

  • Conference paper
  • First Online:
Tests and Proofs (TAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13361))

Included in the following conference series:

Abstract

A common problem in verification is to ensure that the formal specification models the real-world system, i.e., the implementation, faithfully. Testing is a technique that can help to bridge the gap between a formal specification and its implementation.

Fuzzing in general and grammar-based fuzzing in particular are successfully used for finding bugs in implementations. Traditional fuzzing applications rely on an implicit test specification that informally can be described as “the program under test does not crash”.

In this paper, we present an approach using grammar-based fuzzing to ensure the conformance of a formal specification, namely the formal semantics of the Solidity Programming language, to a real-world implementation. For this, we derive an executable test-oracle from the formal semantics of Solidity in Isabelle/HOL. The derived test oracle is used during the fuzzing of the implementation to validate that the formal semantics and the implementation are in conformance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Availability

Our formalization, the test framework, and the evaluation results are available under BSD license (SPDX-License-Identifier: BSD-2-Clause) [33].

Notes

  1. 1.

    This is the currently supported default version of the Truffle test framework.

References

  1. Solidity. https://github.com/ethereum/solidity. Accessed 29 Mar 2022

  2. Ahrendt, W., Bubel, R.: Functional verification of smart contracts via strong data integrity. In: Margaria, T., Steffen, B. (eds.) ISoLA 2020. LNCS, vol. 12478, pp. 9–24. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61467-6_2

    Chapter  Google Scholar 

  3. Feo-Arenis, S., Westphal, B., Dietsch, D., Muñiz, M., Andisha, S., Podelski, A.: Ready for testing: ensuring conformance to industrial standards through formal verification. Formal Aspects Comput. 28(3), 499–527 (2016). https://doi.org/10.1007/s00165-016-0365-3

    Article  Google Scholar 

  4. Armstrong, J.: Programming Erlang: Software for a Concurrent World. Pragmatic Bookshelf (2013)

    Google Scholar 

  5. Bartoletti, M., Galletta, L., Murgia, M.: A Minimal core calculus for solidity contracts. In: Pérez-Solà, C., Navarro-Arribas, G., Biryukov, A., Garcia-Alfaro, J. (eds.) DPM/CBT -2019. LNCS, vol. 11737, pp. 233–243. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-31500-9_15

    Chapter  Google Scholar 

  6. Bereczky, P., Horpácsi, D., Kőszegi, J., Szeier, S., Thompson, S.: Validating formal semantics by property-based cross-testing. In: IFL 2020: Proceedings of the 32nd Symposium on Implementation and Application of Functional Languages, IFL 2020, pp. 150–161. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3462172.3462200

  7. Blazy, S., Leroy, X.: Mechanized semantics for the Clight subset of the C language. J. Autom. Reason. 43(3), 263–288 (2009)

    Article  Google Scholar 

  8. Brucker, A.D., Herzberg, M.: Formalizing (Web) standards. In: Dubois, C., Wolff, B. (eds.) TAP 2018. LNCS, vol. 10889, pp. 159–166. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-92994-1_9

    Chapter  Google Scholar 

  9. Brucker, A.D., Wolff, B.: On theorem prover-based testing. Formal Aspects Comput. 25(5), 683–721 (2013). https://doi.org/10.1007/s00165-012-0222-y

    Article  Google Scholar 

  10. Bulwahn, L.: The new quickcheck for Isabelle. In: Hawblitzel, C., Miller, D. (eds.) CPP 2012. LNCS, vol. 7679, pp. 92–108. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35308-6_10

    Chapter  Google Scholar 

  11. Chen, J., et al.: A survey of compiler testing. ACM Comput. Surv. 53(1) (2020). https://doi.org/10.1145/3363562

  12. Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: The Fifth ACM SIGPLAN International Conference on Functional Programming, pp. 268–279. ACM Press (2000). https://doi.org/10.1145/351240.351266

  13. ConsenSys Software Inc.: Ganache. https://www.trufflesuite.com/docs/ganache/. Accessed 1 May 2021

  14. ConsenSys Software Inc.: Truffle. https://www.trufflesuite.com/truffle. Accessed 1 May 2021

  15. Crafa, S., Di Pirro, M., Zucca, E.: Is solidity solid enough? In: Bracciali, A., Clark, J., Pintore, F., Rønne, P.B., Sala, M. (eds.) FC 2019. LNCS, vol. 11599, pp. 138–153. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-43725-1_11

    Chapter  Google Scholar 

  16. Duncan, A.G., Hutchison, J.S.: Using attributed grammars to test designs and implementations. In: Proceedings of the 5th International Conference on Software Engineering, ICSE 1981, pp. 170–178. IEEE Press (1981)

    Google Scholar 

  17. Felderer, M., Büchler, M., Johns, M., Brucker, A.D., Breu, R., Pretschner, A.: Security testing: a survey. Adv. Comput. 101, 1–51 (2016). https://doi.org/10.1016/bs.adcom.2015.11.003

    Article  Google Scholar 

  18. Filaretti, D., Maffeis, S.: An executable formal semantics of PHP. In: Jones, R. (ed.) ECOOP 2014. LNCS, vol. 8586, pp. 567–592. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44202-9_23

    Chapter  Google Scholar 

  19. Gill, A., Runciman, C.: Haskell program coverage. In: Haskell Workshop, Haskell 2007, pp. 1–12. ACM (2007). https://doi.org/10.1145/1291201.1291203

  20. Godefroid, P., Kiezun, A., Levin, M.Y.: Grammar-based whitebox fuzzing. SIGPLAN Not. 43(6), 206–215 (2008). https://doi.org/10.1145/1379022.1375607

  21. Guagliardo, P., Libkin, L.: A formal semantics of SQL queries, its validation, and applications. Proc. VLDB Endow. 11(1), 27–39 (2017). https://doi.org/10.14778/3151113.3151116

  22. Hanford, K.V.: Automatic generation of test cases. IBM Syst. J. 9(4), 242–257 (1970)

    Article  Google Scholar 

  23. Hodován, R., Kiss, A., Gyimóthy, T.: Grammarinator: a grammar-based open source fuzzer. In: Automating TEST Case Design, A-TEST 2018, pp. 45–48. ACM (2018). https://doi.org/10.1145/3278186.3278193

  24. Holler, C., Herzig, K., Zeller, A.: Fuzzing with code fragments. In: 21st USENIX Security Symposium (USENIX Security 12), pp. 445–458. USENIX Association, Bellevue, August 2012

    Google Scholar 

  25. Horl, J., Aichernig, B.K.: Validating voice communication requirements using lightweight formal methods. IEEE Softw. 17(3), 21–27 (2000). https://doi.org/10.1109/52.896246

    Article  Google Scholar 

  26. Jiao, J., Kan, S., Lin, S.W., Sanan, D., Liu, Y., Sun, J.: Semantic understanding of smart contracts: executable operational semantics of Solidity. In: SP, pp. 1695–1712. IEEE (2020)

    Google Scholar 

  27. Kappelmann, K., Bulwahn, L., Willenbrink, S.: Speccheck - specification-based testing for Isabelle/ML. Arch. Formal Proofs (2021). https://isa-afp.org/entries/SpecCheck.html. Formal Proof Development

  28. Kifetew, F.M., Tiella, R., Tonella, P.: Combining stochastic grammars and genetic programming for coverage testing at the system level. In: Le Goues, C., Yoo, S. (eds.) SSBSE 2014. LNCS, vol. 8636, pp. 138–152. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09940-8_10

    Chapter  Google Scholar 

  29. Kristoffersen, F., Walter, T.: TTCN: towards a formal semantics and validation of test suites. Comput. Netw. ISDN Syst. 29(1), 15–47 (1996). https://doi.org/10.1016/S0169-7552(96)00016-5

    Article  Google Scholar 

  30. Majumdar, R., Xu, R.G.: Directed test generation using symbolic grammars. In: The 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering: Companion Papers, pp. 553–556. Association for Computing Machinery, New York (2007). https://doi.org/10.1145/1295014.1295039

  31. Marlow, S.: Haskell 2010 language report (2010). https://www.haskell.org/onlinereport/haskell2010/

  32. Marmsoler, D., Brucker, A.D.: A denotational semantics of solidity in Isabelle/HOL. In: Calinescu, R., Păsăreanu, C.S. (eds.) SEFM 2021. LNCS, vol. 13085, pp. 403–422. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-92124-8_23https://www.brucker.ch/bibliography/abstract/marmsoler.ea-solidity-semantics-2021

    Chapter  Google Scholar 

  33. Marmsoler, D., Brucker, A.D.: A denotational semantics of Solidity in Isabelle/HOL: implementation and test data (2021). https://doi.org/10.5281/zenodo.5573225

  34. Mavridou, A., Laszka, A., Stachtiari, E., Dubey, A.: VeriSolid: correct-by-design smart contracts for Ethereum. In: Goldberg, I., Moore, T. (eds.) FC 2019. LNCS, vol. 11598, pp. 446–465. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32101-7_27

    Chapter  Google Scholar 

  35. Nipkow, T., Wenzel, M., Paulson, L.C. (eds.): Isabelle/HOL. LNCS, vol. 2283. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45949-9

    Book  Google Scholar 

  36. Online: Solidity documentation. https://docs.soliditylang.org/en/v0.5.16/. Accessed 1 May 2021

  37. Politz, J.G., Carroll, M.J., Lerner, B.S., Pombrio, J., Krishnamurthi, S.: A tested semantics for getters, setters, and eval in JavaScript. In: Proceedings of the 8th Symposium on Dynamic Languages, DLS 2012, pp. 1–16. Association for Computing Machinery, New York (2012). https://doi.org/10.1145/2384577.2384579

  38. Purdom, P.: A sentence generator for testing parsers. BIT Numer. Math. 12(3), 366–375 (1972)

    Article  Google Scholar 

  39. Rouş, G., Şerbănută, T.F.: An overview of the K semantic framework. J. Log. Algebraic Program. 79(6), 397–434 (2010). https://doi.org/10.1016/j.jlap.2010.03.012. Membrane computing and programming

  40. The Coq development team: The Coq proof assistant reference manual. LogiCal Project (2004). Version 8.0

    Google Scholar 

  41. Wood, G.: Ethereum: a secure decentralised generalised transation ledger (version 2021-04-21). Technical report (2021)

    Google Scholar 

  42. Yang, Z., Lei, H.: Lolisa: formal syntax and semantics for a subset of the Solidity programming language in mathematical tool Coq. Math. Probl. Eng. 2020, 6191537 (2020)

    Google Scholar 

Download references

Acknowledgements

We would like to thank Tobias Nipkow for useful discussions about the compliance testing. Moreover, we would like to thank Silvio Degenhardt for his support with implementing the semantics.

Author information

Authors and Affiliations

  1. University of Exeter, Exeter, UK

    Diego Marmsoler & Achim D. Brucker

Authors
  1. Diego Marmsoler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Achim D. Brucker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Achim D. Brucker .

Editor information

Editors and Affiliations

  1. Faculty of Informatics, TU Wien, Vienna, Austria

    Laura Kovács

  2. Royal Institute of Technology, Stockholm, Sweden

    Karl Meinke

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Marmsoler, D., Brucker, A.D. (2022). Conformance Testing of Formal Semantics Using Grammar-Based Fuzzing. In: Kovács, L., Meinke, K. (eds) Tests and Proofs. TAP 2022. Lecture Notes in Computer Science, vol 13361. Springer, Cham. https://doi.org/10.1007/978-3-031-09827-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-09827-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-09826-0

  • Online ISBN: 978-3-031-09827-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics