Skip to main content
Springer Nature Link
Log in

Configurable Benchmarks for C Model Checkers

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13260))

Included in the following conference series:

  • 2245 Accesses

Abstract

Software model checkers employ many different techniques. During various competitions, the capabilities of these verification tools are compared on a wide variety of benchmarks. Our aim is to get insight into which code characteristics are “hard” for software model checkers. To that end, we present a software tool that automatically generates C benchmark programs that are intended as stress tests for software model checkers. The parameters of the generated C programs, e.g., program size, types of operation, are controllable, and programs can be tweaked, e.g., floats can be replaced by integers and pointer dereferencing can be used for variable accesses. Our tool enables a systematic comparison of software verifiers. We illustrate its usage by evaluating the top verifiers from the SV-COMP 2022 reachability category and analyze what makes benchmarks hard for these tools and how well these tools scale, both in terms of code related to the property at hand as well as in terms of code that is unrelated to it.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/ultimate-pa/ultimate/issues/578.

  2. 2.

    https://sv-comp.sosy-lab.org/2022/systems.php.

  3. 3.

    https://gitlab.com/sosy-lab/benchmarking/competition-scripts/#instructions-for-execution-and-reproduction.

  4. 4.

    https://stat.ethz.ch/R-manual/R-devel/library/stats/html/wilcox.test.html.

  5. 5.

    https://github.com/diffblue/2ls/issues/159.

References

  1. Berger, P., Katoen, J.-P., Ábrahám, E., Waez, M.T.B., Rambow, T.: Verifying auto-generated C code from simulink. In: Havelund, K., Peleska, J., Roscoe, B., de Vink, E. (eds.) FM 2018. LNCS, vol. 10951, pp. 312–328. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-95582-7_18

    Chapter  Google Scholar 

  2. Beyer, D.: Software verification: 10th comparative evaluation (SV-COMP 2021). In: TACAS 2021. LNCS, vol. 12652, pp. 401–422. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72013-1_24

    Chapter  Google Scholar 

  3. Beyer, D., Keremoglu, M.E.: CPAchecker: a tool for configurable software verification. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 184–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_16

    Chapter  Google Scholar 

  4. Beyer, D., Lemberger, T.: Software verification: testing vs. model checking. In: HVC 2017. LNCS, vol. 10629, pp. 99–114. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70389-3_7

    Chapter  Google Scholar 

  5. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1–29 (2019)

    Article  Google Scholar 

  6. Chalupa, M., Novák, J., Strejcek, J.: Symbiotic 8: parallel and targeted test generation - (competition contribution). In: FASE. LNCS, vol. 12649, pp. 368–372. Springer (2021)

    Google Scholar 

  7. Chalupa, M., Strejček, J.: Evaluation of program slicing in software verification. In: Ahrendt, W., Tapia Tarifa, S.L. (eds.) IFM 2019. LNCS, vol. 11918, pp. 101–119. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34968-4_6

    Chapter  Google Scholar 

  8. Cordeiro, L.C., Kroening, D., Schrammel, P.: Benchmarking of java verification tools at the software verification competition (SV-COMP). ACM SIGSOFT Softw. Eng. Notes 43(4), 56 (2018)

    Article  Google Scholar 

  9. Gadelha, M.Y.R., Menezes, R., Monteiro, F.R., Cordeiro, L.C., Nicole, D.A.: ESBMC: scalable and precise test generation based on the floating-point theory - (competition contribution). In: FASE. LNCS, vol. 12076, pp. 525–529. Springer (2020)

    Google Scholar 

  10. Groce, A., Havelund, K., Holzmann, G., Joshi, R., Xu, R.-G.: Establishing flight software reliability: testing, model checking, constraint-solving, monitoring and learning. Ann. Math. Artif. Intell. 70(4), 315–349 (2014). https://doi.org/10.1007/s10472-014-9408-8

    Article  MathSciNet  MATH  Google Scholar 

  11. Heizmann, M., et al.: Ultimate automizer and the search for perfect interpolants. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 447–451. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-89963-3_30

    Chapter  Google Scholar 

  12. Howar, F., Jasper, M., Mues, M., Schmidt, D., Steffen, B.: The RERS challenge: towards controllable and scalable benchmark synthesis. Int. J. Softw. Tools Technol. Transf. 23(6), 917–930 (2021). https://doi.org/10.1007/s10009-021-00617-z

    Article  Google Scholar 

  13. Jasper, M.: Synthesizing realistic verification tasks. Ph.D. thesis, Technical University of Dortmund, Germany (2021)

    Google Scholar 

  14. Jhala, R., Majumdar, R.: Software model checking. ACM Comput. Surv. 41(4), 21:1–21:54 (2009)

    Google Scholar 

  15. Klinger, C., Christakis, M., Wüstholz, V.: Differentially testing soundness and precision of program analyzers. In: ISSTA, pp. 239–250. ACM (2019)

    Google Scholar 

  16. Malík, V., Schrammel, P., Vojnar, T.: 2LS: heap analysis and memory safety. In: TACAS 2020. LNCS, vol. 12079, pp. 368–372. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45237-7_22

    Chapter  Google Scholar 

  17. Méry, D., Singh, N.K.: Automatic code generation from event-b models. In: SoICT, pp. 179–188. ACM (2011)

    Google Scholar 

  18. Richter, C., Wehrheim, H.: PeSCo: predicting sequential combinations of verifiers. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 229–233. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_19

    Chapter  Google Scholar 

  19. Sharir, M., Pnueli, A., et al.: Two Approaches to Interprocedural Data Flow Analysis. New York University, Courant Institute of Mathematical Sciences (1978)

    Google Scholar 

  20. Vorobyov, K., Krishnan, P.: Comparing model checking and static program analysis: a case study in error detection approaches. In: Proceedings of SSV (2010)

    Google Scholar 

  21. Westhofen, L., Berger, P., Katoen, J.-P.: Benchmarking software model checkers on automotive code. In: Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds.) NFM 2020. LNCS, vol. 12229, pp. 133–150. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-55754-6_8

    Chapter  Google Scholar 

  22. Zhang, C., Su, T., Yan, Y., Zhang, F., Pu, G., Su, Z.: Finding and understanding bugs in software model checkers. In: ESEC/SIGSOFT FSE, pp. 763–773. ACM (2019)

    Google Scholar 

Download references

Acknowledgments

We thank Fabian Hippler and Felix Faber for their continuing support and work.

Author information

Authors and Affiliations

  1. RWTH Aachen University, Aachen, Germany

    Xaver Fink, Philipp Berger & Joost-Pieter Katoen

Authors
  1. Xaver Fink
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philipp Berger
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joost-Pieter Katoen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Philipp Berger .

Editor information

Editors and Affiliations

  1. University of Southern California, Los Angeles, CA, USA

    Jyotirmoy V. Deshmukh

  2. Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA

    Klaus Havelund

  3. National Institute of Aerospace, Hampton, VA, USA

    Ivan Perez

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fink, X., Berger, P., Katoen, JP. (2022). Configurable Benchmarks for C Model Checkers. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds) NASA Formal Methods. NFM 2022. Lecture Notes in Computer Science, vol 13260. Springer, Cham. https://doi.org/10.1007/978-3-031-06773-0_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06773-0_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06772-3

  • Online ISBN: 978-3-031-06773-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics