Skip to main content
SpringerLink
Log in

Timeout Prediction for Software Analyses

  • Conference paper
  • First Online:
Software Engineering and Formal Methods (SEFM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14323))

Included in the following conference series:

  • 231 Accesses

Abstract

Software verification tools automatically prove the correctness of programs with respect to user supplied specifications. Today, such tools implement a range of different types of analyses. As different analyses are good at different sorts of verification tasks, state-of-the-art tools often employ sequential compositions of analyses in which every analysis gets a fixed time slot assigned for verification. As a consequence, however, one analysis might consume parts of the overall available time although it does not finish within its time slot.

In this paper, we propose timeout prediction as a way to determine when an analysis should get its full time slot and when to prematurely stop it. Our technique for timeout prediction employs machine learning to predict whether a given analysis will terminate on a given verification task (within a time limit) or will time out. To this end, we develop static as well as dynamic features of verification tasks and analyses. Values of static features can be statically determined for tasks; dynamic features are determined while an analysis is already running. Our experimental evaluation shows that we can predict timeouts with a high accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://gitlab.com/sosy-lab/benchmarking/sv-benchmarks/-/tree/svcomp23.

  2. 2.

    https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html.

  3. 3.

    https://github.com/scikit-learn/scikit-learn.

References

  1. Beyer, D., Dangl, M.: Strategy selection for software verification based on boolean features. In: Margaria, T., Steffen, B. (eds.) ISoLA 2018. LNCS, vol. 11245, pp. 144–159. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03421-4_11

    Chapter  Google Scholar 

  2. Beyer, D., Henzinger, T.A., Théoduloz, G.: Configurable software verification: concretizing the convergence of model checking and program analysis. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 504–518. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73368-3_51

    Chapter  MATH  Google Scholar 

  3. Beyer, D., Kanav, S., Richter, C.: Construction of verifier combinations based on off-the-shelf verifiers. In: FASE 2022. LNCS, vol. 13241, pp. 49–70. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99429-7_3

    Chapter  Google Scholar 

  4. Beyer, D., Keremoglu, M.E.: CPAchecker: a tool for configurable software verification. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 184–190. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-22110-1_16

    Chapter  Google Scholar 

  5. Beyer, D., Keremoglu, M.E., Wendler, P.: Predicate abstraction with adjustable-block encoding. In: FMCAD 2010, pp. 189–197. IEEE (2010)

    Google Scholar 

  6. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. STTT 21(1), 1–29 (2019)

    Article  Google Scholar 

  7. Chalupa, M., Henzinger, T.A.: BUBAAK: runtime monitoring of program verifiers. In: Sankaranarayanan, S., Sharygina, N. (eds.) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp. 535–540. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_32

  8. Chalupa, M., Mihalkovič, V., Řechtáčková, A., Zaoral, L., Strejček, J.: Symbiotic 9: string analysis and backward symbolic execution with loop folding. In: TACAS 2022. LNCS, vol. 13244, pp. 462–467. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99527-0_32

    Chapter  Google Scholar 

  9. Chalupa, M., Strejcek, J., Vitovská, M.: Joint forces for memory safety checking revisited. Int. J. Softw. Tools Technol. Transf. 22(2), 115–133 (2020). https://doi.org/10.1007/s10009-019-00526-2

    Article  Google Scholar 

  10. Clarke, E., Grumberg, O., Jha, S., Lu, Y., Veith, H.: Counterexample-guided abstraction refinement. In: Emerson, E.A., Sistla, A.P. (eds.) CAV 2000. LNCS, vol. 1855, pp. 154–169. Springer, Heidelberg (2000). https://doi.org/10.1007/10722167_15

    Chapter  Google Scholar 

  11. Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: ACM 1977, pp. 238–252. ACM (1977). https://doi.org/10.1145/512950.512973

  12. Czech, M., Hüllermeier, E., Jakobs, M., Wehrheim, H.: Predicting rankings of software verification tools. In: SWAN@ESEC/SIGSOFT FSE 2017, pp. 23–26. ACM (2017). https://doi.org/10.1145/3121257.3121262

  13. Dangl, M., Löwe, S., Wendler, P.: CPAchecker with support for recursive programs and floating-point arithmetic. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 423–425. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46681-0_34

    Chapter  Google Scholar 

  14. Demyanova, Y., Pani, T., Veith, H., Zuleger, F.: Empirical software metrics for benchmarking of verification tools. Formal Methods Syst. Des. 50(2), 289–316 (2017). https://doi.org/10.1007/s10703-016-0264-5

    Article  MATH  Google Scholar 

  15. Ernst, G.: KORN–software verification with horn clauses (Competition Contribution). In: Sankaranarayanan, S., Sharygina, N. (eds) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp 559–564. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_36

  16. Ernst, G., Huisman, M., Mostowski, W., Ulbrich, M.: VerifyThis – verification competition with a human factor. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 176–195. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_12

    Chapter  Google Scholar 

  17. Graf, S., Saidi, H.: Construction of abstract state graphs with PVS. In: Grumberg, O. (ed.) CAV 1997. LNCS, vol. 1254, pp. 72–83. Springer, Heidelberg (1997). https://doi.org/10.1007/3-540-63166-6_10

    Chapter  Google Scholar 

  18. Hajdu, Á., Micskei, Z.: Efficient strategies for CEGAR-based model checking. J. Autom. Reason. 64(6), 1051–1091 (2020). https://doi.org/10.1007/s10817-019-09535-x

    Article  MathSciNet  MATH  Google Scholar 

  19. Healy, A., Monahan, R., Power, J.F.: Predicting SMT solver performance for software verification. In: F-IDE@FM 2016, EPTCS, vol. 240, pp. 20–37 (2016). https://doi.org/10.4204/EPTCS.240.2

  20. Heizmann, M., Hoenicke, J., Podelski, A.: Software model checking for people who love automata. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 36–52. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39799-8_2

    Chapter  Google Scholar 

  21. Leeson, W., Dwyer, M.B.: Graves-CPA: a graph-attention verifier selector (Competition Contribution). In: TACAS 2022. LNCS, vol. 13244, pp. 440–445. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99527-0_28

    Chapter  Google Scholar 

  22. Leeson, W., Dwyer, M.B., Filieri, A.: Sibyl: improving software engineering tools with SMT selection. In: Proceedings of ICSE (2023)

    Google Scholar 

  23. Luckow, K., et al.: JDart: a dynamic symbolic analysis framework. In: Chechik, M., Raskin, J.-F. (eds.) TACAS 2016. LNCS, vol. 9636, pp. 442–459. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49674-9_26

    Chapter  Google Scholar 

  24. Luo, S., Xu, H., Bi, Y., Wang, X., Zhou, Y.: Boosting symbolic execution via constraint solving time prediction (experience paper). In: ISSTA 2021, pp. 336–347. ACM (2021). https://doi.org/10.1145/3460319.3464813

  25. Richter, C., Hüllermeier, E., Jakobs, M., Wehrheim, H.: Algorithm selection for software validation based on graph kernels. Autom. Softw. Eng. 27(1), 153–186 (2020). https://doi.org/10.1007/s10515-020-00270-x

    Article  Google Scholar 

  26. Richter, C., Wehrheim, H.: PeSCo: predicting sequential combinations of verifiers. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 229–233. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-17502-3_19

    Chapter  Google Scholar 

  27. Richter, C., Wehrheim, H.: Attend and represent: a novel view on algorithm selection for software verification. In: ASE 2020, pp. 1016–1028. IEEE (2020). https://doi.org/10.1145/3324884.3416633

  28. Saan, S. et al.: GOBLINT: autotuning thread-modular abstract interpretation. In: Sankaranarayanan, S., Sharygina, N. (eds.) Tools and Algorithms for the Construction and Analysis of Systems. TACAS 2023. LNCS, vol. 13994, pp. 547–552. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30820-8_34

  29. Scott, J., Niemetz, A., Preiner, M., Nejati, S., Ganesh, V.: MachSMT: a machine learning-based algorithm selector for SMT solvers. In: TACAS 2021. LNCS, vol. 12652, pp. 303–325. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72013-1_16

    Chapter  Google Scholar 

  30. Thoben, N.: Online Performance Prediction of Software Verification using Machine Learning. Master’s thesis, University of Oldenburg, Department of Computer Science (2023)

    Google Scholar 

  31. Tulsian, V., Kanade, A., Kumar, R., Lal, A., Nori, A.V.: MUX: algorithm selection for software model checkers. In: MSR 2014, pp. 132–141. ACM (2014). https://doi.org/10.1145/2597073.2597080

  32. Wahl, T.: The k-induction principle (2013). http://www.ccs.neu.edu/home/wahl/Publications/k-induction.pdf

Download references

Author information

Authors and Affiliations

  1. Department of Computing Science, University of Oldenburg, Oldenburg, Germany

    Nicola Thoben, Jan Haltermann & Heike Wehrheim

Authors
  1. Nicola Thoben
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Haltermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heike Wehrheim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicola Thoben .

Editor information

Editors and Affiliations

  1. NOVA University Lisbon, Caparica, Portugal

    Carla Ferreira

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Tim A. C. Willemse

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Thoben, N., Haltermann, J., Wehrheim, H. (2023). Timeout Prediction for Software Analyses. In: Ferreira, C., Willemse, T.A.C. (eds) Software Engineering and Formal Methods. SEFM 2023. Lecture Notes in Computer Science, vol 14323. Springer, Cham. https://doi.org/10.1007/978-3-031-47115-5_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47115-5_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47114-8

  • Online ISBN: 978-3-031-47115-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation