Abstract
We present a system which automatically generates an exhaustive set of black-box test-cases, up to a given size, for units under test requiring complex preconditions. The key of the approach is to translate a formal precondition into a set of constraints belonging to the decidable logics of SMT solvers. By checking the satisfiability of the constraints, then the models returned by the solver automatically synthesize the cases. We also show how to use SMT solvers to automatically check the validity of the test-case results, by using the postcondition as an oracle, and also how to complement the black-box cases with white-box ones automatically generated. Finally, we use the solver to perform what we call automatic partial verification of the program. In summary, we present a system in which exhaustive black-box and white-box testing, result validation, and partial verification, can all be done automatically. The only extra effort required from programmers is to write formal specifications.
Work partially funded by the Spanish Ministry of Economy and Competitiveness, under the grant TIN2017-86217-R, and by the Madrid Regional Government, under the grant S2018/TCS-4339, co-funded by the European Union EIE funds.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Acronym of Computer Assisted Validation by Analysis, Transformation and Proof.
- 2.
- 3.
Static Single Assignment.
References
Anand, S., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013). https://doi.org/10.1016/j.jss.2013.02.061
Aracil, M., García, P., Peña, R.: A tool for black-box testing in a multilanguage verification platform. In: Proceedings of the XVII Jornadas sobre Programación y Lenguajes, PROLE 2017, Tenerife, Spain, September 2017, pp. 1–15 (2017)
Barrett, C.W., Sebastiani, R., Seshia, S.A., Tinelli, C.: Satisfiability modulo theories. In: Biere, A., Heule, M., van Maaren, H., Walsh, T. (eds.) Handbook of Satisfiability, Frontiers in Artificial Intelligence and Applications, vol. 185, pp. 825–885. IOS Press (2009). https://doi.org/10.3233/978-1-58603-929-5-825
Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on Java predicates. In: Frankl, P.G. (ed.) Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 2002, Roma, Italy, 22–24 July 2002, pp. 123–133. ACM (2002). https://doi.org/10.1145/566172.566191, http://doi.acm.org/10.1145/566172.566191
Cadar, C., Dunbar, D., Engler, D.R.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Draves, R., van Renesse, R. (eds.) 8th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2008(December), pp. 8–10, 2008. San Diego, California, USA, Proceedings, pp. 209–224. USENIX Association (2008). http://www.usenix.org/events/osdi08/tech/full_papers/cadar/cadar.pdf
Cadar, C., Sen, K.: Symbolic execution for software testing: three decades later. Commun. ACM 56(2), 82–90 (2013). https://doi.org/10.1145/2408776.2408795
Casso, I., Morales, J.F., López-García, P., Hermenegildo, M.V.: An integrated approach to assertion-based random testing in prolog. In: Gabbrielli, M. (ed.) LOPSTR 2019. LNCS, vol. 12042, pp. 159–176. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45260-5_10
Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: Odersky, M., Wadler, P. (eds.) Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP 2000), Montreal, Canada, 18–21 September 2000, pp. 268–279. ACM (2000). https://doi.org/10.1145/351240.351266, http://doi.acm.org/10.1145/351240.351266
Flanagan, C., Sabry, A., Duba, B.F., Felleisen, M.: The essence of compiling with continuations. In: Cartwright, R. (ed.) Proceedings of the Conference on Programming Language Design and Implementation (PLDI 1993), pp. 237–247. ACM (1993). https://doi.org/10.1145/155090.155113, http://doi.acm.org/10.1145/155090.155113
Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. In: Sarkar, V., Hall, M.W. (eds.) Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, Chicago, IL, USA, 12–15 June 2005, pp. 213–223. ACM (2005). https://doi.org/10.1145/1065010.1065036
Hughes, J.: Software testing with QuickCheck. In: Horváth, Z., Plasmeijer, R., Zsók, V. (eds.) CEFP 2009. LNCS, vol. 6299, pp. 183–223. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17685-2_6
Jackson, D., Schechter, I., Shlyakhter, I.: Alcoa: the Alloy constraint analyzer. In: Ghezzi, C., Jazayeri, M., Wolf, A.L. (eds.) Proceedings of the 22nd International Conference on on Software Engineering, ICSE 2000, Limerick Ireland, 4–11 June 2000, pp. 730–733. ACM (2000). https://doi.org/10.1145/337180.337616
Khurshid, S., Marinov, D.: TestEra: specification-based testing of Java programs using SAT. Autom. Softw. Eng. 11(4), 403–434 (2004). https://doi.org/10.1023/B:AUSE.0000038938.10589.b9
Montenegro, M., Nieva, S., Peña, R., Segura, C.: Liquid types for array invariant synthesis. In: D’Souza, D., Narayan Kumar, K. (eds.) ATVA 2017. LNCS, vol. 10482, pp. 289–306. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68167-2_20
Montenegro, M., Peña, R., Sánchez-Hernández, J.: A generic intermediate representation for verification condition generation. In: Falaschi, M. (ed.) LOPSTR 2015. LNCS, vol. 9527, pp. 227–243. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-27436-2_14
de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24
Pasareanu, C.S., Visser, W.: A survey of new trends in symbolic execution for software testing and analysis. STTT 11(4), 339–353 (2009)
Peña, R., Sánchez-Hernández, J.: White-box path generation in recursive programs. In: Byrski, A., Hughes, J. (eds.) TFP 2020. LNCS, vol. 12222, pp. 121–135. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57761-2_6
Reynolds, A., Blanchette, J.C., Cruanes, S., Tinelli, C.: Model finding for recursive functions in SMT. In: Olivetti, N., Tiwari, A. (eds.) IJCAR 2016. LNCS (LNAI), vol. 9706, pp. 133–151. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40229-1_10
Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: Wermelinger, M., Gall, H.C. (eds.) Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon, Portugal, 5–9 September 2005, pp. 263–272. ACM (2005). https://doi.org/10.1145/1081706.1081750
Toda, T., Soh, T.: Implementing efficient all solutions SAT solvers. ACM J. Exp. Algorithmics 21(1), 1.12:1–1.12:44 (2016). https://doi.org/10.1145/2975585
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Peña, R., Sánchez-Hernández, J., Garrido, M., Sagredo, J. (2023). SMT-Based Test-Case Generation and Validation for Programs with Complex Specifications. In: Lopez-Garcia, P., Gallagher, J.P., Giacobazzi, R. (eds) Analysis, Verification and Transformation for Declarative Programming and Intelligent Systems. Lecture Notes in Computer Science, vol 13160. Springer, Cham. https://doi.org/10.1007/978-3-031-31476-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-31476-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31475-9
Online ISBN: 978-3-031-31476-6
eBook Packages: Computer ScienceComputer Science (R0)