Skip to main content
SpringerLink
Log in

Property-driven benchmark generation: synthesizing programs of realistic structure

  • RERS
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

We present a systematic approach to the automatic generation of platform-independent benchmarks of realistic structure and tailored complexity for evaluating verification tools for reactive systems. The idea is to mimic a systematic constraint-driven software development process by automatically transforming randomly generated temporal-logic-based requirement specifications on the basis of a sequence of property-preserving, randomly generated structural design decisions into executable source code of a chosen target language or platform. Our automated transformation process steps through dedicated representations in terms of Büchi automata, Mealy machines, decision diagram models, and code models. It comprises LTL synthesis, model checking, property-oriented expansion, path condition extraction, theorem proving, SAT solving, and code motion. This setup allows us to address different communities via a growing set of programming languages, tailored sets of programming constructs, different notions of observation, and the full variety of LTL properties—ranging from mere reachability over general safety properties to arbitrary liveness properties. The paper illustrates the corresponding tool chain along accompanying examples, emphasizes the current state of development, and sketches the envisioned potential and impact of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Of course, dedicated real(-istic) problems are also extremely important, but because of their “singularity” and a priori unknown properties, they are not suitable for a careful, wide-range profile analysis.

  2. In particular “mystery” and “multi” caches. See http://www.geocaching.com/.

  3. We chose Mealy machines as our intermediate model structure because of their input/output distinction. Of course labeled transition systems [36] or I/O automata [23] could have been chosen as well.

  4. This is due to the implicit universal quantification over paths.

  5. We rely here on the intuitive clarity of this definition and stick to this informal definition as we are treating a number of different kinds of artifacts during benchmark generation. A formal treatment would therefore be very tedious without major benefit.

  6. The Mealy machine that arises from \(M\) taking all (“may”) transitions.

  7. The (partial) Mealy machine that arises from taking only the “must” transitions of \(M\).

  8. In order to emphasize the essence of the POE pattern, we decided to illustrate it on a simple directed graph rather than on a deterministic Mealy machine.

  9. Of course, LTL also allows for specifying reachability properties, e.g., \(\mathrm{\mathbf {F} }\mathrm{error}_i\). However, as failed assertions or uncaught exceptions are a common source of errors, we also want to address tools built to detect these kind of errors without supporting the full range of LTL properties.

References

  1. Almeida, E.E., Luntz, J.E., Tilbury, D.M.: Event-condition–action systems for reconfigurable logic control. IEEE Trans. Autom. Sci. Eng. 4(2), 167–181 (2007)

  2. Apt, K.R., Olderog, E.-R.: Verification of Sequential and Concurrent Programs. Texts and Monographs in Computer Science. Springer, New York, NY, USA (1991)

  3. Bauer, O., Geske, M., Isberner, M.: Analyzing program behavior through active automata learning. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0333-2 (2014)

  4. Beyer, D., Stahlbauer, A.: BDD-based software verification. Applications to event-condition–action systems. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0334-1 (2014)

  5. Beyer, D. Competition on software verification—(SV-COMP). In: Proceedings of of 18th Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2012), vol. 7214 of LNCS, pp. 504–524. Springer (2012)

  6. Biere, A., Heule, M.J.H., van Maaren, H., Walsh, T. (editors): Handbook of Satisfiability, vol. 185 of Frontiers in Artificial Intelligence and Applications. IOS Press, Amsterdam, NL (2009)

  7. Blom, S.C.C., van de Pol, J.C., Weber, M.: Ltsmin: distributed and symbolic reachability. In: Touili, T., Cook, B., Jackson, P. (eds.) Computer Aided Verification. Edinburgh, vol. 6174 of Lecture Notes in Computer Science, pp. 354–359. Springer, Berlin (2010)

  8. Briggs, P., Cooper, K.D.: Effective partial redundancy elimination. In: Proceedings of ACM SIGPLAN Conf. Prog. Lang. Design and Impl. (PLDI’94), pp. 159–170 (1994)

  9. Burkart, O., Steffen, B.: Model checking the full modal Mu-Calculus for infinite sequential processes. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) Automata. Languages and Programming, vol. 1256 of Lecture Notes in Computer Science, pp. 419–429. Springer, Berlin Heidelberg (1997)

  10. Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge, MA, USA (2001)

  11. Combe, D., de la Higuera, C., Zulu, J.-C.J.: An interactive learning competition. In: Revised Selected Papers of 8th International Workshop on Finite-State Methods and Natural Language Processing (FSMNLP 2009), vol. 6062, pp. 139–146. Springer (2010)

  12. Dhamdhere, D.M.: A new algorithm for composite hoisting and strength reduction optimisation (+ Corrigendum). Int. J. Comp. Math. 27, 1–14 (1989)

    Article  Google Scholar 

  13. Dwyer, M.B., Avrunin, G.S., Corbett, J.C.: Patterns in property specifications for finite-state verification. In: Proceedings of the 1999 Int. Conf. on Software Engineering, pp. 411–420. IEEE (1999)

  14. Floyd, R.W.: Assigning meaning to programs. In: Proceedongs of Symposium on Applied Mathematics, vol. 19 of Mathematical aspects of computer science, pp. 19–32. American Mathematical Society (1967)

  15. Gastin, P., Oddoux, D.: Fast LTL to Büchi Automata Translation. In: Berry, G., Comon, H., Finkel, A., (eds) Proceedings of the 13th International Conference on Computer Aided Verification (CAV’01), vol. 2102 of Lecture Notes in Computer Science, pp. 53–65, Paris, France. Springer (2001)

  16. Giannakopoulou, D., Lerda, F.: From states to transitions: improving translation of LTL formulae to Büchi Automata. In: Proceedings of the 22nd IFIP WG 6.1 International Conference Houston on Formal Techniques for Networked and Distributed Systems, FORTE ’02, pp. 308–326, London, UK. Springer (2002)

  17. Hayes-Roth, F.: Rule-based systems. Commun. ACM 28(9), 921–932 (1985)

  18. Hoare, C.A.R.: An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)

    Article  MATH  Google Scholar 

  19. Holzmann, G.J.: The SPIN Model Checker—Primer and Reference Manual. Addison-Wesley, Boston, MA, USA (2004)

  20. Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D., Pasareanu, C.S.: Rigorous examination of reactive systems. The RERS challenges 2012 and 2013. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0337-y (2014)

  21. Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D.: The RERS grey-box challenge 2012: analysis of event-condition–action systems. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change, vol. 7609 of Lecture Notes in Computer Science, pp. 608–614. Springer, Berlin Heidelberg (2012)

  22. Huth, M.: Model checking modal transition systems using kripke structures. In: Revised Papers from the Third International Workshop on Verification, Model Checking, and Abstract Interpretation, VMCAI ’02, pp. 302–316, London. Springer (2002)

  23. Kaynar, D.K., Lynch, N.A., Segala, R., Vaandrager, F.W.: Timed I/O Automata: a mathematical framework for modeling and analyzing real-time systems. In: Proceedings of the 24th IEEE Real-Time Systems Symposium (RTSS 2003), pp. 166–177. IEEE Computer Society (2003)

  24. Klebanov, V., Müller, P., Shankar, N., Leavens, G.T., Wüstholz, V., Alkassar, E., Arthan, R., Bronish, D., Chapman, R., Cohen, E., Hillebrand, M.A., Rustan, J.K., Leino, M., Monahan, R., Piessens, F., Polikarpova, N., Ridge, T., Smans, J., Tobies, S., Tuerk, T., Ulbrich, M., Weiß, B.: The 1st verified software competition: experience report. In: Proceedings of the 17th Int. Symposium on Formal Methods (FM 2011), vol. 6664 of LNCS, pp. 154–168 (2011)

  25. Knoop, J., Rüthing, O., Steffen, B.: Expansion-based removal of semantic partial redundancies. In: Compiler Construction, 8th International Conference, CC’99, Held as Part of the European Joint Conferences on the Theory and Practice of Software, ETAPS’99, Amsterdam, The Netherlands, 22–28 March, 1999, Proceedings, vol. 1575 of LNCS, pp. 91–106. Springer (1999)

  26. Knoop, J., Rüthing, O., Steffen, B.: Lazy code motion. In: Proceedings of the ACM SIGPLAN’92 Conference on Programming Language Design and Implementation (PLDI), pp. 224–234. ACM (1992)

  27. Knoop, J., Rüthing, O., Steffen, B.: Lazy strength reduction. J. Program. Lang. 1, 71–91 (1993)

  28. Knoop, J., Rüthing, O., Steffen, B.: Optimal code motion: theory and practice. ACM Trans. Program. Lang. Syst. 16(4), 1117–1155 (1994)

  29. Knoop, J., Rüthing, O., Steffen, B.: Partial dead code elimination. In: Proceedings of the ACM SIGPLAN’94 Conference on Programming Language Design and Implementation (PLDI), pp. 147–158. ACM (1994)

  30. Knoop, J., Rüthing, O., Steffen, B. The power of assignment motion. In: Proceedings of the ACM SIGPLAN’95 Conference on Programming Language Design and Implementation (PLDI). ACM (1995)

  31. Lamport, L.: What good is temporal logic? Inf. Process. 83, 657–668 (1983)

  32. Larsen, K.G.: Modal specifications. In: Automatic Verification Methods for Finite State Systems, pp. 232–246 (1989)

  33. Margaria, T., Steffen, B.: Continuous model-driven engineering. IEEE Computer 42(10), 106–109 (2009)

  34. McCarthy, D.R., Dayal, U.: The architecture of an active data base management system. In: Proceedings of the 1989 ACM SIGMOD International Conference on Management of Data, pp. 215–224. ACM Press (1989)

  35. Mealy, G.H.: A method for synthesizing sequential circuits. Bell Syst. Tech. J. 34(5), 1045–1079 (1955)

  36. Milner, R.: Communication and Concurrency. PHI Series in Computer Science. Prentice-Hall Inc, Upper Saddle River (1989)

  37. Morel, E., Renvoise, C.: Global optimization by suppression of partial redundancies. Commun. ACM 22(2), 96–103 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  38. Morse, J., Cordeiro, L., Nicole, D., Fischer, B.: Applying symbolic bounded model checking to the 2012 RERS greybox challenge. Soft. Tools Technol. Transf. doi:10.1007/s10009-014-0335-0 (2014)

  39. Robinson, A., Voronkov, A. (eds): Handbook of Automated Reasoning, vol. I and II. Elsevier, Amsterdam, NL (2001)

  40. Rosen, B.K., Wegman, M.N., Kenneth Zadeck, F.: Global value numbers and redundant computations. In: Conference Record of the Fifteenth Annual ACM Symposium on Principles of Programming Languages. ACM Press (1988)

  41. Rüthing, O., Knoop, J., Steffen, B.: Sparse code motion. In: Proceedings of the 27th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2000), pp. 170–183. ACM (2000)

  42. SCCE: Service Centered Continuous Engineering. http://scce.info. Accessed 26 Feb 2014

  43. Schordan, M., Prantl, A.: Combining static analysis and state transition graphs for verification of event-condition–action systems in the RERS 2012 and 2013 challenges. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0338-x (2014)

  44. Snelting, G., Robschnik, T., Krinke, J.: Efficient path conditions in dependence graphs for software safety analysis. ACM Trans. Softw. Eng. Methodol. (TOSEM) 15(4), 410–457 (2006)

  45. Steffen, B., Claßen, A., Klein, M., Knoop, J., Margaria, T.: The fixpoint-analysis machine. In: Lee, I., Smolka, S.A. (eds) CONCUR ’95: Concurrency Theory, volume 962 of Lecture Notes in Computer Science, pp. 72–87. Springer, Berlin, Heidelberg (1995)

  46. Steffen, B., Howar, F., Isberner, M., Naujokat, S., Margaria, T.: Tailored generation of concurrent benchmarks. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0339-9 (2014)

  47. Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation. In: International SPIN Symposium on Model Checking of Software (SPIN2013), vol. 7976 of LNCS, pp. 341–357. Springer (2013)

  48. Steffen, B., Knoop, J., Rüthing, O.: Efficient code motion and an adaption to strength reduction. In: Proceedings of the International Joint Conference on Theory and Practice of Software Development (TAPSOFT’91). Volume 2: Advances in Distributed Computing (ADC) and Colloquium on Combining Paradigms for Software Developmemnt (CCPSD), vol. 494 of LNCS, pp. 394–415. Springer (1991)

  49. Steffen, B., Knoop, J., Rüthing, O.: The value flow graph: a program representation for optimal program transformations. In: Jones, N.D. (ed.) 3rd European Symposium on Programming (ESOP’90). Lecture Notes in Computer Science, vol. 432, pp. 389–405. Springer, Berlin Heidelberg (1990)

  50. Steffen, B., Knoop, J.: Finite constants: characterizations of a new decidable set of constants. In: Kreczmar, A., Mirkowska, G. (eds) Mathematical Foundations of Computer Science (MFCS’89), vol. 379 of LNCS, pp. 481–491. Springer (1989)

  51. Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds) Haifa Verification Conference, vol. 4383 of Lecture Notes in Computer Science, pp. 92–108. Springer (2006)

  52. Steffen, B., Narayan, P.: Full life-cycle support for end-to-end processes. IEEE Computer 40(11), 64–73 (2007)

  53. Steffen, B., Rüthing, O.: Quality engineering: leveraging heterogeneous information—(Invited Talk). In: Proceedings of the 12th Int. Conf. on Verification, Model Checking, and Abstract Interpretation (VMCAI 2011), LNCS, pp. 23–37 (2011)

  54. Steffen, B.: Characteristic formulae. In: Ausiello, G., Dezani-Ciancaglini, M., Rocca, S.R. (eds.) Automata. Languages and Programming, vol. 372 of Lecture Notes in Computer Science, pp. 723–732. Springer, Berlin Heidelberg (1989)

  55. Steffen, B.: Property-oriented expansion. In: Cousot, R., Schmidt, D.A. (eds.) Third International Symposium on Static Analysis (SAS ’96). Lecture Notes in Computer Science, vol. 1145, pp. 22–41. Springer, Berlin, Heidelberg (1996)

  56. Steffen, B.: Unifying models. In: Reischuk, R., Morvan, M. (eds.) STACS 97. Lecture Notes in Computer Science, vol. 1200, pp. 1–20. Springer, Berlin, Heidelberg (1997)

  57. van de Pol, J., Ruys, T.C., te Brinke, S.: Thoughtful brute force attack of the RERS 2012 and 2013 challenges. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0324-3 (2014)

  58. Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in C compilers. In: Proceedings of the 32nd ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI 2011) (2011)

Download references

Author information

Authors and Affiliations

  1. Chair for Programming Systems, Technische Universität Dortmund, Dortmund,  44227, Germany

    Bernhard Steffen, Malte Isberner, Stefan Naujokat & Maren Geske

  2. Chair Service and Software Engineering, Universität Potsdam, Potsdam,  14482, Germany

    Tiziana Margaria

Authors
  1. Bernhard Steffen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Malte Isberner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Naujokat
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tiziana Margaria
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Maren Geske
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Naujokat.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steffen, B., Isberner, M., Naujokat, S. et al. Property-driven benchmark generation: synthesizing programs of realistic structure. Int J Softw Tools Technol Transfer 16, 465–479 (2014). https://doi.org/10.1007/s10009-014-0336-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0336-z

Keywords

Search

Navigation