Skip to main content
SpringerLink
Log in

Tailored generation of concurrent benchmarks

  • RERS
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

In this paper, we show how to extend our approach to property-driven benchmark generation (PDBG) to support concurrency in the benchmarks: we systematically produce multi-process PROMELA code of known and tailored complexity that can then serve as benchmark for the evaluation of analysis and verification tools for concurrent software systems. Key to this extension is the way in which we control the corresponding property profiles, i.e., the statements about the validity of a predefined set of LTL properties, along the PDBG tool chain. It is based on the idea to introduce intricate structural elements like synchronization and interference at a local and therefore controllable level, only concerning a few parallel components, and subsequently obfuscating the locality of these structures at the global level via property-preserving transformations of both the system components and the properties. This leads to a scalable approach which we illustrate along some concrete examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. A property profile consists of a number of (typically temporal) properties together with an imposed verdict “satisfied” or “failed”. Determining the verdicts for given benchmarks and properties is the challenge.

  2. Of course, dedicated real(-istic) problems are also extremely important, but because of their “singularity” and a priori unknown properties, they are often not suitable for a careful, wide-range profile analysis.

  3. cf. Sect. 6.3 for a more complete presentation of profile parameters

  4. http://www.rers-challenge.org.

  5. Section 6 will provide a more detailed discussion on the impact of the LIGOC principle.

  6. In our context this essentially this requires that all considered properties ignore stuttering, which, e.g., excludes the use of a next operator (cf. Sect. 4).

  7. Please note that assignments in other ASSoCS may be executed an arbitrary number of times, whereas this may be limited for assignments in the same ASSoCs because of synchronization constraints.

  8. Note that for the sake of a more compact representation, we restricted ourselves to only modeling communication or resource usage patterns, omitting the eating and thinking actions of the philosophers.

  9. Except for some technical restrictions like forbidding the next time operator.

  10. Please note that the failure of properties is preserved because they root in properties of the SoCs and the language of the overall system projected onto the alphabets of the individual SoCs remains unchanged.

References

  1. Abadi, M., Lamport, L.: Conjoining specifications. ACM Trans. Program. Lang. Syst. 17(3), 507–535 (1995)

    Article  Google Scholar 

  2. Biere, A., Heule, M.J.H., van Maaren, H., Walsh, T. (eds.): Handbook of Satisfiability. Frontiers in Artificial Intelligence and Applications, vol. 185. IOS Press, Amsterdam (2009)

  3. Briggs, P., Cooper, K.D.: Effective partial redundancy elimination. In: Proceedings of the ACM SIGPLAN’93 Conference on Programming Language Design and Implementation (PLDI’94), pp. 159–170 (1994)

  4. Burkart, O., Steffen, B.: Model checking the full modal mu-calculus for infinite sequential processes. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) Automata. Languages and Programming. Lecture Notes in Computer Science, vol. 1256, pp. 419–429. Springer, Berlin (1997)

  5. Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge (2001)

  6. Elrad, T., Francez, N.: Decomposition of distributed programs into communication-closed layers. Sci. Comput. Program. 2, 155–173 (1982)

    Article  MATH  Google Scholar 

  7. Gastin, P., Oddoux, D.: Fast LTL to Büchi automata translation. In: Gérard, B., Hubert, C., Alain, F. (eds.) Proceedings of CAV’01. LNCS, vol 2102, pp. 53–65, Paris, France. Springer, Berlin (2001)

  8. Giannakopoulou, D., Lerda, F.: From states to transitions: improving translation of LTL formulae to Büchi automata. In: Proceedings of FORTE’02, pp. 308–326, London, UK. Springer, Berlin (2002)

  9. Hoare, C.A.R.: Communicating Sequential Processes. Prentice-Hall Inc., Upper Saddle River (1985)

    MATH  Google Scholar 

  10. Holzmann, G.J.: Holzmann. The SPIN Model Checker—Primer and Reference Manual. Addison-Wesley, Reading (2004)

    Google Scholar 

  11. Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D.: The RERS grey-box challenge 2012: analysis of event-condition-action systems. In: Margaria, T., Steffen, B. (eds.) Proceedings of ISOLA’12. LNCS, vol. 7609, pp. 608–614. Springer, Berlin (2012)

  12. Knoop, J., Rüthing, O., Steffen, B.: Lazy code motion. In Proceedings of the ACM SIGPLAN’92 Conference on Programming Language Design and Implementation (PLDI), pp. 224–234. ACM (1992)

  13. Knoop, J., Rüthing, O., Steffen, B.: Lazy strength reduction. J. Program. Lang. 1, 71–91 (1993)

    Google Scholar 

  14. Kupferman, O., Vardi, M.Y.: Vacuity detection in temporal model checking. In: Pierre, L., Kropf, T. (eds.) Correct Hardware Design and Verification Methods. LNCS, vol. 1703, pp. 82–98. Springer, Berlin (1999)

  15. Larsen, K.G.: Modal Specifications. In: Automatic Verification Methods for Finite State Systems, pp. 232–246 (1989)

  16. Milner, R.: Communication and Concurrency. PHI Series in Computer Science. Prentice-Hall Inc, Upper Saddle River (1989)

  17. Morel, E., Renvoise, C.: Global optimization by suppression of partial redundancies. Commun. ACM 22(2), 96–103 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  18. Naujokat, S., Lybecait, M., Steffen, B., Kopetzki, D., Margaria, T.: Full generation of domain-specific graphical modeling tools: a meta\(^2\)modeling approach (2014 under submission)

  19. Naujokat, S., Traonouez, L-M., Isberner, M., Steffen, B., Axel, L.: Domain-specific code generator modeling: a case study for multi-faceted concurrent systems. In: Proceedings of the 6th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation (ISoLA 2014) (2014 to appear)

  20. Robinson, A., Voronkov, A. (eds.): Handbook of Automated Reasoning, vol. I & II. Elsevier, Amsterdam (2001)

  21. Rosen, B.K., Wegman, M.N., Zadeck, F.K.: Global value numbers and redundant computations. In: Conference Record of the Fifteenth Annual ACM Symposium on Principles of Programming Languages. ACM Press, New York (1988)

  22. Snelting, G., Robschnik, T., Krinke, J.: Efficient path conditions in dependence graphs for software safety analysis. ACM Trans. Softw. Eng. Methodol (TOSEM) 15(4), 410–457 (2006)

    Article  Google Scholar 

  23. Steffen, B.: Characteristic formulae. In: Ausiello, G., Dezani-Ciancaglini, M., Rocca, S.R. (eds.) Automata, Languages and Programming. Lecture Notes in Computer Science, vol. 372, pp. 723–732. Springer, Berlin (1989)

    Chapter  Google Scholar 

  24. Steffen, B.: Unifying models. In: Reischuk, R., Morvan, M. (eds.) STACS 97. Lecture Notes in Computer Science, vol. 1200, pp. 1–20. Springer, Berlin (1997)

  25. Steffen, B., Claßen, A., Klein, M., Knoop, J., Margaria, T.: The fixpoint-analysis machine. In: Lee, I., Smolka, S.A. (eds.) CONCUR ’95: Concurrency Theory. Lecture Notes in Computer Science, vol. 962, pp. 72–87. Springer, Berlin (1995)

  26. Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation. In: International SPIN Symposium on Model Checking of Software (SPIN2013). LNCS, vol. 7976, pp. 341–357. Springer, Berlin (2013)

  27. Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation: synthesizing programs of realistic structure. Softw. Tools Technol. Transf. doi:10.1007/s10009-014-0336-z (2014)

  28. Steffen, B., Knoop, J.: Finite constants: characterizations of a new decidable set of constants. In: Kreczmar, A., Mirkowska, G. (eds.) Mathematical Foundations of Computer Science (MFCS’89). LNCS, vol. 379, pages 481–491. Springer, Berlin (1989)

  29. Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds.) Haifa Verification Conference. LNCS, vol. 4383, pp. 92–108. Springer, Berlin (2006)

Download references

Author information

Authors and Affiliations

  1. Chair for Programming Systems, Technische Universität Dortmund, 44227 , Dortmund, Germany

    Bernhard Steffen, Malte Isberner & Stefan Naujokat

  2. Carnegie Mellon University, Moffett Field, CA, USA

    Falk Howar

  3. Chair Service and Software Engineering, Universität Potsdam, 14482 , Potsdam, Germany

    Tiziana Margaria

Authors
  1. Bernhard Steffen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Falk Howar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malte Isberner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefan Naujokat
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tiziana Margaria
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Naujokat.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Steffen, B., Howar, F., Isberner, M. et al. Tailored generation of concurrent benchmarks. Int J Softw Tools Technol Transfer 16, 543–558 (2014). https://doi.org/10.1007/s10009-014-0339-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0339-9

Keywords

Search

Navigation