Skip to main content
SpringerLink
Log in

Fast detection of concurrency errors by state space traversal with randomization and early backtracking

  • Regular Paper
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

State space traversal is a very popular approach to detect concurrency errors and test concurrent programs. However, it is not practically feasible for complex programs with many thread interleavings and a large state space. Many techniques explore only a part of the state space in order to find errors quickly—building upon the observation that errors can often be found in a particular small part of the state space. Great improvements in performance have been achieved also through randomization. In the context of this research direction, we present the DFS-RB algorithm that augments the standard algorithm for depth-first traversal with early backtracking. Specifically, it is possible to backtrack early from a state before all outgoing transitions have been explored. The DFS-RB algorithm is non-deterministic—it uses random numbers, together with values of several parameters, to determine when and how early backtracking takes place in the search. To evaluate DFS-RB, we performed a large experimental study with our prototype implementation in Java Pathfinder on several Java programs. The results show that DFS-RB achieves better performance in terms of speed and error detection than many state-of-the-art techniques for many benchmarks in our set. Nevertheless, it is difficult to find a single configuration of DFS-RB that works well for many different benchmarks. We designed a ranking algorithm whose purpose is to identify configurations that yield overall consistently good performance with a small variation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. We used the release of JPF corresponding to the commit number 715 in the repository for JPF v6.

  2. We used the version defined by the commit number 29 in the repository that contains JPF v8.

References

  1. Abdulla, P., Aronis, S., Jonsson, B., Sagonas, K.: Optimal dynamic partial order reduction. In: Proceedings of POPL. ACM (2014)

  2. Barnat, J., Brim, L., Rockai, P.: On-the-fly parallel model checking algorithm that is optimal for verification of weak LTL properties. Sci. Comput. Program. 77(12), 1272–1288 (2012)

    Article  MATH  Google Scholar 

  3. Behrmann, G., Hune, T., Vaandrager, F.: Distributing timed model checking—how the search order matters. In: Proceedings of CAV, LNCS, vol. 1855 (2000)

  4. Biere, A.: PicoSAT essentials. J. Satisf. Boolean Model. Comput. (JSAT) 4, 75–97 (2008)

    MATH  Google Scholar 

  5. Bisiani, R.: Beam Search. Encyclopedia of Artificial Intelligence. Wiley, New York (1992)

    Google Scholar 

  6. Burckhardt, S., Kothari, P., Musuvathi, M., Nagarakatte, S.: A Randomized scheduler with probabilistic guarantees of finding bugs. In: Proceedings of ASPLOS. ACM (2010)

  7. Ciardo, G., Gluckman, J., Nicol, D.: Distributed state space generation of discrete-state stochastic models. INFORMS J. Comput. 10(1), 82–93 (1998)

    Article  Google Scholar 

  8. Coons, K.E., Burckhardt, S., Musuvathi, M.: GAMBIT: effective unit testing for concurrency libraries. In: Proceedings of PPoPP. ACM (2010)

  9. Dwyer, M.B., Elbaum, S.G., Person, S., Purandare, R.: Parallel randomized state-space search. In: Proceedings of ICSE, IEEE CS (2007)

  10. Dwyer, M., Hatcliff, J., Robby, Ranganath, V.: Exploiting object escape and locking information in artial-order reductions for concurrent object-oriented programs. Formal Methods Syst. Des. 25(2–3), 199–240 (2004)

    Article  MATH  Google Scholar 

  11. Dwyer, M.B., Person, S., Elbaum, S.G.: Controlling factors in evaluating path-sensitive error detection techniques. In: Proceedings of SIGSOFT FSE. ACM (2006)

  12. Edelkamp, S., Leue, S., Lluch-Lafuente, A.: Directed explicit-state model checking in the validation of communication protocols. Int. J. Softw. Tools Technol. Transf. 5(2–3), 247–267 (2004)

    Article  MATH  Google Scholar 

  13. Edelkamp, S., Schuppan, V., Bosnacki, D., Wijs, A., Fehnker, A., Aljazzar, H.: Survey on directed model checking. In: Proceedings of 5th International Workshop on Model Checking and Artificial Intelligence, LNCS, vol. 5348 (2008)

  14. Een, N., Sorensson, N.: An extensible SAT-solver. In: Proceedings of SAT, LNCS, vol. 2919 (2003)

  15. Emmi, M., Qadeer, S., Rakamaric, Z.: Delay-bounded scheduling. In: Proceedings of POPL. ACM (2011)

  16. Engels, T.A.N., Groote, J.F., van Weerdenburg, M.J., Willemse, T.A.C.: Search algorithms for automated validation. J. Logic Algebr. Program. 78(4), 274–287 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  17. Farzan, A., Holzer, A., Razavi, N., Veith, H.: Con2colic testing. In: Proceedings of ESEC/FSE. ACM (2013)

  18. Flanagan, C., Godefroid, P.: Dynamic partial-order reduction for model checking software. In: Proceedings of POPL. ACM (2005)

  19. Godefroid, P.: Partial-order methods for the verification of concurrent systems. In: LNCS, vol. 1032 (1996)

  20. Gomes, C., Selman, B., Kautz, H.: Boosting combinatorial search through randomization. In: Proceedings of AAAI (1998)

  21. Groce, A., Visser, W.: Heuristics for model checking Java programs. Int. J. Softw. Tools Technol. Transf. 6(4), 260–276 (2004)

    Article  MATH  Google Scholar 

  22. Gueta, G., Flanagan, C., Yahav, E., Sagiv, M.: Cartesian partial-order reduction. In: Proceedings of SPIN, LNCS, vol. 4595 (2007)

  23. Holzmann, G.J., Bosnacki, D.: The design of a multicore extension of the SPIN model checker. IEEE Trans. Softw. Eng. 33(10), 659–674 (2007)

    Article  Google Scholar 

  24. Holzmann, G.J., Joshi, R., Groce, A.: Tackling large verification problems with the swarm tool. In: Proceedings of SPIN, LNCS, vol. 5156 (2008)

  25. Holzmann, G.J., Joshi, R., Groce, A.: Swarm verification. In: Proceedings of ASE, IEEE CS (2008)

  26. Jagannath, V., Kirn, M., Lin, Y., Marinov, D.: Evaluating machine-independent metrics for state-space exploration. In: Proceedings of ICST, IEEE CS (2012)

  27. Jones, M., Mercer, E.: Explicit state model checking with hopper. In: Proceedings of SPIN, LNCS, vol. 2989 (2004)

  28. Jones, M., Sorber, J.: Parallel search for LTL violations. Int. J. Softw. Tools Technol. Transf. 7(1), 31–42 (2005)

    Article  Google Scholar 

  29. Kalibera, T., Hagelberg, J., Pizlo, F., Plsek, A., Titzer, B., Vitek, J.: CDx: a family of real-time Java benchmarks. In: Proceedings of JTRES. ACM (2009)

  30. Laarman, A., Langerak, R., van de Pol, J., Weber, M., Wijs, A.: Multi-core nested depth-first search. In: Proceedings of ATVA, LNCS, vol. 6996 (2011)

  31. Laarman, A., van de Pol, J., Weber, M.: Boosting multi-core reachability performance with shared hash tables. In: Proceedings of FMCAD. IEEE (2010)

  32. Lerda, F., Sisto, R.: Distributed-memory model checking with SPIN. In: Proceedings of SPIN, LNCS, vol. 1680 (1999)

  33. Luby, M., Sinclair, A., Zuckerman, D.: Optimal speedup of Las Vegas algorithms. Inf. Process. Lett. 47(4), 173–180 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  34. van Moorsel, A.P.A., Wolter, K.: Analysis of restart mechanisms in software systems. IEEE Trans. Softw. Eng. 32(8), 547–558 (2006)

    Article  Google Scholar 

  35. Musuvathi, M., Qadeer, S.: Iterative context bounding for systematic testing of multithreaded programs. In: Proceedings of PLDI. ACM (2007)

  36. Musuvathi, M., Qadeer, S., Ball, T., Basler, G., Nainar, P.A., Neamtiu, I.: Finding and reproducing heisenbugs in concurrent programs. In: Proceedings of OSDI, USENIX (2008)

  37. Parízek, P., Kalibera, T.: Efficient detection of errors in Java components using random environment and restarts. In: Proceedings of TACAS, LNCS, vol. 6015 (2010)

  38. Parízek, P., Lhoták, O.: Randomized backtracking in state space traversal. In: Proceedings of SPIN, LNCS, vol. 6823 (2011)

  39. Parízek, P., Lhoták, O.: Identifying future field accesses in exhaustive state space traversal. In: Proceedings of ASE, IEEE CS (2011)

  40. Parízek, P.: Hybrid analysis for partial order reduction of programs with arrays. In: Proceedings of VMCAI, LNCS, vol. 9583 (2016)

  41. Parízek, P.: Fast error detection with hybrid analyses of future accesses. In: Proceedings of SAC, MUSEPAT Track. ACM (2016)

  42. Qadeer, S.: Daisy File System. Joint CAV/ISSTA special event on specification, verification and testing of concurrent software (2004)

  43. Qadeer, S., Rehof, J.: Context-bounded model checking of concurrent software. In: Proceedings of TACAS, LNCS, vol. 3440 (2005)

  44. Rabinovitz, I., Grumberg, O.: Bounded model checking of concurrent programs. In: Proceedings of CAV, LNCS, vol. 3576 (2005)

  45. Renault, E., Duret-Lutz, A., Kordon, F., Poitrenaud, D.: Parallel explicit model checking for generalized Büchi automata. In: Proceedings of TACAS, LNCS, vol. 9035 (2015)

  46. Rungta, N., Mercer, E.: Generating counter-examples through randomized guided search. In: Proceedings of SPIN, LNCS, vol. 4595 (2007)

  47. Rungta, N., Mercer, E.: Clash of the titans: tools and techniques for hunting bugs in concurrent programs. In: Proceedings of PADTAD. ACM (2009)

  48. Sen, K.: Effective random testing of concurrent programs. In: Proceedings of ASE. ACM (2007)

  49. Seppi, K., Jones, M., Lamborn, P.: Guided model checking with a bayesian meta-heuristic. Fundamenta Informaticae 70(1–2), 111–126 (2006)

    MathSciNet  MATH  Google Scholar 

  50. Smith, L.A., Bull, J.M., Obdrzalek, J.: A parallel Java grande benchmark suite. Supercomputing. ACM (2001). https://www2.epcc.ed.ac.uk/computing/research_activities/java_grande/index_1.html

  51. Spearman, C.: The proof and measurement of association between two things. Am. J. Psychol. 15(1), 72–101 (1904)

    Article  Google Scholar 

  52. Stern, U., Dill, D.L.: Parallelizing the murphi verifier. In: Proceedings of CAV, LNCS, vol. 1254 (1997)

  53. Thomson, P., Donaldson, A.F., Betts, A.: Concurrency testing using schedule bounding: an empirical study. In: Proceedings of PPoPP. ACM (2014)

  54. Udupa, A., Desai, A., Rajamani, S.: Depth bounded explicit-state model checking. In: Proceedings of SPIN, LNCS, vol. 6823 (2011)

  55. Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)

    Google Scholar 

  56. Walsh, T.: Search in a small world. In: Proceedings of IJCAI, Morgan Kaufmann (1999)

  57. Wehrle, M., Kupferschmid, S., Podelski, A.: Transition-based directed model checking. In: Proceedings of TACAS, LNCS, vol. 5505 (2009)

  58. Wehrle, M., Kupferschmid, S.: Context-enhanced directed model checking. In: Proceedings of SPIN, LNCS, vol. 6349 (2010)

  59. Wijs, A., Bosnacki, D.: Many-core on-the-fly model checking of safety properties using GPUs. Int. J. Softw. Tools Technol. Transf. 18(2), 169–185 (2016)

    Article  Google Scholar 

  60. Wijs, A., Lisser, B.: Distributed extended beam search for quantitative model checking. In: Proceedings of MoChArt, LNCS, vol. 4428 (2006)

  61. Yang, Y., Chen, X., Gopalakrishnan, G.: Inspect: a runtime model checker for multithreaded C programs. Technical Report UUCS-08-004, University of Utah (2008)

  62. Concurrency tool comparison repository, https://facwiki.cs.byu.edu/vv-lab/index.php/Concurrency_Tool_Comparison

  63. Java pathfinder, http://babelfish.arc.nasa.gov/trac/jpf/

  64. jPapaBench, http://d3s.mff.cuni.cz/~malohlava/projects/jpapabench/

  65. Parallel Java benchmarks, https://bitbucket.org/psl-lab/pjbench

Download references

Acknowledgements

The first phase of this work was partially supported by the Czech Science Foundation project 14-11384S, and the second phase was partially supported by the Czech Science Foundation project 18-17403S. It was also partially supported by the Natural Sciences and Engineering Research Council of Canada.

Author information

Authors and Affiliations

  1. Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic

    Pavel Parízek

  2. David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, ON, Canada

    Ondřej Lhoták

Authors
  1. Pavel Parízek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ondřej Lhoták
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pavel Parízek.

A Results of experiments with twelve benchmarks

A Results of experiments with twelve benchmarks

Table 12 Results for Elevator
Full size table
Table 13 Results for Alarm Clock
Full size table
Table 14 Results for Linked List
Full size table
Table 15 Results for Producer Consumer
Full size table
Table 16 Results for RAX Extended
Full size table
Table 17 Results for Replicated Workers
Full size table
Table 18 Results for jPapaBench
Full size table
Table 19 Results for Monte Carlo
Full size table
Table 20 Results for CDx
Full size table
Table 21 Results for Cache4j
Full size table
Table 22 Results for QSortMT
Full size table

Here we provide Tables 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22 with concrete data on error detection performance that were used to create the graphs in Fig. 8. We created one table per benchmark to enable easy comparison of the error detection performance of different configurations and techniques on each benchmark. Each table is divided into three segments. The top segment contains the performance data for selected configurations of DFS-RB from the overall top 10 list (which reflects the combined score). We selected the configurations at positions 1, 2, 3, and 10 in the list in order to show the best ones while also covering the whole range. The middle segment of each table shows the performance data for the state-of-the-art techniques that we included in our experimental comparison. Finally, the bottom segment contains the data for the combination of the overall best configuration of DFS-RB with the state-of-the-art techniques.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Parízek, P., Lhoták, O. Fast detection of concurrency errors by state space traversal with randomization and early backtracking. Int J Softw Tools Technol Transfer 21, 365–400 (2019). https://doi.org/10.1007/s10009-018-0484-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-018-0484-7

Keywords

Search

Navigation