Skip to main content
SpringerLink
Log in

Generating effective tests for concurrent programs via AI automated planning techniques

  • VVPS-11
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Testing concurrent programs is a challenging problem due to interleaving explosion: even for a fixed set of inputs, there is a huge number of concurrent runs that need to be tested to account for scheduler behavior. Testing all possible schedules is not practical. Consequently, most effective testing algorithms only test a select subset of runs. For example, limiting testing to runs that contain data races or atomicity violations has been shown to capture a large proportion of concurrency bugs. In this paper we present a general approach to concurrent program testing that is based on techniques from artificial intelligence (AI) automated planning. We propose a framework for predicting concurrent program runs that violate a collection of generic correctness specifications for concurrent programs, namely runs that contain data races, atomicity violations, or null-pointer dereferences. Our prediction is based on observing an arbitrary run of the program, and using information collected from this run to model the behavior of the program, and to predict new runs that contain bugs with one of the above noted violation patterns. We characterize the problem of predicting such new runs as an AI sequential planning problem with the temporally extended goal of achieving a particular violation pattern. In contrast to many state-of-the-art approaches, in our approach feasibility of the predicted runs is guaranteed and, therefore, all generated runs are fully usable for testing. Moreover, our planning-based approach has the merit that it can easily accommodate a variety of violation patterns which serve as the selection criteria for guiding search in the state space of concurrent runs. This is achieved by simply modifying the planning goal. We have implemented our approach using state-of-the-art AI planning techniques and tested it within the Penelope concurrent program testing framework [35]. Nevertheless, the approach is general and is amenable to a variety of program testing frameworks. Our experiments with a benchmark suite showed that our approach is very fast and highly effective, finding all known bugs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. A data race occurs when two threads access a shared memory location at the same time and at least one of the accesses is a write access.

  2. A code unit is not atomic if it is interrupted during an execution by statements from another thread, and the interaction cannot be ruled out as harmless by presenting an equivalent execution in which this interruption does not occur.

  3. http://www.alphaworks.ibm.com/tech/contest/.

References

  1. Baier J.A., McIlraith, S.A.: Planning with first-order temporally extended goals using heuristic search. In: Proceedings of the 21st National Conference on Artificial Intelligence, vol. 1, AAAI’06, pp. 788–795. AAAI Press, Menlo Park (2006)

  2. Chen, F., Roşu, G.: Parametric and sliced causality. In: Proceedings of the 19th International Conference on Computer Aided Verification, CAV’07, pp. 240–253. Springer, Berlin (2007)

  3. Feng, C., Serbanuta, T., Rosu, G.: jPredictor: a predictive runtime analysis tool for java. In: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, pp. 221–230. ACM, New York (2008)

  4. Christiaens, M., De Bosschere, K.: TRaDe, a topological approach to on-the-fly race detection in java programs. In: Proceedings of the 2001 Symposium on JavaTM Virtual Machine Research and Technology Symposium, vol. 1, pp. 15. JVM’01. USENIX Association, Berkeley (2001)

  5. Dinning, A., Schonberg, E.: An empirical comparison of monitoring algorithms for access anomaly detection. SIGPLAN Notice 25, 1–10 (1990)

    Article  Google Scholar 

  6. Azadeh, F., Madhusudan, P., Razavi, N., Sorrentino, F.: Predicting null-pointer dereferences in concurrent programs. In: Proceedings of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’12, pp. 47–56 (2012)

  7. Flanagan, C., Freund, S.N.: Atomizer: a dynamic atomicity checker for multithreaded programs. SIGPLAN Notice 39(1), 256–267 (2004)

    Article  Google Scholar 

  8. Flanagan, C., Freund, S.N.: FastTrack: efficient and precise dynamic race detection. Commun. ACM 53, 93–101 (2010)

    Article  Google Scholar 

  9. Flanagan, C., Freund, S.N.: Velodrome: a sound and complete dynamic atomicity checker for multithreaded programs. SIGPLAN Notice 43, 293–303 (2008)

    Article  Google Scholar 

  10. Gerevini, A., Haslum, P., Long, D., Saetti, A., Dimopoulos, Y.: Deterministic planning in the fifth international planning competition: PDDL3 and experimental evaluation of the planners. Artif. Intell. 173(5–6), 619–668 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  11. Gregory, P., Long, D., Fox, M., Christopher Beck, J.: Planning modulo theories: extending the planning paradigm. In: Proceedings of the 22nd International Conference on Automated Planning and Scheduling, ICAPS2012, pp. 65–73 (2012)

  12. Groce, A., Visser, W.: Model checking java programs using structural heuristics. In: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA ’02, pp. 12–21 (2002)

  13. Haslum, P., Grastien, A.: Diagnosis as planning: two case studies. In: Proceedings of the International Scheduling and Planning Applications, Workshop, SPARK’11, pp. 37–44 (2011)

  14. Hoffmann, J.: FF: the fast-forward planning system. AI Mag. 22, 57–62 (2001)

    Google Scholar 

  15. Hoffmann, J.: The metric-FF planning system: translating ”ignoring delete lists” to numeric state variables. J. Artif. Intell. Res. (JAIR) 20, 291–341 (2003)

    MATH  Google Scholar 

  16. Hoffmann, J., Gomes, C., Selman, B., Kautz, H.: SAT encodings of state-space reachability problems in numeric domains. In: Proceedings of the 20th International Joint Conference on Artificial Intelligence, vol. 7, pp. 1918–1923, IJCAI’07 (2007)

  17. Kautz, H.A., Selman, B.: Unifying SAT-based and graph-based planning. In: Proceedings of the Sixteenth International Joint Conference on, Artificial Intelligence, IJCAI’99, pp. 318–325 (1999)

  18. Lamport, L.: Ti clocks, and the ordering of events in a distributed system. Commun. ACM 21, 558–565 (1978)

    Article  MATH  Google Scholar 

  19. Marino, D., Musuvathi, M., Narayanasamy, S.: LiteRace: effective sampling for lightweight data-race detection. SIGPLAN Notice 44(6), 134–143 (2009)

    Article  Google Scholar 

  20. McDermott Drew, V.: PDDL—the Planning domain definition language. Technical Report TR-98-003/DCS TR-1165, Yale Center for Computational Vision and Control (1998)

  21. Musuvathi, M., Qadeer, S.: Chess: systematic stress testing of concurrent software. In: Proceedings of the 16th International Conference on Logic-Based Program Synthesis and Transformation, LOPSTR’06, pp. 15–16. Springer, Berlin (2007)

  22. Nau, D., Ghallab, M., Traverso, P.: Automated Planning: Theory & Practice. Morgan Kaufmann, San Francisco (2004)

    Google Scholar 

  23. O’Callahan, R., Choi, J.: Hybrid dynamic data race detection. In: Proceedings of the 9th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’03, pp. 167–178. ACM, New York (2003)

  24. Park, C., Sen, K.: Randomized active atomicity violation detection in concurrent programs. In: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE’16, pp. 135–145. ACM, New York (2008)

  25. Park, S., Lu, S., Zhou, Y.: CTrigger: exposing atomicity violation bugs from their hiding places. SIGPLAN Notice 44(3), 25–36 (2009)

    Article  Google Scholar 

  26. Pnueli, A.: The temporal logic of programs. In: Proceedings of the 18th Annual Symposium on Foundations of Computer Science, SFCS ’77, pp. 46–57. IEEE Computer Society, Washington, DC (1977)

  27. Pozniansky, E., Schuster, A.: MultiRace: efficient on-the-fly data race detection in multithreaded C++ programs: research articles. Concurr. Comput. Pract. Exp. 19, 327–340 (2007)

    Article  Google Scholar 

  28. Rapps, S., Weyuker, E.J.: Selecting software test data using data flow information. IEEE Trans. Softw. Eng. 11(4), 367–375 (1985)

    Article  MATH  Google Scholar 

  29. Rintanen, J.: Planning with specialized SAT solvers. In: Proceedings of the 25th AAAI Conference on Artificial Intelligence, AAAI’11, pp. 1563–1566 (2011)

  30. Said, M., Wang, C., Yang, Z., Sakallah, K.: Generating data race witnesses by an SMT-based analysis. In: Proceedings of the 3rd International Conference on NASA Formal Methods, NFM’11, pp. 313–327. Springer, Berlin (2011)

  31. Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., Anderson, T.: Eraser: a dynamic data race detector for multithreaded programs. ACM Trans. Comput. Syst. 15(4), 391–411 (1997)

    Article  Google Scholar 

  32. Serbanuta, T., Chen, F., Rosu, G.: Maximal causal models for sequentially consistent systems. In: Runtime Verification, RV’12, Lecture Notes in Computer Science (2012)

  33. Sinha, N., Wang, C.: Staged concurrent program analysis. In: Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’10, pp. 47–56 (2010)

  34. Sinha, N., Wang, C.: On interference abstractions. In: Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’11, pp. 423–434 (2011)

  35. Sorrentino, F., Farzan, A., Madhusudan, P.: Penelope: weaving threads to expose atomicity violations. In: Proceedings of the 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE ’10, pp. 37–46. ACM, New York (2010)

  36. von Praun, C., Gross, T.R.: Object race detection. In: Proceedings of the 16th ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA ’01, pp. 70–82. ACM, New York (2001)

  37. Wang, C., Limaye, R., Ganai, M., Gupta, A.: Trace-based symbolic analysis for atomicity violations. In: Proceedings of the 16th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, TACAS’10, pp. 328–342. Springer, Berlin (2010)

  38. Wang, L., Stoller, S.D.: Accurate and efficient runtime detection of atomicity errors in concurrent programs. In: Proceedings of the 11th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’06, pp. 137–146. ACM, New York (2006)

  39. Wang, L., Stoller, S.D.: Runtime analysis of atomicity for multi-threaded programs. IEEE Trans. Softw. Eng. 32, 93–110 (2006)

    Article  Google Scholar 

  40. Yi, J., Sadowski, C., Flanagan, C.: SideTrack: generalizing dynamic atomicity analysis. In: Proceedings of the 7th Workshop on Parallel and Distributed Systems: Testing, Analysis, and Debugging, PADTAD ’09, pp. 8:1–8:10. ACM, New York (2009)

  41. Yu, Y., Rodeheffer, T., Chen, W.: RaceTrack: efficient detection of data race conditions via adaptive tracking. SIGOPS Oper. Syst. Rev. 39(5), 221–234 (2005)

    Article  Google Scholar 

Download references

Acknowledgments

The authors gratefully acknowledge funding from the Natural Sciences and Engineering Research Council of Canada (NSERC).

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Toronto, Toronto, Canada

    Niloofar Razavi, Azadeh Farzan & Sheila A. McIlraith

Authors
  1. Niloofar Razavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azadeh Farzan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheila A. McIlraith
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Niloofar Razavi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Razavi, N., Farzan, A. & McIlraith, S.A. Generating effective tests for concurrent programs via AI automated planning techniques. Int J Softw Tools Technol Transfer 16, 49–65 (2014). https://doi.org/10.1007/s10009-013-0277-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-013-0277-y

Keywords

Search

Navigation