Abstract
Coverage metrics play a crucial role in testing. They allow one to estimate how well a program has been tested and/or to control the testing process. Several concurrency-related coverage metrics have been proposed, but most of them do not reflect concurrent behaviour accurately enough. In this paper, we propose several new metrics that are suitable primarily for saturation-based or search-based testing of concurrent software. Their distinguishing feature is that they are derived from various dynamic analyses designed for detecting synchronisation errors in concurrent software. In fact, the way these metrics are obtained is generic, and further metrics can be obtained in a similar way from other analyses. The underlying motivation is that, within such analyses, behavioural aspects crucial for occurrence of various bugs are identified, and hence it makes sense to track how well the occurrence of such phenomena is covered by testing. Next, coverage tasks of the proposed as well as some existing metrics are combined with an abstract identification of the threads participating in generation of the phenomena captured in the concerned tasks. This way, further, more precise metrics are obtained. Finally, an empirical evaluation of the proposed metrics, which confirms that several of them are indeed more suitable for saturation-based and search-based testing than the previously known metrics, is presented.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Bensalem, S., Havelund, K.: Dynamic Deadlock Analysis of Multi-threaded Programs. In: Ur, S., Bin, E., Wolfsthal, Y. (eds.) HVC 2005. LNCS, vol. 3875, pp. 208–223. Springer, Heidelberg (2006)
Bron, A., Farchi, E., Magid, Y., Nir, Y., Ur, S.: Applications of Synchronization Coverage. In: Proc. of PPoPP 2005. ACM Press (2005)
Clarke, E., Grumberg, O., Peled, D.: Model Checking. MIT Press (1999)
Edelstein, O., Farchi, E., Goldin, E., Nir, Y., Ratsaby, G., Ur, S.: Framework for Testing Multi-threaded Java Programs. Concurrency and Computation: Practice and Experience 15(3-5), 485–499 (2003)
Elmas, T., Qadeer, S., Tasiran, S.: Goldilocks: A Race and Transaction-aware Java Runtime. In: Proc. of PLDI 2007. ACM Press (2007)
Hovemeyer, D., Pugh, W.: Finding Concurrency Bugs in Java. In: Proc. of PODC 2004. ACM Press (2004)
Joshi, P., Park, C.-S., Sen, K., Naik, M.: A Randomized Dynamic Program Analysis Technique for Detecting Real Deadlocks. In: Proc. of PLDI 2009. ACM Press (2009)
Kahlon, V., Yang, Y., Sankaranarayanan, S., Gupta, A.: Fast and Accurate Static Data-Race Detection for Concurrent Programs. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 226–239. Springer, Heidelberg (2007)
Křena, B., Letko, Z., Vojnar, T., Ur, S.: A Platform for Search-based Testing of Concurrent Software. In: Proc. of PADTAD 2010. ACM Press (2010)
Lu, S., Tucek, J., Qin, F., Zhou, Y.: Avio: Detecting Atomicity Violations via Access Interleaving Invariants. In: Proc. of ASPLOS 2006. ACM Press (2006)
Manson, J., Pugh, W., Adve, S.V.: The Java Memory Model. In: Proc. of POPL 2005. ACM Press (2005)
McMinn, P.: Search-based Software Test Data Generation: A Survey: Research Articles. Software Testing, Verification, and Reliability 14(2), 105–156 (2004)
Musuvathi, M., Qadeer, S., Ball, T., Basler, G., Nainar, P.A., Neamtiu, I.: Finding and Reproducing Heisenbugs in Concurrent Programs. In: Proc. of OSDI 2008. USENIX Association (2008)
Pozniansky, E., Schuster, A.: Efficient On-the-fly Data Race Detection in Multithreaded C++ Programs. In: Proc. of PPoPP 2003. ACM Press (2003)
Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., Anderson, T.: Eraser: A Dynamic Data Race Detector for Multi-threaded Programs. In: Proc. of SOSP 1997. ACM Press (1997)
Sherman, E., Dwyer, M.B., Elbaum, S.: Saturation-based Testing of Concurrent Programs. In: Proc. of ESEC/FSE 2009. ACM Press (2009)
Soriano, J., Jimenez, M., Cantera, J.M., Hierro, J.J.: Delivering Mobile Enterprise Services on Morfeo’s MC Open Source Platform. In: Proc. of MDM 2006. IEEE CS (2006)
Trainin, E., Nir-Buchbinder, Y., Tzoref-Brill, R., Zlotnick, A., Ur, S., Farchi, E.: Forcing Small Models of Conditions on Program Interleaving for Detection of Concurrent Bugs. In: Proc. of PADTAD 2009. ACM Press (2009)
von Praun, C., Gross, T.R.: Object Race Detection. In: Proc. of OOPSLA 2001. ACM Press (2001)
Williams, A., Thies, W., Ernst, M.D.: Static Deadlock Detection for Java Libraries. In: Gao, X.-X. (ed.) ECOOP 2005. LNCS, vol. 3586, pp. 602–629. Springer, Heidelberg (2005)
Yang, C.-S.D., Souter, A.L., Pollock, L.L.: All-du-path Coverage for Parallel Programs. In: Proc. of ISSTA 1998. ACM Press (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Křena, B., Letko, Z., Vojnar, T. (2012). Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software. In: Khurshid, S., Sen, K. (eds) Runtime Verification. RV 2011. Lecture Notes in Computer Science, vol 7186. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29860-8_14
Download citation
DOI: https://doi.org/10.1007/978-3-642-29860-8_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29859-2
Online ISBN: 978-3-642-29860-8
eBook Packages: Computer ScienceComputer Science (R0)