research-article

Synthesizing racy tests

Authors:

Malavika Samak,

Murali Krishna Ramanathan,

Suresh JagannathanAuthors Info & Claims

PLDI '15: Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation

Pages 175 - 185

https://doi.org/10.1145/2737924.2737998

Published: 03 June 2015 Publication History

Abstract

Subtle concurrency errors in multithreaded libraries that arise because of incorrect or inadequate synchronization are often difficult to pinpoint precisely using only static techniques. On the other hand, the effectiveness of dynamic race detectors is critically dependent on multithreaded test suites whose execution can be used to identify and trigger races. Usually, such multithreaded tests need to invoke a specific combination of methods with objects involved in the invocations being shared appropriately to expose a race. Without a priori knowledge of the race, construction of such tests can be challenging. In this paper, we present a lightweight and scalable technique for synthesizing precisely these kinds of tests. Given a multithreaded library and a sequential test suite, we describe a fully automated analysis that examines sequential execution traces, and produces as its output a concurrent client program that drives shared objects via library method calls to states conducive for triggering a race. Experimental results on a variety of well-tested Java libraries yield 101 synthesized multithreaded tests in less than four minutes. Analyzing the execution of these tests using an off-the-shelf race detector reveals 187 harmful races, including several previously unreported ones. Our implementation, named NARADA, and the results of our experiments are available at http://www.csa.iisc.ernet.in/~sss/tools/narada.

References

[1]

D. F. Bacon, R. E. Strom, and A. Tarafdar. Guava: A dialect of Java without data races. In Proceedings of the 15th ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications, OOPSLA ’00, pages 382–400, 2000.

Digital Library

[2]

E. Bodden and K. Havelund. Aspect-oriented race detection in Java. IEEE Transactions on Software Engineering, 36(4):509–527, 2010.

Digital Library

[3]

S. Burckhardt, P. Kothari, M. Musuvathi, and S. Nagarakatte. A randomized scheduler with probabilistic guarantees of finding bugs. In Proceedings of the Fifteenth Edition of ASPLOS on Architectural Support for Programming Languages and Operating Systems, ASPLOS XV, pages 167–178, 2010.

Digital Library

[4]

C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation, OSDI’08, pages 209–224, 2008.

Digital Library

[5]

D. Engler and K. Ashcraft. RacerX: Effective, static detection of race conditions and deadlocks. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), pages 237–252, 2003.

Digital Library

[6]

M. Eslamimehr and J. Palsberg. Race directed scheduling of concurrent programs. In Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’14, pages 301–314, 2014.

Digital Library

[7]

C. Flanagan and S. N. Freund. Fasttrack: Efficient and precise dynamic race detection. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’09, pages 121–133, 2009.

Digital Library

[8]

P. Godefroid, N. Klarlund, and K. Sen. Dart: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’05, pages 213–223, 2005.

Digital Library

[9]

V. Jagannath, M. Gligoric, D. Jin, Q. Luo, G. Rosu, and D. Marinov. Improved multithreaded unit testing. In Proceedings of the 19th ACM SIGSOFT Symposium and the European Conference on Foundations of Software Engineering, ESEC/FSE ’11, pages 223–233, 2011.

Digital Library

[10]

G. Jin, W. Zhang, D. Deng, B. Liblit, and S. Lu. Automated concurrency-bug fixing. In Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation, OSDI’12, pages 389–400, 2012.

Digital Library

[11]

P. Joshi, C.-S. Park, K. Sen, and M. Naik. A randomized dynamic program analysis technique for detecting real deadlocks. In Proceedings of the 2009 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’09, pages 110–120, 2009.

Digital Library

[12]

S. McPeak, C.-H. Gros, and M. K. Ramanathan. Scalable and incremental software bug detection. In Proceedings of the ACM SIGSOFT Symposium on Foundations of Software Engineering, ESEC/FSE 2013, pages 554–564, 2013.

Digital Library

[13]

M. Musuvathi and S. Qadeer. Iterative context bounding for systematic testing of multithreaded programs. In Proceedings of the 2007 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’07, pages 446–455, 2007.

Digital Library

[14]

S. Nagarakatte, S. Burckhardt, M. M. Martin, and M. Musuvathi. Multicore acceleration of priority-based schedulers for concurrency bug detection. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, pages 543–554, 2012.

Digital Library

[15]

M. Naik, A. Aiken, and J. Whaley. Effective static race detection for Java. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2006.

Digital Library

[16]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering, ICSE ’07, pages 75–84, 2007.

Digital Library

[17]

D. Perelman, S. Gulwani, D. Grossman, and P. Provost. Test-driven synthesis. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014.

Digital Library

[18]

B. Petrov, M. Vechev, M. Sridharan, and J. Dolby. Race detection for web applications. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, pages 251–262, 2012.

Digital Library

[19]

E. Pozniansky and A. Schuster. Efficient on-the-fly data race detection in multithreaded C++ programs. In Proceedings of the Ninth ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP ’03, pages 179–190, 2003.

Digital Library

[20]

M. Pradel and T. R. Gross. Fully automatic and precise detection of thread safety violations. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’12, pages 521–530, 2012.

Digital Library

[21]

P. Pratikakis, J. S. Foster, and M. Hicks. Locksmith: Context-sensitive correlation analysis for race detection. In Proceedings of the 2006 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’06, pages 320–331, 2006.

Digital Library

[22]

M. Samak and M. K. Ramanathan. Multithreaded test synthesis for deadlock detection. In Proceedings of the 2014 ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA ’14, pages 473–489, 2014.

Digital Library

[23]

M. Samak and M. K. Ramanathan. Trace driven dynamic deadlock detection and reproduction. In Proceedings of the 2014 ACM SIGPLAN Conference on Principles and Practices of Parallel Programming, PPoPP ’14, pages 29–42, 2014.

Digital Library

[24]

S. Savage, M. Burrows, G. Nelson, P. Sobalvarro, and T. Anderson. Eraser: a dynamic data race detector for multithreaded programs. ACM Trans. Comput. Syst., 15(4):391–411, Nov. 1997.

Digital Library

[25]

K. Sen. Race directed random testing of concurrent programs. In Proceedings of the 2008 ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’08, pages 11–21, 2008.

Digital Library

[26]

Y. Smaragdakis, J. Evans, C. Sadowski, J. Yi, and C. Flanagan. Sound predictive race detection in polynomial time. In Proceedings of the 39th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL ’12, pages 387–400, 2012.

Digital Library

[27]

R. Surendran, R. Raman, S. Chaudhuri, J. Mellor-Crummey, and V. Sarkar. Test-driven repair of data races in structured parallel programs. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014.

Digital Library

[28]

R. Vallee-Rai, E. Gagnon, L. Hendren, P. Lam, P. Pominville, and V. Sundaresan. Optimizing java bytecode using the soot framework: Is it feasible? In In International Conference on Compiler Construction, LNCS 1781, pages 18–34, 2000.

Digital Library

[29]

B. Xin, W. N. Sumner, and X. Zhang. Efficient program execution indexing. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2008.

Digital Library

[30]

J. Yu, S. Narayanasamy, C. Pereira, and G. Pokam. Maple: A coverage-driven testing tool for multithreaded programs. In Proceedings of the ACM Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA), pages 485–502, 2012.

Digital Library

Cited By

Kuliamin V(2024)A Survey of Software Dynamic Analysis MethodsProgramming and Computing Software10.1134/S036176882401007950:1(90-114)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1134/S0361768824010079
Stoica BLu SMusuvathi MNath SFedorova ANarayanan DDi Luna GQuerzoni L(2023)WAFFLE: Exposing Memory Ordering Bugs Efficiently with Active Delay InjectionProceedings of the Eighteenth European Conference on Computer Systems10.1145/3552326.3567507(111-126)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.1145/3552326.3567507
Zhao YWang ZLiu SSun JChen JChen X(2023)Achieving High MAP-Coverage Through Pattern Constraint ReductionIEEE Transactions on Software Engineering10.1109/TSE.2022.314448049:1(99-112)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3144480
Show More Cited By

Index Terms

Synthesizing racy tests
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
2. Theory of computation
  1. Semantics and reasoning
    1. Program semantics

Recommendations

Synthesizing racy tests
PLDI '15

Subtle concurrency errors in multithreaded libraries that arise because of incorrect or inadequate synchronization are often difficult to pinpoint precisely using only static techniques. On the other hand, the effectiveness of dynamic race detectors is ...
Synthesizing tests for detecting atomicity violations
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Using thread-safe libraries can help programmers avoid the complexities of multithreading. However, designing libraries that guarantee thread-safety can be challenging. Detecting and eliminating atomicity violations when methods in the libraries are ...
Race directed random testing of concurrent programs
PLDI '08

Bugs in multi-threaded programs often arise due to data races. Numerous static and dynamic program analysis techniques have been proposed to detect data races. We propose a novel randomized dynamic analysis technique that utilizes potential data race ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PLDI '15: Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation

June 2015

630 pages

ISBN:9781450334686

DOI:10.1145/2737924

General Chair:
David Grove
IBM Research, USA
,
Program Chair:
Steve Blackburn
Australian National University, Australia

ACM SIGPLAN Notices Volume 50, Issue 6
PLDI '15
June 2015
630 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/2813885
Editor:
Andy Gill
University of Kansas, Lawrence, KS
Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

PLDI '15

Sponsor:

SIGPLAN

PLDI '15: ACM SIGPLAN Conference on Programming Language Design and Implementation

June 13 - 17, 2015

OR, Portland, USA

Acceptance Rates

Overall Acceptance Rate 406 of 2,067 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
507
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kuliamin V(2024)A Survey of Software Dynamic Analysis MethodsProgramming and Computing Software10.1134/S036176882401007950:1(90-114)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1134/S0361768824010079
Stoica BLu SMusuvathi MNath SFedorova ANarayanan DDi Luna GQuerzoni L(2023)WAFFLE: Exposing Memory Ordering Bugs Efficiently with Active Delay InjectionProceedings of the Eighteenth European Conference on Computer Systems10.1145/3552326.3567507(111-126)Online publication date: 8-May-2023
https://dl.acm.org/doi/10.1145/3552326.3567507
Zhao YWang ZLiu SSun JChen JChen X(2023)Achieving High MAP-Coverage Through Pattern Constraint ReductionIEEE Transactions on Software Engineering10.1109/TSE.2022.314448049:1(99-112)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3144480
Chabbi MRamanathan MJhala RDillig I(2022)A study of real-world data races in GolangProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523720(474-489)Online publication date: 9-Jun-2022
https://dl.acm.org/doi/10.1145/3519939.3523720
Yavuz T(2022)SIFT: A Tool for Property Directed Symbolic Execution of Multithreaded Software2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00049(433-443)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00049
Gong SAltinbüken DFonseca PManiatis P(2021)SnowboardProceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles10.1145/3477132.3483549(66-83)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3477132.3483549
Manes VHan HHan CCha SEgele MSchwartz EWoo M(2021)The Art, Science, and Engineering of Fuzzing: A SurveyIEEE Transactions on Software Engineering10.1109/TSE.2019.294656347:11(2312-2331)Online publication date: 1-Nov-2021
https://doi.org/10.1109/TSE.2019.2946563
Terragni VPezzè M(2021)Statically driven generation of concurrent tests for thread‐safe classesSoftware Testing, Verification and Reliability10.1002/stvr.177431:4Online publication date: 4-May-2021
https://doi.org/10.1002/stvr.1774
Khanna DPurandare RSharma S(2020)Verifying and Testing Concurrent Programs using Constraint Solver based Approaches2020 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME46990.2020.00105(834-838)Online publication date: Sep-2020
https://doi.org/10.1109/ICSME46990.2020.00105
Li GLu SMusuvathi MNath SPadhye RBrecht TWilliamson C(2019)Efficient scalable thread-safety-violation detectionProceedings of the 27th ACM Symposium on Operating Systems Principles10.1145/3341301.3359638(162-180)Online publication date: 27-Oct-2019
https://dl.acm.org/doi/10.1145/3341301.3359638
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten