Abstract
Some Java libraries have underdetermined specifications that allow more than one correct output for the same input, e.g., an output array may have its elements in any order. While such specifications have a number of advantages (e.g., a library can change while still satisfying the specification), the non-determinism inherent in underdetermined specifications can lead to failures in client code that erroneously assumes behaviors based on the library implementation instead of only the specification. Our recent work introduced the NonDex approach for detecting such erroneous assumptions by checking client code against models of library methods, which encode all behaviors allowed by the specifications
We present NonDex for JPF, which includes JPF models for 11 methods from the Java standard library (i.e., all methods that JPF supports from the current methods in Non-Dex). We use these models to systematically explore state spaces of 46 tests from student homework submissions. Our experiments show several interesting results, which provide new insights into the complexity of exploring the behaviors of code that uses underdetermined APIs and the structure of state spaces that arise in the exploration, and provide basis for future work on better detecting faults in tests that invoke underdetermined APIs as well as developing tool support for writing and maintaining more robust test suites
- Ben Lambeth's GitHub. https://github.com/azy2.Google Scholar
- Checkstyle Pull Request. https://github.com/checkstyle/checkstyle/pull/3393.Google Scholar
- Class#getDeclaredFields Javadoc. https://docs.oracle.com/javase/8/docs/api/java/lang/Class.html#getDeclaredFields--.Google Scholar
- Class#getFields in Java PathFinder. http://babelfish.arc.nasa.gov/hg/jpf/jpf-core/file/0069194b1048/src/peers/gov/nasa/jpf/vm/JPF java lang Class.java#l580.Google Scholar
- M. B. Dwyer, S. Person, and S. Elbaum. Controlling factors in evaluating path-sensitive error detection techniques. In FSE, pages 92--104, 2006. Google ScholarDigital Library
- A. Gyori, B. Lambeth, A. Shi, O. Legunsen, and D. Marinov. NonDex: A tool for detecting and debugging wrong assumptions on Java API specifications. In FSE Demo 2016 to appear. Google ScholarDigital Library
- HashSet Javadoc. https://docs.oracle.com/javase/8/docs/api/java/util/HashSet.html.Google Scholar
- C. Huo and J. Clause. Improving oracle quality by detecting brittle assertions and unused inputs in tests. In FSE, pages 621--631, 2014. Google ScholarDigital Library
- JUnit 4.11 - What's new? Test execution order. http://randomallsorts.blogspot.com/2012/12/junit-411-whats-new-test-execution-order.html.Google Scholar
- D. Knuth. Seminumerical algorithms, the art of computer programming, vol. 2 1997.Google Scholar
- NonDex Source Code. https://github.com/TestingResearchIllinois/NonDex.Google Scholar
- R. Pelánek. Properties of state spaces and their applications. STTT, 10(5):443--454, 2008. Google ScholarDigital Library
- R. Sedgewick. Permutation generation methods. ACM CSUR, 9(2):137--164, 1977. Google ScholarDigital Library
- A. Shi, A. Gyori, O. Legunsen, and D. Marinov. Detecting assumptions on deterministic implementations of non-deterministic specifications. In ICST, pages 80--90, 2016. Google ScholarCross Ref
Index Terms
- Exploring Underdetermined Specifications using Java PathFinder
Recommendations
NonDex: a tool for detecting and debugging wrong assumptions on Java API specifications
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software EngineeringWe present NonDex, a tool for detecting and debugging wrong assumptions on Java APIs. Some APIs have underdetermined specifications to allow implementations to achieve different goals, e.g., to optimize performance. When clients of such APIs assume ...
Finding Polluter Tests Using Java PathFinder
Tests that modify (i.e., "pollute") the state shared among tests in a test suite are called \polluter tests". Finding these tests is im- portant because they could result in di erent test outcomes based on the order of the tests in the test suite. Prior ...
State extensions for java pathfinder
ICSE '08: Proceedings of the 30th international conference on Software engineeringJava PathFinder (JPF) is an explicit-state model checker for Java programs. JPF implements a backtrackable Java Virtual Machine (JVM) that provides non-deterministic choices and control over thread scheduling. JPF is itself implemented in Java and runs ...
Comments