Tristan Allwood
Cristian Cadar
ABSTRACT
This paper presents a new lightweight technique for automatically generating high coverage test suites for Haskell library code. Our approach combines four main features to increase test coverage: (1) automatically inferring the constructors and functions needed to generate test data; (2) using needed narrowing to take advantage of Haskell's lazy evaluation semantics; (3) inspecting elements inside returned data structures through the use of case statements, and (4) efficiently handling polymorphism by lazily instantiating all possible instances.
We have implemented this technique in IRULAN, a fully automatic tool for systematic black-box unit testing of Haskell library code. We have designed IRULAN to generate high coverage test suites and detect common programming errors in the process. We have applied IRULAN to over 50 programs from the spectral and real suites of the nofib benchmark and show that it can effectively generate high-coverage test suites---exhibiting 70.83% coverage for spectral and 59.78% coverage for real---and find errors in these programs.
Our techniques are general enough to be useful for several other types of testing, and we also discuss our experience of using IRULAN for property and regression testing.
- T. Allwood, S. Peyton Jones, and S. Eisenbach. Finding the Needle: Stack Traces for GHC. In Haskell, 2009. Google Scholar
Digital Library
- A. Andoni, D. Daniliuc, S. Khurshid, and D. Marinov. Evaluating the "Small Scope Hypothesis", 2002.Google Scholar
- Antoy, Echahed, and Hanus. A Needed Narrowing Strategy. JACM: Journal of the ACM, 47, 2000. Google ScholarDigital Library
- P. Boonstoppel, C. Cadar, and D. Engler. RWset: Attacking Path Explosion in Constraint-Based Test Generation. In TACAS, 2008. Google ScholarDigital Library
- C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In ISSTA, 2002. Google ScholarDigital Library
- C. Cadar, D. Dunbar, and D. R. Engler. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In OSDI, 2008. Google ScholarDigital Library
- K. Claessen and J. Hughes. QuickCheck: A lightweight tool for random testing of Haskell programs. In ICFP, 2000. Google ScholarDigital Library
- D. C. Cooper. Theorem proving in arithmetic without multiplication. In Machine Intelligence, 1972.Google Scholar
- S. Fischer and H. Kuchen. Systematic generation of glass-box test cases for functional logic programs. In PPDP, 2007. Google ScholarDigital Library
- J. E. Forrester and B. P. Miller. An empirical study of the robustness of windows NT applications using random testing, 2000.Google Scholar
- A. Gill and C. Runciman. Haskell Program Coverage. In Haskell, 2007. Google ScholarDigital Library
- P. Godefroid, M. Y. Levin, and D. A. Molnar. Automated Whitebox Fuzz Testing. In NDSS, 2008.Google Scholar
- B. Korel. Automated test data generation for programs with procedures. In ISSTA, 1996. Google ScholarDigital Library
- F. Lindblad. Property directed generation of first-order test data. In TFP, 2007.Google Scholar
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. CACM, 1990. Google ScholarDigital Library
- N. Mitchell and C. Runciman. A Static Checker for Safe Pattern Matching in Haskell. In TFP, 2007.Google Scholar
- N. Mitchell and C. Runciman. Not All Patterns, But Enough - an automatic verifier for partial but sufficient pattern matching. In Haskell Symposium, 2008. Google ScholarDigital Library
- C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classification of test inputs. In ECOOP 2005. Google ScholarDigital Library
- C. Pacheco, S. Lahiri, M. Ernst, and T. Ball. Feedback-directed random test generation. Technical Report MSR-TR-2006-125, Microsoft Research, 2006.Google Scholar
- W. Partain. The nofib benchmark suite of Haskell programs. In Functional Programming, Workshops in Computing, 1992. Google ScholarDigital Library
- S. L. Peyton Jones. The Implementation of Functional Programming Languages. Prentice--Hall, 1987. Google ScholarDigital Library
- C. Runciman, M. Naylor, and F. Lindblad. SmallCheck and Lazy SmallCheck: Automatic exhaustive testing for small values. In Haskell Symposium, 2008. Google ScholarDigital Library
- W. Visser, C. S. Pasareanu, and S. Khurshid. Test input generation with Java PathFinder. In ISSTA, 2004. Google ScholarDigital Library
- J. Yang, P. Twohey, D. Engler, and M. Musuvathi. Using Model Checking to Find Serious File System Errors. In OSDI, 2004. Google ScholarDigital Library
Index Terms
- High coverage testing of Haskell programs
Recommendations
MuCheck: an extensible tool for mutation testing of haskell programs
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and AnalysisThis paper presents MuCheck, a mutation testing tool for Haskell programs. MuCheck is a counterpart to the widely used QuickCheck random testing tool for functional programs, and can be used to evaluate the efficacy of QuickCheck property definitions. ...
Selection and Prioritization of Test Cases by Combining White-Box and Black-Box Testing Methods
ECBS-EERC '13: Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based SystemsIn this paper, we present a methodology that combines both white-box and black-box testing, in order to improve testing quality for a given class of embedded systems. The goal of this methodology is generation of test cases for the new functional ...
Multi-objective black-box test case selection for system testing
GECCO '17: Proceedings of the Genetic and Evolutionary Computation ConferenceTesting is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression ...
Comments