Abstract
Since test cases cannot be exhaustive, any effective test case generation strategy must identify the execution states most likely to uncover bugs. The key issue is to define criteria for selecting such interesting states.
If the units being tested are classes in object-oriented programming, it seems attractive to rely on the boolean queries present in each class, which indeed define criteria on the states of the corresponding objects, and — in contract-equipped O-O software — figure prominently in preconditions, postconditions and invariants. As these queries are part of the class specification and hence relevant to its clients, one may conjecture that the resulting partition of the state space is also relevant for tests.
We explore this conjecture by examining whether relying on the boolean queries of a class to extract abstract states improves the results of black-box testing. The approach uses constraint-solving and proof techniques to generate objects that satisfy the class invariants, then performs testing by relying on postconditions as test oracles. The resulting strategy, in our experiments on library classes used in production software, finds significantly more bugs than random testing.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Barnettl, M., Rustan, K., Leinol, M., Schultel, W.: The Spec# programming system: An overview. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, Springer, Heidelberg (2005)
Ball, T.: A theory of predicate-complete test coverage and generation. In: 3rd International Symposium on Formal Methods for Components and Objects, pp. 1–22 (2004)
Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on Java predicates. In: ISSTA 2002. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 123–133. ACM Press, New York (2002)
Csallner, C., Smaragdakis, Y.: Check ’n’ crash: combining static checking and testing. In: Inverardi, P., Jazayeri, M. (eds.) ICSE 2005. LNCS, vol. 4309, pp. 422–431. Springer, Heidelberg (2006)
Ciupa, I., Leitner, A.: Automatic testing based on design by contract. In: Proceedings of Net.ObjectDays 2005 (6th Annual International Conference on Object-Oriented and Internet-based Technologies, Concepts and Applications for a Networked World), pp. 545–557 (2005)
Detlefs, D., Nelson, G., Saxe, J.B.: Simplify: A theorem prover for program checking. Technical Report HPL-2003-148, HP Labs (2003), http://research.compaq.com/SRC/esc/Simplify.html
Duran, J., Ntafos, S.: An evaluation of random testing. IEEE Transactions on Software Engineering SE-10, 438–444 (1984)
Hamie, A.: Towards verifying Java realization of OCL-constrained design models using JML. In: Proceedings of 6th IASTED International Conference on Software Engineering and Applications, ACTA Press, MIT, Cambridge, MA, USA (2002)
Hamlet, D., Taylor, R.: Partition testing does not inspire confidence. IEEE Transactions on Software Engineering 16(12), 1402–1411 (1990)
Hamlet, R.: Random testing. In: Marciniak, J. (ed.) Encyclopedia of Software Engineering, pp. 970–978. Wiley, Chichester (1994)
Jackson, D.: Alloy: Alightweight object modeling notation. ACM Trans. Soft. Eng. Methodology 11(2), 256–290 (2002)
Kramer, R.: iContract - the JavaTM design by contractTM tool. In: Proceedings of Object-Oriented Language and Systems, pp. 295–307. IEEE Computer Society, Washington, DC, USA (1998)
Leavens, G.T., Baker, A.L.: Enhancing the pre- and postcondition technique for more expressive specifications. In: World Congress on Formal Methods, pp. 1087–1106 (1999)
Leavens, G.T., Cheon, Y., Clifton, C., Ruby, C., Cok, D.R.: How the design of jml accommodates both runtime assertion checking and formal verification. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2002. LNCS, vol. 2852, pp. 262–284. Springer, Heidelberg (2003)
Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - A survey. In: Proc. The IEEE, pp. 1090–1123. IEEE Computer Society Press, Los Alamitos (1996)
Liu, L., Leitner, A., Offutt, J.: Using contracts to automate forward class testing. Journal of System and Software (submitted)
Meyer, B.: Reusable Software: The Base Object-Oriented Libraries. Prentice Hall, Englewood Cliffs (1994)
Meyer, B.: Object-Oriented Software Construction, 2nd edn. Prentice-Hall, Englewood Cliffs (1997)
Meyer, B.: Eiffel: The Language, Prentice Hall, 1991, revised edn. in progress at (2006), http://se.ethz.ch/~meyer/ongoing/etl/
Marinov, D., Khurshid, S.: TestEra: A novel framework for automated testing of Java programs. In: ASE. Proc. 16th IEEE International Conference on Automated Software Engineering, pp. 22–34. IEEE Computer Society Press, Los Alamitos (2001)
Nimmer, J.W., Ernst, M.D.: Invariant inference for static checking: An empirical evaluation. In: Daemen, J., Rijmen, V. (eds.) FSE 2002. LNCS, vol. 2365, pp. 11–20. Springer, Heidelberg (2002)
Ostrand, T.J., Balcer, M.J.: The Category-Partition method for specifying and generating functional test. Comm. ACM 31(6), 676–686 (1988)
Richtersl, M., Gogolla, M.: On formalizing the UML object constraint language OCL. In: Ling, T.-W., Ram, S., Lee, M.L. (eds.) Conceptual Modeling – ER 1998. LNCS, vol. 1507, Springer, Heidelberg, Singapore (1998)
SICStus Prolog User’s Manual, http://www.sics.se/sicstus/docs/latest/pdf/sicstus.pdf
Whaley, J., Martin, M.C., Lam, M.S.: Automatic extraction of object-oriented component interface. In: ISSTA 2002, pp. 218–228 (2002)
Xie, T., Notkin, D.: Automatic extraction of object-oriented observer abstractions from unit-test executions. In: Davies, J., Schulte, W., Barnett, M. (eds.) ICFEM 2004. LNCS, vol. 3308, pp. 290–305. Springer, Heidelberg (2004)
Yorsh, G., Ball, T., Sagiv, M.: Testing, abstraction, theorem proving: better together! In: ISSTA 2006, pp. 145-156 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Liu, L.(., Meyer, B., Schoeller, B. (2007). Using Contracts and Boolean Queries to Improve the Quality of Automatic Test Generation. In: Gurevich, Y., Meyer, B. (eds) Tests and Proofs. TAP 2007. Lecture Notes in Computer Science, vol 4454. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73770-4_7
Download citation
DOI: https://doi.org/10.1007/978-3-540-73770-4_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-73769-8
Online ISBN: 978-3-540-73770-4
eBook Packages: Computer ScienceComputer Science (R0)