Skip to main content
SpringerLink
Log in

TestEra: Specification-Based Testing of Java Programs Using SAT

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

TestEra is a framework for automated specification-based testing of Java programs. TestEra requires as input a Java method (in sourcecode or bytecode), a formal specification of the pre- and post-conditions of that method, and a bound that limits the size of the test cases to be generated. Using the method's pre-condition, TestEra automatically generates all nonisomorphic test inputs up to the given bound. It executes the method on each test input, and uses the method postcondition as an oracle to check the correctness of each output. Specifications are first-order logic formulae. As an enabling technology, TestEra uses the Alloy toolset, which provides an automatic SAT-based tool for analyzing first-order logic formulae. We have used TestEra to check several Java programs including an architecture for dynamic networks, the Alloy-alpha analyzer, a fault-tree analyzer, and methods from the Java Collection Framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Adjie-Winoto, W., Schwartz, E., Balakrishnan, H., and Lilley, J. 1999. The design and implementation of an intentional naming system. In Proc. 17th ACM Symposium on Operating Systems Principles (SOSP), Kiawah Island, Dec.

  • Aertryck, L.V., Benveniste, M., and Le Metayer, D. 1997. CASTING: A formally based software test generation method. In Proc. First IEEE International Conference on Formal Engineering Methods, Hiroshima, Japan, Nov.

  • Ball, T. and Rajamani, S.K. 2001. Automatically validating temporal safety properties of interfaces. In Proc. 8th International SPIN Workshop on Model Checking of Software, pp. 103–122.

  • Ball, T. and Rajamani, S.K. 2002. The SLAM project: Debugging system software via static analysis. In Proc. 29th Annual ACM Symposium on the Principles of Programming Languages (POPL), pp. 1–3.

  • Beck, K. and Gamma, E. 1998. Test infected: Programmers love writing tests. Java Report, 3(7), July.

  • Beizer, B. 1990. Software Testing Techniques. International Thomson Computer Press.

  • Boyapati, C., Khurshid, S., and Marinov, D. 2002. Korat: Automated testing based on Java predicates. In Proc. International Symposium on Software Testing and Analysis (ISSTA), July, pp. 123–133.

  • Chang, J. and Richardson, D.J. 1999. Structural specification-based testing: Automated support and experimental evaluation. In Proc. 7th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), Sep., pp. 285–302.

  • Cheon, Y. and Leavens, G.T. 2002. A simple and practical approach to unit testing: The JML and junit way. In Proc. European Conference on Object-Oriented Programming (ECOOP), June.

  • Corbett, J., Dwyer, M., Hatcliff, J., Pasareanu, C., Laubach, R.S., and Zheng, H. 2000. Bandera: Extracting finitestate models from Java source code. In Proc. 22nd International Conference on Software Engineering (ICSE), June.

  • Cormen, T.H., Leiserson, C.E., and Rivest, R.L. 1990. Introduction to Algorithms. Cambridge, MA: The MIT Press.

    Google Scholar 

  • Crawford, J., Ginsberg, M., Luks, E., and Roy, A. 1996. Symmetry-breaking predicates for search problems. In Proc. Fifth International Conference on Principles of Knowledge Representation and Reasoning.

  • Dahm, M. Byte code engineering library. http://bcel.sourceforge. net/.

  • Demartini, C., Iosif, R., and Sisto, R. 1999. A deadlock detection tool for concurrent Java programs. Software-Practice and Experience, July

  • Detlefs, D.L., Rustan, K., Leino, M., Nelson, G., and Saxe, J.B. 1998. Extended static checking. Research Report 159, Compaq Systems Research Center.

  • Dick, J. and Faivre, A. 1993. Automating the generation and sequencing of test cases from model-based specifi-cations. In Proc. Formal Methods Europe (FME), pp. 268–284.

  • Donat, M.R. 1997. Automating formal specification based testing. In Proc. Conference on Theory and Practice of Software Development, Lille, France, vol. 1214, pp. 833–847.

    Google Scholar 

  • Foundations of Software Engineering, Microsoft Research. The AsmL test generator tool. http://research.microsoft.com/fse/asml/doc/AsmLTester.html

  • Godefroid, P. 1997. Model checking for programming languages using VeriSoft. In Proc. 24th Annual ACM Symposium on the Principles of Programming Languages (POPL), Paris, France, Jan. pp. 174–186.

  • Goodenough, J. and Gerhart, S. 1975. Toward a theory of test data selection. IEEE Transactions on Software Engineering, June.

  • Grieskamp, W., Gurevich, Y., Schulte, W., and Veanes, M. 2002. Generating finite state machines from abstract state machines. In Proc. International Symposium on Software Testing and Analysis (ISSTA), July. pp. 112–122.

  • Gurevich, Y. 1995. Evolving algebras 1993: Lipari guide. In Specification and Validation Methods, Oxford University Press. pp. 9–36.

  • Holzmann, G. 1997. The model checker SPIN. IEEE Transactions on Software Engineering, 23(5), May.

  • Horcher, H.-M. 1995. Improving software tests using Z specifications. In Proc. 9th International Conference of Z Users, The Z Formal Specification Notation.

  • Jackson, D. 2001. Micromodels of software: Modelling and analysis with Alloy, http://sdg.lcs.mit.edu/alloy/ book.pdf.

  • Jackson, D. 2002. Alloy: A lightweight object modeling notation. ACM Transactions on Software Engineering and Methodology (TOSEM), 11(2), April.

  • Jackson, D., Schechter, I., and Shlyakhter, I. 2000. ALCOA: The Alloy constraint analyzer. In Proc. 22nd International Conference on Software Engineering (ICSE), Limerick, Ireland, June.

  • Jackson, D. and Vaziri, M. 2000. Finding bugs with a constraint solver. In Proc. International Symposium on Software Testing and Analysis (ISSTA), Portland, OR, Aug.

  • Jones, C.B. 1990. Systematic Software Development Using VDM. Prentice-Hall.

  • Khurshid, S. and Jackson, D. 2000. Exploring the design of an intentional naming scheme with an automatic constraint analyzer. In Proc. 15th IEEE International Conference on Automated Software Engineering (ASE), Grenoble, France, Sep.

  • Khurshid, S. and Marinov, D. 2001. Checking Java implementation of a naming architecture using TestEra. In Scott D. Stoller and Willem Visser, editors, Electronic Notes in Theoretical Computer Science (ENTCS), vol. 55. Elsevier Science Publishers.

  • Khurshid, S., Marinov, D., and Jackson, D. 2002. An analyzable annotation language. In Proc. ACM SIGPLAN 2002 Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), Seattle, WA, Nov.

  • Khurshid, S., Marinov, D., Shlyakhter, I., and Jackson, D. 2003. A case for efficient solution enumeration. In Proc. Sixth International Conference on Theory and Applications of Satisfiability Testing (SAT 2003), Santa Margherita Ligure, Italy, May.

  • Khurshid, S., Pasareanu, C., and Visser, W. 2003. Generalized symbolic execution for model checking and testing. In Proc. 9th International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS), Warsaw, Poland, April.

  • Kuncak, V., Lam, P., and Rinard, M. 2002. Role analysis. In Proc. 29th Annual ACM Symposium on the Principles of Programming Languages (POPL), Portland, OR, Jan.

  • Leavens, G.T., Baker, A.L., and Ruby, C. 1998. Preliminary design of JML: A behavioral interface specification language for Java. Technical Report TR 98-06i, Department of Computer Science, Iowa State University, June. (last revision: Aug. 2001).

  • Legeard, B., Peureux, F., and Utting, M. 2002. Automated boundary testing from Z and B. In Proc. Formal Methods Europe (FME), Copenhagen, Denmark, July.

  • Lev-Ami, T., Reps, T., Sagiv, M., and Wilhelm, R. 2000. Putting static analysis to work for verification: A case study. In Proc. International Symposium on Software Testing and Analysis.

  • Lev-Ami T. and Sagiv, M. 2000. TVLA: A system for implementing static analyses. In Proc. Static Analysis Symposium, Santa Barbara, CA, June.

  • Marinov, D., Andoni, A., Daniliuc, D., Khurshid, S., and Rinard, M. 2003. An evaluation of exhaustive testing for data structures. Technical Report MIT-LCS-TR-921, MIT CSAIL, Cambridge, MA, September.

  • Marinov D. and Khurshid, S. 2002. VAlloy: Virtual functions meet a relational language. In Proc. Formal Methods Europe (FME), Copenhagen, Denmark, July.

  • McMillan, K. 1993. Symbolic Model Checking. Kluwer Academic Publishers.

  • Moeller, A. and Schwartzbach, M.I. 2001. The pointer assertion logic engine. In Proc. SIGPLAN Conference on Programming Languages Design and Implementation, Snowbird, UT, June.

  • Offutt, J. and Abdurazik, A. 1999. Generating tests from UML specifications. In Proc. Second International Conference on the Unified Modeling Language, October.

  • Rumbaugh, J., Jacobson, I., and Booch, G. 1998. The Unified Modeling Language Reference Manual. Addison-Wesley Object Technology Series.

  • Sagiv, M., Reps, T., and Wilhelm, R. 1998. Solving shape-analysis problems in languages with destructive updating. ACM Transactions on Programming Languages and Systems (TOPLAS), January.

  • Sankar, S. and Hayes, R. 1994. Specifying and testing software components using ADL. Technical Report SMLI TR-94-23, Sun Microsystems Laboratories, Inc., Mountain View, CA, April.

    Google Scholar 

  • Schwartz, E. 1999. Design and implementation of intentional names. Master's thesis, MITLaboratory for Computer Science, Cambridge, MA, June.

    Google Scholar 

  • Shlyakhter, I. 2001. Generating effective symmetry-breaking predicates for search problems. In Proc. Workshop on Theory and Applications of Satisfiability Testing, June.

  • Sloane, N.J.A., Plouffe, S., Borwein, J.M., and Corless, R.M. 1996. The encyclopedia of integer sequences. SIAM Review, 38(2), http://www.research. att.com/~njas/sequences/Seis.html.

  • Spivey, J.M. 1992. The Z Notation: A Reference Manual. 2nd edition, Prentice Hall.

  • Stocks, P. and Carrington, D. 1996. A framework for specification-based testing. IEEE Transactions on Software Engineering, 22(11):777–793.

    Google Scholar 

  • Vaziri, M. and Jackson, D. 2003. Checking properties of heap-manipulating procedures with a constraint solver. In Proc. 9th International Conference on Tools and Algorithms for Construction and Analysis of Systems (TACAS), Warsaw, Poland, April.

  • Visser, W., Havelund, K., Brat, G., and Park, S.J. 2000. Model checking programs. In Proc. 15th IEEE International Conference on Automated Software Engineering (ASE), Grenoble, France.

Download references

Author information

Authors and Affiliations

  1. MIT Lab for Computer Science, 200 Technology Square, Cambridge, MA, 02139, USA

    Sarfraz Khurshid & Darko Marinov

Authors
  1. Sarfraz Khurshid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Darko Marinov
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Khurshid, S., Marinov, D. TestEra: Specification-Based Testing of Java Programs Using SAT. Automated Software Engineering 11, 403–434 (2004). https://doi.org/10.1023/B:AUSE.0000038938.10589.b9

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:AUSE.0000038938.10589.b9

Search

Navigation