ABSTRACT
Producing specifications by dynamic (runtime) analysis of program executions is potentially unsound, because the analyzed executions may not fully characterize all possible executions of the program. In practice, how accurate are the results of a dynamic analysis? This paper describes the results of an investigation into this question, determining how much specifications generalized from program runs must be changed in order to be verified by a static checker. Surprisingly, small test suites captured nearly all program behavior required by a specific type of static checking; the static checker guaranteed that the implementations satisfy the generated specifications, and ensured the absence of runtime exceptions. Measured against this verification task, the generated specifications scored over 90% on precision, a measure of soundness, and on recall, a measure of completeness.This is a positive result for testing, because it suggests that dynamic analyses can capture all semantic information of interest for certain applications. The experimental results demonstrate that a specific technique, dynamic invariant detection, is effective at generating consistent, sufficient specifications for use by a static checker. Finally, the research shows that combining static and dynamic analyses over program specifications has benefits for users of each technique, guaranteeing soundness of the dynamic analysis and lessening the annotation burden for users of the static analysis.
- {AFMS96} David Abramson, Ian Foster, John Michalakes, and Rok Socič. Relative debugging: A new methodology for debugging scientific applications. Communications of the ACM, 39(11):69-77, November 1996.]] Google ScholarDigital Library
- {Bal99} Thomas Ball. The concept of dynamic analysis. In ESEC/FSE, pages 216-234, September 6-10, 1999.]] Google ScholarDigital Library
- {BBM97} Nicolaj Bjørner, Anca Browne, and Zohar Manna. Automatic generation of invariants and intermediate assertions. Theoretical Computer Science, 173(1):49-87, February 1997.]] Google ScholarDigital Library
- {BG93} Ivan Bratko and Marko Grobelnik. Inductive learning applied to program construction and verification. In José Cuena, editor, AIFIPP '92, pages 169-182. North-Holland, 1993.]] Google ScholarDigital Library
- {BG97} Bernard Boigelot and Patrice Godefroid. Automatic synthesis of specifications from the dynamic observation of reactive programs. In TACAS '97, pages 321-333, Twente, April 1997.]] Google ScholarDigital Library
- {BLS96} Saddek Bensalem, Yassine Lakhnech, and Hassen Saidi. Powerful techniques for the automatic generation of invariants. In CAV, pages 323-335, July 31-August 3, 1996.]] Google ScholarDigital Library
- {CC77} Patrick M. Cousot and Radhia Cousot. Automatic synthesis of optimal invariant assertions: Mathematical foundations. In Proceedings of the ACM Symposium on Artificial Intelligence and Programming Languages, pages 1-12, Rochester, NY, August 1977.]] Google ScholarDigital Library
- {CDH+00} James Corbett, Matthew Dwyer, John Hatcliff, Corina Pasareanu, Robby, Shawn Laubach, and Hongjun Zheng. Bandera: Extracting finite-state models from Java source code. In ICSE, pages 439-448, June 7-9, 2000.]] Google ScholarDigital Library
- {CFE97} Brad Calder, Peter Feller, and Alan Eustace. Value profiling. In MICRO-97, pages 259-269, December 1-3, 1997.]] Google ScholarDigital Library
- {CHK+93} Allen Cypher, Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, editors. Watch What I Do: Programming by Demonstration. MIT Press, Cambridge, MA, 1993.]] Google ScholarDigital Library
- {Coh94} William W. Cohen. Grammatically biased learning: Learning logic programs using an explicit antecedent description language. Artificial Intelligence, 68:303-366, August 1994.]] Google ScholarDigital Library
- {CW98} Jonathan E. Cook and Alexander L. Wolf. Event-based detection of concurrency. In FSE, pages 35-45, November 1998.]] Google ScholarDigital Library
- {Das00} Manuvir Das. Unification-based pointer analysis with directional assignments. In PLDI, pages 35-46, June 18-23, 2000.]] Google ScholarDigital Library
- {Det96} David L. Detlefs. An overview of the Extended Static Checking system. In Proceedings of the First Workshop on Formal Methods in Software Practice, pages 1-9, January 1996.]]Google Scholar
- {DLNS98} David L. Detlefs, K. Rustan M. Leino, Greg Nelson, and James B. Saxe. Extended static checking. SRC Research Report 159, Compaq Systems Research Center, December 18, 1998.]]Google Scholar
- {Dod02} Nii Dodoo. Selecting predicates for conditional invariant detection using cluster analysis. Master's thesis, MIT Dept. of EECS, 2002.]]Google Scholar
- {ECGN00} Michael D. Ernst, Adam Czeisler, William G. Griswold, and David Notkin. Quickly detecting relevant program invariants. In ICSE, pages 449-458, June 2000.]] Google ScholarDigital Library
- {ECGN01} Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. Dynamically discovering likely program invariants to support program evolution. IEEE TSE, 27(2):1-25, February 2001. A previous version appeared in ICSE, pages 213-224, Los Angeles, CA, USA, May 1999.]] Google ScholarDigital Library
- {EGHT94} David Evans, John Guttag, James Horning, and Yang Meng Tan. LCLint: A tool for using specifications to check code. In FSE, pages 87-97, December 1994.]] Google ScholarDigital Library
- {Ern00} Michael D. Ernst. Dynamically Discovering Likely Program Invariants. PhD thesis, University of Washington Department of Computer Science and Engineering, Seattle, Washington, August 2000.]]Google ScholarDigital Library
- {FJL01} Cormac Flanagan, Rajeev Joshi, and K. Rustan M. Leino. Annotation inference for modular checkers. Information Processing Letters, 2(4):97-108, February 2001.]] Google ScholarDigital Library
- {FL01} Cormac Flanagan and K. Rustan M. Leino. Houdini, an annotation assistant for ESC/Java. In Formal Methods Europe, volume 2021 of LNCS, pages 500-517, Berlin, Germany, March 2001.]] Google ScholarDigital Library
- {GG90} Stephen Garland and John Guttag. LP, the Larch Prover. In M. Stickel, editor, Proceedings of the Tenth International Conference on Automated Deduction, volume 449 of LNCS, Kaiserslautern, West Germany, 1990. Springer-Verlag.]]Google Scholar
- {GJM91} Carlo Ghezzi, Mehdi Jazayeri, and Dino Mandrioli. Fundamentals of Software Engineering. Prentice Hall, Englewood Cliffs, NJ, 1 edition, 1991.]] Google ScholarDigital Library
- {GLV97} Stephen J. Garland, Nancy A. Lynch, and Mandana Vaziri. IOA: A language for specifying, programming, and validating distributed systems. Technical report, MIT Laboratory for Computer Science, 1997.]]Google Scholar
- {Har02} Michael Harder. Improving test suites via generated specifications. Master's thesis, MIT Dept. of EECS, May 2002.]]Google Scholar
- {HJv01} Marieke Huisman, Bart P. F. Jacobs, and Joachim A. G. M. van den Berg. A case study in class library verification: Java's Vector class. International Journal on Software Tools for Technlogy Transfer, 2001.]]Google Scholar
- {HL02} Sudheendra Hangal and Monica S. Lam. Tracking down software bugs using automatic anomaly detection. In ICSE, May 2002.]]Google ScholarCross Ref
- {HP00} Klaus Havelund and Thomas Pressburger. Model checking Java programs using Java PathFinder. International Journal on Software Tools for Technology Transfer, 2(4):366-381, 2000.]]Google ScholarCross Ref
- {HRWY98} Mary Jean Harrold, Gregg Rothermel, Rui Wu, and Liu Yi. An empirical investigation of program spectra. In PASTE '98, pages 83-90, June 16, 1998.]] Google ScholarDigital Library
- {JvH+98} Bart Jacobs, Joachim van den Berg, Marieke Huisman, Martijn van Berkum, Ulrich Hensel, and Hendrik Tews. Reasoning about Java classes. In OOPSLA, pages 329-340, Vancouver, BC, Canada, October 18-22, 1998.]] Google ScholarDigital Library
- {KEGN01} Yoshio Kataoka, Michael D. Ernst, William G. Griswold, and David Notkin. Automated support for program refactoring using invariants. In ICSM, pages 736-743, November 2001.]] Google ScholarDigital Library
- {KM97} Matt Kaufmann and J Strother Moore. An industrial strength theorem prover for a logic based on Common Lisp. IEEE TSE, 23(4):203-213, April 1997.]] Google ScholarDigital Library
- {Lam88} David Alex Lamb. Software Engineering: Planning for Change. Prentice Hall, Englewood Cliffs, NJ, 1988.]] Google ScholarDigital Library
- {LBR99} Gary T. Leavens, Albert L. Baker, and Clyde Ruby. JML: A notation for detailed design. In Haim Kilov, Bernhard Rumpe, and Ian Simmonds, editors, Behavioral Specifications of Businesses and Systems, pages 175-188. Kluwer Academic Publishers, Boston, 1999.]]Google ScholarCross Ref
- {LBR00} Gary T. Leavens, Albert L. Baker, and Clyde Ruby. Preliminary design of JML: A behavioral interface specification language for Java. Technical Report 98-06m, Iowa State University, Department of Computer Science, February 2000. See www.cs.iastate.edu/~leavens/JML.html.]]Google Scholar
- {LDW00} Tessa Lau, Pedro Domingos, and Daniel S. Weld. Version space algebra and its application to programming by demonstration. In ICML, Stanford, CA, June 2000.]] Google ScholarDigital Library
- {LG01} Barbara Liskov and John Guttag. Program Development in Java: Abstraction, Specification, and Object-Oriented Design. Addison-Wesley, Boston, MA, 2001.]]Google ScholarDigital Library
- {LN98} K. Rustan M. Leino and Greg Nelson. An extended static checker for Modula-3. In Compiler Construction '98, pages 302-305, April 1998.]] Google ScholarDigital Library
- {LNS00} K. Rustan M. Leino, Greg Nelson, and James B. Saxe. ESC/Java user's manual. Technical Report 2000-002, Compaq Systems Research Center, Palo Alto, California, October 12, 2000.]]Google Scholar
- {LT89} Nancy A. Lynch and Mark R. Tuttle. An introduction to Input/Output automata. CWI-Quarterly, 2(3):219-246, September 1989.]]Google Scholar
- {NCOD97} Gleb Naumovich, Lori A. Clarke, Leon J. Osterweil, and Matthew B. Dwyer. Verification of concurrent software with FLAVERS. In ICSE, pages 594-595, May 1997.]] Google ScholarDigital Library
- {NE01} Jeremy W. Nimmer and Michael D. Ernst. Static verification of dynamically detected program invariants: Integrating Daikon and ESC/Java. In Proceedings of RV'01, First Workshop on Runtime Verification, Paris, France, July 23, 2001.]]Google Scholar
- {Nim02} Jeremy W. Nimmer. Automatic generation and checking of program specifications. Master's thesis, MIT Dept. of EECS, May 2002.]]Google Scholar
- {NWEL02} Toh Ne Win, Michael Ernst, and Nancy Lynch. Static and dynamic analysis of I/O automata. Technical Report 841, MIT Lab for Computer Science, May 2002.]]Google Scholar
- {O'C01} Robert O'Callahan. Generalized Aliasing as a Basis for Program Analysis Tools. PhD thesis, Carnegie-Mellon University, Pittsburgh, PA, May 2001.]]Google ScholarDigital Library
- {ORS92} S. Owre, J. M. Rushby, and N. Shankar. PVS: A prototype verification system. In Proceedings of the 11th International Conference on Automated Deduction (CADE-11), volume 607, pages 748-752, Saratoga Springs, NY, June 1992.]] Google ScholarDigital Library
- {ORSvH95} Sam Owre, John Rushby, Natarajan Shankar, and Friedrich von Henke. Formal verification for fault-tolerant architectures: Prolegomena to the design of PVS. IEEE TSE, 21(2):107-125, February 1995. Special Section---Best Papers of FME (Formal Methods Europe) '93.]] Google ScholarDigital Library
- {PC86} David Lorge Parnas and Paul C. Clements. A rational design process: How and why to fake it. IEEE TSE, SE-12(2):251-257, February 1986.]] Google ScholarDigital Library
- {Pfe92} Frank Pfenning. Dependent types in logic programming. In Frank Pfenning, editor, Types in Logic Programming, chapter 10, pages 285-311. MIT Press, Cambridge, MA, 1992.]]Google Scholar
- {Pre92} Roger S. Pressman. Software Engineering: A Practitioner's Approach. McGraw-Hill, New York, third edition, 1992.]]Google Scholar
- {Qui90} J. Ross Quinlan. Learning logical definitions from relations. Machine Learning, 5:239-266, 1990.]] Google ScholarDigital Library
- {RBDL97} Thomas Reps, Thomas Ball, Manuvir Das, and James Larus. The use of program profiling for software maintenance with applications to the year 2000 problem. In ESEC/FSE, pages 432-449, September 22-25, 1997.]] Google ScholarDigital Library
- {Rin00} Jussi Rintanen. An iterative algorithm for synthesizing invariants. In AAAI/IAAI, pages 806-811, Austin, TX, July 30-August 3, 2000.]] Google ScholarDigital Library
- {RKS02} Orna Raz, Philip Koopman, and Mary Shaw. Semantic anomaly detection in online data sources. In ICSE, May 2002.]]Google ScholarCross Ref
- {Sal68} Gerard Salton. Automatic Information Organization and Retrieval. McGraw-Hill, 1968.]] Google ScholarDigital Library
- {Sem94} Semiconductor Industry Association. The national technology roadmap for semiconductors. San Jose, CA, 1994.]]Google Scholar
- {Som96} Ian Sommerville. Software Engineering. Addison-Wesley, Wokingham, England, fifth edition, 1996.]]Google Scholar
- {SS98} Avinash Sodani and Gurindar S. Sohi. An empirical analysis of instruction repetition. In ASPLOS, pages 35-45, October 1998.]] Google ScholarDigital Library
- {vR79} C. J. van Rijsbergen. Information Retrieval. Butterworths, London, second edition, 1979.]] Google ScholarDigital Library
- {Weg74} Ben Wegbreit. The synthesis of loop predicates. Communications of the ACM, 17(2):102-112, February 1974.]] Google ScholarDigital Library
- {Wei99} Mark Allen Weiss. Data Structures and Algorithm Analysis in Java. Addison Wesley Longman, 1999.]]Google ScholarDigital Library
- Automatic generation of program specifications
Recommendations
Automatic generation of program specifications
Producing specifications by dynamic (runtime) analysis of program executions is potentially unsound, because the analyzed executions may not fully characterize all possible executions of the program. In practice, how accurate are the results of a ...
Automatic test program generation for pipelined processors
ICCAD '94: Proceedings of the 1994 IEEE/ACM international conference on Computer-aided designSimulation-based verification has both advantages and disadvantages compared with formal verification. Our demand is to find a practical way to verify actual microprocessors. This paper presents an efficient test program generation method for simulation-...
Informality in Program Specifications
This paper is concerned with the need for computer-based tools which help human designers formulate formal process-oriented specifications. It first determines some attributes of a suitable process-oriented specification language, then examines the ...
Comments