Skip to main content
Log in

Automatic Generation of Test Oracles—From Pilot Studies to Application

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

We describe a progression from pilot studies to development and use of domain-specific verification and validation (V&V) automation. Our domain is the testing of an AI planning system that forms a key component of an autonomous spacecraft. We used pilot studies to ascertain opportunities for, and suitability of, automating various analyses whose results would contribute to V&V in our domain. These studies culminated in development of an automatic generator of automated test oracles. This was then applied and extended in the course of testing the spacecraft's AI planning system.

Richardson et al. (1992, In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105–118), presents motivation for automatic test oracles, and considered the issues and approaches particular to test oracles derived from specifications. Our work, carried through from conception to application, confirms many of their insights. Generalizing from our specific domain, we present some additional insights and recommendations concerning the use of test oracles for V&V of knowledge-based systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Allen, J.F. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832-843.

    Google Scholar 

  • Andrews, J.H. 1998. Testing using log file analysis: Tools, methods, and issues. In Proceedings of the 13th IEEE International Conference on Automated Software Engineering, (Honolulu, Hawaii, IEEE Computer Society, pp. 157-166.

    Google Scholar 

  • Cohen, D. 1989. Compiling complex database transition triggers. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, Portland, Oregon, ACM Press, pp. 225-234.

    Google Scholar 

  • Dillon, L.K. and Ramakrishna, Y.S. 1996. Generating oracles from your favorite temporal logic specifications. In Proceedings 4th ACM SIGSOFT Symposium Foundations of Software Engineering, San Francisco, ACM Press, pp. 106-117.

    Google Scholar 

  • Dillon, L. and Yu, Q. 1994. Oracles for checking temporal properties of concurrent systems. In Proceedings 2nd ACM SIGSOFT Symposium Foundations of Software Engineering, New Orleans, ACM Press, pp. 140-153.

    Google Scholar 

  • Douglass, B.P. 1998. Real-Time UML. Developing Efficient Objects for Embedded Systems. Reading MA: Addison-Wessley.

    Google Scholar 

  • DS1, 1998. http://nmp.jpl.nasa.gov/ds1/.

  • Feather, M.S. 1998. Rapid application of lightweight formal methods for consistency analyses. IEEE Transactions on Software Engineering, 24(11):949-959.

    Google Scholar 

  • Feather, M.S. and Smith, B. 1998. V&V of a spacecraft's autonomous planner through extended automation. In Proceedings of the 23rd Annual Software Engineering Workshop, NASA Goddard, MD, pp. 321-326.

    Google Scholar 

  • Gutjahr, W.J. 1999. Partition testing vs. random testing: Influence of uncertainty. IEEE Transactions on Software Engineering, 25(5):661-674.

    Google Scholar 

  • Jagadeesan, L.J., Proter, A., Puchol, C., Ramming, J.C., and Votta, L.G. 1997. Specification-based testing of reactive software: Tools and experiments. In Proceedings of the 19th International Conference on Software Engineering, Boston, MA, pp. 525-535.

  • Jonsson, A.K., Morris, P.H., Muscettola, N., Rajan, K., and Smith, B.D. 2000. Planning in interplanetary space: Theory and practice. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling Systems, Breckenridge, Colorado, AAAI Press, pp. 177-186.

    Google Scholar 

  • Mars Climate Orbiter, 1999. Mars Climate Orbiter Mishap Investigation Report, Nov 10, 1999. ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf.

  • NMP, 1999. http://nmp.jpl.nasa.gov/ds1/tech/autora.html.

  • Mikk, E., Lakhnech, Y., Siegel, M., and Holzmann, G. 1999. Implementing statecharts in Promela/SPIN. In Proceedings of the 2nd IEEE Workshop on Industrial-Stength Formal Specification Techniques, IEEE Computer Society, pp. 90-101.

  • Pell, B., Bernard, D.E., Chien, S.A., Gat, E., Muscettola, N., Nayak, P.P., Wagner, M.D., and Williams, B.C. 1996. A remote agent prototype for spacecraft autonomy. In Proceedings of the SPIE Conference on Optical Science, Engineering and Instrumentation.

  • Pell, B., Bernard, D.E., Chien, S.A., Gat, E., Muscettola, N., Nayak, P.P., Wagner, M.D., and Williams, B.C. 1997. An autonomous spacecraft agent prototype. In Proceedings First International Conference on Autonomous Agents, ACM Press.

  • Reasoning SDK. Reasoning, Inc. http://www.reasoning.com.

  • Reyes, A.A. and Richardson, D.J. 1998. Specification-based testing of ada units with low encapsulation. In Proceedings of the 13th IEEE International Conference on Automated Software Engineering, (Honolulu, Hawaii, IEEE Computer Society, pp. 22-31.

    Google Scholar 

  • Richardson, D.J., Aha, S.L., and O'Malley, T.O. 1992. Specification-based test oracles for reactive systems. In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105-118.

  • Smith, B.D., Feather, M.S., and Muscettola, N. 2000. Challenges and methods in testing the remote agent planner. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling Systems, Breckenridge, Colorado, AAAI Press, pp. 254-263.

    Google Scholar 

  • Wasserman, H. and Blum, M. 1997. Software reliability viarun-time result-checking. JACM 44(6):826-845.

    Google Scholar 

  • Wile, D. 1997. Abstract Syntax from Concrete Syntax. In Proceedings of the 19th International Conference on Software Engineering, Boston, MA, pp. 472-480.

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Feather, M.S., Smith, B. Automatic Generation of Test Oracles—From Pilot Studies to Application. Automated Software Engineering 8, 31–61 (2001). https://doi.org/10.1023/A:1008711707946

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008711707946

Navigation