Abstract
We describe a progression from pilot studies to development and use of domain-specific verification and validation (V&V) automation. Our domain is the testing of an AI planning system that forms a key component of an autonomous spacecraft. We used pilot studies to ascertain opportunities for, and suitability of, automating various analyses whose results would contribute to V&V in our domain. These studies culminated in development of an automatic generator of automated test oracles. This was then applied and extended in the course of testing the spacecraft's AI planning system.
Richardson et al. (1992, In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105–118), presents motivation for automatic test oracles, and considered the issues and approaches particular to test oracles derived from specifications. Our work, carried through from conception to application, confirms many of their insights. Generalizing from our specific domain, we present some additional insights and recommendations concerning the use of test oracles for V&V of knowledge-based systems.
Similar content being viewed by others
References
Allen, J.F. 1983. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832-843.
Andrews, J.H. 1998. Testing using log file analysis: Tools, methods, and issues. In Proceedings of the 13th IEEE International Conference on Automated Software Engineering, (Honolulu, Hawaii, IEEE Computer Society, pp. 157-166.
Cohen, D. 1989. Compiling complex database transition triggers. In Proceedings of the ACM SIGMOD International Conference on the Management of Data, Portland, Oregon, ACM Press, pp. 225-234.
Dillon, L.K. and Ramakrishna, Y.S. 1996. Generating oracles from your favorite temporal logic specifications. In Proceedings 4th ACM SIGSOFT Symposium Foundations of Software Engineering, San Francisco, ACM Press, pp. 106-117.
Dillon, L. and Yu, Q. 1994. Oracles for checking temporal properties of concurrent systems. In Proceedings 2nd ACM SIGSOFT Symposium Foundations of Software Engineering, New Orleans, ACM Press, pp. 140-153.
Douglass, B.P. 1998. Real-Time UML. Developing Efficient Objects for Embedded Systems. Reading MA: Addison-Wessley.
DS1, 1998. http://nmp.jpl.nasa.gov/ds1/.
Feather, M.S. 1998. Rapid application of lightweight formal methods for consistency analyses. IEEE Transactions on Software Engineering, 24(11):949-959.
Feather, M.S. and Smith, B. 1998. V&V of a spacecraft's autonomous planner through extended automation. In Proceedings of the 23rd Annual Software Engineering Workshop, NASA Goddard, MD, pp. 321-326.
Gutjahr, W.J. 1999. Partition testing vs. random testing: Influence of uncertainty. IEEE Transactions on Software Engineering, 25(5):661-674.
Jagadeesan, L.J., Proter, A., Puchol, C., Ramming, J.C., and Votta, L.G. 1997. Specification-based testing of reactive software: Tools and experiments. In Proceedings of the 19th International Conference on Software Engineering, Boston, MA, pp. 525-535.
Jonsson, A.K., Morris, P.H., Muscettola, N., Rajan, K., and Smith, B.D. 2000. Planning in interplanetary space: Theory and practice. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling Systems, Breckenridge, Colorado, AAAI Press, pp. 177-186.
Mars Climate Orbiter, 1999. Mars Climate Orbiter Mishap Investigation Report, Nov 10, 1999. ftp://ftp.hq.nasa.gov/pub/pao/reports/1999/MCO_report.pdf.
NMP, 1999. http://nmp.jpl.nasa.gov/ds1/tech/autora.html.
Mikk, E., Lakhnech, Y., Siegel, M., and Holzmann, G. 1999. Implementing statecharts in Promela/SPIN. In Proceedings of the 2nd IEEE Workshop on Industrial-Stength Formal Specification Techniques, IEEE Computer Society, pp. 90-101.
Pell, B., Bernard, D.E., Chien, S.A., Gat, E., Muscettola, N., Nayak, P.P., Wagner, M.D., and Williams, B.C. 1996. A remote agent prototype for spacecraft autonomy. In Proceedings of the SPIE Conference on Optical Science, Engineering and Instrumentation.
Pell, B., Bernard, D.E., Chien, S.A., Gat, E., Muscettola, N., Nayak, P.P., Wagner, M.D., and Williams, B.C. 1997. An autonomous spacecraft agent prototype. In Proceedings First International Conference on Autonomous Agents, ACM Press.
Reasoning SDK™. Reasoning, Inc. http://www.reasoning.com.
Reyes, A.A. and Richardson, D.J. 1998. Specification-based testing of ada units with low encapsulation. In Proceedings of the 13th IEEE International Conference on Automated Software Engineering, (Honolulu, Hawaii, IEEE Computer Society, pp. 22-31.
Richardson, D.J., Aha, S.L., and O'Malley, T.O. 1992. Specification-based test oracles for reactive systems. In Proceedings of the 14th International Conference on Software Engineering, Melbourne, Australia, pp. 105-118.
Smith, B.D., Feather, M.S., and Muscettola, N. 2000. Challenges and methods in testing the remote agent planner. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling Systems, Breckenridge, Colorado, AAAI Press, pp. 254-263.
Wasserman, H. and Blum, M. 1997. Software reliability viarun-time result-checking. JACM 44(6):826-845.
Wile, D. 1997. Abstract Syntax from Concrete Syntax. In Proceedings of the 19th International Conference on Software Engineering, Boston, MA, pp. 472-480.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Feather, M.S., Smith, B. Automatic Generation of Test Oracles—From Pilot Studies to Application. Automated Software Engineering 8, 31–61 (2001). https://doi.org/10.1023/A:1008711707946
Issue Date:
DOI: https://doi.org/10.1023/A:1008711707946