Abstract
The overall aim of this paper is to provide a general setting for quantitative quality measures of knowledge-based system behaviour that is widely applicable to many knowledge-based systems. We propose a general approach that we call degradation studies: an analysis of how system output changes as a function of degrading system input, such as incomplete or incorrect data or knowledge. To show the feasibility of our approach, we have applied it in a case study. We have taken a large and realistic vegetation-classification system, and have analysed its behaviour under various varieties of incomplete and incorrect input. This case study shows that degradation studies can reveal interesting and surprising properties of the system under study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cohen PR (1995) Empirical methods for artificial intelligence. MIT Press, Cambridge, MA
Dean T, Boddy M (1988) An analysis of time-dependent planning. In Proceedings of the 7th national conference on artificial intelligence. Kaufmann, Saint Paul, MN, August 1988, pp 49–54
Hayes-Roth F (1984) Knowledge-based expert systems—the state of the art in the US. In Fox J (ed) Expert systems: state of the art report. Pergamon, Oxford, UK
IEEE (1990) IEEE standard glossary of software engineering terminology IEEE standard 610.12-1990, ISBN 1-55937-067-X
Menzies T, van Harmelen F (1999) Evaluating knowledge-engineering techniques. Int J Hum Comput Stud 51(4):715–727
Preece AD, Talbot S, Vignollet L (1997) Evaluation of verification tools for knowledge-based systems. Int J Hum Comput Stud 47(5):629–658
Puppe F, Poeck K, Gappa U, et al (1994) Wiederverwendbare bausteine für eine konfigurierbare Diagnostik-shell. Künstliche Intelligenz 94(2):13–18
Puppe F, Gappa U, Poeck K, et al (1996) Wissenbasierte Diagnose- und Informationssysteme: mit Anwendungen des Expertensystem-Shell-Baukastens D3. Springer, Berlin Heidelberg New York
Salton G, McGill MJ (1983) Introduction to modern information retrieval. McGraw-Hill, New York
Schumann J, Fischer B (1997) NORA/HAMMR: making deduction-based software component retrieval practical. In: Proceedings of the 12th international conference on automated software engineering, IEEE Press, Lake Tahoe, CA, November 1997, pp 246–254
Shadbolt NR, O’Hara K, Crow L (1999) The experimental evaluation of knowledge acquisition techniques and methods: history, problems and new directions. Int J Hum Comput Stud 51(4):729–755
ten Teije A, van Harmelen F (1996) Computing approximate diagnoses by using approximate entailment. In Aiello G, Doyle J (eds). Proceedings of the 5th international conference on principles of knowledge representation and reasoning. Kaufmann, Cambridge, MA, November 1996, pp 256–265
ten Teije A, van Harmelen F (1997) Exploiting domain knowledge for approximate diagnosis (1997) In: Pollack ME (ed) Proceedings of the 15th international joint conference on artificial intelligence. Kaufmann, Nogoya, Japan, August 1997, pp 454–459
van Harmelen F, ten Teije A (1998) Characterising approximate problem-solving by partial pre- and postconditions. In: Prade H (ed) Proceedings of the 13th European conference on artificial intelligence. Wiley, Brighton, UK, August 1998, pp 78–82
Zilberstein S (1996) Using anytime algorithms in intelligent systems. Artif Intell Mag 17(3):73–83
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Groot, P., ten Teije, A. & van Harmelen, F. A quantitative analysis of the robustness of knowledge-based systems through degradation studies. Knowl Inf Syst 7, 224–245 (2005). https://doi.org/10.1007/s10115-003-0140-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10115-003-0140-7