Simulations for Comparing Knowledge Acquisition and Machine Learning

Abstract

This paper introduces a new class of systematic experimental studies targeted towards a better understanding of the strengths and weaknesses of knowledge acquisition (KA) methods. We model a domain along with the behaviour of a domain expert. Using these models we can simulate the KA process and observe which factors of the domain, the expert or the KA technique, affect the overall result of the KA process. On the basis of our models, we can also compare the performance of our KA techniques against the performance of automatic KA techniques, i.e. against machine learning techniques.

We present a number of results from our modelling approach. These results include the surprising fact that in some domains, building a decision tree by consulting an expert for providing a correct discriminating attribute along with a correct threshold value for a presented case, may still be inferior to an automatic method, such as C4.5, using the same set of cases. Furthermore, we obtained new insights into characteristics of the knowledge representation scheme being used (Ripple Down Rules) as well as guidelines for experts when providing knowledge. Finally, we advocate our methodological approach for studying KA techniques to also being much more widely used in machine learning research. We consider our approach as an important methodological complement to the extensive performance comparisons in machine learning research using ‘natural datasets’.