Skip to main content
SpringerLink
Log in

Dynamic evaluation of the development process of knowledge-based information systems

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Codifying expert domain knowledge is a difficult and expensive task. To evaluate the quality of the outcome, often the same domain expert or a colleague of similar expertise is relied on to undertake a direct evaluation of the knowledge-based system or indirectly by preparing appropriate test data. During an incremental knowledge acquisition process, a data stream is available, and the knowledge base is observed and amended by an expert each time it produces an error. Using the kept record of the system’s performance, we propose an evaluation process to estimate its effectiveness as it gets evolved. We instantiate this process for an incremental knowledge acquisition methodology, Ripple Down Rules. We estimate the added value in each knowledge base update. Using these values, the decision makers in the organisation employing the knowledge-based information system can apply a cost-benefit analysis of the continuation of the incremental knowledge acquisition process. They can then determine when this process, involving keeping an expert online, should be terminated. As a result, the expert is not kept on-line longer than it is absolutely necessary. Hence, a major expense in deploying the information system—the cost of keeping a domain expert on-line—is reduced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aggarwal C, Yu P (2010) On clustering massive text and categorical data streams. Knowl Inf Syst 24: 171–196

    Article  Google Scholar 

  2. Akerkar R, Sajja P (2010) Knowledge-based systems. Jones and Bartlett, Ontario

    Google Scholar 

  3. Barouni-Ebrahimi M, Ghorbani A (2008) An interactive search assistant architecture based on intrinsic query stream characteristics. Comput Intell 24(2): 158–190

    Article  MathSciNet  Google Scholar 

  4. Baumeister J, Freiberg M (2011) Knowledge visualization for evaluation tasks. Knowl Inf Syst 29: 349–378

    Article  Google Scholar 

  5. Berners-Lee T, Hall W et al (2006) Creating a Science of the Web. Science 313(5788): 769–771

    Article  Google Scholar 

  6. Beydoun G (2009) Formal concept analysis for an e-learning semantic web. Expert Syst Appl 36(8): 10952–10961

    Article  Google Scholar 

  7. Beydoun G, Gonzalez-Perez C, Henderson-Sellers B et al (2006) Developing and evaluating a generic metamodel for MAS work products. In: Garcia A, Choren R, Lucena C et al (eds) Software engineering for multi-agent systems IV: research issues and practical applications. Springer, Berlin, LNCS 3914, pp 126–142

  8. Beydoun G, Hoffmann A (1998) Simultaneous modelling and knowledge acquisition using NRDR. 5th Pacific Rim conference on artificial intelligence (PRICAI98). Springer, Singapore

    Google Scholar 

  9. Beydoun G, Hoffmann A (2000) Incremental acquisition of search knowledge. Int J Hum Comput Stud 52(3): 493–530

    Article  Google Scholar 

  10. Beydoun G, Hoffmann A (2000) Monitoring knowledge acquisition instead of evaluating knowledge bases. 12th European conference on knowledge acquisition and knowledge management (EKAW2000). Springer, France

    Google Scholar 

  11. Beydoun G, Hoffmann A (2001) Theoretical basis for hierarchical incremental acquisition. Int J Hum Comput Stud 54(3): 407–452

    Article  MATH  Google Scholar 

  12. Beydoun G, Hoffmann A, Hamade R (2010) Automating dimensional tolerancing using Ripple Down Rules (RDR). Expert Syst Appl 37(7): 5101–5109

    Article  Google Scholar 

  13. Beydoun G, Lopez-Lorca A, García-Sánchez F et al (2011) How do we measure and improve the quality of a hierarchical ontology?. J Syst Softw 84(12): 2363–2373

    Article  Google Scholar 

  14. Bichindaritz I (2009) Prototypical cases for retrieval, reuse, and knowledge maintenance in biomedical case-based reasoning. Comput Intell 25(3): 214–234

    Article  MathSciNet  Google Scholar 

  15. Bichindaritz I, Montani S (2009) Introduction to the special issue on case-based reasoning in the health sciences. Comput Intell 25(3): 161–164

    Article  MathSciNet  Google Scholar 

  16. Brewster C, O’Hara K (2004) Knowledge representation with ontologies: the present and future. IEEE Intell Syst 19(1): 72–81

    Article  Google Scholar 

  17. Cao T, Compton P (2006) Knowledge acquisition evaluation using simulated experts managing knowledge in a world of networks. Springer, Berlin, pp 35–42

    Book  Google Scholar 

  18. Compton P, Kang B, Preston P et al (1993) Knowledge acquisition without knowledge analysis. European Knowledge Acquisition Workshop (EKAW93). Springer, Berlin

    Google Scholar 

  19. Compton P, Peters L, Edwards G et al (2006) Experience with Ripple Down Rules. Knowl Based Syst 19(5): 356–362

    Article  Google Scholar 

  20. Davis R (1979) Interactive transfer of expertise: acquisition of new inference rules. Artif Intell 12: 121–157

    Article  Google Scholar 

  21. Edwards G (1996) Reflective expert systems in clinical pathology (MD thesis), University of New South Wales

  22. Farahat AK, Kamel M (2011) Statistical semantics for enhancing document clustering. Knowl Inf Syst 28: 365–393

    Article  Google Scholar 

  23. Grogono P, Batarekh A, Preece A et al (1991) Expert system evaluation techniques: a selected bibliography. Expert Syst 8(4): 227–239

    Article  Google Scholar 

  24. Hoffmann A, Kwok R, Compton P (2001) Simulations for comparing knowledge acquisition and machine learning. Australian artificial intelligence conference (AI2001), pp 273–284

  25. Kelil A, Wang S, Jiang Q et al (2010) A general measure of similarity for categorical sequences. Knowl Inf Syst 24: 197–220

    Article  Google Scholar 

  26. Kivinen J, Mannila H, Ukkonen E (1993) Learning rules with local exceptions. ACM Conference on Computational Theory. Santa Cruz, USA

    Google Scholar 

  27. Kwok RBH (2000) Translations of ripple down rules into logic formalisms. The 12th European Knowledge Acquisition Conference (EKAW2000). Springer, France

    Google Scholar 

  28. Littau D, Boley D (2009) Clustering very large data sets using a low memory matrix factored representation. Comput Intell 25(2): 114–135

    Article  MathSciNet  Google Scholar 

  29. Liu H, Lin Y, Han J (2011) Methods for mining frequent items in data streams: an overview. Knowl Inf Syst 26: 1–30

    Article  Google Scholar 

  30. Menzies T (1998) Evaluation issues with critical success metrics. 11th Banff Knowledge Acquisition for Knowledge Base System Workshop (KAW99). SRDG Publications, Canada

  31. Mitchell TM (1997) Machine learning. McGraw-Hill, Singapore

    MATH  Google Scholar 

  32. Nick M, Althoff K, Tautz C (1999) Facilitating the practical evaluation of organizational memories using the goal-question-metric technique. 12th Banff knowledge acquisition for knowledge-based systems workshop (KAW99). SRDG publications, Canada

  33. Ning H, Shihan D (2006) Structure-based ontology evaluation. IEEE International conference on e-business engineering (ICEBE’06), pp 132–137

  34. Punera K, Rajan S, Ghosh J (2006) Automatic construction of N-ary tree based taxonomies. In: Sixth IEEE International conference on data mining—workshops (ICDMW’06). University of Texas, Austin

  35. Sadraei E, Aurum A, Beydoun G et al (2007) A field study of the requirements engineering practice in Australian software industry. Requir Eng 12(3): 145–162

    Article  Google Scholar 

  36. Scheffer T (1995) Learning rules with nested exceptions. International Workshop on Artificial Intelligence

  37. Simmuteit S, Schleif F, Villmann T et al (2010) Evolving trees for the retrieval of mass spectrometry-based bacteria fingerprints. Knowl Inf Syst 25: 327–343

    Article  Google Scholar 

  38. Sun S, Huang Z, Zhong H et al (2010) Efficient monitoring of skyline queries over distributed data streams. Knowl Inf Syst 25: 575–606

    Article  Google Scholar 

  39. Völker J, Vrandecic D, Sure Y et al (2005) Automatic evaluation of ontologies (AEON). The 4th International Semantic Web Conference (ISWC 2005). Springer, Berlin

  40. Wada T, Horiuchi T, Motoda H et al (1998) A new look at default knowledge in ripple down rules method. Pacific Rim Knowledge Acquisition Workshop (PKAW98). National Univeristy of Singapore, Singapore

  41. Wobcke W, Chan R, Limaru A (2006) A call handling assistant for mobile devices. International Conference on Intelligent Agent Technology (IAT06). IEEE/WIC/ACM, Hong Kong

  42. Zeng Y, Zhong N et al (2011) User-centric query refinement and processing using granularity-based strategies. Knowl Inf Syst 27: 419–450

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Systems and Technology, University of Wollongong, Wollongong, Australia

    Ghassan Beydoun

  2. School of Computer Science and Engineering, University of New South Wales, Sydney, Australia

    Achim Hoffmann

Authors
  1. Ghassan Beydoun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Achim Hoffmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ghassan Beydoun.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Beydoun, G., Hoffmann, A. Dynamic evaluation of the development process of knowledge-based information systems. Knowl Inf Syst 35, 233–247 (2013). https://doi.org/10.1007/s10115-012-0491-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-012-0491-z

Keywords

Search

Navigation