Abstract
As modern applications tend to stretch between large, ever-growing datasets and increasing demand for meaningful content at the user end, more elaborate and sophisticated knowledge extraction technologies are needed. Towards this direction, the inherently contradicting technologies of deductive software agents and inductive data mining have been integrated, in order to address knowledge intensive problems. However, there exists no generalized evaluation methodology for assessing the efficiency of such applications. On the one hand, existing data mining evaluation methods focus only on algorithmic precision, ignoring overall system performance issues. On the other hand, existing systems evaluation techniques are insufficient, as the emergent intelligent behavior of agents introduce unpredictable factors of performance. In this paper, we present a generalized methodology for performance evaluation of intelligent agents that employ knowledge models produced through data mining. The proposed methodology consists of concise steps for selecting appropriate metrics, defining measurement methodologies and aggregating the measured performance indicators into thorough system characterizations. The paper concludes with a demonstration of the proposed methodology to a real world application, in the Supply Chain Management domain.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Galitsky, B., Pampapathi, R.: Deductive and inductive reasoning for processing the claims of unsatisfied customers. In: Chung, P.W.H., Hinde, C.J., Ali, M. (eds.) IEA/AIE 2003. LNCS, vol. 2718, pp. 21–30. Springer, Heidelberg (2003)
Fernandes, A.A.A.: Combining inductive and deductive inference in knowledge management tasks. In: DEXA ’00: Proceedings of the 11th International Workshop on Database and Expert Systems Applications, Washington, DC, USA, p. 1109. IEEE Computer Society Press, Los Alamitos (2000)
Kero, B., et al.: An overview of database mining techniques. In: KDOOD/TDOOD, pp. 1–8 (1995)
Symeonidis, A.L., Mitkas, P.A.: Agent Intelligence Through Data Mining. Springer, Heidelberg (2005)
Zadeh, L.A.: In quest of performance metrics for intelligent systems – a challenge that cannot be met with existing methods. In: Proc. of the Third International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, USA, 13-15 August (2002)
Nelson, A.L., Grant, E., Henderson, T.C.: Competitive relative performance evaluation of neural controllers for competitive game playing with teams of real mobile robots. In: Proc. of the Third International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, USA, 13-15 August (2002)
Zimmerman, N., et al.: Performance evaluation of tools and techniques for representing cost-based decision criteria for on-road autonomous navigation. In: Proc. of the Third International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, USA, 13-15 August (2002)
Scholtz, J., Antonishek, B., Young, J.: Evaluation of human-robot interaction in the nist reference search and rescue test arenas. In: Proc. of the Fourth International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, USA, 13-15 August (2002)
Burke, J.L., et al.: Task performance metrics in human-robot interaction: Taking a systems approach. In: Proc. of the Fourth International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, USA, 13-15 August (2002)
McCann, J.A., Huebscher, M.C.: Evaluation Issues in Autonomic Computing. In: Jin, H., et al. (eds.) GCC 2004. LNCS, vol. 3252, pp. 597–608. Springer, Heidelberg (2004)
Shih, T.K.: Evolution of mobile agents. In: Proc. of the First International Workshop on Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, MD, USA, 12-14 August (2000)
Krippendorff, K.: A Dictionary of Cybernetics. The American Society of Cybernetics, Norfolk, VA, USA (1986)
Kitchenham, B.A.: Evaluating software engineering methods and tool, part 2: selecting an appropriate evaluation method technical criteria. SIGSOFT Softw. Eng. Notes 21(2), 11–15 (1996), doi:10.1145/227531.227533
Grabisch, M., Orlovski, S.A., Yager, R.R.: Fuzzy aggregation of numerical preferences. In: Fuzzy Sets in Decision Analysis, Operations Research and Statistics, pp. 31–68 (1998)
Kontogounis, I., et al.: A robust agent design for dynamic scm environments. In: Antoniou, G., et al. (eds.) SETN 2006. LNCS (LNAI), vol. 3955, pp. 127–136. Springer, Heidelberg (2006)
Collins, J., et al.: The supply chain management game for the 2005 trading agent competition. Technical report, CMU (2004)
Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques. Morgan Kaufmann, San Francisco (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Dimou, C., Symeonidis, A.L., Mitkas, P.A. (2007). Evaluating Knowledge Intensive Multi-agent Systems. In: Gorodetsky, V., Zhang, C., Skormin, V.A., Cao, L. (eds) Autonomous Intelligent Systems: Multi-Agents and Data Mining. AIS-ADM 2007. Lecture Notes in Computer Science(), vol 4476. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72839-9_6
Download citation
DOI: https://doi.org/10.1007/978-3-540-72839-9_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72838-2
Online ISBN: 978-3-540-72839-9
eBook Packages: Computer ScienceComputer Science (R0)