Christos Dimou
ABSTRACT
Although the need for well-established engineering approaches in Intelligent Systems (IS) performance evaluation is urging, currently no widely accepted methodology exists, mainly due to lack of consensus on relevant definitions and scope of applicability, multi-disciplinary issues and immaturity of the field of IS. Even existing well-tested evaluation methodologies applied in other domains, such as (traditional) software engineering, prove inadequate to address the unpredictable emerging factors of the behavior of intelligent components. In this paper, we present a generic methodology and associated tools for evaluating the performance of IS, by exploiting the software agent paradigm as a representative modeling concept for intelligent systems. Based on the assessment of observable behavior of agents or multi-agent systems, the proposed methodology provides a concise set of guidelines and representation tools for evaluators to use. The methodology comprises three main tasks, namely metrics selection, monitoring agent activities for appropriate measurements, and aggregation of the conducted measurements. Coupled to this methodology is the Evaluator Agent Framework, which aims at the automation of most of the provided steps of the methodology, by providing Graphical User Interfaces for metrics organization and results presentation, as well as a code generating module that produces a skeleton of a monitoring agent. Once this agent is completed with domain-specific code, it is appended to the runtime of a multi-agent system and collects information from observable events and messages. Both the evaluation methodology and the automation framework are tested and demonstrated in Symbiosis, a MAS simulation environment for competing groups of autonomous entities.
- F. Bellifemine, A. Poggi, and G. Rimassa. Developing multi-agent systems with jade. In Eleventh International Workshop on Agent Theories, Architectures, and Languages (ATAL-2000), Boston, MA, USA, 2000. Google Scholar
Digital Library
- C. Dimou, A. Symeonidis, and P. Mitkas. Evaluating knowledge intensive multi-agent systems. In Proceedings of the Autonomous Information Systems - Agents and Data Mining Conference, St. Petersburg, Russia, 2007. Google ScholarDigital Library
- W. G. Multiagent systems. A modern approach to distributed artificial intelligence. The MIT Press, 1999.Google Scholar
- J. H. Gennari, M. A. Musen, R. W. Fergerson, W. E. Grosso, M. Crubzy, H. Eriksson, N. F. Noy, and S. W. Tu. The evolution of protg: An environment for knowledge-based systems development.Google Scholar
- N. Jennings. An agent-based approach for building complex software systems. Commun. ACM, 44(4):35--41, 2001. Google ScholarDigital Library
- L. P. Kaelbling, M. L. Littman, and A. P. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4:237--285, 1996. Google ScholarDigital Library
- F. Krebs and H. Bossel. Emergent value orientation in self-organization of an animat. Ecological Modelling, 96(1):143--164, 1997.Google ScholarCross Ref
- K. Krippendorff. A Dictionary of Cybernetics. The American Society of Cybernetics, Norfolk, VA, USA, 1986.Google Scholar
- F. Menczer, W. Street, and M. Degeratu. Evolving heterogeneous neural agents by local selection. In V. Honavar, M. Patel, and K. Balakrishnan, editors, Advances in the Evolutionary Synthesis of Neural Systems. MIT Press, Cambridge, MA, 2000.Google Scholar
- A. Newell and B. Buchanan. Artificial intelligence. Encyclopedia of Science and Technology, 2(1):146--150, 1997.Google Scholar
- H. Nwana. Software agents: An overview. Knowledge Engineering Review, 11:1--40, 1996.Google ScholarCross Ref
- T. R. Payne and P. Edwards. Learning mechanisms for information filtering agents. In J. L. Nealon and N. S. Taylor, editors, Proceedings of the UK Intelligent Agents Workshop, pages 163--183, Oxford, 1997. SGES Publications.Google Scholar
- V. Renganarayanan, A. Nalla, and A. Helal. Internet agents for effective collaboration, 2001.Google Scholar
- A. Symeonidis. Agent Intelligence through Data Mining. Springer Science and Business Media, 2005. Google ScholarDigital Library
- A. Turing. Computing machinery and intelligence. Mind, 59(1):443--460, 1950.Google Scholar
- F. Tzima, A. Symeonidis, and P. Mitkas. Symbiosis: Using predatorprey games as a test bed for studying competitive co-evolution. In Proceedings of the Knowledge Intensive Multi-Agent Systems (KIMAS-07), Boston, MA, USA, 2007.Google ScholarCross Ref
- M. Yokoo, E. Durfee, T. Ishida, and K. Kuwabara. Distributed constraint satisfaction for formalizing distributed problem solving. In International Conference on Distributed Computing Systems, pages 614--621, 1992.Google ScholarCross Ref
- L. Zadeh. Fuzzy sets. Information and Control, 8(1):338--353, 1965.Google ScholarCross Ref
- L. A. Zadeh. In quest of performance metrics for intelligent systemsa challenge that cannot be met with existing methods. In Proc. of the Third International Workshop on Performance Metrics for Intelligent Systems (PERMIS), 13--15 August 2002.Google Scholar
Recommendations
Design patterns for multi-agent programming
Various agent-based programming languages and frameworks have been proposed to support the development of autonomous agents and multi-agent systems. They have provided a valuable contribution to the identification and operationalisation of agent ...
MASD: Multi-agent systems development methodology
In recent years, multi-agent systems gained growing acceptance as a required technology to develop complex distributed systems. As result, there is an increased need for practical methodology for developing such systems. This paper presents a new Multi-...
A Plan Based Coalition Formation Model for Multi-agent Systems
WI-IAT '11: Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Volume 02This article addresses the coalition formation problem in a multi-agent context where agents plan their activities dynamically and use these plans to coordinate their actions and form suitable coalitions. In most coalition formation methods, when ...
Comments