Skip to main content
SpringerLink
Log in

A systematic approach to evaluating domain-specific modeling language environments for multi-agent systems

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Multi-agent systems (MASs) include multiple interacting agents within an environment to provide a solution for complex systems that cannot be easily solved with individual agents or monolithic systems. However, the development of MASs is not trivial due to the various agent properties such as autonomy, responsiveness, and proactiveness, and the need for realization of the many different agent interactions. To support the development of MASs various domain-specific modeling languages (DSMLs) have been introduced that provide a declarative approach for modeling and supporting the generation of agent-based systems. To be effective, the proposed DSMLs need to meet the various stakeholder concerns and the related quality criteria for the corresponding MASs. Unfortunately, very often the evaluation of the DSML is completely missing or has been carried out in idiosyncratic approach. If the DSMLs are not well defined, then implicitly this will have an impact on the quality of the MASs. In this paper, we present an evaluation framework and systematic approach for assessing existing or newly defined DSMLs for MASs. The evaluation is specific for MAS DSMLs and targets both the language and the corresponding tools. To illustrate the evaluation approach, we first present SEA_ML, which is a model-driven MAS DSML for supporting the modeling and generation of agent-based systems. The evaluation of SEA_ML is based on a multi-case study research approach and provides both qualitative evaluation and quantitative analysis. We report on the lessons learned considering the adoption of the evaluation approach as well as the SEA_ML for supporting the generation of agent-based systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. We keep the descriptions of the case studies short in this paper in order to keep focus on the evaluation. However, detailed descriptions of each case study, their models according to the SEA_ML specifications with including all modeling and code generation tools of SEA_ML can be found online inside a bundle at: http://mas.ube.ege.edu.tr/downloads/sea_ml_bundle.zip.

References

  • Abran, A. (2010). Software metrics and software metrology. New Jersey: Wiley and IEEE-CS Press.

    Book  MATH  Google Scholar 

  • AOS. (2001). Agent Oriented Software Pty., Ltd. JACK Environment. http://www.aosgrp.com/products/jack/. Last Accessed August 2015.

  • Barisic, A. (2013). Evaluating the quality in use of domain-specific languages in an agile way. In Proceedings of the doctoral symposium at 16th international conference on model driven engineering languages and systems (MODELS), September, 2013, CEUR-WS.

  • Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011a). How to reach a usable DSL? Moving toward a systematic evaluation. Electronic Communications of the EASST, 50, 1–12.

    Google Scholar 

  • Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2011b). Quality in use of domain-specific languages: A case study. In Proceedings of the 3rd ACM SIGPLAN workshop on evaluation and usability of programming languages and tools (pp. 65–72). PLATEAU’11, New York, USA.

  • Barisic, A., Amaral, V., Goulao, M., & Barroca, B. (2012a). Evaluating the usability of domain-specific languages. In M. Mernik (Ed.), Formal and practical aspects of domain-specific languages: Recent developments (pp. 386–407). Hershey: IGI Global.

    Google Scholar 

  • Barisic, A., Amaral, V., Goulao, M., & Monteiro, M. P. (2012b). Patterns for evaluating usability of domain-specific languages. In Proceedings of the 19th conference on pattern languages of programs (PLoP), SPLASH.

  • Basili, V., Shull, F., & Lanubile, F. (1999). Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4), 456–473.

    Article  Google Scholar 

  • Bayona-Oré, S., Calvo-Manzano, J. A., Cuevas, G., & San-Feliu, T. (2014). Critical success factors taxonomy for software process deployment. Software Quality Journal, 22(1), 21–48.

    Article  Google Scholar 

  • Bellifemine, F., Poggi, A., & Rimassa, G. (2001). Developing multi-agent systems with a FIPA-compliant agent framework. Software: Practice and Experience, 31(2), 103–128.

    MATH  Google Scholar 

  • Berners-Lee, T., Hendler, J., & Lassila, O. (2001). The semantic web. Scientific American, 284(5), 34–43.

    Article  Google Scholar 

  • Bernon, C., Cossentino, M., Gleizes, M.-P., Turci, P., & Zambonelli, F. (2005). A study of some multi-agent meta-models. Lecture Notes in Computer Science, 3382, 62–77.

    Article  Google Scholar 

  • Beydoun, G., Low, G. C., Henderson-Sellers, B., Mouratidis, H., Gomez-Sanz, J. J., Pavon, J., & Gonzalez-Perez, C. (2009). FAML: A generic metamodel for MAS development. IEEE Transactions on Software Engineering, 35(6), 841–863.

    Article  Google Scholar 

  • Bobkowska, A. (2005). A methodology of visual modeling language evaluation. In SOFSEM 2005: Theory and practice of computer systems. Lecture Notes in Computer Science (Vol. 3381).

  • Bresciani, P., Perini, A., Giorgini, P., Giunchiglia, F., & Mylopoulos, J. (2004). Tropos: An agent-oriented software development methodology. Autonomous Agents and Multi-Agent Systems, 8(3), 203–236.

    Article  MATH  Google Scholar 

  • Carver, J., Jaccheri, L., Morasca, S., & Shull, F. (2010). A checklist for integrating student empirical studies with research and teaching goals. Empirical Software Engineering, 15(1), 35–59.

    Article  Google Scholar 

  • Challenger, M., Demirkol, S., Getir, S., Mernik, M., Kardas, G., & Kosar, T. (2014). On the use of a domain-specific modeling language in the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 28, 111–141.

    Article  Google Scholar 

  • Challenger, M., Getir, S., Demirkol, S., & Kardas, G. (2011). A domain specific metamodel for semantic web enabled multi-agent systems. Lecture Notes in Business Information Processing, 83, 177–186.

    Article  Google Scholar 

  • Ciobanu, G., & Juravle, C. (2012). Flexible software architecture and language for mobile agents. Concurrency and Computation: Practice and Experience, 24(6), 559–571.

    Article  Google Scholar 

  • Clark, T., Evans, A., Sammut, P., & Willans, J. (2004). Language driven development and MDA. MDA Journal, 2–13. http://www.bptrends.com/bpt/wp-content/publicationfiles/10-04%20COL%20MDA%20-%20Frankel%20-%20Xactium.pdf. Last Accessed August 2015.

  • CMU/SEI. (2010). Software Engineering Institute, CMMI for Acquisition, Version 1.3, Technical Report, CMU/SEI-2010-TR-032, Carnegie Mellon (November 2012).

  • Demirkol, S., Challenger, M., Getir, S., Kosar, T., Kardas, G., & Mernik, M. (2013). A DSL for the development of software agents working within a semantic web environment. Computer Science and Information Systems, Special Issue on Advances in Model Driven Engineering, Languages and Agents, 10(4), 1525–1556.

    Google Scholar 

  • Demirkol, S., Getir, S., Challenger, M., & Kardas, G. (2011). Development of an agent based E-barter System. In: International symposium on innovations in intelligent systems and applications (INISTA) (pp. 193–198). IEEE Computer Society, Istanbul-Turkey.

  • Demirli, E., & Tekinerdogan, B. (2011). Software language engineering of architectural viewpoints. In Proceeding of the 5th European Conference on Software Architecture (ECSA 2011), Lecture Notes on Computer Science (Vol. 6903, pp. 336–343).

  • Eclipse. (2004). The Eclipse Foundation, Eclipse Integrated Development Environment (IDE). http://www.eclipse.org/. Last Accessed August 2015.

  • Fister, I, Jr, Fister, I., Mernik, M., & Brest, J. (2011). Design and implementation of domain-specific language easytime. Computer Languages, Systems and Structures, 37(4), 151–167.

    Article  Google Scholar 

  • Fowler, M. (2011). Domain-specific languages. Boston: Addison-Wesley.

    Google Scholar 

  • Fuentes-Fernandez, R., Garcia-Magarino, I., Gomez-Rodriguez, A. M., & Gonzalez-Moreno, J. C. (2010). A technique for defining agent-oriented engineering processes with tool support. Engineering Applications of Artificial Intelligence, 23(3), 432–444.

    Article  Google Scholar 

  • Garcia, A. F., Sant’Anna, C., Figueiredo, E., Kulesza, U., Pereira de Lucena, C. J., & von Staa, A. (2005). Modularizing design patterns with aspects: A quantitative study. In Proceedings of the 4th international conference on Aspect-oriented software development (AOSD 2005) (pp. 3–14).

  • Gascuena, J. M., Navarro, E., & Fernandez-Caballero, A. (2012). Model-driven engineering techniques for the development of multi-agent systems. Engineering Applications of Artificial Intelligence, 25(1), 159–173.

    Article  Google Scholar 

  • Getir, S., Challenger, M., Demirkol, S., & Kardas, G. (2012). The semantics of the interaction between agents and web services on the semantic web. In Proceedings of the 7th IEEE international workshop on engineering semantic agent systems (ESAS 2012), held in conjunction with the 36th IEEE signature conference on computers, software, and applications (COMPSAC 2012) (pp. 619–624).

  • Getir, S., Challenger, M., & Kardas, G. (2014). The formal semantics of a domain-specific modeling language for semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 23(3), 1450005.

    Article  Google Scholar 

  • Gray, J., Tolvanen, J.-P., Kelly, S., Gokhale, A., Neema, S., & Sprinkle, J. (2007). Domain-specific modeling. In P. A. Fishwick (Ed.), Handbook of dynamic system modeling (pp. 1–7). Boca Raton: CRC Press.

    Google Scholar 

  • Hahn, C. (2008). A domain specific language for multiagent systems. In Proceedings of the 7th autonomous agents and multiagent systems conference (AAMAS’08), Estoril, Portugal (pp. 233–240).

  • Hahn, C., Madrigal-Mora, C., & Fischer, K. (2009). A platform-independent metamodel for multiagent systems. Autonomous Agents and Multi-Agent Systems, 18(2), 239–266.

    Article  Google Scholar 

  • Hahn, C., Nesbigall, S., Warwas, S., Zinnikus, I., Fischer, K., & Klusch, M. (2008). Integration of multiagent systems and semantic web services on a platform independent level. In Proceedings of IEEE/WIC/ACM international conference on web intelligence and intelligent agent technology (WI-IAT 2008), Sydney, Australia (pp. 200–206).

  • Hannemann, J., & Kiczales, G. (2002). Design patterns implementation in Java and AspectJ. In Proceeding of object oriented programming systems languages and applications (OOPSLA’02) (pp. 161–173).

  • ISO/IEC. (1998). International Organization for Standardization (Ed.), ISO/IEC TR 15504-2. Information Technology-Software Process Assessment—Part 2: A reference model for processes and process capability. Case Postale 56, CH-1211 Geneva, Switzerland.

  • ISO/IEC. (2011). ISO/IEC 25010. Systems and software engineering systems and software quality requirements and evaluation (SQuaRE), System and software quality models, International Standards Organization/International Electrotechnical Commission.

  • Jackson, D. (2002). Alloy: A lightweight object modeling notation. ACM Transactions on Software Engineering and Methodology, 11(2), 256–290.

    Article  Google Scholar 

  • Jouault, F., Allilaire, F., Bezivin, J., & Kurtev, I. (2008). ATL: A model transformation tool. Science of Computer Programming, 72(1–2), 31–39.

    Article  MathSciNet  MATH  Google Scholar 

  • Kahlaoui, A., Abran, A., & Lefebvre, E. (2008). DSML success factors and their assessment criteria. Metrics News, 13(1), 43–51.

    Google Scholar 

  • Kahraman, G., & Bilgen, S. (2013). A framework for qualitative assessment of domain-specific languages. Software & Systems Modeling. doi:10.1007/s10270-013-0387-8.

    Google Scholar 

  • Kardas, G., Challenger, M., Yildirim, S., & Yamuc, A. (2012). Design and implementation of a multi-agent stock trading system. Software Practice and Experience, 42, 1247–1273.

    Article  Google Scholar 

  • Kardas, G., Goknil, A., Dikenelli, O., & Topaloglu, N. Y. (2009). Model driven development of semantic web enabled multi-agent systems. International Journal of Cooperative Information Systems, 18(2), 261–308.

    Article  Google Scholar 

  • Kelly, S., & Tolvanen, J.-P. (2008). Domain-specific modeling- enabling full code generation. Los Alamitos: IEEE Computer Society Publications.

    Book  Google Scholar 

  • Khedker, U. P. (1997). What makes a good programming language? Technical Report TR-97-upk-1, Department of Computer Science, University of Pune.

  • Kirstan, S., & Zimmermann, J. (2010). Evaluating costs and benefits of model-based development of embedded software systems in the car industry—Results of a qualitative Case Study. In Proceedings of ECMFA 2010 workshop C2 M:EEMDD-from Code Centric to Model Centric: Evaluating the effectiveness of MDD (pp 18–29).

  • Kitchenham, B. A., Linkman, S., & Law, D. (1997). DESMET: A methodology for evaluating software engineering methods and tools. Computing and Control Engineering Journal, 8(3), 120–126.

    Article  Google Scholar 

  • Kosar, T., Mernik, M., & Carver, J. C. (2012). Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments. Empircal Software Engineering, 17, 276–304.

    Article  Google Scholar 

  • Kosar, T., Oliveira, N., Mernik, M., Varanda Pereira, M. J., Crepinsek, M., Carneiro da Cruz, D., & Henriques, P. R. (2010). Comparing general-purpose and domain-specific languages: An empirical study. Computer Science and Information Systems, 7(2), 247–264.

    Article  Google Scholar 

  • Marin, B., Pastor, O., & Abran, A. (2010). Towards an accurate functional size measurement procedure for conceptual models in an MDA environment. Journal of Data and Knowledge Engineering, 69(5), 472–490.

    Article  Google Scholar 

  • Martin, D., Burstein, M., Hobbs, J., Lassila, O., McDermott, D., McIlraith, S., Narayanan, S., Paolucci, M., Parsia, B., Payne, T., Sirin, E., Srinivasan, N., & Sycara, K. (2004). OWL-S: Semantic markup for web services. W3C Member Submission. http://www.w3.org/Submission/OWL-S/. Last Accessed August 2015.

  • Mernik, M., Heering, J., & Sloane, A. (2005). When and how to develop domain-specific languages. ACM Computing Surveys, 37(4), 316–344.

    Article  Google Scholar 

  • Mernik, M., & Zumer, V. (2005). Incremental programming language development. Computer Languages, Systems and Structures, 31(1), 1–16.

    Article  MATH  Google Scholar 

  • MOFScript. (2005). The Eclipse Foundation, MOFScript model-to-text transformation language and tool. http://www.eclipse.org/gmt/mofscript/. Last Accessed August 2015.

  • Mohagheghi, P., & Dehlen, V. (2008). Where is the proof? A review of experiences from applying MDE in industry. In: I. Schieferdecker, & A. Hartman (Eds.) ECMDA-FA 2008. Lecture notes in computer science (Vol. 5095, pp. 432–443).

  • Mohagheghi, P., Gilani, W., Stefanescu, A., & Fernandez, M. A. (2013). An empirical study of the state of the practice and acceptance of model-driven engineering in four industrial cases. Empirical Software Engineering, 18, 89–116.

    Article  Google Scholar 

  • Monteiro, M. P., & Fernandes, J. M. (2005). Towards a catalog of aspect oriented refactorings. In Proceedings of the 4th international conference on aspect oriented software development (AOSD’05) (pp. 111–122).

  • OCL. (2012). Object Management Group, Object Constraint Language (OCL). http://www.omg.org/spec/OCL/2.3.1/. Last Accessed August 2015.

  • Odell, J., Nodine, M., & Levy, R. (2005). A metamodel for agents, roles, and groups. Agent-oriented software engineering. Lecture Notes in Computer Science, 3382, 78–92.

    Article  Google Scholar 

  • Oldevik, J., Neple, T., Gronmo, R., Aagedal, J., & Berre, A. J. (2005). Toward standardized model to text transformations. Lecture Notes in Computer Science, 3748, 239–253.

    Article  Google Scholar 

  • Omicini, A., Ricci, A., & Viroli, M. (2008). Artifacts in the A&A meta-model for multi-agent systems. Autonomous Agents and Multi-Agent Systems, 17(3), 432–456.

    Article  Google Scholar 

  • Padgham, L., & Winikoff, M. (2004). Developing intelligent agent systems: A practical guide. New York: Wiley.

    Book  MATH  Google Scholar 

  • Paige, R. F., Ostroff, J. S., & Brooke, P. J. (2000). Principles for modeling language design. Information and Software Technology, 42(10), 665–675.

    Article  Google Scholar 

  • Pavon, J., Gomez-Sanz, J. J., & Fuentes, R. (2006). Model driven development of multi-agent systems. Lecture Notes in Computer Science, 4066, 284–298.

    Article  Google Scholar 

  • Pešović, D., Vidaković, M., Ivanović, M., Budimac, Z., & Vidaković, J. (2011). Usage of agents in document management. Computer Science and Information Systems, 8, 193–210.

    Article  Google Scholar 

  • Pokahr, A., Braubach, L., & Lamersdorf, W. (2005). JADEX: A BDI reasoning engine. In R. H. Bordini, et al. (Eds.), Multi-agent programming languages, platforms and applications (pp. 149–174). Berlin: Springer.

    Chapter  Google Scholar 

  • Rao, A., & Georgeff, M. (1995). BDI agents: From theory to practice. In Proceedings of the 1st international conference on multi-agent systems (ICMAS-95), San Francisco (pp. 312–319).

  • Rougemaille, S., Migeon, F., Maurel, C., & Gleizes, M.-P. (2007). Model driven engineering for designing adaptive multi-agent systems. Lecture Notes in Artificial Intelligence, 4995, 318–332.

    Google Scholar 

  • Runeson, P., & Höst, M. (2009). Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering, 14, 131–164.

    Article  Google Scholar 

  • Russell, S. J., & Norvig, P. (2003). Artificial intelligence: A modern approach (2nd ed.). USA: Pearson Education.

    MATH  Google Scholar 

  • Saritas, H. B., & Kardas, G. (2014). A model driven architecture for the development of smart card software. Computer Languages, Systems and Structures, 40(2), 53–72.

    Article  Google Scholar 

  • Schmid, K., & John, I. (2002). Developing, validating and evolving an approach to product line benefit and risk assessment. In Proceedings of the 28th Euro-micro conference (EUROMICRO’02) (pp. 272–283).

  • Schmidt, D. C. (2006). Guest editor’s introduction: Model-driven engineering. IEEE Computer, 39(2), 25–31.

    Article  Google Scholar 

  • Shadbolt, N., Hall, W., & Berners-Lee, T. (2006). The semantic web revisited. IEEE Intelligent Systems, 21(3), 96–101.

    Article  Google Scholar 

  • Shih, D. H., Huang, S. Y., & Yen, D. C. (2005). A new reverse auction agent system for M-commerce using mobile agents. Computer Standards and Interfaces, 27, 383–395.

    Article  Google Scholar 

  • Smith, R. (1980). The Contract Net protocol: High Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers, 29(12), 1104–1113.

    Article  Google Scholar 

  • Sprinkle, J., Mernik, M., Tolvanen, J.-P., & Spinellis, D. (2009). Guest editors’ introduction: What kinds of nails need a domain-specific hammer? IEEE Software, 26(4), 15–18.

    Article  Google Scholar 

  • Sycara, K. (1998). Multiagent systems. AI Magazine, 19(4), 79–92.

    Google Scholar 

  • Sycara, K., Paolucci, M., Ankolekar, A., & Srinivasan, N. (2003). Automated discovery, interaction and composition of semantic web services. Journal of Web Semantics, 1(1), 27–46.

    Article  Google Scholar 

  • Tekinerdogan, B., Bozbey, S., Mester, Y., Turançiftci, E., & Alkışlar, L. (2011). An aspect-oriented tool framework for developing process-sensitive embedded user assistance systems. In Transactions on aspect-oriented software development VIII. Lecture notes computer science (Vol. 6580, pp. 196–220).

  • Tekinerdogan, B., & Demirli, E. (2013). Evaluation framework for software architecture viewpoint languages. In Proceedings of 9th international ACM Sigsoft conference on the quality of software architectures (QoSA 2013) (pp. 89–98).

  • van Deursen, A., Klint, P., & Visser, J. (2000). Domain-specific languages: An annotated bibliography. ACM Sigplan Notices, 35(6), 26–36.

    Article  Google Scholar 

  • Varanda-Pereira, M. J., Mernik, M., Da-Cruz, D., & Henriques, P. R. (2008). Program comprehension for domain-specific languages. Computer Science and Information Systems, 5(2), 1–17.

    Article  Google Scholar 

  • Warmer, J., & Kleppe, A. G. (1998). The object constraint language: Precise modeling with UML. Boston: Addison-Wesley.

    Google Scholar 

  • Wile, D. (2004). Lessons learned from real DSL experiments. Science of Computer Programming, 51, 265–290.

    Article  MathSciNet  Google Scholar 

  • Wooldridge, M., & Jennings, N. R. (1995). Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2), 115–152.

    Article  Google Scholar 

  • Zambonelli, F., Jennings, N. R., & Wooldrige, M. (2003). Developing multiagent systems: The Gaia methodology. ACM Transactions on Software Engineering and Methodologies, 12(3), 317–370.

    Article  Google Scholar 

Download references

Acknowledgments

This study is partially funded by the Scientific and Technological Research Council of Turkey (TUBITAK) Electric, Electronic and Informatics Research Group (EEEAG) under Grant 109E125. The authors wish to thank the former members of the SEA_ML project, Dr. Marjan Mernik, Dr. Tomaz Kosar, Sinem Getir and Sebla Demirkol for their valuable contributions on design and implementation of SEA_ML. Finally, the authors would like to thank the participants, various Ph.D. and M.Sc. students, who actively got involved in the language evaluation and helped to the realization of the empirical study presented in this paper.

Author information

Authors and Affiliations

  1. International Computer Institute, Ege University, Izmir, Turkey

    Moharram Challenger & Geylani Kardas

  2. Information Technology Group, Wageningen University, Wageningen, The Netherlands

    Bedir Tekinerdogan

Authors
  1. Moharram Challenger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geylani Kardas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bedir Tekinerdogan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Geylani Kardas.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Challenger, M., Kardas, G. & Tekinerdogan, B. A systematic approach to evaluating domain-specific modeling language environments for multi-agent systems. Software Qual J 24, 755–795 (2016). https://doi.org/10.1007/s11219-015-9291-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-015-9291-5

Keywords

Search

Navigation