Skip to main content
SpringerLink
Log in

Quality in model-driven engineering: a tertiary study

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Model-driven engineering (MDE) is believed to have a significant impact in software quality. However, researchers and practitioners may have a hard time locating consolidated evidence on this impact, as the available information is scattered in several different publications. Our goal is to aggregate consolidated findings on quality in MDE, facilitating the work of researchers and practitioners in learning about the coverage and main findings of existing work as well as identifying relatively unexplored niches of research that need further attention. We performed a tertiary study on quality in MDE, in order to gain a better understanding of its most prominent findings and existing challenges, as reported in the literature. We identified 22 systematic literature reviews and mapping studies and the most relevant quality attributes addressed by each of those studies, in the context of MDE. Maintainability is clearly the most often studied and reported quality attribute impacted by MDE. Eighty out of 83 research questions in the selected secondary studies have a structure that is more often associated with mapping existing research than with answering more concrete research questions (e.g., comparing two alternative MDE approaches with respect to their impact on a specific quality attribute). We briefly outline the main contributions of each of the selected literature reviews. In the collected studies, we observed a broad coverage of software product quality, although frequently accompanied by notes on how much more empirical research is needed to further validate existing claims. Relatively, little attention seems to be devoted to the impact of MDE on the quality in use of products developed using MDE.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. The DARE criteria are based on those used by the Center for Reviews and Dissemination of the University of York, for assessing the eligibility of systematic reviews to be included in their Database of Abstracts of Reviews of Effects (DARE). http://www.crd.york.ac.uk/CRDWeb/.

  2. Although Genero et al. (2011) published as a SLR, its 5 research questions are of a mapping nature. Under these circumstances, the quality classifications, which can be seen as adequacy for inclusion classifications, can be less strict.

References

  • Agner, L. T. W., Soares, I. W., Stadzisz, P. C., & Simão, J. M. (2013). A Brazilian survey on UML and model-driven practices for embedded software development. Journal of Systems and Software, 86(4), 997–1005. doi:10.1016/j.jss.2012.11.023.

    Article  Google Scholar 

  • Ameller, D., Franch, X., Gómez, C., Araujo, J., Svensson, R. B., Biffl, S., Cabot, J., Cortellessa, V., Daneva, M., Fernández, D. M., Moreira, A., Muccini, H., Vallecillo, A., Wimmer, M., Amaral, V., Brunelièrek, H., Burgueño, L., Goulão, M., Schätz, B., & Teufl, S. (2015) Handling non-functional requirements in model-driven development: An ongoing industrial survey. In 23rd IEEE international requirements engineering conference (RE 2015).

  • Badreddin, O., Lethbridge, T. C., & Elassar, M. (2013). Modeling practices in open source software. In Open source software: Quality verification (pp. 127–139). Springer. doi:10.1007/978-3-642-38928-3_9.

  • Badreddin, O., Sturm, A., Hamou-Lhadj, A., Lethbridge, T., Dixon, W., & Simmons, R. (2015). The effects of education on students’ perception of modeling in software engineering. In First International workshop on human factors in modeling (HuFaMo 2015), CEUR workshop proceedings

  • Biolchini, J., Mian, P. G., Natali, A. C. C., & Travassos, G. H. (2005). Systematic review in software engineering. Tech. Rep. RT–ES 679/05, System engineering and computer science department COPPE/UFRJ. http://www.cin.ufpe.br/~in1037/leitura/systematicReviewSE-COPPE.pdf.

  • Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., & Khalil, M. (2007). Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software, 80(4), 571–583. doi:10.1016/j.jss.2006.07.009.

    Article  Google Scholar 

  • Budgen, D., Kitchenham, B. A., Charters, S. M., Turner, M., Brereton, P., & Linkman, S. G. (2008). Presenting software engineering results using structured abstracts: A randomised experiment. Empirical Software Engineering, 13(4), 435–468. doi:10.1007/s10664-008-9075-7.

    Article  Google Scholar 

  • Cruzes, D. S., & Dybå, T. (2011). Research synthesis in software engineering: A tertiary study. Information and Software Technology, 53(5), 440–455. doi:10.1016/j.infsof.2011.01.004.

    Article  Google Scholar 

  • Cuadrado, J. S., Molina, J. G., & Tortosa, M. M. (2006). Rubytl: A practical, extensible transformation language. In A. Rensink & J. Warmer (Eds.), Model driven architecture–Foundations and applications (pp. 158–172). Bilbao, Spain: Springer.

  • Cuadrado, J. S., Izquierdo, J. L. C., & Molina, J. G. (2014). Applying model-driven engineering in small software enterprises. Science of Computer Programming, 89, 176–198. doi:10.1016/j.scico.2013.04.007.

    Article  Google Scholar 

  • Dybå, T., & Dingsøyr, T. (2008). Strength of evidence in systematic reviews in software engineering. In Proceedings of the second acm-ieee international symposium on empirical software engineering and measurement (pp. 178–187), ACM. doi:10.1145/1414004.1414034.

  • Fernández-Sáez, A. M., Genero, M., Caivano, D., & Chaudron, M. R. V. (2015). On the use of UML documentation in software maintenance: Results from a survey in industry. In ACM/IEEE 18th International conference on model driven engineering languages and systems (MODELS 2015) (pp. 292–301). ACM/IEEE.

  • Forward, A., Lethbridge, T., & Badreddin, O. (2010). Problems and opportunities for model-centric vs code-centric development: A survey of software professionals. In 5th Workshop “From code centric to model centric: Evaluating the effectiveness of MDD (C2M: EEMDD)”, University Pierre & Marie Curie, Paris.

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16,569–16,572. doi:10.1073/pnas.0507655102.

    Article  Google Scholar 

  • Hutchinson, J., Whittle, J., Rouncefield, M., & Kristoffersen, S. (2011). Empirical assessment of MDE in industry. In Proceedings of the 33rd international conference on software engineering (pp. 471–480). ACM, New York, NY, USA, ICSE ’11. doi:10.1145/1985793.1985858.

  • Hutchinson, J., Whittle, J., & Rouncefield, M. (2014). Model-driven engineering practices in industry: Social, organizational and managerial factors that lead to success or failure. Science of Computer Programming, 89, 144–161. doi:10.1016/j.scico.2013.03.017.

    Article  Google Scholar 

  • ISO/IEC. (2011). IEC25010:2011 Systems and software engineering–Systems and software Quality Requirements and Evaluation (SQuaRE)–System and software quality models. International Organization for Standardization, 34.

  • Kitchenham, B., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. In Technical report, Version 2.3 EBSE technical report. EBSE, Keele University and Durham University Joint Report. http://www.dur.ac.uk/ebse/resources/Systematic-reviews-5-8.pdf.

  • Kitchenham, B., Brereton, O. P., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009). Systematic literature reviews in software engineering-a systematic literature review. Information and Software Technology, 51(1), 7–15. doi:10.1016/j.infsof.2008.09.009.

    Article  Google Scholar 

  • Kitchenham, B., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M., et al. (2010). Systematic literature reviews in software engineering-a tertiary study. Information and Software Technology, 52(8), 792–805. doi:10.1016/j.infsof.2010.03.006.

    Article  Google Scholar 

  • Kitchenham, B. A., Dyba, T., & Jorgensen, M. (2004). Evidence-based software engineering. In Proceedings of the 26th international conference on software engineering (pp. 273–281). IEEE Computer Society. doi:10.1109/ICSE.2004.1317449.

  • Kitchenham, B. A., Budgen, D., & Brereton, O. P. (2011). Using mapping studies as the basis for further research-a participant-observer case study. Information and Software Technology, 53(6), 638–651. doi:10.1016/j.infsof.2010.12.011.

    Article  Google Scholar 

  • Mohagheghi, P., & Conradi, R. (2007). Quality, productivity and economic benefits of software reuse: A review of industrial studies. Empirical Software Engineering, 12(5), 471–516. doi:10.1007/s10664-007-9040-x.

    Article  Google Scholar 

  • Mohagheghi, P., Gilani, W., Stefanescu, A., & Fernandez, M. A. (2013). An empirical study of the state of the practice and acceptance of model-driven engineering in four industrial cases. Empirical Software Engineering, 18(1), 89–116. doi:10.1007/s10664-012-9196-x.

    Article  Google Scholar 

  • OMG (2015). OMG unified modeling language ™(OMG UML). Tech. Rep. formal/2015-03-01, Object Management Group. http://www.omg.org/spec/UML/2.5.

  • Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008). Systematic mapping studies in software engineering. In 12th international conference on evaluation and assessment in software engineering

  • Petre, M. (2013). Uml in practice. In D. Notkin, B. H. C. Cheng, & K. Pohl (Eds.), Proceedings of the 2013 international conference on software engineering (pp. 722–731). San Francisco, CA: IEEE Press. doi:10.1109/ICSE.2013.6606618.

  • Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. New York: Wiley.

    Google Scholar 

  • Schmidt, D. C. (2006). Guest editor’s introduction: Model-driven engineering. Computer, 39(2), 0025–31. doi:10.1109/MC.2006.58.

    Article  Google Scholar 

  • da Silva, A. R. (2015). Model-driven engineering: A survey supported by the unified conceptual model. Computer Languages, Systems and Structures, 43, 139–155. doi:10.1016/j.cl.2015.06.001.

    Google Scholar 

  • Torchiano, M., Tomassetti, F., Ricca, F., Tiso, A., & Reggio, G. (2013). Relevance, benefits, and problems of software modelling and model driven techniques-a survey in the Italian industry. Journal of Systems and Software, 86(8), 2110–2126. doi:10.1016/j.jss.2013.03.084.

    Article  Google Scholar 

  • Whittle, J., Hutchinson, J., Rouncefield, M., Burden, H., & Heldal, R. (2015). A taxonomy of tool-related issues affecting the adoption of model-driven engineering. Software and Systems Modeling, pp. 1–19. doi:10.1007/s10270-015-0487-8.

  • Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th international conference on evaluation and assessment in software engineering (pp. 38:1–38:10). ACM, New York, NY, USA, EASE ’14. doi:10.1145/2601248.2601268.

  • Zhou, Y., Zhang, H., Huang, X., Yang, S., Babar, M. A., Tang, H. (2015). Quality assessment of systematic reviews in software engineering: A tertiary study. In Proceedings of the 19th international conference on evaluation and assessment in software engineering (pp. 14:1–14:14). ACM, New York, NY, USA, EASE ’15. doi:10.1145/2745802.2745815.

Included Secondary Studies

  • Budgen, D., Burn, A. J., Brereton, O. P., Kitchenham, B. A., & Pretorius, R. (2011). Empirical evidence about the UML: A systematic literature review. Software: Practice and Experience, 41(4), 363–392. doi:10.1002/spe.1009.

    Google Scholar 

  • Delgado, A., Ruiz, F., de Guzmán, I. G. R., & Piattini, M. (2013). Main principles on the integration of SOC and MDD paradigms to business processes: A systematic review. In J. Cordeiro, M. Virvou, & B. Shishkov (Eds.), Software and Data Technologies (pp. 88–108). Athens, Greece: Springer. doi:10.1007/978-3-642-29578-2_6.

  • Domínguez, E., Pérez, B., Rubio, Á. L., & Zapata, M. A. (2012). A systematic review of code generation proposals from state machine specifications. Information and Software Technology, 54(10), 1045–1066. doi:10.1016/j.infsof.2012.04.008.

    Article  Google Scholar 

  • Genero, M., Fernández-Saez, A. M., Nelson, H. J., Poels, G., & Piattini, M. (2011). A systematic literature review on the quality of UML models. Journal of Database Management, 22(3), 46–70. doi:10.4018/978-1-4666-2044-5.ch012.

    Article  Google Scholar 

  • Giachetti, G., Valverde, F., & Marín, B. (2012). Interoperability for model-driven development: Current state and future challenges. In Sixth international conference on research challenges in information science (RCIS 2012) (pp. 1–10). IEEE. doi:10.1109/RCIS.2012.6240445.

  • Giraldo, F. D., España, S., & Pastor, O. (2014). Analysing the concept of quality in model-driven engineering literature: A systematic review. In Eighth international conference on research challenges in information science (RCIS 2014), (pp. 1–12). IEEE. doi:10.1109/RCIS.2014.6861030.

  • Giraldo, F. D., España, S., Giraldo, W. J., & Pastor, O. (2015). Modelling language quality evaluation in model-driven information systems engineering: a roadmap. In 9th International conference on research challenges in information science (RCIS 2015) (pp. 64–69). IEEE. doi:10.1109/RCIS.2015.7128864.

  • González, C. A., & Cabot, J. (2014). Formal verification of static software models in mde: A systematic review. Information and Software Technology, 56(8), 821–838. doi:10.1016/j.infsof.2014.03.003.

    Article  Google Scholar 

  • Hansson, S., Zhao, Y., & Burden, H. (2014). How MAD are we? empirical evidence for model-driven agile development. In 3rd Workshop on Extreme Modeling, XM 2014, CEUR Workshop Proceedings (pp. 2–11). vol 1239. URL http://ceur-ws.org/Vol-1239/XM2014#page=9.

  • Jensen, J., & Jaatun, M. G. (2011) Security in model driven development: A survey. In Availability, reliability and security (ARES), 2011 Sixth International conference on (pp. 704–709). IEEE. doi:10.1109/ARES.2011.110.

  • Loniewski, G., Insfran, E., & Abrahão, S. (2010). A systematic review of the use of requirements engineering techniques in model-driven development. In Model driven engineering languages and systems (pp. 213–227). Springer. doi:10.1007/978-3-642-16129-216.

  • Lucas, F. J., Molina, F., & Toval, A. (2009). A systematic review of UML model consistency management. Information and Software Technology, 51(12), 1631–1645. doi:10.1016/j.infsof.2009.04.009.

    Article  Google Scholar 

  • Malavolta, I., & Muccini, H. (2014). A study on MDE approaches for engineering wireless sensor networks. In Software engineering and advanced applications (SEAA), 2014 40th EUROMICRO conference on (pp. 149–157). IEEE. doi:10.1109/SEAA.2014.61.

  • Mehmood, A., & Jawawi, D. N. (2013). Aspect-oriented model-driven code generation: A systematic mapping study. Information and Software Technology, 55(2), 395–411. doi:10.1016/j.infsof.2012.09.003.

    Article  Google Scholar 

  • Misbhauddin, M., & Alshayeb, M. (2015). Uml model refactoring: A systematic literature review. Empirical Software Engineering, 20(1), 206–251. doi:10.1007/s10664-013-9283-7.

    Article  Google Scholar 

  • Mohagheghi, P., & Dehlen, V. (2008). Where is the proof?—a review of experiences from applying mde in industry. In Schieferdecker, I., Hartman, A. (eds) Model driven architecture—foundations and applications, Lecture notes in computer Science (pp. 432–443). vol 5095, Springer, Berlin. doi:10.1007/978-3-540-69100-631.

  • Mohagheghi, P., Dehlen, V., & Neple, T. (2009). Definitions and approaches to model quality in model-based software development—A review of literature. Information and Software Technology, 51(12), 1646–1669. doi:10.1016/j.infsof.2009.04.004.

    Article  Google Scholar 

  • Neto, V. V. G., Guessi, M., Oliveira, L. B. R., Oquendo, F., & Nakagawa, E. Y. (2014). Investigating the model-driven development for systems-of-systems. In Proceedings of the 2014 European Conference on software architecture workshops (pp. 22:1–22:8). ACM, New York, NY, USA, ECSAW ’14. doi:10.1145/2642803.2642825.

  • Nguyen, P. H., Klein, J., le Traon, Y., & Kramer, M. E. (2013) A systematic review of model-driven security. In 20th Asia–Pacific software engineering conference (APSEC 2013) (pp. 432–441). vol 1. doi:10.1109/APSEC.2013.64.

  • Santiago, I., Jiménez, Á., Vara, J. M., De Castro, V., Bollati, V. A., & Marcos, E. (2012). Model-driven engineering as a new landscape for traceability management: A systematic literature review. Information and Software Technology, 54(12), 1340–1356. doi:10.1016/j.infsof.2012.07.008.

    Article  Google Scholar 

  • Szvetits, M., & Zdun, U. (2013). Systematic literature review of the objectives, techniques, kinds, and architectures of models at runtime. Software and Systems Modeling, pp. 1–39. doi:10.1007/s10270-013-0394-9.

  • Yue, T., Briand, L. C., & Labiche, Y. (2011). A systematic review of transformation approaches between user requirements and analysis models. Requirements Engineering, 16(2), 75–99. doi:10.1007/s00766-010-0111-y.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank FCT/MEC NOVA LINCS PEst UID/ CEC/04516/ 2013 for the financial support to this work.

Author information

Authors and Affiliations

  1. NOVA LINCS, Departamento de Informática, Universidade Nova de Lisboa, Lisbon, Portugal

    Miguel Goulão & Vasco Amaral

  2. Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia

    Marjan Mernik

Authors
  1. Miguel Goulão
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vasco Amaral
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marjan Mernik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miguel Goulão.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goulão, M., Amaral, V. & Mernik, M. Quality in model-driven engineering: a tertiary study. Software Qual J 24, 601–633 (2016). https://doi.org/10.1007/s11219-016-9324-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9324-8

Keywords

Search

Navigation