Skip to main content
SpringerLink
Log in

On the Replicability of Experimental Tool Evaluations in Model-Based Development

Lessons Learnt from a Systematic Literature Review Focusing on MATLAB/Simulink

  • Conference paper
  • First Online:
Systems Modelling and Management (ICSMM 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1262))

Included in the following conference series:

Abstract

Research on novel tools for model-based development differs from a mere engineering task by providing some form of evidence that a tool is effective. This is typically achieved by experimental evaluations. Following principles of good scientific practice, both the tool and the models used in the experiments should be made available along with a paper. We investigate to which degree these basic prerequisites for the replicability of experimental results are met by recent research reporting on novel methods, techniques, or algorithms supporting model-based development using MATLAB/Simulink. Our results from a systematic literature review are rather unsatisfactory. In a nutshell, we found that only 31% of the tools and 22% of the models used as experimental subjects are accessible. Given that both artifacts are needed for a replication study, only 9% of the tool evaluations presented in the examined papers can be classified to be replicable in principle. Given that tools are still being listed among the major obstacles of a more widespread adoption of model-based principles in practice, we see this as an alarming signal. While we are convinced that this can only be achieved as a community effort, this paper is meant to serve as starting point for discussion, based on the lessons learnt from our study.

This work has been supported by the German Ministry of Research and Education (BMBF) under grant 01IS18091B in terms of the research project SimuComp.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://dl.acm.org.

  2. 2.

    https://ieeexplore.ieee.org/Xplore/home.jsp.

  3. 3.

    https://www.sciencedirect.com.

  4. 4.

    https://dblp.uni-trier.de.

  5. 5.

    ScienceDirect presented an initial selection of 217 papers on their web interface, out of which 214 could be downloaded.

  6. 6.

    https://www.jabref.org.

  7. 7.

    https://www.researchgate.net.

  8. 8.

    https://www.se-rwth.de/materials/cncviewscasestudy.

  9. 9.

    https://www.dropbox.com.

  10. 10.

    https://cloud.google.com/bigquery.

References

  1. Arrieta, A., Wang, S., Arruabarrena, A., Markiegi, U., Sagardui, G., Etxeberria, L.: Multi-objective black-box test case selection for cost-effectively testing simulation models. In: Proceedings of the Genetic and Evolutionary Computation Conference GECCO 2018, pp. 1411–1418. Association for Computing Machinery, New York (2018)

    Google Scholar 

  2. Arrieta, A., Wang, S., Markiegi, U., Arruabarrena, A., Etxeberria, L., Sagardui, G.: Pareto efficient multi-objective black-box test case selection for simulation-based testing. Inf. Softw. Technol. 114, 137–154 (2019)

    Article  Google Scholar 

  3. Bertram, V., Maoz, S., Ringert, J.O., Rumpe, B., von Wenckstern, M.: Component and connector views in practice: an experience report. In: Proceedings of the ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems MODELS 2017, pp. 167–177. IEEE Press (2017)

    Google Scholar 

  4. Bertram, V., Maoz, S., Ringert, J.O., Rumpe, B., von Wenckstern, M.: Component and connector views in practice: an experience report. In: 2017 ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems (MODELS), pp. 167–177. IEEE (2017)

    Google Scholar 

  5. Boll, A., Kehrer, T.: The download link of all digital artifacts of this paper. https://doi.org/10.6084/m9.figshare.13019183.v1

  6. Brambilla, M., Cabot, J., Wimmer, M.: Model-driven software engineering in practice. Synth. Lect. Softw. Eng. 3(1), 1–207 (2017)

    Article  Google Scholar 

  7. Chowdhury, S.A.: Understanding and improving cyber-physical system models and development tools. In: 2018 IEEE/ACM 40th International Conference on Software Engineering: Companion (ICSE-Companion), pp. 452–453, May 2018

    Google Scholar 

  8. Chowdhury, S.A., Mohian, S., Mehra, S., Gawsane, S., Johnson, T.T., Csallner, C.: Automatically finding bugs in a commercial cyber-physical system development tool chain with SLforge. In: 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 981–992, May 2018

    Google Scholar 

  9. Chowdhury, S.A., Shrestha, S.L., Johnson, T.T., Csallner, C.: SLEMI: Equivalence Modulo Input (EMI) based mutation of CPS models for finding compiler bugs in Simulink. In: Proceedings of 42nd ACM/IEEE International Conference on Software Engineering (ICSE). ACM (2020, To appear)

    Google Scholar 

  10. Chowdhury, S.A., Varghese, L.S., Mohian, S., Johnson, T.T., Csallner, C.: A curated corpus of simulink models for model-based empirical studies. In: 2018 IEEE/ACM 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS), pp. 45–48. IEEE (2018)

    Google Scholar 

  11. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)

    Article  Google Scholar 

  12. Elberzhager, F., Rosbach, A., Bauer, T.: Analysis and testing of Matlab simulink models: a systematic mapping study. In: Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to Testing Automation JAMAICA 2013, pp. 29–34. Association for Computing Machinery, New York (2013)

    Google Scholar 

  13. France, R., Rumpe, B.: Model-driven development of complex software: a research roadmap. In: Future of Software Engineering (FOSE 2007), pp. 37–54. IEEE (2007)

    Google Scholar 

  14. Gallego-Calderon, J., Natarajan, A.: Assessment of wind turbine drive-train fatigue loads under torsional excitation. Eng. Struct. 103, 189–202 (2015)

    Article  Google Scholar 

  15. Gerlitz, T., Kowalewski, S.: Flow sensitive slicing for MATLAB/Simulink models. In: 2016 13th Working IEEE/IFIP Conference on Software Architecture (WICSA), pp. 81–90, April 2016

    Google Scholar 

  16. Gusenbauer, M., Haddaway, N.R.: Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed and 26 other resources. Res. Synth. Methods 11, 181–217 (2019)

    Article  Google Scholar 

  17. Hebig, R., Quang, T.H., Chaudron, M.R., Robles, G., Fernandez, M.A.: The quest for open source projects that use UML: mining GitHub. In: Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems, pp. 173–183 (2016)

    Google Scholar 

  18. Holling, D., Hofbauer, A., Pretschner, A., Gemmar, M.: Profiting from unit tests for integration testing. In: 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 353–363, April 2016

    Google Scholar 

  19. Hussain, A., Sher, H.A., Murtaza, A.F., Al-Haddad, K.: Improved restricted control set model predictive control (iRCS-MPC) based maximum power point tracking of photovoltaic module. IEEE Access 7, 149422–149432 (2019)

    Article  Google Scholar 

  20. Jiang, Z., Wu, X., Dong, Z., Mu, M.: Optimal test case generation for Simulink models using slicing. In: 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 363–369, July 2017

    Google Scholar 

  21. Kehrer, T., Kelter, U., Ohrndorf, M., Sollbach, T.: Understanding model evolution through semantically lifting model differences with SiLift. In: 28th IEEE International Conference on Software Maintenance (ICSM), pp. 638–641. IEEE (2012)

    Google Scholar 

  22. Kehrer, T., Kelter, U., Pietsch, P., Schmidt, M.: Adaptability of model comparison tools. In: Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering, pp. 306–309. IEEE (2012)

    Google Scholar 

  23. Khelifi, A., Ben Lakhal, N.M., Gharsallaoui, H., Nasri, O.: Artificial neural network-based fault detection. In: 2018 5th International Conference on Control, Decision and Information Technologies (CoDIT), pp. 1017–1022, April 2018

    Google Scholar 

  24. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering (2007)

    Google Scholar 

  25. Kitchenham, B., et al.: Systematic literature reviews in software engineering-a tertiary study. Inf. Softw. Technol. 52(8), 792–805 (2010)

    Article  Google Scholar 

  26. Kuroki, Y., Yoo, M., Yokoyama, T.: A Simulink to UML model transformation tool for embedded control software development. In: IEEE International Conference on Industrial Technology, ICIT 2016, Taipei, Taiwan, 14–17 March 2016, pp. 700–706. IEEE (2016)

    Google Scholar 

  27. Kusmenko, E., Shumeiko, I., Rumpe, B., von Wenckstern, M.: Fast simulation preorder algorithm. In: Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development MODELSWARD 2018, pp. 256–267. SCITEPRESS - Science and Technology Publications, Lda, Setubal, PRT (2018)

    Google Scholar 

  28. Lamprecht, A.L., et al.: Towards fair principles for research software. Data Sci. 1–23 (2019, Preprint)

    Google Scholar 

  29. Langer, P., Mayerhofer, T., Wimmer, M., Kappel, G.: On the usage of UML: initial results of analyzing open UML models. In: Modellierung 2014 (2014)

    Google Scholar 

  30. Liggesmeyer, P., Trapp, M.: Trends in embedded software engineering. IEEE Softw. 26(3), 19–25 (2009)

    Article  Google Scholar 

  31. Matinnejad, R., Nejati, S., Briand, L.C., Bruckmann, T.: Automated test suite generation for time-continuous Simulink models. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 595–606, May 2016

    Google Scholar 

  32. Matinnejad, R., Nejati, S., Briand, L.C., Bruckmann, T.: Test generation and test prioritization for Simulink models with dynamic behavior. IEEE Trans. Softw. Eng. 45(9), 919–944 (2019)

    Article  Google Scholar 

  33. Nejati, S., Gaaloul, K., Menghi, C., Briand, L.C., Foster, S., Wolfe, D.: Evaluating model testing and model checking for finding requirements violations in Simulink models. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ESEC/FSE 2019, pp. 1015–1025. Association for Computing Machinery, New York (2019)

    Google Scholar 

  34. Norouzi, P., Kıvanç, Ö.C., Üstün, Ö.: High performance position control of double sided air core linear brushless DC motor. In: 2017 10th International Conference on Electrical and Electronics Engineering (ELECO), pp. 233–238, November 2017

    Google Scholar 

  35. Oussalem, O., Kourchi, M., Rachdy, A., Ajaamoum, M., Idadoub, H., Jenkal, S.: A low cost controller of PV system based on Arduino board and INC algorithm. Mater. Today: Proc. 24, 104–109 (2019)

    Google Scholar 

  36. Rao, A.C., Raouf, A., Dhadyalla, G., Pasupuleti, V.: Mutation testing based evaluation of formal verification tools. In: 2017 International Conference on Dependable Systems and Their Applications (DSA), pp. 1–7, October 2017

    Google Scholar 

  37. Rashid, M., Anwar, M.W., Khan, A.M.: Toward the tools selection in model based system engineering for embedded systems-a systematic literature review. J. Syst. Softw. 106, 150–163 (2015)

    Article  Google Scholar 

  38. Rebaya, A., Gasmi, K., Hasnaoui, S.: A Simulink-based rapid prototyping workflow for optimizing software/hardware programming. In: 2018 26th International Conference on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1–6. IEEE (2018)

    Google Scholar 

  39. Sanchez, B., Zolotas, A., Rodriguez, H.H., Kolovos, D., Paige, R.: On-the-fly translation and execution of OCL-like queries on simulink models. In: 2019 ACM/IEEE 22nd International Conference on Model Driven Engineering Languages and Systems (MODELS), pp. 205–215. IEEE (2019)

    Google Scholar 

  40. Shaw, M.: What makes good research in software engineering? Int. J. Softw. Tools Technol. Trans. 4(1), 1–7 (2002)

    Article  Google Scholar 

  41. Silva, R., Neiva, F.: Systematic literature review in computer science - a practical guide, November 2016

    Google Scholar 

  42. Stapić, Z., López, E.G., Cabot, A.G., de Marcos Ortega, L., Strahonja, V.: Performing systematic literature review in software engineering. In: CECIIS 2012–23rd International Conference (2012)

    Google Scholar 

  43. Stephan, M., Cordy, J.R.: Identifying instances of model design patterns and antipatterns using model clone detection. In: Proceedings of the Seventh International Workshop on Modeling in Software Engineering MiSE 2015, pp. 48–53. IEEE Press (2015)

    Google Scholar 

  44. Strathmann, T., Oehlerking, J.: Verifying properties of an electro-mechanical braking system. In: 2nd Workshop on Applied Verification of Continuous and Hybrid Systems (ARCH 2015), April 2015

    Google Scholar 

  45. Tomita, T., Ishii, D., Murakami, T., Takeuchi, S., Aoki, T.: A scalable Monte-Carlo test-case generation tool for large and complex simulink models. In: 2019 IEEE/ACM 11th International Workshop on Modelling in Software Engineering (MiSE). pp. 39–46, May 2019

    Google Scholar 

  46. Völter, M., Stahl, T., Bettin, J., Haase, A., Helsen, S.: Model-Driven Software Development: Technology, Engineering, Management. Wiley, Hoboken (2013)

    Google Scholar 

  47. Whittle, J., Hutchinson, J., Rouncefield, M., Burden, H., Heldal, R.: Industrial adoption of model-driven engineering: are the tools really the problem? In: Moreira, A., Schätz, B., Gray, J., Vallecillo, A., Clarke, P. (eds.) MODELS 2013. LNCS, vol. 8107, pp. 1–17. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-41533-3_1

    Chapter  Google Scholar 

  48. Wilkinson, M.D., et al.: The fair guiding principles for scientific data management and stewardship. Sci. Data 3, 1–9 (2016)

    Article  Google Scholar 

  49. Wille, D., Babur, Ö., Cleophas, L., Seidl, C., van den Brand, M., Schaefer, I.: Improving custom-tailored variability mining using outlier and cluster detection. Sci. Comput. Program. 163, 62–84 (2018)

    Article  Google Scholar 

  50. Yang, Y., Jiang, Y., Gu, M., Sun, J.: Verifying Simulink Stateflow model: Timed automata approach. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering ASE 2016, pp. 852–857. Association for Computing Machinery, New York (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Humboldt-Universität zu Berlin, Berlin, Germany

    Alexander Boll & Timo Kehrer

Authors
  1. Alexander Boll
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Timo Kehrer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Alexander Boll or Timo Kehrer .

Editor information

Editors and Affiliations

  1. Eindhoven University of Technology, Eindhoven, The Netherlands

    Önder Babur

  2. University of Antwerp, Antwerp, Belgium

    Joachim Denil

  3. Technical University of Munich, Munich, Germany

    Birgit Vogel-Heuser

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Boll, A., Kehrer, T. (2020). On the Replicability of Experimental Tool Evaluations in Model-Based Development. In: Babur, Ö., Denil, J., Vogel-Heuser, B. (eds) Systems Modelling and Management. ICSMM 2020. Communications in Computer and Information Science, vol 1262. Springer, Cham. https://doi.org/10.1007/978-3-030-58167-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58167-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58166-4

  • Online ISBN: 978-3-030-58167-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation