Skip to main content
SpringerLink
Log in

Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

It is a long-standing hypothesis that the concise and customized notation of a DSL improves the performance of developers when compared with a GPL. For non-technical domains—e.g., science—, this hypothesis lacks empirical evidence. Given this lack of empirical evidence, we evaluate a DSL for ecological modeling designed and implemented by us with regard to performance improvements of developers as compared to a GPL. We conduct an online survey with embedded controlled experiments among ecologists to assess the correctness and time spent of the participants when using a DSL for ecosystem simulation specifications compared with a GPL-based solution. We observe that (1) solving tasks with the DSL, the participants’ correctness point score was —depending on the task— on average 61 % up to 63 % higher than with the GPL-based solution and their average time spent per task was reduced by 31 % up to 56 %; (2) the participants subjectively find it easier to work with the DSL, and (3) more than 90 % of the subjects are able to carry out basic maintenance tasks concerning the infrastructure of the DSL used in our treatment, which is based on another internal DSL embedded into Java. The tasks of our experiments are simplified and our web-based editor components do not offer full IDE-support. Our findings indicate that the development of further DSL for the specific needs of the ecological modeling community should be a worthwhile investment to increase its members’ productivity and to enhance the reliability of their scientific results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. 1 http://www.ices.dk

  2. For this purpose, we use CodeMirror (http://codemirror.net), for which we implemented a Sprat Ecosystem DSL syntax coloring plug-in.

  3. 3 http://www.r-project.org

  4. These tests are all formulated to test against normality (the alternative hypothesis is that the sample is not normal). This means that the test, strictly speaking, can never show that a sample is likely to be normal because, as Bortz and Döring (2006) put it, “a non-significant result says nothing.”

References

  • Almorsy M, Grundy J, Sadus R, van Straten W, Barnes DG, Kaluza O (2013) A suite of domain-specific visual languages for scientific software application modelling. In: Symposium on Visual Languages and Human-Centric Computing (VL/HCC), 2013. IEEE, pp 91–94

  • April A, Abran A (2012) Software Maintenance Management: Evaluation and Continuous Improvement. Wiley

  • Basili VR, Caldiera G, Rombach HD (1994) Goal question metric paradigm. In: Encyclopedia of Software Engineering. Wiley, pp 528–532

  • Basili VR, Cruzes D, Carver JC, Hochstein LM, Hollingsworth JK, Zelkowitz MV, Shull F (2008) Understanding the high-performance-computing community: a software engineer’s perspective. IEEE Softw 25(4):29–36

    Article  Google Scholar 

  • Bortz J, Döring N (2006) Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler, 4th edn. Springer

  • Bortz J, Schuster C (2010) Statistik für Human- und Sozialwissenschaftler, 7th edn. Springer

  • Consel C, Marlet R (1998) Architecture software using a methodology for language development. In: Principles of Declarative Programming, LNCS, vol 1490. Springer, pp 170–194

  • Faulk S, Loh E, Vanter MLVD, Squires S, Votta LG (2009) Scientific computing’s productivity gridlock: How software engineering can help. Comput Sci Eng 11:30–39

    Article  Google Scholar 

  • Fowler M (2010) Domain-Specific Languages. Addison-Wesley

  • Galesic M (2006) Dropouts on the web: Effects of interest and burden experienced during an online survey. J Off Stat 22(2):313–328

    Google Scholar 

  • Hevner AR, Linger RC, Webb Collins R, Pleszkoch MG, Walton GH (2005) The impact of function extraction technology on next-generation software engineering. Tech. rep. Carnegie Mellon University

  • Hoisl B, Sobernig S, Strembeck M (2014) Comparing three notations for defining scenario-based model tests: a controlled experiment. In: 9th International Conference on the Quality of Information and Communications Technology (QUATIC), pp 95–104

  • Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Shull f, Singer J, DI Sjøberg (eds) Guide to advanced empirical software engineering. Springer, pp 201–228

  • Johanson AN (2016) Data and scripts for the Sprat Ecosystem DSL survey. doi:http://dx.doi.org/10.5281/zenodo.61373

  • Johanson AN, Hasselbring W (2014a) Hierarchical combination of internal and external domain-specific languages for scientific computing. In: Proceedings of the 2014 European Conference on Software Architecture Workshops, ACM, ECSAW’14, pp 17:1–17:8

  • Johanson AN, Hasselbring W (2014b) Sprat: Hierarchies of domain-specific languages for marine ecosystem simulation engineering. In: Proceedings TMS SpringSim’14, SCS, pp 187–192

  • Johanson AN, Hasselbring W, Oschlies A, Worm B (2016) Evaluating hierarchical domain-specific languages for computational science: Applying the Sprat approach to a marine ecosystem model. In: Carver j, Hong NPC, Thiruvathukal GK (eds) Software Engineering for Science. Chapman and Hall

  • Jones B, Kenward M (2014) Design and Analysis of Cross-Over Trials. Taylor & Francis

  • Kieburtz R, Hook J (1995) Software design for reliability and reuse (sdrr) project phase I final scientific and technical report. Tech. rep. Pacific Software Research Center

  • Kieburtz RB, McKinney L, Bell JM, Hook J, Kotov A, Lewis J, Oliva DP, Sheard T, Smith I, Walton L (1996) A software engineering experiment in software component generation. In: Proceedings of the 18th international conference on software engineering (ICSE’96), pp 542–552

  • Knight J (2002) Safety critical systems: challenges and directions. In: Proceedings ICSE’02. IEEE, pp 547–550

  • Kolovos DS, Paige RF, Kelly T, Polack FA (2006) Requirements for domain-specific languages. In: Proceedings of ECOOP Workshop on Domain-Specific Program Development (DSPD)

  • Korman AK (1971) Industrial and Organizational Psychology. Prentice-Hall

  • Kosar T, Martı PE, Barrientos PA, Mernik M, et al (2008) A preliminary study on various implementation approaches of domain-specific language. Inf Softw Technol 50(5):390–405

    Article  Google Scholar 

  • Kosar T, Mernik M, Carver JC (2012) Program comprehension of domain-specific and general-purpose languages: Comparison using a family of experiments. Empir Softw Eng 17(3):276–304

    Article  Google Scholar 

  • Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Inf Softw Technol 71:77–91

    Article  Google Scholar 

  • Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22(140):5–55

    Google Scholar 

  • Meliá S, Cachero C, Hermida JM, Aparicio E (2015) Comparison of a textual versus a graphical notation for the maintainability of mde domain models: an empirical pilot study. Softw Qual J:1–27

  • Mernik M, Heering J, Sloane AM (2005) When and how to develop domain-specific languages. ACM Comput Surv (CSUR) 37(4):316–344

    Article  Google Scholar 

  • Palyart M, Lugato D, Ober I, Bruel J (2012) MDE4HPC: An approach for using model-driven engineering in high-performance computing. In: Proceedings SDL’11: Integrating System and Software Modeling, LNCS, vol 7083, pp 247–261

  • Prabhu P, Jablin TB, Raman A, Zhang Y, Huang J, Kim H, Johnson NP, Liu F, Ghosh S, Beardl S, Oh T, Zoufaly M, Walker D, August DI (2011) A survey of the practice of computational science. In: State of the Practice Reports, ACM, SC’11, pp 19:1–19:12

  • Ricca F, Scanniello G, Torchiano M, Reggio G, Astesiano E (2010) On the effectiveness of screen mockups in requirements engineering: results from an internal replication

  • Shapiro SS, Wilk MB, Chen HJ (1968) A comparative study of various tests for normality. J Am Stat Assoc 63(324):1343–1372

    Article  MathSciNet  Google Scholar 

  • Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: IEEE/ACM 37Th IEEE international conference on software engineering (ICSE 2015), pp 9–19

  • da Silva AR (2015) Model-driven engineering: a survey supported by the unified conceptual model. Comput Lang Syst Struct 43:139–155

    Google Scholar 

  • van Solingen R (1999) Berghout, E. McGraw-Hill, The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development

  • Stahl T, Völter M (2006) Model-Driven Software development: Technology, Engineering, Management. Wiley

  • Webb Collins R, Hevner AR, Walton GH, Linger RC (2008) The impacts of function extraction technology on program comprehension: a controlled experiment. Inf Softw Technol 50(11):1165–1179

    Article  Google Scholar 

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1(6):80–83

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer

Download references

Author information

Authors and Affiliations

  1. Software Engineering Group, Kiel University, 24098, Kiel, Germany

    Arne N. Johanson & Wilhelm Hasselbring

Authors
  1. Arne N. Johanson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wilhelm Hasselbring
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arne N. Johanson.

Additional information

Communicated by: Romain Robbes

Appendix

Appendix

1.1 Appendix A: Meta-Model of the Sprat Ecosystem DSL

The meta-model is shown in Fig. 9.

Fig. 9
figure 9

UML diagram of the Sprat Ecosystem DSL meta-model

Full size image

1.2 Appendix B: Academic Background Questionnaire

  1. 1.

    Your name [free-form question; optional]

  2. 2.

    Your e-mail address [free-form question; optional]

  3. 3.

    What is your highest educational qualification? (e.g., Bachelor, Master, Ph.D. etc.) [free-form question]

  4. 4.

    In which discipline did you acquire this qualification? [free-form question]

  5. 5.

    What is your current occupation? (e.g., bachelor/master/Ph.D. student, post-doc, professor etc.) [free-form question]

1.3 Appendix C: Programming Experience Questionnaire

  1. 1.

    Have you ever written a computer program using a programming language? (e.g., C, Matlab, Java, R etc.) [yes/no]

  2. 2.

    Have you ever been required to write a computer program for work/for your studies? [yes/no]

  3. 3.

    If applicable, which programming languages (including languages for special purposes like R or Matlab) have you used so far? [free-form question; optional]

  4. 4.

    How would you rate the level of your programming skills? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]

  5. 5.

    How experienced are you with the programming language C/C++? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]

  6. 6.

    How interested are you in programming in general? [6-point Likert scale]

  7. 7.

    If applicable, what was the primary source of information for learning how to program? (e.g., internet, book, a colleague/friend etc.) [free-form question; optional]

  8. 8.

    If applicable: I am confident that the programs I have written are correct (i.e., that they actually compute what I intended them to). [6-point Likert scale]

1.4 Appendix D: Parametrization and Recording Exercise

The exercises are shown in Figs. 10 and 11 (parametrization exercise) as well as Figs. 12 and 13 (recording exercise). Note that in the screenshots of the survey, the Sprat Ecosystem DSL is always referred to as EcoDSL; this was the working title of the DSL when conducting the survey.

  1. 1.

    Concepts of marine ecosystem simulations—such as fish species—can easily be specified with the Sprat Ecosystem DSL. [6-point Likert scale]

  2. 2.

    Concepts of marine ecosystem simulations—such as fish species—can easily be specified with C++. [6-point Likert scale]

  3. 3.

    The Sprat Ecosystem DSL seems simple to use. [6-point Likert scale]

  4. 4.

    C++ seems simple to use. [6-point Likert scale]

  5. 5.

    The Sprat Ecosystem DSL programs were easy to understand. [6-point Likert scale]

  6. 6.

    The C++ programs were easy to understand. [6-point Likert scale]

  7. 7.

    The Sprat Ecosystem DSL seems too technical to specify concepts of marine ecosystem simulations. [6-point Likert scale]

  8. 8.

    C++ seems too technical to specify concepts of marine ecosystem simulations. [6-point Likert scale]

  9. 9.

    It is easy to introduce changes to existing Sprat Ecosystem DSL programs. [6-point Likert scale]

  10. 10.

    It is easy to introduce changes to existing C++ programs. [6-point Likert scale]

  11. 11.

    I was able to understand the meaning of the Sprat Ecosystem DSL source code quickly. [6-point Likert scale]

  12. 12.

    I was able to understand the meaning of C++ source code quickly. [6-point Likert scale]

  13. 13.

    Which difficulties did you have while using the Sprat Ecosystem DSL? [free-form question; optional]

  14. 14.

    Which suggestions do you have for improving the Sprat Ecosystem DSL programming language? [free-form question; optional]

Fig. 10
figure 10

Parametrization exercise with the Sprat Ecosystem DSL

Full size image
Fig. 11
figure 11

Parametrization exercise with C++

Full size image
Fig. 12
figure 12

Recording exercise with the Sprat Ecosystem DSL

Full size image
Fig. 13
figure 13

Recording exercise with C++

Full size image

1.5 Appendix E: Exercise in Modifying the Sprat Ecosystem DSL

The exercise is shown in Fig. 14.

  1. 1.

    How would you rate your expertise regarding the Java programming language? [non-programmer (never written a program, difficulty with reading), beginner (problems writing easy programs, can read), regular (can write basic programs, no problems reading), advanced (can write complex programs), expert (years of experience)]

  2. 2.

    The meaning of the Java code was easy to comprehend. [6-point Likert scale]

  3. 3.

    It was easy to recognize the statements in the Java code that were relevant for completing the task. [6-point Likert scale]

Fig. 14
figure 14

Exercise in modifying the Sprat Ecosystem DSL

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Johanson, A.N., Hasselbring, W. Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empir Software Eng 22, 2206–2236 (2017). https://doi.org/10.1007/s10664-016-9483-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-016-9483-z

Keywords

Search

Navigation