Skip to main content
SpringerLink
Log in

Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Domain-specific languages (DSLs) allow developers to write code at a higher level of abstraction compared with general-purpose languages (GPLs). Developers often use DSLs to reduce the complexity of GPLs. Our previous study found that developers performed program comprehension tasks more accurately and efficiently with DSLs than with corresponding APIs in GPLs. This study replicates our previous study to validate and extend the results when developers use IDEs to perform program comprehension tasks. We performed a dependent replication of a family of experiments. We made two specific changes to the original study: (1) participants used IDEs to perform the program comprehension tasks, to address a threat to validity in the original experiment and (2) each participant performed program comprehension tasks on either DSLs or GPLs, not both as in the original experiment. The results of the replication are consistent with and expanded the results of the original study. Developers are significantly more effective and efficient in tool-based program comprehension when using a DSL than when using a corresponding API in a GPL. The results indicate that, where a DSL is available, developers will perform program comprehension better using the DSL than when using the corresponding API in a GPL.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1 https://lpm.feri.um.si/projects/ReplicationLabPackage.zip

  2. 2 http://lpm.feri.um.si/projects/ReplicationQuestionnaires.zip

  3. 3 https://lpm.feri.um.si/projects/ReplicationLabPackage.zip

References

  • Albuquerque D, Cafeo B, Garcia A, Barbosa S, Abrahao S, Ribeiro A (2015) Quantifying usability of domain-specific languages: an empirical study on software maintenance. J Syst Softw 101:245–259

    Article  Google Scholar 

  • Baldassarre MT, Carver J, Dieste O, Juristo N (2014) Replication types: Towards a shared taxonomy. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, New York, pp 18:1–18:4

  • Barišić A, Amaral V, Goulão M (2017) Usability driven DSL development with USE-ME. Computer Languages, Systems & Structures. In: Press

  • Bentley J (1986) Programming pearls: little languages. Commun ACM 29 (8):711–721

    Article  Google Scholar 

  • Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: 1st international workshop on replication in empirical software engineering

  • Carver JC, Syriani E, Gray J (2011) Assessing the frequency of empirical evaluation in software modeling research. In: Proceedings of the 1st international workshop on experiences and empirical studies in software modeling (EESSMOD), pp 28–37

  • Chiş A, Denker M, Gîrba T, Nierstrasz O (2015) Practical domain-specific debuggers using the moldable debugger framework. Comput Lang Syst Struct 44:89–113

    Google Scholar 

  • Consel C, Marlet R (1998) Architecturing software using a methodology for language development. In: Proceedings of the 10th international symposium on programming language implementation and logic programming, vol 1490, pp 170–194

  • Cornelissen B, Zaidman A, Van Deursen A (2011) A controlled experiment for program comprehension through trace visualization. IEEE Trans Softw Eng 37 (3):341–355

    Article  Google Scholar 

  • van Deursen A, Klint P (1998) Little languages: little maintenance. J Softw Maint 10(2):75–92

    Article  Google Scholar 

  • van Deursen A, Klint P (2002) Domain-specific language design requires feature descriptions. J Comput Inf Technol 10(1):1–17

    Article  MATH  Google Scholar 

  • van Deursen A, Klint P, Visser J (2000) Domain-specific languages: an annotated bibliography. ACM SIGPLAN Not 35(6):26–36

    Article  Google Scholar 

  • Erdweg S, Giarrusso PG, Rendel T (2012) Language composition untangled. In: Proceedings of the twelfth workshop on language descriptions, tools, and applications (LDTA’12). ACM, New York, pp 7:1–7:8

  • Gansner ER, Koutsofios E, North S (2009) Drawing graphs with dot. Tech. rep., AT&T Bell Laboratories, Murray Hill. http://www.graphviz.org/pdf/dotguide.pdf

    Google Scholar 

  • Goulão M, Amaral V, Mernik M (2016) Quality in model-driven engineering: a tertiary study. Softw Qual J 24(3):601–633

    Article  Google Scholar 

  • Häser F, Felderer M, Breu R (2016) Is business domain language support beneficial for creating test case specifications: a controlled experiment. Inf Softw Technol 79:52–62

    Article  Google Scholar 

  • Hermans F, Pinzger M, Van Deursen A (2009) Domain-specific languages in practice: a user study on the success factors. In: Model driven engineering languages and systems, lecture notes in computer science, vol 5795. Springer, Berlin, pp 423–437

  • Hevner AR, Linger RC, Collins Webb R, Pleszkoch M, Prowell S, Walton G (2005) The impact of function extraction technology on next-generation software engineering. Tech. Rep CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon University

    Google Scholar 

  • Hoisl B, Sobernig S, Strembeck M (2014) Comparing three notations for defining scenario-based model tests: a controlled experiment. In: Proceedings of the 9th international conference on the quality of information and communications technology (QUATIC), pp 95–104

  • Hudak P (1998) Modular domain specific languages and tools. In: Proceedings of the 5th international conference on software reuse (JCSR ’98), IEEE Computer Society, pp 134–142

  • Johanson AN, Hasselbring W (2016) Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empir Softw Eng 22(4):2206– 2236

    Article  Google Scholar 

  • Jung HW, Kim SG, Chung CS (2004) Measuring software product quality: a survey of iso/iec 9126. IEEE Softw 21(5):88–92

    Article  Google Scholar 

  • Kitchenham B (2008) The role of replications in empirical software engineering—a word of warning. Empir Softw Eng 13(2):219–221

    Article  Google Scholar 

  • Kosar T, Oliveira N, Mernik M, Varanda Pereira MJ, Črepinšek M, da Cruz D, Henriques PR (2010) Comparing general-purpose and domain-specific languages: an empirical study . Computer Science and Information Systems 7(2):247–264

    Article  Google Scholar 

  • Kosar T, Mernik M, Carver J (2012) Program comprehension of domain-specific and general-purpose languages: comparison using a family of experiments. Empir Softw Eng 17(3):276–304

    Article  Google Scholar 

  • Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Inf Softw Technol 71:77–91

    Article  Google Scholar 

  • MacVittie LA (2006) XAML in a nutshell. O’Reilly Media, Inc, Sebastopol

    Google Scholar 

  • Mauw S, Wiersma W, Willemse T (2004) Language-driven system design. Int J Softw Eng Knowl Eng 6(14):625–664

    Article  Google Scholar 

  • Meliá S, Cachero C, Hermida JM, Aparicio E (2016) Comparison of a textual versus a graphical notation for the maintainability of MDE domain models: an empirical pilot study. Softw Qual J 24(3):709–735

    Article  Google Scholar 

  • Mernik M (2013) An object-oriented approach to language compositions for software language engineering. J Syst Softw 86(9):2451–2464

    Article  Google Scholar 

  • Mernik M, Heering J, Sloane A (2005) When and how to develop domain-specific languages. ACM Comput Surv 37(4):316–344

    Article  Google Scholar 

  • Nugroho A (2009) Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf Softw Technol 51(12):1670–1685

    Article  Google Scholar 

  • Prähofer H, Schatz R, Wirth C, Hurnaus D, Mössenböck H (2013) Monaco – a domain-specific language solution for reactive process control programming with hierarchical components. Comput Lang Syst Struct 39(3):67–94

    Google Scholar 

  • Reis LV, Di Iorio VO, Bigonha RS (2015) An on-the-fly grammar modification mechanism for composing and defining extensible languages. Comput Lang Syst Struct 42:46–59

    MATH  Google Scholar 

  • Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218

    Article  Google Scholar 

  • Sprinkle J, Mernik M, Tolvanen JP, Spinellis D (2009) Guest editors introduction: what kinds of nails need a domain-specific hammer? IEEE Softw 26 (4):15–18

    Article  Google Scholar 

  • Storey MA (2005) Theories, methods and tools in program comprehension: past, present and future. In: Proceedings of the 13th international workshop on program comprehension (IWPC’05), IEEE Computer Society, pp 181–191

  • Umuhoza E, Brambilla M, Ripamonti D, Cabot J (2015) An empirical study on simplification of business process modeling languages. In: Proceedings of the 2015 ACM SIGPLAN international conference on software language engineering. ACM, New York, pp 13–24

  • Varanda Pereira MJ, Fonseca J, Henriques PR (2016) Ontological approach for DSL development. Comput Lang Syst Struct 45:35–52

    Google Scholar 

  • Wile DS (2001) Supporting the DSL spectrum. J Comput Inf Technol 9(4):263–287

    Article  MATH  Google Scholar 

  • Williams M (2002) Microsoft visual c# (core reference). Microsoft Press, Redmond

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering and Computer Science, University of Maribor, Koroška cesta 46, 2000, Maribor, Slovenia

    Tomaž Kosar, Sašo Gaberc & Marjan Mernik

  2. Department of Computer Science, University of Alabama, Tuscaloosa, AL, USA

    Jeffrey C. Carver

Authors
  1. Tomaž Kosar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sašo Gaberc
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jeffrey C. Carver
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marjan Mernik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tomaž Kosar.

Additional information

Communicated by: Sven Apel

The authors acknowledge the financial support from the Slovenian Research Agency (research core funding No. P2-0041).

Appendices

Appendix A: Training Session

To illustrate the type of information the participants saw during training, Fig. 1 provides an example of one of the slides from the DSL GD domain training session. For the sake of clarity in the paper, we simplified the slide from its original use in the training session. The original slide along with the slides for all training sessions for all applications are available in the online lab packageFootnote 3. This slide shows example program code (in the middle of the slide) followed by the program output (below the code). During the training session for GD DSL, we explained all language constructs from GD domain (digraph, graph, node, connection) and the individual properties for each construct (for nodes, fixedsize, shape, width, and height).

Fig. 1
figure 1

Slide with DSL program in GD domain used in tutorial session

Full size image

Appendix B: Code Snippets and Output from each DSL

Snippet from FD questionnaire on DSL (question 9)

figure e

Snippet from GD questionnaire on DSL (question 6)

figure f

Snippet from GUI questionnaire on DSL (question 6)

figure g

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kosar, T., Gaberc, S., Carver, J.C. et al. Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments. Empir Software Eng 23, 2734–2763 (2018). https://doi.org/10.1007/s10664-017-9593-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9593-2

Keywords

Search

Navigation