Abstract
Domain-specific languages (DSLs) allow developers to write code at a higher level of abstraction compared with general-purpose languages (GPLs). Developers often use DSLs to reduce the complexity of GPLs. Our previous study found that developers performed program comprehension tasks more accurately and efficiently with DSLs than with corresponding APIs in GPLs. This study replicates our previous study to validate and extend the results when developers use IDEs to perform program comprehension tasks. We performed a dependent replication of a family of experiments. We made two specific changes to the original study: (1) participants used IDEs to perform the program comprehension tasks, to address a threat to validity in the original experiment and (2) each participant performed program comprehension tasks on either DSLs or GPLs, not both as in the original experiment. The results of the replication are consistent with and expanded the results of the original study. Developers are significantly more effective and efficient in tool-based program comprehension when using a DSL than when using a corresponding API in a GPL. The results indicate that, where a DSL is available, developers will perform program comprehension better using the DSL than when using the corresponding API in a GPL.
Similar content being viewed by others
References
Albuquerque D, Cafeo B, Garcia A, Barbosa S, Abrahao S, Ribeiro A (2015) Quantifying usability of domain-specific languages: an empirical study on software maintenance. J Syst Softw 101:245–259
Baldassarre MT, Carver J, Dieste O, Juristo N (2014) Replication types: Towards a shared taxonomy. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering. ACM, New York, pp 18:1–18:4
Barišić A, Amaral V, Goulão M (2017) Usability driven DSL development with USE-ME. Computer Languages, Systems & Structures. In: Press
Bentley J (1986) Programming pearls: little languages. Commun ACM 29 (8):711–721
Carver JC (2010) Towards reporting guidelines for experimental replications: A proposal. In: 1st international workshop on replication in empirical software engineering
Carver JC, Syriani E, Gray J (2011) Assessing the frequency of empirical evaluation in software modeling research. In: Proceedings of the 1st international workshop on experiences and empirical studies in software modeling (EESSMOD), pp 28–37
Chiş A, Denker M, Gîrba T, Nierstrasz O (2015) Practical domain-specific debuggers using the moldable debugger framework. Comput Lang Syst Struct 44:89–113
Consel C, Marlet R (1998) Architecturing software using a methodology for language development. In: Proceedings of the 10th international symposium on programming language implementation and logic programming, vol 1490, pp 170–194
Cornelissen B, Zaidman A, Van Deursen A (2011) A controlled experiment for program comprehension through trace visualization. IEEE Trans Softw Eng 37 (3):341–355
van Deursen A, Klint P (1998) Little languages: little maintenance. J Softw Maint 10(2):75–92
van Deursen A, Klint P (2002) Domain-specific language design requires feature descriptions. J Comput Inf Technol 10(1):1–17
van Deursen A, Klint P, Visser J (2000) Domain-specific languages: an annotated bibliography. ACM SIGPLAN Not 35(6):26–36
Erdweg S, Giarrusso PG, Rendel T (2012) Language composition untangled. In: Proceedings of the twelfth workshop on language descriptions, tools, and applications (LDTA’12). ACM, New York, pp 7:1–7:8
Gansner ER, Koutsofios E, North S (2009) Drawing graphs with dot. Tech. rep., AT&T Bell Laboratories, Murray Hill. http://www.graphviz.org/pdf/dotguide.pdf
Goulão M, Amaral V, Mernik M (2016) Quality in model-driven engineering: a tertiary study. Softw Qual J 24(3):601–633
Häser F, Felderer M, Breu R (2016) Is business domain language support beneficial for creating test case specifications: a controlled experiment. Inf Softw Technol 79:52–62
Hermans F, Pinzger M, Van Deursen A (2009) Domain-specific languages in practice: a user study on the success factors. In: Model driven engineering languages and systems, lecture notes in computer science, vol 5795. Springer, Berlin, pp 423–437
Hevner AR, Linger RC, Collins Webb R, Pleszkoch M, Prowell S, Walton G (2005) The impact of function extraction technology on next-generation software engineering. Tech. Rep CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon University
Hoisl B, Sobernig S, Strembeck M (2014) Comparing three notations for defining scenario-based model tests: a controlled experiment. In: Proceedings of the 9th international conference on the quality of information and communications technology (QUATIC), pp 95–104
Hudak P (1998) Modular domain specific languages and tools. In: Proceedings of the 5th international conference on software reuse (JCSR ’98), IEEE Computer Society, pp 134–142
Johanson AN, Hasselbring W (2016) Effectiveness and efficiency of a domain-specific language for high-performance marine ecosystem simulation: a controlled experiment. Empir Softw Eng 22(4):2206– 2236
Jung HW, Kim SG, Chung CS (2004) Measuring software product quality: a survey of iso/iec 9126. IEEE Softw 21(5):88–92
Kitchenham B (2008) The role of replications in empirical software engineering—a word of warning. Empir Softw Eng 13(2):219–221
Kosar T, Oliveira N, Mernik M, Varanda Pereira MJ, Črepinšek M, da Cruz D, Henriques PR (2010) Comparing general-purpose and domain-specific languages: an empirical study . Computer Science and Information Systems 7(2):247–264
Kosar T, Mernik M, Carver J (2012) Program comprehension of domain-specific and general-purpose languages: comparison using a family of experiments. Empir Softw Eng 17(3):276–304
Kosar T, Bohra S, Mernik M (2016) Domain-specific languages: a systematic mapping study. Inf Softw Technol 71:77–91
MacVittie LA (2006) XAML in a nutshell. O’Reilly Media, Inc, Sebastopol
Mauw S, Wiersma W, Willemse T (2004) Language-driven system design. Int J Softw Eng Knowl Eng 6(14):625–664
Meliá S, Cachero C, Hermida JM, Aparicio E (2016) Comparison of a textual versus a graphical notation for the maintainability of MDE domain models: an empirical pilot study. Softw Qual J 24(3):709–735
Mernik M (2013) An object-oriented approach to language compositions for software language engineering. J Syst Softw 86(9):2451–2464
Mernik M, Heering J, Sloane A (2005) When and how to develop domain-specific languages. ACM Comput Surv 37(4):316–344
Nugroho A (2009) Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf Softw Technol 51(12):1670–1685
Prähofer H, Schatz R, Wirth C, Hurnaus D, Mössenböck H (2013) Monaco – a domain-specific language solution for reactive process control programming with hierarchical components. Comput Lang Syst Struct 39(3):67–94
Reis LV, Di Iorio VO, Bigonha RS (2015) An on-the-fly grammar modification mechanism for composing and defining extensible languages. Comput Lang Syst Struct 42:46–59
Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218
Sprinkle J, Mernik M, Tolvanen JP, Spinellis D (2009) Guest editors introduction: what kinds of nails need a domain-specific hammer? IEEE Softw 26 (4):15–18
Storey MA (2005) Theories, methods and tools in program comprehension: past, present and future. In: Proceedings of the 13th international workshop on program comprehension (IWPC’05), IEEE Computer Society, pp 181–191
Umuhoza E, Brambilla M, Ripamonti D, Cabot J (2015) An empirical study on simplification of business process modeling languages. In: Proceedings of the 2015 ACM SIGPLAN international conference on software language engineering. ACM, New York, pp 13–24
Varanda Pereira MJ, Fonseca J, Henriques PR (2016) Ontological approach for DSL development. Comput Lang Syst Struct 45:35–52
Wile DS (2001) Supporting the DSL spectrum. J Comput Inf Technol 9(4):263–287
Williams M (2002) Microsoft visual c# (core reference). Microsoft Press, Redmond
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Sven Apel
The authors acknowledge the financial support from the Slovenian Research Agency (research core funding No. P2-0041).
Appendices
Appendix A: Training Session
To illustrate the type of information the participants saw during training, Fig. 1 provides an example of one of the slides from the DSL GD domain training session. For the sake of clarity in the paper, we simplified the slide from its original use in the training session. The original slide along with the slides for all training sessions for all applications are available in the online lab packageFootnote 3. This slide shows example program code (in the middle of the slide) followed by the program output (below the code). During the training session for GD DSL, we explained all language constructs from GD domain (digraph, graph, node, connection) and the individual properties for each construct (for nodes, fixedsize, shape, width, and height).
Appendix B: Code Snippets and Output from each DSL
Snippet from FD questionnaire on DSL (question 9)

Snippet from GD questionnaire on DSL (question 6)

Snippet from GUI questionnaire on DSL (question 6)

Rights and permissions
About this article
Cite this article
Kosar, T., Gaberc, S., Carver, J.C. et al. Program comprehension of domain-specific and general-purpose languages: replication of a family of experiments using integrated development environments. Empir Software Eng 23, 2734–2763 (2018). https://doi.org/10.1007/s10664-017-9593-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-017-9593-2