Abstract
Domain-specific languages promise to improve productivity and quality of software development by providing problem-adequate abstractions to developers. Projectional language workbenches, in turn, allow the definition of modular and extensible domain specific languages, generators, and development environments. While recent advances in language engineering have enabled the definition of DSLs and tooling in a modular and cost-effective way, the quality assurance of their implementation is still challenging. In this paper, we discuss our work on testing different aspects of the implementation of domain specific languages and associated tools, and present several approaches to increase the automation of language testing. We illustrate these approaches with the Jetbrains MPS language workbench and our experience with testing mbeddr, a set of domain specific languages and tools on top of C tailored to embedded software development. Based on the experience gained from the mbeddr project, we extract generic lessons for practitioners as well as challenges which need more research.
Similar content being viewed by others
Notes
In principle, users could open the files in which MPS stores its models and modify the XML directly, potentially leading to invalid models. However, such low-level modifications are possible in many tools (UML tools, Simulink, MS Word) and we classify those more as sabotage and not error prevention. This is why we do not discuss this possibility any further and assume that users will modify their models exclusively with the IDE.
We use EMMA for measuring coverage http://emma.sourceforge.net/.
Note: due to the way in which MPS generates Java code from the DSLs used to implement different language definition aspects (e.g. default constructors, catch-blocks) and how EMMA measures the coverage, it is often technically impossible to achieve 100% line coverage.
References
Amrani, M., Combemale, B., Lucio, L., Selim, G.M.K., Dingel, J., Traon, Y.L., Vangheluwe, H., & Cordy, J.R. (2015). Formal verification techniques for model transformations: a tridimensional classification. Journal of Object Technology, 14(3), 1:1–43.
Campagne, F (2014). The MPS language workbench. CreateSpace Publishing.
Chis, A., Denker, M., Gîrba, T., & Nierstrasz, O. (2015). Practical domain-specific debuggers using the moldable debugger framework. Computer Languages Systems & Structures, 44, 89–113.
Clarke, E.M., Kroening, D., & Lerda, F. (2004). A tool for checking ANSI-C programs. In 10th International conference tools and algorithms for the construction and analysis of systems.
Ellison, C.M. III (2012). A formal semantics of C with applications. University of Illinois at Urbana-Champaign.
Erdweg, S., Storm, T., Völter, M., & et al (2013). The state of the art in language workbenches. In Software language engineering, LNCS. Springer.
Eysholdt, M. (2014). Executable specifications for xtext. Website. http://www.xpect-tests.org/.
Fetscher, B., Claessen, K., Pałka, M., Hughes, J., & Findler, R.B. (2015). Making random judgments: automatically generating well-typed terms from the definition of a type-system, (pp. 383–405). Berlin: Springer.
Free Software Foundation (2015). The GNU Project Debugger.
Gargantini, A., Riccobene, E., & Scandurra, P. (2009). A semantic framework for metamodel-based languages. Automated Software Engineering, 16(3-4), 415–454.
Garousi, V., & Zhi, J. (2013). A survey of software testing practices in Canada. Journal of Systems and Software, 86(5), 1354–1376.
JetBrains (2017). JetBrains MPS Documentation. https://www.jetbrains.com/mps/documentation/.
Kats, L.C., Vermaas, R., & Visser, E. (2011). Integrated language definition testing: enabling test-driven language development. In ACM SIGPLAN Notices (Vol. 46, pp. 139–154). ACM.
Lämmel, R. (2001). Grammar testing. In Proceedings of the 4th international conference on fundamental approaches to software engineering.
Molotnikov, Z., Völter, M., & Ratiu, D. (2014). Automated domain-specific c verification with mbeddr. InProceedings of the 29th ACM/IEEE international conference on automated software engineering (pp. 539–550). ACM.
Pałka, M.H., Claessen, K., Russo, A., & Hughes, J. (2011). Testing an optimising compiler by generating random lambda terms. In Proceedings of the 6th international workshop on automation of software test, AST ’11 (pp. 91–97). New York: ACM.
Pavletic, D., Raza, S.A., Voelter, M., Kolb, B., & Kehrer, T. (2013). Extensible debuggers for extensible languages. Softwaretechnik-Trends, 33, 2.
Pavletic, D., Raza, S.A., Dummann, K., & Haßlbauer, K. (2015a). Testing extensible language debuggers. In T. Mayerhofer, P. Langer, E. Seidewitz, & J. Gray (Eds.), Proceedings of the 1st International Workshop on Executable Modeling co-located with ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS 2015). Ottawa, Canada, September 27, 2015., volume 1560 of CEUR Workshop Proceedings (pp. 34–40). CEUR-WS.org.
Pavletic, D., Voelter, M., Raza, S. A., Kolb, B., & Kehrer, T. (2015b). Extensible debugger framework for extensible languages. In J. A. de la Puente & T. Vardanega (Eds.), Reliable software technologies - Ada-Europe 2015 - 20th Ada-Europe international conference on reliable software technologies, Madrid Spain, June 22-26 2015, proceedings, volume 9111 of lecture notes in computer science (pp. 33–49). Springer.
Ratiu, D., Voelter, M., Schaetz, B., & Kolb, B. (2012). Language engineering as enabler for incrementally defined formal analyses. In Proceedings of the workshop on formal methods in software engineering: rigorous and agile approaches (FORMSERA’2012).
Ratiu, D., Voelter, M., Kolb, B., & Schaetz, B. (2013). Using language engineering to lift languages and analyses at the domain level. In Proceedings the 5th NASA formal methods symposium (NFM’13).
Ratiu, D., & Voelter, M. (2016). Automated testing of dsl implementations Experiences from building mbeddr. In Proceedings of the 11th international workshop on automation of software test, AST ’16. New York: ACM.
Ratiu, D., & Ulrich, A. (2017). Increasing usability of spin-based c code verification using a harness definition language leveraging model-driven code checking to practitioners. In Proceedings of the 24th ACM SIGSOFT international SPIN symposium on model checking of software.
Tolvanen, J.-P., & Kelly, S. (2016). Model-driven development challenges and solutions - experiences with domain-specific modelling in industry. In Proceedings of the 4th international conference on model-driven engineering and software development - Volume 1: IndTrackMODELSWARD (pp. 711–719).
Voelter, M. (2011). Language and IDE development, modularization and composition with MPS. In Generative and transformational techniques in software engineering, lecture notes in computer science.
Voelter, M., & Lisson, S. (2014). Supporting diverse notations in mps’projectional editor. In GEMOC@ MoDELS (pp. 7–16).
Voelter, M, Ratiu, D, Schätz, B, & Kolb, B (2012). mbeddr: an extensible C-based programming language and IDE for embedded systems. In SPLASH ’12.
Voelter, M., Ratiu, D., Kolb, B., & Schaetz, B (2013a). mbeddr: instantiating a language workbench in the embedded software domain. Automated Software Engineering, 20(3), 339–390.
Voelter, M., Benz, S., Dietrich, C., Engelmann, B., Helander, M., Kats, L, Visser, E., & Wachsmuth, G. (2013b). DSL Engineering. dslbook.org.
Voelter, M., Deursen, A.v., Kolb, B., & Eberle, S. (2015). Using C language extensions for developing embedded software: a case study (Vol. 50). ACM.
Voelter, M., Kolb, B., Szabó, T., Ratiu, D., & van Deursen, A. (2017). Lessons learned from developing mbeddr: a case study in language engineering with mps. Software & Systems Modeling.
Wu, H., Gray, J.G., & Mernik, M. (2009). Unit testing for domain-specific languages. In W. M. Taha (Ed.), Domain-specific languages, IFIP TC 2 working conference, DSL 2009, Oxford, UK, July 15-17, 2009, proceedings, volume 5658 of lecture notes in computer science (pp. 125–147). Springer.
Wu, H., Monahan, R., & Power, J.F. (2012). Metamodel instance generation: a systematic literature review. Computing Research Repository (CoRR).
Yang, X., Chen, Y., Eide, E., & Regehr, J. (2011). Finding and understanding bugs in c compilers. SIGPLAN Notices, 46(6), 283–294.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ratiu, D., Voelter, M. & Pavletic, D. Automated testing of DSL implementations—experiences from building mbeddr. Software Qual J 26, 1483–1518 (2018). https://doi.org/10.1007/s11219-017-9390-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11219-017-9390-6