Skip to main content
SpringerLink
Log in

Automated testing of DSL implementations—experiences from building mbeddr

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Domain-specific languages promise to improve productivity and quality of software development by providing problem-adequate abstractions to developers. Projectional language workbenches, in turn, allow the definition of modular and extensible domain specific languages, generators, and development environments. While recent advances in language engineering have enabled the definition of DSLs and tooling in a modular and cost-effective way, the quality assurance of their implementation is still challenging. In this paper, we discuss our work on testing different aspects of the implementation of domain specific languages and associated tools, and present several approaches to increase the automation of language testing. We illustrate these approaches with the Jetbrains MPS language workbench and our experience with testing mbeddr, a set of domain specific languages and tools on top of C tailored to embedded software development. Based on the experience gained from the mbeddr project, we extract generic lessons for practitioners as well as challenges which need more research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

Notes

  1. https://www.jetbrains.com/mps/.

  2. http://mbeddr.com

  3. https://www.plm.automation.siemens.com/en_us/products/lms/imagine-lab/embedded-software-designer.shtml https://www.plm.automation.siemens.com/en_us/products/lms/imagine-lab/embedded-software-designer.shtml

  4. In principle, users could open the files in which MPS stores its models and modify the XML directly, potentially leading to invalid models. However, such low-level modifications are possible in many tools (UML tools, Simulink, MS Word) and we classify those more as sabotage and not error prevention. This is why we do not discuss this possibility any further and assume that users will modify their models exclusively with the IDE.

  5. https://www.gnu.org/software/gdb/.

  6. We use EMMA for measuring coverage http://emma.sourceforge.net/.

  7. Note: due to the way in which MPS generates Java code from the DSLs used to implement different language definition aspects (e.g. default constructors, catch-blocks) and how EMMA measures the coverage, it is often technically impossible to achieve 100% line coverage.

References

  • Amrani, M., Combemale, B., Lucio, L., Selim, G.M.K., Dingel, J., Traon, Y.L., Vangheluwe, H., & Cordy, J.R. (2015). Formal verification techniques for model transformations: a tridimensional classification. Journal of Object Technology, 14(3), 1:1–43.

    Article  Google Scholar 

  • Campagne, F (2014). The MPS language workbench. CreateSpace Publishing.

  • Chis, A., Denker, M., Gîrba, T., & Nierstrasz, O. (2015). Practical domain-specific debuggers using the moldable debugger framework. Computer Languages Systems & Structures, 44, 89–113.

    Article  Google Scholar 

  • Clarke, E.M., Kroening, D., & Lerda, F. (2004). A tool for checking ANSI-C programs. In 10th International conference tools and algorithms for the construction and analysis of systems.

    Chapter  Google Scholar 

  • Ellison, C.M. III (2012). A formal semantics of C with applications. University of Illinois at Urbana-Champaign.

  • Erdweg, S., Storm, T., Völter, M., & et al (2013). The state of the art in language workbenches. In Software language engineering, LNCS. Springer.

  • Eysholdt, M. (2014). Executable specifications for xtext. Website. http://www.xpect-tests.org/.

  • Fetscher, B., Claessen, K., Pałka, M., Hughes, J., & Findler, R.B. (2015). Making random judgments: automatically generating well-typed terms from the definition of a type-system, (pp. 383–405). Berlin: Springer.

    Google Scholar 

  • Free Software Foundation (2015). The GNU Project Debugger.

  • Gargantini, A., Riccobene, E., & Scandurra, P. (2009). A semantic framework for metamodel-based languages. Automated Software Engineering, 16(3-4), 415–454.

    Article  Google Scholar 

  • Garousi, V., & Zhi, J. (2013). A survey of software testing practices in Canada. Journal of Systems and Software, 86(5), 1354–1376.

    Article  Google Scholar 

  • JetBrains (2017). JetBrains MPS Documentation. https://www.jetbrains.com/mps/documentation/.

  • Kats, L.C., Vermaas, R., & Visser, E. (2011). Integrated language definition testing: enabling test-driven language development. In ACM SIGPLAN Notices (Vol. 46, pp. 139–154). ACM.

  • Lämmel, R. (2001). Grammar testing. In Proceedings of the 4th international conference on fundamental approaches to software engineering.

  • Molotnikov, Z., Völter, M., & Ratiu, D. (2014). Automated domain-specific c verification with mbeddr. InProceedings of the 29th ACM/IEEE international conference on automated software engineering (pp. 539–550). ACM.

  • Pałka, M.H., Claessen, K., Russo, A., & Hughes, J. (2011). Testing an optimising compiler by generating random lambda terms. In Proceedings of the 6th international workshop on automation of software test, AST ’11 (pp. 91–97). New York: ACM.

  • Pavletic, D., Raza, S.A., Voelter, M., Kolb, B., & Kehrer, T. (2013). Extensible debuggers for extensible languages. Softwaretechnik-Trends, 33, 2.

    Article  Google Scholar 

  • Pavletic, D., Raza, S.A., Dummann, K., & Haßlbauer, K. (2015a). Testing extensible language debuggers. In T. Mayerhofer, P. Langer, E. Seidewitz, & J. Gray (Eds.), Proceedings of the 1st International Workshop on Executable Modeling co-located with ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS 2015). Ottawa, Canada, September 27, 2015., volume 1560 of CEUR Workshop Proceedings (pp. 34–40). CEUR-WS.org.

  • Pavletic, D., Voelter, M., Raza, S. A., Kolb, B., & Kehrer, T. (2015b). Extensible debugger framework for extensible languages. In J. A. de la Puente & T. Vardanega (Eds.), Reliable software technologies - Ada-Europe 2015 - 20th Ada-Europe international conference on reliable software technologies, Madrid Spain, June 22-26 2015, proceedings, volume 9111 of lecture notes in computer science (pp. 33–49). Springer.

  • Ratiu, D., Voelter, M., Schaetz, B., & Kolb, B. (2012). Language engineering as enabler for incrementally defined formal analyses. In Proceedings of the workshop on formal methods in software engineering: rigorous and agile approaches (FORMSERA’2012).

  • Ratiu, D., Voelter, M., Kolb, B., & Schaetz, B. (2013). Using language engineering to lift languages and analyses at the domain level. In Proceedings the 5th NASA formal methods symposium (NFM’13).

    Google Scholar 

  • Ratiu, D., & Voelter, M. (2016). Automated testing of dsl implementations Experiences from building mbeddr. In Proceedings of the 11th international workshop on automation of software test, AST ’16. New York: ACM.

  • Ratiu, D., & Ulrich, A. (2017). Increasing usability of spin-based c code verification using a harness definition language leveraging model-driven code checking to practitioners. In Proceedings of the 24th ACM SIGSOFT international SPIN symposium on model checking of software.

  • Tolvanen, J.-P., & Kelly, S. (2016). Model-driven development challenges and solutions - experiences with domain-specific modelling in industry. In Proceedings of the 4th international conference on model-driven engineering and software development - Volume 1: IndTrackMODELSWARD (pp. 711–719).

  • Voelter, M. (2011). Language and IDE development, modularization and composition with MPS. In Generative and transformational techniques in software engineering, lecture notes in computer science.

  • Voelter, M., & Lisson, S. (2014). Supporting diverse notations in mps’projectional editor. In GEMOC@ MoDELS (pp. 7–16).

  • Voelter, M, Ratiu, D, Schätz, B, & Kolb, B (2012). mbeddr: an extensible C-based programming language and IDE for embedded systems. In SPLASH ’12.

  • Voelter, M., Ratiu, D., Kolb, B., & Schaetz, B (2013a). mbeddr: instantiating a language workbench in the embedded software domain. Automated Software Engineering, 20(3), 339–390.

    Article  Google Scholar 

  • Voelter, M., Benz, S., Dietrich, C., Engelmann, B., Helander, M., Kats, L, Visser, E., & Wachsmuth, G. (2013b). DSL Engineering. dslbook.org.

  • Voelter, M., Deursen, A.v., Kolb, B., & Eberle, S. (2015). Using C language extensions for developing embedded software: a case study (Vol. 50). ACM.

  • Voelter, M., Kolb, B., Szabó, T., Ratiu, D., & van Deursen, A. (2017). Lessons learned from developing mbeddr: a case study in language engineering with mps. Software & Systems Modeling.

  • Wu, H., Gray, J.G., & Mernik, M. (2009). Unit testing for domain-specific languages. In W. M. Taha (Ed.), Domain-specific languages, IFIP TC 2 working conference, DSL 2009, Oxford, UK, July 15-17, 2009, proceedings, volume 5658 of lecture notes in computer science (pp. 125–147). Springer.

  • Wu, H., Monahan, R., & Power, J.F. (2012). Metamodel instance generation: a systematic literature review. Computing Research Repository (CoRR).

  • Yang, X., Chen, Y., Eide, E., & Regehr, J. (2011). Finding and understanding bugs in c compilers. SIGPLAN Notices, 46(6), 283–294.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Siemens AG, Munich, Germany

    Daniel Ratiu

  2. Independent/Itemis AG, Stuttgart, Germany

    Markus Voelter

  3. Itemis AG, Stuttgart, Germany

    Domenik Pavletic

Authors
  1. Daniel Ratiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Markus Voelter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Domenik Pavletic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Ratiu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ratiu, D., Voelter, M. & Pavletic, D. Automated testing of DSL implementations—experiences from building mbeddr. Software Qual J 26, 1483–1518 (2018). https://doi.org/10.1007/s11219-017-9390-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-017-9390-6

Keywords

Search

Navigation