skip to main content
10.1145/3193992.3194000acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Testing meets static and runtime verification

Published: 02 June 2018 Publication History

Abstract

Test driven development (TDD) is a technique where test cases are used to guide the development of a system. This technique introduces several advantages at the time of developing a system, e.g. writing clean code, good coverage for the features of the system, and evolutionary development. In this paper we show how the capabilities of a testing focused development methodology based on TDD and model-based testing, can be enhanced by integrating static and runtime verification into its workflow. Considering that the desired system properties capture data- as well as control-oriented aspects, we integrate TDD with (static) deductive verification as an aid in the development of the data-oriented aspects, and we integrate model-based testing with runtime verification as an aid in the development of the control-oriented aspects. As a result of this integration, the proposed development methodology features the benefits of TDD and model-based testing, enhanced with, for instance, early detection of bugs which may be missed by TDD, regarding data aspects, and the validation of the overall system with respect to the model, regarding the control aspects.

References

[1]
2012. Quviq AB: QuickCheck Documentation v1.26.2. (June 2012).
[2]
2018. Bank system repository. github.com/mchimento/Bank. (January 2018).
[3]
2018. SeleniumHQ. http://www.seleniumhq.org/. (2018). Accessed: 2018-01-25.
[4]
Wolfgang Ahrendt, Bernhard Beckert, Richard Bubel, Reiner Hähnle, Peter H. Schmitt, and Mattias Ulbrich (Eds.). 2016. Deductive Software Verification---The KeY Book. Springer.
[5]
Wolfgang Ahrendt, Jesús Mauricio Chimento, Gordon J. Pace, and Gerardo Schneider. 2015. A Specification Language for Static and Runtime Verification of Data and Control Properties. In FM'15. LNCS, Vol. 9109. Springer, 108--125.
[6]
Wolfgang Ahrendt, Jesús Mauricio Chimento, Gordon J. Pace, and Gerardo Schneider. 2017. Verifying data- and control-oriented properties combining static and runtime verification: theory and tools. Form Methods Syst Des 51, 1 (2017).
[7]
Wolfgang Ahrendt, Christoph Gladisch, and Mihai Herda. 2016. Proof-based Test Case Generation, 415--452. In Ahrendt et al. {4}.
[8]
Micael Andersson. 2014. Test Driven Development and Automated Testin. Course given at Chalmers reporting on his experience teaching TDD to Volvo software developers. (2014).
[9]
Paolo Arcaini, Angelo Gargantini, and Elvinia Riccobene. 2013. Combining Model-Based Testing and Runtime Monitoring for Program Testing in the Presence of Nondeterminism. In ICST'13 2013. 178--187.
[10]
D. Astels. 2003. Test Driven Development: A Practical Guide. Prentice Hall PTR.
[11]
Stefan Bechtold, Sam Brannen, Johannes Link, Matthias Merdes, Marc Philipp, and Christian Stein. 2018. JUnit 5 User Guide (version 5.0.3).
[12]
Yves Bertot, Pierre Castéran, Gèrard Huet, and Christine Paulin-Mohring. 2004. Coq'Art : the calculus of inductive constructions. Springer.
[13]
Omar Chebaro, Nikolai Kosmatov, Alain Giorgetti, and Jacques Julliand. 2011. The SANTE Tool: Value Analysis, Program Slicing and Test Generation for C Program Debugging. In TAP'11. 78--83.
[14]
David Chelimsky. 2010. The RSpec Book. Behaviour-Driven Development with RSpec, Cucumber, and Friends. The Pragmatic Bookshelf.
[15]
David R. Cok. 2011. OpenJML: JML for Java 7 by Extending OpenJDK. Springer.
[16]
Christian Colombo, Mark Micallef, and Mark Scerri. 2014. Verifying Web Applications: From Business Level Specifications to Automated Model-Based Testing. In MBT'14. 14--28.
[17]
Christian Colombo, Gordon J. Pace, and Gerardo Schneider. 2009. Dynamic Event-Based Runtime Monitoring of Real-Time and Contextual Properties. In FMICS'08 (LNCS), Vol. 5596. Springer, 135--149.
[18]
Christian Colombo, Gordon J. Pace, and Gerardo Schneider. 2009. LARVA - A Tool for Runtime Monitoring of Java Programs. In SEFM'09.
[19]
Christoph Csallner and Yannis Smaragdakis. 2005. Check 'n' crash: combining static checking and testing. In (ICSE'05). 422--431.
[20]
Stijn de Gouw, Jurriaan Rot, Frank S. de Boer, Richard Bubel, and Reiner Hähnle. 2015. OpenJDK's Java.utils.Collection.sort() Is Broken: The Good, the Bad and the Worst Case. In CAV'15. 273--289.
[21]
Leonardo M. de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In TACAS (LNCS), Vol. 4963. Springer, 337--340.
[22]
Kevin Falzon and Gordon Pace. 2012. Combining Testing and Runtime Verification Techniques. In Model-based Methodologies for Pervasive and Embedded Software, Vol. LNCS 7706.
[23]
Jean-Christophe Filliâtre. 2011. Deductive software verification. International Journal on Software Tools for Technology Transfer 13, 5 (2011), 397--403.
[24]
A. Francalanza, L. Aceto, A. Achilleos, D. P. Attard, I. Cassar, D. Della Monica, and A. Ingólfsdóttir. 2017. A Foundation for Runtime Monitoring. In RV'17. 8--29.
[25]
Klaus Havelund and Grigore Roşu. 2001. Runtime Verification. In Computer Aided Verification (CAV'01) satellite workshop (ENTCS), Vol. 55.
[26]
Martin Hentschel, Reiner Hähnle, and Richard Bubel. 2016. Symbolic Execution, 385--389. In Ahrendt et al. {4}.
[27]
C. A. R. Hoare. 1969. An Axiomatic Basis for Computer Programming. Commun. ACM 12, 10 (Oct. 1969), 576--580.
[28]
Marieke Huisman, Wolfgang Ahrendt, Daniel Grahl, and Martin Hentschel. 2016. Formal Specification with the Java Modeling Language, 193--241. In Ahrendt et al. {4}.
[29]
J. C. King. 1976. Symbolic Execution and Program Testing. Commun. ACM 19, 7 (July 1976), 385--394.
[30]
Jason Koenig and K. Rustan M. Leino. 2012. Getting Started with Dafny: A Guide. In Software Safety and Security. NATO Science for Peace and Security Series - D: Information and Communication Security, Vol. 33. IOS Press, 152--181.
[31]
G. T. Leavens, E. Poll, C. Clifton, Y. Cheon, C. Ruby, D. Cok, P. Müller, J. Kiniry, and P. Chalin. 2007. JML Reference Manual.
[32]
Martin Leucker and Christian Schallhart. 2009. A Brief Account of Runtime Verification. J. Log. Algebr. Program. 78, 5 (2009), 293--303.
[33]
G. Petiot, B. Botella, J. Julliand, N. Kosmatov, and J. Signoles. 2014. Instrumentation of Annotated C Programs for Test Generation. In SCAM'14. 105--114.
[34]
Amir Pnueli. 1977. The temporal logic of programs. In Proc. 18th IEEE Symposium on Foundation of Computer Science. 46--57.
[35]
Giles Reger, Helena Cuenca Cruz, and David E. Rydeheard. 2015. MarQ: Monitoring at Runtime with QEA. In TACAS (LNCS), Vol. 9035. Springer, 596--610.
[36]
J. C. Reynolds. 2009. Theories of Programming Languages. Cambridge University Press.
[37]
Nikolai Tillmann and Jonathan de Halleux. 2008. Pex-White Box Test Generation for .NET. In TAP (LNCS), Vol. 4966. Springer, 134--153.
[38]
J. Tschannen, C. A. Furia, M. Nordio, and B. Meyer. 2011. Usable Verification of Object-Oriented Programs by Combining Static and Dynamic Techniques. In SEFM (LNCS). 382--398.
[39]
Mark Utting and Bruno Legeard. 2007. Practical Model-Based Testing - A Tools Approach. Morgan Kaufmann. I--XIX, 1--433 pages.
[40]
Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A Taxonomy of Model-based Testing Approaches. Softw. Test. Verif. Reliab. 22, 5 (2012), 297--312.
[41]
Makarius Wenzel. 2016. The Isabelle/Isar Reference Manual.
[42]
N. Williams, B. Marre, P. Mouy, and M. Roger. 2005. PathCrawler: Automatic Generation of Path Tests by Combining Static and Dynamic Analysis. In EDCC'05. Springer, 281--292.

Cited By

View all
  • (2023)Residual Runtime Verification via Reachability AnalysisVerified Software. Theories, Tools and Experiments.10.1007/978-3-031-25803-9_9(148-166)Online publication date: 1-Feb-2023
  • (2022)Capturing program models with BISMProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507239(1857-1861)Online publication date: 25-Apr-2022
  • (2022)Runtime Verification Meets Controller SynthesisLeveraging Applications of Formal Methods, Verification and Validation. Verification Principles10.1007/978-3-031-19849-6_22(382-396)Online publication date: 17-Oct-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
FormaliSE '18: Proceedings of the 6th Conference on Formal Methods in Software Engineering
June 2018
101 pages
ISBN:9781450357180
DOI:10.1145/3193992
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 June 2018

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Java
  2. runtime verification
  3. static verification
  4. test driven development

Qualifiers

  • Research-article

Conference

ICSE '18
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)3
Reflects downloads up to 12 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Residual Runtime Verification via Reachability AnalysisVerified Software. Theories, Tools and Experiments.10.1007/978-3-031-25803-9_9(148-166)Online publication date: 1-Feb-2023
  • (2022)Capturing program models with BISMProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing10.1145/3477314.3507239(1857-1861)Online publication date: 25-Apr-2022
  • (2022)Runtime Verification Meets Controller SynthesisLeveraging Applications of Formal Methods, Verification and Validation. Verification Principles10.1007/978-3-031-19849-6_22(382-396)Online publication date: 17-Oct-2022
  • (2021)Realization of a Model-Based DevOps Process for Industrial Safety Critical Cyber Physical Systems2021 4th IEEE International Conference on Industrial Cyber-Physical Systems (ICPS)10.1109/ICPS49255.2021.9468213(597-604)Online publication date: 10-May-2021
  • (2021)The effect of Test-Driven Development and Behavior-Driven Development on Project Success Factors: A Systematic Literature Review Based Study2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE)10.1109/ICCCEEE49695.2021.9429593(1-9)Online publication date: 26-Feb-2021
  • (2021)Runtime Verification: Passing on the BatonFormal Methods in Outer Space10.1007/978-3-030-87348-6_5(89-107)Online publication date: 17-Oct-2021
  • (2020)Salsa: static analysis of serialization featuresProceedings of the 22nd ACM SIGPLAN International Workshop on Formal Techniques for Java-Like Programs10.1145/3427761.3428343(18-25)Online publication date: 23-Jul-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media