Skip to main content
SpringerLink
Log in

Comparing Correctness-by-Construction with Post-Hoc Verification—A Qualitative User Study

  • Conference paper
  • First Online:
Formal Methods. FM 2019 International Workshops (FM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12233))

Included in the following conference series:

Abstract

Correctness-by-construction (CbC) is a refinement-based methodology to incrementally create formally correct programs. Programs are constructed using refinement rules which guarantee that the resulting implementation is correct with respect to a pre-/postcondition specification. In contrast, with post-hoc verification (PhV) a specification and a program are created, and afterwards verified that the program satisfies the specification. In the literature, both methods are discussed with specific advantages and disadvantages. By letting participants construct and verify programs using CbC and PhV in a controlled experiment, we analyzed the claims in the literature. We evaluated defects in intermediate code snapshots and discovered a trial-and-error construction process to alter code and specification. The participants appreciated the good feedback of CbC and state that CbC is better than PhV in helping to find defects. Nevertheless, some defects in the constructed programs with CbC indicate that the participants need more time to adapt the CbC process.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    see https://github.com/TUBS-ISF/CorC and  [34] for explanation of the editor.

  2. 2.

    https://github.com/Runge93/UserstudyCbCPhV.

  3. 3.

    The calculation is explained in the work by Feigenspan et al.  [18]. They derived with stepwise regression testing that the experience in comparison to classmates with factor 0.441 summed up with the logical programming experience with factor 0.286 is the best indicator for programming experience.

  4. 4.

    https://www.ueq-online.org/.

  5. 5.

    Statistical hypothesis test to compare two independent samples which are normally distributed.

References

  1. Abrial, J.: Modeling in Event-B: System and Software Engineering. Cambridge University Press, Cambridge (2010)

    MATH  Google Scholar 

  2. Abrial, J.R., Abrial, J.R.: The B-Book: Assigning Programs to Meanings. Cambridge University Press, Cambridge (2005)

    MATH  Google Scholar 

  3. Abrial, J.R., Butler, M., Hallerstede, S., Hoang, T.S., Mehta, F., Voisin, L.: Rodin: an open toolset for modelling and reasoning in event-B. Int. J. Softw. Tools Technol. Transf. 12(6), 447–466 (2010)

    Google Scholar 

  4. Ahrendt, W., Beckert, B., Bubel, R., Hähnle, R., Schmitt, P.H., Ulbrich, M.: Deductive Software Verification-The KeY Book: From Theory to Practice, vol. 10001. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-319-49812-6

    Book  Google Scholar 

  5. Amighi, A., Blom, S., Darabi, S., Huisman, M., Mostowski, W., Zaharieva-Stojanovski, M.: Verification of concurrent systems with VerCors. In: Bernardo, M., Damiani, F., Hähnle, R., Johnsen, E.B., Schaefer, I. (eds.) SFM 2014. LNCS, vol. 8483, pp. 172–216. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07317-0_5

    Chapter  Google Scholar 

  6. Back, R.-J.: Invariant based programming: basic approach and teaching experiences. Formal Aspects Comput. 21(3), 227–244 (2009)

    Google Scholar 

  7. Back, R.-J., Eriksson, J., Myreen, M.: Testing and verifying invariant based programs in the SOCOS environment. In: Gurevich, Y., Meyer, B. (eds.) TAP 2007. LNCS, vol. 4454, pp. 61–78. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-73770-4_4

    Chapter  Google Scholar 

  8. Back, R.-J., Wright, J.: Refinement Calculus: A Systematic Introduction. Springer, Heidelberg (2012)

    Google Scholar 

  9. Barnes, J.G.P.: High Integrity Software: The Spark Approach to Safety and Security. Pearson Education, London (2003)

    Google Scholar 

  10. Barnett, M., Fähndrich, M., Leino, K.R.M., Müller, P., Schulte, W., Venter, H.: Specification and verification: the Spec# experience. Commun. ACM 54(6), 81–91 (2011)

    Google Scholar 

  11. Barnett, M., Leino, K.R.M., Schulte, W.: The Spec# programming system: an overview. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, pp. 49–69. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-30569-9_3

    Chapter  Google Scholar 

  12. Beckert, B., Grebing, S., Böhl, F.: A usability evaluation of interactive theorem provers using focus groups. In: Canal, C., Idani, A. (eds.) SEFM 2014. LNCS, vol. 8938, pp. 3–19. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15201-1_1

    Chapter  Google Scholar 

  13. Beckert, B., Grebing, S., Böhl, F.: How to put usability into focus: using focus groups to evaluate the usability of interactive theorem provers. Electron. Proc. Theor. Comput. Sci. 167, 4–13 (2014)

    MathSciNet  Google Scholar 

  14. Cohen, E., et al.: VCC: a practical system for verifying concurrent C. In: Berghofer, S., Nipkow, T., Urban, C., Wenzel, M. (eds.) TPHOLs 2009. LNCS, vol. 5674, pp. 23–42. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03359-9_2

    Chapter  Google Scholar 

  15. Cok, D.R.: OpenJML: JML for Java 7 by extending OpenJDK. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 472–479. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20398-5_35

    Chapter  Google Scholar 

  16. Cuoq, P., Kirchner, F., Kosmatov, N., Prevosto, V., Signoles, J., Yakobowski, B.: Frama-C. In: Eleftherakis, G., Hinchey, M., Holcombe, M. (eds.) SEFM 2012. LNCS, vol. 7504, pp. 233–247. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33826-7_16

    Chapter  Google Scholar 

  17. Dijkstra, E.W.: A Discipline of Programming. Prentice Hall, Upper Saddle River (1976)

    MATH  Google Scholar 

  18. Feigenspan, J., Kästner, C., Liebig, J., Apel, S., Hanenberg, S.: Measuring programming experience. In: 2012 IEEE 20th International Conference on Program Comprehension (ICPC), pp. 73–82. IEEE (2012)

    Google Scholar 

  19. Gries, D.: The Science of Programming. Springer, Heidelberg (1987)

    MATH  Google Scholar 

  20. Hall, A., Chapman, R.: Correctness by construction: developing a commercial secure system. IEEE Softw. 19(1), 18–25 (2002)

    Google Scholar 

  21. Hentschel, M., Hähnle, R., Bubel, R.: Can formal methods improve the efficiency of code reviews? In: Ábrahám, E., Huisman, M. (eds.) IFM 2016. LNCS, vol. 9681, pp. 3–19. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33693-0_1

    Chapter  Google Scholar 

  22. Jacobs, B., Smans, J., Piessens, F.: A quick tour of the VeriFast program verifier. In: Ueda, K. (ed.) APLAS 2010. LNCS, vol. 6461, pp. 304–311. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17164-2_21

    Chapter  Google Scholar 

  23. Johnson, B., Song, Y., Murphy-Hill, E., Bowdidge, R.: Why don’t software developers use static analysis tools to find bugs? In: Proceedings of the 2013 International Conference on Software Engineering, pp. 672–681. IEEE Press (2013)

    Google Scholar 

  24. Khazeev, M., Rivera, V., Mazzara, M., Johard, L.: Initial steps towards assessing the usability of a verification tool. In: Ciancarini, P., Litvinov, S., Messina, A., Sillitti, A., Succi, G. (eds.) SEDA 2016. AISC, vol. 717, pp. 31–40. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-70578-1_4

    Chapter  Google Scholar 

  25. Kourie, D.G., Watson, B.W.: The Correctness-by-Construction Approach to Programming. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-27919-5

    Book  MATH  Google Scholar 

  26. Leino, K.R.M.: Specification and verification of object-oriented software. Eng. Methods Tools Softw. Saf. Secur. 22, 231–266 (2009)

    Google Scholar 

  27. Leino, K.R.M.: Dafny: an automatic program verifier for functional correctness. In: Clarke, E.M., Voronkov, A. (eds.) LPAR 2010. LNCS (LNAI), vol. 6355, pp. 348–370. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17511-4_20

    Chapter  MATH  Google Scholar 

  28. Meyer, B.: Eiffel*: a language and environment for software engineering. J. Syst. Softw. 8(3), 199–246 (1988)

    Google Scholar 

  29. Meyer, B.: Applying “design by contract”. Computer 25(10), 40–51 (1992)

    Google Scholar 

  30. Morgan, C.: Programming from Specifications, 2nd edn. Prentice Hall, Upper Saddle River (1994)

    MATH  Google Scholar 

  31. Oliveira, M.V.M., Cavalcanti, A., Woodcock, J.: ArcAngel: a tactic language for refinement. Formal Aspects Comput. 15(1), 28–47 (2003)

    MATH  Google Scholar 

  32. Pearce, D.J., Groves, L.: Whiley: a platform for research in software verification. In: Erwig, M., Paige, R.F., Van Wyk, E. (eds.) SLE 2013. LNCS, vol. 8225, pp. 238–248. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-02654-1_13

    Chapter  Google Scholar 

  33. Petiot, G., Kosmatov, N., Botella, B., Giorgetti, A., Julliand, J.: Your proof fails? Testing helps to find the reason. In: Aichernig, B.K.K., Furia, C.A.A. (eds.) TAP 2016. LNCS, vol. 9762, pp. 130–150. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-41135-4_8

    Chapter  Google Scholar 

  34. Runge, T., Schaefer, I., Cleophas, L., Thüm, T., Kourie, D., Watson, B.W.: Tool support for correctness-by-construction. In: Hähnle, R., van der Aalst, W. (eds.) FASE 2019. LNCS, vol. 11424, pp. 25–42. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-16722-6_2

    Chapter  Google Scholar 

  35. Tschannen, J., Furia, C.A., Nordio, M., Polikarpova, N.: AutoProof: auto-active functional verification of object-oriented programs. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 566–580. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-46681-0_53

    Chapter  Google Scholar 

  36. Watson, B.W., Kourie, D.G., Schaefer, I., Cleophas, L.: Correctness-by-construction and post-hoc verification: a marriage of convenience? In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9952, pp. 730–748. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47166-2_52

    Chapter  Google Scholar 

Download references

Acknowledgment

We would like to thank Alexander Knüppel and Domenik Eichhorn for their help with the user study. The hints and suggestions of Alexander helped to construct the final version of the study. Thanks to Domenik for setting up the tools.

Author information

Authors and Affiliations

  1. TU Braunschweig, Braunschweig, Germany

    Tobias Runge & Ina Schaefer

  2. University of Ulm, Ulm, Germany

    Thomas Thüm

  3. TU Eindhoven, Eindhoven, The Netherlands

    Loek Cleophas

  4. Stellenbosch University, Stellenbosch, South Africa

    Loek Cleophas & Bruce W. Watson

  5. Centre for Artificial Intelligence Research, Stellenbosch, South Africa

    Bruce W. Watson

Authors
  1. Tobias Runge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Thüm
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Loek Cleophas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ina Schaefer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bruce W. Watson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Tobias Runge , Thomas Thüm , Loek Cleophas , Ina Schaefer or Bruce W. Watson .

Editor information

Editors and Affiliations

  1. McMaster University, Hamilton, ON, Canada

    Emil Sekerinski

  2. University of Porto, Porto, Portugal

    Nelma Moreira

  3. University of Minho, Braga, Portugal

    José N. Oliveira

  4. Argo Ai, Munich, Germany

    Daniel Ratiu

  5. University of Pisa, Pisa, Italy

    Riccardo Guidotti

  6. University of Liverpool, Liverpool, UK

    Marie Farrell

  7. University of Liverpool, Liverpool, UK

    Matt Luckcuck

  8. University of Exeter, Exeter, UK

    Diego Marmsoler

  9. University of Minho, Braga, Portugal

    José Campos

  10. University of Newcastle, Newcastle upon Tyne, UK

    Troy Astarte

  11. Claude Bernard University, Lyon, France

    Laure Gonnord

  12. Nazarbayev University, Nur-Sultan, Kazakhstan

    Antonio Cerone

  13. University of Surrey, Guildford, UK

    Luis Couto

  14. University of Surrey, Guildford, UK

    Brijesh Dongol

  15. University of Giessen, Giessen, Germany

    Martin Kutrib

  16. University of Lisbon, Lisbon, Portugal

    Pedro Monteiro

  17. Airbus Operations S.A.S., Toulouse, France

    David Delmas

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Runge, T., Thüm, T., Cleophas, L., Schaefer, I., Watson, B.W. (2020). Comparing Correctness-by-Construction with Post-Hoc Verification—A Qualitative User Study. In: Sekerinski, E., et al. Formal Methods. FM 2019 International Workshops. FM 2019. Lecture Notes in Computer Science(), vol 12233. Springer, Cham. https://doi.org/10.1007/978-3-030-54997-8_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-54997-8_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-54996-1

  • Online ISBN: 978-3-030-54997-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation