Skip to main content
SpringerLink
Log in

Formal Verification of Developer Tests: A Research Agenda Inspired by Mutation Testing

  • Conference paper
  • First Online:
Book cover Leveraging Applications of Formal Methods, Verification and Validation: Engineering Principles (ISoLA 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12477))

Included in the following conference series:

Abstract

With the current emphasis on DevOps, automated software tests become a necessary ingredient for continuously evolving, high-quality software systems. This implies that the test code takes a significant portion of the complete code base—test to code ratios ranging from 3:1 to 2:1 are quite common.

We argue that “testware” provides interesting opportunities for formal verification, especially because the system under test may serve as an oracle to focus the analysis. As an example we describe five common problems (mainly from the subfield of mutation testing) and how formal verification may contribute. We deduce a research agenda as an open invitation for fellow researchers to investigate the peculiarities of formally verifying testware.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Finding duplicated code with CPD (2020). https://pmd.github.io/latest/pmd_userdocs_cpd.html. Accessed July 2020

  2. Abdi, M., Rocha, H., Demeyer, S.: Test amplification in the pharo smalltalk ecosystem. In: Proceedings IWST 2019 International Workshop on Smalltalk Technologies. ESUG (2019)

    Google Scholar 

  3. Agibalov, A.: What is a normal “functional lines of code” to “test lines of code” ratio? (2015). https://softwareengineering.stackexchange.com/questions/156883/. Accessed Aug 2020

  4. Aichernig, B.K., Lorber, F., Tiran, S.: Formal test-driven development with verified test cases. In: Proceedings MODELSWARD 2014 2nd International Conference on Model-Driven Engineering and Software Development, pp. 626–635 (2014)

    Google Scholar 

  5. Athanasiou, D., Nugroho, A., Visser, J., Zaidman, A.: Test code quality and its relation to issue handling performance. IEEE Trans. Softw. Eng. 40(11), 1100–1125 (2014). https://doi.org/10.1109/TSE.2014.2342227

    Article  Google Scholar 

  6. Bass, L., Weber, I., Zhu, L.: DevOps: A Software Architect’s Perspective. Addison-Wesley Longman Publishing Co. Inc., Boston (2015)

    Google Scholar 

  7. van Bladel, B., Demeyer, S.: A novel approach for detecting Type-IV clones in test code. In: Proceedings IWSC 2019 IEEE 13th International Workshop on Software Clones, pp. 102–118. IEEE (2019). https://doi.org/10.1109/IWSC.2019.8665855

  8. van Bladel, B., Demeyer, S.: Clone detection in test code: an empirical evaluation. In: Proceedings SANER 2020 International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 492–500. IEEE (2020). https://doi.org/10.1109/SANER48275.2020.9054798

  9. Brillout, A., et al.: Mutation-based test case generation for simulink models. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 208–227. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17071-3_11

    Chapter  Google Scholar 

  10. Cordy, J.R., Roy, C.K.: The NiCad clone detector. In: 2011 IEEE 19th International Conference on Program Comprehension, pp. 219–220. IEEE (2011)

    Google Scholar 

  11. Danglot, B., Vera-Perez, O., Yu, Z., Zaidman, A., Monperrus, M., Baudry, B.: A snowballing literature study on test amplification. J. Syst. Softw. 157, 110398 (2019)

    Article  Google Scholar 

  12. Danglot, B., Vera-Pérez, O.L., Baudry, B., Monperrus, M.: Automatic test improvement with dspot: a study with ten mature open-source projects. Empirical Softw. Eng. 24, 2603–2635 (2019)

    Article  Google Scholar 

  13. Daniel, B., Jagannath, V., Dig, D., Marinov, D.: Reassert: Suggesting repairs for broken unit tests. In: Proceedings ASE 2009 International Conference on Automated Software Engineering, pp. 433–444. IEEE CS (2009). https://doi.org/10.1109/ASE.2009.17

  14. Devroey, X., Perrouin, G., Papadakis, M., Legay, A., Schobbens, P.Y., Heymans, P.: Model-based mutant equivalence detection using automata language equivalence and simulations. J. Syst. Softw. 141, 1–15 (2018). https://doi.org/10.1016/j.jss.2018.03.010

    Article  Google Scholar 

  15. Fewster, M., Graham, D.: Software Test Automation: Effective Use of Test Execution Tools. ACM Press Series. Addison-Wesley (1999)

    Google Scholar 

  16. Göde, N., Koschke, R.: Incremental clone detection. In: 2009 13th European Conference on Software Maintenance and Reengineering, pp. 219–228. IEEE (2009)

    Google Scholar 

  17. Hähnle, R.: Quo vadis formal verification? In: Ahrendt, W., Beckert, B., Bubel, R., Hähnle, R., Schmitt, P.H., Ulbrich, M. (eds.) Deductive Software Verification - The KeY Book: From Theory to Practice, pp. 1–19. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49812-6_1

  18. Hall, A.: Seven myths of formal methods. IEEE Softw. 7(5), 11–19 (1990). https://doi.org/10.1109/52.57887

    Article  Google Scholar 

  19. Hasanain, W., Labiche, Y., Eldh, S.: An analysis of complex industrial test code using clone analysis. In: Proceedings QRS 2018 IEEE International Conference on Software Quality, Reliability and Security, pp. 482–489. IEEE (2018). https://doi.org/10.1109/QRS.2018.00061

  20. Hiep, H.D.A., Maathuis, O., Bian, J., de Boer, F.S., van Eekelen, M., de Gouw, S.: Verifying openjdk’s linkedlist using key. In: Biere, A., Parker, D. (eds.) Tools and Algorithms for the Construction and Analysis of Systems, pp. 217–234. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45237-7_13

  21. Jacobs, B., Smans, J., Philippaerts, P., Vogels, F., Penninckx, W., Piessens, F.: VeriFast: a powerful, sound, predictable, fast verifier for C and Java. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 41–55. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-20398-5_4

    Chapter  Google Scholar 

  22. Jenkins, J.: Velocity culture. In: Keynote Address at the Velocity 2011 Conference (2011)

    Google Scholar 

  23. Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011). https://doi.org/10.1109/TSE.2010.62

    Article  Google Scholar 

  24. King, K.N., Offutt, A.J.: A fortran language system for mutation-based software testing. Softw. Pract. Exp. 21(7), 685–718 (1991). https://doi.org/10.1002/spe.4380210704

    Article  Google Scholar 

  25. Kintis, M., Malevris, N.: MEDIC: a static analysis framework for equivalent mutant identification. Inf. Softw. Technol. 68, 1–17 (2015). https://doi.org/10.1016/j.infsof.2015.07.009

    Article  Google Scholar 

  26. Koschke, R.: Survey of research on software clones. In: Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik (2007)

    Google Scholar 

  27. Li, N., Offutt, J.: Test oracle strategies for model-based testing. IEEE Trans. Softw. Eng. 43(4), 372–395 (2016). https://doi.org/10.1109/TSE.2016.2597136

    Article  Google Scholar 

  28. Lu, Z.X., Vercammen, S., Demeyer, S.: Semi-automatic test case expansion for mutation testing. In: Proceedings VST 2020 IEEE Workshop on Validation, Analysis and Evolution of Software Tests, pp. 1–7 (2020). https://doi.org/10.1109/VST50071.2020.9051637

  29. Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: Proceedings FSE 2014 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 643–453. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2635868.2635920

  30. Rob, M.: Everything you need to know about tesla software updates (2014). https://www.teslarati.com/everything-need-to-know-tesla-software-updates/. Accessed May 2020

  31. Madeyski, L., Orzeszyna, W., Torkar, R., Jozala, M.: Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014). https://doi.org/10.1109/TSE.2013.44

    Article  Google Scholar 

  32. Marcozzi, M., Bardin, S., Kosmatov, N., Papadakis, M., Prevosto, V., Correnson, L.: Time to clean your test objectives. In: Proceedings ICSE 2018 40th International Conference on Software Engineering, pp. 456–467. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3180155.3180191

  33. Offutt, A.J., Pan, J.: Automatically detecting equivalent mutants and infeasible paths. Softw. Test. Verification Reliab. 7(3), 165–192 (1997). https://doi.org/10.1002/(SICI)1099-1689(199709)7:3<165::AID-STVR143>3.0.CO;2-U

  34. Papadakis, M., Jia, Y., Harman, M., Le Traon, Y.: Trivial compiler equivalence: a large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In: Proceedings of the 37th International Conference on Software Engineering, Piscataway, NJ, USA, vol. 1, pp. 936–946. IEEE Press (2015). https://doi.org/10.1109/ICSE.2015.103

  35. Papadakis, M., Kintis, M., Zhang, J., Jia, Y., Traon, Y.L., Harman, M.: Mutation testing advances: an analysis and survey. Adv. Comput. 112, 275–378 (2019). https://doi.org/10.1016/bs.adcom.2018.03.015

    Article  Google Scholar 

  36. Parsai, A., Demeyer, S.: Do null-type mutation operators help prevent null-type faults? In: Catania, B., Královič, R., Nawrocki, J., Pighizzini, G. (eds.) SOFSEM 2019. LNCS, vol. 11376, pp. 419–434. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10801-4_33

    Chapter  Google Scholar 

  37. Parsai, A., Demeyer, S.: Comparing mutation coverage against branch coverage in an industrial setting. Int. J. Softw. Tools Technol. Transfer (2020). https://doi.org/10.1007/s10009-020-00567-y

  38. Parsai, A., Demeyer, S., De Busser, S.: C++11/14 mutation operators based on common fault patterns. In: Medina-Bulo, I., Merayo, M.G., Hierons, R. (eds.) ICTSS 2018. LNCS, vol. 11146, pp. 102–118. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99927-2_9

    Chapter  Google Scholar 

  39. Roy, C.K., Cordy, J.R.: Benchmarks for software clone detection: a ten-year retrospective. In: 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (JSS), pp. 26–37. IEEE (2018)

    Google Scholar 

  40. Roy, C.K., Cordy, J.R., Koschke, R.: Comparison and evaluation of code clone detection techniques and tools: a qualitative approach. Sci. Comput. Program. 74(7), 470–495 (2009)

    Article  MathSciNet  Google Scholar 

  41. Shi, A., Bell, J., Marinov, D.: Mitigating the effects of flaky tests on mutation testing. In: Proceedings ISSTA 2019 the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 112–122. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3293882.3330568

  42. Svajlenko, J., Islam, J.F., Keivanloo, I., Roy, C.K., Mia, M.M.: Towards a big data curated benchmark of inter-project code clones. In: 2014 IEEE International Conference on Software Maintenance and Evolution, pp. 476–480 (2014)

    Google Scholar 

  43. Tillmann, N., Schulte, W.: Unit tests reloaded: parameterized unit testing with symbolic execution. IEEE Softw. 23(4) (2006). https://doi.org/10.1109/MS.2006.117

  44. Tonella, P.: Evolutionary testing of classes. In: Proceedings ISSTA 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 119–128. Association for Computing Machinery, New York (2004). https://doi.org/10.1145/1007512.1007528

  45. Van Rompaey, B., Du Bois, B., Demeyer, S., Rieger, M.: On the detection of test smells: a metrics-based approach for general fixture and eager test. IEEE Trans. Softw. Eng. 33(12), 800–817 (2007). https://doi.org/10.1109/TSE.2007.70745

    Article  Google Scholar 

  46. Vercammen, S., Demeyer, S., Borg, M., Eldh, S.: Speeding up mutation testing via the cloud: lessons learned for further optimisations. In: Proceedings ESEM 2018 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 26:1–26:9. ACM, New York (2018). https://doi.org/10.1145/3239235.3240506

  47. Xie, T.: Augmenting automatically generated unit-test suites with regression oracle checking. In: Thomas, D. (ed.) ECOOP 2006. LNCS, vol. 4067, pp. 380–403. Springer, Heidelberg (2006). https://doi.org/10.1007/11785477_23

    Chapter  Google Scholar 

  48. Zaidman, A., Rompaey, B.V., van Deursen, A., Demeyer, S.: Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Int. J. Empirical Softw. Eng. 16(3), 325–364 (2011). https://doi.org/10.1007/s10664-010-9143-7

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universiteit Antwerpen, Antwerp, Belgium

    Serge Demeyer, Ali Parsai, Sten Vercammen, Brent van Bladel & Mehrdad Abdi

  2. Flanders Make vzw, Kortrijk, Belgium

    Serge Demeyer

Authors
  1. Serge Demeyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Parsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sten Vercammen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brent van Bladel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mehrdad Abdi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Serge Demeyer , Ali Parsai or Sten Vercammen .

Editor information

Editors and Affiliations

  1. University of Limerick and Lero, Limerick, Ireland

    Tiziana Margaria

  2. TU Dortmund, Dortmund, Germany

    Bernhard Steffen

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Demeyer, S., Parsai, A., Vercammen, S., van Bladel, B., Abdi, M. (2020). Formal Verification of Developer Tests: A Research Agenda Inspired by Mutation Testing. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation: Engineering Principles. ISoLA 2020. Lecture Notes in Computer Science(), vol 12477. Springer, Cham. https://doi.org/10.1007/978-3-030-61470-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61470-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61469-0

  • Online ISBN: 978-3-030-61470-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation