skip to main content
10.1145/3377812.3382156acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
demonstration

mCoq: mutation analysis for Coq verification projects

Published: 01 October 2020 Publication History

Abstract

Software developed and verified using proof assistants, such as Coq, can provide trustworthiness beyond that of software developed using traditional programming languages and testing practices. However, guarantees from formal verification are only as good as the underlying definitions and specification properties. If properties are incomplete, flaws in definitions may not be captured during verification, which can lead to unexpected system behavior and failures. Mutation analysis is a general technique for evaluating specifications for adequacy and completeness, based on making small-scale changes to systems and observing the results. We demonstrate mCoq, the first mutation analysis tool for Coq projects. mCoq changes Coq definitions, with each change producing a modified project version, called a mutant, whose proofs are exhaustively checked. If checking succeeds, i.e., the mutant is live, this may indicate specification incompleteness. Since proof checking can take a long time, we optimized mCoq to perform incremental and parallel processing of mutants. By applying mCoq to popular Coq libraries, we found several instances of incomplete and missing specifications manifested as live mutants. We believe mCoq can be useful to proof engineers and researchers for analyzing software verification projects. The demo video for mCoq can be viewed at: https://youtu.be/QhigpfQ7dNo.

References

[1]
Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press.
[2]
Thomas Ball and Orna Kupferman. 2008. Vacuity in Testing. In Tests and Proofs. 4--17.
[3]
Ahmet Celik, Karl Palmskog, Marinela Parovic, Emilio Jesús Gallego Arias, and Milos Gligoric. 2019. Mutation Analysis for Coq. In International Conference on Automated Software Engineering. 539--551.
[4]
Coq Development Team. 2019. Coq Manual: Conversion rules. https://coq.inria.fr/distrib/V8.10.2/refman/language/cic.html.
[5]
Coq Development Team. 2019. Coq Manual: Utilities. https://coq.inria.fr/distrib/V8.10.2/refman/practical-tools/utilities.html.
[6]
Leonardo de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. 2015. The Lean Theorem Prover (System Description). In International Conference on Automated Deduction. 378--388.
[7]
Pedro Fonseca, Kaiyuan Zhang, Xi Wang, and Arvind Krishnamurthy. 2017. An Empirical Study on the Correctness of Formally Verified Distributed Systems. In European Conference on Computer Systems. 328--343.
[8]
Gradle Team. 2019. Gradle. https://gradle.org.
[9]
Alex Groce, Iftekhar Ahmed, Carlos Jensen, Paul E. McKenney, and Josie Holmes. 2018. How Verified (or Tested) is My Code? Falsification-driven Verification and Testing. Automated Software Engineering 25, 4 (2018), 917--960.
[10]
Catalin Hritcu, John Hughes, Benjamin C. Pierce, Antal Spector-Zabusky, Dimitrios Vytiniotis, Arthur Azevedo de Amorim, and Leonidas Lampropoulos. 2013. Testing Noninterference, Quickly. In International Conference on Functional Programming. 455--468.
[11]
JaCoCo Development Team. 2019. JaCoCo Java Code Coverage Library. https://www.eclemma.org/jacoco.
[12]
Daniel Kästner, Ulrich Wünsche, Jörg Barrho, Marc Schlickling, Bernhard Schommer, Michael Schmidt, Christian Ferdinand, Xavier Leroy, and Sandrine Blazy. 2018. CompCert: Practical Experience on Integrating and Qualifying a Formally Verified Optimizing Compiler. In European Congress on Embedded Real Time Software and Systems. 1--9.
[13]
Duc Le, Mohammad Amin Alipour, Rahul Gopinath, and Alex Groce. 2014. MuCheck: An Extensible Tool for Mutation Testing of Haskell Programs. In International Symposium on Software Testing and Analysis. 429--432.
[14]
Xavier Leroy. 2009. Formal Verification of a Realistic Compiler. Commun. ACM 52, 7 (2009), 107--115.
[15]
John McCarthy. 1960. Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I. Commun. ACM 3, 4 (1960), 184--195.
[16]
Tobias Nipkow, Lawrence C. Paulson, and Markus Wenzel. 2002. Isabelle/HOL --- A Proof Assistant for Higher-Order Logic. Springer.
[17]
OPAM Team. 2019. OCaml Package Manager. https://opam.ocaml.org.
[18]
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Mutation Testing Advances: An Analysis and Survey. Advances in Computers 112 (2019), 275--378.
[19]
Talia Ringer, Karl Palmskog, Ilya Sergey, Milos Gligoric, and Zachary Tatlock. 2019. QED at Large: A Survey of Engineering of Formally Verified Software. (2019), 102--281.
[20]
Talia Ringer, Nathaniel Yazdani, John Leo, and Dan Grossman. 2018. Adapting Proof Automation to Adapt Proofs. In Certified Programs and Proofs. 115--129.

Cited By

View all
  • (2023)Baldur: Whole-Proof Generation and Repair with Large Language ModelsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616243(1229-1241)Online publication date: 30-Nov-2023
  • (2023)Passport: Improving Automated Formal Verification Using IdentifiersACM Transactions on Programming Languages and Systems10.1145/359337445:2(1-30)Online publication date: 26-Jun-2023
  • (2022)Using Mutations to Analyze Formal SpecificationsCompanion Proceedings of the 2022 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity10.1145/3563768.3563960(81-83)Online publication date: 29-Nov-2022

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings
June 2020
357 pages
ISBN:9781450371223
DOI:10.1145/3377812
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

In-Cooperation

  • KIISE: Korean Institute of Information Scientists and Engineers
  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Check for updates

Author Tags

  1. Coq
  2. deductive verification
  3. mutation analysis
  4. proof assistants

Qualifiers

  • Demonstration

Funding Sources

  • US National Science Foundation

Conference

ICSE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)7
  • Downloads (Last 6 weeks)1
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Baldur: Whole-Proof Generation and Repair with Large Language ModelsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616243(1229-1241)Online publication date: 30-Nov-2023
  • (2023)Passport: Improving Automated Formal Verification Using IdentifiersACM Transactions on Programming Languages and Systems10.1145/359337445:2(1-30)Online publication date: 26-Jun-2023
  • (2022)Using Mutations to Analyze Formal SpecificationsCompanion Proceedings of the 2022 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity10.1145/3563768.3563960(81-83)Online publication date: 29-Nov-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media