skip to main content
10.1145/3144555.3144556acmconferencesArticle/Chapter ViewAbstractPublication PagessospConference Proceedingsconference-collections
research-article

The Cogent Case for Property-Based Testing

Published: 28 October 2017 Publication History

Abstract

Property-based testing can play an important role in reducing the cost of formal verification: It has been demonstrated to be effective at detecting bugs and finding inconsistencies in specifications, and thus can eliminate effort wasted on fruitless proof attempts. We argue that in addition, property-based testing enables an incremental approach to a fully verified system, by allowing replacement of automatically generated tests of properties stated in the specification by formal proofs. We demonstrate this approach on the verification of systems code, discuss the implications on systems design, and outline the integration of property-based testing into the Cogent framework.

References

[1]
Sidney Amani. 2016. A Methodology for Trustworthy File Systems. PhD Thesis. CSE, UNSW, Sydney, Australia.
[2]
Sidney Amani, Alex Hixon, Zilin Chen, Christine Rizkallah, Peter Chubb, Liam O'Connor, Joel Beeren, Yutaka Nagashima, Japheth Lim, Thomas Sewell, Joseph Tuong, Gabriele Keller, Toby Murray, Gerwin Klein, and Gernot Heiser. 2016. Cogent: Verifying High-Assurance File System Implementations. In ASPLOS. Atlanta, GA, USA, 175--188.
[3]
Sidney Amani and Toby Murray. 2015. Specifying a Realistic File System. In Workshop on Models for Formal Analysis of Real Systems. Suva, Fiji, 1--9.
[4]
Thomas Arts, John Hughes, Ulf Norell, and Hans Svensson. 2015. Testing AUTOSAR software with QuickCheck. In Eighth IEEE International Conference on Software Testing, Verification and Validation, ICST 2015 Workshops, Graz, Austria, April 13-17, 2015. 1--4. https://doi.org/10.1109/ICSTW.2015.7107466
[5]
Jean-Philippe Bernardy, Mathieu Bosepflug, Ryan R. Newton, Simon Peyton Jones, and Arnaud Spiwack. 2017. Retrofitting Linear Types. (2017). https://www.microsoft.com/en-us/research/wp-content/uploads/2017/03/haskell-linear-submitted.pdf.
[6]
Achim D. Brucker and Burkhart Wolff. 2013. On theorem prover-based testing. Formal Aspects of Computing 25, 5 (2013), 683--721. http://dx.doi.org/10.1007/s00165-012-0222-y
[7]
Lukas Bulwahn. 2012. The New Quickcheck for Isabelle: Random, Exhaustive and Symbolic Testing Under One Roof. In Proceedings of the Second International Conference on Certified Programs and Proofs (CPP'12). Berlin, Heidelberg, 92--108. http://dx.doi.org/10.1007/978-3-642-35308-6_10
[8]
Matthieu Carlier, Catherine Dubois, and Arnaud Gotlieb. 2012. A First Step in the Design of a Formally Verified Constraint-Based Testing Tool: FocalTest. Tests and Proofs: 6th International Conference, TAP 2012, Prague, Czech Republic, May 31-June 1, 2012. Proceedings (2012), 35--50.
[9]
Manuel M. T. Chakravarty. 1999. C -> HASKELL, or Yet Another Interfacing Tool. In Implementation of Functional Languages, 11th International Workshop, IFL'99, Lochem, The Netherlands, September 7-10, 1999, Selected Papers. 131--148. https://doi.org/10.1007/10722298_8
[10]
Zilin Chen. 2017. Cogent⇑: Giving Systems Engineers A Stepping Stone (Extended abstract). In The workshop on Type-Driven Development (TyDe'17). https://www.cse.unsw.edu.au/~zilinc/tyde17.pdf.
[11]
Koen Claessen and John Hughes. 2000. QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. In 5th ICFP. New York, NY, USA, 268--279. http://doi.acm.org/10.1145/351240.351266
[12]
Koen Claessen, Michal Palka, Nicholas Smallbone, John Hughes, Hans Svensson, Thomas Arts, and Ulf Wiger. 2009. Finding Race Conditions in Erlang with QuickCheck and PULSE. In Proceedings of the 14th ACM SIGPLAN International Conference on Functional Programming (ICFP '09). New York, NY, USA, 149--160. http://doi.acm.org/10.1145/1596550.1596574
[13]
Cogent 2017. Cogent Homepage. (2017). http://ts.data61.csiro.au/projects/TS/cogent.pml.
[14]
Arthur Azevedo de Amorim, Nathan Collins, André DeHon, Delphine Demange, Cătălin Hriţcu, David Pichardie, Benjamin C. Pierce, Randy Pollack, and Andrew Tolmach. 2014. A Verified Information-Flow Architecture. In POPL. San Diego, CA, USA, 165--178.
[15]
Willem-Paul de Roever and Kai Engelhardt. 1998. Data Refinement: Model-Oriented Proof Methods and their Comparison. Number 47 in Cambridge Tracts Theoretical Comp. Sci. United Kingdom.
[16]
FUSE Developers. 2017. The FUSE Project. https://sourceforge.net/projects/fuse. (2017).
[17]
Peter Dybjer, Qiao Haiyan, and Makoto Takeyama. 2003. Combining Testing and Proving in Dependent Type Theory. In Proceedings of the 16th International Conference on Theorem Proving in Higher Order Logics (TPHOLS '03). Berlin, Heidelberg, 188--203. http://dx.doi.org/10.1007/10930755_12
[18]
GHC Team. 2017. GHC User's Guide Documentation, Version 8.2.1. User's Guide. 463--466 pages. https://downloads.haskell.org/~ghc/latest/docs/users_guide.pdf.
[19]
Thomas Hallgren, Mark P. Jones, Rebekah Leslie, and Andrew Tolmach. 2005. A principled approach to operating system construction in Haskell. In 10th ICFP. Tallinn, Estonia, 116--128.
[20]
Cătălin Hriţcu, John Hughes, Benjamin C. Pierce, Antal Spector-Zabusky, Dimitrios Vytiniotis, Arthur Azevedo de Amorim, and Leonidas Lampropoulos. 2013. Testing Noninterference, Quickly. In Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming (ICFP '13). New York, NY, USA, 455--468. http://doi.acm.org/10.1145/2500365.2500574
[21]
John Hughes. 2016. Experiences with QuickCheck: Testing the Hard Stuff and Staying Sane. In A List of Successes That Can Change the World - Essays Dedicated to Philip Wadler on the Occasion of His 60th Birthday. 169--186. https://doi.org/10.1007/978-3-319-30936-1_9
[22]
Galois Inc. 2017. The Haskell Lightweight Virtual Machine (HaLVM) source archive. https://github.com/GaloisInc/HaLVM. (2017).
[23]
Gerwin Klein, Kevin Elphinstone, Gernot Heiser, June Andronick, David Cock, Philip Derrin, Dhammika Elkaduwe, Kai Engelhardt, Rafal Kolanski, Michael Norrish, Thomas Sewell, Harvey Tuch, and Simon Winwood. 2009. seL4: Formal Verification of an OS Kernel. In SOSP. Big Sky, MT, USA, 207--220.
[24]
Leonidas Lampropoulos, Diane Gallois-Wong, Cătălin Hriţcu, John Hughes, Benjamin C. Pierce, and Li-yao Xia. 2017. Beginner's Luck: A Language for Property-based Generators. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages (POPL 2017). New York, NY, USA, 114--129. http://doi.acm.org/10.1145/3009837.3009868
[25]
David R. MacIver. 2016. Integrated vs Type-based Shrinking. Article. http://hypothesis.works/articles/integrated-shrinking.
[26]
Neil Mitchell. 2017. The derive package. (2017). https://hackage.haskell.org/package/derive.
[27]
Wojciech Mostowski, Thomas Arts, and John Hughes. 2017. Modelling of Autosar Libraries for Large Scale Testing. In Proceedings 2nd Workshop on Models for Formal Analysis of Real Systems, MARS@ETAPS 2017, Uppsala, Sweden, 29th April 2017. 184--199. https://doi.org/10.4204/EPTCS.244.7
[28]
Tobias Nipkow and Gerwin Klein. 2014. Concrete Semantics with Isabelle/HOL.
[29]
Liam O'Connor. 2016. Applications of Applicative Proof Search. In Proceedings of the 1st International Workshop on Type-Driven Development (TyDe 2016). New York, NY, USA, 43--55. http://doi.acm.org/10.1145/2976022.2976030
[30]
Liam O'Connor, Zilin Chen, Christine Rizkallah, Sidney Amani, Japheth Lim, Toby Murray, Yutaka Nagashima, Thomas Sewell, and Gerwin Klein. 2016. Refinement Through Restraint: Bringing Down the Cost of Verification. In ICFP. Nara, Japan.
[31]
Zoe Paraskevopoulou, Cătălin Hriţcu, Maxime Dénès, Leonidas Lampropoulos, and Benjamin C. Pierce. 2015. Foundational Property-Based Testing. In ITP 2015 - 6th conference on Interactive Theorem Proving (Lecture Notes in Computer Science), Vol. 9236. Nanjing, China. https://hal.inria.fr/hal-01162898
[32]
Rust 2014. The Rust Programming Language. http://rustlang.org. (2014). Accessed March 2015.
[33]
Jacob Stanley. 2017. Hedgehog will eat all your bugs. Open Source Project. https://github.com/hedgehogqa/haskell-hedgehog.

Cited By

View all
  • (2022)CerberusProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security10.1145/3548606.3560595(1871-1885)Online publication date: 7-Nov-2022
  • (2022)Quickstrom: property-based acceptance testing with LTL specificationsProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523728(1025-1038)Online publication date: 9-Jun-2022
  • (2021)Verification of Operating Systems for Internet of Things in Smart Cities From the Assembly Perspective Using Isabelle/HOLIEEE Access10.1109/ACCESS.2020.30474119(2854-2863)Online publication date: 2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
PLOS '17: Proceedings of the 9th Workshop on Programming Languages and Operating Systems
October 2017
62 pages
ISBN:9781450351539
DOI:10.1145/3144555
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Cogent
  2. Formal methods
  3. QuickCheck
  4. Refinement
  5. Systems software

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SOSP '17
Sponsor:

Acceptance Rates

Overall Acceptance Rate 17 of 32 submissions, 53%

Upcoming Conference

SOSP '25
ACM SIGOPS 31st Symposium on Operating Systems Principles
October 13 - 16, 2025
Seoul , Republic of Korea

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)2
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)CerberusProceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security10.1145/3548606.3560595(1871-1885)Online publication date: 7-Nov-2022
  • (2022)Quickstrom: property-based acceptance testing with LTL specificationsProceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3519939.3523728(1025-1038)Online publication date: 9-Jun-2022
  • (2021)Verification of Operating Systems for Internet of Things in Smart Cities From the Assembly Perspective Using Isabelle/HOLIEEE Access10.1109/ACCESS.2020.30474119(2854-2863)Online publication date: 2021
  • (2021)Cogent: uniqueness types and certifying compilationJournal of Functional Programming10.1017/S095679682100023X31Online publication date: 27-Oct-2021
  • (2019)Mutation analysis for CoqProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00057(539-551)Online publication date: 10-Nov-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media