research-article

Targeted property-based testing

Authors:

Andreas Löscher,

Konstantinos SagonasAuthors Info & Claims

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 46 - 56

https://doi.org/10.1145/3092703.3092711

Published: 10 July 2017 Publication History

Abstract

We introduce targeted property-based testing, an enhanced form of property-based testing that aims to make the input generation component of a property-based testing tool guided by a search strategy rather than being completely random. Thus, this testing technique combines the advantages of both search-based and property-based testing. We demonstrate the technique with the framework we have built, called Target, and show its effectiveness on three case studies. The first of them demonstrates how Target can employ simulated annealing to generate sensor network topologies that form configurations with high energy consumption. The second case study shows how the generation of routing trees for a wireless network equipped with directional antennas can be guided to fulfill different energy metrics. The third case study employs Target to test the noninterference property of information-flow control abstract machine designs, and compares it with a sophisticated hand-written generator for programs of these abstract machines.

Supplementary Material

Auxiliary Archive (issta17-mainid42-s.zip)

This artifact consists of the targeted property-based testing framework Target and the experiments and case studies that are described in the paper "Targeted Property-Based Testing" [1]. Target is nowadays fully integrated in the the open-source QuickCheck-inspired property-based testing tool PropEr [2]. [1] https://doi.org/10.1145/3092703.3092711 [2] http://proper.softlab.ntua.gr/

Download
2390.98 MB

References

[1]

Wasif Afzal, Richard Torkar, and Robert Feldt. 2009. A systematic review of searchbased testing for non-functional system properties. Information and Software Technology 51, 6 (2009), 957–976. https://

Digital Library

[2]

Ian F. Akyildiz, Weilian Su, Yogesh Sankarasubramaniam, and Erdai Cayirci. 2002. Wireless sensor networks: a survey. Computer Networks 38, 4 (2002), 393–422. https://

Digital Library

[3]

Andrea Arcuri and Lionel Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis. ACM, New York, NY, USA, 265–275. https://

Digital Library

[4]

Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger. 2006. Testing Telecoms Software with Quviq QuickCheck. In Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang. ACM, New York, NY, USA, 2–10. https://

Digital Library

[5]

Dimitris Bertsimas and Omid Nohadani. 2010. Robust Optimization with Simulated Annealing. J. of Global Optimization 48, 2 (Oct. 2010), 323–334. https://

Digital Library

[6]

Kwok Ping Chan, Tsong Yueh Chen, and Dave Towey. 2002. Restricted Random Testing. In Proceedings of the 7th European Conference on Software Quality. Springer, Berlin, Heidelberg, 321–330. https://

Digital Library

[7]

Tsong Yueh Chen, F.-C. Kuo, Robert G. Merkel, and S. P. Ng. 2004. Mirror adaptive random testing. Information and Software Technology 46, 15 (2004), 1001–1010.

[8]

https://

[9]

Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of Test Case Diversity. J. Syst. Softw. 83, 1 (Jan. 2010), 60–66. https://

Digital Library

[10]

Tsong Yueh Chen, Hing Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Proceedings of the 9th Asian Computing Science Conference, Michael J. Maher (Ed.). Springer, Berlin, Heidelberg, 320–329. https://

[11]

Tsong Yueh Chen and Robert Merkel. 2007. Quasi-Random Testing. IEEE Transactions on Reliability 56, 3 (Sept. 2007), 562–568. https://

[12]

903293

[13]

Koen Claessen and John Hughes. 2000. QuickCheck: a Lightweight Tool for Random Testing of Haskell Programs. In Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming. ACM, New York, NY, USA, 268–279. https://

Digital Library

[14]

Gordon Fraser and Andrea Arcuri. 2011. Evolutionary Generation of Whole Test Suites. In Proceedings of the 11th International Conference on Quality Software (QSIC ’11). IEEE Computer Society, Washington, DC, USA, 31–40. https://

Digital Library

[15]

Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test Generation Through Programming in UDITA. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE ’10). ACM, New York, NY, USA, 225–234. https://

Digital Library

[16]

Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based Software Engineering: Trends, Techniques and Applications. ACM Comput. Surv. 45, 1, Article 11 (Dec. 2012), 61 pages. https://

Digital Library

[17]

Cătălin Hriţcu, Leonidas Lampropoulos, Antal Spector-Zabusky, Arthur Azevedo De Amorim, Maxime Dénès, John Hughes, Benjamin C. Pierce, and Dimitrios Vytiniotis. 2016. Testing Noninterference, Quickly. Journal of Functional Programming 26 (2016), e4. https://

[18]

Leonidas Lampropoulos, Benjamin C. Pierce, Cătălin Hriţcu, John Hughes, Zoe Paraskevopoulou, and Li-yao Xia. 2017.

[19]

Beginner’s Luck: A Language for Property-based Generators. In Proceedings of the 44th Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages (POPL 2017). ACM, New York, NY, USA, 114–129. https://

Digital Library

[20]

Leonidas Lampropoulos and Konstantinos Sagonas. 2012. Automatic WSDLguided Test Case Generation for PropEr Testing of Web Services. In Proceedings 8th International Workshop on Automated Specification and Verification of Web Systems. 3–16. https://

[21]

Andreas Löscher, Konstantinos Sagonas, and Thiemo Voigt. 2015. Property-based Testing of Sensor Networks. In Sensing, Communication, and Networking, 12th Annual IEEE International Conference on. IEEE, 100–108. https://

[22]

Johannes Mayer. 2005. Lattice-based Adaptive Random Testing. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE ’05). ACM, New York, NY, USA, 333–336. https://

Digital Library

[23]

1101963

[24]

Phil McMinn. 2004. Search-based Software Test Data Generation: A Survey. Softw. Test. Verif. Reliab. 14, 2 (June 2004), 105–156. https://

Digital Library

[25]

Webb Miller and David L. Spooner. 1976. Automatic Generation of Floating-Point Test Data. IEEE Transactions on Software Engineering SE-2, 3 (Sept. 1976), 223–226. https://

Digital Library

[26]

Alexander G. Nikolaev and Sheldon H. Jacobson. 2010. Simulated annealing. In Handbook of Metaheuristics. Springer, 1–39.

[27]

Martin Nilsson. 2009. Directional antennas for wireless sensor networks. In Proc. 9th Scandinavian Workshop on Wireless Adhoc Networks (Adhoc’09).

[28]

Alessandro Orso and Gregg Rothermel. 2014. Software Testing: A Research Travelogue (2000–2014). In Proceedings of the Future of Software Engineering. ACM, New York, NY, USA, 117–132. https://

Digital Library

[29]

Erik Öström, Luca Mottola, Martin Nilsson, and Thiemo Voigt. 2010. Smart Antennas Made Practical: The SPIDA Way. In Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN ’10). ACM, New York, NY, USA, 438–439. https://

Digital Library

[30]

Manolis Papadakis and Konstantinos Sagonas. 2011. A PropEr Integration of Types and Function Specifications with Property-based Testing. In Proceedings of the 10th ACM SIGPLAN Workshop on Erlang. ACM, New York, NY, USA, 39–50. https://

Digital Library

[31]

QuickCheck 2016. QuickCheck. (2016).

[32]

https://en.wikipedia.org/wiki/ QuickCheck

Cited By

Jiang ZWen MCao JShi XJin HFilkov VRay BZhou M(2024)Towards Understanding the Effectiveness of Large Language Models on Directed Test Input GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695513(1408-1420)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695513
Xiong YSu TWang JSun JPu GSu ZFilkov VRay BZhou M(2024)General and Practical Property-based Testing for Android AppsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694986(53-64)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694986
Sokolowski DSpielmann DSalvaneschi GRoychoudhury APaiva AAbreu RStorey M(2024)Unleashing the Giants: Enabling Advanced Testing for Infrastructure as CodeProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3643078(300-301)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3643078
Show More Cited By

Index Terms

Targeted property-based testing
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
2. Theory of computation
  1. Design and analysis of algorithms
    1. Mathematical optimization
      1. Discrete optimization
        Optimization with randomized search heuristics
        Simulated annealing

Recommendations

JQF: coverage-guided property-based testing in Java
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis

We present JQF, a platform for performing coverage-guided fuzz testing in Java. JQF is designed both for practitioners, who wish to find bugs in Java programs, as well as for researchers, who wish to implement new fuzzing algorithms.

Practitioners ...
ArbitCheck: A Highly Automated Property-Based Testing Tool for Java
ICSTW '14: Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops

Lightweight property-based testing tools are becoming popular these days. With property-based testing, developers can test properties of the system under test against large varieties of randomly generated inputs without writing test cases. Despite the ...
Automatic generation of test models and properties from UML models with OCL constraints
OCL '12: Proceedings of the 12th Workshop on OCL and Textual Modelling

Model-Based Testing and Property-Based Testing are two testing methodologies that usually facilitate the automation of the generation of test cases, using either models or properties as basis to derive complete test suites. In doing so, they also ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2017

447 pages

ISBN:9781450350761

DOI:10.1145/3092703

General Chair:
Tevfik Bultan
University of California at Santa Barbara, USA
,
Program Chair:
Koushik Sen
University of California at Berkeley, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Conference

ISSTA '17

Sponsor:

SIGSOFT

ISSTA '17: International Symposium on Software Testing and Analysis

July 10 - 14, 2017

CA, Santa Barbara, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
551
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)10

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jiang ZWen MCao JShi XJin HFilkov VRay BZhou M(2024)Towards Understanding the Effectiveness of Large Language Models on Directed Test Input GenerationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695513(1408-1420)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695513
Xiong YSu TWang JSun JPu GSu ZFilkov VRay BZhou M(2024)General and Practical Property-based Testing for Android AppsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3694986(53-64)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3694986
Sokolowski DSpielmann DSalvaneschi GRoychoudhury APaiva AAbreu RStorey M(2024)Unleashing the Giants: Enabling Advanced Testing for Infrastructure as CodeProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings10.1145/3639478.3643078(300-301)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639478.3643078
Sokolowski DSpielmann DSalvaneschi G(2024)Automated Infrastructure as Code Program TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.339307050:6(1585-1599)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3393070
Zhou YMa FChen YRen MJiang Y(2023)CLFuzz: Vulnerability Detection of Cryptographic Algorithm Implementation via Semantic-aware FuzzingACM Transactions on Software Engineering and Methodology10.1145/362816033:2(1-28)Online publication date: 23-Dec-2023
https://dl.acm.org/doi/10.1145/3628160
de Vries EMcDonell TVazou N(2023)falsify: Internal Shrinking Reimagined for HaskellProceedings of the 16th ACM SIGPLAN International Haskell Symposium10.1145/3609026.3609733(97-109)Online publication date: 30-Aug-2023
https://dl.acm.org/doi/10.1145/3609026.3609733
Shi JKeles AGoldstein HPierce BLampropoulos L(2023)Etna: An Evaluation Platform for Property-Based Testing (Experience Report)Proceedings of the ACM on Programming Languages10.1145/36078607:ICFP(878-894)Online publication date: 31-Aug-2023
https://dl.acm.org/doi/10.1145/3607860
Corgozinho AValente MRocha H(2023)How Developers Implement Property-Based Tests2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00049(380-384)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00049
Sokolowski DSalvaneschi G(2023)Towards Reliable Infrastructure as Code2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C)10.1109/ICSA-C57050.2023.00072(318-321)Online publication date: Mar-2023
https://doi.org/10.1109/ICSA-C57050.2023.00072
Goldstein HPierce B(2022)Parsing randomnessProceedings of the ACM on Programming Languages10.1145/35632916:OOPSLA2(89-113)Online publication date: 31-Oct-2022
https://dl.acm.org/doi/10.1145/3563291
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten