Andreas Löscher
ABSTRACT
We introduce targeted property-based testing, an enhanced form of property-based testing that aims to make the input generation component of a property-based testing tool guided by a search strategy rather than being completely random. Thus, this testing technique combines the advantages of both search-based and property-based testing. We demonstrate the technique with the framework we have built, called Target, and show its effectiveness on three case studies. The first of them demonstrates how Target can employ simulated annealing to generate sensor network topologies that form configurations with high energy consumption. The second case study shows how the generation of routing trees for a wireless network equipped with directional antennas can be guided to fulfill different energy metrics. The third case study employs Target to test the noninterference property of information-flow control abstract machine designs, and compares it with a sophisticated hand-written generator for programs of these abstract machines.
Supplemental Material
Available for Download
This artifact consists of the targeted property-based testing framework Target and the experiments and case studies that are described in the paper "Targeted Property-Based Testing" [1]. Target is nowadays fully integrated in the the open-source QuickCheck-inspired property-based testing tool PropEr [2]. [1] https://doi.org/10.1145/3092703.3092711 [2] http://proper.softlab.ntua.gr/
- Wasif Afzal, Richard Torkar, and Robert Feldt. 2009. A systematic review of searchbased testing for non-functional system properties. Information and Software Technology 51, 6 (2009), 957–976. https:// Google Scholar
Digital Library
- Ian F. Akyildiz, Weilian Su, Yogesh Sankarasubramaniam, and Erdai Cayirci. 2002. Wireless sensor networks: a survey. Computer Networks 38, 4 (2002), 393–422. https:// Google ScholarDigital Library
- Andrea Arcuri and Lionel Briand. 2011. Adaptive Random Testing: An Illusion of Effectiveness?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis. ACM, New York, NY, USA, 265–275. https:// Google ScholarDigital Library
- Thomas Arts, John Hughes, Joakim Johansson, and Ulf Wiger. 2006. Testing Telecoms Software with Quviq QuickCheck. In Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang. ACM, New York, NY, USA, 2–10. https:// Google ScholarDigital Library
- Dimitris Bertsimas and Omid Nohadani. 2010. Robust Optimization with Simulated Annealing. J. of Global Optimization 48, 2 (Oct. 2010), 323–334. https:// Google ScholarDigital Library
- Kwok Ping Chan, Tsong Yueh Chen, and Dave Towey. 2002. Restricted Random Testing. In Proceedings of the 7th European Conference on Software Quality. Springer, Berlin, Heidelberg, 321–330. https:// Google ScholarDigital Library
- Tsong Yueh Chen, F.-C. Kuo, Robert G. Merkel, and S. P. Ng. 2004. Mirror adaptive random testing. Information and Software Technology 46, 15 (2004), 1001–1010.Google ScholarCross Ref
- https://Google Scholar
- Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of Test Case Diversity. J. Syst. Softw. 83, 1 (Jan. 2010), 60–66. https:// Google ScholarDigital Library
- Tsong Yueh Chen, Hing Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Proceedings of the 9th Asian Computing Science Conference, Michael J. Maher (Ed.). Springer, Berlin, Heidelberg, 320–329. https://Google Scholar
- Tsong Yueh Chen and Robert Merkel. 2007. Quasi-Random Testing. IEEE Transactions on Reliability 56, 3 (Sept. 2007), 562–568. https://Google ScholarCross Ref
- 903293Google Scholar
- Koen Claessen and John Hughes. 2000. QuickCheck: a Lightweight Tool for Random Testing of Haskell Programs. In Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming. ACM, New York, NY, USA, 268–279. https:// Google ScholarDigital Library
- Gordon Fraser and Andrea Arcuri. 2011. Evolutionary Generation of Whole Test Suites. In Proceedings of the 11th International Conference on Quality Software (QSIC ’11). IEEE Computer Society, Washington, DC, USA, 31–40. https:// Google ScholarDigital Library
- Milos Gligoric, Tihomir Gvero, Vilas Jagannath, Sarfraz Khurshid, Viktor Kuncak, and Darko Marinov. 2010. Test Generation Through Programming in UDITA. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE ’10). ACM, New York, NY, USA, 225–234. https:// Google ScholarDigital Library
- Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based Software Engineering: Trends, Techniques and Applications. ACM Comput. Surv. 45, 1, Article 11 (Dec. 2012), 61 pages. https:// Google ScholarDigital Library
- Cătălin Hriţcu, Leonidas Lampropoulos, Antal Spector-Zabusky, Arthur Azevedo De Amorim, Maxime Dénès, John Hughes, Benjamin C. Pierce, and Dimitrios Vytiniotis. 2016. Testing Noninterference, Quickly. Journal of Functional Programming 26 (2016), e4. https://Google ScholarCross Ref
- Leonidas Lampropoulos, Benjamin C. Pierce, Cătălin Hriţcu, John Hughes, Zoe Paraskevopoulou, and Li-yao Xia. 2017.Google Scholar
- Beginner’s Luck: A Language for Property-based Generators. In Proceedings of the 44th Annual ACM SIGPLANSIGACT Symposium on Principles of Programming Languages (POPL 2017). ACM, New York, NY, USA, 114–129. https:// Google ScholarDigital Library
- Leonidas Lampropoulos and Konstantinos Sagonas. 2012. Automatic WSDLguided Test Case Generation for PropEr Testing of Web Services. In Proceedings 8th International Workshop on Automated Specification and Verification of Web Systems. 3–16. https://Google Scholar
- Andreas Löscher, Konstantinos Sagonas, and Thiemo Voigt. 2015. Property-based Testing of Sensor Networks. In Sensing, Communication, and Networking, 12th Annual IEEE International Conference on. IEEE, 100–108. https://Google Scholar
- Johannes Mayer. 2005. Lattice-based Adaptive Random Testing. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE ’05). ACM, New York, NY, USA, 333–336. https:// Google ScholarDigital Library
- 1101963Google Scholar
- Phil McMinn. 2004. Search-based Software Test Data Generation: A Survey. Softw. Test. Verif. Reliab. 14, 2 (June 2004), 105–156. https:// Google ScholarDigital Library
- Webb Miller and David L. Spooner. 1976. Automatic Generation of Floating-Point Test Data. IEEE Transactions on Software Engineering SE-2, 3 (Sept. 1976), 223–226. https:// Google ScholarDigital Library
- Alexander G. Nikolaev and Sheldon H. Jacobson. 2010. Simulated annealing. In Handbook of Metaheuristics. Springer, 1–39.Google Scholar
- Martin Nilsson. 2009. Directional antennas for wireless sensor networks. In Proc. 9th Scandinavian Workshop on Wireless Adhoc Networks (Adhoc’09).Google Scholar
- Alessandro Orso and Gregg Rothermel. 2014. Software Testing: A Research Travelogue (2000–2014). In Proceedings of the Future of Software Engineering. ACM, New York, NY, USA, 117–132. https:// Google ScholarDigital Library
- Erik Öström, Luca Mottola, Martin Nilsson, and Thiemo Voigt. 2010. Smart Antennas Made Practical: The SPIDA Way. In Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN ’10). ACM, New York, NY, USA, 438–439. https:// Google ScholarDigital Library
- Manolis Papadakis and Konstantinos Sagonas. 2011. A PropEr Integration of Types and Function Specifications with Property-based Testing. In Proceedings of the 10th ACM SIGPLAN Workshop on Erlang. ACM, New York, NY, USA, 39–50. https:// Google ScholarDigital Library
- QuickCheck 2016. QuickCheck. (2016).Google Scholar
- https://en.wikipedia.org/wiki/ QuickCheckGoogle Scholar
Index Terms
- Targeted property-based testing
Recommendations
JQF: coverage-guided property-based testing in Java
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisWe present JQF, a platform for performing coverage-guided fuzz testing in Java. JQF is designed both for practitioners, who wish to find bugs in Java programs, as well as for researchers, who wish to implement new fuzzing algorithms.
Practitioners ...
ArbitCheck: A Highly Automated Property-Based Testing Tool for Java
ICSTW '14: Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation WorkshopsLightweight property-based testing tools are becoming popular these days. With property-based testing, developers can test properties of the system under test against large varieties of randomly generated inputs without writing test cases. Despite the ...
Automatic generation of test models and properties from UML models with OCL constraints
OCL '12: Proceedings of the 12th Workshop on OCL and Textual ModellingModel-Based Testing and Property-Based Testing are two testing methodologies that usually facilitate the automation of the generation of test cases, using either models or properties as basis to derive complete test suites. In doing so, they also ...
Comments