research-article

Feedback-controlled random test generation

Authors:

Kazunori Sakamoto,

Fuyuki Ishikawa,

Shinichi HonidenAuthors Info & Claims

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

Pages 316 - 326

https://doi.org/10.1145/2771783.2771805

Published: 13 July 2015 Publication History

Abstract

Feedback-directed random test generation is a widely used technique to generate random method sequences. It leverages feedback to guide generation. However, the validity of feedback guidance has not been challenged yet. In this paper, we investigate the characteristics of feedback-directed random test generation and propose a method that exploits the obtained knowledge that excessive feedback limits the diversity of tests. First, we show that the feedback loop of feedback-directed generation algorithm is a positive feedback loop and amplifies the bias that emerges in the candidate value pool. This over-directs the generation and limits the diversity of generated tests. Thus, limiting the amount of feedback can improve diversity and effectiveness of generated tests. Second, we propose a method named feedback-controlled random test generation, which aggressively controls the feedback in order to promote diversity of generated tests. Experiments on eight different, real-world application libraries indicate that our method increases branch coverage by 78% to 204% over the original feedback-directed algorithm on large-scale utility libraries.

References

[1]

A. Arcuri and L. Briand. Adaptive random testing: An illusion of effectiveness? In Proceedings of the 2011 International Symposium on Software Testing and Analysis, ISSTA’11, pages 265–275, 2011.

Digital Library

[2]

T. Y. Chen, F.-C. Kuo, R. G. Merkel, and T. H. Tse. Adaptive random testing: The art of test case diversity. J. Syst. Softw., 83(1):60–66, Jan. 2010.

Digital Library

[3]

T. Y. Chen, H. Leung, and I. Mak. Adaptive random testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, volume 3321 of Lecture Notes in Computer Science, pages 320–329. 2004.

Digital Library

[4]

I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. Artoo: Adaptive random testing for object-oriented software. In Proceedings of the 30th International Conference on Software Engineering, ICSE’08, pages 71–80, 2008.

Digital Library

[5]

A. P. Felt, E. Chin, S. Hanna, D. Song, and D. Wagner. Android permissions demystified. In Proceedings of the 18th ACM Conference on Computer and Communications Security, CCS’11, pages 627–638, 2011.

Digital Library

[6]

G. Fraser and A. Arcuri. Sound empirical evidence in software testing. In Proceedings of the 34th International Conference on Software Engineering, ICSE’12, pages 178–188, 2012.

Digital Library

[7]

G. Fraser and A. Arcuri. Evosuite: On the challenges of test case generation in the real world. In Proceedings of the 2013 IEEE 6th International Conference on Software Testing, Verification and Validation, ICST’13, pages 362–369, 2013.

Digital Library

[8]

G. Fraser and A. Arcuri. Whole test suite generation. IEEE Trans. Softw. Eng., 39(2):276–291, Feb. 2013.

Digital Library

[9]

P. Garg, F. Ivancic, G. Balakrishnan, N. Maeda, and A. Gupta. Feedback-directed unit test generation for c/c++ using concolic execution. In Proceedings of the 2013 International Conference on Software Engineering, ICSE’13, pages 132–141, 2013.

Digital Library

[10]

P. Godefroid, N. Klarlund, and K. Sen. Dart: Directed automated random testing. SIGPLAN Not., 40(6):213–223, June 2005.

Digital Library

[11]

R. Gopinath, C. Jensen, and A. Groce. Code coverage for suite evaluation by developers. In Proceedings of the 36th International Conference on Software Engineering, ICSE’14, pages 72–82, 2014.

Digital Library

[12]

A. Groce, C. Zhang, E. Eide, Y. Chen, and J. Regehr. Swarm testing. In Proceedings of the 2012 International Symposium on Software Testing and Analysis, ISSTA’12, pages 78–88, 2012.

Digital Library

[13]

L. Inozemtseva and R. Holmes. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering, ICSE’14, pages 435–445, 2014.

Digital Library

[14]

H. Jaygarl, K.-S. Lu, and C. K. Chang. Genred: A tool for generating and reducing object-oriented test cases. In Proceedings of the 2010 IEEE 34th Annual Computer Software and Applications Conference, COMPSAC ’10, pages 127–136, 2010.

Digital Library

[15]

P. McMinn. Search-based software test data generation: A survey: Research articles. Softw. Test. Verif. Reliab., 14(2):105–156, June 2004.

Digital Library

[16]

C. Pacheco and M. D. Ernst. Randoop: Feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN Conference on Object-oriented Programming Systems and Applications Companion, OOPSLA’07, pages 815–816, 2007.

Digital Library

[17]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering, ICSE’07, pages 75–84, 2007.

Digital Library

[18]

M. Pradel and T. R. Gross. Leveraging test generation and specification mining for automated bug detection without false positives. In Proceedings of the 34th International Conference on Software Engineering, ICSE’12, pages 288–298, 2012.

Digital Library

[19]

B. Robinson, M. D. Ernst, J. H. Perkins, V. Augustine, and N. Li. Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE’11, pages 23–32, 2011.

Digital Library

[20]

K. Sen, D. Marinov, and G. Agha. Cute: A concolic unit testing engine for c. In Proceedings of the 10th European Software Engineering Conference Held Jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE’13, pages 263–272, 2005.

Digital Library

[21]

A. Shahbazi, A. F. Tappenden, and J. Miller. Centroidal voronoi tessellations—a new approach to random testing. IEEE Trans. Softw. Eng., 39(2):163–183, Feb. 2013.

Digital Library

[22]

S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte. Mseqgen: Object-oriented unit-test generation via mining source code. In Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’09, pages 193–202, 2009.

Digital Library

[23]

N. Tillmann and J. De Halleux. Pex: White box test generation for .net. In Proceedings of the 2nd International Conference on Tests and Proofs, TAP’08, pages 134–153, 2008.

Digital Library

[24]

N. Tillmann and W. Schulte. Parameterized unit tests. In Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ESEC/FSE’13, pages 253–262, 2005.

Digital Library

[25]

J. Yi, D. Qi, S. H. Tan, and A. Roychoudhury. Expressing and checking intended changes via software change contracts. In Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA’13, pages 1–11, 2013.

Digital Library

[26]

C. Zhang, A. Groce, and M. A. Alipour. Using test case reduction and prioritization to improve symbolic execution. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA’14, pages 160–170, 2014.

Digital Library

[27]

S. Zhang, Y. Bu, X. S. Wang, and M. D. Ernst. Dependence-guided random test generation. CSE 503 Course Project Report, University of Washington, May 2010.

[28]

W. Zheng, Q. Zhang, M. Lyu, and T. Xie. Random unit-test generation with mut-aware sequence recommendation. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ASE’10, pages 293–296, 2010.

Digital Library

Cited By

Elgendy IHierons RMcMinn P(2025)A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing TechniquesSoftware Testing, Verification and Reliability10.1002/stvr.191435:2Online publication date: 17-Jan-2025
https://doi.org/10.1002/stvr.1914
Mandrioli CMaggio M(2022)Testing Self-Adaptive Software With Probabilistic Guarantees on Performance Metrics: Extended and Comparative ResultsIEEE Transactions on Software Engineering10.1109/TSE.2021.310113048:9(3554-3572)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TSE.2021.3101130
Chen BLiu YPeng XWu YQin S(2022)Baton: symphony of random testing and concolic testing through machine learning and taint analysisScience China Information Sciences10.1007/s11432-020-3403-266:3Online publication date: 11-Nov-2022
https://doi.org/10.1007/s11432-020-3403-2
Show More Cited By

Index Terms

Feedback-controlled random test generation
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
Fault detection effectiveness of source test case generation strategies for metamorphic testing
MET '18: Proceedings of the 3rd International Workshop on Metamorphic Testing

Metamorphic testing is a well known approach to tackle the oracle problem in software testing. This technique requires the use of source test cases that serve as seeds for the generation of follow-up test cases. Systematic design of test cases is ...
Improving random test sets using the diversity oriented test data generation
RT '07: Proceedings of the 2nd international workshop on Random testing: co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007)

We present a measure that characterizes the diversity of a test set from the perspective of the input domain of the program under test. By using a metaheuristic algorithm, randomly generated test sets (RTS) are evolved towards Diversity Oriented Test ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2015: Proceedings of the 2015 International Symposium on Software Testing and Analysis

July 2015

447 pages

ISBN:9781450336208

DOI:10.1145/2771783

General Chair:
Michal Young
University of Oregon, USA
,
Program Chair:
Tao Xie
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Artifact

Author Tags

Qualifiers

Research-article

Conference

ISSTA '15

Sponsor:

SIGSOFT

ISSTA '15: International Symposium on Software Testing and Analysis

July 13 - 17, 2015

MD, Baltimore, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Elgendy IHierons RMcMinn P(2025)A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing TechniquesSoftware Testing, Verification and Reliability10.1002/stvr.191435:2Online publication date: 17-Jan-2025
https://doi.org/10.1002/stvr.1914
Mandrioli CMaggio M(2022)Testing Self-Adaptive Software With Probabilistic Guarantees on Performance Metrics: Extended and Comparative ResultsIEEE Transactions on Software Engineering10.1109/TSE.2021.310113048:9(3554-3572)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TSE.2021.3101130
Chen BLiu YPeng XWu YQin S(2022)Baton: symphony of random testing and concolic testing through machine learning and taint analysisScience China Information Sciences10.1007/s11432-020-3403-266:3Online publication date: 11-Nov-2022
https://doi.org/10.1007/s11432-020-3403-2
Bonnaventure WKhanfir ABartel APapadakis MTraon Y(2021)Confuzzion: A Java Virtual Machine Fuzzer for Type Confusion Vulnerabilities2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS54544.2021.00069(586-597)Online publication date: Dec-2021
https://doi.org/10.1109/QRS54544.2021.00069
Molina FPonzio PAguirre NFrias M(2021)EvoSpex: An Evolutionary Algorithm for Learning Postconditions2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)10.1109/ICSE43902.2021.00112(1223-1235)Online publication date: May-2021
https://doi.org/10.1109/ICSE43902.2021.00112
Holmes JAhmed IBrindescu CGopinath RZhang HGroce A(2020)Using Relative Lines of Code to Guide Automated Test Generation for PythonACM Transactions on Software Engineering and Methodology10.1145/340889629:4(1-38)Online publication date: 26-Sep-2020
https://dl.acm.org/doi/10.1145/3408896
Mandrioli CMaggio MDevanbu PCohen MZimmermann T(2020)Testing self-adaptive software with probabilistic guarantees on performance metricsProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409685(1002-1014)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3409685
Liang HPei XJia XShen WZhang J(2018)Fuzzing: State of the ArtIEEE Transactions on Reliability10.1109/TR.2018.283447667:3(1199-1218)Online publication date: Sep-2018
https://doi.org/10.1109/TR.2018.2834476
Pillai AGupta V(2018)User Feedback-Based Test Suite Management: A Research FrameworkProceedings of First International Conference on Smart System, Innovations and Computing10.1007/978-981-10-5828-8_78(825-830)Online publication date: 9-Jan-2018
https://doi.org/10.1007/978-981-10-5828-8_78
Peleg HRasin DYahav E(2017)Generating Tests by ExampleVerification, Model Checking, and Abstract Interpretation10.1007/978-3-319-73721-8_19(406-429)Online publication date: 29-Dec-2017
https://doi.org/10.1007/978-3-319-73721-8_19
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten