Article

Is adaptive random testing really better than random testing

Authors:

F.-C. KuoAuthors Info & Claims

RT '06: Proceedings of the 1st international workshop on Random testing

Pages 64 - 69

https://doi.org/10.1145/1145735.1145745

Published: 17 July 2006 Publication History

Abstract

Adaptive Random Testing (ART) is an enhancement of Random Testing (RT). It is known when ART can outperform RT and when it cannot. Previous studies assumed that the test cases are selected with replacement. It is unknown whether selection without replacement can enhance the effectiveness of ART and also RT. Our studies include whether ART outperforms RT because it evenly spreads test cases or it generates fewer duplicate test cases, whether the input domain type and size have an impact on the effectiveness of ART, whether ART can still outperform RT when there exists only one single failure-causing input (in that case, failure-causing inputs do not cluster together). This study comprehends our understandings about whether ART is really better than RT.

References

[1]

P. E. Ammann and J. C. Knight. Data diversity: an approach to software fault tolerance. IEEE Transactions on Computers, 37(4):418--425, 1988.

Digital Library

[2]

P. G. Bishop. The variation of software survival times for different operational input profiles. In Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23), pages 98--107. IEEE Computer Society Press, 1993.

[3]

K. P. Chan. T. Y. Chen. and D. Towey. Restricted random testing: Adaptive random testing by exclusion. Accepted to appear in International Journal of Software Engineering and Knowledge Engineering, 2006.

[4]

T. Y. Chen, G. Eddy, R. G. Merkel, and P. K. Wong. Adaptive random testing through dynamic partitioning. In Proceedings of the 4th International Conference on Quality Software (QSIC 04), pages 79--86, Braunschweig, Germany, 2004. IEEE Computer Society Press.

Digital Library

[5]

T. Y. Chen and D. H. Huang. Adaptive random testing by localization. In Proceedings of the 11th Asia-Pacific Software Engineering Conference (APSEC'04), pages 292--298. IEEE Computer Society, 2004.

Digital Library

[6]

T. Y. Chen, F. C. Kuo, R. G. Merkel, and S. P. Ng. Mirror adaptive random testing. Information and Software Technology, 46(15):1001--1010, 2004.

[7]

T. Y. Chen, F. C. Kuo, and Z. Q. Zhou. On the relationships between the distribution of failure-causing inputs and effectiveness of adaptive random testing. In Proceedings of the 17th International Conference on Software Engineering and Knowledge Engineering (SEKE 2005)), pages 306--311, Taipei, Taiwan, 2005.

[8]

T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference, volume 3321 of Lecture Notes in Computer Science, pages 320--329, 2004.

Digital Library

[9]

T. Y. Chen and R. Merkel. An upper bound on software testing effectiveness. submitted for publication.

[10]

R. Cobb and H. D. Mills. Engineering software under statistical quality control. IEEE Software, 7(6):45--54, 1990.

Digital Library

[11]

T. Dabóczi, I. Kollár, G. Simon, and T. Megyeri. Automatic testing of graphical user interfaces. In Proceedings of the 20th IEEE Instrumentation and Measurement Technology Conference 2003 (IMTC '03), pages 441--445, Vail, CO, USA, 2003.

[12]

G. B. Finelli. Nasa software failure characterization experiments. Reliability Engineering and System Safety. 32(1-2):155--169, 1991.

[13]

J. E. Forrester and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th USENIX Windows Systems Symposium, pages 59--68, Seattle, 2000.

Digital Library

[14]

R. Hamlet. Random testing. In J. Marciniak, editor, Encyclopedia of Software Engineering. John Wiley & Sons, second edition, 2002.

[15]

I. K. Mak. On the effectiveness of random testing. Master's thesis, Department of Computer Science, University of Melbourne, 1997.

[16]

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32--44, 1990.

Digital Library

[17]

B. P. Miller. D. Koski, C. P. Lee, V. Maganty, R. Murthy. A. Natarajan, and J. Steidl. Fuzz revisited: A re-examination of the reliability of UNIX utilities and services. Technical Report. CS-TR-1995-1268, University of Wisconsin, 1995.

[18]

E. Miller. Website testing. http://www.soft.com/eValid/Technology/White.Papers/website.testing.html. Software Research, Inc., 2005.

[19]

G. J. Myers. The Art of Software Testing. Wiley, New York, second edition, 1979.

Digital Library

[20]

N. Nyman. In defense of monkey testing: Random testing can find bugs, even in well engineered software. http://www.softtest.org/sigs/material/nnyman2.htm, Microsoft Corporation.

[21]

K. Sen. D. Marinov, and G. Agha. Cute: a concolic unit testing engine for c. In ESEC/FSE-13: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, pages 263--272, New York, NY, USA, 2005. ACM Press.

Digital Library

[22]

D. Slutz. Massive stochastic testing of SQL. In Proceedings of the 24th International Conference on Very Large Databases (VLDB 98), pages 618--622, 1998.

Digital Library

[23]

T. Yoshikawa, K. Shimura, and T. Ozawa. Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software (QSIC 2003), pages 20--24. IEEE Computer Society Press, 2003.

Digital Library

Cited By

Huang RCui CTowey DSun WLian J(2023)VPP-ART: An Efficient Implementation of Fixed-Size-Candidate-Set Adaptive Random Testing Using Vantage Point PartitioningIEEE Transactions on Reliability10.1109/TR.2022.321860272:4(1632-1647)Online publication date: Dec-2023
https://doi.org/10.1109/TR.2022.3218602
Huang RSun WXu YChen HTowey DXia X(2021)A Survey on Adaptive Random TestingIEEE Transactions on Software Engineering10.1109/TSE.2019.294292147:10(2052-2083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TSE.2019.2942921
Huang RChen HSun WTowey D(2021)Candidate test set reduction for adaptive random testing: An overheads reduction techniqueScience of Computer Programming10.1016/j.scico.2021.102730(102730)Online publication date: Oct-2021
https://doi.org/10.1016/j.scico.2021.102730
Show More Cited By

Index Terms

Is adaptive random testing really better than random testing

Recommendations

Adaptive random testing by balancing
RT '07: Proceedings of the 2nd international workshop on Random testing: co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007)

Adaptive Random Testing (ART) is an effective improvement of Random Testing (RT). It is based on the observation that failure-causing inputs tend to be clustered together. ART, therefore, proposes to have randomly selected test cases being more evenly ...
Enhancing adaptive random testing in high dimensional input domains
SAC '07: Proceedings of the 2007 ACM symposium on Applied computing

Adaptive random testing (ART) is an enhancement of random testing (RT). It can detect failures more effectively than RT when failure-causing inputs are clustered. Having test cases both randomly selected and evenly spread is the key to the success of ...
Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RT '06: Proceedings of the 1st international workshop on Random testing

July 2006

84 pages

ISBN:159593457X

DOI:10.1145/1145735

Program Chairs:
Johannes Mayer
Ulm University, Germany
,
Robert Merkel
Swinburne University of Technology, Australia

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ISSTA06

ISSTA06: International Symposium on Software Testing and Analysis 2006

July 20, 2006

Maine, Portland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
788
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Huang RCui CTowey DSun WLian J(2023)VPP-ART: An Efficient Implementation of Fixed-Size-Candidate-Set Adaptive Random Testing Using Vantage Point PartitioningIEEE Transactions on Reliability10.1109/TR.2022.321860272:4(1632-1647)Online publication date: Dec-2023
https://doi.org/10.1109/TR.2022.3218602
Huang RSun WXu YChen HTowey DXia X(2021)A Survey on Adaptive Random TestingIEEE Transactions on Software Engineering10.1109/TSE.2019.294292147:10(2052-2083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TSE.2019.2942921
Huang RChen HSun WTowey D(2021)Candidate test set reduction for adaptive random testing: An overheads reduction techniqueScience of Computer Programming10.1016/j.scico.2021.102730(102730)Online publication date: Oct-2021
https://doi.org/10.1016/j.scico.2021.102730
Zhang ZDai JZhao LQin S(2019)A Web Services Testing Approach based on Difference Measurement and Adaptive Random TestingProceedings of the 2019 the 9th International Conference on Communication and Network Security10.1145/3371676.3371703(1-5)Online publication date: 15-Nov-2019
https://dl.acm.org/doi/10.1145/3371676.3371703
Huang RXie XChen TLu Y(2012)Adaptive Random Test Case Generation for Combinatorial TestingProceedings of the 2012 IEEE 36th Annual Computer Software and Applications Conference10.1109/COMPSAC.2012.15(52-61)Online publication date: 16-Jul-2012
https://dl.acm.org/doi/10.1109/COMPSAC.2012.15
Parizi RGhani AAbdullah RAtan R(2011)Empirical evaluation of the fault detection effectiveness and test effort efficiency of the automated AOP testing approachesInformation and Software Technology10.1016/j.infsof.2011.05.00453:10(1062-1083)Online publication date: Oct-2011
https://doi.org/10.1016/j.infsof.2011.05.004
Liu HKuo FChen T(2011)Comparison of adaptive random testing and random testing under various testing and debugging scenariosSoftware: Practice and Experience10.1002/spe.111342:8(1055-1074)Online publication date: 2-Sep-2011
https://doi.org/10.1002/spe.1113
Zhifang Liu Xiaopeng Gao Long X(2010)Adaptive random testing of mobile application2010 2nd International Conference on Computer Engineering and Technology10.1109/ICCET.2010.5485442(V2-297-V2-301)Online publication date: Apr-2010
https://doi.org/10.1109/ICCET.2010.5485442
AboElFotoh MDean TMayor RNg JCouturier CKark AMartin P(2009)An empirical evaluation of a language-based security testing techniqueProceedings of the 2009 Conference of the Center for Advanced Studies on Collaborative Research10.1145/1723028.1723043(112-121)Online publication date: 2-Nov-2009
https://dl.acm.org/doi/10.1145/1723028.1723043
Kuo F(2009)An Indepth Study of Mirror Adaptive Random TestingProceedings of the 2009 Ninth International Conference on Quality Software10.1109/QSIC.2009.15(51-58)Online publication date: 24-Aug-2009
https://dl.acm.org/doi/10.1109/QSIC.2009.15
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten