research-article

An upper bound on software testing effectiveness

Authors:

Tsong Yueh Chen,

Robert MerkelAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 17, Issue 3

Article No.: 16, Pages 1 - 27

https://doi.org/10.1145/1363102.1363107

Published: 27 June 2008 Publication History

Abstract

Failure patterns describe typical ways in which inputs revealing program failure are distributed across the input domain—in many cases, clustered together in contiguous regions. Based on these observations several debug testing methods have been developed. We examine the upper bound of debug testing effectiveness improvements possible through making assumptions about the shape, size and orientation of failure patterns. We consider the bounds for testing strategies with respect to minimizing the F-measure, maximizing the P-measure, and maximizing the E-measure. Surprisingly, we find that the empirically measured effectiveness of some existing methods that are not based on these assumptions is close to the theoretical upper bound of these strategies. The assumptions made to obtain the upper bound, and its further implications, are also examined.

References

[1]

Ammann, P. E. and Knight, J. C. 1988. Data diversity: An approach to software fault tolerance. IEEE Trans. Comput. 37, 4, 418--425.

Digital Library

[2]

Bishop, P. G. 1993. The variation of software survival times for different operational input profiles. In FTSC-23. Digest of Papers, the Twenty-Third International Symposium on Fault-Tolerant Computing. IEEE Computer Society Press, 98--107.

[3]

Chan, F. T., Chen, T. Y., Mak, I. K., and Yu, Y. T. 1996. Proportional sampling strategy: Guidelines for software testing practitioners. Inform. Softw. Tech. 38, 12, 775--782.

[4]

Chen, T. Y., Kuo, F.-C., and Merkel, R. 2006. On the statistical properties of testing effectiveness measures. J. Syst. Softw. 79, 591--601.

Digital Library

[5]

Chen, T. Y., Kuo, F.-C., Merkel, R., and Ng, S. 2004a. Mirror adaptive random testing. Inform. Softw. Tech. 46, 15, 1001--1010.

[6]

Chen, T. Y., Leung, H., and Mak, I. K. 2004b. Adaptive random testing. In Proceedings of the 9th Asian Computing Science Conference. Lecture Notes in Computer Science, vol. 3321. 320--329.

Digital Library

[7]

Chen, T. Y. and Yu, Y. T. 1996. On the expected number of failures detected by subdomain testing and random testing. IEEE Trans. Softw. Eng. 22, 2, 109--119.

Digital Library

[8]

Ciupa, I., Leitner, A., Oriol, M., and Meyer, B. 2006. Object distance and its application to adaptive random testing of object-oriented programs. In Proceedings of the 1st International Workshop on Random Testing. ACM Press, Portland, Maine, 55--63.

Digital Library

[9]

Dickinson, W., Leon, D., and Podgurski, A. 2001. Pursuing failure: the distribution of program failures in a profile space. In ESEC/FSE-9: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM Press, 246--255.

Digital Library

[10]

Duran, J. W. and Ntafos, S. C. 1984. Evaluation of random testing. IEEE Trans. Softw. Eng. 10, 4, 438--444.

Digital Library

[11]

Frankl, P. G., Hamlet, R. G., Littlewood, B., and Strigini, L. 1998. Evaluating testing methods by delivered reliability. IEEE Trans. Softw. Eng. 21, 8, 586--601.

Digital Library

[12]

Frankl, P. G. and Weyuker, E. J. 1993. A formal analysis of the fault-detecting ability of testing methods. IEEE Trans. Softw. Eng. 19, 3, 202--213.

Digital Library

[13]

Hamlet, R. and Taylor, R. 1990. Partition testing does not inspire confidence. IEEE Trans. Softw. Eng. 16, 12, 1402--1411.

Digital Library

[14]

Kuo, F.-C. 2006. On adaptive random testing. Ph.D. thesis, Swinburne University of Technology, Hawthorn, Australia.

[15]

Merkel, R. 2005. Analysis and enhancements of adaptive random testing. Ph.D. thesis, Swinburne University of Technology, Hawthorn, Australia.

[16]

Morasca, S. and Serra-Capizzano, S. 2004. On the analytical comparison of testing techniques. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA2004). ACM, 154--164.

Digital Library

[17]

Weisstein, E. W. 2004. Tessellation. From Mathworld—a Wolfram Web Resource, http://mathworld.wolfram.com/Tessellation.html.

[18]

White, L. and Cohen, E. 1980. A domain strategy for computer program testing. IEEE Trans. Softw. Eng. 6, 247--257.

Digital Library

Cited By

Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Ying ZTowey DBellotti AChen TZhou Z(2024)SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domainsInformation and Software Technology10.1016/j.infsof.2024.107528(107528)Online publication date: Jul-2024
https://doi.org/10.1016/j.infsof.2024.107528
Shu THe ZYin XDing ZZhou M(2024)Model-based diversity-driven learn-to-rank test case prioritizationExpert Systems with Applications10.1016/j.eswa.2024.124768255(124768)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124768
Show More Cited By

Index Terms

An upper bound on software testing effectiveness

Recommendations

Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random ...
On the statistical properties of testing effectiveness measures
Special issue: Quality software

We examine the statistical variability of three commonly used software testing effectiveness measures-the E-measure (expected number of failures detected), P-measure (probability of detecting at least one failure), and F-measure (number of tests ...
Does Adaptive Random Testing Deliver a Higher Confidence than Random Testing?
QSIC '08: Proceedings of the 2008 The Eighth International Conference on Quality Software

Random testing (RT) is a fundamental software testing technique. Motivated by the rationale that neighbouring test cases tend to cause similar execution behaviours, adaptive random testing (ART) was proposed as an enhancement of RT, which enforces ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 17, Issue 3

June 2008

133 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/1363102

Issue’s Table of Contents

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2008

Accepted: 01 April 2007

Revised: 01 January 2007

Received: 01 June 2006

Published in TOSEM Volume 17, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

80
Total Citations
View Citations
1,520
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)2

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ying ZTowey DChen TZhou Z(2024)MT-PART: Metamorphic-Testing-Based Adaptive Random Testing Through Partitioning2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00158(1184-1193)Online publication date: 2-Jul-2024
https://doi.org/10.1109/COMPSAC61105.2024.00158
Ying ZTowey DBellotti AChen TZhou Z(2024)SFIDMT-ART: A metamorphic group generation method based on Adaptive Random Testing applied to source and follow-up input domainsInformation and Software Technology10.1016/j.infsof.2024.107528(107528)Online publication date: Jul-2024
https://doi.org/10.1016/j.infsof.2024.107528
Shu THe ZYin XDing ZZhou M(2024)Model-based diversity-driven learn-to-rank test case prioritizationExpert Systems with Applications10.1016/j.eswa.2024.124768255(124768)Online publication date: Dec-2024
https://doi.org/10.1016/j.eswa.2024.124768
Ying ZTowey DGraham Bellotti AQuan Zhou Z(2024)MRGS‐ART: Metamorphic Relation and Group Selection Based on Adaptive Random TestingSoftware Testing, Verification and Reliability10.1002/stvr.190835:1Online publication date: 27-Nov-2024
https://doi.org/10.1002/stvr.1908
Huang RChen HSun WTowey D(2023)Extended Abstract of Candidate Test Set Reduction for Adaptive Random Testing: An Overheads Reduction Technique2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER56733.2023.00102(853-854)Online publication date: Mar-2023
https://doi.org/10.1109/SANER56733.2023.00102
Liu YZhang X(2022)Adaptive Random Testing for Multiagent Path Finding SystemsIEEE Transactions on Reliability10.1109/TR.2022.314632371:1(295-308)Online publication date: Mar-2022
https://doi.org/10.1109/TR.2022.3146323
Tiwari DZhang LMonperrus MBaudry B(2022)Production Monitoring to Improve Test SuitesIEEE Transactions on Reliability10.1109/TR.2021.310131871:3(1381-1397)Online publication date: Sep-2022
https://doi.org/10.1109/TR.2021.3101318
Ying ZBellotti ATowey DChen TZhou Z(2022)Using Metamorphic Relation Violation Regions to Support a Simulation Framework for the Process of Metamorphic Testing2022 IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC54236.2022.00274(1722-1727)Online publication date: Jun-2022
https://doi.org/10.1109/COMPSAC54236.2022.00274
Huang RChen HSun WTowey D(2022)Candidate test set reduction for adaptive random testingScience of Computer Programming10.1016/j.scico.2021.102730214:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.scico.2021.102730
Huang RSun WXu YChen HTowey DXia X(2021)A Survey on Adaptive Random TestingIEEE Transactions on Software Engineering10.1109/TSE.2019.294292147:10(2052-2083)Online publication date: 1-Oct-2021
https://doi.org/10.1109/TSE.2019.2942921
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents