Skip to main content
SpringerLink
Log in

Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Adaptive random testing (ART) achieves better failure-detection effectiveness than random testing by increasing the diversity of test cases. However, the intention of ensuring even spread of test cases inevitably causes an overhead problem. Although two basic forgetting strategies (i.e. random forgetting and consecutive retention) were proposed to reduce the computation cost of ART, they only considered the temporal distribution of test cases. In the paper, we presented a distance-aware forgetting strategy for the fixed size candidate set version of ART (DF-FSCS), in which the spatial distribution of test cases is taken into consideration. For a given candidate, the test cases out of its “sight” are ignored to reduce the distance computation cost. At the same time, the dynamic adjustment for partitioning and the second-round forgetting are adopted to ensure the linear complexity of DF-FSCS algorithm. Both simulation analysis and empirical study are employed to investigate the efficiency and effectiveness of DF-FSCS. The experimental results show that DF-FSCS significantly outperforms the classical ART algorithm FSCS-ART in efficiency, and has comparable failure-detection effectiveness. Com-pared with two basic forgetting methods, DF-FSCS is better in both efficiency and effectiveness. In contrast with a typical linear-time ART algorithm RBCVT-Fast, our algorithm requires less computational overhead and exhibits the similar failure-detection capability. In addition, DF-FSCS has more reliable performance than RBCVT-Fast in detecting failures for the programs with high-dimensional input domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hamlet R. Random testing. In: Marciniak J J, ed. Encyclopedia of Software Engineering. 2nd ed. Chichester: John Wiley and Sons, 2002. 1507–1513

    Google Scholar 

  2. Chen T Y, Leung H, Mak I K. Adaptive random testing. In: Proceedings of the 9th Asian Computing Science Conference, Chiang Mai, 2004. 320–329

    MATH  Google Scholar 

  3. Chen T Y, Kuo F-C, Merkel R G, et al. Adaptive random testing: the ART of test case diversity. J Syst Softw, 2010, 83: 60–66

    Article  Google Scholar 

  4. Orso A, Rothermel G. Software testing: a research travelogue (2000-2014). In: Proceedings of Future of Software Engineering (FOSE’14), Hyderabad, 2014. 117–132

    Google Scholar 

  5. Chen T Y, Kuo F-C, Towey D, et al. A revisit of three studies related to random testing. Sci China Inf Sci, 2015, 58: 052104

    Google Scholar 

  6. Shi Q, Chen Z, Fang C, et al. Measuring the diversity of a test set with distance entropy. IEEE Trans Reliab, 2016, 65: 19–27

    Article  Google Scholar 

  7. Feldt R, Poulding S, Clark D, et al. Test set diameter: quantifying the diversity of sets of test cases. In: Proceedings of the 9th IEEE International Conference on Software Testing, Verification and Validation (ICST’16), Chicago, 2016. 223–233

    Google Scholar 

  8. Chen T Y, Merkel R G. An upper bound on software testing effectiveness. ACM Trans Softw Eng Methodol, 2008, 17: 1–27

    Article  Google Scholar 

  9. Chen T Y. Fundamentals of test case selection: diversity, diversity, diversity. In: Proceedings of the 2nd International Conference on Software Engineering and Data Mining (SEDM’10), Chengdu, 2010. 723–724

    Google Scholar 

  10. Mariani L, Pezzè M, Zuddas D. Recent advances in automatic black-box testing. Adv Comput, 2015, 99: 157–193

    Article  Google Scholar 

  11. Arcuri A, Briand L. Adaptive random testing: an illusion of effectiveness? In: Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA’11), Toronto, 2011. 265–275

    Google Scholar 

  12. Chan K P, Chen T Y, Towey D. Forgetting test cases. In: Proceedings of the 30th Annual International Computer Software and Applications Conference (COMPSAC’06), Chicago, 2006. 485–494

    Google Scholar 

  13. Rigaux P, Scholl M, Voisard A. Spatial Databases: With Application to GIS. San Francisco: Morgan Kaufmann Publishers, 2002. 267–309

    Book  Google Scholar 

  14. Shahbazi A, Tappenden A F, Miller J. Centroidal voronoi tessellations—a new approach to random testing. IEEE Trans Softw Eng, 2013, 39: 163–183

    Article  Google Scholar 

  15. Finelli G B. NASA software failure characterization experiments. Reliab Eng Syst Saf, 1991, 32: 155–169

    Article  Google Scholar 

  16. Bishop P G. The variation of software survival time for different operational input profiles. In: Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23), Toulouse, 1993. 98–107

    Google Scholar 

  17. Schneckenburger C, Mayer J. Towards the determination of typical failure patterns. In: Proceedings of the 4th International Workshop on Software Quality Assurance (SOQUA’07), Dubrovnik, 2007. 90–93

    Google Scholar 

  18. Chen T Y, Tse T H, Yu Y T. Proportional sampling strategy: a compendium and some insights. J Syst Softw, 2001, 58: 65–81

    Article  Google Scholar 

  19. Chen T Y, Huang D H. Adaptive random testing by localization. In: Proceedings of the 11th Asia-Pacific Software Engineering Conference (APSEC’04), Busan, 2004. 292–298

    Google Scholar 

  20. Chen T Y, Kuo F-C, Merkel R. On the statistical properties of testing effectiveness measures. J Syst Softw, 2006, 79: 591–601

    Article  Google Scholar 

  21. Kuo F-C. On adaptive random testing. Dissertation for the Doctoral Degree. Melbourne: Swinburne University of Technology, 2006. 24–26

    Google Scholar 

  22. ACM. Collected Algorithms from ACM: Volume 1, Algorithms 1–220. New York: Association for Computer Machinery, 1980

    Google Scholar 

  23. Press W H, Teukolsky S A, Vetterling W T, et al. Numerical Recipes: The Art of Scientific Computing. 3rd ed. Cambridge: Cambridge University Press, 2007

    MATH  Google Scholar 

  24. Jia Y, Harman M. An analysis and survey of the development of mutation testing. IEEE Trans Softw Eng, 2011, 37: 649–678

    Article  Google Scholar 

  25. Chen T Y, Kuo F-C, Merkel R G, et al. Mirror adaptive random testing. Inform Softw Tech, 2004, 46: 1001–1010

    Article  Google Scholar 

  26. Chen T Y, Merkel R, Eddy G, et al. Adaptive random testing through dynamic partitioning. In: Proceedings of the 4th International Conference on Quality Software (QSIC’04), Braunschweig, 2004. 79–86

    Google Scholar 

  27. Chen T Y, Huang D, Zhou Z. On adaptive random testing through iterative partitioning. J Inf Sci Eng, 2011, 27: 1449–1472

    Google Scholar 

  28. Chow C, Chen T Y, Tse T H. The ART of divide and conquer: an innovative approach to improving the efficiency of adaptive random testing. In: Proceedings of the 13th International Conference on Quality Software (QSIC’13), Nanjing, 2013. 268–275

    Google Scholar 

  29. Barus A C, Chen T Y, Kuo F-C, et al. A cost-effective random testing method for programs with non-numeric inputs. IEEE Trans Comput, 2016, 65: 3509–3523

    MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 61462030), Australian Research Council Linkage Grant (Grant No. LP100200208), Natural Science Foun-dation of Jiangxi Province (Grant Nos. 20162BCB23036, 20151BAB207018), and Science Foundation of Jiangxi Educational Committee (Grant No. GJJ150465).

Author information

Authors and Affiliations

  1. School of Software and Communication Engineering, Jiangxi University of Finance and Economics, Nanchang, 330013, China

    Chengying Mao

  2. Department of Computer Science and Software Engineering, Swinburne University of Technology, Melbourne, 3122, Australia

    Tsong Yueh Chen & Fei-Ching Kuo

Authors
  1. Chengying Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tsong Yueh Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei-Ching Kuo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chengying Mao.

Additional information

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, C., Chen, T.Y. & Kuo, FC. Out of sight, out of mind: a distance-aware forgetting strategy for adaptive random testing. Sci. China Inf. Sci. 60, 092106 (2017). https://doi.org/10.1007/s11432-016-0087-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-016-0087-0

Keywords

Search

Navigation