skip to main content
research-article

Developing and Evaluating Objective Termination Criteria for Random Testing

Published: 18 July 2019 Publication History

Abstract

Random testing is a software testing technique through which programs are tested by generating and executing random inputs. Because of its unstructured nature, it is difficult to determine when to stop a random testing process. Faults may be missed if the process is stopped prematurely, and resources may be wasted if the process is run too long. In this article, we propose two promising termination criteria, “All Equivalent” (AEQ) and “All Included in One” (AIO), applicable to random testing. These criteria stop random testing once the process has reached a code-coverage-based saturation point after which additional testing effort is unlikely to provide additional effectiveness. We model and implement them in the context of a general random testing process composed of independent random testing sessions. Thirty-six experiments involving GUI testing and unit testing of Java applications have demonstrated that the AEQ criteria is generally able to stop the process when a code coverage equal or very near to the saturation level is reached, while AIO is able to stop the process earlier in cases it reaches the saturation level of coverage. In addition, the performance of the two criteria has been compared against other termination criteria adopted in the literature.

References

[1]
2015. ISO/IEC/IEEE international standard--Software and systems engineering--Software testing--Part 4: Test techniques. ISO/IEC/IEEE 29119-4:2015 (Dec. 2015), 1--149.
[2]
Moataz A. Ahmed and Irman Hermadi. 2008. GA-based multiple paths test data generator. Comput. Operat. Res. 35, 10 (Oct. 2008), 3107--3124.
[3]
Domenico Amalfitano, Nicola Amatucci, Anna Rita Fasolino, Porfirio Tramontana, Emily Kowalczyk, and Atif M. Memon. 2015. Exploiting the saturation effect in automatic random testing of android applications. In Proceedings of the 2nd ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft’15). IEEE Press, Los Alamitos, CA, 33--43. http://dl.acm.org/citation.cfm?id=2825041.2825046
[4]
Domenico Amalfitano, Anna Rita Fasolino, Salvatore De Carmine, Atif Memon, and Porfirio Tramontana. 2012. Using GUI ripping for automated testing of android applications. In Proceedings of the 27th IEEE international Conference on Automated Software Engineering (ASE’12). IEEE Computer Society, Los Alamitos, CA.
[5]
D. Amalfitano, V. Riccio, N. Amatucci, V. D. Simone, and A. R. Fasolino. 2019. Combining automated GUI exploration of Android apps with capture and replay through machine learning. Inf. Softw. Technol. 105 (2019), 95--116.
[6]
J. H. Andrews, A. Groce, M. Weston, and R. G. Xu. 2008. Random test run length and effectiveness. In Proceedings of the 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE’08). 19--28.
[7]
A. Arcuri. 2012. A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Trans. Softw. Eng. 38, 3 (May 2012), 497--519.
[8]
Andrea Arcuri and Lionel Briand. 2011. Adaptive random testing: An illusion of effectiveness? In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA’11). ACM, New York, NY, 265--275.
[9]
A. Arcuri and L. Briand. 2012. Formal analysis of the probability of interaction fault detection using random testing. IEEE Trans. Softw. Eng. 38, 5 (Sep. 2012), 1088--1099.
[10]
Andrea Arcuri and Lionel Briand. 2014. A Hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw. Test. Verif. Reliab. 24, 3 (May 2014), 219--250.
[11]
Andrea Arcuri and Gordon Fraser. 2013. Parameter tuning or default values? An empirical investigation in search-based software engineering. Empir. Softw. Eng. 18, 3 (2013), 594--623.
[12]
Andrea Arcuri, Muhammad Zohaib Iqbal, and Lionel Briand. 2012. Random testing: Theoretical results and practical implications. IEEE Trans. Softw. Eng. 38, 2 (Mar. 2012), 258--277.
[13]
Thomas R. Arnold. 1998. Visual Test 6 Bible (1st ed.). IDG Books Worldwide, Inc., Foster City, CA.
[14]
Sebastian Bauersfeld and Tanja E. J. Vos. 2012. GUITest: A Java library for fully automated GUI robustness testing. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE’12). ACM, New York, NY, 330--333.
[15]
Marcel Böhme and Soumya Paul. 2014. On the efficiency of automated testing. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14). ACM, New York, NY, 632--642.
[16]
L. Briand and D. Pfahl. 1999. Using simulation for assessing the real impact of test coverage on defect coverage. In Proceedings of the 10th International Symposium on Software Reliability Engineering 1999. 148--157.
[17]
P. M. S. Bueno and M. Jino. 2000. Identification of potentially infeasible program paths by monitoring the search for test data. In Proceedings of the 15th IEEE International Conference on Automated Software Engineering (ASE’00). 209--218.
[18]
Paulo Marcos Siqueira Bueno and Mario Jino. 2002. Automatic test data generation for program paths using genetic algorithms. Int. J. Softw. Eng. Knowl. Eng. 12, 06 (2002), 691--709.
[19]
M. Böhme and S. Paul. 2016. A probabilistic analysis of the efficiency of automated software testing. IEEE Trans. Softw. Eng. 42, 4 (Apr. 2016), 345--360.
[20]
Xia Cai and Michael R. Lyu. 2005. The effect of code coverage on fault detection under different testing profiles. In Proceedings of the 1st International Workshop on Advances in Model-based Testing (A-MOST’05). ACM, New York, NY, 1--7.
[21]
Mariano Ceccato, Alessandro Marchetto, Leonardo Mariani, Cu D. Nguyen, and Paolo Tonella. 2012. An empirical study about the effectiveness of debugging when random test cases are used. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Press, Los Alamitos, CA, 452--462. http://dl.acm.org/citation.cfm?id=2337223.2337277
[22]
Mariano Ceccato, Alessandro Marchetto, Leonardo Mariani, Cu D. Nguyen, and Paolo Tonella. 2015. Do automatically generated test cases make debugging easier? An experimental assessment of debugging effectiveness and efficiency. ACM Trans. Softw. Eng. Methodol. 25, 1, Article 5 (Dec. 2015), 38 pages.
[23]
Wen-Kui Chang and Shuen-Lin Jeng. 2007. Practical stopping criteria for validating safety-critical software by estimating impartial reliability. Appl. Math. Model. 31, 7 (2007), 1411--1424.
[24]
Mei-Hwa Chen, M. R. Lyu, and W. E. Wong. 1996. An empirical study of the correlation between code coverage and reliability estimation. In Proceedings of the 3rd International Symposium on Software Metrics: From Measurement to Empirical Results (METRICS’96). IEEE Computer Society, Los Alamitos, CA, 133--. http://dl.acm.org/citation.cfm?id=525586.823872
[25]
Mei-Hwa Chen, M. R. Lyu, and W. E. Wong. 1997. Incorporating code coverage in the reliability estimation for fault-tolerant software. In Proceedings of the 16th Symposium on Reliable Distributed Systems 1997. 45--52.
[26]
Sanping Chen and S. Mills. 1996. A binary Markov process model for random testing. IEEE Trans. Softw. Eng. 22, 3 (Mar. 1996), 218--223.
[27]
T. Y. Chen, F. C. Kuo, R. G. Merkel, and T. H. Tse. 2010. Adaptive random testing: The ART of test case diversity. J. Syst. Softw. 83, 1 (2010), 60--66.
[28]
Tsong Yueh Chen and Robert Merkel. 2008. An upper bound on software testing effectiveness. ACM Trans. Softw. Eng. Methodol. 17, 3, Article 16 (Jun. 2008), 27 pages.
[29]
T. Y. Chen, T. H. Tse, and Y. T. Yu. 2001. Proportional sampling strategy: A compendium and some insights. J. Syst. Softw. 58, 1 (Aug. 2001), 65--81.
[30]
Ilinca Ciupa, Andreas Leitner, Manuel Oriol, and Bertrand Meyer. 2007. Experimental assessment of random testing for object-oriented software. In Proceedings of the 2007 International Symposium on Software Testing and Analysis (ISSTA’07). ACM, New York, NY, 84--94.
[31]
I. Ciupa, A. Pretschner, A. Leitner, M. Oriol, and B. Meyer. 2008. On the predictability of random tests for object-oriented software. In Proceedings of the 2008 1st International Conference on Software Testing, Verification, and Validation. 72--81.
[32]
Ilinca Ciupa, Alexander Pretschner, Manuel Oriol, Andreas Leitner, and Bertrand Meyer. 2011. On the number and nature of faults found by random testing. Softw. Test. Verif. Reliab. 21, 1 (2011), 3--28.
[33]
P. A. Currit, M. Dyer, and H. D. Mills. 1986. Certifying the reliability of software. IEEE Trans. Softw. Eng. SE-12, 1 (Jan. 1986), 3--11.
[34]
Michael Dyer. 1987. A formal approach to software error removal. J. Syst. Softw. 7, 2 (1987), 109--114.
[35]
R. A. Fisher. 1922. On the interpretation of Chi squared from contingency tables, and the calculation of P. J. Roy. Stat. Soc. 85, 1 (1922), 87--94. http://www.jstor.org/stable/2340521
[36]
Justin E. Forrester and Barton P. Miller. 2000. An empirical study of the robustness of windows NT applications using random testing. In Proceedings of the 4th Conference on USENIX Windows Systems Symposium—Volume 4 (WSS’00). USENIX Association, Berkeley, CA, 6--6. http://dl.acm.org/citation.cfm?id=1267102.1267108
[37]
Phyllis G. Frankl and Oleg Iakounenko. 1998. Further empirical studies of test effectiveness. In Proceedings of the 6th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT’98/FSE-6). ACM, New York, NY, 153--162.
[38]
G. Fraser and A. Arcuri. 2011. Evolutionary generation of whole test suites. In Proceedings of the 2011 11th International Conference on Quality Software. 31--40.
[39]
G. Fraser and A. Arcuri. 2012. Sound empirical evidence in software testing. In Proceedings of the 2012 34th International Conference on Software Engineering (ICSE’12). 178--188.
[40]
Gordon Fraser and Andrea Arcuri. 2013. Whole test suite generation. IEEE Trans. Softw. Eng. 39, 2 (Feb. 2013), 276--291.
[41]
G. Fraser and A. Gargantini. 2009. Experiments on the test case length in specification based test case generation. In Proceedings of the ICSE Workshop on Automation of Software Test (AST’09). 18--26.
[42]
Carlo A. Furia, Bertrand Meyer, Manuel Oriol, Andrey Tikhomirov, and Yi Wei. 2013. The search for the laws of automatic random testing. In Proceedings of the 28th Annual ACM Symposium on Applied Computing (SAC’13). ACM, New York, NY, 1211--1216.
[43]
P. Garg. 1994. Investigating coverage-reliability relationship and sensitivity of reliability to errors in the operational profile. In Proceedings of the 1st International Conference on Software Testing, Reliability and Quality Assurance 1994. 21--35.
[44]
Moheb R. Girgis. 2005. Automatic test data generation for data flow testing using a genetic algorithm. Journal of Universal Computer Science (JUCS) 11, 6 (Jun. 2005), 898--915.
[45]
Patrice Godefroid, Adam Kiezun, and Michael Y. Levin. 2008. Grammar-based whitebox fuzzing. SIGPLAN Not. 43, 6 (Jun. 2008), 206--215.
[46]
Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. SIGPLAN Not. 40, 6 (Jun. 2005), 213--223.
[47]
A. L. Goel. 1985. Software reliability models: Assumptions, limitations, and applicability. IEEE Trans. Softw. Eng. SE-11, 12 (Dec. 1985), 1411--1423.
[48]
S. S. Gokhale, T. Philip, P. N. Marinos, and K. S. Trivedi. 1996. Unification of finite failure non-homogeneous Poisson process models through test coverage. In Proceedings of the 7th International Symposium on Software Reliability Engineering 1996. 299--307.
[49]
S. S. Gokhale and K. S. Trivedi. 1998. Log-logistic software reliability growth model. In Proceedings of the 3rd IEEE International High-Assurance Systems Engineering Symposium 1998. 34--41.
[50]
Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. 2012. Swarm testing. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA’12). ACM, New York, NY, 78--88.
[51]
Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. EXSYST: Search-based GUI testing. In Proceedings of the 34th International Conference on Software Engineering (ICSE’12). IEEE Press, Los Alamitos, CA, 1423--1426. http://dl.acm.org/citation.cfm?id=2337223.2337435
[52]
Florian Gross, Gordon Fraser, and Andreas Zeller. 2012. Search-based system testing: High coverage, no false alarms. In Proceedings of the 2012 International Symposium on Software Testing and Analysis (ISSTA’12). ACM, New York, NY, 67--77.
[53]
Michael Grottke. 2002. A vector Markov model for structural coverage growth and the number of failure occurrences. In Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE’02). IEEE Computer Society, Los Alamitos, CA, 304. http://dl.acm.org/citation.cfm?id=851033.856318
[54]
A. Hajjar, T. Chen, I. Munn, A. Andrews, and M. Bjorkman. 2001. High quality behavioral verification using statistical stopping criteria. In Proceedings of the Design, Automation and Test in Europe, 2001. Conference and Exhibition 2001. 411--418.
[55]
A. Hajjar, T. Chen, I. Munn, A. Andrews, and M. Bjorkman. 2001. Stopping criteria comparison: Towards high quality behavioral verification. In Proceedings of the 2001 International Symposium on Quality Electronic Design. 31--37.
[56]
Dick Hamlet. 2006. When only random testing will do. In Proceedings of the 1st International Workshop on Random Testing (RT’06). ACM, New York, NY, 1--9.
[57]
Richard Hamlet. 2002. Random Testing. John Wiley 8 Sons, New York, NY.
[58]
Shuai Hao, Bin Liu, Suman Nath, William G. J. Halfond, and Ramesh Govindan. 2014. PUMA: Programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys’14). ACM, New York, NY, 204--217.
[59]
B. Hofer, B. Peischl, and F. Wotawa. 2009. GUI savvy end-to-end testing with smart monkeys. In Proceedings of the ICSE Workshop on Automation of Software Test (AST’09). 130--137.
[60]
W. E. Howden. 1997. Confidence-based reliability and statistical coverage estimation. In Proceedings of the 8th International Symposium on Software Reliability Engineering, 1997. 283--291.
[61]
Monica Hutchins, Herb Foster, Tarak Goradia, and Thomas Ostrand. 1994. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE’94). IEEE Computer Society Press, Los Alamitos, CA, 191--200. http://dl.acm.org/citation.cfm?id=257734.257766
[62]
P. Jalote and Y. R. Muralidhara. 1994. A coverage based model for software reliability estimation. In Proceedings of the 1st International Conference on Software Testing, Reliability and Quality Assurance 1994. 6--10.
[63]
Saileshwar Krishnamurthy and Aditya P. Mathur. 1996. On predicting reliability of modules using code coverage. In Proceedings of the 1996 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON’96). IBM Press, 22-. http://dl.acm.org/citation.cfm?id=782052.782074
[64]
M. Lil, Y. Wei, D. Desovski, H. Nejad, S. Ghose, B. Cukic, and C. Smidts. 2004. Validation of a methodology for assessing software reliability. In Proceedings of the 15th International Symposium on Software Reliability Engineering (ISSRE’04). 66--76.
[65]
Jin-Cherng Lin and Pu-Lin Yeh. 2001. Automatic test data generation for path testing using GAs. Inf. Sci. 131, 1--4 (Jan. 2001), 47--64.
[66]
B. Littlewood and D. Wright. 1997. Some conservative stopping rules for the operational testing of safety critical software. IEEE Trans. Softw. Eng. 23, 11 (Nov. 1997), 673--683.
[67]
H. Liu, X. Xie, J. Yang, Y. Lu, and T. Y. Chen. 2010. Adaptive random testing by exclusion through test profile. In Proceedings of the 2010 10th International Conference on Quality Software. 92--101.
[68]
Z. Liu, X. Gao, and X. Long. 2010. Adaptive random testing of mobile application. In Proceedings of the 2nd International Conference on Computer Engineering and Technology (ICCET’10), Vol. 2. IEEE, V2--297.
[69]
Michael R Lyu et al. 1996. Handbook of Software Reliability Engineering. Vol. 222. IEEE Computer Society, Los Alamitos, CA.
[70]
M. R. Lyu, Zubin Huang, S. K. S. Sze, and Xia Cai. 2003. An empirical study on testing and fault tolerance for software reliability engineering. In Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE’03). 119--130.
[71]
L. Ma, C. Artho, C. Zhang, H. Sato, J. Gmeiner, and R. Ramler. 2015. GRT: Program-analysis-guided random testing (T). In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE’15). 212--223.
[72]
Aravind Machiry, Rohan Tahiliani, and Mayur Naik. 2013. Dynodroid: An input generation system for Android apps. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’13). ACM, New York, NY, 224--234.
[73]
Riyadh Mahmood, Nariman Mirzaei, and Sam Malek. 2014. EvoDroid: Segmented evolutionary testing of Android apps. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14). ACM, New York, NY, 599--609.
[74]
Y. K. Malaiya, M. N. Li, J. M. Bieman, and R. Karcich. 2002. Software reliability growth with test coverage. IEEE Trans. Reliabil. 51, 4 (Dec. 2002), 420--426.
[75]
Y. K. Malaiya, Naixin Li, J. Bieman, R. Karcich, and B. Skibbe. 1994. The relationship between test coverage and reliability. In Proceedings of the 5th International Symposium on Software Reliability Engineering 1994. 186--195.
[76]
N. Malevris and E. Petrova. 2000. On the determination of an appropriate time for ending the software testing process. In Proceedings of the 1st Asia-Pacific Conference on Quality Software 2000. 81--88.
[77]
Nashat Mansour and Miran Salame. 2004. Data generation for path testing. Softw. Qual. J. 12, 2 (Jun. 2004), 121--136.
[78]
Leonardo Mariani, Mauro Pezzè, Oliviero Riganelli, and Mauro Santoro. 2014. Automatic testing of GUI-based applications. Softw. Test. Verif. Reliab. 24, 5 (Aug. 2014), 341--366.
[79]
Leonardo Mariani, Mauro PezzÃl, and Daniele Zuddas. 2015. Recent advances in automatic black-box testing. In Advances in Computers, Vol. 99. Elsevier, 157--193.
[80]
Lorenzo Martignoni, Roberto Paleari, Giampaolo Fresi Roglia, and Danilo Bruschi. 2010. Testing system virtual machines. In Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA’10). ACM, New York, NY, 171--182.
[81]
Tim Menzies and Bojan Cukic. 2000. When to test less. IEEE Softw. 17, 5 (2000), 107--112.
[82]
James Miller, Marek Reformat, and Howard Zhang. 2006. Automatic test data generation using genetic algorithm and program dependence graphs. Inf. Softw. Technol. 48, 7 (2006), 586--605.
[83]
Rajeev Motwani and Prabhakar Raghavan. 1995. Randomized Algorithms. Cambridge University Press, New York, NY.
[84]
Akbar Siami Namin and James H. Andrews. 2009. The influence of size and coverage on test suite effectiveness. In Proceedings of the 18th International Symposium on Software Testing and Analysis (ISSTA’09). ACM, New York, NY, 57--68.
[85]
Simeon C. Ntafos. 2001. On comparisons of random, partition, and proportional partition testing. IEEE Trans. Softw. Eng. 27, 10 (Oct. 2001), 949--960.
[86]
P. Oehlert. 2005. Violating assumptions with fuzzing. IEEE Secur. Priv. 3, 2 (Mar. 2005), 58--62.
[87]
M. Oriol. 2012. Random testing: Evaluation of a law describing the number of faults found. In Proceedings of the 2012 IEEE 5th International Conference on Software Testing, Verification and Validation. 201--210.
[88]
Alessandro Orso and Gregg Rothermel. 2014. Software testing: A research travelogue (2000--2014). In Proceedings of the on Future of Software Engineering (FOSE’14). ACM, New York, NY, 117--132.
[89]
Carlos Pacheco and Michael D. Ernst. 2005. Eclat: Automatic generation and classification of test inputs. In Proceedings of the 19th European Conference on Object-Oriented Programming (ECOOP’05). Springer-Verlag, Berlin, 504--527.
[90]
C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. 2007. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). 75--84.
[91]
David L. Parnas, A. John van Schouwen, and Shu Po Kwan. 1990. Evaluation of safety-critical software. Commun. ACM 33, 6 (Jun. 1990), 636--648.
[92]
Min Pei, Erik D. Goodman, Zongyi Gao, and Kaixiang Zhong. 1994. Automated software test data generation using a Genetic Algorithm. Technical Report. Michigan State University.
[93]
Paul Piwowarski, Mitsuru Ohba, and Joe Caruso. 1993. Coverage measurement experience during function test. In Proceedings of the 15th International Conference on Software Engineering (ICSE’93). IEEE Computer Society Press, Los Alamitos, CA, 287--301. http://dl.acm.org/citation.cfm?id=257572.257635
[94]
James Ramsey and Victor R. Basili. 1985. Analyzing the test process using structural coverage. In Proceedings of the 8th International Conference on Software Engineering (ICSE’85). IEEE Computer Society Press, Los Alamitos, CA, 306--312.
[95]
Abdul Rauf, Sajid Anwar, M. Arfan Jaffar, and Arshad Ali Shahid. 2010. Automated GUI test coverage analysis using GA. In Proceedings of the 7th International Conference on Information Technology: New Generations (ITNG’10). 1057--1062.
[96]
Lenin Ravindranath, Suman Nath, Jitendra Padhye, and Hari Balakrishnan. 2014. Automatic and scalable fault detection for mobile applications. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys’14). ACM, New York, NY, 190--203.
[97]
Urko Rueda, Tanja E. J. Vos, and I. S. W. B. Prasetya. 2015. Unit testing tool competition: Round three. In Proceedings of the 8th International Workshop on Search-Based Software Testing (SBST’15). IEEE Press, Piscataway, NJ, 19--24. http://dl.acm.org/citation.cfm?id=2821339.2821346
[98]
M. Sahinoglu. 2003. An empirical Bayesian stopping rule in testing and verification of behavioral models. IEEE Trans. Instrum. Meas. 52, 5 (Oct. 2003), 1428--1443.
[99]
M. Sahinoglu, A. von Mayrhauser, A. Hajjar, T. Chen, and C. Anderson. 1999. On the efficiency of a compound Poisson stopping rule for mixed strategy testing. In Proceedings of the 1999 IEEE Aerospace Conference, Vol. 5. 93--98 vol. 5.
[100]
Rohan Sharma, Milos Gligoric, Andrea Arcuri, Gordon Fraser, and Darko Marinov. 2011. Testing container classes: Random or systematic? In Proceedings of the 14th International Conference on Fundamental Approaches to Software Engineering: Part of the Joint European Conferences on Theory and Practice of Software (FASE’11/ETAPS’11). Springer-Verlag, Berlin, 262--277. http://dl.acm.org/citation.cfm?id=1987434.1987460
[101]
Elena Sherman, Matthew B. Dwyer, and Sebastian Elbaum. 2009. Saturation-based testing of concurrent programs. In Proceedings of the the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE’09). ACM, New York, NY, 53--62.
[102]
P. Thévenod-Fosse and H. Waeselynck. 1991. An investigation of statistical software testing. Softw. Test. Verif. Reliabil. 1, 2 (1991), 5--25.
[103]
Petar Tsankov, Mohammad Torabi Dashti, and David Basin. 2012. SecFuzz: Fuzz-testing security protocols. In Proceedings of the 7th International Workshop on Automation of Software Test (AST’12). IEEE Press, Piscataway, NJ, 1--7. http://dl.acm.org/citation.cfm?id=2663608.2663610
[104]
Alan Veevers and Adam C. Marshall. 1994. A relationship between software coverage metrics and reliability. Softw. Test. Verif. Reliabil. 4, 1 (1994), 3--8.
[105]
Y. Wei, S. Gebhardt, B. Meyer, and M. Oriol. 2010. Satisfying test preconditions through guided object selection. In Proceedings of the 2010 3rd International Conference on Software Testing, Verification and Validation. 303--312.
[106]
Yi Wei, Bertrand Meyer, and Manuel Oriol. 2012. Is branch coverage a good measure of testing effectiveness? In Empirical Software Engineering and Verification, Bertrand Meyer and Martin Nordio (Eds.). Lecture Notes in Computer Science, Vol. 7007. Springer, Berlin, 194--212.
[107]
Elaine J. Weyuker and Bingchiang Jeng. 1991. Analyzing partition testing strategies. IEEE Trans. Softw. Eng. 17, 7 (Jul. 1991), 703--711.
[108]
J. A. Whittaker and M. G. Thomason. 1994. A Markov chain model for statistical software testing. IEEE Trans. Softw. Eng. 20, 10 (Oct. 1994), 812--824.
[109]
C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer, Berlin.

Cited By

View all
  • (2024)Bounding Random Test Set Size with Computational Learning TheoryProceedings of the ACM on Software Engineering10.1145/36608191:FSE(2538-2560)Online publication date: 12-Jul-2024
  • (2024)Extrapolating Coverage Rate in Greybox FuzzingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639198(1-12)Online publication date: 20-May-2024
  • (2023)Website Functionality Testing Using the Automation Tool2023 IEEE International Conference on Data and Software Engineering (ICoDSE)10.1109/ICoDSE59534.2023.10291853(256-260)Online publication date: 7-Sep-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology
ACM Transactions on Software Engineering and Methodology  Volume 28, Issue 3
July 2019
278 pages
ISSN:1049-331X
EISSN:1557-7392
DOI:10.1145/3343019
  • Editor:
  • Mauro Pezzè
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2019
Accepted: 01 May 2019
Revised: 01 February 2019
Received: 01 March 2018
Published in TOSEM Volume 28, Issue 3

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Random testing
  2. code coverage
  3. saturation effect

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)38
  • Downloads (Last 6 weeks)11
Reflects downloads up to 05 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Bounding Random Test Set Size with Computational Learning TheoryProceedings of the ACM on Software Engineering10.1145/36608191:FSE(2538-2560)Online publication date: 12-Jul-2024
  • (2024)Extrapolating Coverage Rate in Greybox FuzzingProceedings of the IEEE/ACM 46th International Conference on Software Engineering10.1145/3597503.3639198(1-12)Online publication date: 20-May-2024
  • (2023)Website Functionality Testing Using the Automation Tool2023 IEEE International Conference on Data and Software Engineering (ICoDSE)10.1109/ICoDSE59534.2023.10291853(256-260)Online publication date: 7-Sep-2023
  • (2022)Metamorphic Testing for Edge Real-Time Face Recognition and Intrusion Detection Solution2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall)10.1109/VTC2022-Fall57202.2022.10012836(1-5)Online publication date: Sep-2022
  • (2022)Testing Self-Adaptive Software With Probabilistic Guarantees on Performance Metrics: Extended and Comparative ResultsIEEE Transactions on Software Engineering10.1109/TSE.2021.310113048:9(3554-3572)Online publication date: 1-Sep-2022
  • (2022)On the use of spectral algorithms for the prediction of short-lived volatile fission product release: Methodology for bounding numerical errorNuclear Engineering and Technology10.1016/j.net.2021.10.02854:4(1195-1205)Online publication date: Apr-2022
  • (2021) testar – scriptless testing through graphical user interface Software Testing, Verification and Reliability10.1002/stvr.177131:3Online publication date: 25-Apr-2021
  • (2020)Testing self-adaptive software with probabilistic guarantees on performance metricsProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409685(1002-1014)Online publication date: 8-Nov-2020
  • (2020)Adaptive random testing based on flexible partitioningIET Software10.1049/iet-sen.2019.032514:5(493-505)Online publication date: Oct-2020
  • (2020)A Technique for Parallel GUI Testing of Android ApplicationsTesting Software and Systems10.1007/978-3-030-64881-7_11(169-185)Online publication date: 2-Dec-2020

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media