Abstract
We propose a novel regression test case prioritization technique targeting to detect ‘hard to detect’ faults. A ‘hard to detect fault’ is a fault that is detected by only one test case. In our technique, we first seed a large number of bugs into a program to create mutants. Each mutant is executed with the test suite and their execution results are recorded in a fault matrix. Using the fault matrix, we first prioritize the test cases based on their ‘hard to detect’ values. Our technique assigns higher priority to those test cases which reveal bugs in a program-component that are hard to expose. Subsequently, the remaining test cases are prioritized based on their fault detection capability. Our experimental results show that on an average our proposed TCP technique performs 43.82% better than existing TCP techniques.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In this paper, we have used the terms ‘bug’ and ‘fault’ interchangeably.
References
Busjaeger B, Xie T (2016) Learning for test prioritization: an industrial case study, in: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 975–980
Dhareula P, Ganpati A (2020) Flower pollination algorithm for test case prioritization in regression testing. In: ICT Analysis and Applications, Springer 155–167
Dutta A, Kumar S, Godboley S (2019) Enhancing test cases generated by concolic testing. In: Proceedings of the 12th Innovations on Software Engineering Conference (formerly known as India Software Engineering Conference), pp. 1–11
Elbaum S, Malishevsky AG, Rothermel G (2002) Test case prioritization: a family of empirical studies. IEEE Transact Softw Eng 28(2):159–182. https://doi.org/10.1109/32.988497
Elbaum S, Rothermel G, Kanduri S, Malishevsky AG (2004) Selecting a cost-effective test case prioritization technique. Softw Qual J 12(3):185–210. https://doi.org/10.1023/B:SQJO.0000034708.84524.22
Elbaum S, Malishevsky A, Rothermel G (2001) Incorporating varying test costs and fault severities into test case prioritization. In: Proceedings of the 23rd International Conference on Software Engineering. ICSE 2001, pp. 329–338. https://doi.org/10.1109/ICSE.2001.919106
Gligoric M, Groce A, Zhang C, Sharma R, Alipour MA, Marinov D (2015) Guidelines for coverage-based comparisons of non-adequate test suites. ACM Transact Softw Eng Methodol (TOSEM) 24(4):1–33
Gökçe N, Eminov M, Belli F (2006) Coverage-based, prioritized testing using neural network clustering. In: International Symposium on Computer and Information Sciences, Springer, pp. 1060–1071
Gregg R, Mary Jean H, Dedhia, (1999) Regression test selection for c++ software. Tech rep, USA
Harrold MJ, Jones JA, Li T, Liang D, Orso A, Pennings M, Sinha S, Spoon SA, Gujarathi A (2001) Regression test selection for java software. SIGPLAN Not 36(11):312–326. https://doi.org/10.1145/504311.504305
Jones JA, Harrold MJ (2003) Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans Software Eng 29(3):195–209
Just R, Jalali D, Ernst MD (2014) Defects4J: A Database of existing faults to enable controlled testing studies for Java programs. In: ISSTA 2014, Proceedings of the 2014 International Symposium on Software Testing and Analysis, San Jose, CA, USA, pp. 437–440, tool demo
Korel B, Tahat LH, Harman M (2005) Test prioritization using system models. In: 21st IEEE International Conference on Software Maintenance (ICSM’05), 2005, pp. 559–568. https://doi.org/10.1109/ICSM.2005.87
Krishnamoorthi R, Sahaaya Arul Mary SA (2009) Factor oriented requirement coverage based system test case prioritization of new and regression test cases. Inf Softw Technol 51(4):799–808. https://doi.org/10.1016/j.infsof.2008.08.007
Larsen L, Harrold MJ (1996) Slicing object-oriented software. In: Proceedings of the 18th International Conference on Software Engineering, ICSE ’96, IEEE Computer Society, Washington, DC, USA, pp. 495–505. http://dl.acm.org/citation.cfm?id=227726.227837
Lou Y, Hao D, Zhang L (2015) Mutation-based test-case prioritization in software evolution. In: IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). IEEE 2015:46–57
Malishevsky AG, Ruthruff JR, Rothermel G, Elbaum S (2006) Cost-cognizant test case prioritization. Tech rep
Mall R (2018) Fundamentals of software engineering. PHI Learning Pvt. Ltd., Delhi
Mirarab S, Tahvildari L (2008) An empirical study on bayesian network-based approach for test case prioritization. In: 2008 1st International Conference on Software Testing, Verification, and Validation, IEEE, pp. 278–287
muJava (2016) https://cs.gmu.edu/offutt/mujava/
Nguyen Q-V, Madeyski L (2014) Problems of mutation testing and higher order mutation testing. Adv Intell Syst Comput 282:157–172. https://doi.org/10.1007/978-3-319-06569-4_12
Panda N, Acharya AA, Mohapatra D (2019) Test scenario prioritization for object-oriented systems using uml diagram. Int J Syst Assur Eng Manag. https://doi.org/10.1007/s13198-019-00759-z
Panigrahi CR, Mall R (2010) Model-based regression test case prioritization. In: Prasad SK, Vin HM, Sahni S, Jaiswal MP, Thipakorn B (eds) Information systems. Technology and Management, Springer, Berlin Heidelberg, Berlin, Heidelberg, pp 380–385
Panigrahi CR, Mall R (2013) An approach to prioritize the regression test cases of object-oriented programs. CSI Transact ICT 1(2):159–173. https://doi.org/10.1007/s40012-013-0011-7
Panigrahi CR, Mall R (2013) A heuristic-based regression test case prioritization approach for object-oriented programs. Innovations Syst Softw Eng 10:155–163
Rothermel Harrold (1994) Selecting regression tests for object-oriented software. In: Proceedings 1994 International Conference on Software Maintenance, pp. 14–25. https://doi.org/10.1109/ICSM.1994.336793
Rothermel G, Elbaum S, Kinneer A, Do H, Software-artifact infrastructure repository, http://sir.unl.edu/portal
Rothermel G, Untch RH, Chengyun Chu, Harrold MJ (1999) Test case prioritization: an empirical study. In: Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM’99). Software Maintenance for Business Change (Cat. No.99CB36360), pp. 179–188
Rothermel G, Untch RH, C Chengyun, Harrold MJ (2001) Prioritizing test cases for regression testing. IEEE Transact Softw Eng 27(10):929–948. https://doi.org/10.1109/32.962562
Spieker H, Gotlieb A, Marijan D, Mossige M, Reinforcement learning for automatic test case prioritization and selection in continuous integration, arXiv preprint arXiv:1811.04122
Srikanth H, Williams L, Osborne J (2005) System test case prioritization of new and regression test cases. In: 2005 International Symposium on Empirical Software Engineering, 2005., p. 10. https://doi.org/10.1109/ISESE.2005.1541815
Tan P-N, Steinbach M, Kumar V (2016) Introduction to data mining. Pearson Education, India
Vescan A, Pintea C.-M, Pop PC, Test case prioritization–ant algorithm with faults severity, Logic Journal of the IGPL
Zhao X, Wang Z, Fan X, Wang Z (2015) A clustering-bayesian network based approach for test case prioritization. In: 2015 IEEE 39th Annual Computer Software and Applications Conference, Vol. 3, IEEE, pp. 542–547
Funding
No funding used.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Biswas, S., Rathi, R., Dutta, A. et al. A regression test case prioritization technique targeting ‘hard to detect’ faults. Int J Syst Assur Eng Manag 13, 1066–1081 (2022). https://doi.org/10.1007/s13198-021-01385-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13198-021-01385-4