skip to main content
10.1145/3379597.3387458acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article
Open access

RTPTorrent: An Open-source Dataset for Evaluating Regression Test Prioritization

Published: 18 September 2020 Publication History

Abstract

The software engineering practice of automated testing helps programmers find defects earlier during development. With growing software projects and longer-running test suites, frequency and immediacy of feedback decline, thereby making defects harder to repair. Regression test prioritization (RTP) is concerned with running relevant tests earlier to lower the costs of defect localization and to improve feedback.
Finding representative data to evaluate RTP techniques is non-trivial, as most software is published without failing tests. In this work, we systematically survey a wide range of RTP literature regarding whether their dataset uses real or synthetic defects or tests, whether they are publicly available, and whether datasets are reused. We observed that some datasets are reused, however, many projects study only few projects and these rarely resemble real-world development activity.
In light of these threats to ecological validity, we describe the construction and characteristics of a new dataset, named RTPTorrent, based on 20 open-source Java programs.
Our dataset allows researchers to evaluate prioritization heuristics based on version control meta-data, source code, and test results from fine-grained, automated builds over 9 years of development history. We provide reproducible baselines for initial comparisons and make all data publicly available.
We see this as a step towards better reproducibility, ecological validity, and long-term availability of studied software in the field of test prioritization.

References

[1]
Everton L. G. Alves, Patrícia D. L. Machado, Tiago Massoni, and Miryung Kim. 2016. Prioritizing test cases for early detection of refactoring faults. Softw. Test., Verif. Reliab. 26, 5 (2016), 402--426.
[2]
Everton L. G. Alves, Patrícia D. L. Machado, Tiago Massoni, and Samuel T. C. Santos. 2013. A refactoring-based approach for test case selection and prioritization. In AST. IEEE Computer Society, 93--99.
[3]
Md. Junaid Arafeen and Hyunsook Do. 2013. Test Case Prioritization Using Requirements-Based Clustering. In ICST. IEEE Computer Society, 312--321.
[4]
Maral Azizi and Hyunsook Do. 2018. A collaborative filtering recommender system for test case prioritization in web applications. In Proceedings of the 33rd Annual ACM Symposium on Applied Computing, SAC 2018, Pau, France, April 09-13, 2018. 1560--1567.
[5]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In Proceedings of the 14th International Conference on Mining Software Repositories, MSR 2017, Buenos Aires, Argentina, May 20-28, 2017. 447--450. https://doi.org/10.1109/MSR.2017.24
[6]
Árpád Beszédes, Tamás Gergely, Lajos Schrettner, Judit Jász, Laszlo Lango, and Tibor Gyimóthy. 2012. Code coverage-based regression test selection and prioritization in WebKit. In ICSM. IEEE Computer Society, 46--55.
[7]
Yi Bian, Serkan Kirbas, Mark Harman, Yue Jia, and Zheng Li. 2015. Regression Test Case Prioritisation for Guava. In SSBSE (Lecture Notes in Computer Science), Vol. 9275. Springer, 221--227.
[8]
Virginia Braun and Victoria Clarke. 2006. Using Thematic Analysis in Psychology. Qualitative research in psychology 3, 2 (2006), 77--101.
[9]
Benjamin Busjaeger and Tao Xie. 2016. Learning for test prioritization: an industrial case study. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016. 975--980.
[10]
Ryan Carlson, Hyunsook Do, and Anne Denton. 2011. A clustering approach to improving test case prioritization: An industrial case study. In ICSM. IEEE Computer Society, 382--391.
[11]
Junjie Chen, Yiling Lou, Lingming Zhang, Jianyi Zhou, Xiaoleng Wang, Dan Hao, and Lu Zhang. 2018. Optimizing test prioritization via test distribution analysis. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, November 04-09, 2018. 656--667.
[12]
Jinfu Chen, Lili Zhu, Tsong Yueh Chen, Rubing Huang, Dave Towey, Fei-Ching Kuo, and Yuchi Guo. 2016. An Adaptive Sequence Approach for OOS Test Case Prioritization. In ISSRE Workshops. IEEE Computer Society, 205--212.
[13]
Jinfu Chen, Lili Zhu, Tsong Yueh Chen, Dave Towey, Fei-Ching Kuo, Rubing Huang, and Yuchi Guo. 2018. Test case prioritization for object-oriented software: An adaptive random sequence approach based on clustering. Journal of Systems and Software 135 (2018), 107--125.
[14]
Pedro de Alcântara dos Santos Neto, Ricardo Britto, Thiago Soares, Werney Ayala, Jonathas Cruz, and Ricardo A. L. Rabêlo. 2012. Regression Testing Prioritization Based on Fuzzy Inference Systems. In SEKE. Knowledge Systems Institute Graduate School, 273--278.
[15]
Hyunsook Do, Sebastian G. Elbaum, and Gregg Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Software Engineering 10, 4 (2005), 405--435. https://doi.org/10.1007/s10664-005-3861-2
[16]
Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2010. The Effects of Time Constraints on Test Case Prioritization: A Series of Controlled Experiments. IEEE Trans. Software Eng. 36, 5 (2010), 593--617.
[17]
Hyunsook Do and Gregg Rothermel. 2005. A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults. In ICSM. IEEE Computer Society, 411--420.
[18]
Hyunsook Do and Gregg Rothermel. 2006. An empirical study of regression testing techniques incorporating context and lifetime factors and improved cost-benefit models. In SIGSOFT FSE. ACM, 141--151.
[19]
Hyunsook Do and Gregg Rothermel. 2006. On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques. IEEE Trans. Software Eng. 32, 9 (2006), 733--752.
[20]
Hyunsook Do and Gregg Rothermel. 2008. Using sensitivity analysis to create simplified economic models for regression testing. In ISSTA. ACM, 51--62.
[21]
Hyunsook Do, Gregg Rothermel, and Alex Kinneer. 2006. Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis. Empirical Software Engineering 11, 1 (2006), 33--70.
[22]
Sepehr Eghbali and Ladan Tahvildari. 2016. Test Case Prioritization Using Lexicographical Ordering. IEEE Trans. Software Eng. 42, 12 (2016), 1178--1195.
[23]
Sebastian Elbaum, Andrew Mclaughlin, and John Penix. 2014. The Google Dataset of Testing Results. https://code.google.com/p/google-shared-dataset-of-test-suite-results
[24]
Sebastian G. Elbaum, David Gable, and Gregg Rothermel. 2001. Understanding and Measuring the Sources of Variation in the Prioritization of Regression Test Suites. In IEEE METRICS. IEEE Computer Society, 169.
[25]
Sebastian G. Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. 2001. Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization. In ICSE. IEEE Computer Society, 329--338.
[26]
Sebastian G. Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. 2002. Test Case Prioritization: A Family of Empirical Studies. IEEE Trans. Software Eng. 28, 2 (2002), 159--182.
[27]
Emelie Engström, Per Runeson, and Andreas Ljung. 2011. Improving Regression Testing Transparency and Efficiency with History-Based Prioritization - An Industrial Case Study. In ICST. IEEE Computer Society, 367--376.
[28]
Michael G. Epitropakis, Shin Yoo, Mark Harman, and Edmund K. Burke. 2015. Empirical evaluation of pareto efficient multi-objective regression test case prioritisation. In Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, Baltimore, MD, USA, July 12-17, 2015. 234--245.
[29]
Chunrong Fang, Zhenyu Chen, Kun Wu, and Zhihong Zhao. 2014. Similarity-based test case prioritization using ordered sequences of program entities. Software Quality Journal 22, 2 (2014), 335--361.
[30]
Yalda Fazlalizadeh, Alireza Khalilian, Mohammad Abdollahi Azgomi, and Saeed Parsa. 2009. Incorporating Historical Test Case Performance Data and Resource Constraints into Test Case Prioritization. In Tests and Proofs, Third International Conference, TAP 2009, Zurich, Switzerland, July 2-3, 2009. Proceedings. 43--57.
[31]
Deepak Garg, Amitava Datta, and Tim French. 2013. A novel bipartite graph approach for selection and prioritisation of test cases. ACM SIGSOFT Software Engineering Notes 38, 6 (2013), 1--6.
[32]
Alberto González-Sanchez, Éric Piel, Rui Abreu, Hans-Gerhard Groß, and Arjan J. C. van Gemund. 2011. Prioritizing tests for software fault diagnosis. Softw., Pract. Exper. 41, 10 (2011), 1105--1129.
[33]
Georgios Gousios. 2013. The GHTorrent dataset and tool suite. In Proceedings of the 10th Working Conference on Mining Software Repositories (San Francisco, CA, USA) (MSR '13). IEEE Press, Piscataway, NJ, USA, 233--236. http://dl.acm.org/citation.cfm?id=2487085.2487132
[34]
Maria Grant and Andrew Booth. 2009. A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies. Health Information & Libraries Journal 26, 2 (2009), 91--108. https://doi.org/10.1111/j.1471-1842.2009.00848.x
[35]
Alireza Haghighatkhah, Mika M"antyl"a, Markku Oivo, and Pasi Kuvaja. 2018. Test prioritization in continuous integration environments. Journal of Systems and Software 146 (2018), 80--98.
[36]
Shifa-e-Zehra Haidry and Tim Miller. 2013. Using Dependency Structures for Prioritization of Functional Test Suites. IEEE Trans. Software Eng. 39, 2 (2013), 258--275.
[37]
Dan Hao, Lu Zhang, Lei Zang, Yanbo Wang, Xingxia Wu, and Tao Xie. 2016. To Be Optimal or Not in Test-Case Prioritization. IEEE Trans. Software Eng. 42, 5 (2016), 490--504.
[38]
Dan Hao, Lingming Zhang, Lu Zhang, Gregg Rothermel, and Hong Mei. 2014. A Unified Test Case Prioritization Approach. ACM Trans. Softw. Eng. Methodol. 24, 2 (2014), 10:1-10:31.
[39]
Dan Hao, Xu Zhao, and Lu Zhang. 2013. Adaptive Test-Case Prioritization Guided by Output Inspection. In COMPSAC. IEEE Computer Society, 169--179.
[40]
Ramzi A. Haraty, Nashat Mansour, Lama Moukahal, and Iman Khalil. 2016. Regression Test Cases Prioritization Using Clustering and Code Change Relevance. International Journal of Software Engineering and Knowledge Engineering 26, 5 (2016), 733--768.
[41]
Christopher Henard, Mike Papadakis, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016. 523--534.
[42]
Charitha Hettiarachchi, Hyunsook Do, and Byoungju Choi. 2016. Risk-based test case prioritization using a fuzzy expert system. Information & Software Technology 69 (2016), 1--15.
[43]
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 197--207.
[44]
Shan-Shan Hou, Lu Zhang, Tao Xie, and Jiasu Sun. 2008. Quota-constrained test-case prioritization for regression testing of service-centric systems. In ICSM. IEEE Computer Society, 257--266.
[45]
Yu-Chi Huang, Chin-Yu Huang, Jun-Ru Chang, and Tsan-Yuan Chen. 2010. Design and Analysis of Cost-Cognizant Test Case Prioritization Using Genetic Algorithm with Test History. In COMPSAC. IEEE Computer Society, 413--418.
[46]
Yu-Chi Huang, Kuan-Li Peng, and Chin-Yu Huang. 2012. A history-based cost-cognizant test case prioritization technique in regression testing. Journal of Systems and Software 85, 3 (2012), 626--637.
[47]
M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. 1994. Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria. In Proceedings of 16th International Conference on Software Engineering. 191--200. https://doi.org/10.1109/ICSE.1994.296778
[48]
Dennis Jeffrey and Neelam Gupta. 2006. Test Case Prioritization Using Relevant Slices. In COMPSAC (1). IEEE Computer Society, 411--420.
[49]
Dennis Jeffrey and Neelam Gupta. 2008. Experiments with test case prioritization using relevant slices. Journal of Systems and Software 81, 2 (2008), 196--221.
[50]
Bo Jiang and W. K. Chan. 2010. On the Integration of Test Adequacy, Test Case Prioritization, and Statistical Fault Localization. In QSIC. IEEE Computer Society, 377--384.
[51]
Bo Jiang and Wing Kwong Chan. 2015. Input-based adaptive randomized test case prioritization: A local beam search approach. Journal of Systems and Software 105 (2015), 91--106.
[52]
Bo Jiang, Wing Kwong Chan, and T. H. Tse. 2015. PORA: Proportion-Oriented Randomized Algorithm for Test Case Prioritization. In QRS. IEEE, 131--140.
[53]
Bo Jiang, Zhenyu Zhang, Wing Kwong Chan, and T. H. Tse. 2009. Adaptive Random Test Case Prioritization. In ASE. IEEE Computer Society, 233--244.
[54]
James A. Jones and Mary Jean Harrold. 2003. Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage. IEEE Trans. Software Eng. 29, 3 (2003), 195--209.
[55]
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In International Symposium on Software Testing and Analysis, ISSTA '14, San Jose, CA, USA - July 21 - 26, 2014. 437--440. https://doi.org/10.1145/2610384.2628055
[56]
René Just, Gregory M. Kapfhammer, and Franz Schweiggert. 2012. Using Non-redundant Mutation Operators and Test Suite Prioritization to Achieve Efficient and Scalable Mutation Analysis. In ISSRE. IEEE Computer Society, 11--20.
[57]
Jeongho Kim, Hohyeon Jeong, and Eunseok Lee. 2017. Failure history data-based test case prioritization for effective regression test. In Proceedings of the Symposium on Applied Computing, SAC 2017, Marrakech, Morocco, April 3-7, 2017. 1409--1415.
[58]
Jung-Min Kim and Adam A. Porter. 2002. A history-based test prioritization technique for regression testing in resource constrained environments. In ICSE. ACM, 119--129.
[59]
Sejun Kim and Jongmoon Baik. 2010. An effective fault aware test case prioritization by incorporating a fault localization technique. In ESEM. ACM.
[60]
Bogdan Korel and George Koutsogiannakis. 2009. Experimental Comparison of Code-Based and Model-Based Test Prioritization. In ICST Workshops. IEEE Computer Society, 77--84.
[61]
Bogdan Korel, George Koutsogiannakis, and Luay Ho Tahat. 2008. Application of system models in regression test suite prioritization. In ICSM. IEEE Computer Society, 247--256.
[62]
Bogdan Korel, Luay Ho Tahat, and Mark Harman. 2005. Test Prioritization Using System Models. In ICSM. IEEE Computer Society, 559--568.
[63]
R. Krishnamoorthi and S. A. Sahaaya Arul Mary. 2009. Requirement Based System Test Case Prioritization of New and Regression Test Cases. International Journal of Software Engineering and Knowledge Engineering 19, 3 (2009), 453--475.
[64]
Jung-Hyun Kwon, In-Young Ko, Gregg Rothermel, and Matt Staats. 2014. Test Case Prioritization Based on Information Retrieval Concepts. In APSEC (1). IEEE Computer Society, 19--26.
[65]
Yves Ledru, Alexandre Petrenko, and Sergiy Boroday. 2009. Using String Distances for Test Case Prioritisation. In ASE. IEEE Computer Society, 510--514.
[66]
Yves Ledru, Alexandre Petrenko, Sergiy Boroday, and Nadine Mandran. 2012. Prioritizing test cases with string distances. Autom. Softw. Eng. 19, 1 (2012), 65--95.
[67]
Zheng Li, Yi Bian, Ruilian Zhao, and Jun Cheng. 2013. A Fine-Grained Parallel Multi-objective Test Case Prioritization on GPU. In SSBSE (Lecture Notes in Computer Science), Vol. 8084. Springer, 111--125.
[68]
Jingjing Liang, Sebastian G. Elbaum, and Gregg Rothermel. 2018. Redefining prioritization: continuous prioritization for continuous integration. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018. 688--698.
[69]
Yiling Lou, Junjie Chen, Lingming Zhang, and Dan Hao. 2019. Chapter One - A Survey on Regression Test-Case Prioritization. In Advances in Computers, Atif M. Memon (Ed.). Vol. 113. Elsevier, 1--46. https://doi.org/10.1016/bs.adcom.2018.10.001
[70]
Yiling Lou, Dan Hao, and Lu Zhang. 2015. Mutation-based test-case prioritization in software evolution. In ISSRE. IEEE Computer Society, 46--57.
[71]
Yafeng Lu, Yiling Lou, Shiyang Cheng, Lingming Zhang, Dan Hao, Yangfan Zhou, and Lu Zhang. 2016. How does regression test prioritization perform in real-world software evolution?. In Proceedings of the 38th International Conference on Software Engineering, ICSE 2016, Austin, TX, USA, May 14-22, 2016. 535--546.
[72]
Qi Luo, Kevin Moran, and Denys Poshyvanyk. 2016. A large-scale empirical comparison of static and dynamic test case prioritization techniques. In Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, Seattle, WA, USA, November 13-18, 2016. 559--570.
[73]
Junpeng Lv, Bei-Bei Yin, and Kai-Yuan Cai. 2013. On the Gain of Measuring Test Case Prioritization. In COMPSAC. IEEE Computer Society, 627--632.
[74]
Zengkai Ma and Jianjun Zhao. 2008. Test Case Prioritization Based on Analysis of Program Structure. In APSEC. IEEE Computer Society, 471--478.
[75]
Camila Loiola Brito Maia, Rafael Augusto Ferreira do Carmo, Fabricio Gomes de Freitas, Gustavo Augusto Lima de Campos, and Jerffeson Teixeira de Souza. 2010. Automated Test Case Prioritization with Reactive GRASP. Adv. Software Engineering 2010 (2010), 428521:1-428521:18.
[76]
Ruchika Malhotra and Divya Tiwari. 2013. Development of a framework for test case prioritization using genetic algorithm. ACM SIGSOFT Software Engineering Notes 38, 3 (2013), 1--6.
[77]
Christoph Malz, Nasser Jazdi, and Peter Göhner. 2012. Prioritization of Test Cases Using Software Agents and Fuzzy Logic. In ICST. IEEE Computer Society, 483--486.
[78]
Dusica Marijan. 2015. Multi-perspective Regression Test Prioritization for Time-Constrained Environments. In QRS. IEEE, 157--162.
[79]
Dusica Marijan, Arnaud Gotlieb, and Sagar Sen. 2013. Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study. In ICSM. IEEE Computer Society, 540--543.
[80]
S. A. Sahaaya Arul Mary and R. Krishnamoorthi. 2011. Time-Aware and Weighted Fault Severity Based Metrics for Test Case Prioritization. International Journal of Software Engineering and Knowledge Engineering 21, 1 (2011), 129--142.
[81]
Wes Masri and Marwa El-Ghali. 2009. Test case filtering and prioritization based on coverage of combinations of program elements. In WODA. 29--34.
[82]
Hong Mei, Dan Hao, Lingming Zhang, Lu Zhang, Ji Zhou, and Gregg Rothermel. 2012. A Static Approach to Prioritizing JUnit Test Cases. IEEE Trans. Software Eng. 38, 6 (2012), 1258--1275.
[83]
Breno Miranda, Emilio Cruciani, Roberto Verdecchia, and Antonia Bertolino. 2018. FAST approaches to scalable similarity-based test case prioritization. In Proceedings of the 40th International Conference on Software Engineering, ICSE 2018, Gothenburg, Sweden, May 27 - June 03, 2018. 222--232.
[84]
Daniel Di Nardo, Nadia Alshahwan, Lionel C. Briand, and Yvan Labiche. 2013. Coverage-Based Test Case Prioritisation: An Industrial Case Study. In ICST. IEEE Computer Society, 302--311.
[85]
Tanzeem Bin Noor and Hadi Hemmati. 2015. A similarity-based approach for test case prioritization using historical failure data. In ISSRE. IEEE Computer Society, 58--68.
[86]
Dario Di Nucci, Annibale Panichella, Andy Zaidman, and Andrea De Lucia. 2015. Hypervolume-Based Search for Test Case Prioritization. In SSBSE (Lecture Notes in Computer Science), Vol. 9275. Springer, 157--172.
[87]
Subhrakanta Panda, Dishant Munjal, and Durga Prasad Mohapatra. 2016. A Slice-Based Change Impact Analysis for Regression Test Case Prioritization of Object-Oriented Programs. Adv. Software Engineering 2016 (2016), 7132404:1-7132404:20.
[88]
Chhabi Rani Panigrahi and Rajib Mall. 2010. Model-based regression test case prioritization. ACM SIGSOFT Software Engineering Notes 35, 6 (2010), 1--7.
[89]
Jos'e Antonio Parejo, Ana B. S'anchez, Sergio Segura, Antonio Ruiz Cort'es, Roberto E. Lopez-Herrejon, and Alexander Egyed. 2016. Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems and Software 122 (2016), 287--310.
[90]
Hyuncheol Park, Hoyeon Ryu, and Jongmoon Baik. 2008. Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing. In SSIRI. IEEE Computer Society, 39--46.
[91]
Xiao Qu and Myra B. Cohen. 2013. A Study in Prioritization for Higher Strength Combinatorial Testing. In ICST Workshops. IEEE Computer Society, 285--294.
[92]
Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 1999. Test Case Prioritization: An Empirical Study. In ICSM. IEEE Computer Society, 179--188.
[93]
Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. Prioritizing Test Cases For Regression Testing. IEEE Trans. Software Eng. 27, 10 (2001), 929--948.
[94]
Ripon K. Saha, Lingming Zhang, Sarfraz Khurshid, and Dewayne E. Perry. 2015. An Information Retrieval Approach for Regression Test Prioritization Based on Program Changes. In ICSE (1). IEEE Computer Society, 268--279.
[95]
Sreedevi Sampath, Renée C. Bryce, Gokulanand Viswanath, Vani Kandimalla, and Akif Günes Koru. 2008. Prioritizing User-Session-Based Test Cases for Web Applications Testing. In ICST. IEEE Computer Society, 141--150.
[96]
Amanda Schwartz and Hyunsook Do. 2016. Cost-effective regression testing through Adaptive Test Prioritization strategies. Journal of Systems and Software 115 (2016), 61--81.
[97]
Mark Sherriff, Mike Lake, and Laurie Williams. 2007. Prioritization of Regression Tests using Singular Value Decomposition with Empirical Change Records. In ISSRE. IEEE Computer Society, 81--90.
[98]
Cristian Simons and Emerson Cabrera Paraiso. 2010. Regression test cases prioritization using Failure Pursuit Sampling. In ISDA. IEEE, 923--928.
[99]
Adam M. Smith, Joshua Geiger, Gregory M. Kapfhammer, and Mary Lou Soffa. 2007. Test suite reduction and prioritization with call trees. In ASE. ACM, 539--540.
[100]
Adam M. Smith and Gregory M. Kapfhammer. 2009. An empirical study of incorporating cost into test suite reduction and prioritization. In Proceedings of the 2009 ACM Symposium on Applied Computing (SAC), Honolulu, Hawaii, USA, March 9-12, 2009. 461--467.
[101]
Helge Spieker, Arnaud Gotlieb, Dusica Marijan, and Morten Mossige. 2017. Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, Santa Barbara, CA, USA, July 10-14, 2017. 12--22.
[102]
Hema Srikanth and Sean Banerjee. 2012. Improving test efficiency through system test prioritization. Journal of Systems and Software 85, 5 (2012), 1176--1187.
[103]
Hema Srikanth, Sean Banerjee, Laurie Williams, and Jason A. Osborne. 2014. Towards the prioritization of system test cases. Softw. Test., Verif. Reliab. 24, 4 (2014), 320--337.
[104]
Amitabh Srivastava and Jay Thiagarajan. 2002. Effectively prioritizing tests in development environment. In ISSTA. ACM, 97--106.
[105]
Praveen Ranjan Srivastava, Krishan Kumar, and G. Raghurama. 2008. Test case prioritization based on requirements and risk factors. ACM SIGSOFT Software Engineering Notes 33, 4 (2008).
[106]
Matt Staats, Pablo Loyola, and Gregg Rothermel. 2012. Oracle-Centric Test Case Prioritization. In ISSRE. IEEE Computer Society, 311--320.
[107]
Heiko Stallbaum, Andreas Metzger, and Klaus Pohl. 2008. An Automated Technique for Risk-based Test Case Generation and Prioritization. In AST. ACM, 67--70.
[108]
Per Erik Strandberg, Daniel Sundmark, Wasif Afzal, Thomas J. Ostrand, and Elaine J. Weyuker. 2016. Experience Report: Automated System Level Regression Test Prioritization Using Multiple Factors. In ISSRE. IEEE Computer Society, 12--23.
[109]
Luay Ho Tahat, Bogdan Korel, Mark Harman, and Hasan Ural. 2012. Regression test suite prioritization using system models. Softw. Test., Verif. Reliab. 22, 7 (2012), 481--506.
[110]
Stephen W. Thomas, Hadi Hemmati, Ahmed E. Hassan, and Dorothea Blostein. 2014. Static test case prioritization using topic models. Empirical Software Engineering 19, 1 (2014), 182--212.
[111]
Paolo Tonella, Paolo Avesani, and Angelo Susi. 2006. Using the Case-Based Ranking Methodology for Test Case Prioritization. In ICSM. IEEE Computer Society, 123--133.
[112]
Filippos I. Vokolos and Phyllis G. Frankl. 1998. Empirical Evaluation of the Textual Differencing Regression Testing Technique. In 1998 International Conference on Software Maintenance, ICSM 1998, Bethesda, Maryland, USA, November 16-19, 1998. 44--53. https://doi.org/10.1109/ICSM.1998.738488
[113]
Kristen R. Walcott, Mary Lou Soffa, Gregory M. Kapfhammer, and Robert S. Roos. 2006. TimeAware test suite prioritization. In ISSTA. ACM, 1--12.
[114]
Rongcun Wang, Shujuan Jiang, and Deng Chen. 2015. Similarity-based regression test case prioritization. In SEKE. KSI Research Inc. and Knowledge Systems Institute Graduate School, 358--363.
[115]
Shuai Wang, Shaukat Ali, Tao Yue, Øyvind Bakkeli, and Marius Liaaen. 2016. Enhancing test case prioritization in an industrial setting with resource awareness and multi-objective search. In ICSE (Companion Volume). ACM, 182--191.
[116]
Song Wang, Jaechang Nam, and Lin Tan. 2017. QTEP: quality-aware test case prioritization. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. 523--534.
[117]
Ying Wang, Zhiliang Zhu, Bo Yang, Fangda Guo, and Hai Yu. 2018. Using reliability risk analysis to prioritize test cases. Journal of Systems and Software 139 (2018), 14--31.
[118]
W. Eric Wong, Joseph Robert Horgan, Aditya P. Mathur, and Alberto Pasquini. 1997. Test Set Size Minimization and Fault Detection Effectiveness: A Case Study in a Space Application. In COMPSAC. IEEE Computer Society, 522--528.
[119]
Kun Wu, Chunrong Fang, Zhenyu Chen, and Zhihong Zhao. 2012. Test case prioritization incorporating ordered sequence of program elements. In AST. IEEE Computer Society, 124--130.
[120]
Shin Yoo, Mark Harman, Paolo Tonella, and Angelo Susi. 2009. Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA 2009, Chicago, IL, USA, July 19-23, 2009. 201--212.
[121]
Hoijin Yoon and Byoungju Choi. 2011. A Test Case Prioritization Based on Degree of Risk Exposure and its Empirical Study. International Journal of Software Engineering and Knowledge Engineering 21, 2 (2011), 191--209.
[122]
Dongjiang You, Zhenyu Chen, Baowen Xu, Bin Luo, and Chen Zhang. 2011. An empirical study on the effectiveness of time-aware test case prioritization techniques. In Proceedings of the 2011 ACM Symposium on Applied Computing (SAC), TaiChung, Taiwan, March 21-24, 2011. 1451--1456.
[123]
Fang Yuan, Yi Bian, Zheng Li, and Ruilian Zhao. 2015. Epistatic Genetic Algorithm for Test Case Prioritization. In SSBSE (Lecture Notes in Computer Science), Vol. 9275. Springer, 109--124.
[124]
Chaoqiang Zhang, Alex Groce, and Mohammad Amin Alipour. 2014. Using test case reduction and prioritization to improve symbolic execution. In International Symposium on Software Testing and Analysis, ISSTA '14, San Jose, CA, USA - July 21-26, 2014. 160--170.
[125]
Lingming Zhang, Dan Hao, Lu Zhang, Gregg Rothermel, and Hong Mei. 2013. Bridging the gap between the total and additional test-case prioritization strategies. In 35th International Conference on Software Engineering, ICSE '13, San Francisco, CA, USA, May 18-26, 2013. 192--201.
[126]
Lu Zhang, Shan-Shan Hou, Chao Guo, Tao Xie, and Hong Mei. 2009. Time-aware test-case prioritization using integer linear programming. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA 2009, Chicago, IL, USA, July 19-23, 2009. 213--224.
[127]
Lingming Zhang, Ji Zhou, Dan Hao, Lu Zhang, and Hong Mei. 2009. Prioritizing JUnit test cases in absence of coverage information. In ICSM. IEEE Computer Society, 19--28.
[128]
Xiaofang Zhang, Tsong Yueh Chen, and Huai Liu. 2014. An Application of Adaptive Random Sequence in Test Case Prioritization. In SEKE. Knowledge Systems Institute Graduate School, 126--131.
[129]
Xiaofang Zhang, Xiaoyuan Xie, and Tsong Yueh Chen. 2016. Test Case Prioritization Using Adaptive Random Sequence with Category-Partition-Based Distance. In QRS. IEEE, 374--385.

Cited By

View all
  • (2025)A systematic literature review on Agile, Cloud, and DevOps integrationInformation and Software Technology10.1016/j.infsof.2024.107569177:COnline publication date: 1-Jan-2025
  • (2024)Faster Feedback with AI? A Test Prioritization StudyCompanion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming10.1145/3660829.3660837(32-40)Online publication date: 11-Mar-2024
  • (2024)Hierarchy-Aware Regression Test Prioritization2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00041(343-354)Online publication date: 28-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MSR '20: Proceedings of the 17th International Conference on Mining Software Repositories
June 2020
675 pages
ISBN:9781450375177
DOI:10.1145/3379597
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Dataset
  2. GitHub
  3. Java
  4. Regression Test Prioritization
  5. TravisCI

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

MSR '20
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)193
  • Downloads (Last 6 weeks)15
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)A systematic literature review on Agile, Cloud, and DevOps integrationInformation and Software Technology10.1016/j.infsof.2024.107569177:COnline publication date: 1-Jan-2025
  • (2024)Faster Feedback with AI? A Test Prioritization StudyCompanion Proceedings of the 8th International Conference on the Art, Science, and Engineering of Programming10.1145/3660829.3660837(32-40)Online publication date: 11-Mar-2024
  • (2024)Hierarchy-Aware Regression Test Prioritization2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00041(343-354)Online publication date: 28-Oct-2024
  • (2024)Agile Regression Testing2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00054(457-459)Online publication date: 27-May-2024
  • (2024)Contrasting test selection, prioritization, and batch testing at scaleEmpirical Software Engineering10.1007/s10664-024-10589-830:1Online publication date: 29-Nov-2024
  • (2024)Regression test prioritization leveraging source code similarity with tree kernelsJournal of Software: Evolution and Process10.1002/smr.2653Online publication date: 15-Feb-2024
  • (2023)Scalable and Accurate Test Case Prioritization in Continuous Integration ContextsIEEE Transactions on Software Engineering10.1109/TSE.2022.318484249:4(1615-1639)Online publication date: 1-Apr-2023
  • (2023)PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00021(57-61)Online publication date: May-2023
  • (2023)Test Case Prioritization using Transfer Learning in Continuous Integration Environments2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00023(191-200)Online publication date: May-2023
  • (2022)ReCoverProceedings of the 19th International Conference on Mining Software Repositories10.1145/3524842.3528490(196-200)Online publication date: 23-May-2022
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media