skip to main content
10.1145/3555228.3555231acmotherconferencesArticle/Chapter ViewAbstractPublication PagessbesConference Proceedingsconference-collections
research-article

A Configurable Test Case Prioritization Technique for Early Fault Detection and Low Test Case Spreading

Published: 05 October 2022 Publication History

Abstract

Developers often use test suites as safety nets to avoid functionality regression. However, regression testing can be costly and time-consuming. Test case prioritization (TCP) techniques try to reduce this burden by reordering the tests of a given suite to achieve a certain testing goal. The literature presents a great number of TCP techniques. Most works evaluate the performance of TCP techniques by using the rate of test cases that fail per fault (APFD). However, other aspects can be considered when evaluating prioritization results. For instance, the ability to reduce the spreading of failing test cases (M-Spreading), since a better grouping often provides more information regarding faults. This paper proposes Additional-Spreading, a configurable test case prioritization strategy that allows testers to favor either APFD or M-Spreading. In a study with open-source projects we evidence the configuration power of Additional-Spreading and that it can provide results similar to two traditional techniques (Total and Additional) when properly configured.

References

[1]
Everton LG Alves, Patrícia DL Machado, Tiago Massoni, and Miryung Kim. 2016. Prioritizing test cases for early detection of refactoring faults. Software Testing, Verification and Reliability 26, 5(2016), 402–426.
[2]
Everton L.G. Alves, Tiago Massoni, and Patrícia Duarte de Lima Machado. 2017. Test coverage of impacted code elements for detecting refactoring faults: An exploratory study. Journal of Systems and Software 123 (2017), 223 – 238. https://doi.org/10.1016/j.jss.2016.02.001
[3]
Ana Emília Victor Barbosa Coutinho, Emanuela Gadelha Cartaxo, and Patrícia Duarte de Lima Machado. 2016. Analysis of distance functions for similarity-based test suite reduction in the context of model-based testing. Software Quality Journal 24, 2 (2016), 407–445.
[4]
Thomaz Diniz, Everton LG Alves, Anderson GF Silva, and Wilkerson L Andrade. 2019. Reducing the Discard of MBT Test Cases using Distance Functions. In Proceedings of the XXXIII Brazilian Symposium on Software Engineering. 337–346.
[5]
Hyunsook Do, Siavash Mirarab, Ladan Tahvildari, and Gregg Rothermel. 2010. The effects of time constraints on test case prioritization: A series of controlled experiments. IEEE Transactions on Software Engineering 36, 5 (2010), 593–617.
[6]
Martin Fowler. 2018. Refactoring - Improving the Design of Existing Code (2 ed.). Addison-Wesley.
[7]
Dan Hao, Lu Zhang, and Hong Mei. 2016. Test-case prioritization: achievements and challenges. Frontiers of Computer Science 10, 5 (2016), 769–777.
[8]
Dan Hao, Lu Zhang, Lei Zang, Yanbo Wang, Xingxia Wu, and Tao Xie. 2016. To be optimal or not in test-case prioritization. IEEE Transactions on Software Engineering 42, 5 (2016), 490–505.
[9]
Anna Huang. 2008. Similarity measures for text document clustering. In Proceedings of the sixth new zealand computer science research student conference (NZCSRSC2008), Christchurch, New Zealand, Vol. 4. 9–56.
[10]
Rubing Huang, Yunan Zhou, Weiwen Zong, Dave Towey, and Jinfu Chen. 2017. An Empirical Comparison of Similarity Measures for Abstract Test Case Prioritization. In Computer Software and Applications Conference (COMPSAC), 2017 IEEE 41st Annual, Vol. 1. IEEE, 3–12.
[11]
Bo Jiang, Zhenyu Zhang, Wing Kwong Chan, and TH Tse. 2009. Adaptive random test case prioritization. In Proceedings of the 2009 IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 233–244.
[12]
B. Jiang, Z. Zhang, T. H. Tse, and T. Y. Chen. 2009. How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization. In 2009 33rd Annual IEEE International Computer Software and Applications Conference, Vol. 1. 99–106. https://doi.org/10.1109/COMPSAC.2009.23
[13]
René Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. ACM, 437–440.
[14]
Muhammad Khatibsyarbini, Mohd Adham Isa, Dayang N.A. Jawawi, and Rooster Tumeng. 2018. Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology 93 (2018), 74 – 93. https://doi.org/10.1016/j.infsof.2017.08.014
[15]
Vladimir Iosifovich Levenshtein. 1966. Binary codes capable of correcting deletions, insertions and reversals.Soviet Physics Doklady 10, 8 (feb 1966), 707–710. Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.
[16]
Zheng Li, Mark Harman, and Robert M Hierons. 2007. Search algorithms for regression test case prioritization. IEEE Transactions on software engineering 33, 4 (2007).
[17]
Yafeng Lu, Yiling Lou, Shiyang Cheng, Lingming Zhang, Dan Hao, Yangfan Zhou, and Lu Zhang. 2016. How does regression test prioritization perform in real-world software evolution?. In Software Engineering (ICSE), 2016 IEEE/ACM 38th International Conference on. IEEE, 535–546.
[18]
Andre Hasudungan Lubis, Ali Ikhwan, and Phak Len Eh Kan. 2018. Combination of levenshtein distance and rabin-karp to improve the accuracy of document equivalence level. International Journal of Engineering & Technology 7, 2.27(2018), 17–21.
[19]
Qi Luo, Kevin Moran, and Denys Poshyvanyk. 2016. A large-scale empirical comparison of static and dynamic test case prioritization techniques. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 559–570.
[20]
Suphakit Niwattanakul, Jatsada Singthongchai, Ekkachai Naenudorn, and Supachanun Wanapu. 2013. Using of Jaccard coefficient for keywords similarity. In Proceedings of the international multiconference of engineers and computer scientists, Vol. 1. 380–384.
[21]
Francisco G Oliveira Neto, Richard Torkar, and Patrícia DL Machado. 2016. Full modification coverage through automatic similarity-based test case selection. Information and Software Technology 80 (2016), 124–137.
[22]
R Pradeepa and Vimala K Devi. 2013. Effectiveness of Testcase Prioritization using APFD Metric: Survey. In IJCA Proceedings on International Conference on Research Trends in Computer Technologies. 1–4.
[23]
Gregg Rothermel and Mary Jean Harrold. 1997. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology (TOSEM) 6, 2(1997), 173–210.
[24]
Gregg Rothermel, Roland H Untch, Chengyun Chu, and Mary Jean Harrold. 1999. Test case prioritization: An empirical study. In Software Maintenance, 1999.(ICSM’99) Proceedings. IEEE International Conference on. IEEE, 179–188.
[25]
Gregg Rothermel, Roland H. Untch, Chengyun Chu, and Mary Jean Harrold. 2001. Prioritizing test cases for regression testing. IEEE Transactions on software engineering 27, 10 (2001), 929–948.
[26]
Thomas A Runkler and James C Bezdek. 2000. Automatic keyword extraction with relational clustering and Levenshtein distances. In Ninth IEEE International Conference on Fuzzy Systems. FUZZ-IEEE 2000 (Cat. No. 00CH37063), Vol. 2. IEEE, 636–640.
[27]
Yogesh Singh, Arvinder Kaur, Bharti Suri, and Shweta Singhal. 2012. Systematic literature review on regression test prioritization techniques. Informatica 36, 4 (2012).
[28]
Wesley NM Torres, Everton LG Alves, and Patrícia DL Machado. 2019. An Empirical Study on the Spreading of Fault Revealing Test Cases in Prioritized Suites. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 129–138.
[29]
Shin Yoo and Mark Harman. 2012. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability 22, 2(2012), 67–120.
[30]
Shin Yoo, Mark Harman, and David Clark. 2013. Fault localization prioritization: Comparing information-theoretic and coverage-based approaches. ACM Transactions on Software Engineering and Methodology (TOSEM) 22, 3(2013), 19.
[31]
Lingming Zhang, Dan Hao, Lu Zhang, Gregg Rothermel, and Hong Mei. 2013. Bridging the gap between the total and additional test-case prioritization strategies. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 192–201.
[32]
Lingming Zhang, Miryung Kim, and Sarfraz Khurshid. 2012. FaultTracer: a change impact and regression fault analysis tool for evolving Java programs. In Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 40.

Index Terms

  1. A Configurable Test Case Prioritization Technique for Early Fault Detection and Low Test Case Spreading

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SBES '22: Proceedings of the XXXVI Brazilian Symposium on Software Engineering
    October 2022
    457 pages
    ISBN:9781450397353
    DOI:10.1145/3555228
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 05 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. APFD
    2. M-Spreading
    3. metric
    4. prioritization
    5. test case

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    SBES 2022
    SBES 2022: XXXVI Brazilian Symposium on Software Engineering
    October 5 - 7, 2022
    Virtual Event, Brazil

    Acceptance Rates

    Overall Acceptance Rate 147 of 427 submissions, 34%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 64
      Total Downloads
    • Downloads (Last 12 months)9
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 15 Feb 2025

    Other Metrics

    Citations

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media