skip to main content
10.1145/3001867.3001868acmconferencesArticle/Chapter ViewAbstractPublication PagesfosdConference Proceedingsconference-collections
research-article

Fine-grained test case prioritization for integration testing of delta-oriented software product lines

Published: 30 October 2016 Publication History

Abstract

Software product line (SPL) testing is a challenging task, due to the huge number of variants sharing common functionalities to be taken into account for efficient testing. By adopting the concept of regression testing, incremental SPL testing strategies cope with this challenge by exploiting the reuse potential of test artifacts between subsequent variants under test. In previous work, we proposed delta-oriented test case prioritization for incremental SPL integration testing, where differences between architecture test model variants allow for reasoning about the order of reusable test cases to be executed. However, the prioritization left two issues open, namely (1) changes to component behavior are ignored, which may also influence component interactions and, (2) the weighting and ordering of similar test cases result in an unintended clustering of test cases. In this paper, we extend the test case prioritization technique by (1) incorporating changes to component behavior allowing for a more fine-grained analysis and (2) defining a dissimilarity measure to avoid clustered test case orders. We prototyped our test case prioritization technique and evaluated its applicability and effectiveness by means of a case study from the automotive domain showing positive results.

References

[1]
M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-based prioritization in software productline testing. In SPLC, pages 197–206, 2014.
[2]
K. Androutsopoulos, D. Clark, M. Harman, J. Krinke, and L. Tratt. State-based Model Slicing: A Survey. CSUR, 45 (4):53:1–53:36, 2013.
[3]
H. Baller and M. Lochau. Towards Incremental Test Suite Optimization for Software Product Lines. In FOSD, pages 30–36. ACM, 2014.
[4]
H. Baller, S. Lity, M. Lochau, and I. Schaefer. Multi-objective test suite optimization for incremental product family testing. In ICST, pages 303–312, 2014.
[5]
A. Bertolino, P. Inverardi, H. Muccini, and A. Rosetti. An approach to integration testing based on architectural descriptions. In ICECCS, pages 77–84, 1997.
[6]
D. Clarke, M. Helvensteijn, and I. Schaefer. Abstract delta modeling. In GPCE, pages 13–22, 2010.
[7]
P. Da Mota Silveira Neto, I. do Carmo Machado, Y. Cavalcanti, E. de Almeida, V. Garcia, and S. de Lemos Meira. A Regression Testing Approach for Software Product Lines Architectures. In SBCARS, pages 41–50, 2010.
[8]
M. Dukaczewski, I. Schaefer, R. Lachmann, and M. Lochau. Requirements-based delta-oriented spl testing. In PLEASE, pages 49–52, 2013.
[9]
J. A. Duraes and H. S. Madeira. Emulation of software faults: A field data study and a practical approach. TSE, 32(11):849– 867, 2006.
[10]
E. Engström. Exploring Regression testing and software product line testing - research and state of practice. Lic dissertation, Lund University, May 2010.
[11]
E. Engström and P. Runeson. Software product line testing - a systematic mapping study. JIST, 53:2–13, 2011.
[12]
A. Ensan, E. Bagheri, M. Asadi, D. Gasevic, and Y. Biletskiy. Goal-oriented test case selection and prioritization for product line feature models. In ITNG, pages 291–298, 2011.
[13]
D. Harel and P. S. Thiagarajan. Message sequence charts. In In UML for Real: Design of Embedded Real-Time Systems, pages 77–105, 2003.
[14]
H. Hemmati, A. Arcuri, and L. Briand. Achieving scalable model-based testing through test case diversity. TOSEM, 22 (1):1–42, 2013.
[15]
M. F. Johansen, Ø. Haugen, and F. Fleurey. An algorithm for generating t-wise covering arrays from large feature models. In SPLC, pages 46–55, 2012.
[16]
R. Lachmann, S. Lity, S. Lischke, S. Beddig, S. Schulze, and I. Schaefer. Delta-oriented test case prioritization for integration testing of software product lines. In SPLC, pages 81–90, 2015.
[17]
S. Lity, R. Lachmann, M. Lochau, and I. Schaefer. Deltaoriented software product line test models - the body comfort system case study. Technical report, TU Braunschweig, 2013.
[18]
S. Lity, T. Morbach, T. Thüm, and I. Schaefer. Applying Incremental Model Slicing to Product-Line Regression Testing. In ICSR, 2016.
[19]
M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-based Testing of Delta-oriented Software Product Lines. In TAP, pages 67–82, 2012.
[20]
M. Lochau, S. Lity, R. Lachmann, I. Schaefer, and U. Goltz. Delta-oriented model-based integration testing of large-scale systems. JSS, 91:63–84, 2014.
[21]
R. E. Lopez-Herrejon, J. Javier Ferrer, F. Chicano, E. N. Haslinger, A. Egyed, and E. Alba. A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In GECCO, pages 1255–1262, 2014.
[22]
H. Muccini and A. van der Hoek. Towards Testing Product Line Architectures. ENTCS, 82(6):99 – 109, 2003.
[23]
S. Oster, M. Zink, M. Lochau, and M. Grechanik. Pairwise feature-interaction testing for spls: Potentials and limitations. In SPLC, pages 6:1–6:8, 2011.
[24]
G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Pairwise testing for software product lines: comparison of two approaches. SQJ, 20(3-4):605–643, 2012.
[25]
C. Pietsch, T. Kehrer, U. Kelter, D. Reuling, and M. Ohrndorf. Sipl a delta-based modeling framework for software product line engineering. In ASE, pages 852–857, 2015.
[26]
K. Pohl, G. Böckle, and F. van der Linden. Software Product Line Engineering. Springer, 2005.
[27]
X. Qu, M. Cohen, and K. Woolf. Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization. In ICSM, pages 255–264, 2007.
[28]
X. Qu, M. B. Cohen, and G. Rothermel. Configurationaware Regression Testing: An Empirical Study of Sampling and Prioritization. In ISSTA, pages 75–86, 2008.
[29]
S. Reis, A. Metzger, and K. Pohl. Integration Testing in Software product Line Engineering: A Model-Based Technique. In FASE, pages 321–335. Springer-Verlag, 2007.
[30]
G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. TSE, Vol.27 No.10: 929–948, 2001.
[31]
P. Runeson and E. Engström. Software product line testing - a 3d regression testing problem. In ICST, pages 742–746, 2012.
[32]
M. Utting and B. Legeard. Practical Model-based Testing. Morgan Kaufmann, 2007.
[33]
E. Uzuncaova, S. Khurshid, and D. Batory. Incremental Test Generation for Software Product Lines. TSE, 36(3):309–322, 2010.
[34]
M. Varshosaz, H. Beohar, and M. R. Mousavi. Delta-Oriented FSM-Based Testing. In ICFEM, volume 9407 of LNCS, pages 366–381. Springer, 2015.
[35]
S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. JSTVR, 22(2):67–120, 2007.

Cited By

View all
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • (2023)Coverage-Driven Test Automation for Highly-Configurable Railway SystemsProceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3571788.3571794(23-30)Online publication date: 25-Jan-2023
  • (2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
  • Show More Cited By

Index Terms

  1. Fine-grained test case prioritization for integration testing of delta-oriented software product lines

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    FOSD 2016: Proceedings of the 7th International Workshop on Feature-Oriented Software Development
    October 2016
    84 pages
    ISBN:9781450346474
    DOI:10.1145/3001867
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 30 October 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Delta-Oriented Software Product Lines
    2. Model-Based Integration Testing
    3. Test Case Prioritization

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    SPLASH '16
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 17 of 28 submissions, 61%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)14
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
    • (2023)Coverage-Driven Test Automation for Highly-Configurable Railway SystemsProceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3571788.3571794(23-30)Online publication date: 25-Jan-2023
    • (2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
    • (2022)Towards developer support for merging forked test casesProceedings of the 26th ACM International Systems and Software Product Line Conference - Volume A10.1145/3546932.3547002(131-141)Online publication date: 12-Sep-2022
    • (2022)Looking For Novelty in Search-Based Software Product Line TestingIEEE Transactions on Software Engineering10.1109/TSE.2021.305785348:7(2317-2338)Online publication date: 1-Jul-2022
    • (2020)Optimization of automated executions based on integration test configurations of embedded software2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW50294.2020.00064(358-363)Online publication date: Oct-2020
    • (2020)Determining Software Inter-Dependency Patterns for Integration Testing by applying Machine learning on Logs and Telemetry data2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO)10.1109/ICRITO48877.2020.9197868(1080-1084)Online publication date: Jun-2020
    • (2020)Automating system test case classification and prioritization for use case-driven testing in product linesEmpirical Software Engineering10.1007/s10664-020-09853-4Online publication date: 18-Aug-2020
    • (2018)Similarity analysis of product-line variantsProceedings of the 22nd International Systems and Software Product Line Conference - Volume 110.1145/3233027.3233044(226-235)Online publication date: 10-Sep-2018
    • (2017)Delta-oriented product prioritization for similarity-based product-line testingProceedings of the 2nd International Workshop on Variability and Complexity in Software Design10.5555/3106050.3106059(34-40)Online publication date: 20-May-2017
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media