skip to main content
10.1145/2791060.2791073acmotherconferencesArticle/Chapter ViewAbstractPublication PagessplcConference Proceedingsconference-collections
research-article

Delta-oriented test case prioritization for integration testing of software product lines

Published: 20 July 2015 Publication History

Abstract

Software product lines have potential to allow for mass customization of products. Unfortunately, the resulting, vast amount of possible product variants with commonalities and differences leads to new challenges in software testing. Ideally, every product variant should be tested, especially in safety-critical systems. However, due to the exponentially increasing number of product variants, testing every product variant is not feasible. Thus, new concepts and techniques are required to provide efficient SPL testing strategies exploiting the commonalities of software artifacts between product variants to reduce redundancy in testing. In this paper, we present an efficient integration testing approach for SPLs based on delta modeling. We focus on test case prioritization. As a result, only the most important test cases for every product variant are tested, reducing the number of executed test cases significantly, as testing can stop at any given point because of resource constraints while ensuring that the most important test cases have been covered. We present the general concept and our evaluation results. The results show a measurable reduction of executed test cases compared to single-software testing approaches.

References

[1]
ISO/DIS 26262-1 - Road vehicles - Functional safety, 2009.
[2]
M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-based prioritization in software product-line testing. SPLC '14, pages 197--206. ACM, 2014.
[3]
S. Amland. Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. J. Syst. Softw., 53: 287--295, 2000.
[4]
H. Baller, S. Lity, M. Lochau, and I. Schaefer. Multi-objective test suite optimization for incremental product family testing. In ICST'14, pages 303--312, 2014.
[5]
A. Bertolino, P. Inverardi, H. Muccini, and A. Rosetti. An approach to integration testing based on architectural descriptions. ICECCS'97, pages 77--84, 1997.
[6]
D. Clarke, M. Helvensteijn, and I. Schaefer. Abstract delta modeling. In GPCE 2010, pages 13--22, 2010.
[7]
P. C. Clements. A survey of architecture description languages. International Workshop on Software Specification and Design, 8:16--25, 1996.
[8]
M. Dukaczewski, I. Schaefer, R. Lachmann, and M. Lochau. Requirements-based delta-oriented spl testing. PLEASE'13, pages 49--52, 2013.
[9]
E. Engström and P. Runeson. Software product line testing - a systematic mapping study. Information and Software Technology, 53:2--13, 2011.
[10]
A. Ensan, E. Bagheri, M. Asadi, D. Gasevic, and Y. Biletskiy. Goal-oriented test case selection and prioritization for product line feature models. ITNG'11, pages 291--298, 2011.
[11]
D. Harel and P. S. Thiagarajan. Message sequence charts. In In UML for Real: Design of Embedded Real-Time Systems, pages 77--105, 2003.
[12]
M. J. Harrold. Testing: A roadmap. In Proceedings of the Conference on The Future of Software Engineering, ICSE '00, pages 61--72. ACM, 2000.
[13]
M. F. Johansen, Ø. Haugen, and F. Fleurey. An algorithm for generating t-wise covering arrays from large feature models. SPLC'12, pages 46--55, 2012.
[14]
C. Kaner. What is a good test case? In STAR'03, 2003.
[15]
S. Lity, R. Lachmann, M. Lochau, and I. Schaefer. Delta-oriented software product line test models - the body comfort system case study. Technical report, TU Braunschweig, 2013.
[16]
M. Lochau, S. Lity, R. Lachmann, I. Schaefer, and U. Goltz. Delta-oriented model-based integration testing of large-scale systems. The Journal of Systems and Software, 91: 63--84, 2014.
[17]
M. Lochau, S. Oster, U. Goltz, and A. Schürr. Model-based Pairwise Testing for Feature Interaction Coverage in Software Product Line Engineering. Software Quality Journal, pages 1--38, 2011.
[18]
M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-based Testing of Delta-oriented Software Product Lines. TAP'12, 2012.
[19]
D. Mondal, H. Hemmati, and S. Durocher. Exploring test suite diversification and code coverage in multi-objective test case selection. In ICST'2015, pages 1--10, April 2015.
[20]
H. Muccini. Using model Differencing for Architecture-level Regression Testing. SEAA'07, 2007.
[21]
G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Pairwise testing for software product lines: comparison of two approaches. Software Quality Journal, 20(3-4):605--643, 2012.
[22]
K. Pohl, G. Böckle, and F. van der Linden. Software Product Line Engineering. Springer, 2005.
[23]
S. Reis, A. Metzger, and K. Pohl. Integration Testing in Software product Line Engineering: A Model-Based Technique. In Fundamental Approaches to Software Engineering, pages 321--335. Springer, 2007.
[24]
G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on software engineering, Vol.27 No.10: 929--948, 2001.
[25]
A. Sanchez, S. Segura, and A. Ruiz-Cortes. A comparison of test case prioritization criteria for software product lines. ICST'14, pages 41--50, 2014.
[26]
I. Schaefer, L. Bettini, V. Bono, F. Damiani, and N. Tanzarella. Delta-oriented programming of software product lines. SPLC'10, pages 77--91, 2010.
[27]
C. Szyperski. Component Software: Beyond Object-Oriented Programming. Addison-Wesley Longman Publishing Co., Inc., 2002.
[28]
S. Thiel and A. Hein. Modeling and using product line variability in automotive systems. IEEE Softw., 19: 66--72, 2002.
[29]
S. Wang, S. Ali, and A. Gotlieb. Minimizing test suites in software product lines using weight-based genetic algorithms. GECCO '13, pages 1493--1500, 2013.
[30]
S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Softw. Test., Verif. Reliab., 22(2):67--120, 2007.

Cited By

View all
  • (2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
  • (2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
  • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
  • Show More Cited By

Index Terms

  1. Delta-oriented test case prioritization for integration testing of software product lines

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    SPLC '15: Proceedings of the 19th International Conference on Software Product Line
    July 2015
    460 pages
    ISBN:9781450336130
    DOI:10.1145/2791060
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    • Vanderbilt University: Vanderbilt University
    • Biglever: BigLever Software, Inc.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 20 July 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. architecture-based testing
    2. delta-oriented software product lines
    3. regression testing
    4. test case prioritization

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    SPLC '15
    Sponsor:
    • Vanderbilt University
    • Biglever

    Acceptance Rates

    SPLC '15 Paper Acceptance Rate 34 of 87 submissions, 39%;
    Overall Acceptance Rate 167 of 463 submissions, 36%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)31
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
    • (2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
    • (2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
    • (2023)An Experimental Evaluation of Path-Based Product Line Integration Testing and Test Coverage MetricsApplied Sciences10.3390/app13231267013:23(12670)Online publication date: 25-Nov-2023
    • (2023)Continuous T-Wise CoverageProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608980(87-98)Online publication date: 28-Aug-2023
    • (2023)Feature-oriented Test Case Selection during Evolution of Highly-Configurable SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608979(76-86)Online publication date: 28-Aug-2023
    • (2023)Coverage-Driven Test Automation for Highly-Configurable Railway SystemsProceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3571788.3571794(23-30)Online publication date: 25-Jan-2023
    • (2023)Compositional Learning for Interleaving Parallel AutomataFoundations of Software Science and Computation Structures10.1007/978-3-031-30829-1_20(413-435)Online publication date: 21-Apr-2023
    • (2023)A Systematic Literature Review on Test Case Prioritization TechniquesAgile Software Development10.1002/9781119896838.ch7(101-159)Online publication date: 8-Feb-2023
    • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media