skip to main content
10.1145/3023956.3023958acmotherconferencesArticle/Chapter ViewAbstractPublication PagesvamosConference Proceedingsconference-collections
research-article

Risk-based integration testing of software product lines

Published: 01 February 2017 Publication History

Abstract

Software product lines (SPL) capture commonalities and variabilities of product families and, thus, enable mass customization of product variants according to customers desired configurations. However, they introduce new challenges to software testing due to a potentially large number of variants. While each variant should be tested, testing resources are limited and, thus, a retest of all, partially redundant, test cases for each variant is not feasible in SPL testing. Coping with these issues has been a major research focus in recent years, leading to different testing approaches. However, risk-based testing has not gained much attention in the SPL domain while being a successful approach for single-software systems. In this paper, we propose a novel risk-based testing approach for SPL integration testing. We incrementally test SPLs by stepping from one variant to the next. For each variant, we automatically compute failure probabilities and failure impacts for its architectural components. To avoid a computational overhead of generating and analyzing each variant, we exploit the variability between variants defined as deltas to focus on important changes. We evaluate our approach using an automotive case study, showing that the risk-based technique leads to positive results compared to random and delta-oriented testing.

References

[1]
R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the Accuracy of Spectrum-based Fault Localization. In Testing: Academic and Industrial Conf. Practice and Research Techn. - MUTATION, pages 89--98, 2007.
[2]
S. Amland. Risk-based testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. J. Sys. and Soft., 53:287--295, 2000.
[3]
X. Bai, R. S. Kenett, and W. Yu. Risk assessment and adaptive group testing of semantic web services. Int. J. of Softw. Eng. and Knowledge Eng., 22(05):595--620, 2012.
[4]
D. Benavides, P. Trinidad, and A. Ruiz-Cortés. Automated Reasoning on Feature Models, pages 491--503. Springer Berlin Heidelberg, 2005.
[5]
D. Clarke, M. Helvensteijn, and I. Schaefer. Abstract Delta Modelling. 25(3):482--527, 2015.
[6]
J. A. Duraes and H. S. Madeira. Emulation of software faults: A field data study and a practical approach. IEEE Trans. Soft. Eng., 32(11):849--867, 2006.
[7]
E. Engström. Exploring Regression testing and software product line testing - research and state of practice. Lic dissertation, Lund University, May 2010.
[8]
M. Felderer, C. Haisjackl, R. Breu, and J. Motz. Integrating manual and automatic risk assessment for risk-based testing. Software Quality Process Autom. in Softw. Develop., pages 159--180, 2012.
[9]
M. Felderer, C. Haisjackl, V. Pekar, and R. Breu. A risk assessment framework for software testing. In Proc. Int'l Symposium Leveraging App. of Formal Meth., Verific. and Validation, pages 292--308. 2014.
[10]
M. Felderer, C. Haisjackl, V. Pekar, and R. Breu. An exploratory study on risk estimation in risk-based testing approaches. In Software Quality Days (SWQD), pages 32--43. 2015.
[11]
M. Felderer and I. Schieferdecker. A taxonomy of risk-based testing. Int. J. Softw. Tools Technol. Transf., 16(5):559--568, 2014.
[12]
H. Hartmann, F. van der Linden, and J. Bosch. Risk based testing for software product line engineering. In Proc. Int'l Software Product Line Conference, pages 227--231, 2014.
[13]
Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Trans. Soft. Eng., 37(5):649--678, Sept 2011.
[14]
M. F. Johansen, Ø. Haugen, and F. Fleurey. An algorithm for generating t-wise covering arrays from large feature models. In Proc. Int'l Software Product Line Conference, pages 46--55, 2012.
[15]
K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-oriented domain analysis (foda) feasibility study. Technical report, S.E. Institute - Carnegie Mellon University, 1990.
[16]
J. Kloos, T. Hussain, and R. Eschbach. Risk-based testing of safety-critical embedded systems driven by fault tree analysis. In Proc. Int'l Conf. Softw. Testing, Verific. and Valid. Workshops, pages 26--33, 2011.
[17]
R. Lachmann, S. Lity, M. Al-Hajjaji, F. E. Fürchtegott, and I. Schaefer. Fine-grained test case prioritization for integration testing of delta-oriented software product lines. In Proc. Int'l Workshop Feature-Oriented Software Development, 2016.
[18]
R. Lachmann, S. Lity, S. Lischke, S. Beddig, S. Schulze, and I. Schaefer. Delta-oriented test case prioritization for integration testing of software product lines. In Proc. Int'l Software Product Line Conference, pages 81--90, 2015.
[19]
S. Lity, R. Lachmann, M. Lochau, and I. Schaefer. Delta-oriented software product line test models - the body comfort system case study. Technical report, TU Braunschweig, 2013.
[20]
M. Lochau, S. Lity, R. Lachmann, I. Schaefer, and U. Goltz. Delta-oriented model-based integration testing of large-scale systems. J. Sys. and Soft., 91:63--84, 2014.
[21]
G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. le Traon. Pairwise testing for software product lines: comparison of two approaches. Software Quality Jour., 20(3--4):605--643, 2012.
[22]
K. Pohl, G. Böckle, and F. van der Linden. Software Product Line Engineering. Springer, 2005.
[23]
G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Trans. Soft. Eng., Vol.27 No.10:929--948, 2001.
[24]
I. Schaefer, L. Bettini, V. Bono, F. Damiani, and N. Tanzarella. Delta-oriented programming of software product lines. In Proc. Int'l Software Product Line Conference, pages 77--91, 2010.
[25]
I. Schaefer, R. Rabiser, D. Clarke, L. Bettini, D. Benavides, G. Botterweck, A. Pathak, S. Trujillo, and K. Villela. Software Diversity: State of the Art and Perspectives. Int. J. Softw. Tools Technol. Transf., 14(5):477--495, 2012.
[26]
H. Stallbaum, A. Metzger, and K. Pohl. An automated technique for risk-based test case generation and prioritization. In Proc. Int'l Workshop Automation of Softw. Test, pages 67--70, 2008.
[27]
S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab., 22(2):67--120, 2007.
[28]
H. Yoon and B. Choi. A test case prioritization based on degree of risk exposure and its empirical study. Int. J. of Softw. Eng. and Knowledge Eng., 21(02):191--209, 2011.

Cited By

View all
  • (2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
  • (2024)MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller SamplesProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672589(47-53)Online publication date: 2-Sep-2024
  • (2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
VaMoS '17: Proceedings of the 11th International Workshop on Variability Modelling of Software-Intensive Systems
February 2017
114 pages
ISBN:9781450348119
DOI:10.1145/3023956
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. model-based testing
  2. risk-based testing
  3. software product lines
  4. test case prioritization

Qualifiers

  • Research-article

Funding Sources

Conference

VaMoS '17

Acceptance Rates

Overall Acceptance Rate 66 of 147 submissions, 45%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
  • (2024)MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller SamplesProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672589(47-53)Online publication date: 2-Sep-2024
  • (2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
  • (2023)An Experimental Evaluation of Path-Based Product Line Integration Testing and Test Coverage MetricsApplied Sciences10.3390/app13231267013:23(12670)Online publication date: 25-Nov-2023
  • (2023)Feature-oriented Test Case Selection during Evolution of Highly-Configurable SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608979(76-86)Online publication date: 28-Aug-2023
  • (2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
  • (2022)Cost-effective learning-based strategies for test case prioritization in continuous integration of highly-configurable softwareEmpirical Software Engineering10.1007/s10664-021-10093-327:6Online publication date: 1-Nov-2022
  • (2021)Using Ant Colony Optimisation to Select Features Having Associated CostsTesting Software and Systems10.1007/978-3-031-04673-5_8(106-122)Online publication date: 10-Nov-2021
  • (2020)Risk-based compatibility analysis in automotive systems engineeringProceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings10.1145/3417990.3421263(1-10)Online publication date: 16-Oct-2020
  • (2020)Model-based security analysis of feature-oriented software product linesACM SIGPLAN Notices10.1145/3393934.327812653:9(93-106)Online publication date: 7-Apr-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media