Skip to main content
SpringerLink
Log in

Effective product-line testing using similarity-based product prioritization

  • Regular Paper
  • Published:
Software & Systems Modeling Aims and scope Submit manuscript

Abstract

A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible due to the potentially exponential number of products in the number of features. Accordingly, several sampling approaches have been proposed to select a presumably minimal, yet sufficient number of products to be tested. Since the time budget for testing is limited or even a priori unknown, the order in which products are tested is crucial for effective product-line testing. Prioritizing products is required to increase the probability of detecting faults faster. In this article, we propose similarity-based prioritization, which can be efficiently applied on product samples. In our approach, we incrementally select the most diverse product in terms of features to be tested next in order to increase feature interaction coverage as fast as possible during product-by-product testing. We evaluate the gain in the effectiveness of similarity-based prioritization on three product lines with real faults. Furthermore, we compare similarity-based prioritization to random orders, an interaction-based approach, and the default orders produced by existing sampling algorithms considering feature models of various sizes. The results show that our approach potentially increases effectiveness in terms of fault detection ratio concerning faults within real-world product-line implementations as well as synthetically seeded faults. Moreover, we show that the default orders of recent sampling algorithms already show promising results, which, however, can still be improved in many cases using similarity-based prioritization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. http://wwwiti.cs.uni-magdeburg.de/iti_db/research/spl-testing/.

  2. Last accessed May 10, 2016.

  3. http://wwwiti.cs.uni-magdeburg.de/iti_db/research/spl-testing/.

References

  1. Abal, I., Brabrand, C., Wasowski, A.: 42 variability bugs in the linux kernel: a qualitative analysis. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 421–432. ACM (2014)

  2. Al-Hajjaji, M.: Scalable sampling and prioritization for product-line testing. In: Proc. Software Engineering (SE), pp. 295–298. Gesellschaft für Informatik (GI) (2015)

  3. Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 197–206. ACM (2014)

  4. Al-Hajjaji, M., Krieter, S., Thüm, T., Lochau, M., Saake, G.: IncLing: efficient product-line testing using incremental pairwise sampling. In: Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), pp. 144–155. ACM (2016)

  5. Al-Hajjaji, M., Meinicke, J., Krieter, S., Schröter, R., Thüm, T., Leich, T., Saake, G.: Tool demo: testing configurable systems with featureIDE. In: Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), pp. 173–177. ACM (2016)

  6. Apel, S., Batory, D., Kästner, C., Saake, G.: Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, Berlin (2013)

    Book  Google Scholar 

  7. Apel, S., von Rhein, A., Wendler, P., Größlinger, A., Beyer, D.: Strategies for product-line verification: case studies and experiments. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 482–491. IEEE (2013)

  8. Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 1–10. ACM (2011)

  9. Bagheri, E., Ensan, F., Gasevic, D.: Grammar-based test generation for software product line feature models. In: Proc. Conf. Centre for Advanced Studies on Collaborative Research (CASCON), pp. 87–101. IBM Corp. (2012)

  10. Baller, H., Lochau, M.: Towards incremental test suite optimization for software product lines. In: Proc. Int’l Workshop Feature-Oriented Software Development (FOSD), pp. 30–36. ACM (2014)

  11. Baller, H., Lity, S., Lochau, M., Schaefer, I.: Multi-objective test suite optimization for incremental product family testing. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 303–312. IEEE (2014)

  12. Bryce, R.C., Colbourn, C.J.: One-test-at-a-time heuristic search for interaction test suites. In: Proc. Int’l Conf. on Genetic and Evolutionary Computation (GECCO), pp. 1082–1089. ACM (2007)

  13. Bryce, R.C., Memon, A.M.: Test suite prioritization by interaction coverage. In: Workshop on Domain Specific Approaches to Software Test Automation: Conjunction with the 6th ESEC/FSE Joint Meeting, DOSTA ’07, pp. 1–7. ACM (2007)

  14. Bürdek, J., Lochau, M., Bauregger, S., Holzer, A., von Rhein, A., Apel, S., Beyer, D.: Facilitating reuse in multi-goal test-suite generation for software product lines. In: Egyed, A., Schaefer, I. (eds.) Fundamental Approaches to Software Engineering, volume 9033 of Lecture Notes in Computer Science, pp. 84–99. Springer (2015)

  15. Carmo Machado, I.D., McGregor, J.D., Cavalcanti, YaC, De Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. J. Inf. Softw. Technol. (IST) 56(10), 1183–1199 (2014)

    Article  Google Scholar 

  16. Cartaxo, E.G., Machado, P.D., Neto, F.G.O.: On the use of a similarity function for test case selection in the context of model-based testing. Softw. Test. Verif. Reliab. (STVR) 21(2), 75–100 (2011)

    Article  Google Scholar 

  17. Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  18. Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pp. 129–139. ACM (2007)

  19. Devroey, X., Perrouin, G., Cordy, M., Schobbens, P.-Y., Legay, A., Heymans, P.: Towards statistical prioritization for software product lines testing. In: Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 10:1–10:7. ACM (2014)

  20. Devroey, X., Perrouin, G., Legay, A., Schobbens, P.-Y., Heymans, P.: Search-based similarity-driven behavioural SPL testing. In: Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 89–96. ACM (2016)

  21. Dietrich, C., Tartler, R., Schröder-Preikschat, W., Lohmann, D.: Understanding linux feature distribution. In: Proc. of Workshop on Modularity in Systems Software (MISS), pp. 15–20. ACM (2012)

  22. Elbaum, S., Malishevsky, A.G., Rothermel, G.: Prioritizing test cases for regression testing. SIGSOFT Softw. Eng. Notes 25(5), 102–112 (2000)

    Article  Google Scholar 

  23. Engström, E., Runeson, P.: Software product line testing—a systematic mapping study. J. Inf. Softw. Technol. (IST) 53(1), 2–13 (2011)

    Article  Google Scholar 

  24. Ensan, A., Bagheri, E., Asadi, M., Gasevic, D., Biletskiy, Y.: Goal-oriented test case selection and prioritization for product line feature models. In: Proc. Int’l Conf. on Information Technology: New Generations (ITNG), pp. 291–298. IEEE (2011)

  25. Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Proc. Int’l Conf. Advanced Information Systems Engineering (CAiSE), vol. 7328, pp. 613–628. Springer (2012)

  26. Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir. Softw. Eng. (EMSE) 16(1), 61–102 (2011)

    Article  Google Scholar 

  27. Hall, R.J.: Fundamental nonmodularity in electronic mail. Autom. Softw. Eng. 12(1), 41–79 (2005)

    Article  Google Scholar 

  28. Hamming, R.W.: Error detecting and error correcting codes. Bell Syst. Tech. J. 29(2), 147–160 (1950)

    Article  MathSciNet  Google Scholar 

  29. Harrold, M.J.: Testing: a roadmap. In: Proc. of the Conf. on The Future of Software Engineering (FSE), pp. 61–72. ACM (2000)

  30. Hemmati, H., Briand, L.: An industrial investigation of similarity measures for model-based test case selection. In: Proc. Int’l Symposium Software Reliability Engineering (ISSRE), pp. 141–150. IEEE (2010)

  31. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: PLEDGE: a product line editor and test generation tool. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 126–129. ACM (2013)

  32. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Le Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize T-wise test configurations for software product lines. IEEE Trans. Softw. Eng. (TSE) 40(7), 650–670 (2014)

    Article  Google Scholar 

  33. Jackson, M., Zave, P.: Distributed feature composition: a virtual architecture for telecommunications services. IEEE Trans. Softw. Eng. (TSE) 24(10), 831–847 (1998)

    Article  Google Scholar 

  34. Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 638–652. Springer (2011)

  35. Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating T-wise covering arrays from large feature models. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 46–55. ACM (2012)

  36. Johansen, M.F., Haugen, Ø., Fleurey, F., Eldegard, A.G., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 269–284. Springer (2012)

  37. Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute (1990)

  38. Kästner, C., Apel, S., ur Rahman, S.S., Rosenmüller, M., Batory, D., Saake, G.: On the impact of the optional feature problem: analysis and case studies. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 181–190. Software Engineering Institute (2009)

  39. Kramer, J., Magee, J., Sloman, M., Lister, A.: CONIC: an integrated approach to distributed computer control systems. IEE Proc. Comput. Dig. Tech. 130(1), 1 (1983)

    Article  Google Scholar 

  40. Kuhn, D.R., Wallace, D.R., Gallo Jr., A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. (TSE) 30(6), 418–421 (2004)

    Article  Google Scholar 

  41. Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., Schaefer, I.: Delta-oriented test case prioritization for integration testing of software product lines. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 81–90. ACM (2015)

  42. Lei, Y., Kacker, R.N., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for T-way software testing. In: Proc. Int’l Conf. Engineering of Computer-Based Systems (ECBS), pp. 549–556. IEEE (2007)

  43. McGregor, J.: Testing a software product line. In: Testing Techniques in Software Engineering, pp. 104–140. Springer, Berlin (2010)

  44. Meinicke, J., Wong, C.-P., Kästner, C., Thüm, T., Saake, G.: On essential configuration complexity: measuring interactions in highly-configurable systems. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 483–494. ACM (2016)

  45. Mendonça, M., Branco, M., Cowan, D.: SPLOT: software product lines online tools. In: Proc. Conf. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), pp. 761–762. ACM (2009)

  46. Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley, New York (2011)

    Google Scholar 

  47. Nguyen, H.V., Kästner, C., Nguyen, T.N.: Exploring variability-aware execution for testing plugin-based web applications. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 907–918. ACM (2014)

  48. Northrop, L., Clements, P.: A framework for software product line practice, version 5.0. SEI (2007)

  49. Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 196–210. Springer (2010)

  50. Perrouin, G., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Automated and scalable T-wise test case generation strategies for software product lines. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 459–468. IEEE (2010)

  51. Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. (SQJ) 20(3–4), 605–643 (2012)

    Article  Google Scholar 

  52. Plath, M., Ryan, M.: Feature integration using a feature construct. Sci. Comput. Program. (SCP) 41(1), 53–84 (2001)

    Article  MATH  Google Scholar 

  53. Rothermel, G., Untch, R., Chu, C., Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. (TSE) 27(10), 929–948 (2001)

    Article  Google Scholar 

  54. Sánchez, A.B., Segura, S., Ruiz-Cortés, A.: A comparison of test case prioritization criteria for software product lines. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 41–50. IEEE (2014)

  55. Sánchez, A., Segura, S., Parejo, J., Ruiz-Cortés, A.: Variability testing in the wild: the drupal case study. Softw. Syst. Model. 1–22 (2015)

  56. Shi, J., Cohen, M.B., Dwyer, M.B.: Integration testing of software product lines using compositional symbolic execution. In: Proc. Int’l Conf. Fundamental Approaches to Software Engineering (FASE), pp. 270–284. Springer (2012)

  57. Thüm, T., Kästner, C., Erdweg, S., Siegmund, N.: Abstract features in feature modeling. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 191–200. IEEE (2011)

  58. Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6:1–6:45 (2014)

    Article  Google Scholar 

  59. Thüm, T., Kästner, C., Benduhn, F., Meinicke, J., Saake, G., Leich, T.: FeatureIDE: an extensible framework for feature-oriented software development. Sci. Comput. Program. (SCP) 79, 70–85 (2014)

    Article  Google Scholar 

  60. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., Burlington (2007)

    Google Scholar 

  61. Visser, W., Havelund, K., Brat, G.P., Park, S.: Model checking programs. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 3–12. Springer (2000)

  62. Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Time-aware test suite prioritization. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pp. 1–12. ACM (2006)

  63. Weiss, D.M.: The product line hall of fame. In: Proc. Int’l Software Product Line Conf. (SPLC), p. 395. IEEE (2008)

  64. Yoo, S., Harman, N.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. (STVR) 22(2), 67–120 (2012)

    Article  Google Scholar 

  65. Yoo, S., Harman, M., Tonella, P., Susi, A.: Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), ISSTA’09, pp. 201–212. ACM (2009)

Download references

Acknowledgements

We are grateful to anonymous reviewers for their useful comments. We also thank Martin Fagereng Johansen for his support in integrating the SPLCATool into FeatureIDE. We are grateful to Remo Lachmann, Reimar Schröter, Fabian Benduhn, and Hauke Baller for helpful comments on a draft and for interesting discussions. This work has been supported by the German Research Foundation (DFG) in the Priority Programme SPP 1593: Design For Future Managed Software Evolution (LO 2198/2-1).

Author information

Authors and Affiliations

  1. University of Magdeburg, Magdeburg, Germany

    Mustafa Al-Hajjaji & Gunter Saake

  2. TU Braunschweig, Braunschweig, Germany

    Thomas Thüm

  3. TU Darmstadt, Darmstadt, Germany

    Malte Lochau

  4. METOP GmbH, University of Magdeburg, Magdeburg, Germany

    Jens Meinicke

Authors
  1. Mustafa Al-Hajjaji
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas Thüm
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Malte Lochau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jens Meinicke
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gunter Saake
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mustafa Al-Hajjaji.

Additional information

Communicated by Prof. Hassan Gomaa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Hajjaji, M., Thüm, T., Lochau, M. et al. Effective product-line testing using similarity-based product prioritization. Softw Syst Model 18, 499–521 (2019). https://doi.org/10.1007/s10270-016-0569-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-016-0569-2

Keywords

Search

Navigation