Skip to main content
SpringerLink
Log in

A systematic test case selection methodology for product lines: results and insights from an industrial case study

  • Experience Report
  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

In the context of product lines, test case selection aims at obtaining a set of relevant test cases for a product from the entire set of test cases available for a product line. While working on a research-based innovation project on automated testing of product lines of Video Conferencing Systems (VCSs) developed by Cisco, we felt the need to devise a cost-effective way of selecting relevant test cases for a product. To fulfill such need, we propose a systematic and automated test selection methodology using: 1) Feature Model for Testing (FM_T) to capture commonalities and variabilities of a product line; 2) Component Family Model for Testing (CFM_T) to model the structure of test case repository; 3) A tool to automatically build restrictions from CFM_T to FM_T and traces from CFM_T to the actual test cases. Using our methodology, a test engineer is only required to select relevant features through FM_T at a higher level of abstraction for a product and the corresponding test cases will be obtained automatically. We evaluate our methodology by applying it to a VCS product line called Saturn with seven commercial products and the results show that our methodology can significantly reduce cost measured as test selection time and at the same time achieves higher effectiveness (feature coverage, feature pairwise coverage and fault detection) as compared with the current manual process. Moreover, we conduct a questionnaire-based study to solicit the views of test engineers who are involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology in their current practice. Finally, we present a set of lessons learnt while applying product line engineering at Cisco for test case selection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Alférez M, Lopez-Herrejon R, Moreira A, Amaral V, Egyed A (2011) Supporting consistency checking between features and software product line use scenarios, In Schmid, K. (Ed.) Top productivity through software reuse. Springer Berlin Heidelberg, pp. 20–35

  • Barros MdO, Neto ACD (2011) Threats to validity in search-based software engineering empirical studies. UNIRIO - Universidade Federal do Estado do Rio de Janeiro0006/2011

  • Benavides D, Segura S, Ruiz-Cortés A (2010) Automated analysis of feature models 20 years later: a literature review. Inf Syst 35:615–636

    Article  Google Scholar 

  • Beuche D (2011) Modeling and building software product lines with pure::variants, proceedings of the 15th international software product line conference, vol 2. ACM, Munich, pp 1–1

    Google Scholar 

  • Bible J, Rothermel G, Rosenblum DS (2001) A comparative study of coarse- and fine-grained safe regression test-selection techniques. ACM Trans Softw Eng Methodol 10:149–183

    Article  MATH  Google Scholar 

  • Cabot J, Clarisó R, Riera D (2014) On the verification of UML/OCL class diagrams using constraint programming. J Syst Softw 93:1–23

    Article  Google Scholar 

  • Chen Y, Probert RL, Sims DP (2002) Specification-based regression test selection with risk analysis, proceedings of the 2002 conference of the centre for advanced studies on collaborative research. IBM Press, Toronto, p 1

    Google Scholar 

  • CiscoSystems C (2010) Cisco telepresence codec c90, data sheet

  • Cohen MB, Dwyer MB, Shi J (2006) Coverage and adequacy in software product line testing, proceedings of the ISSTA 2006 workshop on role of software architecture for testing and analysis. ACM, Portland, pp 53–63

    Google Scholar 

  • Czarnecki K, Antkiewicz M (2005) Mapping features to models: a template approach based on superimposed variants, proceedings of the 4th international conference on generative programming and component engineering. Springer, Tallinn, pp 422–437

    Google Scholar 

  • Czarnecki K, Pietroszek K (2006) Verifying feature-based model templates against well-formedness OCL constraints, proceedings of the 5th international conference on generative programming and component engineering. ACM, Portland, pp 211–220

    Google Scholar 

  • Czarnecki K, Kim CHP, Kalleberg KT (2006) Feature models are views on ontologies, proceedings of the 10th international on software product line conference. IEEE computer society, pp. 41–51

  • Egyed A (2006) Instant consistency checking for the UML, proceedings of the 28th international conference on Software engineering. ACM, Shanghai, pp 381–390

    Google Scholar 

  • Engstrom E (2010a) Regression test selection and product line system testing, software testing, verification and validation (ICST), 2010 third international conference on, pp. 512–515

  • Engstrom E, Runeson P, Skoglund M (2010) A systematic review on regression test selection techniques. Inf Softw Technol 52:14–30

    Article  Google Scholar 

  • Gent IP, Jefferson C, Miguel I (2006) MINION: a fast, scalable, constraint solver, proceedings of the 2006 conference on ecai 2006: 17th European conference on artificial intelligence August 29 -- September 1, 2006, Riva del Garda. Ios Press, Italy, pp 98–102

    Google Scholar 

  • Graves TL, Harrold MJ, Kim J-M, Porter A, Rothermel G (2001) An empirical study of regression test selection techniques. ACM Trans Softw Eng Methodol 10:184–208

    Article  MATH  Google Scholar 

  • Harrold MJ, Souffa ML (1988) An incremental approach to unit testing during maintenance, software maintenance, 1988., proceedings of the conference on, pp. 362–367.

  • Hartmann J, Robson DJ (1990) Techniques for selective revalidation. IEEE Softw 7:31–36

    Article  Google Scholar 

  • Heidenreich F, Kopcsek J, Wende C (2008) FeatureMapper: mapping features to models, companion of the 30th international conference on software engineering. ACM, Leipzig, pp 943–944

    Google Scholar 

  • Heidenreich F, Sanchez P, Santos J, Zschaler S, Alferez M, Araujo J, Fuentes L, Kulesza U, Moreira A, Rashid A (2010) Relating feature models to other models of a software product line: a comparative study of featuremapper and VML, in: Shmuel, K., Mira, M. (Eds.) Transactions on aspect-oriented software development VII. Springer-Verlag, pp. 69–114

  • Kang K, Cohen, Sholom, Hess, James, Novak, William, & Peterson A, (1990) Feature-Oriented Domain Analysis (FODA) feasibility study (CMU/SEI-90-TR-021). Software Engineering Institute, Carnegie Mellon University

  • Lauenroth K, Pohl K (2008) Dynamic consistency checking of domain requirements in product line engineering, International Requirements Engineering, 2008. RE '08. 16th IEEE, pp. 193–202

  • Mansour N, Bahsoon R, Baradhi G (2001) Empirical comparison of regression test selection algorithms. J Syst Softw 57:79–90

    Article  Google Scholar 

  • McGregor J (2001) Testing a software product line (CMU/SEI-2001-TR-022), software engineering institute. Carnegie Mellon University, Pittsburgh

    Google Scholar 

  • Muccini H, van der Hoek A (2003) Towards testing product line architectures. Electron Notes Theor Comput Sci 82:99–109

    Article  Google Scholar 

  • Nebut C, Traon Y, Jezequel J.-M (2006). System testing of product lines: from requirements to test cases, In Käköla T, Duenas J (Eds.) Software product lines. Springer Berlin Heidelberg, pp. 447–477

  • Orso A, Harrold MJ, Rosenblum D, Rothermel G, Soffa ML, Do H (2001) Using component metacontent to support the regression testing of component-based software, software maintenance, 2001. Proceedings. IEEE International Conference on, pp. 716–725

  • Puresystems G (2006) Variant management with pure::variants, technical white paper

  • Scheidemann KD (2006) Optimizing the selection of representative configurations in verification of evolving product lines of distributed embedded systems, software product line conference, 2006 10th International, pp. 75–84

  • Sheskin DJ (2007) Handbook of parametric and nonparametric statistical procedures. Chapman \& Hall/CRC

  • Thaker S, Batory D, Kitchin D, Cook W (2007) Safe composition of product lines, proceedings of the 6th international conference on Generative programming and component engineering. ACM, Salzburg, pp 95–104

    Google Scholar 

  • Vierhauser M, Gr P, #252, nbacher, Egyed A, Rabiser R, Heider W (2010) Flexible and scalable consistency checking on product line variability models, proceedings of the IEEE/ACM international conference on automated software engineering. ACM, Antwerp, Belgium, pp. 63–72

  • Wang S, Gotlieb A, Liaaen M, Briand LC (2012) Automatic selection of test execution plans from a video conferencing system product line, proceedings of the variability for you workshop: variability modeling made useful for everyone. ACM, Innsbruck, pp 32–37

    Book  Google Scholar 

  • Wang S, Ali S, Gotlieb A (2013a) Minimizing test suites in software product lines using weight-based genetic algorithms, proceeding of the fifteenth annual conference on Genetic and evolutionary computation conference. ACM, Amsterdam, pp 1493–1500

    Book  Google Scholar 

  • Wang S, Ali S, Tao Y, Liaaen M, (2013b) Using feature model to support model-based testing of product lines: an Industrial Case Study, Quality Software (QSIC), 2013 13th international conference on, pp. 75–84

  • Wang S, Gotlieb A, Ali S, Liaaen M (2013a) Automated test case selection using feature model: an industrial case study, In Moreira A, Schätz B, Gray J, Vallecillo A, Clarke P (Eds.) Model-driven engineering languages and systems. Springer Berlin Heidelberg, pp. 237–253

  • Warners JP (1998) A linear-time transformation of linear inequalities into conjunctive normal form. Inf Process Lett 68:63–69

    Article  MathSciNet  MATH  Google Scholar 

  • Wohlin C, Runeson P, Host M, Ohlsson MC, Regnell B, Wesslen A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers

  • Yih-Farn C, Rosenblum DS, Vo KP, (1994) TESTTUBE: a system for selective regression testing, software engineering, 1994. Proceedings. ICSE-16., 16th International Conference on, pp. 211–220

  • Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab 22:67–120

    Article  Google Scholar 

  • Zschaler S, Sánchez P, Santos J, Alférez M, Rashid A, Fuentes L, Moreira A, Araújo J, Kulesza U (2010) VML* – a family of languages for variability management in software product lines, In Brand M, Gašević D, Gray J (Eds.) Software Language Engineering. Springer Berlin Heidelberg, pp. 82–102

Download references

Author information

Authors and Affiliations

  1. Certus Software V&V Center, Simula Research Laboratory, Oslo, Norway

    Shuai Wang, Shaukat Ali & Arnaud Gotlieb

  2. Department of Informatics, University of Oslo, Oslo, Norway

    Shuai Wang

  3. Cisco Systems Inc., Oslo, Norway

    Marius Liaaen

  4. P.O. Box 134, 1325, Lysaker, Norway

    Shuai Wang

Authors
  1. Shuai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaukat Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arnaud Gotlieb
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marius Liaaen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shuai Wang.

Additional information

Communicated by: Ebrahim Bagheri, David Benavides, Per Runeson and Klaus Schmid

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Ali, S., Gotlieb, A. et al. A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empir Software Eng 21, 1586–1622 (2016). https://doi.org/10.1007/s10664-014-9345-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-014-9345-5

Keywords

Search

Navigation