Skip to main content
SpringerLink
Log in
Book cover

Computational Intelligence and Quantitative Software Engineering pp 59–87Cite as

Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges

  • Chapter
  • First Online:

Part of the book series: Studies in Computational Intelligence ((SCI,volume 617))

Abstract

Because of economical, technological and marketing reasons today’s software systems are more frequently being built as families where each product variant implements a different combination of features. Software families are commonly called Software Product Lines (SPLs) and over the past three decades have been the subject of extensive research and application. Among the benefits of SPLs are: increased software reuse, faster and easier product customization, and reduced time to market. However, testing SPLs is specially challenging as the number of product variants is usually large making it infeasible to test every single variant. In recent years there has been an increasing interest in applying evolutionary computation techniques for SPL testing. In this chapter, we provide a concise overview of the state of the art and practice in SPL testing with evolutionary techniques as well as to highlight open questions and areas for future research.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.uml.org/.

  2. 2.

    An early version is available in [24].

  3. 3.

    Definitions based on [12, 25].

  4. 4.

    In [30] the algorithm is named PGS. We changed its name for this chapter to avoid confusions with the original algorithm PGS in [27] that was not designed for SPLs.

  5. 5.

    For notational brevity we omit on the vector function and the objective vectors the T that denotes the transpose on vectors.

  6. 6.

    Available at URL: http://minisat.se/MiniSat+.html.

  7. 7.

    An early version is available in [24].

  8. 8.

    http://www.splot-research.org/.

  9. 9.

    http://www.evosuite.org/.

References

  1. Pohl, K., Bockle, G., van der Linden, F.J.: Software Product Line Engineering: Foundations, Principles and Techniques. Springer, Berlin (2005)

    Book  Google Scholar 

  2. Batory, D.S., Sarvela, J.N., Rauschmayer, A.: Scaling step-wise refinement. IEEE Trans. Softw. Eng. 30(6), 355–371 (2004)

    Article  Google Scholar 

  3. van der Linden, F., Schmid, K., Rommes, E.: Software Product Lines in Action—The Best Industrial Practice in Product Line Engineering. Springer, Berlin (2007)

    Google Scholar 

  4. Engström, E., Runeson, P.: Software product line testing—A systematic mapping study. Inf. Softw. Technol. 53(1), 2–13 (2011)

    Article  Google Scholar 

  5. da Mota Silveira Neto, P.A., do Carmo Machado, I., McGregor, J.D., de Almeida, E.S., de Lemos Meira, S.R.: A systematic mapping study of software product lines testing. Inf. Softw. Technol. 53(5), 407–423 (2011)

    Google Scholar 

  6. Lee, J., Kang, S., Lee, D.: A survey on software product line testing. 16th International Software Product Line Conference, pp. 31–40 (2012)

    Google Scholar 

  7. do Carmo Machado, I., McGregor, J.D., de Almeida, E.S.: Strategies for testing products in software product lines. ACM SIGSOFT Softw. Eng. Notes 37(6), 1–8 (2012)

    Google Scholar 

  8. Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. 45(1), 11 (2012)

    Article  Google Scholar 

  9. Eiben, A., Smith, J.: Introduction to Evolutionary Computing. Springer, Berlin (2003)

    Google Scholar 

  10. McMinn, P.: Search-based software testing: past, present and future. In: ICST Workshops, pp. 153–163. IEEE Computer Society (2011)

    Google Scholar 

  11. Kang, K., Cohen, S., Hess, J., Novak, W., Peterson, A.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon University (1990)

    Google Scholar 

  12. Benavides, D., Segura, S., Cortés, A.R.: Automated analysis of feature models 20 years later: a literature review. Inf. Syst. 35(6), 615–636 (2010)

    Article  Google Scholar 

  13. Lopez-Herrejon, R.E., Batory, D.S.: A standard problem for evaluating product-line methodologies. In: Bosch, J. (ed.) GCSE. Volume 2186 of Lecture Notes in Computer Science, pp. 10–24. Springer, Berlin (2001)

    Google Scholar 

  14. Michalewicz, Z., Fogel, D.B.: How to Solve It: Modern Heuristics, 2nd edn. Springer, Berlin (2010)

    Google Scholar 

  15. Goldberg, D.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading (1989)

    Google Scholar 

  16. da Mota Silveira Neto, P.A., Runeson, P., do Carmo Machado, I., de Almeida, E.S., de Lemos Meira, S.R., Engström, E.: Testing software product lines. IEEE Software 28(5), 16–20 (2011)

    Google Scholar 

  17. Wohlin, C., Runeson, P., da Mota Silveira Neto, P.A., Engström, E., do Carmo Machado, I., de Almeida, E.S.: On the reliability of mapping studies in software engineering. J. Syst. Softw. 86(10), 2594–2610 (2013)

    Google Scholar 

  18. do Carmo Machado, I., McGregor, J.D., Cavalcanti, Y.C., de Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. Inf. Softw. Technol. 56(10), 1183–1199 (2014)

    Google Scholar 

  19. Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)

    Article  Google Scholar 

  20. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11:1–11:29 (February 2011)

    Google Scholar 

  21. Yilmaz, C., Fouché, S., Cohen, M.B., Porter, A.A., Demiröz, G., Koc, U.: Moving forward with combinatorial interaction testing. IEEE Comput. 47(2), 37–45 (2014)

    Article  Google Scholar 

  22. de Freitas, F.G., de Souza, J.T.: Ten years of search based software engineering: a bibliometric analysis. In: Cohen, M.B., Cinnéide, M.Ó. (eds.) SSBSE. Volume 6956 of Lecture Notes in Computer Science, pp. 18–32. Springer, Berlin (2011)

    Google Scholar 

  23. Lopez-Herrejon, R.E., Linsbauer, L., Egyed, A.: A systematic mapping study of search-based software engineering for software product lines. Inf. Softw. Technol. J. (to appear)

    Google Scholar 

  24. Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Linsbauer, L., Egyed, A., Alba, E.: A hitchhiker’s guide to search-based software engineering for software product lines. CoRR abs/1406.2823 (2014)

    Google Scholar 

  25. Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. 16th International Software Product Line Conference, pp. 46–55 (2012)

    Google Scholar 

  26. Lopez-Herrejon, R.E., Egyed, A.: Towards interactive visualization support for pairwise testing software product lines. In: Telea, A., Kerren, A., Marcus, A. (eds.) VISSOFT, pp. 1–4. IEEE (2013)

    Google Scholar 

  27. Ferrer, J., Kruse, P.M., Chicano, J.F., Alba, E.: Evolutionary algorithm for prioritized pairwise test data generation. InL: Soule, T., Moore, J.H. (eds.) GECCO, pp. 1213–1220. ACM (2012)

    Google Scholar 

  28. Durillo, J.J., Nebro, A.J.: jmetal: a java framework for multi-objective optimization. Adv. Eng. Softw. 42(10), 760–771 (2011)

    Article  Google Scholar 

  29. Trinidad, P., Benavides, D., Ruiz-Cortes, A., Segura, S., Jimenez, A.: Fama framework. In: Software Product Line Conference, 2008. SPLC’08. 12th International (Sept.), pp. 359–359

    Google Scholar 

  30. Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: Towards a benchmark and a comparison framework for combinatorial interaction testing of software product lines. CoRR abs/1401.5367 (2014)

    Google Scholar 

  31. Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Ralyté, J., Franch, X., Brinkkemper, S., Wrycza, S. (eds.) CAiSE. Volume 7328 of Lecture Notes in Computer Science, pp. 613–628. Springer, Berlin (2012)

    Google Scholar 

  32. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans. Softw. Eng. 40(7), 650–670 (2014)

    Article  Google Scholar 

  33. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Pledge: a product line editor and test generation tool. In: SPLC Workshops, pp. 126–129. ACM (2013)

    Google Scholar 

  34. Xu, Z., Cohen, M.B., Motycka, W., Rothermel, G.: Continuous test suite augmentation in software product lines. In: Proceedings SPLC, pp. 52–61 (2013)

    Google Scholar 

  35. Henard, C., Papadakis, M., Traon, Y.L.: Mutation-based generation of software product line test configurations. In: SSBSE, pp. 92–106 (2014)

    Google Scholar 

  36. Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)

    Article  Google Scholar 

  37. Perrouin, G., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Automated and scalable t-wise test case generation strategies for software product lines. In: ICST, pp. 459–468. IEEE Computer Society (2010)

    Google Scholar 

  38. Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In Bosch, J., Lee, J. (eds.) SPLC. Volume 6287 of Lecture Notes in Computer Science, pp. 196–210. Springer, Berlin (2010)

    Google Scholar 

  39. Hervieu, A., Baudry, B., Gotlieb, A.: Pacogen: automatic generation of pairwise test configurations from feature models. In: Dohi, T., Cukic, B. (eds.) ISSRE, pp. 120–129. IEEE (2011)

    Google Scholar 

  40. Lochau, M., Oster, S., Goltz, U., Schürr, A.: Model-based pairwise testing for feature interaction coverage in software product line engineering. Softw. Qual. J. 20(3–4), 567–604 (2012)

    Article  Google Scholar 

  41. Cichos, H., Oster, S., Lochau, M., Schürr, A.: Model-based coverage-driven test suite generation for software product lines. In: Whittle, J., Clark, T., Kühne, T. (eds.) MoDELS. Volume 6981 of Lecture Notes in Computer Science, pp. 425–439. Springer, Berlin (2011)

    Google Scholar 

  42. Calvagna, A., Gargantini, A., Vavassori, P.: Combinatorial testing for feature models using citlab. In: ICST Workshops, pp. 338–347 (2013)

    Google Scholar 

  43. Coello, C.C.: Evolutionary multi-objective optimization website. http://delta.cs.cinvestav.mx/ccoello/EMOO/

  44. Zhang, Y.: Search Based Software Engineering Repository. http://crestweb.cs.ucl.ac.uk/resources/sbse_repository/

  45. Coello, C.C., Lamont, G.B., Veldhuizen, D.A.: Evolutionary Algorithms for Solving Multi-objective Problems, 2nd edn. Genetic and Evolutionary Computation. Springer, Berlin (2007)

    Google Scholar 

  46. Deb, K.: Multi-objective Optimization Using Evolutionary Algorithms, 1st edn. Wiley, New York (June 2001)

    Google Scholar 

  47. Zitzler, E.: Evolutionary multiobjective optimization. In: Handbook of Natural Computing, pp. 871–904 (2012)

    Google Scholar 

  48. Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E.: Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2014, Beijing, China, 6–11 July 2014, pp. 387–396. IEEE (2014)

    Google Scholar 

  49. Lopez-Herrejon, R.E., Chicano, J.F., Ferrer, J., Egyed, A., Alba, E.: Multi-objective optimal test suite computation for software product line pairwise testing. In: ICSM, pp. 404–407. IEEE (2013)

    Google Scholar 

  50. Arito, F., Chicano, F., Alba, E.: On the application of sat solvers to the test suite minimization problem. In: Proceedings of the Symposium of Search Based Software Engineering. Volume 7515 of LNCS, pp. 45–59 (2012)

    Google Scholar 

  51. Wolsey, L.A.: Integer Programming. Wiley, New York (1998)

    Google Scholar 

  52. Sutton, A.M., Whitley, L.D., Howe, A.E.: A polynomial time computation of the exact correlation structure of k-satisfiability landscapes. In: Proceedings of GECCO, pp. 365–372 (2009)

    Google Scholar 

  53. Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: GECCO, pp. 1493–1500 (2013)

    Google Scholar 

  54. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: Proceedings of SPLC, pp. 62–71 (2013)

    Google Scholar 

  55. Marler, R., Arora, J.: Survey of multi-objective optimization methods for engineering. Struct. Multi. Optim. 26(6), 369–395 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  56. Cruz, J., Neto, P.S., Britto, R., Rabelo, R., Ayala, W., Soares, T., Mota, M.: Toward a hybrid approach to generate software product line portfolios. In: IEEE Congress on Evolutionary Computation, pp. 2229–2236 (2013)

    Google Scholar 

  57. Sayyad, A.S., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: a case study in software product lines. In: Proceedings of ICSE, pp. 492–501 (2013)

    Google Scholar 

  58. Sayyad, A.S., Ingram, J., Menzies, T., Ammar, H.: Scalable product line configuration: a straw to break the camel’s back. In: ASE, pp. 465–474 (2013)

    Google Scholar 

  59. Pascual, G.G., Lopez-Herrejon, R.E., Pinto, M., Fuentes, L., Egyed, A.: Applying multiobjective evolutionary algorithms to dynamic software product lines for reconfiguring mobile applications. J. Syst. Softw. (2015, to appear)

    Google Scholar 

  60. Olaechea, R., Rayside, D., Guo, J., Czarnecki, K.: Comparison of exact and approximate multi-objective optimization for software product lines. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 92–101. Florence, Italy, 15–19 Sept 2014. ACM (2014)

    Google Scholar 

  61. Murashkin, A., Antkiewicz, M., Rayside, D., Czarnecki, K.: Visualization and exploration of optimal variants in product line engineering. In: Proceedings of SPLC, pp. 111–115 (2013)

    Google Scholar 

  62. Dubinsky, Y., Rubin, J., Berger, T., Duszynski, S., Becker, M., Czarnecki, K.: An exploratory study of cloning in industrial software product lines. In: Cleve, A., Ricca, F., Cerioli, M. (eds.) CSMR, pp. 25–34. IEEE Computer Society (2013)

    Google Scholar 

  63. Chen, L., Babar, M.A.: A systematic review of evaluation of variability management approaches in software product lines. Inf. Softw. Technol. 53(4), 344–362 (2011)

    Article  Google Scholar 

  64. Lopez-Herrejon, R.E., Linsbauer, L., Galindo, J.A., Parejo, J.A., Benavides, D., Segura, S., Egyed, A.: An assessment of search-based techniques for reverse engineering feature models. J. Syst. Softw. Spec. Issue Search-Based Softw. Eng. (2015)

    Google Scholar 

  65. Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Feature model synthesis with genetic programming. In: Goues, C.L., Yoo, S. (eds.) Search-Based Software Engineering—6th International Symposium, SSBSE 2014, Fortaleza, Brazil, 26–29 Aug 2014. Proceedings. Volume 8636 of Lecture Notes in Computer Science, pp. 153–167. Springer, Berlin (2014)

    Google Scholar 

  66. She, S., Ryssel, U., Andersen, N., Wasowski, A., Czarnecki, K.: Efficient synthesis of feature models. Inf. Softw. Technol. 56(9), 1122–1143 (2014)

    Article  Google Scholar 

  67. Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., Liaaen, M.: Multi-objective test prioritization in software product line testing: an industrial case study. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 32–41. Florence, Italy, 15–19 Sept 2014. ACM (2014)

    Google Scholar 

  68. Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. 20(3–4), 605–643 (2012)

    Article  Google Scholar 

  69. Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Using feature model knowledge to speed up the generation of covering arrays. In: Gnesi, S., Collet, P., Schmid, K. (eds.) VaMoS, p. 16. ACM (2013)

    Google Scholar 

  70. Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Improving casa runtime performance by exploiting basic feature model analysis. CoRR abs/1311.7313 (2013)

    Google Scholar 

  71. Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6 (2014)

    Article  Google Scholar 

  72. Fischer, S., Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Enhancing clone-and-own with systematic reuse for developing software variants. 30th International Conference on Software Maintenance and Evolution (2014, to appear)

    Google Scholar 

  73. Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: SPLC (1), pp. 46–55 (2012)

    Google Scholar 

  74. Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In: Arnold, D.V. (ed.) Genetic and Evolutionary Computation Conference, GECCO’14, Vancouver, BC, Canada, 12–16 July 2014, pp. 1255–1262. ACM (2014)

    Google Scholar 

  75. Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 197–206. Florence, Italy, 15–19 Sept 2014. ACM (2014)

    Google Scholar 

  76. Sánchez, A.B., Segura, S., Cortés, A.R.: A comparison of test case prioritization criteria for software product lines. In: ICST, pp. 41–50 (2014)

    Google Scholar 

  77. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test., Verif. Reliab. 22(2), 67–120 (2012)

    Google Scholar 

Download references

Acknowledgments

This research is partially funded by the Austrian Science Fund (FWF) projects P 25513-N15, P 25289-N15, and Lise Meitner Fellowship M1421-N15, and by the Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2011-28194 and fellowship BES-2012-055967. It is also partially founded by projects 8.06/5.47.4142 (collaboration with the VSB-Tech. Univ. of Ostrava) and 8.06/5.47.4356 (Andalusian Agency of Public Works).

Author information

Authors and Affiliations

  1. Software Systems Engineering Institute, Johannes Kepler University, Linz, Austria

    Roberto E. Lopez-Herrejon & Alexander Egyed

  2. Universidad de Málaga, Andalucía Tech, Sevilla, Spain

    Javier Ferrer, Francisco Chicano & Enrique Alba

Authors
  1. Roberto E. Lopez-Herrejon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Javier Ferrer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francisco Chicano
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander Egyed
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Enrique Alba
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roberto E. Lopez-Herrejon .

Editor information

Editors and Affiliations

  1. Department of Electrical and Computer En, University of Alberta, Edmonton, Alberta, Canada

    Witold Pedrycz

  2. Innopolis University, Bolzano, Italy

    Giancarlo Succi

  3. Center for Applied Software Engineering, Bolzano, Italy

    Alberto Sillitti

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E. (2016). Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges. In: Pedrycz, W., Succi, G., Sillitti, A. (eds) Computational Intelligence and Quantitative Software Engineering. Studies in Computational Intelligence, vol 617. Springer, Cham. https://doi.org/10.1007/978-3-319-25964-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-25964-2_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-25962-8

  • Online ISBN: 978-3-319-25964-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation