Sabrina Souto
Marcelo d'Amorim
ABSTRACT
A software product line (SPL) is a family of programs that are differentiated by features --- increments in functionality. Systematically testing an SPL is challenging because it requires running each test of a test suite against a combinatorial number of programs. Feature models capture dependencies among features and can (1) reduce the space of programs to test and (2) enable accurate categorization of failing tests as failures of programs or the tests themselves, not as failures due to illegal combinations of features. In practice, sadly, feature models are not always available.
We introduce SPLif, the first approach for testing SPLs that does not require the a priori availability of feature models. Our insight is to use a profile of passing and failing test runs to quickly identify failures that are indicative of real problems in test or code rather than specious failures due to illegal feature combinations.
Experimental results on five SPLs and one large configurable system (GCC) demonstrate the effectiveness of our approach. SPLif enabled the discovery of five news bugs in GCC, three of which have already been fixed.
- Option definition files. https://gcc.gnu.org/onlinedocs/gccint/Options.html.Google Scholar
- R. Abreu, P. Zoeteweij, and A. J. C. v. Gemund. Spectrum-based multiple fault localization. In ASE, 2009. Google ScholarDigital Library
- R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. An evaluation of similarity coefficients for software fault localization. In PRDC, pages 39--46, 2006. Google ScholarDigital Library
- R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the accuracy of spectrum-based fault localization. In MUTATION, 2007. Google ScholarDigital Library
- M. Acher, A. Cleve, G. Perrouin, P. Heymans, C. Vanbeneden, P. Collet, and P. Lahire. On extracting feature models from product descriptions. In VaMoS, pages 45--54, 2012. Google ScholarDigital Library
- M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-based Prioritization in Software Product-line Testing. In SPLC, pages 197--206, 2014. Google ScholarDigital Library
- V. Alves, C. Schwanninger, L. Barbosa, A. Rashid, P. Sawyer, P. Rayson, C. Pohl, and A. Rummler. An exploratory study of information retrieval techniques in domain analysis. In SPLC, pages 67--76, 2008. Google ScholarDigital Library
- N. Andersen, K. Czarnecki, S. She, and A. Wasowski. Efficient synthesis of feature models. In SPLC, pages 106--115, 2012. Google ScholarDigital Library
- S. Apel and D. Beyer. Feature cohesion in software product lines: an exploratory study. In ICSE, pages 421--430, 2011. Google ScholarDigital Library
- S. Apel, A. von Rhein, P. Wendler, A. Groblinger, and D. Beyer. Strategies for Product-Line Verification: Case Studies and Experiments. In ICSE, pages 482--491, 2013. Google ScholarDigital Library
- D. S. Batory. Feature-oriented programming and the AHEAD tool suite. In ICSE, pages 702--703, 2004. Google ScholarDigital Library
- P. Borba, M. B. Cohen, A. Legay, and A. Wasowski. Analysis, Test and Verification in the Presence of Variability (Dagstuhl Seminar 13091). Dagstuhl Reports, 3(2):144--170, 2013.Google Scholar
- C. Boyapati, S. Khurshid, and D. Marinov. Korat: Automated testing based on Java predicates. In ISSTA, pages 123--133, 2002. Google ScholarDigital Library
- I. Cabral, M. B. Cohen, and G. Rothermel. Improving the testing and testability of software product lines. In SPLC, pages 241--255, 2010. Google ScholarDigital Library
- Y. Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, and J. Regehr. Taming compiler fuzzers. In PLDI, pages 197--208, 2013. Google ScholarDigital Library
- C. Kastner, A. Rhein, S. Erdweg, J. Pusch, S. Apel, T. Rendel, and K. Ostermann. Toward variability-aware testing. FOSD'12, pages 1--8, 2012. Google ScholarDigital Library
- K. Czarnecki and U. W. Eisenecker. Generative programming: methods, tools, and applications. ACM Press/Addison-Wesley, 2000. Google ScholarDigital Library
- K. Czarnecki and A. Wasowski. Feature diagrams and logics: There and back again. In SPLC, pages 23--34, 2007. Google ScholarDigital Library
- V. Dallmeier, C. Lindig, and A. Zeller. Lightweight defect localization for Java. In ECOOP, 2005. Google ScholarDigital Library
- M. d'Amorim, S. Lauterburg, and D. Marinov. Delta execution for efficient state-space exploration of object-oriented programs. IEEE TSE, 34(5):597--613, 2008. Google ScholarDigital Library
- J.-M. Davril, E. Delfosse, N. Hariri, M. Acher, J. Cleland-Huang, and P. Heymans. Feature model extraction from large collections of informal product descriptions. In ESEC/FSE, pages 290--300, 2013. Google ScholarDigital Library
- J. C. de Campos, R. Abreu, G. Fraser, and M. d'Amorim. Entropy-based Test Generation for Improved Fault Localization. In ASE, pages 257--267, 2013.Google ScholarDigital Library
- DejaGnu. GNU Testing Framework. gnu.org/s/dejagnu.Google Scholar
- Instructions to students on generating new tests for SPLs. http://www.cin.ufpe.br/~sfs/splif/experiments.html.Google Scholar
- FEST. Fixtures for Easy Software Testing. https://code.google.com/p/fest/.Google Scholar
- B. Garvin and M. Cohen. Feature interaction faults revisited: An exploratory study. In ISSRE, pages 90--99, 2011. Google ScholarDigital Library
- B. Garvin, M. Cohen, and M. B. Dwyer. Failure avoidance in configurable systems through feature locality. In Assurances for Self-Adaptive Systems, volume 7740 of LNCS, pages 266--296. 2013.Google ScholarCross Ref
- GCC. GNU Compiler Collection. gcc.gnu.org.Google Scholar
- GCC Documentation. Options That Control Optimization. gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#Optimize-Options.Google Scholar
- GCC Options. GCC Options. https://gcc.gnu.org/onlinedocs/gcc/Option-Summary.html.Google Scholar
- L. Ghandehari, Y. Lei, D. Kung, R. Kacker, and R. Kuhn. Fault localization based on failure-inducing combinations. In ISSRE, pages 168--177, Nov 2013.Google ScholarCross Ref
- E. N. Haslinger, R. E. Lopez-Herrejon, and A. Egyed. On extracting feature models from sets of valid feature combinations. In FASE, pages 53--67, 2013. Google ScholarDigital Library
- C. Henard, M. Papadakis, G. Perrouin, J. Klein, and Y. Le Traon. Towards automated testing and fixing of re-engineered feature models. In ICSE, pages 1245--1248, 2013. Google ScholarDigital Library
- Human-resource management system. 101Companies. http://101companies.org/wiki/@system.Google Scholar
- J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In ICSE, pages 467--477, 2002. Google ScholarDigital Library
- C. Kim, S. Khurshid, and D. Batory. Shared execution for efficiently testing product lines. In ISSRE, pages 221--230, 2012. Google ScholarDigital Library
- C. H. P. Kim, D. Batory, and S. Khurshid. Reducing Combinatorics in Testing Product Lines. In AOSD, pages 57--68, 2011. Google ScholarDigital Library
- C. H. P. Kim, E. Bodden, D. S. Batory, and S. Khurshid. Reducing Configurations to Monitor in a Software Product Line. In RV, pages 285--299, 2010. Google ScholarDigital Library
- C. H. P. Kim, D. Marinov, S. Khurshid, D. Batory, S. Souto, P. Barros, and M. d'Amorim. SPLat: Lightweight Dynamic Analysis for Reducing Combinatorics in Testing Configurable Systems. In ESEC/FSE, pages 257--267, 2013. Google ScholarDigital Library
- V. Le, M. Afshari, and Z. Su. Compiler validation via equivalence modulo inputs. In PLDI, pages 216--226, 2014. Google ScholarDigital Library
- P. Lengauer, V. Bitto, F. Angerer, P. Grünbacher, and H. Mössenböck. Where has all my memory gone?: Determining memory characteristics of product variants using virtual-machine-level monitoring. In VaMoS, pages 1--8, 2013. Google ScholarDigital Library
- R. E. Loepz-Herrejon, J. A. Galindo, D. Benavides, S. Segura, and A. Egyed. Reverse engineering feature models with evolutionary algorithms: An exploratory study. In SSBSE, pages 168--182, 2012. Google ScholarDigital Library
- R. E. Lopez-Herrejon and D. Batory. A standard problem for evaluating product-line methodologies. In GPCE, pages 10--24, 2001. Google ScholarDigital Library
- H. V. Nguyen, C. Kästner, and T. N. Nguyen. Exploring variability-aware execution for testing plugin-based web applications. In ICSE, pages 907--918, 2014. Google ScholarDigital Library
- K. Pohl, G. Böckle, and F. J. v. d. Linden. Software Product Line Engineering: Foundations, Principles and Techniques. Springer-Verlag, 2005. Google ScholarDigital Library
- X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: an empirical study of sampling and prioritization. In ISSTA, pages 75--86, 2008. Google ScholarDigital Library
- A. Rabkin and R. Katz. Static extraction of program configuration options. In ICSE, pages 131--140, 2011. Google ScholarDigital Library
- M. Renieris and S. P. Reiss. Fault localization with nearest neighbor queries. In ASE, 2003.Google ScholarCross Ref
- S. Segura, R. Hierons, D. Benavides, and A. Ruiz-Cortés. Automated test data generation on the analyses of feature models: A metamorphic testing approach. In ICST, pages 35--44, April 2010. Google ScholarDigital Library
- S. She, R. Lotufo, T. Berger, A. Wasowski, and K. Czarnecki. Reverse engineering feature models. In ICSE, pages 461--470, 2011. Google ScholarDigital Library
- J. Shi, M. B. Cohen, and M. B. Dwyer. Integration testing of software product lines using compositional symbolic execution. In FASE, pages 270--284, 2012. Google ScholarDigital Library
- Software Engineering Institute (SEI) SPL website. http://www.sei.cmu.edu/productlines/.Google Scholar
- C. Song, A. Porter, and J. S. Foster. itree: Efficiently Discovering High-Coverage Configurations Using Interaction Trees. In ICSE, pages 903--913, 2012. Google ScholarDigital Library
- J. Swanson, M. B. Cohen, M. B. Dwyer, B. J. Garvin, and J. Firestone. Beyond the rainbow: Self-adaptive failure avoidance in configurable systems. In FSE, pages 377--388, 2014. Google ScholarDigital Library
- E. Uzuncaova. Efficient Specification-based Testing Using Incremental Techniques. PhD thesis, Department of Electrical and Computer Engineering, University of Texas at Austin, Dec. 2008. Google ScholarDigital Library
- N. Weston, R. Chitchyan, and A. Rashid. A framework for constructing semantically composable feature models from natural language requirements. In SPLC, pages 211--220, 2009. Google ScholarDigital Library
- T. Xu, J. Zhang, P. Huang, J. Zheng, T. Sheng, D. Yuan, Y. Zhou, and S. Pasupathy. Do not blame users for misconfigurations. In SOSP, pages 244--259, 2013. Google ScholarDigital Library
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and Understanding Bugs in C Compilers. In PLDI, pages 283--294, 2011. Google ScholarDigital Library
- A. Zeller. Yesterday, my program worked. Today, it does not. Why? In FSE, 1999. Google ScholarDigital Library
- S. Zhang and M. D. Ernst. Automated diagnosis of software configuration errors. In ICSE, pages 312--321, San Francisco, CA, USA, May 2013. Google ScholarDigital Library
- S. Zhang and M. D. Ernst. Which Configuration Option Should I Change? In ICSE, pages 152--163, 2014. Google ScholarDigital Library
- Z. Zhang and J. Zhang. Characterizing failure-causing parameter interactions by adaptive testing. In ISSTA, pages 331--341, 2011. Google ScholarDigital Library
- Faster bug detection for software product lines with incomplete feature models
Recommendations
Continuous test suite augmentation in software product lines
SPLC '13: Proceedings of the 17th International Software Product Line ConferenceSoftware Product Line (SPL) engineering offers several advantages in the development of families of software products. There is still a need, however, to generate test cases for individual products in product lines more efficiently. In this paper we ...
A Comparison of Test Case Prioritization Criteria for Software Product Lines
ICST '14: Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and ValidationSoftware Product Line (SPL) testing is challenging due to the potentially huge number of derivable products. To alleviate this problem, numerous contributions have been proposed to reduce the number of products to be tested while still having a good ...
Multi-objective test generation for software product lines
SPLC '13: Proceedings of the 17th International Software Product Line ConferenceSoftware Products Lines (SPLs) are families of products sharing common assets representing code or functionalities of a software product. These assets are represented as features, usually organized into Feature Models (FMs) from which the user can ...
Comments