Paul Gazzillo
Ugur Koc
ThanhVu Nguyen
ABSTRACT
The complexity of configurable systems has grown immensely, and it is only getting more complex. Such systems are a challenge for software testing and maintenance, because bugs and other defects can and do appear in any configuration. One common requirement for many development tasks is to identify the configurations that lead to a given defect or some other program behavior. We distill this requirement down to a challenge question: given a program location in a source file, what are valid configurations that include the location? The key obstacle is scalability. When there are thousands of configuration options, enumerating all combinations is exponential and infeasible. We provide a set of target programs of increasing difficulty and variations on the challenge question so that submitters of all experience levels can try out solutions. Our hope is to engage the community and stimulate new and interesting approaches to the problem of analyzing configurations.
- Iago Abal, Claus Brabrand, and Andrzej Wasowski. 2014. 42 Variability Bugs in the Linux Kernel: A Qualitative Analysis. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 421--432. Google Scholar
Digital Library
- Bram Adams, Herman Tromp, Kris De Schutter, and Wolfgang De Meuter. 2007. Design recovery and maintenance of build systems. In 23rd IEEE International Conference on Software Maintenance (ICSM 2007), October 2-5, 2007, Paris, France. 114--123.Google ScholarCross Ref
- Sven Apel, Sergiy Kolesnikov, Norbert Siegmund, Christian Kästner, and Brady Garvin. 2013. Exploring Feature Interactions in the Wild: The New Feature-interaction Challenge. In Proceedings of the 5th International Workshop on Feature-Oriented Software Development (FOSD '13). ACM, New York, NY, USA, 1--8. Google ScholarDigital Library
- Thorsten Berger, Steven She, Rafael Lotufo, Krzysztof Czarnecki, and Andrzej Wasowski. 2010. Feature-to-code Mapping in Two Large Product Lines. In Proceedings of the 14th International Conference on Software Product Lines: Going Beyond (SPLC'10). Springer-Verlag, Berlin, Heidelberg, 498--499. http://dl.acm.org/citation.cfm?id=1885639.1885698 Google ScholarDigital Library
- Renée C. Bryce and Charles J. Colbourn. 2006. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Information and Software Technology 48, 10 (Oct. 2006), 960--970.Google ScholarCross Ref
- D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton. 1996. The combinatorial design approach to automatic test generation. IEEE Software 13, 5 (Sept. 1996), 83--88. Google ScholarDigital Library
- M.B. Cohen, Amanda Swearngin, Brady Garvin, Jacob Swanson, Justyna Petke, Kaylei Burke, Katie Macias, Ronald Decker, Wayne Motycka, and Zhen and Wang. {n. d.}. Combinatorial Interaction Testing Portal. ({n. d.}). http://cse.unl.edu/cit-portal, Accessed on 2018-01-16.Google Scholar
- M. B. Cohen, P. B. Gibbons, W. B. Mugridge, and C. J. Colbourn. 2003. Constructing test suites for interaction testing. In 25th International Conference on Software Engineering, 2003. Proceedings. 38--48. Google ScholarDigital Library
- Krzysztof Czarnecki and Krzysztof Pietroszek. 2006. Verifying Feature-based Model Templates Against Well-formedness OCL Constraints. In Proceedings of the 5th International Conference on Generative Programming and Component Engineering (GPCE '06). ACM, New York, NY, USA, 211--220. Google ScholarDigital Library
- Gulsen Demiroz and Cemal Yilmaz. 2012. Cost-aware combinatorial interaction testing. In Proceedings of the Internatinoal Conference on Advances in System Testing and Validation Lifecycles. 9--16.Google Scholar
- Christian Dietrich, Reinhard Tartler, Wolfgang Schröder-Preikschat, and Daniel Lohmann. 2012. A Robust Approach for Variability Extraction from the Linux Build System. In Proceedings of the 16th International Software Product Line Conference - Volume 1 (SPLC '12). ACM, New York, NY, USA, 21--30. Google ScholarDigital Library
- Christian Dietrich, Reinhard Tartler, Wolfgang Schröder-Preikshat, and Daniel Lohmann. 2012. A robust approach for variability extraction from the Linux build system. 21--30. Google ScholarDigital Library
- Emine Dumlu, Cemal Yilmaz, Myra B. Cohen, and Adam Porter. 2011. Feedback Driven Adaptive Combinatorial Testing. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA '11). ACM, New York, NY, USA, 243--253. Google ScholarDigital Library
- Alejandra Garrido and Ralph Johnson. 2005. Analyzing Multiple Configurations of a C Program. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM '05). IEEE Computer Society, Washington, DC, USA, 379--388. Google ScholarDigital Library
- Paul Gazzillo. 2017. Kmax: Finding All Configurations of Kbuild Makefiles Statically. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 279--290. Google ScholarDigital Library
- Paul Gazzillo and Robert Grimm. 2012. SuperC: Parsing All of C by Taming the Preprocessor. In Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI '12). ACM, New York, NY, USA, 323--334. Google ScholarDigital Library
- Christian Kästner. 2016. KconfigReader. https://github.com/ckaestne/kconfigreader. (2016). {Online; accessed 12-Jan-2018}.Google Scholar
- Christian Kästner, Paolo G. Giarrusso, Tillmann Rendel, Sebastian Erdweg, Klaus Ostermann, and Thorsten Berger. 2011. Variability-aware Parsingin the Presence of Lexical Macros and Conditional Compilation. In Proceedings of the 2011 ACM International Conference on Object Oriented Programming Systems Languages and Applications (OOPSLA '11). ACM, New York, NY, USA, 805--824. Google ScholarDigital Library
- Author: Richard Kuhn (NIST), Author: Raghu Kacker (NIST), and Author: Yu Lei (UTA). {n. d.}. Advanced Combinatorial Test Methods for System Reliability. ({n. d.}). https://csrc.nist.gov/publications/detail/journal-article/2010/advanced-combinatorial-test-methods-for-system-reliability DOI:https://content.csrc.nist.gov/publications/detail/journal-article/2010/advanced-combinatorial-test-methods-for-system-reliability.Google Scholar
- Max Lillack, Christian Kästner, and Eric Bodden. 2014. Tracking Load-time Configuration Options. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE '14). ACM, New York, NY, USA, 445--456. Google ScholarDigital Library
- M. Lillack, C. KÃd'stner, and E. Bodden. 2017. Tracking Load-time Configuration Options. IEEE Transactions on Software Engineering PP, 99 (2017), 1--1.Google Scholar
- Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Rohit Gheyi, and Sven Apel. 2016. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proceedings of the 38th International Conference on Software Engineering (ICSE). ACM Press, New York, NY, 643--654. Google ScholarDigital Library
- S. Nadi, T. Berger, C. KÃd'stner, and K. Czarnecki. 2015. Where Do Configuration Constraints Stem From? An Extraction Approach and an Empirical Study. IEEE Transactions on Software Engineering 41, 8 (Aug. 2015), 820--841.Google ScholarDigital Library
- Sarah Nadi and Ric Holt. 2012. Mining Kbuild to Detect Variability Anomalies in Linux. In Proceedings of the 2012 16th European Conference on Software Maintenance and Reengineering (CSMR '12). IEEE Computer Society, Washington, DC, USA, 107--116. Google ScholarDigital Library
- ThanhVu Nguyen, Ugur Koc, Javran Cheng, Jeffrey S. Foster, and Adam A. Porter. 2016. iGen: Dynamic Interaction Inference for Configurable Software. In Foundations of Software Engineering (FSE). ACM, 655--665. Google ScholarDigital Library
- Changhai Nie and Hareton Leung. 2011. A Survey of Combinatorial Testing. ACM Comput. Surv. 43, 2 (Feb. 2011), 11:1--11:29. Google ScholarDigital Library
- Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding Near-optimal Configurations in Product Lines by Random Sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 61--71. Google ScholarDigital Library
- Maxime Ouellet, Ettore Merlo, Neset Sozen, and Martin Gagnon. 2012. Locating Features in Dynamically Configured Avionics Software. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, Piscataway, NJ, USA, 1453--1454. http://dl.acm.org/citation.cfm?id=2337223.2337449 Google ScholarDigital Library
- Elnatan Reisner, Charles Song, Kin-Keung Ma, Jeffrey S. Foster, and Adam Porter. 2010. Using Symbolic Evaluation to Understand Behavior in Configurable Software Systems. In Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering - Volume 1 (ICSE '10). ACM, New York, NY, USA, 445--454. Google ScholarDigital Library
- A. v Rhein, A. Grebhahn, S. Apel, N. Siegmund, D. Beyer, and T. Berger. 2015. Presence-Condition Simplification in Highly Configurable Systems. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. 178--188. Google ScholarDigital Library
- Norbert Siegmund, Stefan Sobernig, and Sven Apel. 2017. Attributed Variability Models: Outside the Comfort Zone. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2017). ACM, New York, NY, USA, 268--278. Google ScholarDigital Library
- Charles Song, Adam Porter, and Jeffrey S. Foster. 2012. iTree: Efficiently Discovering High-coverage Configurations Using Interaction Trees. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, Piscataway, NJ, USA, 903--913. http://dl.acm.org/citation.cfm?id=2337223.2337329 Google ScholarDigital Library
- Jacob Swanson, Myra B. Cohen, Matthew B. Dwyer, Brady J. Garvin, and Justin Firestone. 2014. Beyond the Rainbow: Self-adaptive Failure Avoidance in Configurable Systems. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 377--388. Google ScholarDigital Library
- Kuo-Chung Tai and Yu Lei. 2002. A test generation strategy for pairwise testing. IEEE Transactions on Software Engineering 28, 1 (Jan. 2002), 109--111. Google ScholarDigital Library
- Ahmed Tamrawi, Hoan Anh Nguyen, Hung Viet Nguyen, and Tien N. Nguyen. 2012. Build Code Analysis with Symbolic Evaluation. In Proceedings of the 34th International Conference on Software Engineering (ICSE '12). IEEE Press, Piscataway, NJ, USA, 650--660. http://dl.acm.org/citation.cfm?id=2337223.2337300 Google ScholarDigital Library
- Thomas ThÃijm, Sven Apel, Christian Kästner, Martin Kuhlemann, Ina Schaefer, and Gunter Saake. 2004. Analysis strategies for software product lines. (2004).Google Scholar
- C. Yilmaz, M. B. Cohen, and A. A. Porter. 2006. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering 32, 1 (Jan. 2006), 20--34. Google ScholarDigital Library
- C. Yilmaz, S. FouchÃl', M. B. Cohen, A. Porter, G. Demiroz, and U. Koc. 2014. Moving Forward with Combinatorial Interaction Testing. Computer 47, 2 (Feb. 2014), 37--45. Google ScholarDigital Library
- X. Yuan, M. B. Cohen, and A. M. Memon. 2011. GUI Interaction Testing: Incorporating Event Context. IEEE Transactions on Software Engineering 37, 4 (July 2011), 559--574. Google ScholarDigital Library
- Sai Zhang and Michael D. Ernst. 2014. Which Configuration Option Should I Change?. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, New York, NY, USA, 152--163. Google ScholarDigital Library
Index Terms
- Localizing configurations in highly-configurable systems
Recommendations
Automating Test Reuse for Highly Configurable Software
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume ADealing with highly configurable systems is generally very complex. Hundreds of different analysis techniques have been conceived to deal with different aspects of configurable systems. One large focal point is the testing of configurable software. This ...
Automated test reuse for highly configurable software
AbstractDealing with highly configurable systems is generally very complex. Researchers and practitioners have conceived hundreds of different analysis techniques to deal with different aspects of configurable systems. One large focal point is the testing ...
Limiting recertification in highly configurable systems: analyzing interactions and isolation among configuration options
HotSoS '14: Proceedings of the 2014 Symposium and Bootcamp on the Science of SecurityIn highly configurable systems the configuration space is too big for (re-)certifying every configuration in isolation. In this project, we combine software analysis with network analysis to detect which configuration options interact and which have ...
Comments