skip to main content
10.1145/3377024.3377025acmotherconferencesArticle/Chapter ViewAbstractPublication PagesvamosConference Proceedingsconference-collections
research-article

Evaluating #SAT solvers on industrial feature models

Published: 06 February 2020 Publication History

Abstract

Configurable systems are widely used for families of products that share multiple configuration options. These systems often induce a large configuration space. Handling the variability of such a system is difficult without being able to measure its complexity. Several methods depend on computing the number of valid configurations, such as estimating the effort of an update or effectively reducing the variability of a system. In many cases, it is possible to map a configurable system to propositional logic. Therefore, we use #SAT in order to evaluate variability of such systems. A #SAT solver computes the number of valid assignments of a propositional formula. However, this problem is even harder than SAT. The main contribution of our work is an investigation of the scalability of off-the-shelf #SAT solvers on industrial feature models. Additionally, we examine the correlation between size of a system and the runtime of a solver computing the number of valid configurations. In this paper, we empirically evaluate nine publicly available #SAT solvers on 127 industrial feature models. Our results indicate that current solvers master a majority of the evaluated systems. However, there are large models, for which none of the evaluated solvers scales. Nevertheless, there are even larger and more complex systems for which the solvers scale.

References

[1]
Mathieu Acher, Philippe Collet, Philippe Lahire, and Robert B France. 2011. Slicing feature models. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 424--427.
[2]
Mathieu Acher, Philippe Collet, Philippe Lahire, and Robert B France. 2013. Familiar: A domain-specific language for large scale management of feature models. Science of Computer Programming 78, 6 (2013), 657--681.
[3]
Sofia Ananieva. 2016. Explaining Defects and Identifying Dependencies in Interrelated Feature Models. Ph.D. Dissertation. Institute of Software.
[4]
Sofia Ananieva, Matthias Kowal, Thomas Thüm, and Ina Schaefer. 2016. Implicit constraints in partial feature models. In Proceedings of the 7th International Workshop on Feature-Oriented Software Development. ACM, 18--27.
[5]
Ebrahim Bagheri, Tommaso Di Noia, Dragan Gasevic, and Azzurra Ragone. 2012. Formalizing interactive staged feature model configuration. Journal of Software: Evolution and Process 24, 4 (2012), 375--400.
[6]
Roberto J Bayardo Jr and Joseph Daniel Pehoushek. 2000. Counting models using connected components. In AAAI/IAAI. 157--162.
[7]
David Benavides, Sergio Segura, and Antonio Ruiz-Cortés. 2010. Automated analysis of feature models 20 years later: A literature review. Information Systems 35, 6 (2010), 615--636.
[8]
David Benavides, Sergio Segura, Pablo Trinidad, and Antonio Ruiz-Cortés. 2005. Using Java CSP solvers in the automated analyses of feature models. In International Summer School on Generative and Transformational Techniques in Software Engineering. Springer, 399--408.
[9]
David Benavides, Pablo Trinidad, and Antonio Ruiz-Cortés. 2005. Automated reasoning on feature models. In International Conference on Advanced Information Systems Engineering. Springer, 491--503.
[10]
Armin Biere. 2008. PicoSAT essentials. Journal on Satisfiability, Boolean Modeling and Computation 4 (2008), 75--97.
[11]
Armin Biere, Marijn Heule, and Hans van Maaren. 2009. Handbook of satisfiability. Vol. 185. IOS press.
[12]
Jan Burchard, Tobias Schubert, and Bernd Becker. 2015. Laissez-faire caching for parallel# SAT solving. In International Conference on Theory and Applications of Satisfiability Testing. Springer, 46--61.
[13]
Supratik Chakraborty, Kuldeep S Meel, and Moshe Y Vardi. 2013. A scalable approximate model counter. In International Conference on Principles and Practice of Constraint Programming. Springer, 200--216.
[14]
Paul C Clements, John D McGregor, and Sholom G Cohen. 2005. The structured intuitive model for product line economics (SIMPLE). Technical Report. CARNEGIE-MELLON UNIV PITTSBURGH PA SOFTWARE ENGINEERING INST.
[15]
Adnan Darwiche. 2004. New advances in compiling CNF to decomposable negation normal form. In Proceedings of the 16th European Conference on Artificial Intelligence. Citeseer, 318--322.
[16]
Adnan Darwiche and Pierre Marquis. 2002. A knowledge compilation map. Journal of Artificial Intelligence Research 17 (2002), 229--264.
[17]
David Fernandez-Amoros, Ruben Heradio, Jose A Cerrada, and Carlos Cerrada. 2014. A scalable approach to exact model and commonality counting for extended feature models. IEEE Transactions on Software Engineering 40, 9 (2014), 895--910.
[18]
Carla P Gomes, Ashish Sabharwal, and Bart Selman. 2006. Model counting: A new strategy for obtaining good bounds. In AAAI. 54--61.
[19]
Tarik Hadzic, Sathiamoorthy Subbarayan, Rune M Jensen, Henrik R Andersen, Jesper Møller, and Henrik Hulgaard. 2004. Fast backtrack-free product configuration using a precompiled solution space representation. small 10, 1 (2004), 3.
[20]
Ruben Heradio, David Fernandez-Amoros, Jose A Cerrada, and Ismael Abad. 2013. A literature review on feature diagram product counting and its usage in software product line economic models. International Journal of Software Engineering and Knowledge Engineering 23, 08 (2013), 1177--1204.
[21]
Jinbo Huang and Adnan Darwiche. 2004. Using DPLL for efficient OBDD construction. In International Conference on Theory and Applications of Satisfiability Testing. Springer, 157--172.
[22]
Jinbo Huang and Adnan Darwiche. 2005. DPLL with a trace: From SAT to knowledge compilation. In IJCAI, Vol. 5. 156--162.
[23]
David S Johnson. 1992. The NP-completeness column: an ongoing guide. Journal of algorithms 13, 3 (1992), 502--524.
[24]
Christian Kastner, Thomas Thum, Gunter Saake, Janet Feigenspan, Thomas Leich, Fabian Wielgorz, and Sven Apel. 2009. FeatureIDE: A tool framework for feature-oriented software development. In Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, 611--614.
[25]
Alexander Knüppel, Thomas Thüm, Stephan Mennicke, Jens Meinicke, and Ina Schaefer. 2017. Is there a mismatch between real-world feature models and product-line research?. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 291--302.
[26]
Sebastian Krieter, Reimar Schröter, Thomas Thüm, Wolfram Fenske, and Gunter Saake. 2016. Comparing algorithms for efficient feature-model slicing. In Proceedings of the 20th International Systems and Software Product Line Conference. ACM, 60--64.
[27]
Andreas Kübler, Christoph Zengler, and Wolfgang Küchlin. 2010. Model counting in product configuration. arXiv preprint arXiv:1007.1024 (2010).
[28]
Jean-Marie Lagniez and Pierre Marquis. 2017. An Improved Decision-DNNF Compiler. In IJCAI. 667--673.
[29]
Marcilio Mendonca, Moises Branco, and Donald Cowan. 2009. SPLOT: software product lines online tools. In Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented programming systems languages and applications. ACM, 761--762.
[30]
Marcilio Mendonca, Andrzej Wąsowski, and Krzysztof Czarnecki. 2009. SAT-based analysis of feature models is easy. In Proceedings of the 13th International Software Product Line Conference. Carnegie Mellon University, 231--240.
[31]
Marcilio Mendonca, Andrzej Wasowski, Krzysztof Czarnecki, and Donald Cowan. 2008. Efficient compilation techniques for large scale feature models. In Proceedings of the 7th international conference on Generative programming and component engineering. ACM, 13--22.
[32]
Christian Muise, Sheila McIlraith, J Christopher Beck, and Eric Hsu. 2010. Fast d-DNNF Compilation with sharpSAT. In Workshops at the twenty-fourth AAAI conference on artificial intelligence.
[33]
Daniel-Jesus Munoz, Jeho Oh, Mónica Pinto, Lidia Fuentes, and Don Batory. 2019. Uniform random sampling product configurations of feature models that have numerical features. In Proceedings of the 23rd International Systems and Software Product Line Conference-Volume A. ACM, 39.
[34]
Macha Nikolskaia and Antoine Rauzy. 1998. Heuristics for BDD handling of sum-of-products formulae. In Proceedings of the European Safety and Reliability Association Conference, ESREL'98. Citeseer.
[35]
Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding near-optimal configurations in product lines by random sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 61--71.
[36]
Jeho Oh, Paul Gazzillo, Don Batory, Marijn Heule, and Maggie Myers. 2019. Uniform sampling from kconfig feature models. The University of Texas at Austin, Department of Computer Science, Tech. Rep. TR-19-02 (2019).
[37]
Richard Pohl, Kim Lauenroth, and Klaus Pohl. 2011. A performance comparison of contemporary algorithmic approaches for automated analysis operations on feature models. In Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 313--322.
[38]
Tian Sang, Fahiem Bacchus, Paul Beame, Henry A Kautz, and Toniann Pitassi. 2004. Combining Component Caching and Clause Learning for Effective Model Counting. SAT 4 (2004), 7th.
[39]
Tian Sang, Paul Beame, and Henry Kautz. 2005. Heuristics for fast exact model counting. In International Conference on Theory and Applications of Satisfiability Testing. Springer, 226--240.
[40]
Reimar Schröter, Sebastian Krieter, Thomas Thüm, Fabian Benduhn, and Gunter Saake. 2016. Feature-model interfaces: the highway to compositional analyses of highly-configurable systems. In Proceedings of the 38th International Conference on Software Engineering. ACM, 667--678.
[41]
Thomas Thum, Christian Kastner, Sebastian Erdweg, and Norbert Siegmund. 2011. Abstract features in feature modeling. In 2011 15th International Software Product Line Conference. IEEE, 191--200.
[42]
Marc Thurley. 2006. sharpSAT-counting models with advanced component caching and implicit BCP. In International Conference on Theory and Applications of Satisfiability Testing. Springer, 424--429.
[43]
Takahisa Toda and Takehide Soh. 2016. Implementing efficient all solutions SAT solvers. Journal of Experimental Algorithmics (JEA) 21 (2016), 1--12.
[44]
Lin Xu, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2008. SATzilla: portfolio-based algorithm selection for SAT. Journal of artificial intelligence research 32 (2008), 565--606.

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • (2023)Views on Edits to Variational SoftwareProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608985(141-152)Online publication date: 28-Aug-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
VaMoS '20: Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems
February 2020
184 pages
ISBN:9781450375016
DOI:10.1145/3377024
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 February 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. #SAT
  2. configurable systems
  3. configuration counting
  4. feature models
  5. model counting
  6. product lines

Qualifiers

  • Research-article

Conference

VaMoS '20

Acceptance Rates

Overall Acceptance Rate 66 of 147 submissions, 45%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)46
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
  • (2024)On the Use of Multi-valued Decision Diagrams to Count Valid Configurations of Feature ModelsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672594(96-106)Online publication date: 2-Sep-2024
  • (2023)Views on Edits to Variational SoftwareProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608985(141-152)Online publication date: 28-Aug-2023
  • (2023)Quantum Computing for Feature Model AnalysisProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608971(1-7)Online publication date: 28-Aug-2023
  • (2023)Can Quantum Computing Improve Uniform Random Sampling of Large Configuration Spaces?2023 IEEE/ACM 4th International Workshop on Quantum Software Engineering (Q-SE)10.1109/Q-SE59154.2023.00012(34-41)Online publication date: May-2023
  • (2023)A modular metamodel and refactoring rules to achieve software product line interoperabilityJournal of Systems and Software10.1016/j.jss.2022.111579197:COnline publication date: 1-Mar-2023
  • (2023)Evaluating state-of-the-art # SAT solvers on industrial configuration spacesEmpirical Software Engineering10.1007/s10664-022-10265-928:2Online publication date: 13-Jan-2023
  • (2023)On the benefits of knowledge compilation for feature-model analysesAnnals of Mathematics and Artificial Intelligence10.1007/s10472-023-09906-692:5(1013-1050)Online publication date: 6-Nov-2023
  • (2022)Tseitin or not Tseitin? The Impact of CNF Transformations on Feature-Model AnalysesProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556938(1-13)Online publication date: 10-Oct-2022
  • (2022)Scalable Sampling of Highly-Configurable Systems: Generating Random Instances of the Linux KernelProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556899(1-12)Online publication date: 10-Oct-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media