research-article

Uniform and scalable SAT-sampling for configurable systems

Authors:

David Fernandez-Amoros,

José A. Galindo,

David BenavidesAuthors Info & Claims

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

Article No.: 17, Pages 1 - 11

https://doi.org/10.1145/3382025.3414951

Published: 19 October 2020 Publication History

Abstract

Several relevant analyses on configurable software systems remain intractable because they require examining vast and highly-constrained configuration spaces. Those analyses could be addressed through statistical inference, i.e., working with a much more tractable sample that later supports generalizing the results obtained to the entire configuration space. To make this possible, the laws of statistical inference impose an indispensable requirement: each member of the population must be equally likely to be included in the sample, i.e., the sampling process needs to be "uniform". Various SAT-samplers have been developed for generating uniform random samples at a reasonable computational cost. Unfortunately, there is a lack of experimental validation over large configuration models to show whether the samplers indeed produce genuine uniform samples or not. This paper (i) presents a new statistical test to verify to what extent samplers accomplish uniformity and (ii) reports the evaluation of four state-of-the-art samplers: Spur, QuickSampler, Unigen2, and Smarch. According to our experimental results, only Spur satisfies both scalability and uniformity.

References

[1]

Dimitris Achlioptas, Zayd S. Hammoudeh, and Panos Theodoropoulos. 2018. Fast Sampling of Perfectly Uniform Satisfying Assignments. In 21st International Conference on Theory and Applications of Satisfiability Testing (SAT). Oxford, UK, 135--147.

[2]

Mauricio Alférez, Mathieu Acher, José Galindo, Benoit Baudry, and David Benavides. 2019. Modeling variability in the video domain: language and experience report. Software Quality Journal 27, 1 (2019), 307--347.

Digital Library

[3]

Fadi A. Aloul, Igor L. Markov, and Karem A. Sakallah. 2003. FORCE: A Fast and Easy-to-Implement Variable-Ordering Heuristic. In ACM Great Lakes Symposium on VLSI (GLSVLSI). Washington, D. C., USA, 116--119.

[4]

Fadi A. Aloul, Igor L. Markov, and Karem A. Sakallah. 2004. MINCE: A Static Global Variable-Ordering Heuristic for SAT Search and BDD Manipulation. Journal of Universal Computer Science 10, 12 (2004), 1562--1596.

[5]

Juliana Alves Pereira, Mathieu Acher, Hugo Martin, and Jean-Marc Jézéquel. 2020. Sampling Effect on Performance Prediction of Configurable Systems: A Case Study. In ACM/SPEC International Conference on Performance Engineering (ICPE). Edmonton AB, Canada, 277--288.

[6]

Don S. Batory. 2005. Feature Models, Grammars, and Propositional Formulas. In 9th Software Product Line Conference (SPLC). Rennes, France, 7--20.

Digital Library

[7]

Mihir Bellare, Oded Goldreich, and Erez Petrank. 2000. Uniform Generation of NP-Witnesses Using an NP-Oracle. Information and Computation 163, 2 (2000), 510--526.

Digital Library

[8]

Armin Biere, Marijn Heule, Hans van Maaren, and Toby Walsh. 2009. Handbook of Satisfiability: Volume 185 Frontiers in Artificial Intelligence and Applications. IOS Press.

[9]

Supratik Chakraborty, Daniel J. Fremont, Kuldeep S. Meel, Sanjit A. Seshia, and Moshe Y. Vardi. 2015. On Parallel Scalable Uniform SAT Witness Generation. In 21st International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). London, UK, 304--319.

[10]

Sourav Chakraborty and Kuldeep S. Meel. 2019. On testing of Uniform Samplers. In 33rd Conference on Artificial Intelligence (AAAI). Honolulu, Hawaii, USA, 7777--7784.

[11]

Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. 2013. A Scalable and Nearly Uniform Generator of SAT Witnesses. In 25th International Conference on Computer Aided Verification (CAV). Saint Petersburg, Russia, 608--623.

[12]

Laura M. Chihara and Tim C. Hesterberg. 2011. Mathematical Statistics with Resampling and R. Peds-R-Us Medical Education, Llc.

[13]

Jacon Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences. Routledge.

[14]

Ralph B. D'Agostino. 1986. Goodness-of-Fit-Techniques. CRC Press.

[15]

Leonardo de Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS). Budapest, Hungary, 337--340.

[16]

Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, and Koushik Sen. 2018. Efficient Sampling of SAT Solutions for Testing. In 40th International Conference on Software Engineering (ICSE) (Gothenburg, Sweden). ACM, New York, NY, USA, 549--559.

[17]

David Fernandez-Amoros, Ruben Heradio, Jose A. Cerrada, and Carlos Cerrada. 2014. A Scalable Approach to Exact Model and Commonality Counting for Extended Feature Models. IEEE Transactions on Software Engineering 40, 9 (2014), 895--910.

[18]

David Fernandez-Amoros, Ruben Heradio, Christoph Mayr-Dorn, and Alexander Egyed. 2019. A Kconfig translation to logic with one-way validation system. In 23rd International Systems and Software Product Line Conference (SPLC). Paris, France, 303--308.

Digital Library

[19]

Ivo Grosse, Pedro Bernaola-Galvan, Pedro Carpena, Ramon Roman-Roldan, Jose Oliver, and H. Eugene Stanley. 2002. Analysis of symbolic sequences using the Jensen-Shannon divergence. Physical Review E 65, 2 (2002), 041905/1--0419051/16.

[20]

Jianmei Guo, Dingyu Yang, Norbert Siegmund, Sven Apel, Atrisha Sarkar, Pavel Valov, Krzysztof Czarnecki, Andrzej Wasowski, and Huiqun Yu. 2018. Dataefficient performance learning for configurable systems. Empirical Software Engineering 23, 3 (2018), 1826--1867.

Digital Library

[21]

Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. Empirical Software Engineering 24, 2 (2019), 674--717.

Digital Library

[22]

Ruben Heradio, David Fernandez-Amoros, Christoph Mayr-Dorn, and Alexander Egyed. 2019. Supporting the Statistical Analysis of Variability Models. In 41st International Conference on Software Engineering (ICSE) (Montreal, Quebec, Canada). Montréal, Canada, 843--853.

[23]

Michael Hübner and Jürgen Becker. 2011. Multiprocessor System-on-Chip: Hardware Design and Tool Integration. Springer.

[24]

Alexander Ivrii, Sharad Malik, Kuldeep S. Meel, and Moshe Y. Vardi. 2016. On computing minimal independent support and its applications to sampling and counting. Constraints 21, 1 (2016), 41--58.

Digital Library

[25]

Robert Kabacoff. 2011. R in Action: Data analysis and graphics with R. Manning Publications.

[26]

C. Kaltenecker, A. Grebhahn, N. Siegmund, J. Guo, and S. Apel. 2019. Distance-Based Sampling of Software Configuration Spaces. In 41st International Conference on Software Engineering (ICSE). Montreal, Canada, 1084--1094.

[27]

Daniel Kaplan. 2012. Statistical Modeling: A Fresh Approach. Peds-R-Us Medical Education, Llc.

[28]

Donald E. Knuth. 2009. The Art of Computer Programming, Volume 4, Fascicle 1: Bitwise Tricks & Techniques; Binary Decision Diagrams. Addison-Wesley Professional.

[29]

Sergiy Kolesnikov, Norbert Siegmund, Christian Kästner, Alexander Grebhahn, and Sven Apel. 2019. Tradeoffs in modeling performance of highly configurable software systems. Software & Systems Modeling 18, 3 (2019), 2265--2283.

Digital Library

[30]

Sebastian Krieter. 2019. Enabling Efficient Automated Configuration Generation and Management. In 23rd International Systems and Software Product Line Conference (SPLC). Paris, France, 215--221.

[31]

Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: Yet Another Sampling Algorithm. In International Working Conference on Variability Modelling of Software-Intensive System (VaMoS). Association for Computing Machinery, Magdeburg, Germany.

Digital Library

[32]

Sebastian Krieter, Thomas Thüm, Sandro Schulze, Reimar Schröter, and Gunter Saake. 2018. Propagating configuration decisions with modal implication graphs. In 40th International Conference on Software Engineering (ICSE). Gothenburg, Sweden, 898--909.

Digital Library

[33]

Daniel Lakens. 2013. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology 4, 863 (2013), 1--12.

[34]

Jianhua Lin. 1991. Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory 37, 1 (Jan 1991), 145--151.

Digital Library

[35]

Christoph Meinel and Thorsten Theobald. 1998. Algorithms and Data Structures in VLSI Design: OBDD - Foundations And Applications. Springer, Chapter Ch. 9: Optimizing the Variable Order, 145--170.

[36]

Marcilio Mendonça. 2009. Efficient Reasoning Techniques for Large Scale Feature Models. Ph.D. Dissertation. University of Waterloo.

[37]

Daniel-Jesus Muñoz, Jeho Oh, Mónica Pinto, Lidia Fuentes, and Don Batory. 2019. Uniform Random Sampling Product Configurations of Feature Models That Have Numerical Features. In 23rd International Systems and Software Product Line Conference (SPLC). Paris, France, 289--301.

[38]

Vivek Nair, Tim Menzies, Norbert Siegmund, and Sven Apel. 2017. Using Bad Learners to Find Good Configurations. In 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE) (Paderborn, Germany). Paderborn, Germany, 257--267.

Digital Library

[39]

Nina Narodytska and Toby Walsh. 2007. Constraint and Variable Ordering Heuristics for Compiling Configuration Problems. In International Joint Conference on Artificial Intelligence (IJCAI). Hyderabad, India, 149--154.

[40]

Yehuda Naveh, Michal Rimon, Itai Jaeger, Yoav Katz, Michael Vinov, Eitan Marcus, and Gil Shurek. 2006. Constraint-Based Random Stimuli Generation for Hardware Verification. In 18th Innovative Applications of Artificial Intelligence Conference (IAAI). Boston, Massachusetts, USA, 1720--1727.

[41]

Alexander Nöhrer and Alexander Egyed. 2013. C2O configurator: a tool for guided decision-making. Automated Software Engineering 20, 2 (01 Jun 2013), 265--296.

[42]

Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding Near-optimal Configurations in Product Lines by Random Sampling. In 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE) (Paderborn, Germany). New York, NY, USA, 61--71.

Digital Library

[43]

Jeho Oh, Paul Gazzillo, and Don Batory. 2019. t-wise Coverage by Uniform Sampling. In 23rd International Systems and Software Product Line Conference (SPLC) (Paris, France). ACM, New York, NY, USA, 84--87.

Digital Library

[44]

Jeho Oh, Paul Gazzillo, and Don S. Batory. 2019. t-wise coverage by uniform sampling. In 23rd International Systems and Software Product Line Conference (SPLC). Paris, France, 15:1--15:4.

[45]

Hector Perez-Morago, Ruben Heradio, David Fernandez-Amoros, Roberto Bean, and Carlos Cerrada. 2015. Efficient Identification of Core and Dead Features in Variability Models. IEEE Access 3 (2015), 2333--2340.

[46]

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. 2019. Product Sampling for Product Lines: The Scalability Challenge. In 23rd International Systems and Software Product Line Conference (SPLC). Paris, France, 78--83.

Digital Library

[47]

Quentin Plazar, Mathieu Acher, Gilles Perrouin, Xavier Devroey, and Maxime Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In 12th IEEE Conference on Software Testing, Validation and Verification (ICST). Xian, China, China, 240--251.

[48]

Subhajit Roy, Awanish Pandey, Brendan Dolan-Gavitt, and Yu Hu. 2018. Bug Synthesis: Challenging Bug-Finding Tools with Deep Faults. In 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). Lake Buena Vista, Florida, USA, 224--234.

[49]

Shubham Sharma, Rahul Gupta, Subhajit Roy, and Kuldeep S. Meel. 2018. Knowledge Compilation meets Uniform Sampling. In 22nd International Conference on Logic for Programming, Artificial Intelligence and Reasoning (LPAR), Gilles Barthe, Geoff Sutcliffe, and Margus Veanes (Eds.). Awassa, Ethiopia, 620--636.

[50]

Paul Temple, José A. Galindo, Mathieu Acher, and Jean-Marc Jézéquel. 2016. Using Machine Learning to Infer Constraints for Product Lines. In 20th International Systems and Software Product Line Conference (SPLC). Beijing, China, 209--218.

Digital Library

[51]

Marc Thurley. 2006. sharpSAT - Counting Models with Advanced Component Caching and Implicit BCP. In 9th International Conference on Theory and Applications of Satisfiability Testing (SAT). Seattle, WA, USA, 424--429.

Digital Library

[52]

Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A Classification of Product Sampling for Software Product Lines. In International Systems and Software Product Line Conference (SPLC). Association for Computing Machinery, Gothenburg, Sweden, 1--13.

Digital Library

[53]

Markus Weckesser, Roland Kluge, Martin Pfannemüller, Michael Matthé, Andy Schürr, and Christian Becker. 2018. Optimal Reconfiguration of Dynamic Software Product Lines Based on Performance-Influence Models. In 22nd International Systems and Software Product Line Conference (SPLC). Gothenburg, Sweden, 98--109.

[54]

Jun Yuan, Ken Albin, Adnan Aziz, and Carl Pixley. 2002. Simplifying Constraint Solving in Random Simulation Generation. In 11th IEEE/ACM International Workshop on Logic & Synthesis (IWLS). New Orleans, Louisiana, USA, 185--190.

Cited By

Entekhabi SMostowski WMousavi M(2023)Automated and Efficient Test-Generation for Grid-Based Multiagent Systems: Comparing Random Input Filtering versus Constraint SolvingACM Transactions on Software Engineering and Methodology10.1145/362473633:1(1-32)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3624736
Oh JBatory DHeradio R(2023)Finding Near-optimal Configurations in Colossal Spaces with Statistical GuaranteesACM Transactions on Software Engineering and Methodology10.1145/361166333:1(1-36)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3611663
Ammermann JBittner TEichhorn DSchaefer ISeidl C(2023)Can Quantum Computing Improve Uniform Random Sampling of Large Configuration Spaces?2023 IEEE/ACM 4th International Workshop on Quantum Software Engineering (Q-SE)10.1109/Q-SE59154.2023.00012(34-41)Online publication date: May-2023
https://doi.org/10.1109/Q-SE59154.2023.00012
Show More Cited By

Index Terms

Uniform and scalable SAT-sampling for configurable systems
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines

Recommendations

t-wise Coverage by Uniform Sampling
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Efficiently testing large configuration spaces of Software Product Lines (SPLs) needs a sampling algorithm that is both scalable and provides good t-wise coverage. The 2019 SPLC Sampling Challenge provides large real-world feature models and asks for a ...
Uniform and scalable sampling of highly configurable systems
Abstract
Many analyses on configurable software systems are intractable when confronted with colossal and highly-constrained configuration spaces. These analyses could instead use statistical inference, where a tractable sample accurately predicts results ...
Scalable Sampling of Highly-Configurable Systems: Generating Random Instances of the Linux Kernel
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering

Software systems are becoming increasingly configurable. A paradigmatic example is the Linux kernel, which can be adjusted for a tremendous variety of hardware devices, from mobile phones to supercomputers, thanks to the thousands of configurable ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

October 2020

323 pages

ISBN:9781450375696

DOI:10.1145/3382025

General Chair:
Roberto Erick Lopez-Herrejon
École de technologie supérieure, Canada

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Spanish Ministry of Science, Innovation and Universities
Consejería de Economía, Innovación, Ciencia y Empleo, Junta de Andalucía
TASOVA network
Community of Madrid

Conference

SPLC '20

Sponsor:

SIGSOFT

SPLC '20: 24th ACM International Systems and Software Product Line Conference

October 19 - 23, 2020

Quebec, Montreal, Canada

Acceptance Rates

SPLC '20 Paper Acceptance Rate 17 of 49 submissions, 35%;

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
225
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Entekhabi SMostowski WMousavi M(2023)Automated and Efficient Test-Generation for Grid-Based Multiagent Systems: Comparing Random Input Filtering versus Constraint SolvingACM Transactions on Software Engineering and Methodology10.1145/362473633:1(1-32)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3624736
Oh JBatory DHeradio R(2023)Finding Near-optimal Configurations in Colossal Spaces with Statistical GuaranteesACM Transactions on Software Engineering and Methodology10.1145/361166333:1(1-36)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3611663
Ammermann JBittner TEichhorn DSchaefer ISeidl C(2023)Can Quantum Computing Improve Uniform Random Sampling of Large Configuration Spaces?2023 IEEE/ACM 4th International Workshop on Quantum Software Engineering (Q-SE)10.1109/Q-SE59154.2023.00012(34-41)Online publication date: May-2023
https://doi.org/10.1109/Q-SE59154.2023.00012
Horcas JGalindo JHeradio RFernandez-Amoros DBenavides D(2023)A Monte Carlo tree search conceptual framework for feature model analysesJournal of Systems and Software10.1016/j.jss.2022.111551195:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.jss.2022.111551
Fernandez-Amoros DHeradio RMayr-Dorn CEgyed A(2022)Scalable Sampling of Highly-Configurable Systems: Generating Random Instances of the Linux KernelProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556899(1-12)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556899
Xiang YHuang HZhou YLi SLuo CLin QLi MYang XDwyer MDamian DZeller A(2022)Search-based diverse sampling from real-world software product linesProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510053(1945-1957)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510053
Xiang YYang XHuang HHuang ZLi M(2022)Sampling configurations from software product lines via probability-aware diversification and SAT solvingAutomated Software Engineering10.1007/s10515-022-00348-829:2Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10515-022-00348-8
Luo CSun BQiao BChen JZhang HLin JLin QZhang DSpinellis DGousios GChechik MDi Penta M(2021)LS-sampling: an effective local search based sampling approach for achieving high t-wise coverageProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468622(1081-1092)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468622
Acher MPerrouin GCordy MMousavi MSchobbens P(2021)BURSTProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B10.1145/3461002.3473070(36-40)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461002.3473070
Martin HAcher MPereira JJézéquel JMousavi MSchobbens P(2021)A comparison of performance specialization learning for configurable systemsProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume A10.1145/3461001.3471155(46-57)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461001.3471155
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten