Skip to main content
Springer Nature Link
Log in

Going deeper with optimal software products selection using many-objective optimization and satisfiability solvers

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

In search-based software engineering, one actively studied problem is the optimal software product selection from a feature model using multiple (usually more than three) optimization objectives simultaneously. This can be represented as a many-objective optimization problem. The primary goal of solving this problem is to search for diverse and high-quality valid products as rapidly as possible. Previous studies have shown that combining search-based techniques with satisfiability (SAT) solvers was promising for achieving this goal, but it remained open that how different solvers affect the performance of a search algorithm, and that whether the ways to randomize solutions in the solvers make a difference. Moreover, we may need further investigation on the necessity of mixing different types of SAT solving techniques. In this paper, we address the above open research questions by performing a series of empirical studies on 21 features models, most of which are reverse-engineered from industrial software product lines. We examine four conflict-driven clause learning solvers, two stochastic local search solvers, and two different ways to randomize solutions. Experimental results suggest that the performance can be indeed affected by different SAT solvers, and by the ways to randomize solutions in the solvers. This study serves as a practical guideline for choosing and tuning SAT solvers for the many-objective optimal software product selection problem.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. In evolutionary multi-objective (EMO) community (Li et al. 2015b; Ishibuchi et al. 2017; Xiang et al. 2017a, b), the term “many-objective” refers in particular to four or more objectives.

  2. SAT4J website: http://www.sat4j.org/

  3. (P)lingeling website: http://fmv.jku.at/lingeling/

  4. Glucose website: http://www.labri.fr/perso/lsimon/glucose/

  5. Kodkod is available at http://emina.github.io/kodkod/.

  6. According to Balint and Schöning (2012), several functions can be defined for f(v). In this study, we choose the simple and effective polynomial function as shown in (2).

  7. The LVAT repository can be found at http://code.google.com/p/linux-variability-analysis-tools

  8. The codes of SATVaEA can be downloaded from https://www.researchgate.net/profile/Xiang_Yi9/publications.

  9. The G*Power is available at http://www.gpower.hhu.de/

  10. The original SATVaEA (Xiang et al. 2018) also uses SAT4J with permutated parameters, therefore we keep the name of the algorithm unchanged.

References

  • Balint A, Schöning U (2012) Choosing Probability Distributions for Stochastic Local Search and the Role of Make versus Break. International Conference on Theory and Applications of Satisfiability Testing, Berlin, pp 16–29

  • Balint A, Schöning U (2014) ProbSAT and pprobSAT. In: Belov A., Diepold D., Heule M. J., Järvisalo M (eds) Proceedings of SAT Competition 2014: Solver and Benchmark Descriptions. Department of Computer Science Series of Publications B and Department of Computer Science, University of Helsinki, Helsinki, vol B-2014-2, p 63

  • Balyo T, Biere A, Isera M, Sinza C (2016) SAT Race 2015. Artif Intell 241:45–65

    Article  MathSciNet  Google Scholar 

  • Batory D (2005) Feature models, grammars, and propositional formulas. In: Obbink H., Pohl K. (eds) Proceedings of the 9th International Conference Software Product Lines, SPLC 2005. Springer, Berlin, pp 7–20. https://doi.org/10.1007/11554844_3

  • Berger T, She S, Lotufo R, Wasowski A, Czarnecki K (2010) Variability modeling in the real: a perspective from the operating systems domain. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ASE ’10. ACM, New York, pp 73-82. https://doi.org/10.1145/1858996.1859010

  • Berger T, Lettner D, Rubin J, Grünbacher P, Silva A, Becker M, Chechik M, Czarnecki K (2015) What is a feature?: a qualitative study of features in industrial software product lines. In: Proceedings of the 19th International Conference on Software Product Line (SPLC), pp 16–25

  • Berre DL, Parrain A (2010) The Sat4j library, release 2.2, system description. Journal on Satisfiability. Boolean Model Comput 7:59–64

    Google Scholar 

  • Bezerra LCT, López-Ibáñez M, Stützle T (2016) Automatic component-wise design of multiobjective evolutionary algorithms. IEEE Trans Evol Comput 20(3):403–417. https://doi.org/10.1109/TEVC.2015.2474158

    Article  Google Scholar 

  • Biere A (2010) Lingeling, Plingeling, picoSAT and precoSAT at SAT Race 2010. Tech. rep., Institute for Formal Models and Verification, Johannes Kepler University, Altenbergerstr 69, 4040 Linz, Austria

  • Biere A (2018) CADICAL, LINGELING, PLINGELING, TREENGELING And YALSAT entering the SAT competition 2018. In: Heule M. JH, Järvisalo MJ, Suda M (eds) Proceedings of SAT Competition 2018: Solver and Benchmark Descriptions. Department of Computer Science Series of Publications B and Department of Computer Science, University of Helsinki, Helsinki, vol B-2018-1, pp 13–14

  • Cai S (2013) Faster implementation for walksat. Tech. rep., Queensland Research Lab, NICTA, Australia

  • Chen J (2018a) AbcdSAT and Glucose hack: Various Simplifications and Optimizations for CDCL SAT solvers. In: Heule MJH, Järvisalo MJ, Suda M (eds) Proceedings of SAT Competition 2018: Solver and Benchmark Descriptions. Department of Computer Science Series of Publications B and Department of Computer Science, University of Helsinki, Helsinki, vol B-2018-1, pp 10–12

  • Chen J, Nair V, Krishna R, Menzies T (2018b) ‘Sampling’ as a Baseline Optimizer for Search-based Software Engineering IEEE Transactions on Software Engineering. https://doi.org/10.1109/TSE.2018.2790925

  • Clements P, Northrop L (2001) Software product lines: practices and patterns. Addison-wesley Longman Publishing Co. Inc

  • Coello Coello CA, Lamont GB, Veldhuizen DAV (2007) Evolutionary Algorithms for Solving Multi-objective Problems, 2nd edn. Springer Science + Business Media, LLC, New York

    MATH  Google Scholar 

  • Czarnecki K, Eisenecker U (2000) Generative programming: methods, tools, and applications. Addison-Wesley, Reading

  • Derrac J, Garcia S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1(7):3–18

    Article  Google Scholar 

  • Durillo JJ, Nebro AJ (2011) jMetal: A java framework for multi-objective optimization. Adv Eng Softw 42:760–771

    Article  Google Scholar 

  • Dutra R, Laeufer K, Bachrach J, Sen K (2018) Efficient sampling of SAT solutions for testing. In: 2018 IEEE/ACM 40Th international conference on software engineering (ICSE). IEEE Computer Society, Los Alamitos, pp 549–559. https://doi.org/10.1145/3180155.3180248

  • Eén N, Sörensson N (2003) An Extensible SAT-solver. In: International conference on theory and applications of satisfiability testing. Springer, Berlin, pp 502–518

  • Faul F, Erdfelder E, Buchner A, Lang AG (2009) Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behav Res Methods 41(4):1149–1160. https://doi.org/10.3758/BRM.41.4.1149

    Article  Google Scholar 

  • Guo J, Zulkoski E, Olaechea R, Rayside D, Czarnecki K, Apel S, Atlee JM (2014) Scaling exact multi-objective combinatorial optimization by parallelization. In: ACM/IEEE International conference on automated software engineering, ASE ’14, Vasteras, Sweden, pp 409–420. https://doi.org/10.1145/2642937.2642971

  • Guo J, Liang JH, Shi K, Yang D, Zhang J, Czarnecki K, Ganesh V, Yu H (2017) SMTIBEA: A hybrid multi-objective optimization algorithm for configuring large constrained software product lines Software & Systems Modeling. https://doi.org/10.1007/s10270-017-0610-0

  • Guo J, Shi K (2018) To preserve or not to preserve invalid solutions in search-based software engineering: a case study in software product lines. In: Proceedings of the 40th International Conference on Software Engineering, ICSE ’18. ACM, New York, pp 1027–1038. https://doi.org/10.1145/3180155.3180163

  • Henard C, Papadakis M, Harman M, Traon YL (2015) Combining multi-objective search and constraint solving for configuring large software product lines. In: The 37th international conference on software engineering, vol 1, pp 517–528. https://doi.org/10.1109/ICSE.2015.69

  • Henard C, Papadakis M, Perrouin G, Klein J, Traon YL (2013a) Assessing software product line testing via model-based mutation: an application to similarity testing. In: 2013 IEEE Sixth international conference on software testing, verification and validation workshops, pp 188–197. https://doi.org/10.1109/ICSTW.2013.30

  • Henard C, Papadakis M, Perrouin G, Klein J, Traon YL (2013b) Multi-objective test generation for software product lines. In: Proceedings of the 17th International Software Product Line Conference, SPLC ’13. ACM, New York, pp 62–71. https://doi.org/10.1145/2491627.2491635

  • Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, Le Traon Y (2014) Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650–670. https://doi.org/10.1109/TSE.2014.2327020

    Article  Google Scholar 

  • Harman M (2007) The current state and future of search based software engineering. In: Future of software engineering, FOSE ’07. IEEE Computer Society, Washington, pp 342–357. https://doi.org/10.1109/FOSE.2007.29

  • Harman M, Mansouri SA, Zhang Y (2012) Search-based software engineering: trends, techniques and applications. ACM Comput Surv 45(1):11:1–11:61. https://doi.org/10.1145/2379776.2379787

    Article  Google Scholar 

  • Harman M, Jia Y, Krinke J, Langdon B, Petke J, Zhang Y (2014) Search based software engineering for software product line engineering: a survey and directions for future work. In: Proceedings of the 18th International Software Product Line Conference - Volume 1, SPLC ’14. ACM, New York, pp 5–18. https://doi.org/10.1145/2648511.2648513

  • Hierons RM, Li M, Liu X, Segura S, Zheng W (2016) SIP: Optimal Product selection from feature models using Many-Objective evolutionary optimization. ACM Trans Softw Eng Methodol 25(2):17:1–17:39. https://doi.org/10.1145/2897760

    Article  Google Scholar 

  • Ishibuchi H, Masuda H, Nojima Y (2016) Pareto fronts of many-objective degenerate test problems. IEEE Trans Evol Comput 20(5):807–813. https://doi.org/10.1109/TEVC.2015.2505784

    Article  Google Scholar 

  • Ishibuchi H, Setoguchi Y, Masuda H, Nojima Y (2017) Performance of decomposition-based many-objective algorithms strongly depends on pareto front shapes. IEEE Trans Evol Comput 21(2):169–190. https://doi.org/10.1109/TEVC.2016.2587749

    Article  Google Scholar 

  • Johansen MF, Haugen Ø, Fleurey F (2012) An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of the 16th International Software Product Line Conference-Volume 1. ACM, pp 46–55

  • Kang KC, Cohen SG, Hess JA, Novak WE, Peterson AS (1990) Feature-oriented domain analysis (FODA ) feasibility study. Tech. rep., CMU/SEI-90-TR-21. SEI Georgetown University

  • Li M, Yang S, Liu X (2014) Shift-based density estimation for pareto-based algorithms in many-objective optimization. IEEE Trans Evol Comput 18:348–65

    Article  Google Scholar 

  • Li M, Yang S, Liu X (2015a) Bi-goal evolution for many-objective optimization problems. Artif Intell 228:45–65

    Article  MathSciNet  Google Scholar 

  • Li B, Li J, Tang K, Yao X (2015b) Many-objective evolutionary algorithms: A survey. ACM Comput Surv 48(1):1–35

    Article  MathSciNet  Google Scholar 

  • Li M, Chen T, Yao X (2018) A critical review of: ”a practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering”: Essay on quality indicator selection for sbse. In: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER ’18. ACM, New York, pp 17–20. https://doi.org/10.1145/3183399.3183405

  • Li M, Yao X (2019) Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Comput Surv 52(2):26:1–26:38. https://doi.org/10.1145/3300148

    Article  Google Scholar 

  • Liang JH, Ganesh V, Czarnecki K, Raman V (2015) SAT-Based Analysis of Large Real-world Feature Models is Easy. In: Proceedings of the 19th International Conference on Software Product Line, SPLC ’15. ACM, New York, pp 91–100. https://doi.org/10.1145/2791060.2791070

  • Lopez-Herrejon RE, Chicano F, Ferrer J, Egyed A, Alba E (2013) Multi-objective optimal test suite computation for software product line pairwise testing. In: IEEE International conference on software maintenance, pp 404–407

  • Marques-Silva JP, Sakallah KA (1999) GRASP: A search algorithm for propositional satisfiability. IEEE Trans Comput 48(5):506–521

    Article  MathSciNet  Google Scholar 

  • Mendonca M, Wasowski A, Czarnecki K (2009) SAT-Based Analysis of Feature Models is Easy. In: Proceedings of the 13th International Software Product Line Conference, SPLC ’09. Carnegie Mellon University, Pittsburgh, pp 231–240

  • Olaechea R, Rayside D, Guo J, Czarnecki K (2014) Comparison of exact and approximate multi-objective optimization for software product lines. In: The international software product line conference, pp 92–101

  • Perrouin G, Sen S, Baudry JKB, le Traon Y (2010) Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines. IEEE Sixth Int Conf Softw Test Verif Valid 00:459–468. https://doi.org/10.1109/ICST.2010.43

    Article  Google Scholar 

  • Pohl R, Lauenroth K, Pohl K (2011) A performance comparison of contemporary algorithmic approaches for automated analysis operations on feature models. In: IEEE/ACM International conference on automated software engineering, pp 313–322

  • Rayside D, Estler HC, Jackson D (2009) The guided improvement algorithm for exact, general-purpose, many-objective combinatorial optimization. Tech. rep., MIT-CSAIL-TR-2009-033 MIT CSAIL

  • Sayyad AS, Goseva-Popstojanova K, Menzies T, Ammar H (2013a) On parameter tuning in search based software engineering: a replicated empirical study. In: International workshop on replication in empirical software engineering research, pp 84–90

  • Sayyad AS, Ingram J, Menzies T, Ammar H (2013b) Scalable product line configuration: a straw to break the camel’s back. In: 2013 28Th IEEE/ACM international conference on automated software engineering (ASE), pp 465–474. https://doi.org/10.1109/ASE.2013.6693104

  • Sayyad AS, Menzies T, Ammar H (2013c) On the value of user preferences in search-based software engineering: a case study in software product lines. In: 2013 35Th international conference on software engineering (ICSE), pp 492–501. https://doi.org/10.1109/ICSE.2013.6606595

  • Selman B, Levesque H, Mitchell D (1992) A new method for solving hard satisfiability problems. In: Proceedings of the Tenth National Conference on Artificial Intelligence, AAAI’92. AAAI Press, pp 440–446

  • Selman B, Kautz HA, Cohen B (1994) Noise strategies for improving local search. In: Proceedings of the Twelfth National Conference on Artificial Intelligence (Vol. 1), AAAI ’94. American Association for Artificial Intelligence, Menlo Park, pp 337–343

  • She S, Lotufo R, Berger T, Wasowski A, Czarnecki K (2011) Reverse engineering feature models. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11. ACM, New York, pp 461–470. https://doi.org/10.1145/1985793.1985856

  • Simon L, Audemard G (2009) Predicting learnt clauses quality in modern SAT solver. In: Twenty-first international joint conference on artificial intelligence (IJCAI’09). Pasadena, United States

  • Srinivas M, Patnaik LM (1994) Genetic algorithms: a survey. Computer 27 (6):17–26

    Article  Google Scholar 

  • Tan TH, Xue Y, Chen M, Sun J, Liu Y, Dong JS (2015) Optimizing selection of competing features via feedback-directed evolutionary algorithms. In: Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015), pp 246–256

  • Wang H, Jin Y, Yao X (2017) Diversity assessment in many-objective optimization. IEEE Trans Cybern 47(6):1510–1522. https://doi.org/10.1109/TCYB.2016.2550502

    Article  Google Scholar 

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bullet 1 (6):80–83

    Article  Google Scholar 

  • Xiang Y, Peng J, Zhou Y, Li M, Chen Z (2017a) An angle based constrained many-objective evolutionary algorithm. Appl Intell 47(3):705–720. https://doi.org/10.1007/s10489-017-0929-9

    Article  Google Scholar 

  • Xiang Y, Zhou Y, Li M, Chen Z (2017b) A vector angle based evolutionary algorithm for unconstrained many-objective problems. IEEE Trans Evol Comput 21(1):131–152. https://doi.org/10.1109/TEVC.2016.2587808

    Article  Google Scholar 

  • Xiang Y, Zhou Y, Zheng Z, Li M (2018) Configuring software product lines by combining Many-Objective optimization and SAT solvers. ACM Trans Softw Eng Methodol 26(4):14:1–14:46. https://doi.org/10.1145/3176644

    Article  Google Scholar 

  • Xue Y, Zhong J, Tan TH, Liu Y, Cai W, Chen M, Sun J (2016) IBED: Combining IBEA And DE for optimal feature selection in software product line engineering. Appl Soft Comput 49:1215–1231. https://doi.org/10.1016/j.asoc.2016.07.040

    Article  Google Scholar 

  • Xue Y, Li YF (2018) Multi-objective Integer Programming Approaches for Solving Optimal Feature Selection Problem: A New Perspective on Multi-objective Optimization Problems in SBSE. In: Proceedings of the 40th International Conference on Software Engineering, ICSE ’18. ACM, New York, pp 1231–1242. https://doi.org/10.1145/3180155.3180257

  • Zabih R, Mcallester D (1988) A rearrangement search strategy for determining propositional satisfiability. In: National conference on artificial intelligence, pp 155–160

  • Zhang Q, Li H (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans Evol Comput 11(6):712–731

    Article  Google Scholar 

  • Zhang Q, Zhou A, Zhao S, Suganthan PN, Liu W, Tiwari S (2009) Multiobjective optimization test instances for the CEC 2009 special session and competition. University of Essex, Colchester, UK and Nanyang Technological University, Singapore, Special Session on Performance Assessment of Multi-Objective Optimization Algorithms, Technical Report

  • Zhou Y, Xiang Y, Chen Z, He J, Wang J (2019) A scalar projection and angle based evolutionary algorithm for many-objective optimization problems. IEEE Trans Cybern 49(6):2073–2084. https://doi.org/10.1109/TCYB.2018.2819360

    Article  Google Scholar 

  • Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans Evol Comput 3(4):257–271

    Article  Google Scholar 

  • Zitzler E, Künzli S (2004) Indicator-based selection in multiobjective search. In: Proc. 8th international conference on parallel problem solving from nature, PPSN VIII. Springer, pp 832–842

Download references

Acknowledgments

This work was supported in part by the Fundamental Research Funds for the Central Universities under Grant x2rjD2190840, in part by the National Natural Science Foundation of China under Grant 61773410, Grant 61673403, Grant 61703183 and Grant 61876207, in part by the Guangdong Natural Science Funds for Distinguished Young Scholar under Grant 2014A030306050, in part by the Guangdong High-Level Personnel of Special Support Program under Grant 2014TQ01X664, in part by the International Cooperation Project of Guangzhou under Grant 201807010047, in part by the Science and Technology Program of Guangzhou under Grant 201607010069, Grant 201802010007 and Grant 201804010276, in part by Guangdong Province Key Area R&D Program under Grant 2018B010109003, and in part by the Science Technique Department of Guizhou Province under Grant [2019]1164.

Author information

Authors and Affiliations

  1. School of Software Engineering, South China University of Technology, Guangzhou, 510006, People’s Republic of China

    Yi Xiang, Xiaowei Yang & Han Huang

  2. School of Data and Computer Science, Sun Yat-sen University, Guangzhou, 510006, People’s Republic of China

    Yuren Zhou & Zibin Zheng

  3. Centre of Excellence for Research in Computational Intelligence and Applications, School of Computer Science, University of Birmingham, Birmingham, B15 2TT, UK

    Miqing Li

Authors
  1. Yi Xiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaowei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuren Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zibin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Miqing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Han Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaowei Yang.

Additional information

Communicated by: Mark Harman

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiang, Y., Yang, X., Zhou, Y. et al. Going deeper with optimal software products selection using many-objective optimization and satisfiability solvers. Empir Software Eng 25, 591–626 (2020). https://doi.org/10.1007/s10664-019-09761-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09761-2

Keywords

Search

Navigation