skip to main content
research-article

sgen1: A generator of small but difficult satisfiability benchmarks

Published: 17 March 2010 Publication History

Abstract

The satisfiability problem is known to be NP-Complete; therefore, there should be relatively small problem instances that take a very long time to solve. However, most of the smaller benchmarks that were once thought challenging, especially the satisfiable ones, can be processed quickly by modern SAT-solvers. We describe and make available a generator that produces both unsatisfiable and, more significantly, satisfiable formulae that take longer to solve than any others known. At the two most recent international SAT Competitions, the smallest unsolved benchmarks were created by this generator. We analyze the results of all solvers in the most recent competition when applied to these benchmarks and also present our own more focused experiments.

References

[1]
Aloul, F. A., Ramani, A., Markov, I., and Sakallah, K. 2003. Solving difficult instances of Boolean satisfiability in the presence of symmetry. IEEE Trans. CAD 22, 9, 1117--1137.
[2]
Batory, D. 2005. Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference on Software Product Lines. Springer, Berlin, 7--20.
[3]
clasp. 2009. A conflict-driven no good learning answer set solver. http://www.cs.uni-potsdam.de/clasp/.
[4]
Cook, S. A. 1971. The complexity of theorem-proving procedures. In Proceedings of the 3rd Annual ACM Symposium on Theory of Computing. ACM, New York, 151--158.
[5]
Crawford, J. M., Kearns, M. J., and Shapire, R. E. 1994. The minimal disagreement parity problem as a hard satisfiability problem. Tech. rep., Computational Intelligence Research Laboratory and AT&T Bell Labs.
[6]
Davis, M., Logemann, G., and Loveland, D. 1962. A machine program for theorem-proving. Commun. ACM 5, 7, 394--397.
[7]
Davis, M. and Putnam, H. 1960. A computing procedure for quantification theory. J. ACM 7, 3, 201--215.
[8]
DIMACS. 1993. Satisfiability suggested format. ftp://dimacs.rutgers.edu/pub/challenge/satisfiability/doc/satformat.tex.
[9]
Eén, N. and Sörensson, N. 2003. An extensible sat-solver. In Proceedings of the 6th Annual Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 502--518.
[10]
Fortnow, L. 2009. The status of the P versus NP problem. Commun. ACM 52, 9, 78--86.
[11]
Haanpää, H., Järvisalo, M., Kaski, P., and Niemelä, I. 2006. Hard satisfiable clause sets for benchmarking equivalence reasoning techniques. J. Satisfiability Boolean Model. Comput. 2, 1-4, 27--46.
[12]
Haixia, J., Moore, C., and Selman, B. 2005. From spin glasses to hard satisfiable formulas. In Proceedings of the 7th International Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 199--210.
[13]
Hirsch, E. 2002. Random generator hgen2 of satisfiable formulas in 3-CNF. http://logic.pdmi.ras.ru/~hirsch/benchmarks/.
[14]
Hoos, H. H. 2009. The international conferences on theory and applications of satisfiability testing (sat). http://www.satisfiability.org.
[15]
Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. 1983. Optimization by simulated annealing. Science 220, 4598, 671--680.
[16]
Krishnamachari, B., Xi Xie, A. B. S., and Wicker, S. 2000. Analysis of random noise and random walk algorithms for satisfiability testing. In Proceedings of the 6th Annual Conference on Principles and Practice of Constraint Programming. Springer, Berlin, 278--290.
[17]
le Berre, D. 2009. The sat competitions. http://www.satcompetition.org.
[18]
Li, C. M. and Huang, W. Q. 2005. Diversification and determinism in local search for satisfiability. In Proceedings of the 8th International Conference on Theory and Applications of Satisfiability Testing. Springer, Berlin, 158--172.
[19]
Pham, D. N., Thornton, J. R., Gretton, C., and Sattar, A. 2007. Advances in local search for satisfiability. In Proceedings of the 20th Australian Joint Conference on Artificial Intelligence. Springer, Berlin, 213--222.
[20]
Rintanen, J., Heljanko, K., and Niemel, I. 2006. Planning as satisfiability: Parallel plans and algorithms for plan search. Artif. Intell. 170, 12-13, 1031--1080.
[21]
Safarpour, S. A., Baeckler, G., Yuan, R., and Veneris, A. 2006. Efficient sat-based Boolean matching for FPGA technology mapping. In Proceedings 43rd Design Automation Conference. ACM, New York, 466--471.
[22]
Sinz, C. 2008. SAT-Race 2008. http://baldur.iti.uka.de/sat-race-2008.
[23]
Smith, A., Veneris, A. G., Ali, M. F., and Viglas, A. 2003. Fault diagnosis and logic debugging using Boolean satisfiability. IEEE Trans. CAD 24, 10, 1606--1621.
[24]
Spence, I. 2008. tts: A SAT-solver for small, difficult instances. J. Satisfiability Boolean Model. Comput. 4, 173--190.
[25]
Whitesitt, J. E. 1995. Boolean Algebra and Its Applications. Dover Publications, Mineola, NY.
[26]
Xu, K., Boussemart, F., Hemery, F., and Lecoutre, C. 2007. Random constraint satisfaction: Easy generation of hard (satisfiable) instances. Artif. Intell. 171, 514--534.
[27]
Zhang, L. and Malik, S. 2002. The quest for efficient Boolean satisfiability solvers. In Proceedings of the 14th International Conference on Computer-Aided Verification. Springer-Verlag, Berlin, 17--36.

Cited By

View all
  • (2024)A Generalized Method for Proving Polynomial Calculus Degree Lower BoundsJournal of the ACM10.1145/367566871:6(1-43)Online publication date: 4-Jul-2024
  • (2024)UNSAT Solver Synthesis via Monte Carlo Forest SearchIntegration of Constraint Programming, Artificial Intelligence, and Operations Research10.1007/978-3-031-60597-0_12(170-189)Online publication date: 25-May-2024
  • (2020)Solving SAT (and MaxSAT) with a quantum annealer: Foundations, encodings, and preliminary resultsInformation and Computation10.1016/j.ic.2020.104609275(104609)Online publication date: Dec-2020
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Journal of Experimental Algorithmics
ACM Journal of Experimental Algorithmics  Volume 15, Issue
2010
387 pages
ISSN:1084-6654
EISSN:1084-6654
DOI:10.1145/1671970
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 March 2010
Accepted: 01 December 2009
Revised: 01 November 2009
Received: 01 November 2009
Published in JEA Volume 15

Author Tags

  1. SAT-solvers
  2. Satisfiability benchmarks

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)1
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Generalized Method for Proving Polynomial Calculus Degree Lower BoundsJournal of the ACM10.1145/367566871:6(1-43)Online publication date: 4-Jul-2024
  • (2024)UNSAT Solver Synthesis via Monte Carlo Forest SearchIntegration of Constraint Programming, Artificial Intelligence, and Operations Research10.1007/978-3-031-60597-0_12(170-189)Online publication date: 25-May-2024
  • (2020)Solving SAT (and MaxSAT) with a quantum annealer: Foundations, encodings, and preliminary resultsInformation and Computation10.1016/j.ic.2020.104609275(104609)Online publication date: Dec-2020
  • (2020)MaxSAT Resolution and Subcube SumsTheory and Applications of Satisfiability Testing – SAT 202010.1007/978-3-030-51825-7_21(295-311)Online publication date: 26-Jun-2020
  • (2018)Using Combinatorial Benchmarks to Probe the Reasoning Power of Pseudo-Boolean SolversTheory and Applications of Satisfiability Testing – SAT 201810.1007/978-3-319-94144-8_5(75-93)Online publication date: 9-Jul-2018
  • (2018)In Between Resolution and Cutting Planes: A Study of Proof Systems for Pseudo-Boolean SAT SolvingTheory and Applications of Satisfiability Testing – SAT 201810.1007/978-3-319-94144-8_18(292-310)Online publication date: 9-Jul-2018
  • (2017)Solving SAT and MaxSAT with a Quantum Annealer: Foundations and a Preliminary ReportFrontiers of Combining Systems10.1007/978-3-319-66167-4_9(153-171)Online publication date: 29-Aug-2017
  • (2016)Solving the MAX-SAT problem by binary enhanced fireworks algorithm2016 Sixth International Conference on Innovative Computing Technology (INTECH)10.1109/INTECH.2016.7845071(204-209)Online publication date: Aug-2016
  • (2015)A generalized method for proving polynomial calculus degree lower boundsProceedings of the 30th Conference on Computational Complexity10.5555/2833227.2833251(467-487)Online publication date: 17-Jun-2015
  • (2015)On the interplay between proof complexity and SAT solvingACM SIGLOG News10.1145/2815493.28154972:3(19-44)Online publication date: 17-Aug-2015
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media