Skip to main content
SpringerLink
Log in

Multi-language evaluation of exact solvers in graphical model discrete optimization

  • Published:
Constraints Aims and scope Submit manuscript

Abstract

By representing the constraints and objective function in factorized form, graphical models can concisely define various NP-hard optimization problems. They are therefore extensively used in several areas of computer science and artificial intelligence. Graphical models can be deterministic or stochastic, optimize a sum or product of local functions, defining a joint cost or probability distribution. Simple transformations exist between these two types of models, but also with MaxSAT or linear programming. In this paper, we report on a large comparison of exact solvers which are all state-of-the-art for their own target language. These solvers are all evaluated on deterministic and probabilistic graphical models coming from the Probabilistic Inference Challenge 2011, the Computer Vision and Pattern Recognition OpenGM2 benchmark, the Weighted Partial MaxSAT Evaluation 2013, the MaxCSP 2008 Competition, the MiniZinc Challenge 2012 & 2013, and the CFLib (a library of Cost Function Networks). All 3026 instances are made publicly available in five different formats and seven formulations. To our knowledge, this is the first evaluation that encompasses such a large set of related NP-complete optimization frameworks, despite their tight connections. The results show that a small number of evaluated solvers are able to perform well on multiple areas. By exploiting the variability and complementarity of solver performances, we show that a simple portfolio approach can be very effective. This portfolio won the last UAI Evaluation 2014 (MAP task).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Rational costs are also used in [11].

  2. Script available at http://genoweb.toulouse.inra.fr/degivry/evalgm/scripts/wcsp2markov.py

  3. A 1-hour time limit was used to translate minizinc2 to flatzinc, readable by CP solvers.

  4. Directly lifting a CP model, with its cost variable, to a CFN would be of limited value since all AC in CFN are known to enforce AC on cost functions representing hard constraints.

  5. We restrict the size of cost functions to be less than 106 tuples in our implementation.

  6. http://numberjack.ucc.ie/

  7. http://www.cs.huji.ac.il/project/PASCAL

  8. http://graphmod.ics.uci.edu/uai08/Evaluation/Report/Benchmarks

  9. Alchemy and Promedas were solved by toulbar2 in less than 1 sec. each. CSP instances came from CFLib. ProteinProtein is already present in CVPR under the name of Protein Prediction ProteinInteraction.

  10. The resulting wcsp instances were translated back to uai instances (with _digit2 extension) in order to optimize the same objective function.

  11. http://hci.iwr.uni-heidelberg.de/opengm2

  12. http://costfunction.org/benchmark

  13. http://www.cril.univ-artois.fr/CPAI08 and http://www.cril.univ-artois.fr/lecoutre/ benchmarks.html

  14. http://maxsat.ia.udl.cat:81/13/benchmarks/

  15. http://www.minizinc.org/challenge2012/results2012.html and http://www.minizinc.org/challenge2013/results2013.html

  16. mplp2 http://cs.nyu.edu/dsontag version 2 (using 2.10−7 gap thres.) was tested but the results are not presented in Section 4.3 as it was dominated in most categories by toulbar2.

  17. https://github.com/lotten/daoopt open source version 1.1.2, not including the closed source and unavailable convergent message-passing bound tightening used in the PIC challenge.

  18. http://www.inra.fr/mia/T/toulbar2 version 0.9.8, parameters -A -V -dee -hbfs.

  19. http://www.maxhs.org version 2.51, no parameter.

  20. http://www.gecode.org/ version 4.4.0, using free search.

  21. Using parameter -pe parallel_smp 2 on a SUN Grid Engine to ensure half-load of the cores on the cluster.

  22. More detailed results are available at http://genoweb.toulouse.inra.fr/degivry/evalgm.

  23. The Z-score of a value x in a set of values is \(\frac {x-\mu }{\sigma }\) where μ is the mean of the set and σ its standard deviation.

  24. The normalized total reduction brought by the feature.

  25. https://github.com/9thbit/uai-proteus used a Random Forest classifier and an older version of toulbar2 version 0.9.7, with no parameter. Here we report the results using the same settings as in Sec. 4.3, incop+toulbar2 corresponds to toulbar2 using an extra parameter -i for the initial incop starting solution phase.

  26. See MAP/Proteus entry at http://www.hlt.utdallas.edu/vgogate/uai14-competition/leaders.html.

  27. See MAP/MIP-UAI and MAP/MIP-T-UAI entries.

  28. See MAP/IncTb entry.

References

  1. Allouche, D., Bessiere, C., Boizumault, P., Givry, S., Gutierrez, P., Loudni, S., Métivier, J., & Schiex, T. (2012). Decomposing global cost functions. In Proceedings of AAAI.

  2. Allouche, D., de Givry, S., Katsirelos, G., Schiex, T., & Zytnicki, M. (2015). Anytime hybrid best-first search with tree decomposition for weighted CSP. In Proceedings of CP (pp. 12–28).

  3. Allouche, D., Traoré, S., André, I., Givry, S., Katsirelos, G., Barbe, S., & Schiex, T. (2012). Computational protein design as a cost function network optimization problem. In Proceedings of CP (pp. 840–849).

  4. Amadini, R., Gabbrielli, M., & Mauro, J. (2015). A Multicore Tool for Constraint Solving. In Proceedings of IJCAI (pp. 232–238).

  5. Ansótegui, C., Malitsky, Y., & Sellmann, M. (2014). MaxSAT by Improved Instance-Specific Algorithm Configuration. In Proceedings of AAAI (pp. 2594–2600).

  6. Argelich, J., Cabiscol, A., Lynce, I., & Manyà, F. (2008). Encoding Max-CSP into partial Max-SAT. In Proceedings of ISMVL (pp. 106–111).

  7. Bacchus, F. (2007). GAC via unit propagation. In Proceedings of CP (pp. 133–147).

  8. Bensana, E., Lemaître, M., & Verfaillie, G. (1999). Earth observation satellite management. Constraints, 4(3), 293–299.

    Article  MATH  Google Scholar 

  9. Breiman, L., Friedman, J., Stone, C.J., & Olshen, R.A. (1984). Classification and regression trees: CRC press.

  10. Cabon, B., de Givry, S., Lobjois, L., Schiex, T., & Warners, J. (1999). Radio link frequency assignment. Constraints, 4, 79–89.

    Article  MATH  Google Scholar 

  11. Cooper, M, de Givry, S., Sanchez, M., Schiex, T., Zytnicki, M., & Werner, T. (2010). Soft arc consistency revisited. Artificial Intelligence, 174, 449–478.

    Article  MathSciNet  MATH  Google Scholar 

  12. Cooper, M., de Givry, S., & Schiex, T. (2007). Optimal soft arc consistency. In Proceedings of IJCAI (pp. 68–73).

  13. Cooper, M.C., & Schiex, T. (2004). Arc consistency for soft constraints. Artificial Intelligence, 154(1-2), 199–227.

    Article  MathSciNet  MATH  Google Scholar 

  14. Davies, J., & Bacchus, F. (2011). Solving MAXSAT by solving a sequence of simpler SAT instances. In Proceedings of CP (pp. 225–239).

  15. Davies, J., & Bacchus, F. (2013). Exploiting the power of MIP solvers in MaxSAT. In Proceedings of SAT (pp. 166–181).

  16. Dechter, R. (1999). Bucket elimination: A unifying framework for reasoning. Artificial Intelligence, 113(1–2), 41–85.

    Article  MathSciNet  MATH  Google Scholar 

  17. Fargier, H., Lang, J., Martin-Clouaire, R., & Schiex, T. (1995). A constraint satisfaction framework for decision under uncertainty. In Proceedings of the 11th International Conference on Uncertainty in Artificial Intelligence. Montréal.

  18. Favier, A., Givry, S., Legarra, A., & Schiex, T. (2011). Pairwise decomposition for combinatorial optim. in graphical models. In Proceedings of IJCAI (pp. 2126–2132).

  19. de Givry, S., Prestwich, S., & O’Sullivan, B. (2013). Dead-end elimination for weighted CSP. In Proceedings of CP (pp. 263–272).

  20. Globerson, A., & Jaakkola, T. (2007). Fixing max-product: Convergent message passing algorithms for MAP LP-relaxations. In Proceedings of NIPS (pp. 553–560).

  21. Gomes, C.P., & Selman, B. (2001). Algorithm Portfolios. Artificial Intelligence, 126(1-2), 43–62.

    Article  MathSciNet  MATH  Google Scholar 

  22. Hebrard, E., O’Mahony, E., & O’Sullivan, B. (2010). Constraint Programming and Combinatorial Optimisation in Numberjack. In Proceedings of CP-AI-OR (pp. 181–185).

  23. Huberman, B.A., Lukose, R.M., & Hogg, T. (1997). An economics approach to hard computational problems. Science, 275(5296), 51–54.

    Article  Google Scholar 

  24. Hurley, B., Kotthoff, L., Malitsky, Y., & O’Sullivan, B. (2014). Proteus: A hierarchical portfolio of solvers and transformations. In Proceedings of CP-AI-OR (pp. 301–317).

  25. Jünger, M., Liebling, T., Naddef, D., Nemhauser, G., Pulleyblank, W., Reinelt, G., Rinaldi, G., & Wolsey, L. (Eds.) (2010). 50 years of integer programming 1958–2008: Springer.

  26. Kadioglu, S., Malitsky, Y., Sellmann, M., & Tierney, K. (2010). ISAC – Instance-specific algorithm configuration. In Proceedings of ECAI (pp. 751–756).

  27. Kappes, J., Andres, B., Hamprecht, F., Schnȯrr, C., Nowozin, S., Batra, D., Kim, S., Kausler, B., Krȯger, T., Lellmann, J., Komodakis, N., Savchynskyy, B., & Rother, C. (2015). A comparative study of modern inference techniques for structured discrete energy minimization problems. International Journal of Computer Vision, 115(2), 155–184.

    Article  MathSciNet  Google Scholar 

  28. Kishimoto, A., & Marinescu, R. (2013). Recursive best-first and/or search with overestimation for genetic linkage analysis. In Proceedings of CP workshop on constraint based methods for bioinformatics.

  29. Koller, D., & Friedman, N. (2009). Probabilistic graphical models: Principles and techniques: The MIT Press.

  30. Koster, A. (1999). Frequency assignment: Models and algorithms. Ph.D. thesis.

  31. Kotthoff, L. (2013). LLAMA: leveraging learning to automatically manage algorithms. Tech. Rep. arXiv:1306.1031.

  32. Kotthoff, L. (2014). Algorithm Selection for combinatorial search problems: a survey. AI Magazine, 35(3), 48–60.

    Google Scholar 

  33. Kratica, J., Toṡic, D., Filipović, V., & Ljubić, I. (2001). Solving the simple plant location problem by genetic alg. RAIRO, 35(1), 127–142.

    Article  MathSciNet  MATH  Google Scholar 

  34. Larrosa, J, de Givry, S., Heras, F., & Zytnicki, M. (2005). Existential arc consistency: getting closer to full arc consistency in weighted CSPs. In Proceedings of IJCAI (pp. 84–89).

  35. Larrosa, J., Heras, F, & de Givry, S. (2008). A logical approach to efficient max-sat solving. Artificial Intelligence, 172(2-3), 204–233.

    Article  MathSciNet  MATH  Google Scholar 

  36. Li, C.M., & Manyà, F. (2009). Maxsat. In Handbook of satisfiability, chap. 19: IOS Press.

  37. Meltzer, T., Globerson, A., & Weiss, Y. (2009). Convergent message passing algorithms: a unifying view. In Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence (pp. 393–401): AUAI Press.

  38. Meseguer, P., Rossi, F., & Schiex, T. (2006). Soft constraints processing. In Rossi, F., van Beek, P., & Walsh, T. (Eds.) Handbook of constraint programming, chap. 9: Elsevier.

  39. Nethercote, N., Stuckey, P., Becket, R., Brand, S., Duck, G., & Tack, G. (2007). MiniZinc: Towards a standard CP modelling language, (pp. 529–543).

  40. Neveu, B., Trombettoni, G., & Glover, F. (2004). Id walk: A candidate list strategy with a simple diversification device. In Proceedings of CP (pp. 423–437).

  41. O’Mahony, E., Hebrard, E., Holland, A., Nugent, C., & O’Sullivan, B. (2008). Using case-based reasoning in an algorithm portfolio for constraint solving: Irish Conference on Artificial Intelligence and Cognitive Science.

  42. Otten, L., Ihler, A., Kask, K., & Dechter, R. (2012). Winning the PASCAL 2011 MAP challenge with enhanced AND/OR branch-and-bound. In NIPS DISCML Workshop.

  43. Petit, T., Régin, J., & Bessière, C. (2000). Meta constraints on violations for over constrained problems. In Proceedings of ICTAI (pp. 358–365).

  44. Prusa, D., & Werner, T. (2015). Universality of the local marginal polytope. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(4), 898–904.

    Article  Google Scholar 

  45. Rossi, F., van Beek, P., & Walsh, T. (Eds.) (2006). Handbook of constraint programming: Elsevier.

  46. Sánchez, M, de Givry, S., & Schiex, T. (2008). Mendelian error detection in complex pedigrees using weighted constraint satisfaction techniques. Constraints, 13 (1–2), 130–154.

    Article  MathSciNet  MATH  Google Scholar 

  47. Schlesinger, M. (1976). Syntactic analysis of two-dimensional visual signals in noisy conditions. Kibernetika, 4, 113–130.

    Google Scholar 

  48. Sontag, D., Choe, D., & Li, Y. (2012). Efficiently searching for frustrated cycles in MAP inference. In Proceedings of UAI (pp. 795–804).

  49. Sontag, D., Meltzer, T., Globerson, A., Weiss, Y., & Jaakkola, T. (2008). Tightening LP relaxations for MAP using message-passing. In Proceedings of UAI (pp. 503–510).

  50. Werner, T. (2007). A linear programming approach to max-sum problem. Pattern Analysis and Machine Intelligence, 29(7), 1165–1179.

    Article  Google Scholar 

  51. Xu, L., Hutter, F., Hoos, H.H., & Leyton-Brown, K. (2008). SATzilla: Portfolio-based algorithm selection for SAT. In Journal of artificial intelligence research (pp. 565–606).

Download references

Acknowledgments

We are grateful to the GenoToul (Toulouse) Bioinformatic platform for providing us computational support for this work. This work is supported by Science Foundation Ireland (SFI) Grant 10/IN.1/I3032. The Insight Centre for Data Analytics is supported by SFI Grant SFI/12/RC/2289.

Author information

Authors and Affiliations

  1. Insight Centre for Data Analytics, University College Cork, Cork, Ireland

    Barry Hurley & Barry O’Sullivan

  2. MIAT, UR-875, INRA, 31320, Castanet Tolosan, France

    David Allouche, George Katsirelos, Thomas Schiex, Matthias Zytnicki & Simon de Givry

Authors
  1. Barry Hurley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Barry O’Sullivan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Allouche
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. George Katsirelos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thomas Schiex
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthias Zytnicki
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Simon de Givry
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simon de Givry.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hurley, B., O’Sullivan, B., Allouche, D. et al. Multi-language evaluation of exact solvers in graphical model discrete optimization. Constraints 21, 413–434 (2016). https://doi.org/10.1007/s10601-016-9245-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10601-016-9245-y

Keywords

Search

Navigation