Skip to main content
SpringerLink
Log in

A lagrangian propagator for artificial neural networks in constraint programming

  • Published:
Constraints Aims and scope Submit manuscript

Abstract

This paper discusses a new method to perform propagation over a (two-layer, feed-forward) Neural Network embedded in a Constraint Programming model. The method is meant to be employed in Empirical Model Learning, a technique designed to enable optimal decision making over systems that cannot be modeled via conventional declarative means. The key step in Empirical Model Learning is to embed a Machine Learning model into a combinatorial model. It has been showed that Neural Networks can be embedded in a Constraint Programming model by simply encoding each neuron as a global constraint, which is then propagated individually. Unfortunately, this decomposition approach may lead to weak bounds. To overcome such limitation, we propose a new network-level propagator based on a non-linear Lagrangian relaxation that is solved with a subgradient algorithm. The method proved capable of dramatically reducing the search tree size on a thermal-aware dispatching problem on multicore CPUs. The overhead for optimizing the Lagrangian multipliers is kept within a reasonable level via a few simple techniques. This paper is an extended version of [27], featuring an improved structure, a new filtering technique for the network inputs, a set of overhead reduction techniques, and a thorough experimentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Of course the propagation is likely to be less effective for more complex networks.

  2. Real-valued variables with fixed precision can be modeled via integer variables: e.g. a number in [0,1] with precision 0.01 corresponds to a number ∈{0..100}. This representation requires some care to ensure consistent rounding. More details can be found in [2].

  3. At https://bitbucket.org/m_lombardi/constraints-15-ann-lag-resources

  4. Google OR-tools, at https://developers.google.com/optimization/

  5. At https://bitbucket.org/m_lombardi/constraints-15-ann-lag-resources

References

  1. Audet, C. (2014). A survey on direct search methods for blackbox optimization and their applications. In Mathematics Without Boundaries (pp. 31–56): Springer.

  2. Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2011). Neuron Constraints to Model Complex Real-World Problems. In Proc. of CP (pp. 115–129).

  3. Bartolini, A., Lombardi, M., Milano, M., & Benini, L. (2012). Optimization and Controlled Systems: A Case Study on Thermal Aware Workload Dispatching. Proc. of AAAI.

  4. Basheer, I.A., & Hajmeer, M. (2000). Artificial neural networks: fundamentals, computing, design, and application. Journal of Microbiological Methods, 43(1), 3–31.

    Article  Google Scholar 

  5. Belew, R.K., McInerney, J., & Schraudolph, N.N. (1991). Evolving networks: Using the genetic algorithm with connectionist learning. Proc. of Artificial Life, 511–547.

  6. Belotti, P., Lee, J., Liberti, L., Margot, F., & Wächter, A. (2009). Branching and bounds tightening techniques for non-convex MINLP. Optimization Methods and Software, 24(4-5), 597–634.

    Article  MathSciNet  MATH  Google Scholar 

  7. Bergman, D., Cirė, A. A., & van Hoeve, W.-J. (2015). Lagrangian bounds from decision diagrams. Constraints, 20(3), 346–361.

    Article  MathSciNet  MATH  Google Scholar 

  8. Bonfietti, A., & Lombardi, M. (2012). The weighted average constraint. In Proc. of CP (pp. 191–206): Springer.

  9. Bonfietti, A., Lombardi, M., & Milano, M. (2015). Embedding decision trees and random forests in constraint programming. In Proc. of CPAIOR (pp. 74–90).

  10. Cambazard, H., & Fages, J.-G. (2015). New filtering for atmostnvalue and its weighted variant: A lagrangian approach. Constraints, 20(3), 362–380.

    Article  MathSciNet  MATH  Google Scholar 

  11. Chow, T.T., Zhang, G.Q., Lin, Z., & Song, C.L. (2002). Global optimization of absorption chiller system by genetic algorithm and neural network. Energy and Buildings, 34(1), 103–109.

    Article  Google Scholar 

  12. Conn, A.R., Scheinberg, K., & Vicente, L.N. (2009). Introduction To Derivative-free Optimization, volume 8. Siam.

  13. d’Antonio, G., & Frangioni, A. (2009). Convergence analysis of deflected conditional approximate subgradient methods. SIAM Journal on Optimization, 20(1), 357–386.

    Article  MathSciNet  MATH  Google Scholar 

  14. Focacci, F., Lodi, A., & Milano, M. (1999). Cost-based domain filtering.

  15. Ge, S.S., Hang, C.C., Lee, T.H., & Zhang, T. (2010). Stable adaptive neural network control. Springer Publishing Company, Incorporated.

  16. Gent, I.P., Kotthoff, L., Miguel, I., & Nightingale, P. (2010). Machine learning for constraint solver design – A case study for the alldifferent constraint. CoRR, abs/1008.4326.

  17. Glover, F., Kelly, J.P., & Laguna, M. (1999). New Advances for Wedding optimization and simulation. In Proc. of WSC. IEEE (pp. 255–260).

  18. Gopalakrishnan, K., & Asce, A.M. (2009). Neural Network Swarm Intelligence Hybrid Nonlinear Optimization Algorithm for Pavement Moduli Back-Calculation. Journal of Transportation Engineering, 136(6), 528–536.

    Article  Google Scholar 

  19. Gualandi, S., & Malucelli, F. (2012). Resource constrained shortest paths with a super additive objective function. In Proc. of CP (pp. 299–315): Springer.

  20. Howard, J., Dighe, S., Vangal, S.R., Ruhl, G., Borkar, N., Jain, S., Erraguntla, V., Konow, M., Riepen, M., Gries, M., & et al. (2011). A 48-core ia-32 processor in 45 nm cmos using on-die message-passing and dvfs for performance and power scaling. IEEE Journal of Solid-State Circuits, 46(1), 173–183.

    Article  Google Scholar 

  21. Huang, W., Ghosh, S., & Velusamy, S. (2006). HotSpot: A compact thermal modeling methodology for early-stage VLSI design. IEEE Transactions on VLSI, 14 (5), 501–513.

    Article  Google Scholar 

  22. Hutter, F., Hoos, H. H., Leyton-Brown, K., & Stu̇tzle, T. (2009). Paramils: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36, 267–306.

    MATH  Google Scholar 

  23. Jayaseelan, R., & Mitra, T. (2009). A hybrid local-global approach for multi-core thermal management. In Proc. of ICCAD (pp. 314–320): ACM Press.

  24. Kiranyaz, S., Ince, T., Yildirim, A., & Gabbouj, M. (2009). Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Networks, 22(10), 1448–1462.

    Article  Google Scholar 

  25. Lemaréchal, C. (2001). Lagrangian relaxation. In Computational Combinatorial Optimization (pp. 112–156): Springer.

  26. Ljung, L. (1999). System identification. Wiley Online Library.

  27. Lombardi, M., & Gualandi, S. (2013). A new propagator for two-layer neural networks in empirical model learning. In Proc. of CP (pp. 448–463).

  28. Montana, D.J., & Davis, L. (1989). Training feedforward neural networks using genetic algorithms. In Proc. of IJCAI (pp. 762–767).

  29. Moore, J., Chase, J.S., & Ranganathan, P. (2006). Weatherman: Automated, Online and Predictive Thermal Mapping and Management for Data Centers. In Proc. of ICAC. IEEE (pp. 155–164).

  30. Moré, J.J. (1978). The Levenberg-Marquardt algorithm: implementation and theory. In Numerical analysis (pp. 105–116): Springer.

  31. Queipo, N.V., Haftka, R.T., Shyy, W., Goel, T., Vaidyanathan, R., & Tucker, P.K. (2005). Surrogate-based analysis and optimization. Progress In Aerospace Sciences, 41(1), 1–28.

    Article  Google Scholar 

  32. Sellmann, M. (2004). Theoretical foundations of cp-based lagrangian relaxation. In Proc. of CP (pp. 634–647): Springer.

  33. Sellmann, M., & Fahle, T. (2003). Constraint programming based lagrangian relaxation for the automatic recording problem. Annals of Operations Research, 118 (1–4), 17–33.

    Article  MathSciNet  MATH  Google Scholar 

  34. Slusky, M.R., & van Hoeve, W.J. (2013). A lagrangian relaxation for golomb rulers. In Proc. of CPAIOR (pp. 251–267): Springer.

  35. Van Cauwelaert, S., Lombardi, M., & Schaus, P. (2015). Understanding the potential of propagators. In Proc. of CPAIOR (pp. 427–436).

  36. Zhang, G., Patuwo, B.E., & Hu, M.Y. (1998). Forecasting with artificial neural networks: The state of the art. International Journal of Forecasting, 14(1), 35–62.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Bologna, Viale del Risorgimento 2, 40136, Bologna, Italy

    Michele Lombardi

  2. AntOptima SA, Via Aprica 26, 6900, Lugano, Switzerland

    Stefano Gualandi

Authors
  1. Michele Lombardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefano Gualandi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michele Lombardi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lombardi, M., Gualandi, S. A lagrangian propagator for artificial neural networks in constraint programming. Constraints 21, 435–462 (2016). https://doi.org/10.1007/s10601-015-9234-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10601-015-9234-6

Keywords

Search

Navigation