Learning and Reasoning with Fast Semidefinite Programming and Mixing Methods
Semidefinite programming has long been a theoretically powerful tool for solving relaxations of challenging, often NP-hard optimization problems. However, it has
typically not been practical for most large-scale tasks, owing to the high memory and computational cost of typical solvers for solving SDPs. In this thesis, we aim
to break the barrier and bring SDP’s power back to large-scale machine learning problems. To achieve this, we introduce a series of optimization solvers, operating on
the low-rank or low-cardinality manifolds of the semidefinite variables. We find that in many domains, these methods allow SDP relaxations to exceed the state of the art
in terms of both computational cost and the relevant performance metrics. First, we proposed the Mixing method, a low-rank SDP solver aimed at the classical MAXCUT SDP relaxation. We also show that the Mixing method can
accurately estimate the mode and partition function of the pairwise Markov Random Fields, and scales to millions of variables. Further, we show how to learn the parameters inside SDPs by analytically differentiating through the optimization problem with implicit differentiation and the mixing methods, which leads to a differentiable SAT solver that can be integrated within the loop of larger deep learning
systems. For nonnegative constraints, we propose a separate variant aimed at low cardinality SDPs, and demonstrate how to apply the method to community detection on finding clusters within large-scale networks. Finally, we show that the technique can also be applied to more generic problems, such as a generic linear programming problems (with arbitrarily structured constraints), and we use this approach to develop
a scalable sparse linear programming solver that improves solution speed over existing state-of-the-art commercial solvers
History
Date
2021-08-05Degree Type
- Dissertation
Department
- Machine Learning
Degree Name
- Doctor of Philosophy (PhD)