Sampling from the complement of a polyhedron: An MCMC algorithm for data augmentation

doi:10.1016/j.orl.2020.08.014

Operations Research Letters

Volume 48, Issue 6, November 2020, Pages 744-751

https://doi.org/10.1016/j.orl.2020.08.014 Get rights and content

Abstract

We present an MCMC algorithm for sampling from the complement of a polyhedron. Our approach is based on the Shake-and-bake algorithm for sampling from the boundary of a set and provably covers the complement. We use this algorithm for data augmentation in a machine learning task of classifying a hidden feasible set in a data-driven optimization pipeline. Numerical results on simulated and MIPLIB instances demonstrate that our algorithm, along with a supervised learning technique, outperforms conventional unsupervised baselines.

Introduction

High-dimensional sampling is a fundamental tool that is used in various domains such as machine learning [1], optimization [7], and stochastic modeling [15]. Sampling from a high-dimensional set is a key component of approximation algorithms for problems that cannot be tractably solved with conventional methods.

The literature on high-dimensional sampling primarily addresses the problem of efficiently sampling points that lie within a convex set, with the family of Markov Chain Monte Carlo (MCMC) methods being the most commonly used approach in this setting [9]. Recent applications in ranking have also generated interest in the related problem of sampling from the boundary of convex sets [10]. However, to the best of our knowledge, there has not been prior work on sampling from the complement of a convex set.

In this paper, we consider the task of efficiently sampling from sets defined by the complement of a polyhedron for which there exist many potential applications. For example, the complement operator can be used to represent disjunctions, which when combined with conjunctions, can describe arbitrary sets. We note that both disjunctive sets and MCMC sampling are common tools in mixed-integer programming [4], [12]. Another application is in data-driven optimization, where sampling from the complement of the feasible set of a partially described optimization problem can help train a machine learning model to predict aspects of the decision-making problem [3].

In practical decision-making problems, an optimization model is designed with structural assumptions and parameter estimates. The growing paradigm of data-driven optimization uses historical decisions to inform the construction of these parameters. For example in data-driven robust optimization, an uncertainty set around the parameters of a constraint is created by analyzing prior instances [6]. In another example, the hidden constraints of an optimization problem are learned by training a binary classifier using a data set of historical decisions [3].

The use of machine learning in most data-driven optimization settings is often limited by the fact that in most applications, the available data consists only of past implemented decisions. Decision-makers rarely collect data on unimplemented decisions, meaning that the available information belongs to a single category. Consequently, supervised learning techniques that learn by differentiating between instances of different classes are often untenable and methods are typically limited to statistical frameworks or unsupervised models [3], [6].

Our objective is to solve an optimization problem that has hidden constraints. To this end, we wish to construct a barrier function for use in an interior point method when we do not know the true feasible set, but are instead given a relaxation and a data set of feasible decisions [3]. Our approach is to construct a binary classifier that predicts whether a given decision is feasible or not. By sampling from a known subset of the infeasible region, i.e., the complement of the relaxation, we can augment our initial data set of feasible decisions with unimplemented decisions to train the classifier.

The complement of a polyhedron is a non-convex set, making conventional sampling techniques inappropriate for the task. Our key methodological contribution is an efficient MCMC algorithm for generating a sequence of points from the complement. Our approach extends current techniques that sample from the boundary of the set. We prove that our algorithm is guaranteed to cover the entire complement region and show that this is a sufficient condition to create a binary classifier that learns to distinguish between feasible and infeasible points for high-dimensional problems.

To demonstrate the effectiveness of our approach, we perform several numerical experiments on a variety of optimization problems. For each problem, a set of feasible decisions from an unknown feasible set is provided and we generate an artificial data set of infeasible decisions that lie in the complement of a known polyhedral relaxation using our MCMC algorithm. We then train a classifier to learn a separating boundary between the feasible and the infeasible data set. We compare our approach with several unsupervised density estimation baselines that are not augmented with data sampled from the complement. Using a simulated fractional knapsack problem, we show that our approach is essential for creating classifiers that (i) perform well when a tight separating boundary between feasible and infeasible regions is required; and (ii) when the data set of feasible decisions is small. Further, we consider linearized relaxations of all MIPLIB [14] instances with less than 80 variables and demonstrate that our sampling-based classifier significantly outperforms all baseline models. Code for our experiments are available at https://github.com/rafidrm/mcmc-complement.

Although our focus is on polyhedra, our approach can be adapted to non-linear sets similar to how sampling from the boundary of a polyhedron generalizes to the boundary of arbitrary convex sets. In the Online Appendix, we explore how to sample from the complement of an ellipsoid and prove that our algorithm also covers the complement in this setting. As a result, we demonstrate that our MCMC algorithm has more general applicability, and can be applied to, for instance, problems in robust optimization which commonly involve ellipsoidal uncertainty sets [5].

Section snippets

Preliminaries

Consider a polyhedron $X ≔ {x \in R^{n} ∣ a_{m}^{T} x \leq b_{m}, 1 \leq m \leq M}$ . There exist several algorithms for sampling from the interior $int (X)$ , with the most well-known being the Hit-and-Run (HR) algorithm [16]. Similarly, the Shake-and-Bake (SB) algorithm is the most well-known approach to sampling from the boundary $bd (X)$ [8]. These algorithms fall under the family of MCMC techniques which operate by constructing a sequence of points governed by a proposal function. The sequence describes a Markov chain whose stationary

Sampling from the complement of a polyhedron

Assume that $X$ is full-dimensional and non-empty. Given a polyhedron $X$ , we generate a sequence of $N$ points $D = {x_{i}}_{i = 0}^{N}$ such that $D \subset R^{n} ∖ X$ . In each iteration of SB, a direction vector is sampled and the next point on the boundary is found by moving in the given direction from the current point. Notice, however, that moving in the negative direction from the current point yields points that lie in the complement of the polyhedron, i.e., $x \in R^{n} ∖ X$ .

In our algorithm, we treat SB as a Hidden Markov chain.

Numerical analysis

We implement Algorithm 1 and the corresponding SB-based classifier to learn the hidden feasible set $\hat{X}$ . Given a data set of feasible decisions $\hat{D} = {\{{\hat{x}}_{i}\}}_{i = 0}^{N} \subseteq \hat{X}$ and a relaxation $X$ , we augment our data set by sampling infeasible decisions before training an SB-based classifier to predict whether a decision is feasible or not with respect to $\hat{X}$ .

Classical approaches towards constructing a classifier $D (x)$ would not have a set of infeasible points $D$ and would thus be forced to use some form of

Conclusion

A potential extension of this work lies in sampling from the complement of sets that are prominent in other areas of constrained optimization. To this end, in the Online Appendix, we demonstrate that the Complement SB algorithm can be used to sample points from the complement of ellipsoidal feasible sets. However, these results require several technical extensions. In future work, we hope to generalize our results and prove that the Complement SB algorithm generates a stationary distribution

Acknowledgments

The authors would like to thank the anonymous reviewer, associate editor, and area editor for their helpful comments, and Philip Mar for helpful discussions on extending the theory. This work was supported by the Natural Sciences and Engineering Research Council of Canada .

References (17)

BalasE.
Disjunctive programming
AndrieuC. et al.
An introduction to MCMC for machine learning
Mach. Learn.
(2003)
ArjovskyM. et al.
Towards principled methods for training generative adversarial networks
(2017)
BabierA. et al.
Learning to optimize with hidden constraints
(2018)
BertsimasD. et al.
Theory and applications of robust optimization
SIAM Rev.
(2011)
BertsimasD. et al.
Data-driven estimation in equilibrium using inverse optimization
Math. Program.
(2015)
BertsimasD. et al.
Solving convex programs by random walks
J. ACM
(2004)
BoenderC.G.E. et al.
Shake-and-bake algorithms for generating uniform points on the boundary of bounded polyhedra
Oper. Res.
(1991)

There are more references available in the full text version of this article.

Cited by (3)

Desensitization method of meteorological data based on differential privacy protection
2023, Journal of Cleaner Production
Artificial intelligence (AI) profoundly affects the research of meteorology. As an AI method, deep learning greatly improves the accuracy of weather forecasting. Deep learning training requires a lot of data, while the collection of meteorological data faces some problems, such as the long collection cycle, the high cost, and the desensitization requirement. We propose a deep learning model called MDPGAN (Meteorology Differential Privacy Generative Adversarial Network), which introduces a differential privacy framework to reduce the risk of identifying real data by querying synthetic data. The MDPGAN model can generate synthetic weather data with similar statistical characteristics to real weather data. The data generated by the MDPGAN model meets the requirements of data augmentation and data desensitization at the same time. In this paper, the meteorological data set of Kennedy Airport published by NOAA (National Oceanic and Atmospheric Administration) was used for the experiments of the MDPGAN model. The reliability and validity of generated meteorological data were analyzed and tested. The comparison between the generated data and the real data shows that they have similar statistical characteristics, and the synthetic data has achieved good results in the time series prediction of temperature changes. The MDPGAN model provides a convenient tool for the meteorology researches based on deep learning, which can automatically generate a large amount of safe and reliable data, especially suitable for the meteorology researches with small sample data.
A comprehensive review of task understanding of command-triggered execution of tasks for service robots
2023, Artificial Intelligence Review
Probability Quantization Model for Sample-to-Sample Stochastic Sampling
2022, Arabian Journal for Science and Engineering

View full text

Sampling from the complement of a polyhedron: An MCMC algorithm for data augmentation

Abstract

Introduction

Section snippets

Preliminaries

Sampling from the complement of a polyhedron

Numerical analysis

Conclusion

Acknowledgments

An introduction to MCMC for machine learning

Mach. Learn.

Towards principled methods for training generative adversarial networks

Learning to optimize with hidden constraints

Theory and applications of robust optimization

SIAM Rev.

Data-driven estimation in equilibrium using inverse optimization

Math. Program.

Solving convex programs by random walks

J. ACM

Shake-and-bake algorithms for generating uniform points on the boundary of bounded polyhedra

Oper. Res.