Abstract
Scenario optimization is a broad methodology to perform optimization based on empirical knowledge. One collects previous cases, called “scenarios”, for the set-up in which optimization is being performed, and makes a decision that is optimal for the cases that have been collected. For convex optimization, a solid theory has been developed that provides guarantees of performance, and constraint satisfaction, of the scenario solution. In this paper, we open a new direction of investigation: the risk that a performance is not achieved, or that constraints are violated, is studied jointly with the complexity (as precisely defined in the paper) of the solution. It is shown that the joint probability distribution of risk and complexity is concentrated in such a way that the complexity carries fundamental information to tightly judge the risk. This result is obtained without requiring extra knowledge on the underlying optimization problem than that carried by the scenarios; in particular, no extra knowledge on the distribution by which scenarios are generated is assumed, so that the result is broadly applicable. This deep-seated result unveils a fundamental and general structure of data-driven optimization and suggests practical approaches for risk assessment.












Similar content being viewed by others
Notes
No limitations are imposed on \({\varDelta }\) like e.g. that \({\varDelta }\) is a subset of a Eucledian space or of a vector space, nor is \({\varDelta }\) endowed with a metric or a topology. \({\varDelta }\) is just a generic set that forms a probability space together with \(\mathcal {F}\) and \({\mathbb {P}}\). Hence, ideas like “the sample \(\delta _i\), \(i=1,\ldots ,N\), covers, or fills up, \({\varDelta }\)” are void of any meaning. This generality in the definition of \({\varDelta }\) is important for the widespread applicability of the theory.
We assume that function \(\ell (v,\delta )\) is convex in v for any given value of \(\delta \), while its dependence on \(\delta \) is arbitrary.
Throughout, we assume that a solution exists. If more than one solution exists, a solution is singled out by means of a convex tie-break rule according to the approach of [10].
We remark that \(V(x^*_N)\) quantifies the risk, which refers to the chance-constrained feasibility, while the value is not at issue here.
To this purpose, it is enough to eliminate one by one the constraints and recompute the solution, the support constraints are those whose elimination determines a change in the solution.
\({{\mathcal {Z}}}\) can be any set, without any Euclidean structure. We change notation from \({{\mathcal {X}}}\) to \({{\mathcal {Z}}}\) because in some applications \({{\mathcal {Z}}}\) is the same as \({{\mathcal {X}}}\) augmented with extra elements; concrete examples of decision sets are provided in Sects. 5.1 and 5.2.
Note that only the tie with respect to x is broken by \(t_1(x)\), \(t_2(x)\), \(t_3(x)\), .... On the other hand, for a given \(x^*_m\) the values of \(\xi _i\), \(i=1,\ldots ,m\), remain unambiguously determined at optimum by relation \(\xi ^*_{i,m} = f(x^*_m,\delta _i)\), so that no tie on \(\xi _i\), \(i=1,\ldots ,m\), can persist after the tie on x is broken.
Intuitively, the proportion of violated constraints (empirical risk) is not a valid indicator of the true risk \(V(x^*_N)\) since optimization generates a bias towards larger risks by drifting the solution against the constraints. The excess with respect to the number of violated constraints that appears in the computation of \({\tilde{s}}^*_N\) captures this mechanism and offers one the possibility to obtain tight evaluations of the risk, as quantified by the lower and upper bounds in Eq. (19), independently of the problem under consideration.
The increase is not always monotone.
Other methods have been proposed in the literature to trade the risk for an improved cost. One method consists in allowing the solution to violate a preset proportion of the empirical constraints (chance-constrained problem over the empirical distribution). In the context of scenario optimization, this approach is described in [18], where practically useful, but untight, bounds on the risk are also derived. More generally, the problem of relating the empirical risk to chance-constrained feasibility is dealt with in many papers including [6, 7, 44, 57]. The problem of finding a solution that violates a preset proportion of the empirical constraints is a non-convex problem that is difficult to solve in general. The formulation in (14) is convex and this eases the problem of finding a solution. Interestingly, as already noted, this formulation is amenable to tight evaluations of the risk.
For reproducibility, we inform the reader about the mechanism by which \(q_{j,k}(\delta )\) were generated. Let \(\delta =(\alpha _1,\alpha _2,\gamma _{1,1},\ldots ,\gamma _{50,1},\gamma _{1,2},\ldots ,\gamma _{50,2})\), where, for \(k=1,2\) and \(j=1,\ldots ,50\), \(\alpha _k \sim \mathcal {U}[10,50]\) (i.e., \(\alpha _k\) is uniformly distributed in [10, 50]), \(\gamma _{j,k} \sim {\mathcal {U}}[-2.5,2.5]\), and all these variables are independent one of the others. Then, \(q_{j,k}(\delta ) = (\alpha _k^{\frac{1}{4}} + \gamma _{j,k})^{-1}\), \(j=1,\ldots ,50\), \(k=1,2\). Moreover, in the simulation, we took \(a_1=a_2=1\).
Notice that, strictly speaking, this choice of \(f(x,\delta )\) does not satisfy Assumption 6. Reason is that setting to zero \(f(x,\delta )\) when \(\sum _{j=1}^{50} q_{j,1}(\delta ) x_j - a_1\) and \(\sum _{j=1}^{50} q_{j,2}(\delta ) x_j - a_2\) are negative, as is done in (22), generates regions with positive volume in the domain in \( { {{\mathbb {R}}}^{50} } \) for x where \(f(x,\delta ) = 0\). However, an easy inspection of the derivation of Theorem 4 shows that the requirement of Assumption 6 that, for every x, \({\mathbb {P}}\{\delta : f(x,\delta ) = 0\} = 0\) can be relaxed to requiring that, for every x, \({\mathbb {P}}\{\delta : x \text{ is } \text{ on } \text{ the } \text{ boundary } \text{ of } \text{ the } \text{ constraint } \{f(x,\delta ) \le 0 \} \} = 0\), and the theory goes through unaltered with the only modifications that, throughout, “\(f(x,\delta ) = 0\)” becomes “x is on the boundary of the constraint \(\{f(x,\delta ) \le 0 \}\)”, “\(f(x,\delta ) < 0\)” becomes “x is in the interior of the constraint \(\{f(x,\delta ) \le 0 \}\)”, and “\(f(x,\delta ) \ge 0\)” becomes “x violates or is on the boundary of the constraint \(\{f(x,\delta ) \le 0 \}\)”. While we have preferred in the general presentation the simpler formulation of Assumption 6, this second formulation leads to zero volume regions in the domain in \( { {{\mathbb {R}}}^{50} } \) for x in the present example.
The reason for introducing H is that the theorem will be proven in a slightly more general form where H is any integer \(\ge 1\) and not just 3N. The choice \(H = 3N\) gives satisfactory evaluations in most cases, and this is why Theorem 2 was stated with \(H = 3N\). However, the extra generality allowed by other values of H can turn out to be useful to tighten the bounds \({\underline{\epsilon }}(\cdot )\) and \({\overline{\epsilon }}(\cdot )\) in some cases when N is not too large. This issue is not further discussed in this paper.
References
Alamo, T., Tempo, R., Camacho, E.: A randomized strategy for probabilistic solutions of uncertain feasibility and optimization problems. IEEE Trans. Autom. Control 54(11), 2545–2559 (2009)
Baronio, F., Baronio, M., Campi, M., Caré, A., Garatti, S.: Ventricular defribillation: classification with GEM and a roadmap for future investigations. In: Proceedings of the 56th IEEE Conference on Decision and Control. Melbourne, Australia (2017)
Bayraksan, G., Morton, D.: Assessing solution quality in stochastic programs. Math. Program. 108, 495–514 (2006)
Bayraksan, G., Morton, D.: Assessing solution quality in stochastic programs via sampling. In: Oskoorouchi, M. (ed.) Tutorials in Operations Research, pp. 102–122. Informs (2009)
Ben-Tal, A., Nemirovski, A.: On safe tractable approximations of chance-constrained linear matrix inequalities. Math. Oper. Res. 34(1), 1–25 (2009)
Bertsimas, D., Gupta, V., Kallus, N.: Data-driven robust optimization. Math. Program. 167, 235–292 (2018)
Bertsimas, D., Gupta, V., Kallus, N.: Robust sample average approximation. Math. Program. 171, 217–282 (2018)
Bertsimas, D., Thiele, A.: Robust and data-driven optimization: modern decision-making under uncertainty. In: Tutorials on Operations Research. INFORMS (2006)
Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Calafiore, G., Campi, M.: Uncertain convex programs: randomized solutions and confidence levels. Math. Program. 102(1), 25–46 (2005). https://doi.org/10.1007/s10107-003-0499-y
Calafiore, G., Campi, M.: The scenario approach to robust control design. IEEE Trans. Autom. Control 51(5), 742–753 (2006)
Calafiore, G., El Ghaoui, L.: On distributionally robust chance-constrained linear programs. Math. Program. 130(1), 1–22 (2006)
Carè, A., Ramponi, F.A., Campi, M.C.: A new classification algorithm with guaranteed sensitivity and specificity for medical applications. IEEE Control Syst. Lett. 2, 393–398 (2018)
Campi, M.: Classification with guaranteed probability of error. Mach. Learn. 80, 63–84 (2010)
Campi, M., Calafiore, G., Garatti, S.: Interval predictor models: identification and reliability. Automatica 45(2), 382–392 (2009). https://doi.org/10.1016/j.automatica.2008.09.004
Campi, M., Carè, A.: Random convex programs with \(l_1\)-regularization: sparsity and generalization. SIAM J. Control Optim. 51(5), 3532–3557 (2013)
Campi, M., Garatti, S.: The exact feasibility of randomized solutions of uncertain convex programs. SIAM J. Optim. 19(3), 1211–1230 (2008). https://doi.org/10.1137/07069821X
Campi, M., Garatti, S.: A sampling-and-discarding approach to chance-constrained optimization: feasibility and optimality. J. Optim. Theory Appl. 148(2), 257–280 (2011)
Campi, M., Garatti, S.: Wait-and-judge scenario optimization. Math. Program. 167(1), 155–189 (2018)
Campi, M., Garatti, S., Prandini, M.: The scenario approach for systems and control design. Annu. Rev. Control 33(2), 149–157 (2009). https://doi.org/10.1016/j.arcontrol.2009.07.001
Carè, A., Garatti, S., Campi, M.: FAST—fast algorithm for the scenario technique. Oper. Res. 62(3), 662–671 (2014)
Carè, A., Garatti, S., Campi, M.: Scenario min-max optimization and the risk of empirical costs. SIAM J. Optim. 25(4), 2061–2080 (2015)
Crespo, L., Giesy, D., Kenny, S.: Interval predictor models with a formal characterization of uncertainty and reliability. In: Proceedings of the 53rd IEEE Conference on Decision and Control (CDC), pp. 5991–5996. Los Angeles, CA, USA (2014)
Crespo, L., Kenny, S., Giesy, D.: Random predictor models for rigorous uncertainty quantification. Int. J. Uncertain. Quantif. 5(5), 469–489 (2015)
Crespo, L., Kenny, S., Giesy, D., Norman, R., Blattnig, S.: Application of interval predictor models to space radiation shielding. In: Proceedings of the 18th AIAA Non-Deterministic Approaches Conference. San Diego, CA, USA (2016)
de Mello, T.H.: Variable-sample methods for stochastic optimization. ACM Trans. Model. Comput. Simul. 13, 108–133 (2003)
de Mello, T.H., Bayraksan, G.: Monte Carlo sampling-based methods for stochastic optimization. Surv. Oper. Res. Manag. Sci. 19(1), 56–85 (2014)
Delage, E., Ye, Y.: Distributionally robust optimization under moment uncertainty with application to data-driven problems. Oper. Res. 58(3), 596–612 (2010)
Dentcheva, D.: Optimization models with probabilistic constraints. In: Calafiore, G., Dabbene, F. (eds.) Probabilistic and Randomized Methods for Design Under Uncertainty. Springer, London (2006)
Erdogan, E., Iyengar, G.: Ambiguous chance constrained problems and robust optimization. Math. Program. 107, 37–61 (2006)
Esfahani, P., Sutter, T., Lygeros, J.: Performance bounds for the scenario approach and an extension to a class of non-convex programs. IEEE Trans. Autom. Control 60(1), 46–58 (2015)
Esfahani, P.M., Kuhn, D.: Data-driven distributionally robust optimization using the wasserstein metric: performance guarantees and tractable reformulations. Math. Program. (2017). https://doi.org/10.1007/s10107-017-1172-1
Fabozzi, F., Kolm, P., Pachamanova, D., Focardi, S.: Robust Portfolio Optimization and Management. Wiley, Hoboken (2010)
Garatti, S., Campi, M.: Modulating robustness in control design: principles and algorithms. IEEE Control Syst. Mag. 33(2), 36–51 (2013). https://doi.org/10.1109/MCS.2012.2234964
Goh, J., Sim, M.: Distributionally robust optimization and its tractable approximations. Oper. Res. 58, 902–917 (2010)
Grammatico, S., Zhang, X., Margellos, K., Goulart, P., Lygeros, J.: A scenario approach for non-convex control design. IEEE Trans. Autom. Control 61(2), 334–345 (2016)
Gupta, V.: Near-optimal ambiguity sets for distributionally robust optimization. Manag. Sci. 65(9), 4242–4260 (2019)
Hanasusanto, G., Roitch, V., Kuhn, D., Wiesemann, W.: A distributionally robust perspective on uncertainty quantification and chance constrained programming. Math. Program. 151(1), 35–62 (2015)
Hong, L., Hu, Z., Liu, G.: Monte Carlo methods for value-at-risk and conditional value-at-risk: a review. ACM Trans. Model. Comput. Simul. 24(4), 22:1–22:37 (2014)
Hu, Z., Hong, L.: Kullback–Leiber divergence constrained distributionally robust optimization (2013). http://www.optimization-online.org/DB_HTML/2012/11/3677.html
Jiang, R., Guan, Y.: Data-driven chance constrained stochastic program. Math. Program. 158(1–2), 291–327 (2016)
Lam, H.: Recovering best statistical guarantees via the empirical divergence-based distributionally robust optimization. Oper. Res. 67(4), 1090–1105 (2019)
Linderoth, J., Shapiro, A., Wright, S.: The empirical behavior of sampling methods for stochastic programming. Ann. Oper. Res. 142, 215–241 (2006)
Luedtke, J., Ahmed, S.: A sample approximation approach for optimization with probabilistic constraints. SIAM J. Optim. 19, 674–699 (2008). https://doi.org/10.1137/070702928
Luedtke, J., Ahmed, S., Nemhauser, G.: An integer programming approach for linear programs with probabilistic constraints. Math. Program. 122(2), 247–272 (2010)
Mak, W., Morton, D., Wood, R.: Monte Carlo bounding techniques for determing solution quality in stochastic programs. Oper. Res. Lett. 24, 47–56 (1999)
Margellos, K., Prandini, M., Lygeros, J.: On the connection between compression learning and scenario based single-stage and cascading optimization problems. IEEE Trans. Autom. Control 60(10), 2716–2721 (2015)
Nemirovski, A.: On safe tractable approximations of chance constraints. Eur. J. Oper. Res. 219, 707–718 (2012)
Nemirovski, A., Shapiro, A.: Convex approximations of chance constrained programs. SIAM J. Optim. 17(4), 969–996 (2006). https://doi.org/10.1137/050622328
Nemirovski, A., Shapiro, A.: Scenario approximations of chance constraints. In: Calafiore, G., Dabbene, F. (eds.) Probabilistic and Randomized Methods for Design Under Uncertainty. Springer, London (2006)
Pagnoncelli, B., Ahmed, S., Shapiro, A.: Sample average approximation method for chance constrained programming: theory and applications. J. Optim. Theory Appl. 142(2), 399–416 (2009)
Pagnoncelli, B., Reich, D., Campi, M.: Risk-return trade-off with the scenario approach in practice: a case study in portfolio selection. J. Optim. Theory Appl. 155(2), 707–722 (2012)
Pagnoncelli, B., Vanduffel, S.: A provisioning problem with stochastic payments. Eur. J. Oper. Res. 221(2), 445–453 (2012)
Pflug, G., Wozabal, D.: Ambiguity in portfolio selection. Quant. Finance 7, 435–442 (2007)
Schildbach, G., Fagiano, L., Frei, C., Morari, M.: The scenario approach for stochastic model predictive control with bounds on closed-loop constraint violations. Automatica 50(12), 3009–3018 (2014)
Schildbach, G., Fagiano, L., Morari, M.: Randomized solutions to convex programs with multiple chance constraints. SIAM J. Optim. 23(4), 2479–2501 (2013)
Shapiro, A.: Monte–Carlo sampling methods. In: Ruszczyński, A., Shapiro, A. (eds.) Stochastic Programming, volume 10 of Handbooks in Operations Research and Management Science. Elsevier, London (2003)
Shapiro, A., Dentcheva, D., Ruszczyński, A.: Lectures on Stochastic Programming: Modeling and Theory. MPS-SIAM, Philadelphia (2009)
Shiryaev, A.: Probability. Springer, New York (1996)
Thiele, A.: Robust stochastic programming with uncertain probabilities. IMA J. Manag. Math. 19(3), 289–321 (2008). https://doi.org/10.1093/imaman/dpm011
Van Parys, B., Esfahani, P., Kuhn, D.: From data to decisions: distributionally robust optimization is optimal. (2017). arxiv:1704.04118
Vayanos, P., Kuhn, D., Rustem, B.: A constraint sampling approach for multistage robust optimization. Automatica 48(3), 459–471 (2012)
Welsh, J., Kong, H.: Robust experiment design through randomisation with chance constraints. In: Proceedings of the 18th IFAC World Congress, Milan, Italy (2011)
Welsh, J., Rojas, C.: A scenario based approach to robust experiment design. In: Proceedings of the 15th IFAC Symposium on System Identification. Saint-Malo, France (2009)
Wieseman, W., Kuhn, D., Sim, M.: Distributionally robust convex optimization. Oper. Res. 62, 1358–1376 (2014)
Wozabal, D.: A framework for optimization under ambiguity. Ann. Oper. Res. 193, 21–47 (2012)
Zhang, X., Grammatico, S., Schildbach, G., Goulart, P., Lygeros, J.: On the sample size of random convex programs with structured dependence on the uncertainty. Automatica 60, 182–188 (2015)
Zhou, Z., Cogill, R.: Reliable approximations of probability-constrained stochastic linear-quadratic control. Automatica 49(8), 2435–2439 (2013)
Zymler, S., Kuhn, D., Rustem, B.: Distributionally robust joint chance constraints with second-order moment information. Math. Program. 137(1–2), 167–198 (2013)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research has been supported by the University of Brescia under the Project CLAFITE.
MATLAB code
MATLAB code
The following MATLAB code returns \({\underline{\epsilon }}(k)\) and \({\overline{\epsilon }}(k)\) for user assigned k, N, and \(\beta \).

Rights and permissions
About this article
Cite this article
Garatti, S., Campi, M.C. Risk and complexity in scenario optimization. Math. Program. 191, 243–279 (2022). https://doi.org/10.1007/s10107-019-01446-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-019-01446-4