Abstract
We present an efficient method for solving optimal stopping problems with a probabilistic constraint. The goal is to optimize the expected cumulative cost, but constrained by an upper bound on the probability that the cost exceeds a specified threshold. This probabilistic constraint causes optimal policies to be time-dependent and randomized, however, we show that an optimal policy can always be selected with “piecewise-monotonic” time-dependence and “nearly-deterministic” randomization. We prove these properties using the Bellman optimality equations for a Lagrangian relaxation of the original problem. We present an algorithm that exploits these properties for computational efficiency. Its performance and the structure of optimal policies are illustrated on two numerical examples.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions. Applications of Mathematics. Springer, New York (1993)
Fan, Y., Nie, Y.: Optimal routing for maximizing the travel time reliability. Netw. Spat. Econ. 6(3), 333–344 (2006)
Browne, S.: Optimal investment policies for a firm with a random risk process: exponential utility and minimizing the probability of ruin. Math. Oper. Res. 20(4), 937–958 (1995). https://doi.org/10.1287/moor.20.4.937
Rockafellar, R.T., Uryasev, S.: Optimization of conditional value-at-risk. J. Risk 2, 21–41 (2000)
Ermon, S., Gomes, C., Selman, B., Vladimirsky, A.: Probabilistic planning with non-linear utility functions and worst-case guarantees. In: Proceedings of the 11th International AAMAS Conference, Vol. 2, pp. 965–972 (2012)
Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1999)
White, D.: Dynamic programming and probabilistic constraints. Oper. Res. 22(3), 654–664 (1974)
Pfeiffer, L.: Two approaches to stochastic optimal control problems with a final-time expectation constraint. Appl. Math. Optim. (2016)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, vol. II, 3rd edn. Athena Scientific, Belmont (2007)
Bertsekas, D.P., Tsitsiklis, J.N.: An analysis of stochastic shortest path problems. Math. Oper. Res. 16, 580–595 (1991)
Crandall, M.G.: Viscosity solutions: a primer. In: Viscosity Solutions and applications: lectures given at the 2nd session of the centro internazionale matematico estivo (C.I.M.E.) held in montecatiniterme, Italy, June 12–20, 1995, pp. 1–43. Springer, Berlin (1997)
Krylov, N.V., Balakrishnan, A.V.: Controlled diffusion processes/N. V. Krylov; translated by A. B. Aries. In: Balakrishnan A. V. (ed.) Springer, New York (1980)
Palmer, A.Z.: a C++ implementation of algorithms for PCOS problem (2017). https://github.com/AaronZPalmer/PCOS
Baxter, J., Chacon, R.: Compactness of stopping times. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete 40(3), 169–181 (1977)
Karoui, N.E., Peng, S., Quenez, M.C.: A dynamic maximum principle for the optimization of recursive utilities under constraints. Ann. Appl. Probab. 11, 664–693 (2001)
Acknowledgements
Aaron Zeff Palmer was supported in part as NSF GRFP Fellow 2011122749. Alexander Vladimirsky was supported in part by the NSF Grants DMS-1016150 and DMS-1738010. We would like to thank the associate editor and the reviewers for their carefully reading and suggestions that helped us greatly improve this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Kyriakos G. Vamvoudakis.
Rights and permissions
About this article
Cite this article
Palmer, A.Z., Vladimirsky, A. Optimal Stopping with a Probabilistic Constraint. J Optim Theory Appl 175, 795–817 (2017). https://doi.org/10.1007/s10957-017-1183-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-017-1183-3