Skip to main content
SpringerLink
Log in

Chance-constrained dynamic programming with application to risk-aware robotic space exploration

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Existing approaches to constrained dynamic programming are limited to formulations where the constraints share the same additive structure of the objective function (that is, they can be represented as an expectation of the summation of one-stage costs). As such, these formulations cannot handle joint probabilistic (chance) constraints, whose structure is not additive. To bridge this gap, this paper presents a novel algorithmic approach for joint chance-constrained dynamic programming problems, where the probability of failure to satisfy given state constraints is explicitly bounded. Our approach is to (conservatively) reformulate a joint chance constraint as a constraint on the expectation of a summation of indicator random variables, which can be incorporated into the cost function by considering a dual formulation of the optimization problem. As a result, the primal variables can be optimized by standard dynamic programming, while the dual variable is optimized by a root-finding algorithm that converges exponentially. Error bounds on the primal and dual objective values are rigorously derived. We demonstrate algorithm effectiveness on three optimal control problems, namely a path planning problem, a Mars entry, descent and landing problem, and a Lunar landing problem. All Mars simulations are conducted using real terrain data of Mars, with four million discrete states at each time step. The numerical experiments are used to validate our theoretical and heuristic arguments that the proposed algorithm is both (i) computationally efficient, i.e., capable of handling real-world problems, and (ii) near-optimal, i.e., its degree of conservatism is very low.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. The proposed algorithm is distinct from MPC in that, as a regular DP, it computes a control policy, instead of a sequence of control inputs, backward in time.

  2. Strictly speaking, the risk-to-go function \(r_0^{\lambda }(x_0)\) does not represent the probability of failure, but an upper bound on it, due the reformulation from Problem 1 to Problem 2.

References

  • Acikmese, B., & Ploen, S. (2007). Convex programming approach to powered descent guidance for mars landing. AIAA Journal of Guidance, Control, and Dynamics, 30(5), 1353–1366.

    Article  Google Scholar 

  • Acikmese, B., Carson, J. M., & Blackmore, L. (2013). Lossless convexification of nonconvex control bound and pointing constraints of the soft landing optimal control problem. IEEE Transactions on Control System Technology, 21, 2104–2113.

  • Altman, E. (1999). Constrained Markov decision processes. Stochastic modeling. Boca Raton: Chapman & Hall/CRC.

    Google Scholar 

  • Askew, A. J. (1974). Chance-constraind dynamic programming and the optimization of water resource systems. Water Resources Research, 10, 1099–1106.

    Article  Google Scholar 

  • Askew, A. J. (1974). Optimum reservoir operating policies and the imposition of a reliability constraint. Water Resources Research, 10, 51–56.

    Article  Google Scholar 

  • Atkinson, K. E. (1989). An introduction to numerical analysis (2nd ed.). New York: wiley.

    MATH  Google Scholar 

  • Bertsekas, D. P. (1999). Nonlinear programming. Belmont: Athena Scientific.

    MATH  Google Scholar 

  • Bertsekas, D. P. (1999). Nonlinear Programming (2nd ed.). Belmont: Athena Scientific.

    MATH  Google Scholar 

  • Beutler, F. J., & Ross, K. W. (1985). Optimal policies for controlled markov chains with a constraint. Journal of Mathematical Analysis and Applications, 112, 236–252.

    Article  MATH  MathSciNet  Google Scholar 

  • Biesiadecki, J. J., & Maimone, M. (1985). The mars exploration rover surface mobility flight software: Driving ambition. In IEEE Aerospace Conference Proceedings 2006.

  • Blackmore, L., Acikmese, B., & Scharf, D. P. (2010). Minimum-landing-error powered-descent guidance for mars landing using convex optimization. AIAA Journal of Guidance Control and Dynamics, 33, 1161–1171.

    Article  Google Scholar 

  • Blackmore, L., Li, H., & Williams, B. C. (2006). A probabilistic approach to optimal robust path planning with obstacles. In Proceedings of American Control Conference.

  • Blackmore, L., & Ono, M. (2009). Convex chance constrained predictive control without sampling. In Proceedings of the AIAA Guidance, Navigation and Control Conference.

  • Blackmore, L., Ono, M., Bektassov, A., & Williams, B. C. (2010). A probabilistic particle-control approximation of chance-constrained stochastic predictive control. IEEE Transactions on Robotics, 26(3), 502–517.

    Article  Google Scholar 

  • Charnes, A., & Cooper, W. W. (1959). Chance-constrained programming. Management Science, 6, 73–79.

    Article  MATH  MathSciNet  Google Scholar 

  • Chow, Y. L., & Pavone, M. (2014). A framework for time-consistent, risk-averse model predictive control: Theory and algorithms. In American Control Conference, (pp. 4204–4211).

  • Couchman, P., Cannon, M., & Kouvaritakis, B. (2006). Stochastic MPC with inequality stability constraints. Automatica, 42(12), 2169–2174.

    Article  MATH  MathSciNet  Google Scholar 

  • Estlin, T. A., Bornstein, B. J., Gaines, D. M., Anderson, R. C., Thompson, D. R., Burl, M., et al. (2012). Aegis: Automated science targeting for the mer opportunity rover. ACM Transactions on Intelligent Systems and Technology, 3(3), 50.

    Article  Google Scholar 

  • Fleming, W., & McEneaney, W. (1995). Risk-sensitive control on an infinite time horizon. SIAM Journal on Control and Optimization, 33(6), 1881–1915.

    Article  MATH  MathSciNet  Google Scholar 

  • Harris, M., & Akmee, B. (2014). Maximum divert for planetary landing using convex optimization. Journal of Optimization Theory and Applications, 162(3), 975–995. doi:10.1007/s10957-013-0501-7.

    Article  MATH  MathSciNet  Google Scholar 

  • Hokayem, P., Chatterjee, D., & Lygeros, J. (2013). Chance-constrained LQG with bounded contr ol policies. In IEEE Conference on Decision and Control (pp. 2471–2476). Italy: Florence.

  • Johnson, A., Huertas, A., Werner, R., & Montgomery, J. (2008). Analysis of On-Board Hazard Detection and Avoidance for Safe Lunar Landing. In IEEE Aerospace Conference.

  • Knocke, P. C., Wawrzyniak, G. G., Kennedy, B. M., Desai, P. N., Parker, T. J., Golombek, M. P., Duxbury, T. C., & Kass, D. M. (2004) Mars exploration rovers landing dispersion analysis. In Proceedings of AIAA/AAS Astrodynamics Specialist Conferencec and Exhibit.

  • Kuwata, Y., & Balaram, J. B. (2011). Combined EDL-mobility planning for planetary missions. In Proceedings of Infotech@Aerospace.

  • Kuwata, Y., Pavone, M., & Balaram, J. B. (2012). A risk-constrained multi-stage decision making approach to the architectural analysis of planetary missions. In Proceedings of the IEEE Conference on Decision and Control.

  • Li, P., Wendt, M., & Wozny, G. (2002). A probabilistically constrained model predictive controller. Automatica, 38, 1171–1176.

    Article  MATH  MathSciNet  Google Scholar 

  • McGhan, C. L. R., Serra, R., Ingham, M. D., Ono, M., Estlin, T., Murray, R. M., & Williams, B. C. (2015). A risk-aware architecture for resilient spacecraft operations. In Accepted for publication in the Proceedings of IEEE aerospace conference.

  • Murray, R. M., Day, J. C., Ingham, M. D., Reder, L. J., & Williams, B. C. (2013). Engineering resilient space systems: Final report.

  • National Aeronautics and Space Administration: Gravity Recovery and Interior Laboratory (GRAIL) Launch Press Kit.

  • National Research Council of the National Academies: Vision and voyages for planetary science in the decade 2013-2022 (2010).

  • Nemirovski, A., & Shapiro, A. (2006). Convex approximations of chance constrained programs. SIAM Journal on Optimization, 17, 969–996.

    Article  MATH  MathSciNet  Google Scholar 

  • Oldewurtel, F., Jones, C. N., & Morari, M. (2008). A tractable approximation of chance constrained stochastic MPC based on affine disturbance feedback. In Proceedings of Conference on Decision and Control.

  • Oldewurtel, F., Ulbig, A., Parisio, A., Andersson, G., & Morari, M. (2010). Reducing peak electricity demand in building climate control using real-time pricing and model predictive control. In Proceedings of Conference on Decision and Control.

  • Ono, M. (2012). Joint chance-constrained model predictive control with probabilistic resolvability. In Proceedings of American Control Conference.

  • Ono, M. (2012). Robust, goal-directed plan execution with bounded risk. Ph.D. thesis, Massachusetts Institute of Technology.

  • Ono, M., Topcu, U., Yo, M., & Adachi, S. (2013). Risk-limiting power grid control with an arma-based prediction model. In IEEE 52nd Annual Conference on Decision and Control (CDC), 2013, (pp. 4949–4956).

  • Ono, M., & Williams, B. C. (2008). Iterative risk allocation: A new approach to robust model predictive control with a joint chance constraint. In Proceedings of 47th IEEE Conference on Decision and Control.

  • Parker, J. S., & Anderson, R. L. (2014). Low-energy lunar trajectory design. New York: Wiley.

    Book  Google Scholar 

  • Prakash, R., Burkhart, P., Chen, A., Comeaux, K., Guernsey, C., Kipp, D., Lorenzoni, L., Mendeck, G., Powell, R., Rivellini, T., San Martin, A., Sell, S., Steltzner, A., & Way, D. (2008). Mars science laboratory entry, descent, and landing system overview. In IEEE Conference on Aerospace (pp. 1–18).

  • Rossman, L. A. (1977). Reliability-constrained dynamic programing and randomized release rules in reservoir management. Water Resources Research, 13, 247–255.

    Article  Google Scholar 

  • Schildbach, G., Goulart, P., & Morari, M. (2013). The linear quadratic regulator with chance constraints. In European Control Conference (ECC) (pp. 2746–2751).

  • Schwarm, A. T., & Nikolaou, M. (1999). Chance-constrained model predictive control. AIChE Journal, 45(8), 1743–1752.

    Article  Google Scholar 

  • Sjödin, A. E., Gayme, D. F., & Topcu, U. (2012). Risk-mitigated optimal power flow with high wind penetration. In Proceedings of American Control Conference (2012).

  • Summers, T., Warrington, J., Morari, M., & Lygeros, J. (2014). Stochastic optimal power flow based on convex approximations of chance constraints. In Power Systems Computation Conference. Wroclaw, Poland.

  • van Hessem, D. H. (2004). Stochastic inequality constrained closed-loop model predictive control with application to chemical process operation. Ph.D. thesis, Delft University of Technology.

  • Vitus, M. P., & Tomlin, C. J. (2011). On feedback design and risk allocation in chance constrained control. In Proceedings of IEEE Conference on Decision and Control (pp. 734–739).

  • Williams, J. L., Fisher, J. W., & Willsky, A. S. (2007). Approximate dynamic programming for communication-constrained sensor network management. IEEE Transactions on Signal Processing, 55, 4300–4311.

    Article  MathSciNet  Google Scholar 

  • Yo, M., Ono, M., Williams, B., & Adachi, S. (2013). Risk-limiting, market-based power dispatch and pricing. In European Control Conference (pp. 3038–3045).

  • Zhang, X., Grammatico, S., Schildbach, G., Goulart, P., & Lygeros, J. (2014). On the sample size of randomized MPC for chance-constrained systems with application to building climate control. In European Control Conference (pp. 478–483). Strasbourg, France.

Download references

Acknowledgments

The research described in this paper was conducted at the Jet Propulsion Laboratory, California Institute of Technology, under a contract with the National Aeronautics and Space Administration. Marco Pavone was supported in part by NASA under the Space Technology Research Grants Program, Grant NNX12AQ43G.

Author information

Authors and Affiliations

  1. Jet Propulsion Laboratory, California Institute of Technology, Pasadena, USA

    Masahiro Ono & J. Balaram

  2. Department of Aeronautics and Astronautics, Stanford University, Stanford, USA

    Marco Pavone

  3. Guidance, Control, and Navigation Division, Space Exploration Technologies Corporation (SpaceX), Hawthorne, USA

    Yoshiaki Kuwata

Authors
  1. Masahiro Ono
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marco Pavone
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yoshiaki Kuwata
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. J. Balaram
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Masahiro Ono.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ono, M., Pavone, M., Kuwata, Y. et al. Chance-constrained dynamic programming with application to risk-aware robotic space exploration. Auton Robot 39, 555–571 (2015). https://doi.org/10.1007/s10514-015-9467-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-015-9467-7

Keywords

Search

Navigation