Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Numerical Methods for Continuous-Time Stochastic Control Problems

  • Living reference work entry
  • First Online:
Encyclopedia of Systems and Control

  • 661 Accesses

Introduction

This expository article provides a brief review of numerical methods for stochastic control in continuous time. Leaving most of the technical details out with the broad general audience in mind, it aims to serve as an introductory reference for researchers, practitioners, and students, who wish to know something about numerical methods for stochastic controls.

The study of stochastic control has witnessed tremendous progress in the last few decades; see, for example, Fleming and Rishel (1975), Fleming and Soner (1992), Kushner (1977), and Yong and Zhou (1999) among others, for fundamentals of stochastic controls as well as historical remarks. Much of the development has been accompanied by the needs and progress in science, engineering, as well as finance. Typically, the problems are highly nonlinear, so a closed-form solution is very difficult to obtain. As a result, designing feasible numerical algorithms becomes vitally important. Among the many approximation methods,...

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Bibliography

  • Barles G, Souganidis P (1991) Convergence of approximation schemes for fully nonlinear second order equations. J Asymptot Anal 4:271–283

    MATH  MathSciNet  Google Scholar 

  • Bertsekas DP, Castanon DA (1989) Adaptive aggregation methods for infinite horizon dynamic programming. IEEE Trans Autom Control 34:589–598

    Article  MATH  MathSciNet  Google Scholar 

  • Chancelier P, Gomez C, Quadrat J-P, Sulem A, Blankenship GL, La Vigna A, MaCenary DC, Yan I (1986) An expert system for control and signal processing with automatic FORTRAN program generation. In: Mathematical systems symposium, Stockholm. Royal Institute of Technology, Stockholm

    Google Scholar 

  • Chancelier P, Gomez C, Quadrat J-P, Sulem A (1987) Automatic study in stochastic control. In: Fleming W, Lions PL (eds) IMA volume in mathematics and its applications, vol 10. Springer, Berlin

    Google Scholar 

  • Crandall MG, Lions PL (1983) Viscosity solutions of Hamilton-Jacobi equations. Trans Am Math Soc 277:1–42

    Article  MATH  MathSciNet  Google Scholar 

  • Crandall MG, Ishii H, Lions PL (1992) User’s guide to viscosity solutions of second order partial differential equations. Bull Am Math Soc 27:1–67

    Article  MATH  MathSciNet  Google Scholar 

  • Fleming WH, Rishel RW (1975) Deterministic and stochastic optimal control. Springer, New York

    Book  MATH  Google Scholar 

  • Fleming WH, Soner HM (1992) Controlled Markov processes and viscosity solutions. Springer, New York

    Google Scholar 

  • Jin Z, Wang Y, Yin G (2011) Numerical solutions of quantile hedging for guaranteed minimum death benefits under a regime-switching-jump-diffusion formulation. J Comput Appl Math 235:2842–2860

    Article  MATH  MathSciNet  Google Scholar 

  • Jin Z, Yin G, Zhu C (2012) Numerical solutions of optimal risk control and dividend optimization policies under a generalized singular control formulation. Automatica 48:1489–1501

    Article  MATH  MathSciNet  Google Scholar 

  • Jin Z, Yang HL, Yin G (2013) Numerical methods for optimal dividend payment and investment strategies of regime-switching jump diffusion models with capital injections. Automatica 49:2317–2329

    Article  MathSciNet  Google Scholar 

  • Krylov VN (2000) On the rate of convergence of finite-difference approximations for Bellman’s equations with variable coefficients. Probab Theory Relat Fields 117:1–16

    Article  MATH  Google Scholar 

  • Kushner HJ (1977) Probability methods for approximation in stochastic control and for elliptic equations. Academic, New York

    Google Scholar 

  • Kushner HJ (1990a) Weak convergence methods and singularly perturbed stochastic control and filtering problems. Birkhäuser, Boston

    Book  MATH  Google Scholar 

  • Kushner HJ (1990b) Numiercal methods for stochastic control problems in continuous time. SIAM J Control Optim 28:999–1048

    Article  MATH  MathSciNet  Google Scholar 

  • Kushner HJ (2000) Consistency issues for numerical methods for variance control with applications to optimization in finance. IEEE Trans Autom Control 44:2283–2296

    Article  MathSciNet  Google Scholar 

  • Kushner HJ (2002) Numerical approximations for stochastic differential games. SIAM J Control Optim 40:457–486

    Article  MathSciNet  Google Scholar 

  • Kushner HJ (2008) Numerical methods for controlled stochastic delay systems. Birkhäuser, Boston

    Book  MATH  Google Scholar 

  • Kushner HJ, Dupuis PG (1992) Numerical methods for stochastic control problems in continuous time. Springer, New York

    Book  MATH  Google Scholar 

  • Kushner HJ, Kleinman AJ (1968) Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory. IEEE Trans Autom Control AC-13:344–353

    Article  MathSciNet  Google Scholar 

  • Kushner HJ, Yin G (2003) Stochastic approximation and recursive algorithms and applications, 2nd edn. Springer, New York

    MATH  Google Scholar 

  • Menaldi J (1989) Some estimates for finite difference approximations. SIAM J Control Optim 27:579–607

    Article  MATH  MathSciNet  Google Scholar 

  • Peng S (1990) A general stochastic maximum principle for optimal control problems. SIAM J Control Optim 28:966–979

    Article  MATH  MathSciNet  Google Scholar 

  • Sethi SP, Zhang Q (1994) Hierarchical decision making in stochastic manufacturing systems. Birkhäuser, Boston

    Book  MATH  Google Scholar 

  • Song QS, Yin G (2009) Rates of convergence of numerical methods for controlled regime-switching diffusions with stopping times in the costs. SIAM J Control Optim 48:1831–1857

    Article  MATH  MathSciNet  Google Scholar 

  • Song QS, Yin G, Zhang Z (2006) Numerical method for controlled regime-switching diffusions and regime-switching jump diffusions. Automatica 42:1147–1157

    Article  MATH  MathSciNet  Google Scholar 

  • Song QS, Yin G, Zhang Z (2008) Numerical solutions for stochastic differential games with regime switching. IEEE Trans Autom Control 53:509–521

    Article  MathSciNet  Google Scholar 

  • Warga J (1962) Relaxed variational problems. J Math Anal Appl 4:111–128

    Article  MATH  MathSciNet  Google Scholar 

  • Yan HM, Yin G, Lou SXC (1994) Using stochastic optimization to determine threshold values for control of unreliable manufacturing systems. J Optim Theory Appl 83:511–539

    Article  MATH  MathSciNet  Google Scholar 

  • Yin G, Zhang Q (2013) Continuous-time Markov chains and applications: a two-time-scale approach, 2nd edn. Springer, New York

    Book  Google Scholar 

  • Yin G, Zhu C (2010) Hybrid switching diffusions: properties and applications. Springer, New York

    Book  Google Scholar 

  • Yin G, Liu RH, Zhang Q (2002) Recursive algorithms for stock liquidation: a stochastic optimization approach. SIAM J Optim 13:240–263

    Article  MATH  MathSciNet  Google Scholar 

  • Yin G, Jin H, Jin Z (2009) Numerical methods for portfolio selection with bounded constraints. J Comput Appl Math 233:564–581

    Article  MATH  MathSciNet  Google Scholar 

  • Yong J, Zhou XY (1999) Stochastic controls: Hamiltonian systems and HJB equations. Springer, New York

    Book  MATH  Google Scholar 

  • Zhang J (2006) Rate of convergence of finite difference approximations for degenerate ODEs. Math Comput 75:1755–1778

    Article  MATH  Google Scholar 

Download references

Acknowledgements

Research of this author was supported in part by the Army Research Office under grant W911NF-12-1-0223.

Author information

Authors and Affiliations

  1. Department of Mathematics, Wayne State University, Detroit, MI, USA

    George Yin

Authors
  1. George Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George Yin .

Editor information

Editors and Affiliations

  1. Electrical and Computer Engineering, Boston University, Boston, Massachusetts, USA

    John Baillieul

  2. Automation and Control Solutions, Honeywell, Golden Valley, Minnesota, USA

    Tariq Samad

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag London

About this entry

Cite this entry

Yin, G. (2014). Numerical Methods for Continuous-Time Stochastic Control Problems. In: Baillieul, J., Samad, T. (eds) Encyclopedia of Systems and Control. Springer, London. https://doi.org/10.1007/978-1-4471-5102-9_232-1

Download citation

  • DOI: https://doi.org/10.1007/978-1-4471-5102-9_232-1

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, London

  • Online ISBN: 978-1-4471-5102-9

  • eBook Packages: Springer Reference EngineeringReference Module Computer Science and Engineering

Publish with us

Policies and ethics

Search

Navigation