Skip to main content

Advertisement

SpringerLink
Log in

Control-limit policies for a class of stopping time problems with termination restrictions

  • Original Paper
  • Published:
Optimization Letters Aims and scope Submit manuscript

Abstract

We consider a class of stopping time problems in which the state of the process evolves according to a discrete-time Markov chain and the action which terminates the process is available only in a certain subset of the state space. For the objective of maximizing total expected discounted reward we develop an infinite-horizon Markov decision process formulation and derive two families of sufficient conditions under which the optimal policy exhibits a control-limit structure. We present numerical examples to illustrate the two families of conditions we derive are satisfiable and do not overrule each other. We also demonstrate that a family of our conditions is weaker than those in the literature which guarantee the optimality of control-limit policies for a special case of the problem we study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Abbad, M., Filar, J.A., Bielecki, T.R.: Algorithms for singularly perturbed limiting average Markov control problems. IEEE Trans. Autom. Control 37(9), 1421–1425 (1992)

    Article  MATH  MathSciNet  Google Scholar 

  2. Aetna, Inc (2014) Clinical policy bulletin: liver transplantation. http://www.aetna.com/cpb/medical/data/500_599/0596.html. Accessed 1 June 2014

  3. Alagoz, O., Maillart, L.M., Schaefer, A.J., Roberts, M.S.: The optimal timing of living-donor liver transplantation. Manag. Sci. 50(10), 1420–1430 (2004)

    Article  MATH  Google Scholar 

  4. Barlow, R.E., Proschan, F.: Mathematical Theory of Reliability. Society for Industrial and Applied Mathematics, Philadelphia (1996)

  5. Bergman, B.: Optimal replacement under a general failure model. Adv. Appl. Prob. 10(2), 431–451 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  6. Bernardi, M., Gitto, S., Biselli, M.: The MELD score in patients awaiting liver transplant: strengths and weaknesses. J. Hepatol. 54(6), 1297–1306 (2011)

    Article  Google Scholar 

  7. Buchholz, P., Schulz, I.: Numerical analysis of continuous time Markov decision processes over finite horizons. Comput. Oper. Res. 38(3), 651–659 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  8. David, I., Yechiali, U.: A time-dependent stopping problem with application to live organ transplants. Oper. Res. 33(3), 491–504 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  9. Derman, C., Sacks, J.: Replacement of periodically inspected equipment (an optimal optional stopping rule). Naval Res. Logist. 7(4), 597–607 (1960)

    Article  MATH  MathSciNet  Google Scholar 

  10. Eldor, R., Raz, I.: American Diabetes Association indications for statins in diabetes. Diabetes Care 32(Suppl 2), S384–S391 (2009)

    Article  Google Scholar 

  11. Federgruen, A., Zipkin, P.: An efficient algorithm for computing optimal (s, S) policies. Oper. Res. 32(6), 1268–1285 (1984)

    Article  MATH  MathSciNet  Google Scholar 

  12. Feng, Y., Gallego, G.: Optimal starting times for end-of-season sales and optimal stopping times for promotional fares. Manag. Sci. 41(8), 1371–1391 (1995)

    Article  MATH  Google Scholar 

  13. Icten, Z.G., Shechter, S.M., Maillart, L.M., Nagarajan, M.: Optimal management of a limited number of replacements under Markovian deterioration. IIE Trans. 45(2), 206–214 (2013)

    Article  Google Scholar 

  14. Jia, Q.S.: On state aggregation to approximate complex value functions in large-scale Markov decision processes. IEEE Trans. Autom. Control 56(2), 333–344 (2011)

    Article  Google Scholar 

  15. Kurt, M., Denton, B.T., Schaefer, A.J., Shah, N.D., Smith, S.A.: The structure of optimal statin initiation policies for patients with Type 2 diabetes. IIE Trans. Healthcare Syst. Eng. 1(1), 49–65 (2011)

    Article  Google Scholar 

  16. Miller, B.L.: Finite state continuous time Markov decision processes with a finite planning horizon. SIAM J. Control 6(2), 266–280 (1968)

    Article  MATH  MathSciNet  Google Scholar 

  17. Mordue, A., Parkin, D.W., Baxter, C., Fawcett, G., Stewart, M.: Thresholds for treatment in cataract surgery. J. Publ. Health 16(4), 393–398 (1994)

    Google Scholar 

  18. National Institutes of Health (2014) Guidelines for the use of antiretroviral agents in HIV-1-infected adults and adolescents. http://aidsinfo.nih.gov. Accessed 5 June 2014

  19. Özekici, S., Parlar, M.: Inventory models with unreliable suppliers in a random environment. Ann. Oper. Res. 91, 123–136 (1999)

    Article  MATH  MathSciNet  Google Scholar 

  20. Pham, H.: Optimal stopping, free boundary, and American option in a jump-diffusion model. Appl. Math. Optim. 35(2), 145–164 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  21. Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality. Wiley, Hoboken (2007)

  22. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2009)

  23. Shechter, S.M., Bailey, M.D., Schaefer, A.J., Roberts, M.S.: The optimal time to initiate HIV therapy under ordered health states. Oper. Res. 56(1), 20–33 (2008)

    Article  MATH  Google Scholar 

  24. Stone, L.D.: Theory Optimal Search. Elsevier, Amsterdam (1975)

    MATH  Google Scholar 

  25. Weiss, H.J.: The computation of optimal control limits for a queue with batch services. Manag. Sci. 25(4), 320–328 (1979)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

The authors thank two anonymous reviewers for their comments which improved an earlier version of the manuscript.

Author information

Authors and Affiliations

  1. Department of Industrial and Systems Engineering, School of Engineering and Applied Sciences, The State University of New York at Buffalo, 342 Bell Hall, North Campus, Buffalo, NY, 14260, USA

    Yan Xia & Murat Kurt

Authors
  1. Yan Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Murat Kurt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Murat Kurt.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, Y., Kurt, M. Control-limit policies for a class of stopping time problems with termination restrictions. Optim Lett 9, 1315–1329 (2015). https://doi.org/10.1007/s11590-014-0843-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11590-014-0843-6

Keywords

Search

Navigation