Skip to main content
SpringerLink
Log in

Fitting feature-dependent Markov chains

  • Published:
Journal of Global Optimization Aims and scope Submit manuscript

Abstract

We describe a method for fitting a Markov chain, with a state transition matrix that depends on a feature vector, to data that can include missing values. Our model consists of separate logistic regressions for each row of the transition matrix. We fit the parameters in the model by maximizing the log-likelihood of the data minus a regularizer. When there are missing values, the log-likelihood becomes intractable, and we resort to the expectation-maximization (EM) heuristic. We illustrate the method on several examples, and describe our efficient Python open-source implementation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alarid-Escudero, F., Krijkamp, E., Enns, E., Yang, A., Hunink, M., Pechlivanoglou, P., Jalal, H.: Cohort state-transition models in R: A tutorial. arXiv preprint arXiv:2001.07824, (2020)

  2. Allison, P.: Missing Data. Sage Publications, New York (2001)

    MATH  Google Scholar 

  3. Barratt, S., Dong, Y., Boyd, S.: Low rank forecasting. arXiv preprint arXiv:2101.12414, (2021)

  4. Baum, L.: An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes. Inequalities 3(1), 1–8 (1972)

    MathSciNet  Google Scholar 

  5. Beck, R., Pauker, S.: The Markov process in medical prognosis. Med. Decis. Making 3(4), 419–458 (1983)

    Article  Google Scholar 

  6. Bellman, R.: Dynamic programming. Science 153(3731), 34–37 (1966)

    Article  MATH  Google Scholar 

  7. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  8. Boyle, B.: Estimation of feature-dependent Markov process transition probability matrices. Inf. Control 32(4), 379–384 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  9. Cox, D.: Regression models and life-tables. J. Roy. Stat. Soc.: Ser. B (Methodol). 34(2), 187–202 (1972)

    MathSciNet  MATH  Google Scholar 

  10. Deltour, I., Richardson, S., Le Hesran, J.-Y.: Stochastic algorithms for Markov models estimation with intermittent missing data. Biometrics 55(2), 565–573 (1999)

    Article  MATH  Google Scholar 

  11. Dempster, A., Laird, N., Rubin, D.: Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Stat. Soc.: Ser. B (Methodol). 39(1), 1–22 (1977)

    MathSciNet  MATH  Google Scholar 

  12. Hastie, T., Tibshirani, R., Friedman, J.: The Elements Of Statistical Learning: Data Mining, Inference, And Prediction. Springer Science & Business Media, Germany (2009)

    Book  MATH  Google Scholar 

  13. Incerti, D., Jansen, J.: hesim: Health Economic Simulation Modeling and Decision Analysis, (2021). R package version 0.5.0

  14. Kalbfleisch, J., Lawless, J.: The analysis of panel data under a Markov assumption. J. Am. Stat. Assoc. 80(392), 863–871 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  15. Kemeny, J., Snell, L.: Markov Chains, vol. 6. Springer, New York (1976)

    MATH  Google Scholar 

  16. Korn, E., Whittemore, A.: Methods for analyzing panel studies of acute health effects of air pollution. Biometrics, 795–802, (1979)

  17. Lane, W., Looney, S., Wansley, J.: An application of the Cox proportional hazards model to bank failure. Journal of Banking & Finance 10(4), 511–531 (1986)

    Article  Google Scholar 

  18. Makis, V., Jardine, A.: Optimal replacement in the proportional hazards model. INFOR: Inf. Sys. Oper. Res. 30(1), 172–183 (1992)

    MATH  Google Scholar 

  19. Norris, J.: Markov Chains. Cambridge University Press, Cambridge (1998)

    MATH  Google Scholar 

  20. Page, L., Brin, S., Motwani, R., Winograd, T.: The Pagerank Citation Ranking: Bringing Order To The Web. Technical report, Stanford InfoLab, Netherlands (1999)

    Google Scholar 

  21. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pages 8024–8035, (2019)

  22. Rabiner, L., Juang, B.: An introduction to hidden Markov models. IEEE ASSP Mag. 3(1), 4–16 (1986)

    Article  Google Scholar 

  23. Recht, B., Fazel, M., Parrilo, P.: Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Rev. 52(3), 471–501 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  24. Inc. Retrosheet. Retrosheet. https://retrosheet.org/

  25. Revuz, D.: Markov Chains. Elsevier, Amsterdam (2008)

    MATH  Google Scholar 

  26. Sherlaw-Johnson, C., Gallivan, S., Burridge, J.: Estimating a Markov transition matrix from observational data. J. Oper. Res. Soc. 46(3), 405–410 (1995)

    Article  MATH  Google Scholar 

  27. Sonnenberg, F., Beck, R.: Markov models in medical decision making: a practical guide. Med. Decis. Making 13(4), 322–338 (1993)

    Article  Google Scholar 

  28. Walrand, J.: Probability In Electrical Engineering And Computer Science: An Application-driven Course. Quorum Books, Santa Barbara, California (2014)

    Google Scholar 

  29. Woo, G.: Quantitative terrorism risk assessment. The Journal of Risk Finance, (2002)

  30. Wu, Jeff: On the convergence properties of the EM algorithm. The Annals of Statistics, pages 95–103, (1983)

  31. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. J. R. Stat. Soc.: Ser. B (Statistical Methodology) 68(1), 49–67 (2006)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors gratefully acknowledge conversations and discussions about some of the material in this paper with Trevor Hastie, Emmanuel Candes, Scott Harris, and Paul Bien.

Author information

Authors and Affiliations

  1. Stanford University, Stanford, United States

    Shane Barratt & Stephen Boyd

Authors
  1. Shane Barratt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen Boyd
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen Boyd.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barratt, S., Boyd, S. Fitting feature-dependent Markov chains. J Glob Optim 87, 329–346 (2023). https://doi.org/10.1007/s10898-022-01198-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10898-022-01198-0

Keywords

Search

Navigation