Skip to main content
Springer Nature Link
Log in

On the Distributional Convergence of Temporal Difference Learning

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases: Research Track (ECML PKDD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14172))

  • 1365 Accesses

Abstract

Temporal Difference (TD) learning is one of the most simple but efficient algorithms for policy evaluation in reinforcement learning. Although the finite-time convergence results of the TD algorithm are abundant now, the distributional convergence has still been blank. This paper shows that TD with constant step size simulates Markov chains converging to some stationary distribution under both i.i.d. and Markov chain observation models. We prove that TD enjoys the geometric distributional convergence rate and show how the step size affects the expectation and covariance of the stationary distribution. All assumptions used in our paper are mild and common in the TD community. Our proved results indicate a tradeoff between the convergence speed and accuracy for TD. Based on our theoretical findings, we explain why the Jacobi preconditioner can accelerate the TD algorithms.

J. Dai–Independent Researcher.

This research is supported by the National Natural Science Foundation of China under the grant (12002382).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We call it semi-gradient because the \(\textbf{g}^t\) does not follow the stochastic gradient direction of any fixed objective.

  2. 2.

    The i.i.d. observation model happens if the initial state \(s_0\) is drawn from the stationary distribution.

  3. 3.

    In practice, \(\textbf{J}\) is approximated by a Monte Carlo method.

  4. 4.

    The random walk gradient descent can be regarded as a special MCGD because the random walk is a Markov chain process.

References

  1. Baird, L.: Residual algorithms: reinforcement learning with function approximation. In: Machine Learning, pp. 30–37 (1995)

    Google Scholar 

  2. Bertsekas, D.P.: A new class of incremental gradient methods for least squares problems. SIAM J. Optim. 7(4), 913–926 (1997)

    Google Scholar 

  3. Bhandari, J., Russo, D., Singal, R.: A finite time analysis of temporal difference learning with linear function approximation. In: Conference on learning theory (2018)

    Google Scholar 

  4. Borkar, V.S.: Stochastic approximation: a dynamical systems viewpoint, vol. 48. Springer (2009)

    Google Scholar 

  5. Brosse, N., Durmus, A., Moulines, E.: The promises and pitfalls of stochastic gradient langevin dynamics. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  6. Can, B., Gurbuzbalaban, M., Zhu, L.: Accelerated linear convergence of stochastic momentum methods in Wasserstein distances. In: Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 891–901. PMLR, 09–15 Jun 2019

    Google Scholar 

  7. Dalal, G., Szörényi, B., Thoppe, G., Mannor, S.: Finite sample analyses for td(0) with function approximation. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  8. Dieuleveut, A., Durmus, A., Bach, F.: Bridging the gap between constant step size stochastic gradient descent and markov chains. Ann. Stat. 48(3), 1348–1382 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  9. Duchi, J.C., Agarwal, A., Johansson, M., Jordan, M.I.: Ergodic mirror descent. SIAM J. Optim. 22(4), 1549–1578 (2012)

    Google Scholar 

  10. Gitman, I., Lang, H., Zhang, P., Xiao, L.: Understanding the role of momentum in stochastic gradient methods. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  11. Gupta, A., Haskell, W.B.: Convergence of recursive stochastic algorithms using wasserstein divergence. SIAM J. Math. Data Sci. 3(4), 1141–1167 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  12. Gupta, A., Chen, H., Pi, J., Tendolkar, G.: Some limit properties of markov chains induced by recursive stochastic algorithms. SIAM J. Math. Data Sci. 2(4), 967–1003 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  13. Hu, B., Syed, U.: Characterizing the exact behaviors of temporal difference learning algorithms using markov jump linear system theory. In: Advances in Neural Information Processing Systems, pp. 8477–8488, Vancouver, Canada, December 2019

    Google Scholar 

  14. Johansson, B., Rabi, M., Johansson, M.: A randomized incremental subgradient method for distributed optimization in networked systems. SIAM J. Optim. 20(3), 1157–1170 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  15. Lakshminarayanan, C., Szepesvari, C.: Linear stochastic approximation: how far does constant step-size and iterate averaging go? In: International Conference on Artificial Intelligence and Statistics, pp. 1347–1355 (2018)

    Google Scholar 

  16. Lee, D., He, N.: Target-based temporal-difference learning. In: International Conference on Machine Learning, pp. 3713–3722. PMLR (2019)

    Google Scholar 

  17. Mandt, S., Hoffman, M.D., Blei, D.M.: Stochastic gradient descent as approximate bayesian inference. J. Mach. Learn. Res. 18, 1–35 (2017)

    MathSciNet  MATH  Google Scholar 

  18. Meyn, S.P.: Markov Chains and Stochastic Stability. Markov Chains and Stochastic Stability (1999)

    Google Scholar 

  19. Nlar, E.: Probability and stochastics. Probability and Stochastics (2011). https://doi.org/10.1007/978-0-387-87859-1

  20. Ram, S.S., Nedić, A., Veeravalli, V.V.: Incremental stochastic subgradient algorithms for convex optimization. SIAM J. Optim. 20(2), 691–717 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  21. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 400–407 (1951)

    Google Scholar 

  22. Romoff, J., et al.: Tdprop: does adaptive optimization with jacobi preconditioning help temporal difference learning? In: Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1082–1090 (2021)

    Google Scholar 

  23. Simoncini, V.: Computational methods for linear matrix equations. SIAM Rev. 58(3), 377–441 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  24. Srikant, R., Ying, L.: Finite-time error bounds for linear stochastic approximation and TD learning. In: COLT (2019)

    Google Scholar 

  25. Sun, T., Li, D., Wang, B.: Adaptive random walk gradient descent for decentralized optimization. In: International Conference on Machine Learning, pp. 20790–20809. PMLR (2022)

    Google Scholar 

  26. Sun, T., Shen, H., Chen, T., Li, D.: Adaptive temporal difference learning with linear function approximation. IEEE Trans. Pattern Anal. Mach. Intell. 44(12), 8812–8824 (2021)

    Google Scholar 

  27. Sun, T., Sun, Y., Yin, W.: On markov chain gradient descent. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  28. Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3(1), 9–44 (1988)

    Article  Google Scholar 

  29. Sutton, R.S., Barto, A.G., et al.: Introduction to reinforcement learning, vol. 2. MIT Press, Cambridge (1998)

    Google Scholar 

  30. Tsitsiklis, J.N., Roy, B.V.: An analysis of temporal-difference learning with function approximation. IEEE Trans. Autom. Control (1997)

    Google Scholar 

  31. Villani, C.: Optimal Transport: Old and New, vol. 338. Springer, Berlin (2009). https://doi.org/10.1007/978-3-540-71050-9

  32. Xiong, H., Tengyu, X., Liang, Y., Zhang, W.: Non-asymptotic convergence of adam-type reinforcement learning algorithms under markovian sampling. Proc. AAAI Conf. Artif. Intell. 35, 10460–10468 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Changsha, China

    Jie Dai

  2. College of Computer, National University of Defense Technology, Changsha, China

    Xuguang Chen

Authors
  1. Jie Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuguang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuguang Chen .

Editor information

Editors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Danai Koutra

  2. University of Vienna, Vienna, Austria

    Claudia Plant

  3. Max Planck Institute for Software Systems, Kaiserslautern, Germany

    Manuel Gomez Rodriguez

  4. Politecnico di Torino, Turin, Italy

    Elena Baralis

  5. CENTAI, Turin, Italy

    Francesco Bonchi

Ethics declarations

Ethical Statement

Our paper is devoted to the theoretical aspect of general stochastic algorithm, which does not present any foreseeable societal consequence.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dai, J., Chen, X. (2023). On the Distributional Convergence of Temporal Difference Learning. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14172. Springer, Cham. https://doi.org/10.1007/978-3-031-43421-1_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43421-1_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43420-4

  • Online ISBN: 978-3-031-43421-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships