The Kullback-Leibler divergence rate of two finite or countable ergodic Markov chains is well-known to exist and have an explicit expression as a function of the transition matrices of the chains, allowing access to classical tools for applications, such as minimization under constraints or projections on convex sets. The existence of Rényi divergence rates of ergodic Markov chains has been established in [5], [12]; here we establish explicit expressions for them and prove some properties of the resulting measures of discrepancy between stochastic matrices, opening the way to applications.