Regularized Least Squares Temporal Difference Learning with Nested ℓ₂ and ℓ₁ Penalization

Abstract

The construction of a suitable set of features to approximate value functions is a central problem in reinforcement learning (RL). A popular approach to this problem is to use high-dimensional feature spaces together with least-squares temporal difference learning (LSTD). Although this combination allows for very accurate approximations, it often exhibits poor prediction performance because of overfitting when the number of samples is small compared to the number of features in the approximation space. In the linear regression setting, regularization is commonly used to overcome this problem. In this paper, we review some regularized approaches to policy evaluation and we introduce a novel scheme (L ₂₁) which uses ℓ₂ regularization in the projection operator and an ℓ₁ penalty in the fixed-point step. We show that such formulation reduces to a standard Lasso problem. As a result, any off-the-shelf solver can be used to compute its solution and standardization techniques can be applied to the data. We report experimental results showing that L ₂₁ is effective in avoiding overfitting and that it compares favorably to existing ℓ₁ regularized methods.