Abstract
Reinforcement learning algorithms require an extensive number of samples to perform a specific task. To achieve the same performance on a new task, the agent must learn from scratch. Transfer reinforcement learning is an emerging solution that aims to improve sample efficiency by reusing previously learnt knowledge in new tasks. Successor feature is a technique aiming to reuse representations to leverage that knowledge in unseen tasks. Successor feature has achieved outstanding results on the assumption that the transition dynamics must remain across tasks. Initial Successor feature approach omits settings with different environment dynamics, common among real-life tasks in reinforcement learning problems. Our approach transformed successor feature projects a set of diverse dynamics into a common dynamic distribution. Hence, it is an initial solution to relax the restriction of transference across fixed environment dynamics. Experimental results indicate that the transformed successor feature improves the transfer of knowledge in environments with fixed and diverse dynamics under the control of a simulated robotic arm, a robotic leg, and the cartpole environment.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdolshah, M., Le, H., George, T.K., Gupta, S., Rana, S., Venkatesh, S.: A new representation of successor features for transfer across dissimilar environments. In: International Conference on Machine Learning (ICML), vol. 139, pp. 1–9 (2021)
Abel, D., Arumugam, D., Lehnert, L., Littman, M.: State abstractions for lifelong reinforcement learning. In: Proceedings of the 35th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 80 (2018)
Allen, C., Parikh, N., Gottesman, O., Konidaris, G.: Learning Markov state abstractions for deep reinforcement learning. Adv. Neural Inf. Process. Syst. 34, 8229–8241 (2021)
Barreto, A., et al.: Successor features for transfer in reinforcement learning. In: Advances in Neural Information Processing Systems (NIPS), vol. 30. Barcelona, Spain (2017)
Barreto, A., Hou, S., Borsa, D., Silver, D., Precup, D.: Fast reinforcement learning with generalized policy updates. Proc. Natl. Acad. Sci. 117, 30079–30087 (2020)
Barreto, A., et al.: Transfer in deep reinforcement learning using successor features and generalised policy improvement. In: International Conference on Machine Learning (ICML), pp. 501–510 (2019)
Brantley, K., Mehri, S., Gordon, G.J.: Successor feature sets: generalizing successor representations across policies. In: AAAI Conference on Artificial Intelligence, vol. 35, pp. 11774–11781 (2021)
Carvalho, W., Filos, A., Lewis, R.L., Lee, H., Singh, S.: Composing task knowledge with modular successor feature approximators. In: International Conference on Learning Representations (ICLR) (2023)
Dayan, P.: Improving generalization for temporal difference learning: the successor representation. Neural Comput. 5(4), 613–624 (1993)
Goumiri, I.R., Priest, B.W., Schneider, M.D.: Reinforcement learning via gaussian processes with neural network dual kernels. In: IEEE Conference on Games (CoG), pp. 1–8 (2020)
Hunt, J., Barreto, A., Lillicrap, T., Heess, N.: Composing entropic policies using divergence correction. In: Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 2911–2920 (2019)
Janner, M., Mordatch, I., Levine, S.: \(\gamma \)-models: generative temporal difference learning for infinite-horizon prediction. Adv. Neural Inf. Process. Syst. (NIPS) 33, 1724–1735 (2020)
Kulkarni, T.D., Saeedi, A., Gautam, S., Gershman, S.J.: Deep successor reinforcement learning (2016). preprint on webpage at https://arxiv.org/abs/1606.02396
van der Laan, M.J., Polley, E.C., Hubbard, A.E.: Super learner. Stat. Appl. Genet. Mol. Biol. 6 (2007)
Madarasz, T., Behrens, T.: Better transfer learning with inferred successor maps. In: Advances in Neural Information Processing Systems (NIPS), vol. 32. Vancouver, BC, Canada (2019)
Rezende, D.J., Mohamed, S.: Variational inference with normalizing flows. In: International Conference on Machine Learning (ICML), vol. 37, pp. 1530–1538 (2015)
Schaul, T., Horgan, D., Gregor, K., Silver, D.: Universal value function approximators. In: International Conference on Machine Learning (ICML), vol. 37, pp. 1312–1320. Lille, France (2015)
Tasfi, N., Santana, E., Liboni, L., Capretz, M.: Dynamic successor features for transfer learning and guided exploration. Knowl.-Based Syst. 267, 110401 (2023)
Zhang, J., Springenberg, J.T., Boedecker, J., Burgard, W.: Deep reinforcement learning with successor features for navigation across similar environments. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371–2378 (2017)
Zhu, Z., Lin, K., Jain, A.K., Zhou, J.: Transfer learning in deep reinforcement learning: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 45(11), 13344–13362 (2023)
Acknowledgments
This work was supported by the Australian Research Council through Discovery Early Career Researcher Awards DE220101075 and DE200100245, and by the University of Technology Sydney (UTS) and Australian Technology Network (ATN) through UTS ATN-LATAM Research Scholarship Award.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Garces, K., Xuan, J., Zuo, H. (2024). Transformed Successor Features for Transfer Reinforcement Learning. In: Liu, T., Webb, G., Yue, L., Wang, D. (eds) AI 2023: Advances in Artificial Intelligence. AI 2023. Lecture Notes in Computer Science(), vol 14472. Springer, Singapore. https://doi.org/10.1007/978-981-99-8391-9_24
Download citation
DOI: https://doi.org/10.1007/978-981-99-8391-9_24
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8390-2
Online ISBN: 978-981-99-8391-9
eBook Packages: Computer ScienceComputer Science (R0)