Abstract
We introduce a technique that leverages the power of indirect encodings (IE) from the field of evolutionary computation to improve the speed of evolution in transfer learning control tasks. Although generative models have previously been used to construct IEs, their potential in transfer learning, specifically in reinforcement learning domains, has not yet been utilised. We train three types of generative models: an autoencoder (AE), a variational autoencoder (VAE) and a generative adversarial network (GAN) on the neural network weights of well-performing solutions of a set of paramaterised source domains. The decoder of the AE and VAE or the generator of the GAN is then used as the IE in an evolutionary run on unseen, but related, target domains. We compare against two baselines: a direct encoding (DE) and a DE starting evolution from a controller pre-trained to maximise the average fitness over the set of source domains. We show that, by using these IEs, the speed of learning on the target domains is greatly increased with respect to the baselines.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Both the training of the IEs and the initialisation of the UC have additional preparatory overheads compared to evolution using a DE only. To compare techniques according to the total number of FLOPS, including pretraining, would be particularly meticulous and, more importantly, implementation dependent. We have therefore decided to evaluate with respect to the number of generations inline with evaluation methods used in [4] (no. of generations) and [3] (no. of gradient steps).
- 2.
Often CMA-ES can discover solutions with very large values making it more difficult to train a generative model over.
References
Bentley, P.J., Lim, S.L., Gaier, A., Tran, L.: Coil: Constrained optimization in learned latent space - learning representations for valid solutions. CoRR abs/2202.02163 (2022). https://doi.org/10.48550/arXiv.2202.02163
Chang, O., Kwiatkowski, R., Chen, S., Lipson, H.: Agent embeddings: a latent representation for pole-balancing networks. In: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2019, pp. 656–664. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2019). https://dl.acm.org/doi/10.5555/3306127.3331753
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, vol. 70, pp. 1126–1135. JMLR.org (2017). https://dl.acm.org/doi/10.5555/3305381.3305498
Gaier, A., Asteroth, A., Mouret, J.B.: Discovering representations for black-box optimization. In: Proceedings of the 2020 Genetic and Evolutionary Computation Conference, GECCO 2020, pp. 103–111. Association for Computing Machinery, New York (2020). https://doi.org/10.1145/3377930.3390221
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014). https://doi.org/10.1109/CVPR.2014.81
Goldberg, D.E.: Simple genetic algorithms and the minimal, deceptive problem. In: Davis, L. (ed.) Genetic Algorithms and Simulated Annealing, pp. 74–88. Research Notes in Artificial Intelligence, Pitman, London (1987)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org
Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9, 159–195 (2001). https://doi.org/10.1162/106365601750190398
Jegorova, M., Doncieux, S., Hospedales, T.: Behavioral repertoire via generative adversarial policy networks. IEEE Trans. Cogn. Dev. Syst., 1 (2020). https://doi.org/10.1109/TCDS.2020.3008574
Kalehbasti, P.R., Lepech, M.D., Pandher, S.S.: Augmenting high-dimensional nonlinear optimization with conditional gans. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO 2021, pp. 1879–1880. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3449726.3463675
Kouvaris, K., Clune, J., Kounios, L., Brede, M., Watson, R.A.: How evolution learns to generalise: using the principles of learning theory to understand the evolution of developmental organisation. PLoS Comput. Biol. 13(4), 1–20 (2017). https://doi.org/10.1371/journal.pcbi.1005358
Moreno, M.A., Banzhaf, W., Ofria, C.: Learning an evolvable genotype-phenotype mapping. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, pp. 983–990. Association for Computing Machinery, New York (2018). https://doi.org/10.1145/3205455.3205597
Rakicevic, N., Cully, A., Kormushev, P.: Policy manifold search: Exploring the manifold hypothesis for diversity-based neuroevolution. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 901–909 (2021). https://doi.org/10.1145/3449639.3459320
Watson, R.A., Szathmáry, E.: How can evolution learn? Trends Ecol. Evol. 31, 147–157 (2016). https://doi.org/10.1016/j.tree.2015.11.009
Zhu, Z., Lin, K., Zhou, J.: Transfer learning in deep reinforcement learning: a survey. CoRR abs/2009.07888 (2020). https://arxiv.org/abs/2009.07888
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Butterworth, J., Savani, R., Tuyls, K. (2022). Generative Models over Neural Controllers for Transfer Learning. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds) Parallel Problem Solving from Nature – PPSN XVII. PPSN 2022. Lecture Notes in Computer Science, vol 13398. Springer, Cham. https://doi.org/10.1007/978-3-031-14714-2_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-14714-2_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14713-5
Online ISBN: 978-3-031-14714-2
eBook Packages: Computer ScienceComputer Science (R0)