Abstract
In the rapidly evolving domain of reinforcement learning (RL), which has applications in computer vision and games, our research presents an RL-based Embedding algorithm (EmbRL) that, applied to an autonomous car racing environment, allows for rapid algorithm training with significant results.
EmbRL addresses the challenge of processing high-dimensional camera inputs, which is common in advanced game environments like OpenAI Five and AlphaStar. By employing a pre-trained supervised learning model, our algorithm efficiently transforms these inputs into a set of 1000 class features, which are then processed by a fully connected network (FCN) acting as the RL model.
This method effectively separates the task of understanding the vehicle’s state from the core path-finding and control tasks performed using a separate RL network, simplifying the task of autonomous car racing. Our findings show a remarkable reduction in training time, speeding up training by 230% compared to traditional end-to-end convolutional networks, as well as a significant boost to the reward, highlighting EmbRL’s potential in enhancing the real-time applicability of vision models. This study integrates concepts from established methodologies, incorporating minor modifications that result in significant enhancements in performance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arulkumaran, K., Cully, A., Togelius, J.: Alphastar: an evolutionary computation perspective. In: GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion, pp. 314–315 (2019). https://doi.org/10.1145/3319619.3321894
Becker, M., Lippel, J., Stuhlsatz, A.: Regularized nonlinear discriminant analysis - an approach to robust dimensionality reduction for data visualization. In: VISIGRAPP 2017 - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 4, pp. 116–127 (2017). https://doi.org/10.5220/0006167501160127
Campos, V., et al.: Beyond fine-tuning: transferring behavior in reinforcement learning
Chen, L., et al.: Driving with LLMs: fusing object-level vector modality for explainable autonomous driving. https://github.com/wayveai/Driving-with-LLMs
Chiu, B., Crichton, G., Korhonen, A., Pyysalo, S.: How to train good word embeddings for biomedical NLP, pp. 166–174 (2016). http://www.ncbi.nlm.nih.gov/pmc/
Dai, B., Shen, X., Wang, J.: Embedding learning. J. Am. Stat. Assoc. 2022(537), 307–319 (2020). https://doi.org/10.1080/01621459.2020.1775614
Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., Labi, S.: Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems. Transp. Res. Part C: Emerg. Technol. 156, 104358 (2023). https://doi.org/10.1016/J.TRC.2023.104358
Ermolov, A., Sebe, N.: Latent world models for intrinsically motivated exploration. In: Advances in Neural Information Processing Systems, vol. 34 (2020). https://github.com/htdt/lwm
Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y.N., Natarajan, P.: Transform-retrieve-generate: natural language-centric outside-knowledge visual question answering. In: Computer Vision Pattern Recognition (2022). https://github.com/JaidedAI/EasyOCR
Ha, D., Urgen Schmidhuber, J.: World Models. https://worldmodels.github.io
Ha Google Brain Tokyo, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Advances in Neural Information Processing Systems, vol. 31 (2018). https://worldmodels.github.io
Hafner, D., Research, G., Lillicrap, T., Norouzi, M., Ba, J.: Mastering atari with discrete world models
Henzinger, T.A., Sifakis, J.: The embedded systems design challenge. In: Misra, J., Nipkow, T., Sekerinski, E. (eds.) FM 2006. LNCS, vol. 4085, pp. 1–15. Springer, Heidelberg (2006). https://doi.org/10.1007/11813040_1
Hinton, G., Dean, J.: Distilling the knowledge in a neural network (2015)
Kato, S., et al.: Autoware on board: enabling autonomous vehicles with embedded systems. In: Proceedings - 9th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS 2018, pp. 287–296 (2018). https://doi.org/10.1109/ICCPS.2018.00035
Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. 23(6), 4909–4926 (2022). https://doi.org/10.1109/TITS.2021.3054625
Klimov, O.: Car Racing - Gym Documentation. https://www.gymlibrary.dev/environments/box2d/car_racing/
Li, L.: Towards a Unified Theory of State Abstraction for MDPs (2006). http://anytime.cs.umass.edu/aimath06/proceedings/P21.pdf
Munk, J., Kober, J., Babuska, R.: Learning state representation for deep actor-critic control. In: 2016 IEEE 55th Conference on Decision and Control, CDC 2016, pp. 4667–4673 (2016). https://doi.org/10.1109/CDC.2016.7798980
Patil, R., Boit, S., Gudivada, V., Nandigam, J.: A survey of text representation and embedding techniques in NLP. IEEE Access 11, 36120–36146 (2023). https://doi.org/10.1109/ACCESS.2023.3266377
Pritz, P.J., Ma, L., Leung, K.K.: Jointly-learned state-action embedding for efficient reinforcement learning. In: International Conference on Information and Knowledge Management, Proceedings, pp. 1447–1456 (2021). https://doi.org/10.1145/3459637.3482357
Schmidhuber, J.: On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models Technical report (2015)
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Openai, O.K.: Proximal Policy Optimization Algorithms (2017). https://arxiv.org/abs/1707.06347v2
Shah, D., Osiński, B., ichter, b., Levine, S.: LM-Nav: robotic navigation with large pre-trained models of language, vision, and action. In: Liu, K., Kulic, D., Ichnowski, J. (eds.) Proceedings of The 6th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 205, pp. 492–504. PMLR (2023). https://proceedings.mlr.press/v205/shah23b.html
Shah, R., Kumar, V.: RRL: ResNet as representation for reinforcement learning. In: Proceedings of the 38th International Conference on Machine Learning, vol. 38 (2021)
Shen, S., Mulgaonkar, Y., Michael, N., Kumar, V.: Vision-based state estimation for autonomous rotorcraft MAVs in complex environments. In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 1758–1764 (2013). https://doi.org/10.1109/ICRA.2013.6630808
Stankovic, J.A.: Real-time and embedded systems. ACM Comput. Surv. 28(1) (1996)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. The MIT Press, Cambridge (2015). https://inst.eecs.berkeley.edu/~cs188/sp20/assets/files/SuttonBartoIPRLBook2ndEd.pdf
Tao, R.Y., François-Lavet, V., Pineau, J.: Novelty search in representational space for sample efficient exploration. In: Advances in Neural Information Processing Systems, vol. 34 (2020). https://github.com/taodav/nsrs
Webb, T.P., Prazenica, R.J., Kurdila, A.J., Lind, R.: Vision-based state estimation for autonomous micro air vehicles. 30(3), 816–826 (2007). https://doi.org/10.2514/1.22398
Wurman, P.R., et al.: Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature 602(7896), 223–228 (2022). https://doi.org/10.1038/s41586-021-04357-7
Xu, Y., Hansen, N., Wang, Z., Chan, Y.C., Su, H., Tu, Z.: On the feasibility of cross-task transfer with model-based reinforcement learning. In: The Eleventh International Conference on Learning Representations (ICLR) (2023). https://nicklashansen.github.io/xtra
Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109, 43–76 (2021). https://doi.org/10.1109/JPROC.2020.3004555
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Holen, M., Singh, J., Omlin, C.W., Zhou, J., Knausgård, K.M., Goodwin, M. (2025). Optimizing Autonomous Vehicle Racing Using Reinforcement Learning with Pre-trained Embeddings for Dimensionality Reduction. In: Bramer, M., Stahl, F. (eds) Artificial Intelligence XLI. SGAI 2024. Lecture Notes in Computer Science(), vol 15447. Springer, Cham. https://doi.org/10.1007/978-3-031-77918-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-77918-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-77917-6
Online ISBN: 978-3-031-77918-3
eBook Packages: Computer ScienceComputer Science (R0)