Skip to main content

Optimizing Autonomous Vehicle Racing Using Reinforcement Learning with Pre-trained Embeddings for Dimensionality Reduction

  • Conference paper
  • First Online:
Artificial Intelligence XLI (SGAI 2024)

Abstract

In the rapidly evolving domain of reinforcement learning (RL), which has applications in computer vision and games, our research presents an RL-based Embedding algorithm (EmbRL) that, applied to an autonomous car racing environment, allows for rapid algorithm training with significant results.

EmbRL addresses the challenge of processing high-dimensional camera inputs, which is common in advanced game environments like OpenAI Five and AlphaStar. By employing a pre-trained supervised learning model, our algorithm efficiently transforms these inputs into a set of 1000 class features, which are then processed by a fully connected network (FCN) acting as the RL model.

This method effectively separates the task of understanding the vehicle’s state from the core path-finding and control tasks performed using a separate RL network, simplifying the task of autonomous car racing. Our findings show a remarkable reduction in training time, speeding up training by 230% compared to traditional end-to-end convolutional networks, as well as a significant boost to the reward, highlighting EmbRL’s potential in enhancing the real-time applicability of vision models. This study integrates concepts from established methodologies, incorporating minor modifications that result in significant enhancements in performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/marho13/EmbeddingInput.

References

  1. Arulkumaran, K., Cully, A., Togelius, J.: Alphastar: an evolutionary computation perspective. In: GECCO 2019 Companion - Proceedings of the 2019 Genetic and Evolutionary Computation Conference Companion, pp. 314–315 (2019). https://doi.org/10.1145/3319619.3321894

  2. Becker, M., Lippel, J., Stuhlsatz, A.: Regularized nonlinear discriminant analysis - an approach to robust dimensionality reduction for data visualization. In: VISIGRAPP 2017 - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, vol. 4, pp. 116–127 (2017). https://doi.org/10.5220/0006167501160127

  3. Campos, V., et al.: Beyond fine-tuning: transferring behavior in reinforcement learning

    Google Scholar 

  4. Chen, L., et al.: Driving with LLMs: fusing object-level vector modality for explainable autonomous driving. https://github.com/wayveai/Driving-with-LLMs

  5. Chiu, B., Crichton, G., Korhonen, A., Pyysalo, S.: How to train good word embeddings for biomedical NLP, pp. 166–174 (2016). http://www.ncbi.nlm.nih.gov/pmc/

  6. Dai, B., Shen, X., Wang, J.: Embedding learning. J. Am. Stat. Assoc. 2022(537), 307–319 (2020). https://doi.org/10.1080/01621459.2020.1775614

    Article  MathSciNet  Google Scholar 

  7. Dong, J., Chen, S., Miralinaghi, M., Chen, T., Li, P., Labi, S.: Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems. Transp. Res. Part C: Emerg. Technol. 156, 104358 (2023). https://doi.org/10.1016/J.TRC.2023.104358

    Article  Google Scholar 

  8. Ermolov, A., Sebe, N.: Latent world models for intrinsically motivated exploration. In: Advances in Neural Information Processing Systems, vol. 34 (2020). https://github.com/htdt/lwm

  9. Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y.N., Natarajan, P.: Transform-retrieve-generate: natural language-centric outside-knowledge visual question answering. In: Computer Vision Pattern Recognition (2022). https://github.com/JaidedAI/EasyOCR

  10. Ha, D., Urgen Schmidhuber, J.: World Models. https://worldmodels.github.io

  11. Ha Google Brain Tokyo, D., Schmidhuber, J.: Recurrent world models facilitate policy evolution. In: Advances in Neural Information Processing Systems, vol. 31 (2018). https://worldmodels.github.io

  12. Hafner, D., Research, G., Lillicrap, T., Norouzi, M., Ba, J.: Mastering atari with discrete world models

    Google Scholar 

  13. Henzinger, T.A., Sifakis, J.: The embedded systems design challenge. In: Misra, J., Nipkow, T., Sekerinski, E. (eds.) FM 2006. LNCS, vol. 4085, pp. 1–15. Springer, Heidelberg (2006). https://doi.org/10.1007/11813040_1

    Chapter  Google Scholar 

  14. Hinton, G., Dean, J.: Distilling the knowledge in a neural network (2015)

    Google Scholar 

  15. Kato, S., et al.: Autoware on board: enabling autonomous vehicles with embedded systems. In: Proceedings - 9th ACM/IEEE International Conference on Cyber-Physical Systems, ICCPS 2018, pp. 287–296 (2018). https://doi.org/10.1109/ICCPS.2018.00035

  16. Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Syst. 23(6), 4909–4926 (2022). https://doi.org/10.1109/TITS.2021.3054625

    Article  Google Scholar 

  17. Klimov, O.: Car Racing - Gym Documentation. https://www.gymlibrary.dev/environments/box2d/car_racing/

  18. Li, L.: Towards a Unified Theory of State Abstraction for MDPs (2006). http://anytime.cs.umass.edu/aimath06/proceedings/P21.pdf

  19. Munk, J., Kober, J., Babuska, R.: Learning state representation for deep actor-critic control. In: 2016 IEEE 55th Conference on Decision and Control, CDC 2016, pp. 4667–4673 (2016). https://doi.org/10.1109/CDC.2016.7798980

  20. Patil, R., Boit, S., Gudivada, V., Nandigam, J.: A survey of text representation and embedding techniques in NLP. IEEE Access 11, 36120–36146 (2023). https://doi.org/10.1109/ACCESS.2023.3266377

    Article  Google Scholar 

  21. Pritz, P.J., Ma, L., Leung, K.K.: Jointly-learned state-action embedding for efficient reinforcement learning. In: International Conference on Information and Knowledge Management, Proceedings, pp. 1447–1456 (2021). https://doi.org/10.1145/3459637.3482357

  22. Schmidhuber, J.: On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models Technical report (2015)

    Google Scholar 

  23. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Openai, O.K.: Proximal Policy Optimization Algorithms (2017). https://arxiv.org/abs/1707.06347v2

  24. Shah, D., Osiński, B., ichter, b., Levine, S.: LM-Nav: robotic navigation with large pre-trained models of language, vision, and action. In: Liu, K., Kulic, D., Ichnowski, J. (eds.) Proceedings of The 6th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 205, pp. 492–504. PMLR (2023). https://proceedings.mlr.press/v205/shah23b.html

  25. Shah, R., Kumar, V.: RRL: ResNet as representation for reinforcement learning. In: Proceedings of the 38th International Conference on Machine Learning, vol. 38 (2021)

    Google Scholar 

  26. Shen, S., Mulgaonkar, Y., Michael, N., Kumar, V.: Vision-based state estimation for autonomous rotorcraft MAVs in complex environments. In: Proceedings - IEEE International Conference on Robotics and Automation, pp. 1758–1764 (2013). https://doi.org/10.1109/ICRA.2013.6630808

  27. Stankovic, J.A.: Real-time and embedded systems. ACM Comput. Surv. 28(1) (1996)

    Google Scholar 

  28. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. The MIT Press, Cambridge (2015). https://inst.eecs.berkeley.edu/~cs188/sp20/assets/files/SuttonBartoIPRLBook2ndEd.pdf

  29. Tao, R.Y., François-Lavet, V., Pineau, J.: Novelty search in representational space for sample efficient exploration. In: Advances in Neural Information Processing Systems, vol. 34 (2020). https://github.com/taodav/nsrs

  30. Webb, T.P., Prazenica, R.J., Kurdila, A.J., Lind, R.: Vision-based state estimation for autonomous micro air vehicles. 30(3), 816–826 (2007). https://doi.org/10.2514/1.22398

  31. Wurman, P.R., et al.: Outracing champion Gran Turismo drivers with deep reinforcement learning. Nature 602(7896), 223–228 (2022). https://doi.org/10.1038/s41586-021-04357-7

    Article  Google Scholar 

  32. Xu, Y., Hansen, N., Wang, Z., Chan, Y.C., Su, H., Tu, Z.: On the feasibility of cross-task transfer with model-based reinforcement learning. In: The Eleventh International Conference on Learning Representations (ICLR) (2023). https://nicklashansen.github.io/xtra

  33. Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109, 43–76 (2021). https://doi.org/10.1109/JPROC.2020.3004555

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Martin Holen .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Holen, M., Singh, J., Omlin, C.W., Zhou, J., Knausgård, K.M., Goodwin, M. (2025). Optimizing Autonomous Vehicle Racing Using Reinforcement Learning with Pre-trained Embeddings for Dimensionality Reduction. In: Bramer, M., Stahl, F. (eds) Artificial Intelligence XLI. SGAI 2024. Lecture Notes in Computer Science(), vol 15447. Springer, Cham. https://doi.org/10.1007/978-3-031-77918-3_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-77918-3_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-77917-6

  • Online ISBN: 978-3-031-77918-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics