Skip to main content
SpringerLink
Log in

High-Level Decision-Making Non-player Vehicles

  • Conference paper
  • First Online:
Games and Learning Alliance (GALA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13647))

Included in the following conference series:

Abstract

Availability of realistic driver models, also able to represent various driving styles, is key to add traffic in serious games on automotive driving. We propose a new architecture for behavioural planning of vehicles, that decide their motion taking high-level decisions, such as “keep lane”, “overtake” and “go to rightmost lane”. This is similar to a driver’s high-level reasoning and takes into account the availability of ever more sophisticated Advanced Driving Assistance Systems (ADAS) in current vehicles. Compared to a low-level decision making system, our model performs better both in terms of safety and average speed. As a significant advantage, the hierarchical approach allows to reduce the number of training steps, which is critical for ML models, by more than one order of magnitude. The developed agent seems to show a more realistic behaviour. We also showed feasibility of training models able to differentiate their performance in a way similar to the driving styles. We believe that such agents could be profitably employed in state of the art SGs for driving, improving the realism of single NPVs and overall traffic.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Massoud, R., Berta, R., Poslad, S., De Gloria, A., Bellotti, F.: IoT sensing for reality-enhanced serious games, a fuel-efficient drive use case. Sensors. 21, 3559 (2021). https://doi.org/10.3390/s21103559

    Article  Google Scholar 

  2. Nousias, S., et al.: Exploiting gamification to improve eco-driving behaviour: the GamECAR approach. Electron. Notes Theor. Comput. Sci. 343, 103–116 (2019). https://doi.org/10.1016/j.entcs.2019.04.013

    Article  Google Scholar 

  3. The Good Drive, a serious game for learning to drive - Renault Group. https://www.renaultgroup.com/en/news-on-air/news/the-good-drive-a-serious-game-for-learning-to-drive/. Accessed 22 July 2022

  4. Leurent, E., Mercat, J.: Social Attention for Autonomous Decision-Making in Dense Traffic (2019). https://doi.org/10.48550/arXiv.1911.12250

  5. González, D., Pérez, J., Milanés, V., Nashashibi, F.: A review of motion planning techniques for automated vehicles. IEEE Trans. Intell. Transp. Syst. 17, 1135–1145 (2016). https://doi.org/10.1109/TITS.2015.2498841

    Article  Google Scholar 

  6. Francois-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., Pineau, J.: An introduction to deep reinforcement learning. Found. Trends® Mach. Learn. 11, 219–354 (2018). https://doi.org/10.1561/2200000071

  7. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: A brief survey of deep reinforcement learning. IEEE Signal Process. Mag. 34, 26–38 (2017). https://doi.org/10.1109/MSP.2017.2743240

    Article  Google Scholar 

  8. GitHub - eleurent/highway-env: A minimalist environment for decision-making in autonomous driving. https://github.com/eleurent/highway-env. Accessed 11 July 2022

  9. Brockman, G., et al.: OpenAI Gym. (2016). https://doi.org/10.48550/arXiv.1606.01540

  10. Campodonico, G., et al.: Adapting autonomous agents for automotive driving games. In: de Rosa, F., Marfisi Schottman, I., Baalsrud Hauge, J., Bellotti, F., Dondio, P., Romero, M. (eds.) GALA 2021. LNCS, vol. 13134, pp. 101–110. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-92182-8_10

    Chapter  Google Scholar 

  11. Rana, A., Malhi, A.: Building safer autonomous agents by leveraging risky driving behavior knowledge. In: 2021 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI), pp. 1–6 (2021). https://doi.org/10.1109/CCCI52664.2021.9583209

  12. Rais, M.S., Boudour, R., Zouaidia, K., Bougueroua, L.: Decision making for autonomous vehicles in highway scenarios using Harmonic SK Deep SARSA. Appl. Intell. (2022). https://doi.org/10.1007/s10489-022-03357-y

  13. Zhang, S., Wu, Y., Ogai, H., Inujima, H., Tateno, S.: Tactical decision-making for autonomous driving using dueling double deep Q network with double attention. IEEE Access. 9, 151983–151992 (2021). https://doi.org/10.1109/ACCESS.2021.3127105

    Article  Google Scholar 

  14. SL, UT: Car Driving School Simulator (Android). https://car-driving-school-simulator.en.uptodown.com/android. Accessed 22 July 2022

  15. Gounaridou, A., Siamtanidou, E., Dimoulas, C.: A serious game for mediated education on traffic behavior and safety awareness. Educ. Sci. 11, 127 (2021). https://doi.org/10.3390/educsci11030127

    Article  Google Scholar 

  16. Likitweerawong, K., Palee, P.: The virtual reality serious game for learning driving skills before taking practical test. In: 2018 International Conference on Digital Arts Media Technology, ICDAMT (2018). https://doi.org/10.1109/ICDAMT.2018.8376515

  17. Hrimech, H., et al.: The effects of the use of serious game in eco-driving training. Front. ICT. 3 (2016)

    Google Scholar 

  18. Massoud, R., Poslad, S., Bellotti, F., Berta, R., Mehran, K., Gloria, A.D.: A fuzzy logic module to estimate a driver’s fuel consumption for reality-enhanced serious games. Int. J. Serious Games 5, 45–62 (2018). https://doi.org/10.17083/ijsg.v5i4.266

  19. Lazzaroni, L., Mazzara, A., Bellotti, F., De Gloria, A., Berta, R.: Employing an IoT framework as a generic serious games analytics engine. In: Marfisi-Schottman, I., Bellotti, F., Hamon, L., Klemke, R. (eds.) GALA 2020. LNCS, vol. 12517, pp. 79–88. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63464-3_8

    Chapter  Google Scholar 

  20. Westera, W., et al.: Artificial intelligence moving serious gaming: presenting reusable game AI components. Educ. Inf. Technol. 25(1), 351–380 (2019). https://doi.org/10.1007/s10639-019-09968-2

    Article  Google Scholar 

  21. Tomlinson, S.L., Melder, N.: An architecture overview for AI in racing games. (2015). https://doi.org/10.1201/b16725-44

  22. Perot, E., Jaritz, M., Toromanoff, M., De Charette, R.: End-to-end driving in a realistic racing game with deep reinforcement learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 474–475 (2017). https://doi.org/10.1109/CVPRW.2017.64

  23. Fakhry, A.: Applying a Deep Q Network for OpenAI’s Car Racing Game. https://towardsdatascience.com/applying-a-deep-q-network-for-openais-car-racing-game-a642daf58fc9. Accessed 22 July 2022

  24. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015). https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  25. Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8, 279–292 (1992). https://doi.org/10.1007/BF00992698

    Article  MATH  Google Scholar 

  26. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms (2017). https://doi.org/10.48550/arXiv.1707.06347

  27. Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems. MIT Press (1999)

    Google Scholar 

  28. Wang, T., et al.: Benchmarking Model-Based Reinforcement Learning (2019). https://doi.org/10.48550/arXiv.1907.02057

  29. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., Dormann, N.: Stable-Baselines3: reliable reinforcement learning implementations. J. Mach. Learn. Res. 22, 1–8 (2021)

    MATH  Google Scholar 

  30. Mirchevska, B., Pek, C., Werling, M., Althoff, M., Boedecker, J.: High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2156–2162 (2018). https://doi.org/10.1109/ITSC.2018.8569448

  31. PPO—Stable Baselines3 1.6.1a0 documentation. https://stable-baselines3.readthedocs.io/en/master/modules/ppo.html. Accessed 23 July 2022

  32. Abadi, M., et al.: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016). https://doi.org/10.48550/arXiv.1603.04467

  33. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. 114, 3521–3526 (2017). https://doi.org/10.1073/pnas.1611835114

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical, Electronic and Telecommunication Engineering (DITEN), University of Genoa, Via Opera Pia 11a, 16145, Genoa, Italy

    Alessandro Pighetti, Luca Forneris, Luca Lazzaroni, Francesco Bellotti, Alessio Capello, Marianna Cossu, Alessandro De Gloria & Riccardo Berta

Authors
  1. Alessandro Pighetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luca Forneris
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luca Lazzaroni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francesco Bellotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alessio Capello
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marianna Cossu
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Alessandro De Gloria
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Riccardo Berta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francesco Bellotti .

Editor information

Editors and Affiliations

  1. Tampere University, Tampere, Finland

    Kristian Kiili

  2. Tampere University, Tampere, Finland

    Koskinen Antti

  3. Center for Advanced Pathogen Threat Response and Simulation, Austin, TX, USA

    Francesca de Rosa

  4. Tampere University, Tampere, Finland

    Muhterem Dindar

  5. Pädagogische Hochschule St.Gallen, St. Gallen, St. Gallen, Switzerland

    Michael Kickmeier-Rust

  6. University of Genova, Genova, Italy

    Francesco Bellotti

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pighetti, A. et al. (2022). High-Level Decision-Making Non-player Vehicles. In: Kiili, K., Antti, K., de Rosa, F., Dindar, M., Kickmeier-Rust, M., Bellotti, F. (eds) Games and Learning Alliance. GALA 2022. Lecture Notes in Computer Science, vol 13647. Springer, Cham. https://doi.org/10.1007/978-3-031-22124-8_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22124-8_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22123-1

  • Online ISBN: 978-3-031-22124-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation