Skip to main content

Advertisement

Log in

Reinforcement Learning Based Control Scheme for Emergency Vehicle Preemption with Edge Computing

  • Published:
International Journal of Intelligent Transportation Systems Research Aims and scope Submit manuscript

Abstract

This paper proposes a reinforcement learning-based collaborative multi-agent actor and critic scheme (RL-CMAS) under edge computing architecture for emergency vehicle preemption. The RL-CMAS deployed a parallel training process at the cloud side for building knowledge and well accelerating learning. Priority of message and model of message offloading strategy have been developed. The simulation results show that the proposed RL-CMAS is efficient in detecting even complex data. Finally, a comparison was made with other benchmark methods, namely, Regular scheduling algorithm, Alameddine’s DTOS algorithm, and independent multi-agent actor-critic. The result showed the proposed method outperforming the other three bench marking methods. The proposed RL-CMAS provides reduction in message processing delay, total delay, and an increase of message delivery success ratio of 14.22%, 18.21%, and 8.86% respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Ian, E., Blanchard, C.J., Doig, B.E., Hagel, A.R., Anton, D.A., Zygun, J.B., Kortbeek, D., Gregory Powell, T.S., Williamson, G.H., Fick, Grant, D., Innes: Emergency medical services response time and mortality in an urban setting. Prehospital Emerg. Care 16(1), 142–151 (2012). https://doi.org/10.3109/10903127.2011.614046

    Article  Google Scholar 

  2. He, Q., Head, K.L., Ding, J.: Multi-modal traffic signal control with priority, signal actuation and coordination. Transp. Res. Part C. 46, 65–82 (2014)

  3. Guler, S.I., Menendez, M.: Analytical formulation and empirical evaluation of pre-signals for bus priority. Transp. Res. Part. B 64, 41–53 (2014)

    Article  Google Scholar 

  4. Bell, M.G., Fonzone, A., Polyzoni, C.: Depot location in degradable transport networks. Transp. Res Part B 66, 148–161 (2014)

    Article  Google Scholar 

  5. Salazar-González, J.J., Santos-Hernández, B.: The split-demand one-commodity pickup-and-delivery travelling salesman problem. Transp. Res. Part. B 75, 58–73 (2015)

    Article  Google Scholar 

  6. Liu, K., Li, Q., Zhang, Z.H.: Distributionally robust optimization of an emergency medical service station location and sizing problem with joint chance constraints. Transp. Res. Part. B 119, 79–101 (2019)

    Article  Google Scholar 

  7. He, Q., Irnich, S., Song, Y.: Branch-and-cut-and-price for the vehicle routing problem with time windows and convex node costs. Transp. Sci 53(5), 1409–1426 (2019)

    Article  Google Scholar 

  8. Ansari, S., McLay, L.A., Mayorga, M.E.: A maximum expected covering problem for district design. Transp. Sci 51(1), 376–390 (2015)

    Article  Google Scholar 

  9. Huang, Y.S., Weng, Y.S., Zhou, M.: Design of traffic safety control systems for emergency vehicle preemption using timed petri nets. IEEE Trans. Intell. Transp. Syst 16(4), 2113–2120 (2015)

    Article  Google Scholar 

  10. Hannoun, G.J., Murray-Tuite, P., Heaslip, K., Chantem, T.: Facilitating emergency response vehicles’ movement through a road segment in a connected vehicle environment. IEEE Trans. Intell. Transp. Syst 20(9), 3546–3557 (2018)

    Article  Google Scholar 

  11. Chen, D., Ahn, S., Laval, J., Zheng, Z.: On the periodicity of traffic oscillations and capacity drop: the role of driver characteristics. Transp. Res. Part. B 59, 117–136 (2014)

    Article  Google Scholar 

  12. Rosayyan, P., Subramaniam, S., Ganesan, S.I.: Decentralized emergency service vehicle pre-emption system using RF communication and GNSS-based geo- fencing. IEEE Transactions on Intelligent Transportation Systems, pp. 1–10. https://doi.org/10.1109/TITS.2020.3007671

  13. Mukherjee, M., Shu, L., Wang, D.: Survey of fog omputing: fundamental, network applications, and research challenges. IEEE Commun. Surv. Tutor 20(3), 1826–1857 (2018)

    Article  Google Scholar 

  14. Wu, Q., Shen, J., Yong, B., Wu, J., Li, F., Wang, J., Zhou, Q.: Smart fog based workflow for traffic control networks. Future Gener. Comput. Syst 97, 825–835 (2019)

    Article  Google Scholar 

  15. Liu, J., Li, L., Zhang, F., Dai, Y., Zhang, X., Meng, J., Shen: Secure intelligent traffic light control using fog computing. Future Gener. Comput. Syst 78, 817–824 (2017). https://doi.org/10.1016/j.future.2017.02.017

    Article  Google Scholar 

  16. Jinjian, L., Jacques, B., Arnaud, D., Guillaume, L.: Multi-models machine learning methods for traffic flow estimation from floating car data. Transp. Res. Part C: Emerg Technol. 132 (2021). https://doi.org/10.1016/j.trc.2021.103389

  17. Posor, J.E., Belzner, L., Knapp, A.: Joint action learning for multi-agent cooperation using recurrent reinforcement learning. Digitale Welt. 479–84 (2020). https://doi.org/10.1007/s42354-019-0239-y

  18. Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning, 2017, arXiv preprint arXiv:1706.05296, https://arxiv.org/abs/1706.05296

  19. Saleem, M.A., Mahmood, K., Kumari, S.: Comments on AKM-IoV: Authenticated key management protocol in fog computing-based internet of vehicles deployment. IEEE Internet Things J 7(5), 4671–4675 (2020). https://doi.org/10.1109/JIOT.2019.2923611

    Article  Google Scholar 

  20. Sodhro, A.H., Pirbhulal, S., de Albuquerque, V.H.C.: Artificial intelligence driven mechanism for edge computing-based industrial applications. IEEE Trans. Ind. Inform 15(7), 4235–4243 (2019). https://doi.org/10.1109/TII.2019.2902878

    Article  Google Scholar 

  21. Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J 3(5), 637–646 (2016)

    Article  Google Scholar 

  22. Zhang, F., Li, J., Li, Z.: A TD3-based multi-agent deep reinforcement learning method in mixed cooperation- competition environment. Neurocomputing 411, 206–215 (2020)

    Article  Google Scholar 

  23. Cao, W., Xu, S., An emergency warning message dissemination policy of vehicular ad hoc network: 9th International Symposium on Computational Intelligence and Design (ISCID), 2016, pp. 227–230 (2016). https://doi.org/10.1109/ISCID.2016.2061

  24. Hollinghurst, J., Ganesh, A., Baugé, T.: Latency reduction in communication networks using redundant messages. Paper presented at: Proceedings of the 2017 29th International Teletraffic Congress (ITC 29), pp. 241–249. (2017)

  25. Kausar, S., Habib, M., Shabir, M.Y., et al.: Secure and efficient data transfer using spreading and assimilation in MANET. Softw. Pract. Exp 1(50), 35–50 (2020)

    Google Scholar 

  26. Wu, X., Chen, H., Chen, C., Zhong, M., Xie, S., Guo, Y., Fujita, H.: The autonomous navigation and obstacle avoidance for USVs with ANOA deep reinforcement learning method. Knowl. -Based Syst 196, 1–12 (2020)

    Article  Google Scholar 

  27. Yang, S., Yang, B., Wong, H., Kang, Z.: Cooperative traffic signal control using multi-step return and off-policy asynchronous advantage actor-critic graph algorithm. Knowl. -Based Syst 183, 1–19 (2019)

    Article  Google Scholar 

  28. Rasheed, F., Yau, K.A., Low, Y.: Deep reinforcement learning for traffic signal control under disturbances: a case study on sunway city, Malaysia, Future Gener. Comput. Syst 109, 431–445 (2020)

    Google Scholar 

  29. Wang, H., Li, J., Chen, Q.Y., Ni, D.: Logistic modeling of equilibrium speed–density relationship. Transp. Res. Part A: Policy Pract 45, 554–566 (2011)

  30. Wei, H., Chen, C., Zheng, G., Wu, K., Gayah, V., Xu, K., Li, Z., Presslight: Learning max pressure control to coordinate traffic signals in arterial network. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1290–1298 (2019)

  31. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986). https://doi.org/10.1038/323533a0

  32. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, O.P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative- competitive environments. In: Advances in Neural Information Processing Systems, pp. 6379–6390 (2017)

  33. Fedesoriano.: Traffic Prediction Dataset, Version 1. Retrieved November 2021 from https://www.kaggle.com/fedesoriano/traffic-prediction-dataset (2021)

  34. Haizhong, W., Jia, L., Qian-Yong, C., Daiheng, N., Logistic modeling of the equilibrium speed–density relationship. Transp. Res. Part A: Policy Pract. 45(6), 554–566 (2011)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Prakash Rosayyan.

Ethics declarations

Conflict of Interest

The Authors declared that there is no conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rosayyan, P., Paul, J., Subramaniam, S. et al. Reinforcement Learning Based Control Scheme for Emergency Vehicle Preemption with Edge Computing. Int. J. ITS Res. 21, 48–62 (2023). https://doi.org/10.1007/s13177-022-00334-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13177-022-00334-0

Keywords

Navigation