Skip to main content

Advertisement

Springer Nature Link
Log in

Experience classification for transfer learning in traffic signal control

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In recent years, due to the drastic rise in the number of vehicles and the lack of sufficient infrastructure, traffic jams, air pollution, and fuel consumption have increased in cities. The optimization of timing for traffic lights is one of the solutions for the mentioned problems. Many methods have been introduced to deal with these problems, including reinforcement learning. Although a great number of learning-based methods have been used in traffic signal control, they suffer from poor performance and slow learning convergence. In this paper, a transfer learning-based method for traffic signal control has been proposed. Multi-agent system has also been used for modelling the traffic network and transfer learning has been used to make reinforcement learning agents transfer their experience to each other. Furthermore, a classifier has been utilized to classify the transferred experiences. The results show that using the proposed method leads to a significant improvement on average delay time and convergence time of the learning process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. www.aimsun.com.

References

  1. Abdelghaffar HM, Yang H, Rakha HA (2016) Isolated traffic signal control using a game theoretic framework. In: 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). IEEE, pp 1496–1501

  2. Abdoos M, Mozayani N, Bazzan AL (2011) Traffic light control in non-stationary environments based on multi agent q-learning. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, pp 1580–1585

  3. Abdoos M, Mozayani N, Bazzan AL (2013) Holonic multi-agent system for traffic signals control. Eng Appl Artif Intell 26(5–6):1575–1587

    Article  Google Scholar 

  4. Abdoos M, Mozayani N, Bazzan AL (2014) Hierarchical control of traffic signals using q-learning with tile coding. Appl Intell 40(2):201–213

    Article  Google Scholar 

  5. Ammar HB, Eaton E, Taylor ME, Mocanu DC, Driessens K, Weiss G, Tuyls K (2014) An automated measure of MDP similarity for transfer in reinforcement learning. In: Workshops at the Twenty-Eighth AAAI Conference on Artificial Intelligence

  6. Banerjee B, Taylor ME (2018) Coordination confidence based human-multi-agent transfer learning for collaborative teams. In: AAMAS Adaptive Learning Agents (ALA) Workshop. sn

  7. Brys T, Harutyunyan A, Taylor ME, Nowé A (2015) Policy transfer using reward shaping. In: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, pp 181–188

  8. Da Silva FL, Costa AHR (2019) A survey on transfer learning for multiagent reinforcement learning systems. J Artif Intell Res 64:645–703

    Article  MathSciNet  MATH  Google Scholar 

  9. Ding D, Ding Z, Wei G, Han F (2019) An improved reinforcement learning algorithm based on knowledge transfer and applications in autonomous vehicles. Neurocomputing 361:243–255

    Article  Google Scholar 

  10. Fachantidis A, Partalas I, Taylor ME, Vlahavas I (2015) Transfer learning with probabilistic mapping selection. Adapt Behav 23(1):3–19

    Article  Google Scholar 

  11. Hou Y, Ong YS, Feng L, Zurada JM (2017) An evolutionary transfer reinforcement learning framework for multiagent systems. IEEE Trans Evol Comput 21(4):601–615

    Article  Google Scholar 

  12. Li M, Dai Q (2018) A novel knowledge-leverage-based transfer learning algorithm. Appl Intell 48(8):2355–2372

    Article  Google Scholar 

  13. Mannion P, Duggan J, Howley E (2016) An experimental review of reinforcement learning algorithms for adaptive traffic signal control. In: Autonomic road transport support systems. Springer, Birkhäuser, Cham, pp 47–66

  14. Moshkov MJ (2005) Time complexity of decision trees. In: Transactions on rough sets III. Springer, Berlin, Heidelberg, pp 244–459

  15. Omidshafiei S, Kim DK, Liu M, Tesauro G, Riemer M, Amato C, Campbell M, How JP (2019) Learning to teach in cooperative multiagent reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 33, pp 6128–6136

  16. Prashanth LA, Bhatnagar S (2011) Reinforcement learning with function approximation for traffic signal control. IEEE Trans Intell Transp Syst 12(2):412–421. https://doi.org/10.1109/TITS.2010.2091408

    Article  Google Scholar 

  17. Prashanth L, Bhatnagar S (2011) Reinforcement learning with average cost for adaptive control of traffic lights at intersections. In: 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, pp 1640–1645

  18. Rosyadi AR, Wirayuda TAB, Al-Faraby S (2016) Intelligent traffic light control using collaborative q-learning algorithms. In: 2016 4th International Conference on Information and Communication Technology (ICoICT). IEEE, pp 1–6

  19. Shoeleh F, Asadpour M (2017) Graph based skill acquisition and transfer learning for continuous reinforcement learning domains. Pattern Recognit Lett 87:104–116

    Article  Google Scholar 

  20. Shoeleh F, Asadpour M (2019) Skill based transfer learning with domain adaptation for continuous reinforcement learning domains. Appl Intell 50(2):1–17

    Google Scholar 

  21. Taylor ME, Stone P (2007) Cross-domain transfer for reinforcement learning. In: Proceedings of the 24th International Conference on Machine Learning, ICML ’07. ACM, New York, pp 879–886

  22. Taylor ME, Stone P (2009) Transfer learning for reinforcement learning domains: a survey. J Mach Learn Res 10:1633–1685

    MathSciNet  MATH  Google Scholar 

  23. Taylor ME, Suay HB, Chernova S (2011) Integrating reinforcement learning with human demonstrations of varying ability. In: The 10th International Conference on Autonomous Agents and Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, vol 2, pp 617–624

  24. Thorpe TL, Anderson CW (1996) Traffic light control using sarsa with three state representations. Technical report, Citeseer

  25. Wang Z, Taylor ME (2016) Effective transfer via demonstrations in reinforcement learning: a preliminary study. In: 2016 AAAI Spring Symposium Series

  26. Zhang S, Taylor ME (2018) Enhanced learning from multiple demonstrations with a two-level structured approach. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2018)

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Engineering, Shahid Beheshti University, Tehran, Iran

    Mojtaba Norouzi & Monireh Abdoos

  2. Institute of Informatics, Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil

    Ana L. C. Bazzan

Authors
  1. Mojtaba Norouzi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Monireh Abdoos
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ana L. C. Bazzan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Monireh Abdoos.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Norouzi, M., Abdoos, M. & Bazzan, A.L.C. Experience classification for transfer learning in traffic signal control. J Supercomput 77, 780–795 (2021). https://doi.org/10.1007/s11227-020-03287-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03287-x

Keywords