skip to main content
10.1145/3511808.3557112acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

CTRL: Cooperative Traffic Tolling via Reinforcement Learning

Published: 17 October 2022 Publication History

Abstract

People have been working long to tackle the traffic congestion problem. Among the different measures, traffic tolling has been recognized as an effective way to mitigate citywide congestion. However, traditional tolling methods can not deal with the dynamic traffic flow in cities. Meanwhile, thanks to the development of traffic sensing technology, how to set appropriate dynamic tolling according to real time traffic observations has attracted research attention in recent years.
In this paper, we put the dynamic tolling problem in a reinforcement learning setting and try to tackle the three key challenges of complex state representation, pricing action credit assignment, and route price relative competition. We propose a soft actor-critic method with (1) a route-level state attention, (2) an interpretable and provable reward design, and (3) a competition-aware Q attention. Extensive experiments on real datasets have shown the superior performance of our proposed method. In addition, interesting analysis on pricing actions and vehicle routes have demonstrated why the proposed method can outperform baselines.

References

[1]
Mauricio Arango. 2019. Toll Road with Dynamic Congestion Pricing Using Reinforcement Learning. (2019).
[2]
Kim Thien Bui, Vu Anh Huynh, and Emilio Frazzoli. 2012. Dynamic traffic congestion pricing mechanism with user-centric considerations. In 2012 15th International IEEE Conference on Intelligent Transportation Systems. IEEE, 147--154.
[3]
Yu-Han Chang, Tracey Ho, and Leslie P Kaelbling. 2004. All learning is local: Multi-agent learning in global reward games. (2004).
[4]
Haipeng Chen, Bo An, Guni Sharon, Josiah Hanna, Peter Stone, Chunyan Miao, and Yeng Soh. 2018. Dyetc: Dynamic electronic toll collection for traffic congestion alleviation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[5]
André de Palma, Moez Kilani, and Robin Lindsey. 2005. Congestion pricing on a road network: A study using the dynamic equilibrium simulator METROPOLIS. Transportation Research Part A-policy and Practice, Vol. 39 (2005), 588--611.
[6]
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018a. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. PMLR, 1861--1870.
[7]
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. 2018b. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018).
[8]
Ammar Haydari and Yasin Yilmaz. 2020. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems (2020).
[9]
Junchen Jin and Xiaoliang Ma. 2019. A multi-objective agent-based control approach with application in intelligent traffic signal system. IEEE Transactions on Intelligent Transportation Systems, Vol. 20, 10 (2019), 3900--3912.
[10]
Jiahui Jin, Xiaoxuan Zhu, Biwei Wu, Jinghui Zhang, and Yuxiang Wang. 2021. A dynamic and deadline-oriented road pricing mechanism for urban traffic management. Tsinghua Science and Technology, Vol. 27, 1 (2021), 91--102.
[11]
Dusica Joksimovic, Michiel CJ Bliemer, and Piet HL Bovy. 2005. Optimal toll design problem in dynamic traffic networks with joint route and departure time choice. Transportation Research Record, Vol. 1923, 1 (2005), 61--72.
[12]
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, Vol. 521, 7553 (2015), 436--444.
[13]
Zhenhui Li. n.d. Reinforcement Learning for Traffic Signal Control. https://traffic-signal-control.github.io
[14]
Jiancheng Long, Ziyou Gao, Haozhi Zhang, and Wai Yuen Szeto. 2010. A turning restriction design problem in urban road networks. European Journal of Operational Research, Vol. 206, 3 (2010), 569--578.
[15]
David Metz. 2018. Tackling urban traffic congestion: The experience of London, Stockholm and Singapore. Case Studies on Transport Policy, Vol. 6, 4 (2018), 494--498.
[16]
Hamid Mirzaei, Guni Sharon, Stephen Boyles, Tony Givargis, and Peter Stone. 2018. Enhanced delta-tolling: Traffic optimization via policy gradient reinforce-ment learning. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 47--52.
[17]
Venktesh Pandey and Stephen D Boyles. 2018. Multiagent reinforcement learning algorithm for distributed dynamic pricing of managed lanes. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2346--2351.
[18]
Venktesh Pandey, Evana Wang, and Stephen D Boyles. 2020. Deep reinforcement learning algorithm for dynamic pricing of express lanes with multiple access locations. Transportation Research Part C: Emerging Technologies, Vol. 119 (2020), 102715.
[19]
Wei Qiu, Haipeng Chen, and Bo An. 2019. Dynamic Electronic Toll Collection via Multi-Agent Deep Reinforcement Learning with Edge-Based Graph Convolutional Networks. In IJCAI. 4568--4574.
[20]
Sandeep Saharan, Seema Bawa, and Neeraj Kumar. 2020. Dynamic pricing techniques for Intelligent Transportation System in smart cities: A systematic review. Computer Communications, Vol. 150 (2020), 603--625.
[21]
Guni Sharon, Michael W Levin, Josiah P Hanna, Tarun Rambha, Stephen D Boyles, and Peter Stone. 2017. Network-wide adaptive tolling for connected and automated vehicles. Transportation Research Part C: Emerging Technologies, Vol. 84 (2017), 142--157.
[22]
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. 2017. Value-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:1706.05296 (2017).
[23]
Richard Stuart Sutton. 1984. Temporal credit assignment in reinforcement learning. Ph.,D. Dissertation. University of Massachusetts Amherst.
[24]
Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. 2017. Multiagent cooperation and competition with deep reinforcement learning. PloS one, Vol. 12, 4 (2017), e0172395.
[25]
Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li. 2019a. Colight: Learning network-level cooperation for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 1913--1922.
[26]
Hua Wei, Guanjie Zheng, Vikash Gayah, and Zhenhui Li. 2019b. A Survey on Traffic Signal Control Methods. arXiv preprint arXiv:1904.08117 (2019).
[27]
Hai Yang and Xiaoning Zhang. 2003. Optimal toll design in second-best link-based congestion pricing. Transportation Research Record, Vol. 1857, 1 (2003), 85--92.
[28]
Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. 2019. CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario. The World Wide Web Conference (May 2019). https://doi.org/10.1145/3308558.3314139
[29]
Xiaoning Zhang, H. Michael Zhang, Haijun Huang, Lijun Sun, and Tie-Qiao Tang. 2011. Competitive, cooperative and Stackelberg congestion pricing for multiple regions in transportation networks. Transportmetrica, Vol. 7 (2011), 297 -- 320.
[30]
Pengjun Zhao and Haoyu Hu. 2019. Geographical patterns of traffic congestion in growing megacities: Big data analytics from Beijing. Cities, Vol. 92 (2019), 164--174.
[31]
Bojian Zhou, Michiel Bliemer, Hai Yang, and Jie He. 2015. A trial-and-error congestion pricing scheme for networks with elastic demand and link capacity constraints. Transportation Research Part B: Methodological, Vol. 72 (2015), 77--92.

Cited By

View all
  • (2024)MAGT-toll: A multi-agent reinforcement learning approach to dynamic traffic congestion pricingPLOS ONE10.1371/journal.pone.031382819:11(e0313828)Online publication date: 18-Nov-2024
  • (2024)Frequency Enhanced Pre-training for Cross-City Few-shot Traffic ForecastingMachine Learning and Knowledge Discovery in Databases. Research Track10.1007/978-3-031-70344-7_3(35-52)Online publication date: 22-Aug-2024
  • (2023)Dynamic Tolling in Arc-based Traffic Assignment Models2023 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton)10.1109/Allerton58177.2023.10313516(1-8)Online publication date: 26-Sep-2023

Index Terms

  1. CTRL: Cooperative Traffic Tolling via Reinforcement Learning

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management
      October 2022
      5274 pages
      ISBN:9781450392365
      DOI:10.1145/3511808
      • General Chairs:
      • Mohammad Al Hasan,
      • Li Xiong
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 17 October 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. dynamic tolling
      2. reinforcement learning

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      CIKM '22
      Sponsor:

      Acceptance Rates

      CIKM '22 Paper Acceptance Rate 621 of 2,257 submissions, 28%;
      Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

      Upcoming Conference

      CIKM '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)30
      • Downloads (Last 6 weeks)1
      Reflects downloads up to 10 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)MAGT-toll: A multi-agent reinforcement learning approach to dynamic traffic congestion pricingPLOS ONE10.1371/journal.pone.031382819:11(e0313828)Online publication date: 18-Nov-2024
      • (2024)Frequency Enhanced Pre-training for Cross-City Few-shot Traffic ForecastingMachine Learning and Knowledge Discovery in Databases. Research Track10.1007/978-3-031-70344-7_3(35-52)Online publication date: 22-Aug-2024
      • (2023)Dynamic Tolling in Arc-based Traffic Assignment Models2023 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton)10.1109/Allerton58177.2023.10313516(1-8)Online publication date: 26-Sep-2023

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media