skip to main content
10.1145/3678717.3691208acmconferencesArticle/Chapter ViewAbstractPublication PagesgisConference Proceedingsconference-collections
research-article

EFECTIW-ROTER: Deep Reinforcement Learning Approach for Solving Heterogeneous Fleet and Demand Vehicle Routing Problem With Time-Window Constraints

Published: 22 November 2024 Publication History

Abstract

The heterogeneous fleet and demand vehicle routing problem with time-window constraints (HFDVRPTW) is a crucial optimization problem of significant importance in real-world logistics operations. In this paper, we propose a deep reinforcement learning (DRL)-based method, termed spatial Edge-Feature EnhanCed mulTIgraph fusion encoder With spectral-based embedding and hieRarchical decOder with learnable TEmpoRal positional embedding (EFECTIW-ROTER, pronounced "Effective Router"), to tackle this complex and practical optimization problem. EFECTIW-ROTER utilizes two sparse graphs to represent node connectivity, where nodes correspond to customers and the depot. This sparsity results from the time-window constraints and customers' demand relative to the list of acceptable vehicle attributes specified for service within a heterogeneous fleet, determined by the reachability of the nodes based on these two factors. Leveraging two graph Transformer models, EFECTIW-ROTER's encoding module captures the interactions between the nodes based on these factors. One model encodes customers' heterogeneous demand with spatial edge features based on travel time between the nodes, while the second employs temporal positional embeddings to capture temporal relationships based on time-window ordering. A fusion model is introduced to integrate node interactions based on these graphs. Additionally, a spectral-attention-based pooling ensures effective state representation for the DRL-based method. EFECTIW-ROTER features a hierarchical attention decoder operating in two stages: heterogeneous vehicle selection and node selection. Enhanced with positional embeddings, the decoder is empowered to make effective routing decisions based on time-window constraints' ordering. Experimental results using real-world traffic data from two major Canadian cities confirm EFECTIW-ROTER's better performance over current state-of-the-art DRL-based and heuristic methods. EFECTIW-ROTER reduces travel times while also achieving faster computational times when compared to conventional heuristics. Additional experiments demonstrate its generalizability across larger instances.

References

[1]
Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. 2017. Neural Combinatorial Optimization with Reinforcement Learning. https://openreview.net/forum?id=rJY3vK9eg
[2]
David Buterez, Jon Paul Janet, Steven J Kiddle, Dino Oglic, and Pietro Liò. 2022. Graph neural networks with adaptive readouts. Advances in Neural Information Processing Systems 35 (2022), 19746--19758.
[3]
Jinbiao Chen, Huanhuan Huang, Zizhen Zhang, and Jiahai Wang. 2022. Deep reinforcement learning with two-stage training strategy for practical electric vehicle routing problem with time windows. In International conference on parallel problem solving from nature. Springer, 356--370.
[4]
Lu Duan, Yang Zhan, Haoyuan Hu, Yu Gong, Jiangwen Wei, Xiaodong Zhang, and Yinghui Xu. 2020. Efficiently solving the practical vehicle routing problem: A novel joint learning approach. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 3054--3063.
[5]
Alexandre Duval and Fragkiskos Malliaros. 2022. Higher-order clustering and pooling for graph neural networks. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 426--435.
[6]
Vijay Prakash Dwivedi and Xavier Bresson. 2021. A Generalization of Transformer Networks to Graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications (2021).
[7]
Richard W Eglese, Alan Mercer, and Babak Sohrabi. 2005. The grocery superstore vehicle scheduling problem. Journal of the Operational Research Society 56 (2005), 902--911.
[8]
Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. 2020. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning. PMLR, 3419--3430.
[9]
Oscar M González, Carlos Segura, and Sergio I Valdez Peña. 2018. A parallel memetic algorithm to solve the capacitated vehicle routing problem with time windows. International Journal of Combinatorial Optimization Problems and Informatics 9, 1 (2018), 35.
[10]
Jihee Han, Arash Mozhdehi, Yunli Wang, Sun Sun, and Xin Wang. 2022. Solving a multi-trip vrp with real heterogeneous fleet and time windows based on ant colony optimization: An industrial case study. In Proceedings of the 15th ACM SIGSPATIAL International Workshop on Computational Transportation Science. 1--4.
[11]
Chaug-Ing Hsu, Sheng-Feng Hung, and Hui-Chieh Li. 2007. Vehicle routing problem with time-windows for perishable food delivery. Journal of food engineering 80, 2 (2007), 465--475.
[12]
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam M. Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2018. Music Transformer: Generating Music with Long-Term Structure. In International Conference on Learning Representations. https://api.semanticscholar.org/CorpusID:54477714
[13]
Saeid Kalantari, Reza Safarzadeh Ramhormozi, Yunli Wang, Sun Sun, and Xin Wang. 2023. Trailer allocation and truck routing using bipartite graph assignment and deep reinforcement learning. Transactions in GIS 27, 4 (2023), 996--1020.
[14]
He-Yau Kang and Amy HI Lee. 2018. An enhanced approach for the multiple vehicle routing problem with heterogeneous vehicles and a soft time window. Symmetry 10, 11 (2018), 650.
[15]
AJ Klein. 1987. Microcomputer-based vehicle routing and scheduling: An overview. (1987).
[16]
Wouter Kool, Herke van Hoof, and Max Welling. 2018. Attention, Learn to Solve Routing Problems!. In International Conference on Learning Representations.
[17]
Keyju Lee and Junjae Chae. 2021. A proposal and analysis of new realistic sets of benchmark instances for vehicle routing problems with asymmetric costs. Applied Sciences 11, 11 (2021), 4790.
[18]
Keyju Lee and Junjae Chae. 2023. Estimation of Travel Cost between Geographic Coordinates Using Artificial Neural Network: Potential Application in Vehicle Routing Problems. ISPRS International Journal of Geo-Information 12, 2 (2023), 57.
[19]
Jingwen Li, Yining Ma, Ruize Gao, Zhiguang Cao, Andrew Lim, Wen Song, and Jie Zhang. 2021. Deep reinforcement learning for solving the heterogeneous capacitated vehicle routing problem. IEEE Transactions on Cybernetics 52, 12 (2021), 13572--13585.
[20]
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. 2021. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems 34 (2021), 15816--15829.
[21]
Bo Lin, Bissan Ghaddar, and Jatin Nathwani. 2021. Deep reinforcement learning for the electric vehicle routing problem with time windows. IEEE Transactions on Intelligent Transportation Systems 23, 8 (2021), 11528--11538.
[22]
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, Puneet K Dokania, Mark Coates, Philip Torr, and Ser-Nam Lim. 2023. Graph inductive biases in transformers without message passing. In International Conference on Machine Learning. PMLR, 23321--23337.
[23]
Odalric-Ambrym Maillard, Daniil Ryabko, and Rémi Munos. 2011. Selecting the state-representation in reinforcement learning. Advances in Neural Information Processing Systems 24 (2011).
[24]
Arash Mozhdehi, Mahdi Mohammadizadeh, and Xin Wang. 2024. Edge-DIRECT: A Deep Reinforcement Learning-based Method for Solving Heterogeneous Electric Vehicle Routing Problem with Time Window Constraints. arXiv preprint arXiv:2407.01615 (2024).
[25]
Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. 2018. Reinforcement learning for solving the vehicle routing problem. Advances in neural information processing systems 31 (2018).
[26]
Michael Polacek, Richard F Hartl, Karl Doerner, and Marc Reimann. 2004. A variable neighborhood search for the multi depot vehicle routing problem with time windows. Journal of heuristics 10 (2004), 613--627.
[27]
Christian Prins. 2002. Efficient heuristics for the heterogeneous fleet multitrip VRP with application to a large-scale real case. Journal of Mathematical Modelling and Algorithms 1, 2 (2002), 135--150.
[28]
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 7008--7024.
[29]
Michael Schneider, Andreas Stenger, and Dominik Goeke. 2014. The electric vehicle-routing problem with time windows and recharging stations. Transportation science 48, 4 (2014), 500--520.
[30]
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics.
[31]
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 464--468. https://doi.org/10.18653/v1/N18-2074
[32]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
[33]
Conghui Wang, Zhiguang Cao, Yaoxin Wu, Long Teng, and Guohua Wu. 2024. Deep Reinforcement Learning for Solving Vehicle Routing Problems With Backhauls. IEEE Transactions on Neural Networks and Learning Systems (2024).
[34]
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8 (1992), 229--256.
[35]
David H Wolpert and William G Macready. 1997. No free lunch theorems for optimization. IEEE transactions on evolutionary computation 1, 1 (1997), 67--82.
[36]
Yunqiu Xu, Meng Fang, Ling Chen, Gangyan Xu, Yali Du, and Chengqi Zhang. 2021. Reinforcement learning with multiple relational attention for solving vehicle routing problems. IEEE Transactions on Cybernetics 52, 10 (2021), 11107--11120.
[37]
Hongyu Zang, Xin Li, and Mingzhong Wang. 2022. Simsr: Simple distance-based state representations for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8997--9005.
[38]
Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang, Chengwei Yao, Zhi Yu, and Can Wang. 2019. Hierarchical graph pooling with structure learning. arXiv preprint arXiv:1911.05954 (2019).
[39]
Kenny Qili Zhu. 2000. A new genetic algorithm for VRPTW. In Proceedings of the international conference on artificial intelligence, Vol. 311264.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGSPATIAL '24: Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems
October 2024
743 pages
Publication rights licensed to ACM. ACM acknowledges that this contribution was co-authored by an affiliate of the Crown in Right of Canada. As such, the Crown in Right of Canada retains an equal interest in the copyright. Reprint requests should be forwarded to ACM, and reprints must include clear attribution to ACM and Crown in Right of Canada.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 November 2024

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Attention Model
  2. Combinatorial Optimization
  3. Reinforcement Learning
  4. Spatial-Temporal systems

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

SIGSPATIAL '24
Sponsor:

Acceptance Rates

SIGSPATIAL '24 Paper Acceptance Rate 37 of 122 submissions, 30%;
Overall Acceptance Rate 257 of 1,238 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 56
    Total Downloads
  • Downloads (Last 12 months)56
  • Downloads (Last 6 weeks)35
Reflects downloads up to 14 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media