Abstract
Finding short and convenient routes for vehicles is an important issue on efficient operations of Automated Guided Vehicle (AGV) systems at container terminals. This paper proposes an anisotropic Q-learning method for AGVs to find the shortest-time routes in the guide-path network of cross-lane type according to real-time vehicle states, which includes current and destination positions, heading direction and the number of vehicles at anisotropic four-direction neighboring locations. The vehicle waiting time of AGV systems is discussed and its estimation is suggested to improve the policy to select actions in the Q-learning method. An improved anisotropic Q-learning routing algorithm is developed with the vehicle-waiting-time-estimation based selecting-action policy. The parameter settings and performance of the proposed methods are analyzed based on simulations. The numerical experiments show that the improved anisotropic Q-learning method can provide stable and dynamic solutions for AGV routing, and achieve 9.5% improvement in optimization efficiency compared to the Jeon Learning Method (Jeon et al. in Logist Res 3(1):19–27, 2011).
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Draganjac, I., Miklic, D., Zdenko Kovacic, Z., Vasiljevic, G., Bogdan, S.: Decentralized control of multi-AGV systems in autonomous warehousing applications. IEEE Trans. Autom. Sci. Eng. 13(4), 1433–1447 (2016)
Fanti, M.P., Mangini, A.M., Pedroncelli, G., Ukovich, W.: A decentralized control strategy for the coordination of AGV systems. Control Eng. Pract. 70, 86–97 (2018)
Fazlollahtabar, H., Saidi-Mehrabad, M.: Methodologies to optimize automated guided vehicle scheduling and routing problems: a review study. J. Intell. Rob. Syst. 77, 525–545 (2015)
Ghasemzadeh, G., Behrangi, E., Azgomi, M.A.: Conflict-free scheduling and routing of automated guided vehicles in mesh topologies. Robot. Auton. Syst. 57, 738–748 (2009)
Jeon, S.M., Kim, K.H., Kopfer, H.: Routing automated guide vehicle in container terminals through the Q-learning technique. Logist. Res. 3(1), 19–27 (2011)
Kim, C.W., Tanchoco, J.M.A.: Conflict-free shortest-time bidirectional AGV routeing. Int. J. Prod. Res. 29(12), 2377–2391 (1991)
Kim, K.H., Jeon, S.M., Ryu, K.R.: Deadlock prevention for automated guided vehicles in automated container terminals. OR Spectrum 28, 659–679 (2006)
Li, Q., Adriaansen, A.C., Udding, J.T., Pogromsky, A.Y.: Design and control of automated guided vehicle systems: a case study. In: Proceedings of the 18th World Congress The International Federation of Automatic Control, Milano, Italy, Aug. 28–Sep. 2, pp. 13852–13857 (2011)
Li, Q., Pogromsky, A., Adriaansen, T., Udding, J.T.: A control of collision and deadlock avoidance for automated guided vehicles with a fault-tolerance capability. Int. J. Adv. Robot. Syst. 13(2), 1–24 (2016)
Lim, J.K., Kim, K.H.: Dynamic routing in automated guided vehicle systems. JSME Int. J. 45(1), 323–332 (2002)
Lim, J.K., Lim, J.M., Yoshimoto, K., Kim, K.H., Takahashi, T.: A construction algorithm for designing guide paths of automated guided vehicle system. Int. J. Prod. Res. 40(15), 3981–3994 (2002)
Maza, S., Castagna, P.: A performance-based structural policy for conflict-free routing bi-directional automated guided vehicles. Comput. Ind. 56, 719–733 (2005)
Moorthy, R.L., Guan, W.H., Cheong, N.W., Piaw, T.C.: Cyclic deadlock prediction and avoidance for zone-controlled AGV system. Int. J. Prod. Econ. 83, 309–324 (2003)
Nishi, T., Ando, M., Konishi, M.: Experimental studies on a local rescheduling procedure for dynamic routing of autonomous decentralized AGV systems. Robot. Comput. Integr. Manuf. 22, 154–165 (2006)
Nishi, T., Morinaka, S., Konishi, M.: A distributed routing method for AGVs under motion delay disturbance. Robot. Comput. Integr. Manuf. 23, 517–532 (2007)
Oboth, C., Batta, R., Karwan, M.: Dynamic conflict-free routing of automated guided vehicles. Int. J. Prod. Res. 37(9), 2003–2030 (1999)
Qiu, L., Hsu, W.J.: A bi-directional path layout for conflict-free routing of AGVs. Int. J. Prod. Res. 39(10), 2177–2195 (2001)
Qiu, L., Hsu, W.J., Huang, S.Y., Wang, H.: Scheduling and routing algorithms for AGVs: a survey. Int. J. Prod. Res. 40(3), 745–760 (2002)
Rajotia, S., Shanker, K., Batra, J.L.: A semi-dynamic time window constrained routeing strategy in an AGV system. Int. J. Prod. Res. 36(1), 35–50 (1998)
Srivastava, S.C., Choudhary, A.K., Kumar, S., Tiwari, M.K.: Development of an intelligent agent-based AGV controller for a flexible manufacturing system. Int. J. Adv. Manuf. Technol. 36, 780–797 (2008)
Taghaboni-Dutta, F., Tanchoco, J.M.A.: Comparison of dynamic routing techniques for automated guided vehicle system. Int. J. Prod. Res. 33(10), 2653–2669 (1995)
Vis, I.F.A.: Survey of research in the design and control of automated guided vehicle systems. Eur. J. Oper. Res. 170, 677–709 (2006)
Yoo, J., Sim, E., Cao, C., Park, J.: An algorithm for deadlock avoidance in an AGV System. Int. J. Adv. Manuf. Technol. 26, 659–668 (2005)
Zeng, J., Hsu, W.J.: Conflict-free container routing in mesh yard layouts. Robot. Auton. Syst. 56, 451–460 (2008)
Acknowledgements
This study was supported partially by the National Natural Science Foundation of China (No. 71101014), the fundamental research funds for the central universities (No. DUT16QY47) and the Liaoning Social Science Planning Funds (No. L17BGL014).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhou, P., Lin, L. & Kim, K.H. Anisotropic Q-learning and waiting estimation based real-time routing for automated guided vehicles at container terminals. J Heuristics 29, 207–228 (2023). https://doi.org/10.1007/s10732-020-09463-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10732-020-09463-9