Abstract
Online scheduling has been an attractive field of research for over three decades. Some recent developments suggest that Reinforcement Learning (RL) techniques can effectively deal with online scheduling issues. Driven by an industrial application, in this paper we apply four of the most important RL techniques, namely Q-learning, Sarsa, Watkins’s Q(\(\lambda \)), and Sarsa(\(\lambda \)), to the online single-machine scheduling problem. Our main goal is to provide insights into how such techniques perform in the scheduling process. We will consider the minimization of two different and widely used objective functions: the total tardiness and the total earliness and tardiness of the jobs. The computational experiments show that Watkins’s Q(\(\lambda \)) performs best in minimizing the total tardiness. At the same time, it seems that the RL approaches are not very effective in minimizing the total earliness and tardiness over large time horizons.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Plastic and Rubber 4.0. Piattaforma Tecnologica per la Fabbrica Intelligente (Technological Platform for Smart Factory), URL: https://www.openplast.it/en/homepage-en/.
- 2.
References
Adamu, M.O., Adewumi, A.: A survey of single machine scheduling to minimize weighted number of tardy jobs. J. Ind. Manag. Optim. 10, 219–241 (2014)
Behnamiana, J., Ghomi, S.F., Zandieh, M.: A multi-phase covering pareto-optimal front method to multi-objective scheduling in a realistic hybrid flowshop using a hybrid metaheuristic. Expert Syst. Appl. 36, 11057–11069 (2009)
Brucker, P.: Scheduling Algorithms, 5th edn. Springer Publishing Company, Incorporated (2010)
Castrogiovanni, P., Fadda, E., Perboli, G., Rizzo, A.: Smartphone data classification technique for detecting the usage of public or private transportation modes. IEEE Access 8, 58377–58391 (2020). https://doi.org/10.1109/ACCESS.2020.2982218
Cerone, V., Fadda, E., Regruto, D.: A robust optimization approach to kernel-based nonparametric error-in-variables identification in the presence of bounded noise. In: 2017 American Control Conference (ACC), IEEE (2017). https://doi.org/10.23919/ACC.2017.7963056
Correa, J.R., Wagner, M.R.: Lp-based online scheduling: from single to parallel machines. Math. Program. 119(1), 109–136 (2009)
Fadda, E., Plebani, P., Vitali, M.: Optimizing monitorability of multi-cloud applications. In: Nurcan, S., Soffer, P., Bajec, M., Eder, J. (eds.) Advanced Information Systems Engineering. CAiSE 2016. Lecture Notes in Computer Science, vol. 9694, pp. 411–426. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39696-5_25
Fadda, E., Perboli, G., Squillero, G.: Adaptive batteries exploiting on-line steady-state evolution strategy. In: Squillero, G., Sim, K. (eds.) Applications of Evolutionary Computation. EvoApplications 2017. Lecture Notes in Computer Science, vol. 10199, pp. 329–341. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-55849-3_22
Fadda, E., Manerba, D., Tadei, R., Camurati, P., Cabodi, G.: KPIs for optimal location of charging stations for electric vehicles: the Biella case-study. In: Ganzha, M., Maciaszek, L., Paprzycki, M. (eds.) Proceedings of the 2019 Federated Conference on Computer Science and Information Systems, IEEE, Annals of Computer Science and Information Systems, vol. 18, pp. 123–126 (2019). https://doi.org/10.15439/2019F171
Fadda, E., Manerba, D., Cabodi, G., Camurati, P., Tadei, R.: Evaluation of Optimal Charging Station Location for Electric Vehicles: An Italian Case-Study, pp. 71–87 (2021). https://doi.org/10.1007/978-3-030-58884-7_4
Fadda, E., Manerba, D., Cabodi, G., Camurati, P.E., Tadei, R.: Comparative analysis of models and performance indicators for optimal service facility location. Transp. Res. Part E: Logist. Transp. Rev. 145 (2021)
Fernandez-Viagas, V., Dios, M., Framinan, J.M.: Ecient constructive and composite heuristics for the permutation flowshop to minimise total earliness and tardiness. Comput. Oper. Res. 75, 38–48 (2016)
François-Lavet, V., Fonteneau, R., Ernst, D.: How to discount deep reinforcement learning: towards new dynamic strategies (2015). arXiv:151202011
Gabel, T., Riedmiller, M.: Adaptive reactive job-shop scheduling with reinforcement learning agents. Int. J. Inf. Technol. Intell. Comput. 24(4), 14–18 (2008)
Giusti, R., Iorfida, C., Li, Y., Manerba, D., Musso, S., Perboli, G., Tadei, R., Yuan, S.: Sustainable and de-stressed international supply-chains through the synchro-net approach. Sustainability 11, 1083 (2019). https://doi.org/10.3390/su11041083
Graham, R.L.: Bounds for certain multiprocessing anomalies. Bell Syst. Tech. J. 45(9), 1563–1581 (1966). https://doi.org/10.1002/j.1538-7305.1966.tb01709.x
Graves, S.C.: A review of production scheduling. Oper. Res. 29(4), 646–675 (1981). https://doi.org/10.1287/opre.29.4.646
Kaban, A., Othman, Z., Rohmah, D.: Comparison of dispatching rules in job-shop scheduling problem using simulation: a case study. Int. J. Simul. Model. 11(3), 129–140 (2012). https://doi.org/10.2507/IJSIMM11(3)2.201
Kanet, J.: Minimizing the average deviation of job completion times about a common due date. Nav. Res. Logist. Q. 28, 643–651 (1981)
Koulamas, C.: The single-machine total tardiness scheduling problem: review and extensions. Eur. J. Oper. Res. 202, 1–7 (2010)
Leksakul, K., Techanitisawad, A.: An application of the neural network energy function to machine sequencing. Comput. Manag. Sci. 2, 309–338 (2005)
Li, Y., Carabelli, S., Fadda, E., Manerba, D., Tadei, R., Terzo, O.: Machine learning and optimization for production rescheduling in industry 4.0. In: The International Journal of Advanced Manufacturing Technology, pp. 1–19 (2020). https://doi.org/10.1007/s00170-020-05850-5
Lu, X., Sitters, R., Stougie, L.: A class of on-line scheduling algorithms to minimize total completion time. Oper. Res. Lett. 31(3), 232–236 (2003). https://doi.org/10.1016/S0167-6377(03)00016-6
Marco Silve, N.M., Poss, Michael: Solution algorithms for minimizing the total tardiness with budgeted processing time uncertainty. Eur. J. Oper. Res. 283, 70–82 (2020)
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013). arXiv:13125602
Oliver Herr, G.: Minimising total tardiness for a single machine scheduling problem with family setups and resource constraints. Eur. J. Oper. Res. 248, 123–135 (2016)
Panwalkar, S.S., Iskander, W.: A survey of scheduling rules. Oper. Res. 25(1), 45–61 (1977). https://doi.org/10.1287/opre.25.1.45
Pinedo, M.: Scheduling: Theory, Algorithms, and Systems. Springer, New York, NY, USA (2012)
Rice, J.R.: The algorithm selection problem. In: Advances in Computers, vol. 15, pp. 65–118. Elsevier (1976)
Schaller, J., Valente, J.: Branch-and-bound algorithms for minimizing total earliness and tardiness in a two-machine permutation flow shop with unforced idle allowed. Comput. Oper. Res. 109, 1–11 (2019)
Sharma, H., Jain, S.: Online learning algorithms for dynamic scheduling problems. In: 2011 Second International Conference on Emerging Applications of Information Technology, pp. 31–34 (2011)
Singh, S., Jaakkola, T., Littman, M.L., Szepesvári, C.: Convergence results for single-step on-policy reinforcement-learning algorithms. Mach. Learn. 38(3), 287–308 (2000). https://doi.org/10.1023/A:1007678930559
Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction. MIT press (2018)
Suwa, H., Sandoh, H.: Online Scheduling in Manufacturing: A Cumulative Delay Approach. Springer Science & Business Media (2012)
Takadama, K., Fujita, H.: Toward guidelines for modeling learning agents in multiagent-based simulation: implications from q-learning and sarsa agents. In: International Workshop on Multi-Agent Systems and Agent-Based Simulation, pp. 159–172. Springer (2004). https://doi.org/10.1007/978-3-540-32243-6_13
Watkins, C.J.C.H.: Learning from delayed rewards. Thesis Submitted for Ph.D., King’s College, Cambridge (1989)
Xie, S., Zhang, T., Rose, O.: Online single machine scheduling based on simulation and reinforcement learning. In: Simulation in Produktion und Logistik 2019, Simulation in Produktion und Logistik 2019 (2019)
Ying, K.C.: Minimizing earliness-tardiness penalties for common due date single-machine scheduling problems by a recovering beam search algorithm. Comput. Ind. Eng. 55, 494–502 (2008)
Zhang, T., Xie, S., Rose, O.: Real-time job shop scheduling based on simulation and markov decision processes. In: 2017 Winter Simulation Conference (WSC), IEEE, pp. 3899–3907 (2017). https://doi.org/10.1109/WSC.2017.8248100
Acknowledgements
This research was partially supported by the Plastic and Rubber 4.0 (P&R4.0) research project, POR FESR 2014–2020 - Action I.1b.2.2, funded by Piedmont Region (Italy), Contract No. 319-31. The authors acknowledge all the project partners for their contribution.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Li, Y., Fadda, E., Manerba, D., Roohnavazfar, M., Tadei, R., Terzo, O. (2022). Online Single-Machine Scheduling via Reinforcement Learning. In: Fidanova, S. (eds) Recent Advances in Computational Optimization. WCO 2020. Studies in Computational Intelligence, vol 986. Springer, Cham. https://doi.org/10.1007/978-3-030-82397-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-82397-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82396-2
Online ISBN: 978-3-030-82397-9
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)