Skip to main content
Springer Nature Link
Log in

Adaptive job shop scheduling strategy based on weighted Q-learning algorithm

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

Given the dynamic and uncertain production environment of job shops, a scheduling strategy with adaptive features must be developed to fit variational production factors. Therefore, a dynamic scheduling system model based on multi-agent technology, including machine, buffer, state, and job agents, was built. A weighted Q-learning algorithm based on clustering and dynamic search was used to determine the most suitable operation and to optimize production. To address the large state space problem caused by changes in the system state, four state features were extracted. The dimension of the system state was decreased through the clustering method. To reduce the error between the actual system states and clustering ones, the state difference degree was defined and integrated with the iteration formula of the Q function. To select the optimal state-action pair, improved search and iteration update strategies were proposed. Convergence analysis of the proposed algorithm and simulation experiments indicated that the proposed adaptive strategy is well adaptable and effective in different scheduling environments, and shows better performance in complex environments. The two contributions of this research are as follows: (1) a dynamic greedy search strategy was developed to avoid blind searching in traditional strategy. (2) Weighted iteration update of the Q function, including the weighted mean of the maximum fuzzy earning, was designed to improve the speed and accuracy of the improved learning algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Baker, K. R. (1984). Sequencing rules and due-date assignments in a job shop. Management Science,30(9), 1093–1104.

    Google Scholar 

  • Delgoshaei, A., Ali, A., Ariffin, M. K. A., & Gomes, C. (2016). A multi-period scheduling of dynamic cellular manufacturing systems in the presence of cost uncertainty. Computers & Industrial Engineering,100, 110–132.

    Google Scholar 

  • Frobenius, G. (1912). Uber Matrizen aus nicht negativen Elementen.

  • Gao, L., & Pan, Q. K. (2016). A shuffled multi-swarm micro-migrating birds optimizer for a multi-resource-constrained flexible job shop scheduling problem. Information Sciences,372, 655–676.

    Google Scholar 

  • Huang, G. Q., Zhang, Y. F., Chen, X., & Newman, S. T. (2008). RFID-enabled real-time wireless manufacturing for adaptive assembly planning and control. Journal of Intelligent Manufacturing,19(6), 701–713.

    Google Scholar 

  • Karimi-Nasab, M., Modarres, M., & Seyedhoseini, S. M. (2015). A self-adaptive PSO for joint lot sizing and job shop scheduling with compressible process times. Applied Soft Computing,27, 137–147.

    Google Scholar 

  • Ken, Y., Myungryun, Y., & Takanori, Y. (2013). A proposal of real-time scheduling algorithm based on RMZL and schedulability analysis. Procedia Computer Science,24, 9–14.

    Google Scholar 

  • Kenneth, R. B. (2014). Minimizing earliness and tardiness costs in stochastic scheduling. European Journal of Operational Research,236(2), 445–452.

    Google Scholar 

  • Kundakcı, N., & Kulak, O. (2016). Hybrid genetic algorithms for minimizing makespan in dynamic job shop scheduling problem. Computers & Industrial Engineering,96, 31–51.

    Google Scholar 

  • Kusiak, A., & Li, W. (2010). Short-term prediction of wind power with a clustering approach. Renewable Energy,35(10), 2362–2369.

    Google Scholar 

  • Kusiak, A., Tang, F., & Xu, G. (2011). Multi-objective optimization of hvac system with an evolutionary computation algorithm. Energy,36(5), 2440–2449.

    Google Scholar 

  • Lee, K. K. (2008). Fuzzy rule generation for adaptive scheduling in a dynamic manufacturing environment. Applied Soft Computing,28(4), 1295–1304.

    Google Scholar 

  • Li, X., Peng, Z., Du, B., Guo, J., Xu, W., & Zhuang, K. (2017). Hybrid artificial bee colony algorithm with a rescheduling strategy for solving flexible job shop scheduling problems. Computers & Industrial Engineering,113, 10–26.

    Google Scholar 

  • Liu, B. J., Fan, Y. S., & Liu, Y. (2015). A fast estimation of distribution algorithm for dynamic fuzzy flexible job-shop scheduling problem. Computers & Industrial Engineering,87, 193–201.

    Google Scholar 

  • Liu, Z., Lu, L., & Qi, X. (2018). Cost allocation in rescheduling with machine unavailable period. European Journal of Operational Research,266(1), 16–28.

    Google Scholar 

  • Luo, H., Du, B., Huang, G. Q., Chen, H., & Li, X. (2013). Hybrid flow shop scheduling considering machine electricity consumption cost. International Journal of Production Economics,146(2), 423–439.

    Google Scholar 

  • Minsky, M. L. (1954). Theory of neural-analog reinforcement systems and its application to the brain model problem. Princeton: Princeton University.

    Google Scholar 

  • Mokhtari, H., & Hasani, A. (2017). An energy-efficient multi-objective optimization for flexible job-shop scheduling problem. Computers & Chemical Engineering,104, 339–352.

    Google Scholar 

  • Park, J., Mei, Y., Su, N., Chen, G., & Zhang, M. (2018). An investigation of ensemble combination schemes for genetic programming based hyper-heuristic approaches to dynamic job shop scheduling. Applied Soft Computing,63, 72–86.

    Google Scholar 

  • Park, S. C., Raman, N., & Shaw, M. J. (1997). Adaptive scheduling in dynamic flexible manufacturing systems, a dynamic rule selection approach. IEEE Transactions on Robotics and Automation,13(4), 486–502.

    Google Scholar 

  • Perron, O. (1907). Zur theorie der matrices. Mathematische Annalen,64(2), 248–263.

    Google Scholar 

  • Piroozfard, H., Wong, K. Y., & Wong, W. P. (2018). Minimizing total carbon footprint and total late work criterion in flexible job shop scheduling by using an improved multi-objective genetic algorithm. Resources, Conservation and Recycling,128, 267–283.

    Google Scholar 

  • Polydoros, A. S., & Nalpantidis, L. (2017). Survey of model-based reinforcement learning, applications on robotics. Journal of Intelligent and Robotic Systems,86(2), 153–173.

    Google Scholar 

  • Sadeghzadeh, M., Calvert, D., & Abdullah, H. A. (2015). Self-learning visual servoing of robot manipulator using explanation-based fuzzy neural networks and Q-learning. Journal of Intelligent and Robotic Systems,78(1), 83–104.

    Google Scholar 

  • Shahrabi, J., Adibi, M. A., & Mahootchi, M. (2017). A reinforcement learning approach to parameter estimation in dynamic job shop scheduling. Computers & Industrial Engineering,110, 75–82.

    Google Scholar 

  • Shen, X. N., & Yao, X. (2015). Mathematical modeling and multi-objective evolutionary algorithms applied to dynamic flexible job shop scheduling problems. Information Sciences,298, 198–224.

    Google Scholar 

  • Singh, S., Jaakkola, T., Littman, M. L., & Szepesvari, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning,38(3), 287–308.

    Google Scholar 

  • Tang, Z., Jiang, L. G., Zhou, J. Q., Li, K., & Li, K. (2015). A self-adaptive scheduling algorithm for reduce start time. Future Generation Computer Systems,43–44, 51–60.

    Google Scholar 

  • Theodoridis, S., & Koutroumbas, K. (2003). Pattern recognition (2nd ed.). San Diego: Academic Press.

    Google Scholar 

  • Tian, J., Wang, Q., Fu, R., & Yuan, J. (2016). Online scheduling on the unbounded drop-line batch machines to minimize the maximum delivery completion time. Theoretical Computer Science,617, 65–68.

    Google Scholar 

  • Wang, C., & Jiang, P. (2016). Manifold learning based rescheduling decision mechanism for recessive disturbances in RFID-driven job shops. Journal of Intelligent Manufacturing,29, 1–16.

    Google Scholar 

  • Wang, G. L., Lin, L., & Zhong, S. S. (2009). Clustering state membership-based Q-learning for dynamic scheduling. High Technology Letters,19(4), 428–433. (in Chinese).

    Google Scholar 

  • Wang, S. J., Sun, S., Zhou, B., & Xi, L. F. (2007). Q-Learning based dynamic single machine scheduling. Journal of Shanghai Jiaotong University,41(8), 1227–1232. (in Chinese).

    Google Scholar 

  • Wang, Y. C., & Usher, J. M. (2004). Learning policies for single machine job dispatching. Robotics and Computer-Integrated Manufacturing,20(6), 553–562.

    Google Scholar 

  • Wang, Y. C., & Usher, J. M. (2005). Application of reinforcement learning for agent-based production scheduling. Engineering Applications of Artificial Intelligence,18(1), 73–82.

    Google Scholar 

  • Wang, H. X., & Yan, H. S. (2016). An interoperable adaptive scheduling strategy for knowledgeable manufacturing based on SMGWQ-learning. Journal of Intelligent Manufacturing,27, 1085–1095.

    Google Scholar 

  • Wen, Z., & Sun, H. K. (2017). MATLAB intelligent algorithm. Beijing: Tsinghua University Press. (in Chinese).

    Google Scholar 

  • Wu, X., & Sun, Y. (2018). A green scheduling algorithm for flexible job shop with energy-saving measures. Journal of Cleaner Production,172, 3249–3264.

    Google Scholar 

  • Xiong, H., Fan, H., Jiang, G., & Li, G. (2017). A simulation-based study of dispatching rules in a dynamic job shop scheduling problem with batch release and extended technical precedence constraints. European Journal of Operational Research,257(1), 13–24.

    Google Scholar 

  • Yang, H. B., & Yan, H. S. (2009). An adaptive approach to dynamic scheduling in knowledgeable manufacturing cell. The International Journal of Advanced Manufacturing Technology,42, 312–320.

    Google Scholar 

  • Yao, S., Jiang, Z., Li, N., Zhang, H., & Geng, N. (2011). A multi-objective dynamic scheduling approach using multiple attribute decision making in semiconductor manufacturing. International Journal of Production Economics,130(1), 125–133.

    Google Scholar 

  • Zhang, Z. (2011). Proficient in MATLAB R2011a. Beijing: Beihang University Press. (in Chinese).

    Google Scholar 

  • Zhang, Z., Zheng, L., Li, N., Wang, W., Zhong, S., & Hu, K. (2012). Minimizing mean weighted tardiness in unrelated parallel machine scheduling with reinforcement learning. Computers & Operations Research,39(7), 1315–1324.

    Google Scholar 

Download references

Acknowledgements

This research is supported by the National Natural Science Foundation of China under Grant No. 51705260, and by Natural Science Foundation of the Higher Education Institution of Jiangsu Province under Grant No. 16KJD460005. I thank the editor-in-chief and the anonymous reviewers for their valuable comments and suggestions.

Author information

Authors and Affiliations

  1. School of Information and Control, Nanjing University of Information Science & Technology, Nanjing, 210044, Jiangsu, China

    Yu-Fang Wang

Authors
  1. Yu-Fang Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yu-Fang Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, YF. Adaptive job shop scheduling strategy based on weighted Q-learning algorithm. J Intell Manuf 31, 417–432 (2020). https://doi.org/10.1007/s10845-018-1454-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-018-1454-3

Keywords