Abstract
To address the uncertainty of production environment in knowledgeable manufacturing system, an interoperable knowledgeable dynamic-scheduling system based on multi-agent is designed, wherein an adaptive scheduling mechanism based on the state membership grade weighted Q-learning (known as SMGWQ-learning) is proposed for guiding the equipment agent to select proper scheduling strategy in a dynamic environment. To avoid the side effect of large state space and minimize errors between the clustering and real states, the state membership grade, defined as weight coefficients, is incorporated into the weighted Q-value update so that several Q-values can be updated simultaneously in an iteration. Results from our convergence analysis and simulation experiments show the effectiveness of the proposed strategy that endows the scheduling system with higher intelligence, interoperability and adaptability to environmental changes by self-learning.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aissani, N., Bekrar, A., Trentesaux, D., & Beldjilali, B. (2012). Dynamic scheduling for multi-site companies: A decisional approach based on reinforcement multi-agent learning. Journal of Intelligent Manufacturing, 23(6), 2513–2529.
Aydin, M. E., & Öztemel, E. (2003). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33(2), 169–178.
Belhe, U., & Kusiak, A. (1997). Dynamic scheduling of design activities with resource constraints. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 27(1), 847–861.
Bourenane, M., Mellouk, A., & Benhamamouch, D. (2009). State-dependent packet scheduling for QoS routing in a dynamically changing environment. Telecommunication Systems, 42(3–4), 249–261.
Branke, J., & Mattfeld, D. C. (2005). Anticipation and flexibility in dynamic scheduling. International Journal of Production Research, 43(15), 3103–3129.
Chou, F. D., Chang, P. C., & Wang, H. M. (2006). A hybrid genetic algorithm to minimize makespan for the single batch machine dynamic scheduling problem. International Journal of Advanced Manufacturing Technology, 31(3/4), 350–359.
Cowling, P. I., Ouelhadj, D., & Petrovic, S. (2003). A multi-agent architecture for dynamic scheduling of steel hot rolling. Journal of Intelligent Manufacturing, 14(5), 457–470.
Creighton, D. C., & Nahavandi, S. (2002). The application of a reinforcement learning agent to a multi-product manufacturing facility. Proceedings of IEEE International Conference on Industrial Technology (pp. 1229–1234). Bangkok: Thailand.
Csáji, B. C., Monostori, L., & Kádár, B. (2006). Reinforcement learning in a distributed market-based production control system. Advanced Engineering Informatics, 20(3), 279–88.
Erenay, F. S., Sabuncuoglu, I., Toptal, A., & Tiwari, M. K. (2010). New solution methods for single machine bicriteria scheduling problem: Minimization of average flowtime and number of tardy jobs. European Journal of Operational Research, 201, 89–98.
Hong, J., & Prabhu, V. V. (2004). Distributed reinforcement learning control for batch sequencing and sizing in just-in-time manufacturing systems. Applied Intelligence, 20(1), 71–87.
Huang, G. Q., Zhang, Y. F., Chen, X., & Newman, S. T. (2008). RFID-enabled real-time wireless manufacturing for adaptive assembly planning and control. Journal of Intelligent Manufacturing, 19(6), 710–713.
Kusiak, A., & He, D. W. (1998). Design for agility: A scheduling perspective. Robotics and Computer-Integrated Manufacturing, 14, 415–427.
Lau, J. S. K., Huang, G. Q., Mak, K. L., & Liang, L. (2006). Agent-based modeling of supply chains for distributed scheduling. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 36(5), 847–861.
Luo, H., Huang, G. Q., Zhang, Y. F., Dai, Q. Y., & Chen, X. (2009). Two-stage hybrid batching flowshop scheduling with blocking and machine availability constraints using genetic algorithm. Robotics and Computer-Integrated Manufacturing, 25, 962–971.
Nie, L., Gao, L., Li, P. G., & Li, X. Y. (2013). A GEP-based reactive scheduling policies constructing approach for dynamic flexible job shop scheduling problem with job release dates. Journal of Intelligent Manufacturing, 24(4), 763–774.
Sabar, M., Montreuil, B., & Frayret, J. M. (2012). An agent-based algorithm for personnel shift-scheduling and rescheduling in flexible assembly lines. Journal of Intelligent Manufacturing, 23(6), 2623–2634.
Shnits, B., Rubinovitz, J., & Sinreich, D. (2004). Multicriteria dynamic scheduling methodology for controlling a flexible manufacturing system. International Journal of Production Research, 42(17), 3457–3472.
Singh, S., Jaakkola, T., Littman, M. L., & Szepesvari, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 39(3), 287–308.
Singh, S. S., Tadić, V. B., & Doucet, A. (2007). A policy gradient method for semi-Markov decision processes with application to call admission control. European Journal of Operational Research, 178(3), 808–18.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Theodoridis, S., & Koutroumbas, K. (2003). Pattern recognition (2nd ed.). San Diego: Academic Press.
Wang, G. L., Zhong, S. S., & Lin, L. (2009). Clustering state membership-based Q-learning for dynamic scheduling. Chinese High Technology Letters, 19(4), 428–433. (in Chinese).
Wang, Z. (2010). Problem-oriented knowledge representing, organizing and inference for production operation and management. Technical Report: School of Automation, Southeast University, Nanjing.
Watkins, C., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3–4), 279–292.
Yan, H. S. (2006). A new complicated knowledge representation approach based on knowledge meshes. IEEE Transactions on Knowledge and Data Engineering, 18(1), 47–62.
Yan, H. S., & Liu, F. (2001). Knowledgeable manufacturing system—A new kind of advanced manufacturing system. Computer Integrated Manufacturing Systems, 7(8), 7–11. (in Chinese).
Yan, H. S., & Ma, K. P. (2011). Competitive diffusion process of repurchased products in knowledgeable manufacturing. European Journal of Operational Research, 208(3), 243–252.
Yang, H. B., & Yan, H. S. (2009). An adaptive approach to dynamic scheduling in knowledgeable manufacturing cell. International Journal of Advanced Manufacturing Technology, 42(3–4), 312–320.
Zandieh, M., & Karimi, N. (2011). An adaptive multi-population genetic algorithm to solve the multi-objective group scheduling problem in hybrid flexible flowshop with sequence-dependent setup times. Journal of Intelligent Manufacturing, 22(6), 979–989.
Zhang, W. J., Freiheit, T., & Yang, H. S. (2005). Dynamic scheduling in flexible assembly system based on timed Petri nets model. Robotics & Computer-Integrated Manufacturing, 21(6), 550–558.
Zhao, R. Q. (1991). Knowledge representation and reasoning. Beijing: China Meteorological Press. (in Chinese).
Acknowledgments
This research is supported by a key program of the National Natural Science Foundation of China under Grant 60934008, by the Scientific Research Foundation of Graduate School of Southeast University under grant YBJJ1215, and by the Jiangsu Provincial Program for Scientific Innovation by College Graduates under grant CXLX11_0118. We thank the Editor-in-Chief & Professor Andrew Kusiak, the two reviewers and Professor Li Lu for their valuable comments and suggestions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, HX., Yan, HS. An interoperable adaptive scheduling strategy for knowledgeable manufacturing based on SMGWQ-learning. J Intell Manuf 27, 1085–1095 (2016). https://doi.org/10.1007/s10845-014-0936-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10845-014-0936-1