Abstract
Service composition is an indispensable technology in the cloud manufacturing process to ensure the smooth execution of tasks. To implement effective and accurate service composition strategies, many researchers choose to use Meta-heuristics algorithms with strong optimization capabilities. However, as users’ demand of personalized products increasing, dynamic service composition is essential. Meta-heuristics algorithms lack dynamic adaptability, so they are not suitable for solving complex and dynamic service composition problems. Deep Reinforcement Learning (DRL) algorithm is difficult to reach a stable state, when the hyper-parameters and rewards in the algorithm are not properly designed. To solve these problems, we propose a Multi-strategy Deep Reinforcement Learning (DRL) algorithm, named Multi-D3QN, which combines the basic DQN algorithm, the dueling architecture, the double estimator and the prioritized replay mechanism. Meanwhile, we add some strategies such as instant reward, the ɛ-greedy policy and a heuristic strategy to ensure better performance of the algorithm in dynamic environment. Experiments show that our proposed method not only adapt to the dynamic environment, but also obtain a better solution.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Mourad, M.H., Nassehi, A., Schaefer, D., Newman, S.T.: Assessment of interoperability in cloud manufacturing. Robotics and Computer-Integrated Manufacturing 61, (2020)
Bouzary, H., Chen, F.F.: A classification-based approach for integrated service matching and composition in cloud manufacturing. Robotics and Computer-Integrated Manufacturing 66, (2020)
Zhang, L., et al.: Cloud manufacturing: a new manufacturing paradigm. Enterprise Information Systems 8, 167–187 (2014)
Yang, Y., Yang, B., Wang, S., Jin, T., Li, S.: An enhanced multi-objective grey wolf optimizer for service composition in cloud manufacturing. Applied Soft Computing 87, (2020)
Yang, Y., Yang, B., Wang, S., Liu, W., Jin, T.: An Improved Grey Wolf Optimizer Algorithm for Energy-Aware Service Composition in Cloud Manufacturing. The International Journal of Advanced Manufacturing Technology 105(7–8), 3079–3091 (2019). https://doi.org/10.1007/s00170-019-04449-9
Akbaripour, H., Houshmand, M., van Woensel, T., Mutlu, N.: Cloud manufacturing service selection optimization and scheduling with transportation considerations: mixed-integer programming models. The International Journal of Advanced Manufacturing Technology 95(1–4), 43–70 (2017). https://doi.org/10.1007/s00170-017-1167-3
Liu, Y., Wang, L., Wang, X.V., Xu, X., Zhang, L.: Scheduling in cloud manufacturing: state-of-the-art and research challenges. Int. J. Prod. Res. 57, 4854–4879 (2019)
Lartigau, J., Xu, X., Nie, L., Zhan, D.: Cloud manufacturing service composition based on QoS with geo-perspective transportation using an improved Artificial Bee Colony optimisation algorithm. Int. J. Prod. Res. 53, 4380–4404 (2015)
Que, Y., Zhong, W., Chen, H., Chen, X., Ji, X.: Improved adaptive immune genetic algorithm for optimal QoS-aware service composition selection in cloud manufacturing. Int. J. Adv. Manuf. Technol. 96(9–12), 4455–4465 (2018). https://doi.org/10.1007/s00170-018-1925-x
Wang, H., et al.: Adaptive and large-scale service composition based on deep reinforcement learning. Knowl.-Based Syst. 180, 75–90 (2019)
Liang, H., Wen, X., Liu, Y., Zhang, H., Zhang, L., Wang, L.: Logistics-involved QoS-aware service composition in cloud manufacturing with deep reinforcement learning. Robot. Comput. Integr. Manuf. 67, 101991 (2021)
Quan, L., Wang, Z.-L., Liu, X.: A real-time subtask-assistance strategy for adaptive services composition. IEICE Trans. Inf. Syst. E101D, 1361–1369 (2018)
Zhou, J., Yao, X.: A hybrid artificial bee colony algorithm for optimal selection of QoS-based cloud manufacturing service composition. Int. J. Adv. Manuf. Technol. 88(9–12), 3371–3387 (2016). https://doi.org/10.1007/s00170-016-9034-1
Fazeli, M.M., Farjami, Y., Nickray, M.: An ensemble optimisation approach to service composition in cloud manufacturing. Int. J. Comput. Integr. Manuf. 32, 83–91 (2018)
Yu, L., Zhou, J., Wei, F., Gao, Y., Yang, B., Zhu, H.: Web Service Composition Based on Reinforcement Learning (2015)
Wang, H., Chen, X., Wu, Q., Yu, Q., Zheng, Z., Bouguettaya, A.: Integrating on-policy reinforcement learning with multi-agent techniques for adaptive service composition. In: Franch, X., Ghose, A.K., Lewis, G.A., Bhiri, S. (eds.) ICSOC 2014. LNCS, vol. 8831, pp. 154–168. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45391-9_11
Wang, H., Gu, M., Yu, Q., Fei, H., Li, J., Tao, Y.: Large-scale and adaptive service composition using deep reinforcement learning. In: Maximilien, M., Vallecillo, A., Wang, J., Oriol, M. (eds.) ICSOC 2017. LNCS, vol. 10601, pp. 383–391. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69035-3_27
Yuan, M., Zhou, Z., Cai, X., Sun, C., Gu, W.: Service composition model and method in cloud manufacturing. Robot. Comput. Integr. Manuf. 61, 101840 (2020)
Liu, Z.Z., Song, C., Chu, D.H., Hou, Z.W., Peng, W.P.: An approach for multipath cloud manufacturing services dynamic composition. Int. J. Intell. Syst. 32, 371–393 (2017)
Zhou, J., Yao, X.: Multi-population parallel self-adaptive differential artificial bee colony algorithm with application in large-scale service composition for cloud manufacturing. Appl. Soft Comput. 56, 379–397 (2017)
Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., Freitas, N.: Dueling network architectures for deep reinforcement learning. In: Maria Florina, B., Kilian, Q.W. (eds.) Proceedings of The 33rd International Conference on Machine Learning, vol. 48, pp. 1995–2003. PMLR, Proceedings of Machine Learning Research (2016)
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. In: AAAI (2016)
Schaul, T., Quan, J., Antonoglou, I.: Prioritized experience replay, arXiv preprint arXiv:1511.05952 (2015)
Acknowledgments
This work was supported in part by the National Key Research and Development Project under grant 2019YFB1706101, the Natural Science Foundation of Chongqing, China (No. cstc2020jcyj-msxmX0900).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Zeng, J., Yao, J., Yu, Y., Wu, Y. (2021). Multi-D3QN: A Multi-strategy Deep Reinforcement Learning for Service Composition in Cloud Manufacturing. In: Gao, H., Wang, X. (eds) Collaborative Computing: Networking, Applications and Worksharing. CollaborateCom 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 407. Springer, Cham. https://doi.org/10.1007/978-3-030-92638-0_14
Download citation
DOI: https://doi.org/10.1007/978-3-030-92638-0_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92637-3
Online ISBN: 978-3-030-92638-0
eBook Packages: Computer ScienceComputer Science (R0)