Skip to main content

Reinforcement Learning for Segmented Manufacturing

  • Conference paper
  • First Online:
Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2023)

Abstract

The manufacturing of large components is, in comparison to small components, cost intensive. This is due to the sheer size of the components and the limited scalability in number of produced items. To take advantage of the effects of small component production we segment the large components into smaller parts and schedule the production of these parts on regular-sized machine tools. We propose to apply and adapt recent developments in reinforcement learning in combination with heuristics to efficiently solve the resulting segmentation and assignment problem. In particular, we solve the assignment problem up to a factor of 8 faster and only a few percentages less accurate than a classic solver from operations research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://developers.google.com/optimization/lp/mpsolver?hl=en.

References

  1. Afteni, C., Frumuşanu, G.: A review on optimization of manufacturing process performance. Int. J. Model. Optim. 7(3), 139–144 (2017). https://doi.org/10.7763/IJMO.2017.V7.573

    Article  MATH  Google Scholar 

  2. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34(6), 26–38 (2017). https://doi.org/10.1109/MSP.2017.2743240

    Article  ADS  Google Scholar 

  3. Bambach, P., et al.: LEAP - legged exploration of the aristarchus plateau. Europlanet Sci. Congr. (2022). https://doi.org/10.5194/epsc2022-856

    Article  MATH  Google Scholar 

  4. Bengio, Y., Lodi, A., Prouvost, A.: Machine learning for combinatorial optimization: a methodological tour d’horizon. Eur. J. Oper. Res. 290(2), 405–421 (2021). https://doi.org/10.1016/j.ejor.2020.07.063

    Article  MathSciNet  MATH  Google Scholar 

  5. Cattrysse, D.G., Van Wassenhove, L.N.: A survey of algorithms for the generalized assignment problem. Eur. J. Oper. Res. 60(3), 260–272 (1992). https://doi.org/10.1016/0377-2217(92)90077-M

    Article  MATH  Google Scholar 

  6. Chen, X., Tian, Y.: Learning to perform local rewriting for combinatorial optimization. Adv. Neural. Inf. Process. Syst. 32, 6281–6292 (2019)

    MATH  Google Scholar 

  7. Deza, A., Khalil, E.B.: Machine learning for cutting planes in integer programming: a survey. arXiv preprint arXiv:2302.09166 (2023). https://doi.org/10.48550/arXiv.2302.09166

  8. Dietterich, T.G., et al.: The MAXQ method for hierarchical reinforcement learning. In: ICML. vol. 98, pp. 118–126 (1998)

    Google Scholar 

  9. Hamzehi, S., Bogenberger, K., Franeck, P., Kaltenhäuser, B.: Combinatorial reinforcement learning of linear assignment problems. In: 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 3314–3321. IEEE (2019). https://doi.org/10.1109/ITSC.2019.8916920

  10. Jagdev, H., Browne, J., Keogh, J.: Manufacturing process optimisation—a survey of techniques. In: Proceedings of the Twenty-eighth International: Matador Conference, pp. 205–215. Springer (1990). https://doi.org/10.1007/978-1-349-10890-9_29

  11. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996). https://doi.org/10.1613/jair.301

    Article  MATH  Google Scholar 

  12. Khalil, E.B.: Machine learning for integer programming. In: IJCAI, pp. 4004–4005 (2016)

    Google Scholar 

  13. Kool, W., van Hoof, H., Welling, M.: Attention, learn to solve routing problems! In: International Conference on Learning Representations (2019). https://doi.org/10.48550/arXiv.1803.08475

  14. Kwon, Y.D., Choo, J., Kim, B., Yoon, I., Gwon, Y., Min, S.: POMO: policy optimization with multiple optima for reinforcement learning. Adv. Neural. Inf. Process. Syst. 33, 21188–21198 (2020)

    MATH  Google Scholar 

  15. Lee, M., Xiong, Y., Yu, G., Li, G.Y.: Deep neural networks for linear sum assignment problems. IEEE Wireless Commun. Lett. 7(6), 962–965 (2018). https://doi.org/10.1109/LWC.2018.2843359

    Article  MATH  Google Scholar 

  16. Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, J.E., Stoica, I.: Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118 (2018). https://doi.org/10.48550/arXiv.1807.05118

  17. Lin, X., Hou, Z.J., Ren, H., Pan, F.: Approximate mixed-integer programming solution with machine learning technique and linear programming relaxation. In: 2019 3rd International Conference on Smart Grid and Smart Cities (ICSGSC), pp. 101–107. IEEE (2019). https://doi.org/10.1109/ICSGSC.2019.00-11

  18. Mazyavkina, N., Sviridov, S., Ivanov, S., Burnaev, E.: Reinforcement learning for combinatorial optimization: a survey. Comput. Oper. Res. 134, 105400 (2021). https://doi.org/10.1016/j.cor.2021.105400

    Article  MathSciNet  MATH  Google Scholar 

  19. Mirhoseini, A., et al.: A graph placement methodology for fast chip design. Nature 594(7862), 207–212 (2021). https://doi.org/10.1038/s41586-022-04657-6

    Article  ADS  PubMed  MATH  Google Scholar 

  20. Nazari, M., Oroojlooy, A., Snyder, L., Takác, M.: Reinforcement learning for solving the vehicle routing problem. In: Advances in Neural Information Processing Systems, vol. 31 (2018)

    Google Scholar 

  21. Ouyang, L., et al.: Training language models to follow instructions with human feedback. Adv. Neural. Inf. Process. Syst. 35, 27730–27744 (2022)

    MATH  Google Scholar 

  22. Pan, Z., Wang, L., Wang, J., Lu, J.: Deep reinforcement learning based optimization algorithm for permutation flow-shop scheduling. IEEE Trans. Emerging Top. Comput. Intell. (2021). https://doi.org/10.1109/TETCI.2021.3098354

    Article  MATH  Google Scholar 

  23. Paul, N., Wirtz, T., Wrobel, S., Kister, A.: Multi-agent neural rewriter for vehicle routing with limited disclosure of costs. In: Presented at the Gamification and Multiagent Solutions Workshop within Tenth International Conference on Learning Representations, ICLR (2022). https://doi.org/10.48550/arXiv.2206.05990

  24. Refaei Afshar, R., Zhang, Y., Firat, M., Kaymak, U.: A state aggregation approach for solving knapsack problem with deep reinforcement learning. In: Pan, S.J., Sugiyama, M. (eds.) Proceedings of The 12th Asian Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 129, pp. 81–96 (2020)

    Google Scholar 

  25. Schnellhardt, T., Hemschik, R., Weiß, A., Schoesau, R., Hellmich, A., Ihlenfeldt, S.: Scalable production of large components by industrial robots and machine tools through segmentation. Front. Robot. AI 9 (2022). https://doi.org/10.3389/frobt.2022.1021755

  26. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017). https://doi.org/10.1038/nature24270

  27. Uriarte, L., et al.: Machine tools for large parts. CIRP Ann. 62(2), 731–750 (2013). https://doi.org/10.1016/j.cirp.2013.05.009

    Article  MATH  Google Scholar 

  28. Volk, A.A., et al.: AlphaFlow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning. Nat. Commun. 14(1), 1403 (2023). https://doi.org/10.1038/s41467-023-37139-y

    Article  ADS  PubMed  PubMed Central  MATH  Google Scholar 

  29. Weichert, D., Link, P., Stoll, A., Rüping, S., Ihlenfeldt, S., Wrobel, S.: A review of machine learning for the optimization of production processes. Int. J. Adv. Manuf. Technol. 1889–1902 (2019). https://doi.org/10.1007/s00170-019-03988-5

Download references

Acknowledgements

The research of N. Paul, T. Schnellhardt and M. Fetz was funded by the Fraunhofer lighthouse project “SWAP - Hierarchical swarms as production architecture with optimized utilization”. D. Hecker contributed as part of the Fraunhofer Center for Machine Learning within the Fraunhofer Cluster for Cognitive Internet Technologies. The work of T. Wirtz was supported by the Federal Ministry of Education and Research of Germany and the state of North-Rhine Westphalia as part of the Lamarr-Institute for Machine Learning and Artificial Intelligence. We would also like to thank the reviewers for their valuable feedback which improved the presentation of our work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nathalie Paul .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Paul, N., Kister, A., Schnellhardt, T., Fetz, M., Hecker, D., Wirtz, T. (2025). Reinforcement Learning for Segmented Manufacturing. In: Meo, R., Silvestri, F. (eds) Machine Learning and Principles and Practice of Knowledge Discovery in Databases. ECML PKDD 2023. Communications in Computer and Information Science, vol 2136. Springer, Cham. https://doi.org/10.1007/978-3-031-74640-6_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-74640-6_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-74639-0

  • Online ISBN: 978-3-031-74640-6

  • eBook Packages: Artificial Intelligence (R0)

Publish with us

Policies and ethics