skip to main content
10.1145/3299815.3314451acmconferencesArticle/Chapter ViewAbstractPublication Pagesacm-seConference Proceedingsconference-collections
short-paper

Applying Reinforcement Learning to Plan Manufacturing Material Handling Part 1: Background and Formal Problem Specification

Published: 18 April 2019 Publication History

Abstract

Applying machine learning to improve the efficiency of complex manufacturing processes, particularly logistics and material handling, can be a challenging problem. The interconnectedness of the multiple components that compose such processes and the typically large number of variables required to specify procedures and plans within those processes combine to make it very difficult to map the details of real-world manufacturing processes to an abstract mathematical representation suitable for machine learning methods. In this paper, we report on the application of machine learning methods, in particular reinforcement learning, to generate increasingly efficient plans for material handling to satisfy temporally varying product demands in a representative manufacturing facility. The essential steps in the research included defining a formal representation of a realistically complex material handling plan, defining a set of suitable two-stage plan change operators as reinforcement learning actions, implementing a simulation-based multi-objective reward function that considers multiple components of material handling costs, and abstracting the many possible material handling plans into a state set small enough to enable reinforcement learning. Extensive experimentation with multiple starting plans showed that the reinforcement learning process could consistently reduce the material handling plans' costs over time. This work may be one of the first applications of reinforcement learning with a multi-objective reward function to a realistically complex material handling process. This paper first provides brief background on material handling and reinforcement learning. It then details the formal mathematical representation of a realistic material handling plan for a manufacturing facility and specifies the reinforcement learning operators to be applied to the plan representation. This paper (part 1 of 2) is the first of two companion papers; the second describes the experimentation and results [1].

References

[1]
S. Govindaiah and M. D. Petty. 2019. Applying Reinforcement Learning to Plan Manufacturing Material Handling Part 2: Experimentation and Results. In 2019 ACM Southeast Conference (ACMSE 2019), April 18-20, 2019, Kennesaw, GA, USA. ACM, New York, NY, USA.
[2]
I. N. Pujawan and A. U. Smart. 2012. Factors Affecting Schedule Instability in Manufacturing Companies. International Journal of Production Research, vol. 50, no. 8, pp. 2252--2266.
[3]
J. Banks, J. S. Carson, B. L. Nelson, and D. M. Nicol. 2010. Discrete Event System Simulation (5th. ed.). Prentice Hall, Upper Saddle River, NJ.
[4]
M. P. Stephens and F. E. Meyers. 2013. Manufacturing Facilities Design and Material Handling (5th. ed.). Purdue University Press, West Lafayette, IN.
[5]
R. S. Sutton and A. G. Barto. 1998. Reinforcement Learning: An Introduction. MIT press, Cambridge, MA.
[6]
Y. C. Wang and J. M. Usher. 2007. A Reinforcement Learning Approach for Developing Routing Policies in Multi-agent Production Scheduling. The International Journal of Advanced Manufacturing Technology, vol. 33, no. 3-4, pp. 323--333.
[7]
C. Chen, B. Xia, B. H. Zhou, and L. Xi. 2015. A Reinforcement Learning Based Approach for a Multiple-load Carrier Scheduling Problem. Journal of Intelligent Manufacturing, vol. 26, no. 6, pp. 1233--1245.
[8]
P. Vamplew, R. Dazeley, A. Berry, R. Issabekov and E. Dekker. 2011. Empirical Evaluation Methods for Multiobjective Reinforcement Learning Algorithms. Machine learning, vol. 84, no. 1-2, pp. 51--80.
[9]
W. Zhang and T. G. Dietterich. 2000. Solving Combinatorial Optimization Tasks by Reinforcement Learning: A General Methodology Applied to Resource-constrained Scheduling. Journal of Artificial Intelligence Research, vol. 1, pp. 1--38.
[10]
R. Rooeinfar, S. Raissi, and V. Ghezavati. 2018. Stochastic Flexible Flow Shop Scheduling Problem with Limited Buffers and Fixed Interval Preventive Maintenance: A Hybrid Approach of Simulation and Metaheuristic Algorithms. SIMULATION, p. 0037549718809542, 2018.
[11]
S. Govindaiah and M. D. Petty. 2019. A Discrete Event Simulation-based Multiobjective Reinforcement Learning Reward Function for Optimizing Manufacturing Material Handling. In Proceedings of the 2019 Simulation Innovation Workshop, Orlando FL, February 2019.
[12]
P. C. Fishburn. 1967. Additive Utilities with Incomplete Product Sets: Application to Priorities and Assignments. Operations Research, vol. 15, no. 3, pp. 537--542.

Cited By

View all
  • (2022)Identification of Machine Learning Relevant Energy and Resource Manufacturing Efficiency LeversSustainability10.3390/su14231561814:23(15618)Online publication date: 24-Nov-2022
  • (2022)Reinforcement learning for logistics and supply chain management: Methodologies, state of the art, and future opportunitiesTransportation Research Part E: Logistics and Transportation Review10.1016/j.tre.2022.102712162(102712)Online publication date: Jun-2022
  • (2022)ROS-based architecture for fast digital twin development of smart manufacturing robotized systemsAnnals of Operations Research10.1007/s10479-022-04759-4322:1(75-99)Online publication date: 7-Jun-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ACMSE '19: Proceedings of the 2019 ACM Southeast Conference
April 2019
295 pages
ISBN:9781450362511
DOI:10.1145/3299815
  • Conference Chair:
  • Dan Lo,
  • Program Chair:
  • Donghyun Kim,
  • Publications Chair:
  • Eric Gamess
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Machine Learning
  2. Material Handling
  3. Multi-Objective Learning
  4. Planning
  5. Reinforcement Learning

Qualifiers

  • Short-paper
  • Research
  • Refereed limited

Conference

ACM SE '19
Sponsor:
ACM SE '19: 2019 ACM Southeast Conference
April 18 - 20, 2019
GA, Kennesaw, USA

Acceptance Rates

Overall Acceptance Rate 502 of 1,023 submissions, 49%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)2
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Identification of Machine Learning Relevant Energy and Resource Manufacturing Efficiency LeversSustainability10.3390/su14231561814:23(15618)Online publication date: 24-Nov-2022
  • (2022)Reinforcement learning for logistics and supply chain management: Methodologies, state of the art, and future opportunitiesTransportation Research Part E: Logistics and Transportation Review10.1016/j.tre.2022.102712162(102712)Online publication date: Jun-2022
  • (2022)ROS-based architecture for fast digital twin development of smart manufacturing robotized systemsAnnals of Operations Research10.1007/s10479-022-04759-4322:1(75-99)Online publication date: 7-Jun-2022
  • (2022)A practical guide to multi-objective reinforcement learning and planningAutonomous Agents and Multi-Agent Systems10.1007/s10458-022-09552-y36:1Online publication date: 1-Apr-2022
  • (2021)Applying reinforcement learning to plan manufacturing material handlingDiscover Artificial Intelligence10.1007/s44163-021-00003-31:1Online publication date: 22-Sep-2021
  • (2019)Applying Reinforcement Learning to Plan Manufacturing Material Handling Part 2Proceedings of the 2019 ACM Southeast Conference10.1145/3299815.3314427(16-23)Online publication date: 18-Apr-2019
  • (undefined)Reinforcement Learning for Logistics and Supply Chain Management: Methodologies, State of the Art, and Future OpportunitiesSSRN Electronic Journal10.2139/ssrn.3935816

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media