Elsevier

Computers in Industry

Volume 130, September 2021, 103471
Computers in Industry

Planning for automatic product assembly using reinforcement learning

https://doi.org/10.1016/j.compind.2021.103471Get rights and content

Highlights

  • An intelligent model is proposed for automatic assembly operations of mechanical products.

  • Reinforcement learning is introduced for automatic positioning and fastening of product modules.

  • The proposed method is applied in assembly planning of an industrial machine.

Abstract

Assembly connects functional modules and components of products. The efficient and accurate assembly can improve performance of the product operation and maintenance. It is therefore essential to have an effective method for product assembly. Existing methods of the mechanical product assembly use mainly manual processes that rely on experience of operators. This paper proposes a reinforcement learning method to enable an automatic operation for improved efficiency and accuracy of the mechanical product assembly. A representation of the product assembly is proposed to build a machine learning model. The automatic assembly of product operations is planned by reinforcement learning agents. Constraints of assembly operations are considered to develop searching strategies of the maximum reward for the optimal solution of assembly operations. A quantitative method is proposed to measure efficiency of assembly operations based on the operation time. The proposed method has been applied in the assembly improvement of function modules of an industrial machine.

Introduction

Assembly connects product components and modules to ensure reliable operations of the product. The efficient and accurate assembly can increase the product variety, function performance, manufacturing and maintenance ability (Samy and Elmaraghy, 2012; Luo et al., 2016). An automatic process can improve efficiency and accuracy of operations in product assembly. Existing assembly operations of mechanical products are mainly manual processes. Positioning and fastening of mechanical components and modules in the product assembly depend on experience of operators, which has the low efficiency and inconsistence for the final product completion. Although robotic systems have been introduced in the product assembly, they are only applied in some specific applications such as the mass production of automobile assemblies where the assembly operation is pre-planned. A recent publication reviewed the research of 125 papers for the robotic process automation (Syed et al., 2020). It was concluded that although robots can work 24/7 non-stop and free human operators from repetitive and tedious tasks, a robot is still manually programmed for automating a simple process, which takes a few weeks to prepare the operation. Therefore, it is essential to improve efficiency and accuracy of the product assembly using an automatic process.

Computer-aided assembly methods have been developed for many years to improve efficiency and consistence of the product assembly plan. Design for assembly considers feasibility of the product assembly in the product design stage (Issaoui et al., 2017). Assembly planning searches an optimal operation process of the product assembly based on the design solution, such as sequence planning of product assembly (Ma et al., 2016), tool accessible analysis of assembly (Ma et al., 2017), and evaluation of the assembly efficiency (Soroush et al., 2014).

Different methods have been proposed for efficient planning of the product assembly using computation algorithms and heuristics (Liu et al., 2018). As the complex and multi-factors in assembly planning, the heuristics are commonly applied in assembly planning. The assembly affects operations of product components and modules in product manufacturing and maintenance (Chung and Peng, 2009). Research also considers the assembly improvement for product personalization, adaptability and diversity. One of the methods to improve the product variants and product personalization is to use the open-architecture product (OAP) (Ma et al., 2018). An OAP allows personalized functional modules added on the original product to meet changeable requirements of users in the product lifespan, which requires that the product is assembled easily when a personal module is replaced in the product for product adaptability, extensibility and sustainability (Peng et al., 2013). Therefore, an effective method for the OAP assembly is essential for the product variants and product personalization.

The feasibility analysis of the module connection and manipulation was introduced to improve efficiency of the module assembly (Soroush et al., 2014). Because various connections can cause complex operations in the product assembly, accessible tools are required in product assembly (Chung and Peng, 2009). The operating space evaluation of an assembly is important for the operation feasibility. The accessibility check in assembly also considers positioning and fastening components and modules.

However, these existing methods mainly focus on the process planning in the product design stage applied in scenarios of the design feasibility for assembly sequences, not for the implementation of assembly operations in manufacturing process to efficiently position and fasten components of the product based on the assembly plan. There is a lack of efficient planning methods for automatic assembly operations.

Reinforcement learning (RL) is one of machine learning (ML) methods through a trial-and-error process for maximizing reward (Xu et al., 2017). Neural networks can be used to approximate parameterized functions for gradient descent learning to discover non-linear mappings from the perceptual input of actions (Duan et al., 2016). RL provides a critical enabling technique for intelligent operations of products. It has been effectively applied in fields of robots (Mirowski et al., 2016), Industry 4.0 (Preuveneers and Ilie-Zudor, 2017) and intelligent transportation systems (Mousavi et al., 2017). RL emphasizes interactions of intelligent agents and environments for an effective tool to evaluate and improve operations of a system such as image understanding and natural language processing.

This paper proposes a RL method for automatic assembly planning in positioning and fastening product modules and components. In order to improve efficiency and accuracy of assembly operations, a description model of product assembly is established. A RL model is then built according to the assembly description model. The automatic assembly of product operations is planned by RL agents. A quantitative method is proposed to measure efficiency of assembly operations based on the process time. The proposed method has been used in automatic assembly planning for positioning and fastening function modules of an industrial paper-bag folding machine.

Following parts of this paper are organized as follows. Section 2 reviews the related research on the assembly modeling, planning and evaluation, ML and RL. In Section 3, the assembly description model, RL and evaluation methods are proposed. A case study is conducted in Section 4 to demonstrate the effectiveness of the proposed method, followed by conclusions and further work discussed in Section 5.

Section snippets

Product assembly modeling and process planning

Assembly connects functional modules and components of a product to perform functions of the product. An effective assembly process supports functions of product inputs and outputs for connections, transformations, and interactions between product modules and components (Ma et al., 2018). An efficient assembly method can effectively support the transmission of force, power and motion in the product application. Product adaptability can also be improved through changing functional modules in the

Assembly modeling

The assembly of a mechanical product connects function modules and components of the product using fasteners, which can be represented as follows.Assembly = Connection (Modules, Fasteners)where modules are formed by components based on the product function requirements in product design, fasteners such as bolts and screws are used to connect modules in a finish product.Module=(Topo,Geo,Parm,CFs)Fastener=(Type,Geo,Parm,CFs)CF=(Topo,Geo,Parm)where Topo and Geo are the topology and geometry of

Assembly operations based on reinforcement learning

This case study conducts a search process of RL to connect product function modules. CFPs are used to describe operations of the modules. In Fig. 3, three connection feature pairs (CFPs) are used to determine operations of modules. Using RL, the key to plan operations of the module assembly is to establish corresponding CFPs and build agent actions and rewards based on CFPs. For example, as shown in Fig. 3, operation steps of the assembly agent are as follows.

  • 1)

    Identifying modules of M1 and M2

Conclusions and further work

Automatic assembly is essential to improve the operation accuracy and efficiency in product development. This paper proposed an effective method to improve automatic operations of the mechanical product assembly using reinforcement learning. Main contributions of the work are as follows.

  • An intelligent model is proposed for automatic assembly operations of mechanical products.

  • Reinforcement learning is introduced for automatic positioning and fastening of product modules.

  • The proposed method is

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was partially supported by 1) National Key R&D Program of China (No: 2018YFB1701701) and Science and Technologies Program (No: 2017B090922008) for Jian Zhang, 2) Leading Talent Project of Guangdong Province for Peihua Gu, and 3) Canada NSERC Discovery Grant for Qingjin Peng.

References (74)

  • Y. Shao et al.

    A machine learning based global simulation data mining approach for efficient design changes

    Adv. Eng. Softw.

    (2018)
  • M. Sharp et al.

    A survey of the advancing use and development of machine learning in smart manufacturing

    J. Manuf. Syst.

    (2018)
  • R. Syed et al.

    Robotic process automation: contemporary themes and challenges

    Comput. Ind.

    (2020)
  • J. Yu et al.

    Knowledge-based deep belief network for machining roughness prediction and knowledge discovery

    Comput. Ind.

    (2020)
  • G. Zhao et al.

    Parallelization and optimization of spatial analysis for large scale environmental model data assembly

    Comput. Electron. Agric.

    (2012)
  • M.U. Ahmed et al.

    A machine learning approach to classify pedestrians’ events based on IMU and GPS

    Int. J. Artif. Intell.

    (2019)
  • S.R. Barde et al.

    Optimal preventive maintenance policy based on reinforcement learning of a fleet of military trucks

    J. Intell. Manuf.

    (2019)
  • A. Bhadra et al.

    Email Classification Using Supervised Learning Algorithms. Progress in Advanced Computing and Intelligent Engineering

    (2018)
  • Q. Cheng et al.

    A structure-based approach to evaluation product adaptability in adaptable design

    J. Mech. Sci. Technol.

    (2011)
  • C. Chern et al.

    Designing a decision-support system for new product sales forecasting

    Expert Syst. Appl.

    (2010)
  • A. Das et al.

    Learning cooperative visual dialog agents with deep reinforcement learning

    Proceedings of the IEEE International Conference on Computer Vision, 2951-2960

    (2017)
  • B. Deepak et al.

    Assembly sequence planning using soft computing methods: a review

    Proc. Inst. Mech. Eng. Part E J. Process. Mech. Eng.

    (2019)
  • A. Desai

    Ease of product assembly through a time-based design methodology

    Assem. Autom.

    (2019)
  • Y. Duan et al.

    Fast reinforcement learning via slow reinforcement learning

    arXiv preprint arXiv:161102779

    (2016)
  • S. El-Metwally et al.

    Next-generation sequence assembly: four stages of data processing and computational challenges

    PLoS Comput. Biol.

    (2013)
  • R. Ghazilla et al.

    Development of decision support system for fastener selection in product recovery oriented design

    Int. J. Adv. Manuf. Technol.

    (2014)
  • G. Gonzalez-Badillo et al.

    The development of a physics and constraint-based haptic virtual assembly system

    Assem. Autom.

    (2014)
  • S.K. Gupta et al.

    Intelligent assembly modeling and simulation

    Assem. Autom.

    (2001)
  • T. Haarnoja et al.

    Soft Actor-critic Algorithms and Applications

    (2018)
  • R. Hartl et al.

    Improving the surface quality of friction stir welds using reinforcement learning and Bayesian optimization

    Int. J. Adv. Manuf. Technol.

    (2020)
  • M. Hoefer et al.

    Automated manufacturing process selection during conceptual design

    J. Mech. Design.

    (2017)
  • C. Hu et al.

    Interface adaptability for an industrial painting machine

    Comput. Des. Appl.

    (2014)
  • L. Issaoui et al.

    Modelling and implementation of geometric and technological information for disassembly simulation in CAD environment

    Int. J. Adv. Manuf. Technol.

    (2017)
  • S. Jin et al.

    Design automation and optimization of assembly sequences for complex mechanical systems

    Int. J. Adv. Manuf. Technol.

    (2010)
  • Y. Jin et al.

    Data-driven evolutionary optimization: an overview and case studies

    IEEE Trans. Evol. Comput.

    (2019)
  • M.I. Jordan et al.

    Machine learning: trends, perspectives, and prospects

    Science

    (2015)
  • A. Juliani et al.

    Unity: a general platform for intelligent agents

    arXiv:1809.02627v2

    (2020)
  • Cited by (22)

    View all citing articles on Scopus
    View full text