Skip to main content

Advertisement

Springer Nature Link
Log in

Multi-objective optimization enabling CFRP energy-efficient milling based on deep reinforcement learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The expanding application of Carbon Fiber Reinforced Polymer (CFRP) in industries is drawing increasing attention to energy efficiency improvement and cost reducing during the secondary processing, particularly in milling. Machining parameter optimization is a practical and economical way to achieve this goal. However, the unclear milling mechanism and dynamic machining conditions of CFRP make it challenging. To fill this gap, this paper proposes a DRL-based approach that integrates physics-guided Transformer networks with Twin Delayed Deep Deterministic Policy Gradient (PGTTD3) to optimize CFRP milling parameters with multi-objectives. Firstly, a PG-Transformer-based CFRP milling energy consumption model is proposed, which modifies the existing De-stationary Attention module by integrating external physical variables to enhance modeling accuracy and efficiency. Secondly, a multi-objective optimization model considering energy consumption, milling time and machining cost for CFRP milling is formulated and mapped to a Markov Decision Process, and a reward function is designed. Thirdly, a PGTTD3 approach is proposed for dynamic parameter decision-making, incorporating a time difference strategy to enhance agent training stability and online adjustment reliability. The experimental results show that the proposed method reduces energy consumption, milling time and machining cost by 10.98%, 3.012%, and 14.56% in CFRP milling respectively, compared to the actual averages. The proposed algorithm exhibits excellent performance metrics when compared to state-of-the-art optimization algorithms, with an average improvement in optimization efficiency of over 20% and a maximum enhancement of 88.66%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data in this study are available on request from the corresponding author.

Abbreviations

\({Q}^{\prime}\)  :

Query matrix

\({K}^{\prime}\)  :

Key matrix

\({V}^{\prime}\)  :

Value matrix

\(\Psi\) :

Physical variables

\(SS\left(\cdot \right)\)  :

Self-supervised Self-compensation function

\({\Gamma }^{i}\) :

Optimization task set

\({E}_{tatol}\) :

Total energy (Wh)

\({E}_{\tau }\) :

Real-time energy consumption (Wh)

\({E}_{p}\) :

Total milling energy consumption (Wh)

\({T}_{p}\) :

Total milling process time (s)

\({EC}_{p}\) :

Total machining process cost ($)

\(n\) :

Spindle speed (rpm)

\({f}_{r}\) :

Feed rate (mm/min)

\({F}_{c}\) :

Cutting force (N)

\(HV\) :

Hypervolume

\(PF\) :

Pareto frontier

\(\alpha\) :

Coefficient for controlling guidance degree

\(\beta\) :

Coefficient for controlling guidance degree

\(\xi\) :

Deviation

\(\lambda\) :

Adjustment factor

\({k}_{1}\) :

The electric cost per unit energy ($/Wh)

\({k}_{2}\) :

The indirect cost per unit time ($/s)

\({g}_{j}^{i}\) :

The machining constraint functions

\({s}_{t}\) :

The environmental state at time \(t\)

\({a}_{t}\) :

Action

\({r}_{tr}\) :

Immediate-reward policy

\({r}_{sr}\) :

Sparsely distributed rewards

\({V}_{\pi }\left(s\right)\) :

An expected return

\({U}_{t}\) :

Long-term return

\({Q}^{*}(s,a)\) :

Action-value

\({Q}_{\pi }^{*}({a}_{t}|{s}_{t})\) :

Maximize expected return

\({Q}_{1}(\cdot |{{w}_{1}}^{{Q}_{1}})\),\({Q}_{2}(\cdot |{{w}_{2}}^{{Q}_{2}})\) :

Critic networks

\({\alpha }_{t}\) :

The learning rate

\({\delta }_{t}\) :

The TD error

\(\theta\) :

The actor network weight

\(w\) :

The critic networks weight

CFRP:

Carbon Fiber Reinforced Polymer

DRL:

Deep Reinforcement Learning

DQN:

Deep Q-Network

DPG:

Deterministic Policy Gradient

DDPG:

Deep Deterministic Policy Gradient

DBO:

Dung Beetle Optimizer

GWO:

Grey Wolf Optimization

GA:

Genetic Algorithm

MEC:

Milling Energy Consumption

MDP:

Markov Decision Process

MOGWO:

Multi-Objective Grey Wolf Optimizer

MOPSO:

Multi-Objective Particle Swarm Optimization

NSDBO:

Non-dominated Sorting Dung beetle optimizer

NSGA3:

Non-dominated Sorting Genetic Algorithm III

NSWOA:

Non-dominated Sorting Whale Optimization Algorithm

PG-Transformer:

Physics-Guided Transformer

PGTTD3:

Physics-Guided Transformer and TD3

PSO:

Particle Swarm Optimization

TD3:

Twin Delayed Deep Deterministic Policy Gradient

WOA:

Whale Optimization Algorithm

References

  1. Pimenov DY, Mia M, Gupta MK et al (2022) Resource saving by optimization and machining environments for sustainable manufacturing: A review and future prospects. Renew Sustain Energy Rev 166:112660

    Article  Google Scholar 

  2. Ozkan D, Gok MS, Karaoglanli AC (2020) Carbon fiber reinforced polymer (CFRP) composite materials, their characteristic properties, industrial application areas and their machinability Engineering Design Applications III. Struct Mater Proc 235–53

  3. Xie J, Cai W, Du Y et al (2021) Modelling approach for energy efficiency of machining system based on torque model and angular velocity. J Clean Prod 293:126249

    Article  Google Scholar 

  4. Zou F, Zhong B, Zhang H, et al (2022) Machinability and surface quality during milling CFRP laminates under dry and supercritical CO 2-based cryogenic conditions. Int J Precis Eng Manufact-Green Technol 1–17

  5. Tian C, Zhou G, Zhang J et al (2019) Optimization of cutting parameters considering tool wear conditions in low-carbon manufacturing environment. J Clean Prod 226:706–719

    Article  Google Scholar 

  6. Zhao X, Li C, Tang Y, et al (2024) Reinforcement Learning-Based Cutting Parameter Dynamic Decision Method Considering Tool Wear for a Turning Machining Process. Int J Precis Eng Manufact-Green Technol 1–18.

  7. Zhang X, Yu T, Dai Y et al (2020) Energy consumption considering tool wear and optimization of cutting parameters in micro milling process. Int J Mech Sci 178:105628

    Article  Google Scholar 

  8. Wang F, Bi G, Ning F (2021) Modeling of dynamic milling forces considering the interlaminar effect during milling multidirectional CFRP laminate. J Reinf Plast Compos 40(11–12):437–449

    Article  Google Scholar 

  9. Gao T, Zhang Y, Li C et al (2022) Fiber-reinforced composites in milling and grinding: machining bottlenecks and advanced strategies. Front Mech Eng 17(2):24

    Article  Google Scholar 

  10. Wang H, Pei Z, Cong W (2020) A mechanistic cutting force model based on ductile and brittle fracture material removal modes for edge surface grinding of CFRP composites using rotary ultrasonic machining. Int J Mech Sci 176:105551

    Article  Google Scholar 

  11. Song Y, Cao H, Zheng W, et al (2022) Cutting force modeling of machining carbon fiber reinforced polymer (CFRP) composites: A Review. Compos Struct 116096

  12. Soepangkat BOP, Norcahyo R, Effendi MK et al (2020) Multi-response optimization of carbon fiber reinforced polymer (CFRP) drilling using back propagation neural network-particle swarm optimization (BPNN-PSO). Eng Sci Technol Int J 23(3):700–713

    Google Scholar 

  13. Yaşar N, Korkmaz ME, Gupta MK et al (2021) A novel method for improving drilling performance of CFRP/Ti6AL4V stacked materials. Int J Adv Manufact Technol 117:653–673

    Article  Google Scholar 

  14. Zhang B, Li Y, Wang F et al (2023) Machining inclination selection method for surface milling of CFRP workpieces with low cutting-induced damage. Compos Struct 304:116495

    Article  Google Scholar 

  15. Deng J, Wang F, Fu R et al (2023) Prediction of time-varying dynamics and chatter stability analysis for surface milling of thin-walled curved CFRP workpiece. J Mater Process Technol 322:118186

    Article  Google Scholar 

  16. Zheng G, Chen W, Qian Q et al (2022) TCM in milling processes based on attention mechanism-combined long short-term memory using a sound sensor under different working conditions. Int J Hydromechatronics 5(3):243–259

    Article  Google Scholar 

  17. Peng B, Gao D, Wang M, et al (2024) 3D-STCNN: Spatiotemporal Convolutional Neural Network based on EEG 3D features for detecting driving fatigue. J Data Sci Intell Syst 2(1)

  18. Li C, Qu Z, Wang S (2024) An object detection approach with residual feature fusion and second-order term attention mechanism. CAAI Transact Intell Technol 9(2):411–424

    Article  Google Scholar 

  19. Simani S, Lam Y P, Farsoni S, et al (2023) Dynamic Neural Network Architecture Design for Predicting Remaining Useful Life of Dynamic Processes. J Data Sci Intell Syst

  20. Yang Z, Lu Y, Li W et al (2023) Effect of scanning spacing on the efficiency of CFRP hole-cutting with multi-pass scanning strategy by nanosecond laser. J Manuf Process 102:205–217

    Article  Google Scholar 

  21. Li W, He S, Mao X et al (2023) Multi-agent evolution reinforcement learning method for machining parameters optimization based on bootstrap aggregating graph attention network simulated environment. J Manuf Syst 67:424–438

    Article  Google Scholar 

  22. Lu F, Zhou G, Zhang C et al (2023) Energy-efficient multi-pass cutting parameters optimisation for aviation parts in flank milling with deep reinforcement learning. Robot Comput-Integr Manufact 81:102488

    Article  Google Scholar 

  23. He L, Chen F, Tian P et al (2024) An improved energy management strategy for hybrid electric powered aircraft based on deep reinforcement learning. Aerosp Sci Technol 149:109137

    Article  Google Scholar 

  24. Neto WAF, Cavalcante CAV, Do P (2024) Deep reinforcement learning for maintenance optimization of a scrap-based steel production line. Reliab Eng Syst Saf 249:110199

    Article  Google Scholar 

  25. Wang H, Pei Z, Cong W (2020) A feeding-directional cutting force model for end surface grinding of CFRP composites using rotary ultrasonic machining with elliptical ultrasonic vibration. Int J Mach Tools Manuf 152:103540

    Article  Google Scholar 

  26. Zhang H, Zhu P, Liu Z et al (2021) Research on prediction method of mechanical properties of open-hole laminated plain woven CFRP composites considering drilling-induced delamination damage. Mech Adv Mater Struct 28(24):2515–2530

    Article  Google Scholar 

  27. Zhang S, Jiao F, Wang X et al (2021) Modeling of cutting forces in helical milling of unidirectional CFRP considering carbon fiber fracture. J Manuf Process 68:1495–1508

    Article  Google Scholar 

  28. Callisaya ES, De SampaioAlves MC, Kondo MY et al (2023) Analysis of power consumption during the machining of epoxy based CFRP. Mater Today Commun 37:106993

    Article  Google Scholar 

  29. Song Y, Cao H, Qu D et al (2022) Specific cutting energy optimization of CF/PEEK milling considering size effect. Int J Mech Sci 232:107618

    Article  Google Scholar 

  30. Hameed AS, Mallick B, Mondal K et al (2023) Neural network (NN) based modelling and Multi-objective Swarm Algorithm (MSA) optimization of CNC milling operation. Mater Today: Proc

  31. Darshan S, Desai K, Bhattacharyya A (2023) Neural Network-based Model Parameter Estimation for End Milling of Carbon Fiber Reinforced Polymer (CFRP) Composites. Procedia CIRP 117:365–370

    Article  Google Scholar 

  32. Shunhu H, Feng M, Qingshan G et al (2024) Efficient low-carbon manufacturing for CFRP composite machining based on deep networks. Int J Prod Res 1–12

  33. Aggogeri F, Merlo A, Pellegrini N (2020) Modeling the thermo-mechanical deformations of machine tool structures in CFRP material adopting data-driven prediction schemes. Mechatronics 71:102436

    Article  Google Scholar 

  34. Huang W, Zhang X, Wu C et al (2022) Tool wear prediction in ultrasonic vibration-assisted drilling of CFRP: A hybrid data-driven physics model-based framework. Tribol Int 174:107755

    Article  Google Scholar 

  35. Azimy H, Azimy N, MeghdadiIsfahani AH et al (2023) Analysis of thermal performance and ultrasonic wave power variation on heat transfer of heat exchanger in the presence of nanofluid using the artificial neural network: experimental study and model fitting. J Therm Anal Calorimetry 148(16):8009–23

    Article  Google Scholar 

  36. Kumar P, Sharma M, Singh G et al (2023) Experimental investigations of machining parameters on turning of Ti6Al4V: optimisation using Taguchi method. Int J Interact Des Manufact (IJIDeM) 1–11

  37. Tran VT, Le MH, Vo MT et al (2023) Optimization design for die-sinking EDM process parameters employing effective intelligent method. Cogent Engineering 10(2):2264060

    Article  Google Scholar 

  38. Işik AT, Çakiroğlu R, Günay M (2023) Multiresponse optimization of performance indicators through Taguchi-grey relational analysis in EDM of cemented carbide. CIRP J Manufact Sci Technol 41:490–500

    Article  Google Scholar 

  39. Paulavičius R, Stripinis L, Sutavičiūtė S et al (2023) A novel greedy genetic algorithm-based personalized travel recommendation system. Expert Syst Appl 230:120580

    Article  Google Scholar 

  40. Ayar M, Isazadeh A, Gharehchopogh FS et al (2023) NSICA: Multi-objective imperialist competitive algorithm for feature selection in arrhythmia diagnosis. Comput Biol Med 161:107025

    Article  Google Scholar 

  41. Huang S, Fu Z, Liu C et al (2023) Multi-objective optimization of fiber laser cutting quality characteristics of glass fiber reinforced plastic (GFRP) materials. Opt Laser Technol 167:109720

    Article  Google Scholar 

  42. Li B, Tian X, Zhang M (2022) Modeling and multi-objective optimization method of machine tool energy consumption considering tool wear. Int J Precis Eng Manufact-Green Technol 1–15

  43. Wang C-N, Yang F-C, Nguyen VTT et al (2022) CFD analysis and optimum design for a centrifugal pump using an effectively artificial intelligent algorithm. Micromachines 13(8):1208

    Article  Google Scholar 

  44. Bousnina K, Hamza A, Yahia NB (2023) An integration of PSO-ANN and ANFIS hybrid models to predict surface quality, cost, and energy (QCE) during milling of alloy 2017A. J Eng Res

  45. Bhaskar M, Balaji V, Narendranath S et al (2023) Machining Parameter Optimization of Wire Electrical Discharge Machining for Ni50. 3Ti29. 7Hf20 Alloy Using TOPSIS and Grey Wolf Optimization Technique. J Mater Eng Perform 1–12

  46. Zhong Y, Wang J, Xu J et al (2023) Data-driven width spread prediction model improvement and parameters optimization in hot strip rolling process. Appl Intell 53(21):25752–25770

    Article  Google Scholar 

  47. Cheng B, Wang L, Tan Q et al (2024) A deep reinforcement learning hyper-heuristic to solve order batching problem with mobile robots. Appl Intell 1–23

  48. Gao M, Feng X, Yu H et al (2023) An efficient evolutionary algorithm based on deep reinforcement learning for large-scale sparse multiobjective optimization. Appl Intell 53(18):21116–21139

    Article  Google Scholar 

  49. Zeng L, Liu Q, Shen S et al (2023) Improved Double Deep Q Network-Based Task Scheduling Algorithm in Edge Computing for Makespan Optimization. Tsinghua Sci Technol 29(3):806–817

    Article  Google Scholar 

  50. Ahmed MH, Abohussien A, El-Shafei A et al (2023) Active control of flexible rotors using deep reinforcement learning with application of multi-actor-critic deep deterministic policy gradient. Eng Appl Artif Intell 124:106593

    Article  Google Scholar 

  51. Fan Y, Dong H, Zhao X et al (2024) Path-Following Control of Unmanned Underwater Vehicle Based on an Improved TD3 Deep Reinforcement Learning. IEEE Transact Control Syst Technol

  52. Liu Y, Wu H, Wang J et al (2022) Non-stationary transformers: Exploring the stationarity in time series forecasting. Adv Neural Inf Process Syst 35:9881–9893

    Google Scholar 

  53. Shang K, Ishibuchi H, He L et al (2020) A survey on the hypervolume indicator in evolutionary multiobjective optimization. IEEE Trans Evol Comput 25(1):1–20

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NO. 51975432, 52375508, 52075396), ‘The 14th Five Year Plan’ Hube Province advantaged characteristic disciplines (groups) project of Wuhan University of Science and Technology (2023B0405), The Logistics Education Reform and Research Project (NO. JZW2023252), and Major Project of Hubei Province Science and Technology (NO. 2023BCA006).

Author information

Authors and Affiliations

  1. Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081, China

    Meihang Zhang, Hua Zhang, Lin Zhang & Zhigang Jiang

  2. Key Laboratory of Metallurgical Equipment and Control Technology, Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081, China

    Meihang Zhang, Lin Zhang & Zhigang Jiang

  3. Precision Manufacturing Research Institute of Wuhan University of Science and Technology, Wuhan, 430081, China

    Hua Zhang

  4. School of Automotive and Traffic Engineering, Wuhan University of Science and Technology, Wuhan, 430065, China

    Wei Yan

Authors
  1. Meihang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhigang Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Meihang Zhang: Conceptualization, Methodology, Software, Formal analysis, Data curation, Original draft, Review & editing; Hua Zhang: Conceptualization, Resources, Supervision, Funding acquisition; Wei Yan: Conceptualization, Methodology, Validation, Formal analysis, Resources, Review & editing, Supervision, Funding acquisition; Lin Zhang: Validation, Data curation; Zhigang Jiang: Resources, Funding acquisition.

Corresponding author

Correspondence to Wei Yan.

Ethics declarations

Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, M., Zhang, H., Yan, W. et al. Multi-objective optimization enabling CFRP energy-efficient milling based on deep reinforcement learning. Appl Intell 54, 12531–12557 (2024). https://doi.org/10.1007/s10489-024-05800-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05800-8

Keywords