Skip to main content
SpringerLink
Log in

Method for monitoring and controlling penetration of complex groove welding based on online multi-modal data

  • Published:
Journal of Intelligent Manufacturing Aims and scope Submit manuscript

Abstract

In industrial production, there are problems of hand polishing error and the thermal deformation of weldment, resulting in unstable groove welding. In this article, we propose a model for online monitoring of the penetration of complex groove welding. We build an active–passive cooperative vision system based on gas metal arc welding (GMAW). A mapping relationship from multi-modal data to the backside melting width is established. The multi-modal data consists of laser line images and molten pool images. The groove angle is extracted from the laser line image based on the segmentation model with the addition of online hard example mining. The molten pool image information is extracted based on DenseNet and ASPP model. Then, the above information is reconstructed and fused to predict the backside melting width. The Mean Square Error (MSE) of the predicted backside melting width is better than 0.28 mm for complex grooves and is 57% lower than that without adding an angle, which verifies the model's accuracy. The model has a run time of fewer than 0.015 s, which meets the time requirement for online monitoring. Finally, the backside melting width is controlled based on fuzzy proportional-integral-derivative (PID) control. The MSE of the control result does not exceed 0.11 mm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

References

  • Chang, W. , & Li, J. . (2015). Research on line structured light calibration method. In International Conference on Intelligent Systems Research & Mechatronics Engineering.

  • Chen, S. B., & Wu, J. (2009). Real-Time control of weld pool dynamics during robotic GTAW. In Intelligentized Methodology for Arc Welding Dynamical Processes (pp. 221–273). Springer

  • Chen, Z., Feng, Z., & Chen, J. (2021). Penetration detection of narrow U-groove welding. In Key Technologies of Intelligentized Welding Manufacturing (pp. 71–80). Springer

  • Chen, C., Lv, N., & Chen, S. (2020). Welding penetration monitoring for pulsed gtaw using visual sensor based on aam and random forests. Journal of Manufacturing Processes., 63, 152–162.

    Article  Google Scholar 

  • Cheng, Y., Yu, R., Zhou, Q., Chen, H., Yuan, W., & Zhang, Y. (2021). Real-time sensing of gas metal arc welding process: A literature review and analysis. Journal of Manufacturing Processes, 70, 452–469.

    Article  Google Scholar 

  • Dinham, M., & Gu, F. (2013). Autonomous weld seam identification and localization using eye-in-hand stereo vision for robotic arc welding. Robotics and Computer-Integrated Manufacturing, 29(5), 288–301.

    Article  Google Scholar 

  • Gagnon, D. P., & Kennedy, D. (2011). Behaviour and ultimate tensile strength of partial joint penetration groove welds. Canadian Journal of Civil Engineering, 16(3), 384–399.

    Article  Google Scholar 

  • Gao, J. Q., Qin, G. L., Yang, J. L., Jian-Guo, H. E., Zhang, T., & Chuan-Song, W. U. (2011). Image processing of weld pool and keyhole in nd:Yag laser welding of stainless steel based on visual sensing. Transactions of Nonferrous Metals Society of China, 21(2), 423–428.

    Article  Google Scholar 

  • Gillespie, J., Yeoh, W. Y., Zhao, C., Parab, N. D., Sun, T., Rollett, A. D., ... & Kube, C. M. (2021). In situ characterization of laser-generated molten pools using synchronized ultrasound and high-speed X-ray imaging. The Journal of the Acoustical Society of America, 150(4), 2409–2420.

  • Henri, F., Ville, K., & Anna F. et al. (2009). Visual measurement and tracking in laser hybrid welding. Machine Vision & Applications., 20, 103–118.

    Article  Google Scholar 

  • Hong, Y., Pan, H., Sun, W., Member, S., IEEE, & Jia, Y. (2021). Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes.

  • Kai, W., Xiangdong, J., Jialei, Z., Jingyang, L., & Congwei, L. (2021). Research on the effect of weld groove on the quality and stability of laser-MAG hybrid welding in horizontal position. Welding in the World, 65(9), 1701–1709.

    Article  Google Scholar 

  • Kawahara, M. (1983). Tracking control system using image sensor for arc welding. Automatica, 19(4), 357–363.

    Article  Google Scholar 

  • Kim, J., Lee, J., Chung, M., & Shin, Y. G. (2021). Multiple weld seam extraction from rgb-depth images for automatic robotic welding via point cloud registration. Multimedia Tools and Applications, 80(13), 1–17.

    Google Scholar 

  • Liu, X. F., Wu, C. S., Jia, C. B., & Zhang, G. K. (2017). Visual sensing of the weld pool geometry from the topside view in keyhole plasma arc welding. Journal of Manufacturing Processes, 26(4), 74–83.

    Article  Google Scholar 

  • Lv, N., Xu, Y., Li, S., Yu, X., & Chen, S. (2017). Automated control of welding penetration based on audio sensing technology. Journal of Materials Processing Technology, 250, 81–98.

    Article  Google Scholar 

  • Lv, N., Zhong, J., Chen, H., Lin, T., & Chen, S. (2014). Real-time control of welding penetration during robotic GTAW dynamical process by audio sensing of arc length. The International Journal of Advanced Manufacturing Technology, 74(1), 235–249.

    Article  Google Scholar 

  • Nomura, K., Fukushima, K., Matsumura, T., & Asai, S. (2020). Burn-through prediction and weld depth estimation by deep learning model monitoring the molten pool in gas metal arc welding with gap fluctuation. Journal of Manufacturing Processes, 61(1), 25.

    Google Scholar 

  • Pham, V. Q., Ito, S., & Kozakaya, T. (2017). Biseg: simultaneous instance segmentation and semantic segmentation with fully convolutional networks.

  • Romera, E., Alvarez, J. M., Bergasa, L. M., & Arroyo, R (2017). Efficient ConvNet for real-time semantic segmentation. In 2017 IEEE Intelligent Vehicles Symposium (IV). IEEE.

  • Wang, Y., Xu, X., Zhao, Z., Deng, W., Han, J., Bai, L., ... & Yao, J. (2021). Coordinated monitoring and control method of deposited layer width and reinforcement in WAAM process. Journal of Manufacturing Processes, 71, 306–316.

  • Wu, D., Huang, Y., Chen, H., He, Y., & Chen, S. (2017). Vppaw penetration monitoring based on fusion of visual and acoustic signals using t-sne and dbn model. Materials & Design, 123(6), 1–14.

    Article  Google Scholar 

  • Xiong, J., & Zhang, G. (2013). Online measurement of bead geometry in gmaw-based additive manufacturing using passive vision. Measurement Science & Technology, 24(11), 5103.

    Article  Google Scholar 

  • Xu, P., Tang, X., & Yao, S. (2007). Application of circular laser vision sensor (clvs) on welded seam tracking. Journal of Materials Processing Tech, 205(1–3), 404–410.

    Google Scholar 

  • Yang, L., Liu, Y., Peng, J., & Liang, Z. (2020). A novel system for off-line 3d seam extraction and path planning based on point cloud segmentation for arc welding robot. Robotics and Computer-Integrated Manufacturing, 64(3), 101929.

    Article  Google Scholar 

  • Yu, R., Kershaw, J., Wang, P., & Zhang, Y. (2021). Real-time recognition of arc weld pool using image segmentation network. Journal of Manufacturing Processes, 72, 159–167.

    Article  Google Scholar 

  • Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2016). Pyramid scene parsing network. IEEE Computer Society, 9, 1–5.

    Google Scholar 

  • Zhao, Z., Luo, J., Wang, Y., Bai, L., & Han, J. (2021). Additive seam tracking technology based on laser vision. The International Journal of Advanced Manufacturing Technology, 116(1), 197–211.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant Nos. 62101265, 62271263), the China Postdoctoral Science Foundation (2021M691592), and the Fundamental Research Funds for the Central Universities (No. 30922010705).

Author information

Authors and Affiliations

  1. Jiangsu Key Laboratory of Spectral Imaging and Intelligent Sense, Nanjing University of Science and Technology, Nanjing, 210094, China

    Peng Gao, Zijian Wu, Yiming Wang, Jun Lu & Zhuang Zhao

  2. School of Intelligent Manufacturing, Luoyang Institute of Science and Technology, Luoyang, 471023, China

    Yiming Wang

Authors
  1. Peng Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zijian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yiming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhuang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jun Lu or Zhuang Zhao.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Table 8.

Table 8 Mathematical symbols and meanings
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gao, P., Wu, Z., Wang, Y. et al. Method for monitoring and controlling penetration of complex groove welding based on online multi-modal data. J Intell Manuf 35, 1247–1265 (2024). https://doi.org/10.1007/s10845-023-02107-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10845-023-02107-2

Keywords

Search

Navigation