Abstract
Robotic welding technology is constantly growing with the development of vision technologies. To establish a fully automatic robotic welding system, an automated weld seam extraction process with accurate perception of the target workpiece (i.e., object) is one of the most challenging tasks. Although many methods pertaining to automatic weld seam extraction have been proposed, most of the previous studies are difficult to employ in practical systems because the algorithms are unable to simultaneously handle multiple and occluded seams from arbitrary view directions. In this study, we propose a novel method to extract weld seams based on point cloud registration from various view directions that can handle randomly occluded seams in a workpiece with multiple seam parts. We focus on the shape of the weld seams as a line or curve, which are dominant structures in the field. Initially, we extract all detectable weld seams for each image with a single view direction obtained using an RGB-depth vision sensor. Subsequently, three-dimensional points of weld seams obtained from each view direction are filtered based on the edge cues from RGB images. Finally, the extracted weld seams obtained from the various view directions are merged by employing a point cloud registration technique. The experimental results demonstrate that the proposed method outperforms a state-of-the-art method in terms of detection accuracy. Our proposed algorithm can be employed for dynamic workpiece scenes, which indicates that multiple weld seams can be successfully extracted from arbitrary view directions containing scene occlusions.








Similar content being viewed by others
References
Ahmed SM, Tan YZ, Lee GH, Chew CM, Pang CK (2016) Ob-ject detection and motion planning for automated welding of tubu-lar joints. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, pp. 2610–2615
Akinlar C, Topal C (2011) Edlines: a real-time line segment detectorwith a false detection control. Pattern Recogn Lett 32(13):1633–1642
Besl PJ, McKay ND (1992) Method for registration of 3-d shapes. In: Sensor fusion IV: control paradigms and data structures, International Society for Optics and Photonics, 1611:586–606
Desolneux A, Moisan L, Morel J-M (2000) Meaningful align-ments. Int J Comput Vis 40(1):7–23
Desolneux A, Moisan L, Morel J-M (2001) Edge detection by helmholtz principle. J Mathematic Imaging Vis 14(3):271–284
Dinham M, Fang G (2013) Autonomous weld seam identification andlocalisation using eye-in-hand stereo vision for robotic arc welding. Robot Comput Integr Manuf 29(5):288–301
Dinham M, Fang G (2014) Detection of fillet weld joints using an adaptive line growing algorithm for robotic arc welding. Robot Comp-Integ Manuf 30(3):229–243
Fang Z, Xu D, Tan M (2010) A vision-based self-tuning fuzzy con-troller for fillet weld seam tracking. IEEE/ASME Trans Mechatron 16(3):540–550
Fernandes LA, Oliveira MM (2008) Real-time line detection through an improved hough transform voting scheme. Pattern Recogn 41(1):299–314
Gordon WJ, Riesenfeld RF (1974) B-spline curves and surfaces. In: Computer aided geometric design. Elsevier, pp. 95–126
Hartley R, Zisserman A (2003) Multiple view geometry in computervision. Cambridge University Press, Cambridge
Lee S, Na S-J (2002) A study on automatic seam tracking in pulsedlaser edge welding by using a vision sensor without an auxiliary lightsource. J Manuf Syst 21(4):302
Li J, Li X, Tao D (2008) Kpca for semantic object extraction in images. Pattern Recogn 41(10):3244–3250
Li J, Jing F, Li E (2016) A new teaching system for arc welding robots with auxiliary path point generation module. In: 2016 35th Chinese Control Conference (CCC). IEEE, pp. 6217–6221
Li J, Mi Y, Li G, Ju Z (2019) Cnn-based facial expression recognition from annotated rgb-d images for human–robot interaction. Int J Human Robot 16(04):1941002
Liu J, Fan Z, Olsen SI, Christensen KH, Kristensen JK (2015) Boosting active contours for weld pool visual tracking in automatic arc welding. IEEE Trans Autom Sci Eng 14(2):1096–1108
Liu F, Wang Z, Ji Y (2018) Precise initial weld position identification of a fillet weld seam using laser vision technology. Int J Adv Manuf Technol 99(5–8):2059–2068
Liu Y, Tang Q, Tian X (2019) A discrete method of sphere-pipe intersecting curve for robot welding by offline programming. Robot Comput Integr Manuf 57:404–411
Lowe DG (2004) Distinctive image features from scale-invariant key-points. Int J Comput Vis 60(2):91–110
Lu X, Yao J, Li K, Li L (2015) Cannylines: a parameter-free line segment detector. In: 2015 IEEE International Conference on Image Processing (ICIP). IEEE, pp. 507–511
Maiolino P, Woolley R, Branson D, Benardos P, Popov A, Ratchev S (2017) Flexible robot sealant dispensing cell using rgb-d sensor and off-line programming. Robot Comput Integr Manuf 48:188–195
Muhammad J, Altun H, Abo-Serie E (2018) A robust butt welding seam finding technique for intelligent robotic welding system using active laser vision. Int J Adv Manuf Technol 94(1–4):13–29
Nunes JF, Moreira PM, Tavares JMR (2019 Benchmark rgb-dgait datasets: a systematic review. In: ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Springer, pp. 366–372
Ren T, Zhang A (2019) Rgb-d salient object detection: a review. In: RGB-D Image Analysis and Processing. Springer, pp. 203–220
Rodríguez-Martín M, Rodríguez-Gonzálvez P, González-Aguilera D, Fernández-Hernández J (2017) Feasibility study of a structured light system applied to welding inspection based on articulated coordinate measure machine data. IEEE Sensors J 17(13):4217–4224
Rout A, Deepak B, Biswal B (2019) Advances in weld seam tracking techniques for robotic welding: A review. Robot Comput Integr Manuf 56:12–37
Rusu RB (2010) Semantic 3d object maps for everyday manipulation in human living environments. KI-Künstliche Intelligenz 24(4):345–348
Segal A, Haehnel D, Thrun S (2009) Generalized-icp. In: Robotics: science and systems 2(4):435
Shah HNM, Sulaiman M, Shukor AZ, Kamis Z, Ab Rah-man A (2018) Butt welding joints recognition and location identification byusing local thresholding. Robot Comput Integr Manuf 51:181–188
Silvers GA, Fang G (2014) Human machine interface in weld seamdetection using an rgb-d camera. In: Advanced Materials Research, Trans Tech Publ 875:1967–1971
Von Gioi RG, Jakubowicz J, Morel J-M, Randall G (2008) Lsd:A fast line segment detector with a false detection control. IEEE Trans Pattern Anal Mach Intell 32(4):722–732
Wang N, Zhong K, Shi X, Zhang X (2020) A robust weld seam recognition method under heavy noise based on structured-light vision. Robot Comput Integr Manuf 61:101821
Yang L, Li E, Long T, Fan J, Liang Z (2018) A novel 3-d path extraction method for arc welding robot based on stereo structured lightsensor. IEEE Sensors J 19(2):763–773
Yang L, Liu Y, Peng J, Liang Z (2020) A novel system for off-line 3d seam extraction and path planning based on point cloud segmentationfor arc welding robot. Robot Comput Integr Manuf 64:101929
Sun Y, Weng Y, Luo B, Li G, Tao B, Jiang D, Chen D (2020) Gesture recognition algorithm basedon multi-scale feature fusion in rgb-d images. IET Image Processing
Zhang S (2018) High-speed 3d shape measurement with structured light methods: A review. Opt Lasers Eng 106:119–131
Acknowledgements
This work was supported by Osstem Implant Inc. (No. 0536-20190132, AI-based Panoramic, and CT Image Detection).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
None declared.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kim, J., Lee, J., Chung, M. et al. Multiple weld seam extraction from RGB-depth images for automatic robotic welding via point cloud registration. Multimed Tools Appl 80, 9703–9719 (2021). https://doi.org/10.1007/s11042-020-10138-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10138-7