Skip to main content
SpringerLink
Log in

Cleaning of object surfaces based on deep learning: a method for generating manipulator trajectories using RGB-D semantic segmentation

  • S.I.: AI based Techniques and Applications for Intelligent IoT Systems
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

A mobile robot with a robotic arm needs to be able to autonomously perceive the operating environment and plan the trajectory of the object’s surface in order to perform surface cleaning tasks in a complex, unstructured environment. This study suggests an autonomous trajectory planning technique for cleaning an object’s surface based on RGB-D semantic segmentation, which enables the robotic arm to move the cleaning mechanism on the object’s surface smoothly and steadily and finish the cleaning process. More particularly, it contains the following: (1) A Double Attention Fusion Net (DAFNet) RGB-D semantic segmentation network is proposed, which successfully integrates color texture features and spatial structure features and enhances the semantic segmentation performance of indoor objects. This network is based on the dual attention mechanism (channel attention and spatial attention). (2) The trajectory planning algorithm for the robot arm is created, and the semantically segmented data is clustered using DBCSCAN. In order to achieve autonomous planning of the cleaning trajectory, the target subject is first extracted, and then the working trajectory of the robot arm is generated via the processes of edge detection, slicing, sampling, fitting, etc. We also compare the accuracy of DAFNet semantic segmentation and other algorithms on SUNRGBD and self-built datasets, experiment with trajectory generation for various objects, and evaluate the online surface cleaning procedure. According to the experimental findings, the DAFNet semantic segmentation model is more accurate than the current models. According to the online test, the trajectory generated has a good degree of smoothness and continuity, and the robotic arm is capable of completing the surface cleaning operation effectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Chen H, Fuhlbrigge T, Li X (2009) A review of cad-based robot path planning for spray painting. Ind Robot Int J

  2. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3431–3440

  3. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  4. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  5. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H (2018) Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the European conference on computer vision (ECCV), pp 801–818

  6. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2117–2125

  7. Li X, You A, Zhu Z, Zhao H, Yang M, Yang K, Tan S, Tong Y (2020) Semantic flow for fast and accurate scene parsing. In: European conference on computer vision. Springer, pp 775–793

  8. Li X, Zhao H, Han L, Tong Y, Tan S, Yang K (2020) Gated fully fusion for semantic segmentation. Proc AAAI Conf Artif Intell 34:11418–11425

    Google Scholar 

  9. Hu P, Perazzi F, Heilbron FC, Wang O, Lin Z, Saenko K, Sclaroff S (2020) Real-time semantic segmentation with fast attention. IEEE Robot Autom Lett 6(1):263–270

    Article  Google Scholar 

  10. Li Y, Song L, Chen Y, Li Z, Zhang X, Wang X, Sun J (2020) Learning dynamic routing for semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8553–8562

  11. Hu X, Yang K, Fei L, Wang K (2019) ACNET: attention based network to exploit complementary features for RGBD semantic segmentation. In: IEEE international conference on image processing (ICIP). IEEE, pp 1440–1444

  12. Park S-J, Hong K-S, Lee S (2017) Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation. In: Proceedings of the IEEE international conference on computer vision, pp 4980–4989

  13. Chen L-Z, Lin Z, Wang Z, Yang Y-L, Cheng M-M (2021) Spatial information guided convolution for real-time rgbd semantic segmentation. IEEE Trans Image Process 30:2313–2324

    Article  Google Scholar 

  14. Cao J, Leng H, Lischinski D, Cohen-Or D, Tu C, Li Y (2021) Shapeconv: shape-aware convolutional layer for indoor RGB-D semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7088–7097

  15. Cui H, Dong J, Hou G, Xiao Z, Chen Y, Zhao Z (2013) Analysis on arc-welding robot visual control tracking system. In: International conference on quality, reliability, risk, maintenance, and safety engineering (QR2MSE)

  16. Martínez D, Alenya G, Torras C (2015) Planning robot manipulation to clean planar surfaces. Eng Appl Artif Intell 39:23–32

    Article  Google Scholar 

  17. Chen W, Zhao D (2013) Path planning for spray painting robot of workpiece surfaces. Math Probl Eng. https://doi.org/10.1155/2013/659457

    Article  Google Scholar 

  18. Gasparetto A, Vidoni R, Pillan D, Saccavini E (2012) Automatic path and trajectory planning for robotic spray painting. In: 7th German conference on robotics ROBOTIK 2012. VDE, pp 1–6

  19. Chen H, Xi N (2008) Automated tool trajectory planning of industrial robots for painting composite surfaces. Int J Adv Manuf Technol 35(7):680–696

    Article  Google Scholar 

  20. Atkar PN, Greenfield A, Conner DC, Choset H, Rizzi AA (2005) Uniform coverage of automotive surface patches. Int J Robot Res 24(11):883–898

    Article  Google Scholar 

  21. Wang G, Cheng J, Li R, Chen K (2015) A new point cloud slicing based path planning algorithm for robotic spray painting. In: IEEE international conference on robotics and biomimetics (ROBIO). IEEE, pp 1717–1722

  22. Wong C-C, Yeh L-Y, Liu C-C, Tsai C-Y, Aoyama H (2021) Manipulation planning for object re-orientation based on semantic segmentation keypoint detection. Sensors 21(7):2280

    Article  Google Scholar 

  23. Yin J, Apuroop KGS, Tamilselvam YK, Mohan RE, Ramalingam B, Le AV (2020) Table cleaning task by human support robot using deep learning technique. Sensors 20(6):1698

    Article  Google Scholar 

  24. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp 3–19

  25. Zhao H, Jiang L, Jia J, Torr PH, Koltun V (2021) Point transformer. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 16259–16268

  26. Ester M, Kriegel H-P, Sander J, Xu X et al (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Kdd, vol 96, pp 226–231

  27. Guo G, Wang H, Bell D, Bi Y, Greer K (2003) KNN model-based approach in classification. In: OTM confederated international conferences on the move to meaningful internet systems. Springer, pp 986–996

  28. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52

    Article  Google Scholar 

  29. Wu Y, Wong Y, Loh HT, Zhang Y (2004) Modelling cloud data using an adaptive slicing approach. Comput Aided Des 36(3):231–240

    Article  Google Scholar 

  30. Woo H, Kang E, Wang S, Lee KH (2002) A new segmentation method for point cloud data. Int J Mach Tools Manuf 42(2):167–178

    Article  Google Scholar 

  31. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  32. Intel RealSense\(^{\text{TM}}\) (2018) Depth module D400 series custom calibration; Intel Corporation:Santa Clara, CA, USA

Download references

Acknowledgements

This research was sponsored by Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZX0103) and Natural Science Foundation of Jiangxi Province (20212BAB202026).

Author information

Authors and Affiliations

  1. Academy of Engineering and Applied Technology, Fudan University, Handan Road 220, Shanghai, 200433, China

    Lizhe Qi, Zhongxue Gan, Zhongwei Hua, Daming Du, Wenxuan Jiang & Yunquan Sun

Authors
  1. Lizhe Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongxue Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhongwei Hua
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daming Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenxuan Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yunquan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhongxue Gan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qi, L., Gan, Z., Hua, Z. et al. Cleaning of object surfaces based on deep learning: a method for generating manipulator trajectories using RGB-D semantic segmentation. Neural Comput & Applic 35, 8677–8692 (2023). https://doi.org/10.1007/s00521-022-07930-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07930-x

Keywords

Search

Navigation