Skip to main content
SpringerLink
Log in

Modeling and assessing an intelligent system for safety in human-robot collaboration using deep and machine learning techniques

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The introduction of technological innovations is essential for accident mitigation in work environments. In a human-robot collaboration scenario, the current number of accidents raises a safety problem that must be dealt. This work proposes an intelligent system that aims to address such problems using deep and machine learning techniques. More specifically, this solution is divided into two modules: (i) collision detection between humans and robots and (ii) worker’s clothing detection. We evaluated these modules separately and concluded that the proposed intelligent system is efficient in supporting safe human-robot collaboration. The results achieved a sensitivity level greater than 90% in identifying collisions and an accuracy above 94% in identifying the worker’s clothing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://github.com/wkentaro/labelme

  2. https://github.com/divamgupta/image-segmentation-keras

References

  1. Anvaripour, M, Saif M (2019) Collision detection for human-robot interaction in an industrial setting using force myography and a deep learning approach. In: 2019 IEEE international conference on systems, man and cybernetics (SMC). IEEE, pp 2149–2154

  2. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39(12):2481–2495

    Article  Google Scholar 

  3. Breiman L (2001) Random forests. Machine learning 45(1):5–32

    Article  Google Scholar 

  4. Garcia-Garcia A, Orts-Escolano S, Oprea S, Villena-Martinez V, Martinez-Gonzalez P, Garcia-Rodriguez J (2018) A survey on deep learning techniques for image and video semantic segmentation. Applied Soft Computing 70:41–65

    Article  Google Scholar 

  5. Graczyk M, Lasota T, Trawiński B, Trawiński K (2010) Comparison of bagging, boosting and stacking ensembles applied to real estate appraisal. In Asian conference on intelligent information and database systems. Springer, pp 340–350

  6. Hearst MA, Dumais ST, Osuna E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intelligent Systems and their applications 13(4):18–28

    Article  Google Scholar 

  7. Heo YJ, Kim D, Lee W, Kim H, Park J, Chung WK (2019) Collision detection for industrial collaborative robots: a deep learning approach. IEEE Robotics and Automation Letters 4(2):740–746

    Article  Google Scholar 

  8. Hong C, Yu J, Wan J, Tao D, Wang M (2015) Multimodal deep autoencoder for human pose recovery. IEEE Transactions on Image Processing 24(12):5659–5670

    Article  MathSciNet  Google Scholar 

  9. Jocelyn S, Burlet-Vienney D, Giraud L, Sghaier A (2019) Collaborative robotics: Assessment of safety functions and feedback from workers, users and integrators in quebec

  10. Keller JM, Gray MR, Givens JA (1985) A fuzzy k-nearest neighbor algorithm. IEEE transactions on systems, man, and cybernetics 4:580–585

    Article  Google Scholar 

  11. Kelm A, Laußat L, Meins-Becker A, Platz D, Khazaee MJ, Costin AM, Helmus M, Teizer J (2013) Mobile passive radio frequency identification (rfid) portal for automated and rapid control of personal protective equipment (ppe) on construction sites. Automation in construction 36:38–52

    Article  Google Scholar 

  12. Khalid S, Khalil T, Nasreen S (2014) A survey of feature selection and feature extraction techniques in machine learning. In: 2014 science and information conference. pp 372–378

  13. Kittler J, Hatef M, Duin RPW, Matas J (1998) On combining classifiers. IEEE transactions on pattern analysis and machine intelligence 20(3):226–239

    Article  Google Scholar 

  14. Kleinbaum DG, Dietz K, Gail M, Klein M, Klein M (2002) Logistic regression. Springer, Berlin

    Google Scholar 

  15. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11):2278–2324

    Article  Google Scholar 

  16. LeCun Y, Bengio Y, Hinton Geoffrey (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  17. Lee S, Kim M, Song J (2015) Sensorless collision detection for safe human-robot collaboration. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). pp 2392–2397

  18. Liu X, Deng Z, Yang Y (2019) Recent progress in semantic image segmentation. Artificial Intelligence Review 52(2):1089–1106

    Article  Google Scholar 

  19. Man crushed to death by robot at car factory. http://www.bbc.co.uk/newsbeat/article/33359005/man-crushed-to-death-by-robot-at-car-factory, 205. Accessed 25 Aug 2020

  20. Mittal S, Tatarchenko M, Brox T (2019) Semi-supervised semantic segmentation with high-and low-level consistency. IEEE Transactions on Pattern Analysis and Machine Intelligence

  21. Mohammed A, Schmidt B, Wang L (2017) Active collision avoidance for human-robot collaboration driven by vision sensors. International Journal of Computer Integrated Manufacturing 30(9):970–980

    Article  Google Scholar 

  22. Mohanaiah P, Sathyanarayana P, GuruKumar L (2013) Image texture feature extraction using glcm approach. International journal of scientific and research publications 3(5):1

    Google Scholar 

  23. Nath ND, Behzadan AH, Paal SG (2020) Deep learning for site safety: Real-time detection of personal protective equipment. Automation in Construction 112:103085

    Article  Google Scholar 

  24. Nekrasov V, Shen C, Reid I (2018) Light-weight refinenet for real-time semantic segmentation. arXiv:arXiv:1810.03272

  25. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern recognition 29(1):51–59

    Article  Google Scholar 

  26. Pan B, Shi Z, Xu X, Shi T, Zhang N, Zhu X (2018) Coinnet: Copy initialization network for multispectral imagery semantic segmentation. IEEE Geoscience and Remote Sensing Letters 16(5):816–820

    Article  Google Scholar 

  27. Pengfei L, Lin X, Jing J, Zhang L (2014) Defect detection in fabrics using local binary patterns. In: Tan T, Ruan Q, Wang S, Ma H, Huang K (eds) Advances in Image and Graphics Technologies. Springer, Berlin, pp 274–283

    Google Scholar 

  28. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp 779–788

  29. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  30. Ruck D, Rogers SK, Kabrisky M, Oxley ME, Suter BW (1990) The multilayer perceptron as an approximation to a bayes optimal discriminant function. IEEE Transactions on Neural Networks 1(4):296–298

    Article  Google Scholar 

  31. Ruder S (2016) An overview of gradient descent optimization algorithms. arXiv:arXiv:1609.04747

  32. Salem YB, Nasri S (2010) Automatic recognition of woven fabrics based on texture and using svm. Signal, Image and Video Processing 4:429–434

    Article  Google Scholar 

  33. Sevastopolsky A (2017) Optic disc and cup segmentation methods for glaucoma detection with modification of u-net convolutional neural network. Pattern Recognition and Image Analysis 27(3):618–624

    Article  Google Scholar 

  34. Sharkawy A-N, Aspragathos N (2018) Human-robot collision detection based on neural networks. Int. J. Mech. Eng. Robot. Res 7(2):150–157

    Article  Google Scholar 

  35. Sharkawy A-N, Koustoumpardis PN, Aspragathos NA (2018) Manipulator collision detection and collided link identification based on neural networks. In: International conference on robotics in Alpe-Adria Danube region. Springer, pp 3–12

  36. Sharkawy A-N, Koustoumpardis PN, Aspragathos N (2020) Human-robot collisions detection for safe human-robot interaction using one multi-input-output neural network. Soft Computing 24(9):6687–6719

    Article  Google Scholar 

  37. Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence 39(4):640–651

    Article  Google Scholar 

  38. Shin H, Kim S, Seo K, Rhim, S (2019) A real-time human-robot collision safety evaluation method for collaborative robot. In: 2019 Third IEEE international conference on robotic computing (IRC). IEEE, pp 509–513

  39. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: 3rd International conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  40. Surakarin W, Chongstitvatana P (2015) Classification of clothing with weighted surf and local binary patterns. In: 2015 international computer science and engineering conference (ICSEC). pp 1–4

  41. Takiguchi T, Lee JH, Okamoto, S (2018) Collision avoidance algorithm using deep learning type artificial intelligence for a mobile robot. In: Proceedings of the international multiconference of engineers and computer scientists, vol 1

  42. Tokunaga H, Teramoto Y, Yoshizawa A, Bise R (2019) Adaptive weighting multi-field-of-view cnn for semantic segmentation in pathology. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 12597–12606

  43. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E (2018) Deep learning for computer vision: A brief review. Computational intelligence and neuroscience

  44. Wang Y, Ye X, Yang Y, Zhang W (2017) Hand movement prediction based collision-free human-robot interaction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp 20–21

Download references

Acknowledgements

This work was supported by the Research, Development and Innovation Center, Ericsson Telecommunications Inc., Brazil, Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), and the Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE).

Author information

Authors and Affiliations

  1. Universidade Federal de Pernambuco, Recife, Pernambuco, Brasil

    Iago Richard Rodrigues, Gibson Barbosa, Assis Oliveira Filho, Carolina Cani, Marrone Dantas, Djamel H. Sadok & Judith Kelner

  2. Ericsson Research, Indaiatuba, São Paulo, Brasil

    Ricardo Silva Souza, Maria Valéria Marquezini & Silvia Lins

Authors
  1. Iago Richard Rodrigues
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gibson Barbosa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Assis Oliveira Filho
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carolina Cani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marrone Dantas
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Djamel H. Sadok
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Judith Kelner
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ricardo Silva Souza
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Maria Valéria Marquezini
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Silvia Lins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Iago Richard Rodrigues.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rodrigues, I.R., Barbosa, G., Oliveira Filho, A. et al. Modeling and assessing an intelligent system for safety in human-robot collaboration using deep and machine learning techniques. Multimed Tools Appl 81, 2213–2239 (2022). https://doi.org/10.1007/s11042-021-11643-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11643-z

Keywords

Search

Navigation