Skip to main content

Advertisement

SpringerLink
Log in

HOSA: An End-to-End Safety System for Human-Robot Interaction

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The advancement of collaborative robotics increases process efficiency. However, humans are still part of the loop in many deployment scenarios. They are unpredictable factors that may potentially become at risk. This work proposes a safety system for Human-Robot Interaction (HRI), called HOSA, and discusses the decisions made from its modular architectural design phase to a real scenario implementation. HOSA is an end-to-end system that considers the information provided by sensors available in the environment, the communication network to transport the information, the reasoning in the information, and the interface to present the risks. It applies deep learning algorithms to detect HRI collision risk and the use of Personal Protective Equipment (PPE) based on surveillance camera images. Also, it considers knowledge representation based on Ontology, Software-Defined Wireless Networking (SDWN), and a user interface based on augmented reality. The benefits of the proposed design are evaluated through a use case of a HRI scenario for radio base station maintenance. The architecture scales with the number of devices due to semantic descriptions and an adequately provisioned communication network. It demonstrates the system’s efficiency in detecting risk during HRI tasks and alerting people in the scenario. The conducted experiment shows that the system takes 1.052 seconds to react to a risky situation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code availability

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

References

  1. Microsoft Dynamics 365. 2019 manufacturing trends report (2019). Accessed 2021 Sept 7

  2. Rocha, F., et al.: Rosi: A robotic system for harsh outdoor industrial inspection-system design and applications. J. Intell. Robot. Syst. 103(2), 1–22 (2021)

    Article  Google Scholar 

  3. Vysocky, A., Novak, P.: Human-robot collaboration in industry. MM Sci. J. 9(2), 903–906 (2016)

    Article  Google Scholar 

  4. Villani, V., Pini, F., Leali, F., Secchi, C.: Survey on human-robot collaboration in industrial settings: Safety, intuitive interfaces and applications. Mechatronics 55, 248–266 (2018)

    Article  Google Scholar 

  5. Tsarouchi, P., Makris, S., Chryssolouris, G.: Human-robot interaction review and challenges on task planning and programming. Int. J. Comput. Integrat. Manufact. 29(8), 916–931 (2016)

    Article  Google Scholar 

  6. Chandrasekaran, B. Conrad, J.M.: Human-robot collaboration: A survey 1–8 (2015)

  7. Haddadin, S., et al.: On making robots understand safety: Embedding injury knowledge into control. Int. J. Robot. Res. 31(13), 1578–1602 (2012)

    Article  Google Scholar 

  8. ISO. Iso 31000: 2009. Risk management– Guideline (2009)

  9. Purdy, G.: Iso 31000: 2009–setting a new standard for risk management. Risk Anal. An Int. J. 30(6), 881–886 (2010)

    Article  Google Scholar 

  10. Smith, C. L. Brooks, D.J.: In Chapter 3 - security risk management (eds Smith, C. L. & Brooks, D. J.) Security Science 51–80 (Butterworth-Heinemann, Boston) (2013)

  11. Kirton, D.: Risk assessment [electric machines] 5–1 (1995)

  12. ABNT. Iso 12100: 2013. Segurança de máquinas — Princípios gerais de projeto — Apreciação e redução de riscos (2013)

  13. ISO, E. Safety of machinery–general principles for design–risk assessment and risk reduction (iso 12100:2010) (2010)

  14. ISO, E. Robots and robotic devices — safety requirements for industrial robots — part 1: Robots (iso 10218-1:2011) (2011)

  15. ISO, E. Robots and robotic devices — safety requirements for industrial robots — part 2: Robot systems and integration (iso 10218-2:2011) (2011)

  16. ISO, E. Robots and robotic devices — collaborative robots (iso/ts 15066:2016) (2016)

  17. Zanchettin, A.M., Rocco, P., Chiappa, S., Rossi, R.: Towards an optimal avoidance strategy for collaborative robots. Robot. Comput.-Integrat. Manufact. 59, 47–55 (2019)

    Article  Google Scholar 

  18. Pellegrinelli, S., Orlandini, A., Pedrocchi, N., Umbrico, A., Tolio, T.: Motion planning and scheduling for human and industrial-robot collaboration. CIRP Annals 66(1), 1–4 (2017)

    Article  Google Scholar 

  19. Pedrocchi, N., Vicentini, F., Matteo, M., Tosatti, L.M.: Safe human-robot cooperation in an industrial environment. Int. J. Adv. Robot. Syst. 10(1), 27 (2013)

    Article  Google Scholar 

  20. Michalos, G., et al.: Design considerations for safe human-robot collaborative workplaces. Procedia CIrP 37, 248–253 (2015)

    Article  Google Scholar 

  21. Glogowski, P., Böhmer, A., Hypki, A., Kuhlenkötter, B.: Robot speed adaption in multiple trajectory planning and integration in a simulation tool for human-robot interaction. J. Intell. Robot. Syst. 102(1), 1–20 (2021)

    Article  Google Scholar 

  22. Byner, C., Matthias, B., Ding, H.: Dynamic speed and separation monitoring for collaborative robot applications - concepts and performance. Robot. Comput.-Integrat. Manufact. 58, 239–252 (2019)

    Article  Google Scholar 

  23. Zhang, S., Li, S., Li, X., Xiong, Y., Xie, Z.: A human-robot dynamic fusion safety algorithm for collaborative operations of cobots. J. Intell. Robot. Syst. 104(1), 1–14 (2022)

    Article  Google Scholar 

  24. Papanastasiou, S., et al.: Towards seamless human robot collaboration: integrating multimodal interaction. Int. J. Adv. Manufact. Technol. 105(9), 3881–3897 (2019)

    Article  Google Scholar 

  25. Zanchettin, A.M., Ceriani, N.M., Rocco, P., Ding, H., Matthias, B.: Safety in human-robot collaborative manufacturing environments: Metrics and control. IEEE Trans. Autom. Sci. Eng. 13(2), 882–893 (2015)

    Article  Google Scholar 

  26. Magrini, E., et al.: Human-robot coexistence and interaction in open industrial cells. Robot. Comput.-Integrat. Manufact. 61, 101846 (2020)

    Article  Google Scholar 

  27. Nikolakis, N., Maratos, V., Makris, S.: A cyber physical system (cps) approach for safe human-robot collaboration in a shared workplace. Robot. Comput.-Integrat. Manufact. 56, 233–243 (2019)

    Article  Google Scholar 

  28. Michalos, G., et al.: Seamless human robot collaborative assembly-an automotive case study. Mechatronics 55, 194–211 (2018)

    Article  Google Scholar 

  29. Bell, R.: Introduction to iec 61508(162), 3–12 (2006)

    Google Scholar 

  30. Embitel.: 7 most commonly used sensors for developing industrial iot solutions. Avaiable in: https://www.embitel.com/blog/embedded-blog/7-most-commonly-used-sensors-for-developing-industrial-iot-solutions (2018). Accessed 2021 Sept 7

  31. Silva, I. R. et al.: Assessing deep learning models for human-robot collaboration collision detection in industrial environments 240–255 (2020)

  32. Rodrigues, I.R., et al.: Modeling and assessing an intelligent system for safety in human-robot collaboration using deep and machine learning techniques. Multimed. Tools Appl. 81(2), 2213–2239 (2022)

    Article  Google Scholar 

  33. Patel, A. Jain, S.: Formalisms of representing knowledge. Procedia Comput. Sci. 125, 542–549 (2018). The 6th International Conference on Smart Computing and Communications

  34. Musen, M.A.: The protégé project: a look back and a look forward. AI Matters 1(4), 4–12 (2015)

    Article  Google Scholar 

  35. Borgo, S., Cesta, A., Orlandini, A., Umbrico, A.: Knowledge-based adaptive agents for manufacturing domains. Eng. Comput. 35(3), 755–779 (2019)

    Article  Google Scholar 

  36. Liebig, T. Reasoning with owl-system support and insights (2013)

  37. Dentler, K., Cornet, R., De Ten Teije, A., Keizer, N.: Comparison of reasoners for large ontologies in the owl 2 el profile. Semantic Web 2(2), 71–87 (2011)

    Article  Google Scholar 

  38. Glimm, B., Horrocks, I., Motik, B., Stoilos, G., Wang, Z.: Hermit: an owl 2 reasoner. J. Autom. Reas. 53(3), 245–269 (2014)

    Article  Google Scholar 

  39. Shearer, R., Motik, B. Horrocks, I.: Hermit: A highly-efficient owl reasoner. 432, 91 (2008)

  40. Horrocks, I. et al.: Swrl: A semantic web rule language combining owl and ruleml. W3C Member Submission 21(79), 1–31 (2004)

  41. McGuinness, D. L., Van Harmelen, F. et al.: Owl web ontology language overview. W3C Recommendation 10(10), 2004 (2004)

  42. Varghese, A., Tandur, D.: Wireless requirements and challenges in industry 4, 634–638 (2014)

    Google Scholar 

  43. Osseiran, A., et al.: Scenarios for 5g mobile and wireless communications: the vision of the metis project. IEEE Commun. Magazine 52(5), 26–35 (2014)

    Article  Google Scholar 

  44. Neal, A. D., Sharpe, R. G., Conway, P. P. West, A. A.: Smarti—a cyber-physical intelligent container for industry 4.0 manufacturing. Journal of Manufacturing Systems 52, 63–75 (2019)

  45. Wittenberg, C. Human-cps interaction - requirements and human-machine interaction methods for the industry 4.0. IFAC-PapersOnLine 49(19), 420–425 (2016). 13th IFAC Symposium on Analysis, Design, and Evaluation ofHuman-Machine Systems HMS 2016

  46. Flatt, H., Schriegel, S., Jasperneite, J., Trsek, H., Adamczyk, H.: Analysis of the cyber-security of industry 4.0 technologies based on rami 4.0 and identification of requirements 1–4 (2016)

  47. Chiueh, T.-c. et al.: Supporting real-time traffic on ethernet 282–286 (1994)

  48. Ong, S., Yew, A., Thanigaivel, N., Nee, A.: Augmented reality-assisted robot programming system for industrial applications. Robot. Comput.-Integrat. Manufact. 61, 101820 (2020)

    Article  Google Scholar 

  49. VideoLan: Vlc media player. https://www.videolan.org/vlc/index.html (2006). Accessed 2021 Sept 7

  50. MPlayer Features. http://www.mplayerhq.hu/design7/info.html (2021). Accessed 2021 Sept 7

  51. Syaifudin, Y., Rozi, I., Ariyanto, R., Erfan, R., Adhisuwignjo, S.: Study of performance of real time streaming protocol (rtsp) in learning systems. Int. J. Eng. Technol. (UAE) 7, 216–221 (2018)

    Article  Google Scholar 

  52. loman, A., Ispas, A., Ciotirnae, P., Sanchez-Iborra, R., Cano, M.-D: Performance evaluation of video streaming using mpeg dash, rtsp, and rtmp in mobile networks 144–151 (2015) 

  53. FFmpeg. Ffmpeg. https://ffmpeg.org/ (2022). Accessed 2021 Sept 7

  54. Tantayakul, K., Dhaou, R., Paillassa, B. Panichpattanakul, W.: Experimental analysis in sdn open source environment 334–337 (2017)

  55. Marvel, J.A.: Performance metrics of speed and separation monitoring in shared workspaces. IEEE Trans. Autom. Sci. Eng. 10(2), 405–414 (2013)

    Article  Google Scholar 

  56. Light, R.A.: Mosquitto: server and client implementation of the mqtt protocol. J. Open Source Softw. 2(13), 265 (2017)

    Article  Google Scholar 

Download references

Acknowledgements

This work was financed in part by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE), and Ericsson Research, Brazil.

Funding

This work was supported by the Research, Development and Innovation Center, Ericsson Telecommunications Inc., Brazil, the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq), the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), and the Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE).

Author information

Author notes

    Authors and Affiliations

    1. Networking and Telecommunications Research Group (GPRT), Federal University of Pernambuco, Recife, Pernambuco, Brazil

      Gibson Barbosa, Carolina Ledebour, Assis T. de Oliveira Filho, Iago Richard Rodrigues, Djamel Sadok & Judith Kelner

    2. Innovation Center Ericsson Telecommunications, Indaiatuba, São Paulo, Brazil

      Ricardo Souza

    Authors
    1. Gibson Barbosa
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Carolina Ledebour
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Assis T. de Oliveira Filho
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Iago Richard Rodrigues
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Djamel Sadok
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Judith Kelner
      View author publications

      You can also search for this author in PubMed Google Scholar

    7. Ricardo Souza
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    Gibson Barbosa: Conceptualization, Methodology, Software, Validation, Investigation, Data Curation, Writing - Original Draft. Carolina Ledebour: Conceptualization, Writing - Original Draft, Visualization. Assis T. Oliveira Filho: Methodology, Validation, Investigation, Data Curation, Writing - Review and Editing. Iago Silva: Methodology, Validation, Investigation, Data Curation, Writing - Review and Editing. Djamel Sadok: Conceptualization, Writing - Review and Editing, Supervision, Project administration, Funding acquisition. Judith Kelner: Writing - Review and Editing, Project administration, Funding acquisition. Ricardo Souza: Writing - Review and Editing.

    Corresponding author

    Correspondence to Gibson Barbosa.

    Ethics declarations

    Conflicts of interest

    The authors have no relevant financial or non-financial interests to disclose.

    Ethical standard

    This work does not require ethics approval

    Consent to participate

    This work does not require consent to participate, because it does not involve human subjects.

    Consent to publish

    The authors affirm that human research participants provided informed consent for publication of the images in Figure(s) Fig. 2a, b and c.”

    Additional information

    Publisher’s Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Carolina Ledebour, Assis T. de Oliveira Filho, Iago Richard Rodrigues, Djamel Sadok, Judith Kelner and Ricardo Souza These authors contributed equally to this work.

    Rights and permissions

    Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Barbosa, G., Ledebour, C., de Oliveira Filho, A.T. et al. HOSA: An End-to-End Safety System for Human-Robot Interaction. J Intell Robot Syst 105, 95 (2022). https://doi.org/10.1007/s10846-022-01701-5

    Download citation

    • Received:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1007/s10846-022-01701-5

    Keywords

    Search

    Navigation