Abstract
The new industrial settings are characterized by the presence of human and robots that work in close proximity, cooperating in performing the required job. Such a collaboration, however, requires to pay attention to many aspects. Firstly, it is crucial to enable a communication between this two actors that is natural and efficient. Secondly, the robot behavior must always be compliant with the safety regulations, ensuring always a safe collaboration. In this paper, we propose a framework that enables multi-channel communication between humans and robots by leveraging multimodal fusion of voice and gesture commands while always respecting safety regulations. The framework is validated through a comparative experiment, demonstrating that, thanks to multimodal communication, the robot can extract valuable information for performing the required task and additionally, with the safety layer, the robot can scale its speed to ensure the operator’s safety.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Video of experiments available at https://doi.org/10.5281/zenodo.8083948 [14].
References
Regenbogen, C., Schneider, D.A., Gur, R.E., Schneider, F., Habel, U., Kellermann, T.: Multimodal human communication - targeting facial expressions, speech content and prosody. NeuroImage 60(4), 2346–2356 (2012). https://www.sciencedirect.com/science/article/pii/S1053811912002212
Holler, J., Levinson, S.C.: Multimodal language processing in human communication. Trends Cogn. Sci. 23(8), 639–652 (2019). https://www.sciencedirect.com/science/article/pii/S1364661319301299
Bauer, A., Wollherr, D., Buss, M.: Human-robot collaboration: a survey. Int. J. Humanoid Robot. 05(01), 47–66 (2008). https://doi.org/10.1142/S0219843-608001303
Poria, S., Cambria, E., Bajpai, R., Hussain, A.: A review of affective computing: from unimodal analysis to multimodal fusion. Inf. Fusion 37, 98–125 (2017). https://www.sciencedirect.com/science/article/pii/S1566253517300738
Liu, H., Fang, T., Zhou, T., Wang, L.: Towards robust human-robot collaborative manufacturing: multimodal fusion. IEEE Access 6, 74762–74771 (2018)
Mohd, T.K., Nguyen, N., Javaid, A.Y.: Multi-modal data fusion in enhancing human-machine interaction for robotic applications: a survey. arXiv preprint arXiv:2202.07732 (2022)
Merckaert, K., Convens, B., ju Wu, C., Roncone, A., Nicotra, M.M., Vanderborght, B.: Real-time motion control of robotic manipulators for safe human-robot coexistence. Robot. Comput. Integr. Manuf. 73, 102223 (2022). https://www.sciencedirect.com/science/article/pii/S0736584521001022
Palleschi, A., Hamad, M., Abdolshah, S., Garabini, M., Haddadin, S., Pallottino, L.: Fast and safe trajectory planning: Solving the cobot performance/safety trade-off in human-robot shared environments. IEEE Robot. Autom. Lett. 6(3), 5445–5452 (2021)
Pupa, A., Arrfou, M., Andreoni, G., Secchi, C.: A safety-aware kinodynamic architecture for human-robot collaboration. IEEE Robot. Autom. Lett. 6(3), 4465–4471 (2021)
Burkhardt-Reed, M.M., Long, H.L., Bowman, D.D., Bene, E.R., Oller, D.K.: The origin of language and relative roles of voice and gesture in early communication development. Infant Behav. Dev. 65, 101648 (2021). https://www.sciencedirect.com/science/article/pii/S0163638321001223
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–80 (1997)
Cacace, J., Finzi, A., Lippiello, V.: A robust multimodal fusion framework for command interpretation in human-robot cooperation. In: 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 372–377 (2017)
Iso/ts 15066:2016(e). robots and robotic devices-collaborative robots. Technical specification, International Organization for Standardization, Geneva (2016)
Ferrari, D.: Experimental Validatio video for paper Safe Multimodal Communication in Human-Robot Collaboration (2023). https://doi.org/10.5281/zenodo.8083948
Stanford Artificial Intelligence Laboratory, et al.: Robotic operating system (2023). https://www.ros.org
Grinberg, M.: Flask Web Development: Developing Web Applications with Python. O’Reilly Media, Inc. (2018)
Lugaresi, C., et al.: MediaPipe: a framework for building perception pipelines. arXiv preprint arXiv:1906.08172 (2019)
Tan, J.C.A., Chan, W.P., Robinson, N.L., Croft, E.A., Kulic, D.: A proposed set of communicative gestures for human robot interaction and an RGB image-based gesture recognizer implemented in ROS. arXiv preprint arXiv:2109.09908 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ferrari, D., Pupa, A., Signoretti, A., Secchi, C. (2024). Safe Multimodal Communication in Human-Robot Collaboration. In: Piazza, C., Capsi-Morales, P., Figueredo, L., Keppler, M., Schütze, H. (eds) Human-Friendly Robotics 2023. HFR 2023. Springer Proceedings in Advanced Robotics, vol 29. Springer, Cham. https://doi.org/10.1007/978-3-031-55000-3_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-55000-3_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-54999-1
Online ISBN: 978-3-031-55000-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)