Abstract
This article presents a people perception software architecture and its implementation, focused on the information of interest from the point of view of a social robot. The key modules employed to get the different people features, such as the body parts location, the face and hands information, and the speech, from a set of possible devices and configurations are described. The association and combination of these features using a temporal and geometric fusion system are explained in detail. A high-level interface for Human-Robot interaction using the resulting information from the fused people is proposed. The paper presents experimental results evaluating the relevant aspects of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Bazarevsky, V., Kartynnik, Y., Vakunov, A., Raveendran, K., Grundmann, M.: Blazeface: Sub-millisecond neural face detection on mobile gpus (2019)
Bohus, D., et al.: Platform for situated intelligence. arXiv preprint arXiv:2103.15975 (2021)
Bormann, R., Zwölfer, T., Fischer, J., Hampp, J., Hägele, M.: Person recognition for service robotics applications. In: 13th IEEE-RAS International Conference on Humanoid Robots (Humanoids), pp. 260–267 (2013)
Esteban, P.G., et al.: How to build a supervised autonomous system for robot-enhanced therapy for children with autism spectrum disorder. Paladyn J. Behav. Robot. 8(1), 18–38 (2017)
Geitgey, A.: Face recognition module (2019). https://github.com/ageitgey/face_recognition
Gomez, R., Szapiro, D., Galindo, K., Nakamura, K.: Haru: Hardware design of an experimental tabletop robot assistant. In: 2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 233–240. IEEE (2018)
Hu, N., Bormann, R., Zwölfer, T., Kröse, B.: Multi-user identification and efficient user approaching by fusing robot and ambient sensors. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 5299–5306 (2014)
Kardaris, N., Rodomagoulakis, I., Pitsikalis, V., Arvanitakis, A., Maragos, P.: A platform for building new human-computer interface systems that support online automatic recognition of audio-gestural commands. In: Proceedings of the 24th ACM international conference on Multimedia, pp. 1169–1173 (2016)
Linder, T., Arras, K.O.: People detection, tracking and visualization using ros on a mobile service robot. In: Koubaa, A. (ed.) Robot Operating System (ROS). SCI, vol. 625, pp. 187–213. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-26054-9_8
Lugaresi, C., et al.: Mediapipe: A framework for building perception pipelines (2019)
Mohamed, Y., Lemaignan, S.: Ros for human-robot interaction. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3020–3027. IEEE
Nakadai, K., Okuno, H.G., Mizumoto, T.: Development, deployment and applications of robot audition open source software hark. J. Robot. Mechatron. 29(1), 16–25 (2017)
Quigley, M., et al.: Ros: an open-source robot operating system, vol. 3 (January 2009)
Röhling, T.: Fkie message filters (2018). http://wiki.ros.org/fkie_message_filters
Stefanov, K., Huang, B., Li, Z., Soleymani, M.: Opensense: A platform for multimodal data acquisition and behavior perception. In: Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 660–664 (2020)
Stratou, G., Morency, L.P.: Multisense-context-aware nonverbal behavior analysis framework: A psychological distress use case. IEEE Trans. Affect. Comput. 8(2), 190–203 (2017)
Tsiami, A., Koutras, P., Efthymiou, N., Filntisis, P.P., Potamianos, G., Maragos, P.: Multi3: Multi-sensory perception system for multi-modal child interaction with multiple robots. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 4585–4592 (2018)
Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., André, E.: The social signal interpretation (ssi) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia, pp. 831–834 (2013)
Zhang, F., et al.: Mediapipe hands: On-device real-time hand tracking (2020)
Acknowledgment
The work of F.C. and L.M. is partially supported by Programa Operativo FEDER Andalucia 2014-2020, Consejeria de Economía y Conocimiento (DeepBot, PY20_00817) and the project PLEC2021-007868, funded by MCIN/AEI/10.13039/501100011033 and the European Union NextGenerationEU/PRTR.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ragel, R. et al. (2022). Multi-modal Data Fusion for People Perception in the Social Robot Haru. In: Cavallo, F., et al. Social Robotics. ICSR 2022. Lecture Notes in Computer Science(), vol 13817. Springer, Cham. https://doi.org/10.1007/978-3-031-24667-8_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-24667-8_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-24666-1
Online ISBN: 978-3-031-24667-8
eBook Packages: Computer ScienceComputer Science (R0)