skip to main content
10.1145/3472307.3484675acmconferencesArticle/Chapter ViewAbstractPublication PageshaiConference Proceedingsconference-collections
poster

Towards a Cognitive Framework for Multimodal Person Recognition in Multiparty HRI

Published: 09 November 2021 Publication History

Abstract

The ability to recognize human partners is an important social skill to build personalized and long-term Human-Robot Interactions (HRI). However, in HRI contexts, unfolding in ever-changing and realistic environments, the identification problem presents still significant challenges. Possible solutions consist of relying on a multimodal approach and making robots learn from their first-hand sensory data. To this aim, we propose a framework to allow robots to autonomously organize their sensory experience into a structured dataset suitable for person recognition during a multiparty interaction. Our results demonstrate the effectiveness of our approach and show that it is a promising solution in the quest of making robots more autonomous in their learning process.

References

[1]
Giulia Belgiovine, Francesco Rea, Pablo Barros, Jacopo Zenzeri, and Alessandra Sciutti. 2020. Sensing the Partner: Toward Effective Robot Tutoring in Motor Skill Learning. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 12483 LNAI. Springer Science and Business Media Deutschland GmbH, ”, 296–307. https://doi.org/10.1007/978-3-030-62056-1_25
[2]
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. 2016. Simple online and realtime tracking. In 2016 IEEE international conference on image processing (ICIP). IEEE, IEEE, New York, NY, USA, 3464–3468.
[3]
Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. 2018. VGGFace2: A dataset for recognising faces across pose and age. In Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018. Institute of Electrical and Electronics Engineers Inc., New York, NY, USA, 67–74. https://doi.org/10.1109/FG.2018.00020 arxiv:1710.08092
[4]
Nikhil Churamani, Paul Anton, Marc Brügger, Erik Fließwasser, Thomas Hummel, Julius Mayer, Waleed Mustafa, Hwei Geok Ng, Thi Linh Chi Nguyen, Quan Nguyen, Marcus Soll, Sebastian Springenberg, Sascha Griffiths, Stefan Heinrich, Nicolás Navarro-Guerrero, Erik Strahl, Johannes Twiefel, Cornelius Weber, and Stefan Wermter. 2017. The Impact of Personalisation on Human-Robot Interaction in Learning Scenarios. In Proceedings of the 5th International Conference on Human Agent Interaction (Bielefeld, Germany) (HAI ’17). Association for Computing Machinery, New York, NY, USA, 171–180. https://doi.org/10.1145/3125739.3125756
[5]
Jonas Gonzalez-Billandon, Giulia Belgiovine, Matthew Tata, Alessandra Sciutti, Giulio Sandini, and Francesco Rea. 2021. Self-supervised learning framework for speaker localisation with a humanoid robot. In 2021 IEEE International Conference on Development and Learning (ICDL),. IEEE, ”, 1–7. https://doi.org/10.1109/ICDL49984.2021.9515566
[6]
Jonas Gonzalez-Billandon, Alessandra Sciutti, Matthew Tata, Giulio Sandini, and Francesco Rea. 2020. Audiovisual cognitive architecture for autonomous learning of face localisation by a Humanoid Robot. In 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, ”, 5979–5985.
[7]
Manuel Gunther, Steve Cruz, Ethan M Rudd, and Terrance E Boult. 2017. Toward open-set face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE, ”, 71–80.
[8]
Guodong Guo and Na Zhang. 2019. A survey on deep learning based face recognition. Computer Vision and Image Understanding 189 (dec 2019), 102805. https://doi.org/10.1016/j.cviu.2019.102805
[9]
M. Günther, P. Hu, C. Herrmann, C. H. Chan, M. Jiang, S. Yang, A. R. Dhamija, D. Ramanan, J. Beyerer, J. Kittler, M. Al Jazaery, M. I. Nouyed, G. Guo, C. Stankiewicz, and T. E. Boult. 2017. Unconstrained Face Detection and Open-Set Face Recognition Challenge. In 2017 IEEE International Joint Conference on Biometrics (IJCB). IEE, New-york,NY,USA, 697–706. https://doi.org/10.1109/BTAS.2017.8272759
[10]
Bahar Irfan, Natalia Lyubova, M Garcia Ortiz, and Tony Belpaeme. 2018. Multi-modal open-set person identification in hri. In 2018 ACM/IEEE International Conference on Human-Robot Interaction Social. IEEE, ”.
[11]
Giorgio Metta, Paul Fitzpatrick, and Lorenzo Natale. 2006. YARP: Yet another robot platform. https://doi.org/10.5772/5761
[12]
Giorgio Metta, Lorenzo Natale, Francesco Nori, Giulio Sandini, David Vernon, Luciano Fadiga, Claes von Hofsten, Kerstin Rosander, Manuel Lopes, José Santos-Victor, Alexandre Bernardino, and Luis Montesano. 2010. The iCub humanoid robot: An open-systems platform for research in cognitive development. Neural Networks 23, 8-9 (oct 2010), 1125–1134. https://doi.org/10.1016/J.NEUNET.2010.08.010
[13]
Tony J Prescott, Daniel Camilleri, Uriel Martinez-Hernandez, Andreas Damianou, and Neil D Lawrence. 2019. Memory and mental time travel in humans and social robots. Philosophical Transactions of the Royal Society B 374, 1771(2019), 20180025.
[14]
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. 2021. SpeechBrain: A General-Purpose Speech Toolkit. arxiv:2106.04624 [eess.AS] arXiv:2106.04624.
[15]
Alessandro Roncone, Ugo Pattacini, Giorgio Metta, and Lorenzo Natale. 2016. A Cartesian 6-DoF Gaze Controller for Humanoid Robots. In Robotics: science and systems, Vol. 2016. Robotics: science and systems, New-york,NY,USA.
[16]
Giulio Sandini, Vishwanathan Mohan, Alessandra Sciutti, and Pietro Morasso. 2018. Social Cognition for Human-Robot Symbiosis—Challenges and Building Blocks. Frontiers in neurorobotics 12 (2018), 34.
[17]
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, New-york,NY,USA, 815–823.
[18]
Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, and Peter Corke. 2018. The limits and potentials of deep learning for robotics. The International Journal of Robotics Research 37, 5 (2018), 405–420. https://doi.org/10.1177/0278364918770733
[19]
Dávid Sztahó, György Szaszák, and András Beke. 2019. Deep learning methods in speaker recognition: a review. arxiv:1911.06615 [eess.AS]
[20]
Adriana Tapus, Cristian Ţăpuş, and Maja J Matarić. 2008. User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intelligent Service Robotics 1, 2 (2008), 169–183.
[21]
Alessia Vignolo, Alessandra Sciutti, and John Michael. 2020. Using Robot Adaptivity to Support Learning in Child-Robot Interaction. In International Conference on Social Robotics. Springer, Springer, ”, 428–439.
[22]
Kilian Q Weinberger and Lawrence K Saul. 2009. Distance metric learning for large margin nearest neighbor classification.Journal of machine learning research 10, 2 (2009), 68–69.

Cited By

View all
  • (2024)A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social EnvironmentsACM Transactions on Human-Robot Interaction10.1145/365703013:4(1-50)Online publication date: 29-Apr-2024
  • (2022)Towards the Deployment of a Social Robot at an Elderly Day Care FacilitySocial Robotics10.1007/978-3-031-24670-8_25(277-287)Online publication date: 13-Dec-2022

Index Terms

  1. Towards a Cognitive Framework for Multimodal Person Recognition in Multiparty HRI
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        HAI '21: Proceedings of the 9th International Conference on Human-Agent Interaction
        November 2021
        447 pages
        ISBN:9781450386203
        DOI:10.1145/3472307
        Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 09 November 2021

        Check for updates

        Author Tags

        1. multimodality
        2. multiparty human-robot interaction
        3. person recognition

        Qualifiers

        • Poster
        • Research
        • Refereed limited

        Funding Sources

        Conference

        HAI '21
        Sponsor:
        HAI '21: International Conference on Human-Agent Interaction
        November 9 - 11, 2021
        Virtual Event, Japan

        Acceptance Rates

        Overall Acceptance Rate 121 of 404 submissions, 30%

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)9
        • Downloads (Last 6 weeks)3
        Reflects downloads up to 18 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)A Survey of Multimodal Perception Methods for Human–Robot Interaction in Social EnvironmentsACM Transactions on Human-Robot Interaction10.1145/365703013:4(1-50)Online publication date: 29-Apr-2024
        • (2022)Towards the Deployment of a Social Robot at an Elderly Day Care FacilitySocial Robotics10.1007/978-3-031-24670-8_25(277-287)Online publication date: 13-Dec-2022

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media