Skip to main content
SpringerLink
Log in
Book cover

International Conference on Social Robotics

ICSR 2020: Social Robotics pp 36–47Cite as

HRI Physio Lib: A Software Framework to Support the Integration of Physiological Adaptation in HRI

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12483))

Abstract

The rise of available physiological sensors in recent years can be attributed to the widespread popularity of commercial sensing technologies equipped with health monitoring technology. To make more engaging human-robot interaction (HRI), social robots should have some ability to infer their social partner’s affective state. Measurements of the autonomic nervous system via non-invasive physiological sensors provide a convenient window into a person’s affective state, namely their emotions, behavior, stress, and engagement. HRI research has included physiological sensors in-the-loop, however implementations are often specific to studies, and do not lend well for reusability. To address this gap, we propose a modular, flexible and extensible framework designed to work with popular robot platforms. Our framework will be compatible with both lab and consumer grade sensors, and includes essential tools and processing algorithms for affective state estimation geared towards real-time HRI applications.

This research was undertaken, in part, thanks to funding from the Canada 150 Research Chairs Program and thanks to Alexander Graham Bell Canada Graduate Scholarships.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. EventIDE (2020). http://www.okazolab.com/

  2. iMotions: Unpack Human Behaviour (2020). https://imotions.com/

  3. Barco, A., Albo-Canals, J., Garriga, C.: Engagement based on a customization of an ipod-lego robot for a long-term interaction for an educational purpose. In: Proceeding of ACM/IEEE HRI 2014, pp. 124–125. IEEE (2014)

    Google Scholar 

  4. Batista, D., da Silva, H.P., Fred, A., Moreira, C., Reis, M., Ferreira, H.A.: Benchmarking of the bitalino biomedical toolkit against an established gold standard. Healthc. Technol. Lett. 6(2), 32–36 (2019)

    Article  Google Scholar 

  5. Bian, D., Wade, J., Swanson, A., Weitlauf, A., Warren, Z., Sarkar, N.: Design of a physiology-based adaptive virtual reality driving platform for individuals with ASD. ACM Trans. Accessible Comput. (TACCESS) 12(1), 2 (2019)

    Google Scholar 

  6. Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  7. Caine, K., Šabanovic, S., Carter, M.: The effect of monitoring by cameras and robots on the privacy enhancing behaviors of older adults. In: Proceedings of ACM/IEEE HRI, vol. 2012, pp. 343–350 (2012)

    Google Scholar 

  8. De Waal, F.B.: Putting the altruism back into altruism: the evolution of empathy. Annu. Rev. Psychol. 59, 279–300 (2008)

    Article  Google Scholar 

  9. Del Duchetto, F., Baxter, P., Hanheide, M.: Are you still with me? continuous engagement assessment from a robot’s point of view. arXiv preprint arXiv:2001.03515 (2020)

  10. Dillen, N., Ilievski, M., Law, E., Nacke, L.E., Czarnecki, K., Schneider, O.: Keep calm and ride along: passenger comfort and anxiety as physiological responses to autonomous driving styles. In: Proceeding of CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020)

    Google Scholar 

  11. D’Mello, S., et al.: A time for emoting: When affect-sensitivity is and isn’t effective at promoting deep learning. In: International Conference on Intelligent Tutoring Systems, pp. 245–254 (2010)

    Google Scholar 

  12. Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997)

    Article  Google Scholar 

  13. Fairclough, S., Gilleade, K.: Construction of the biocybernetic loop: a case study. In: Proceeding of ACM ICMI, pp. 571–578 (2012)

    Google Scholar 

  14. Fairclough, S.H.: Fundamentals of physiological computing. Interact. Comput. 21(1–2), 133–145 (2009)

    Article  Google Scholar 

  15. Fan, J., et al.: A robotic coach architecture for elder care (rocare) based on multi-user engagement models. IEEE Trans. Neural Syst. Rehabil. Eng. 25(8), 1153–1163 (2016)

    Article  Google Scholar 

  16. Fan, Y., Lu, X., Li, D., Liu, Y.: Video-based emotion recognition using CNN-RNN and c3d hybrid networks. In: Proceeding of ACM ICMI, vol. 2016, pp. 445–450 (2016)

    Google Scholar 

  17. Fiorini, L., Mancioppi, G., Semeraro, F., Fujita, H., Cavallo, F.: Unsupervised emotional state classification through physiological parameters for social robotics applications. Knowl.-Based Syst. 190, 105217 (2020)

    Article  Google Scholar 

  18. Foster, M.E., Gaschler, A., Giuliani, M.: Automatically classifying user engagement for dynamic multi-party human-robot interaction. Int. J. Soc. Robot. 9(5), 659–674 (2017)

    Article  Google Scholar 

  19. Gonzalez Billandon, J., et al.: Can a robot catch you lying? a machine learning system to detect lies during interactions. Front. Robot. AI 6, 64 (2019)

    Article  Google Scholar 

  20. Gunes, H., Piccardi, M.: Bi-modal emotion recognition from expressive face and body gestures. J. Netw. Comput. Appl. 30(4), 1334–1345 (2007)

    Article  Google Scholar 

  21. Heath, S., et al.: Spatiotemporal aspects of engagement during dialogic storytelling child-robot interaction. Front. Robot. AI 4, 27 (2017)

    Article  Google Scholar 

  22. Kaniusas, E.: Biomedical Signals and Sensors I: Linking Physiological Phenomena and Biosignals. Springer Science & Business Media, Berlin (2012)

    Book  Google Scholar 

  23. Leite, I., Henriques, R., Martinho, C., Paiva, A.: Sensors in the wild: exploring electrodermal activity in child-robot interaction. In: Proceeding of ACM/IEEE HRI 2013, pp. 41–48. IEEE (2013)

    Google Scholar 

  24. Loewe, N., Nadj, M.: Physio-adaptive systems-a state-of-the-art review and future research directions. In: ECIS (2020)

    Google Scholar 

  25. McDuff, D., Blackford, E.: iphys: an open non-contact imaging-based physiological measurement toolbox. In: 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6521–6524. IEEE (2019)

    Google Scholar 

  26. Metta, G., Fitzpatrick, P., Natale, L.: Yarp: yet another robot platform. Int. J. Adv. Robot. Syst. 3(1), 8 (2006)

    Article  Google Scholar 

  27. Motti, V.G., Caine, K.: Users’ privacy concerns about wearables. In: Brenner, M., Christin, N., Johnson, B., Rohloff, K. (eds.) FC 2015. LNCS, vol. 8976, pp. 231–244. Springer, Heidelberg (2015). https://doi.org/10.1007/978-3-662-48051-9_17

    Chapter  Google Scholar 

  28. Mukherjee, S.S., Robertson, N.M.: Deep head pose: gaze-direction estimation in multimodal video. IEEE Trans. Multimedia 17(11), 2094–2107 (2015)

    Article  Google Scholar 

  29. Muñoz, J., Rubio, E., Cameirao, M., Bermúdez, S.: The biocybernetic loop engine: an integrated tool for creating physiologically adaptive videogames. In: Proceeding of International Conference on Physiological Computing Systems, pp. 45–54 (2017)

    Google Scholar 

  30. Muñoz, J.E., Cameirão, M., Bermúdez i Badia, S., Gouveia, E.R.: Closing the loop in exergaming-health benefits of biocybernetic adaptation in senior adults. In: Proceeding of Annual Symposium on Computer-Human Interaction in Play, pp. 329–339 (2018)

    Google Scholar 

  31. Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceeding of ACM ICMI, vol. 2015, pp. 443–449 (2015)

    Google Scholar 

  32. Novak, D.: Engineering issues in physiological computing. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing. HIS, pp. 17–38. Springer, London (2014). https://doi.org/10.1007/978-1-4471-6392-3_2

    Chapter  Google Scholar 

  33. Novak, D., Mihelj, M., Munih, M.: A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing. Interact. Comput. 24(3), 154–172 (2012)

    Article  Google Scholar 

  34. Paiva, A., Leite, I., Boukricha, H., Wachsmuth, I.: Empathy in virtual agents and robots: a survey. ACM Trans. Interact. Intell. Syst. (TiiS) 7(3), 1–40 (2017)

    Article  Google Scholar 

  35. Pasquali, D., Aroyo, A.M., Gonzalez-Billandon, J., Rea, F., Sandini, G., Sciutti, A.: Your eyes never lie: a robot magician can tell if you are lying. In: Proceeding of companion of ACM/IEEE HRI, vol. 2020, pp. 392–394 (2020)

    Google Scholar 

  36. Peck, E.M., Easse, E., Marshall, N., Stratton, W., Perrone, L.F.: Flyloop: a micro framework for rapid development of physiological computing systems. In: Proceeding of ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 152–157 (2015)

    Google Scholar 

  37. Pineda, J., Hecht, E.: Mirroring and mu rhythm involvement in social cognition: are there dissociable subcomponents of theory of mind? Biol. Psychol. 80(3), 306–314 (2009)

    Article  Google Scholar 

  38. Plews, D.J., Scott, B., Altini, M., Wood, M., Kilding, A.E., Laursen, P.B.: Comparison of heart-rate-variability recording with smartphone photoplethysmography, polar h7 chest strap, and electrocardiography. Int. J. Sports Physiol. Perform. 12(10), 1324–1328 (2017)

    Article  Google Scholar 

  39. Quigley, M., et al.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5. Kobe, Japan (2009)

    Google Scholar 

  40. Renard, Y., et al.: Openvibe: an open-source software platform to design, test, and use brain-computer interfaces in real and virtual environments. Presence: Teleoperators Virtual Environ. 19(1), 35–53 (2010)

    Article  MathSciNet  Google Scholar 

  41. Rich, C., Ponsler, B., Holroyd, A., Sidner, C.L.: Recognizing engagement in human-robot interaction. In: Proceeding of ACM/IEEE HRI 2010), pp. 375–382. IEEE (2010)

    Google Scholar 

  42. Rudovic, O., Lee, J., Mascarell-Maricic, L., Schuller, B.W., Picard, R.W.: Measuring engagement in robot-assisted autism therapy: a cross-cultural study. Front. Robot. AI 4, 36 (2017)

    Article  Google Scholar 

  43. Shan, C., Gong, S., McOwan, P.W.: Beyond facial expressions: learning human emotion from body gestures. In: BMVC, pp. 1–10 (2007)

    Google Scholar 

  44. Sjak-Shie, E.E.: Physiodata toolbox (version 0.5) [computer software] (2019). https://PhysioDataToolbox.leidenuniv.nl

  45. Szafir, D., Mutlu, B.: Pay attention! designing adaptive agents that monitor and improve user engagement. In: Proceeding of IGCHI Conference on Human Factors in Computing Systems, pp. 11–20 (2012)

    Google Scholar 

  46. Tsiakas, K., Abujelala, M., Makedon, F.: Task engagement as personalization feedback for socially-assistive robots and cognitive training. Technologies 6(2), 49 (2018)

    Article  Google Scholar 

  47. Vogel, E.A.: About one-in-five americans use a smart watch or fitness tracker (2020). https://www.pewresearch.org/fact-tank/2020/01/09/about-one-in-five-americans-use-a-smart-watch-or-fitness-tracker/

Download references

Author information

Authors and Affiliations

  1. Social and Intelligent Robotics Research Lab (SIRRL), University of Waterloo, Waterloo, Canada

    Austin Kothig, John Muñoz, Hamza Mahdi, Alexander M. Aroyo & Kerstin Dautenhahn

Authors
  1. Austin Kothig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John Muñoz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hamza Mahdi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander M. Aroyo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kerstin Dautenhahn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Austin Kothig .

Editor information

Editors and Affiliations

  1. Pennsylvania State University, University Park, PA, USA

    Alan R. Wagner

  2. University of Nevada Reno, Reno, NV, USA

    David Feil-Seifer

  3. University of Denver, Denver, CO, USA

    Kerstin S. Haring

  4. University of Naples Federico II, Naples, Italy

    Silvia Rossi

  5. Colorado School of Mines, Golden, CO, USA

    Thomas Williams

  6. Wichita State University, Wichita, KS, USA

    Hongsheng He

  7. National University of Singapore, Singapore, Singapore

    Shuzhi Sam Ge

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kothig, A., Muñoz, J., Mahdi, H., Aroyo, A.M., Dautenhahn, K. (2020). HRI Physio Lib: A Software Framework to Support the Integration of Physiological Adaptation in HRI. In: Wagner, A.R., et al. Social Robotics. ICSR 2020. Lecture Notes in Computer Science(), vol 12483. Springer, Cham. https://doi.org/10.1007/978-3-030-62056-1_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62056-1_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62055-4

  • Online ISBN: 978-3-030-62056-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation