Skip to main content
Springer Nature Link
Log in

A Reinforcement Learning Based Cognitive Empathy Framework for Social Robots

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Robots that express human’s social norms, like empathy, are perceived as more friendly, understanding, and caring. However, appropriate human-like empathic behaviors cannot be defined in advance, instead, they must be learned through daily interaction with humans in different situations. Additionally, to learn and apply the correct behaviors, robots must be able to perceive and understand the affective states of humans. This study presents a framework to enable cognitive empathy in social robots, which uses facial emotion recognition to perceive and understand the affective states of human users. The perceived affective state is then provided to a reinforcement learning model to enable a robot to learn the most appropriate empathic behaviors for different states. The proposed framework has been evaluated through an experiment between 28 individual humans and the humanoid robot Pepper. The results show that by applying empathic behaviors selected by the employed learning model, the robot is able to provide participants comfort and confidence and help them enjoy and feel better.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The vocabulary set contained ordinary items, like pen, apple, tape, spoon, pencil, camera, socks, and scissor.

References

  1. Adams RG, Blieszner R, De Vries B (2000) Definitions of friendship in the third age: age, gender, and study location effects. J Aging Stud 14(1):117–133

    Article  Google Scholar 

  2. Asada M (2015) Development of artificial empathy. Neurosci Res 90:41–50

    Article  Google Scholar 

  3. Bagheri E, Bagheri A, Esteban PG, Vanderborgth B (2020) A Novel Model for Emotion Detection from Facial Muscles Activity. In: Silva M, Luís Lima J, Reis L, Sanfeliu A, Tardioli D. (eds) Robot 2019: Fourth Iberian Robotics Conference. ROBOT 2019. Advances in Intelligent Systems and Computing, vol 1093. Springer, Cham, pp 237–249. https://doi.org/10.1007/978-3-030-36150-1_20

  4. Bagheri E, Esteban PG, Cao HL, De Beir A, Lefeber D, Vanderborght B An autonomous cognitive empathy model responsive to users’ facial emotion expressions. ACM Trans Interact Intell Syst (TIIS) (in press)

  5. Bemelmans R, Gelderblom GJ, Jonker P, De Witte L (2012) Socially assistive robots in elderly care: a systematic review into effects and effectiveness. J Am Med Dir Assoc 13(2):114–120

    Article  Google Scholar 

  6. Bentall RP, Haddock G, Slade PD (1994) Cognitive behavior therapy for persistent auditory hallucinations: from theory to therapy. Behav Ther 25(1):51–66

    Article  Google Scholar 

  7. Brave S, Nass C, Hutchinson K (2005) Computers that care: investigating the effects of orientation of emotion exhibited by an embodied computer agent. Int J Hum Comput Stud 62(2):161–178

    Article  Google Scholar 

  8. Brose SW, Weber DJ, Salatin BA, Grindle GG, Wang H, Vazquez JJ, Cooper RA (2010) The role of assistive robotics in the lives of persons with disability. Am J Phys Med Rehabil 89(6):509–521

    Article  Google Scholar 

  9. Cao HL, Esteban P, Bartlett M, Baxter PE, Belpaeme T, Billing E, Cai H, Coeckelbergh M, Costescu C, David D (2019) Robot-enhanced therapy: development and validation of a supervised autonomous robotic system for autism spectrum disorders therapy. IEEE Robot Autom Mag 26(2):49–58

    Article  Google Scholar 

  10. Churamani N, Barros P, Strahl E, Wermter S (2018) Learning empathy-driven emotion expressions using affective modulations. In: 2018 international joint conference on neural networks (IJCNN). IEEE, pp 1–8

  11. Cikara M, Bruneau E, Van Bavel JJ, Saxe R (2014) Their pain gives us pleasure: How intergroup dynamics shape empathic failures and counter-empathic responses. J Exp Soc Psychol 55:110–125

    Article  Google Scholar 

  12. Coeckelbergh M, Pop C, Simut R, Peca A, Pintea S, David D, Vanderborght B (2016) A survey of expectations about the role of robots in robot-assisted therapy for children with ASD: ethical acceptability, trust, sociability, appearance, and attachment. Sci Eng Ethics 22(1):47–65

    Article  Google Scholar 

  13. Davis MH (1983) Measuring individual differences in empathy: evidence for a multidimensional approach. J Pers Soc Psychol 44(1):113

    Article  Google Scholar 

  14. Davis MH (2006) “Empathy” handbook of the sociology of emotions. Springer, New York, pp 443–466

    Book  Google Scholar 

  15. Diette GB, Lechtzin N, Haponik E, Devrotes A, Rubin HR (2003) Distraction therapy with nature sights and sounds reduces pain during flexible bronchoscopy: a complementary approach to routine analgesia. Chest 123(3):941–948

    Article  Google Scholar 

  16. Ekman P (2002) Facial action coding system (FACS). A human face. https://ci.nii.ac.jp/naid/10025007347/en/

  17. Goldberg LR (1992) The development of markers for the big-five factor structure. Psychol Assess 4(1):26

    Article  Google Scholar 

  18. Hatfield E, Cacioppo J, Rapson R (1994) Emotional contagion: Cambridge studies in emotion and social interaction. Cambridge University Press, Cambridge. errors-in-variables regression model when the variances of the measurement errors vary between the observations. Statistics in Medicine, vol. 21, pp. 1089–1101

  19. Heerink M, Kröse B, Evers V, Wielinga B (2010) Assessing acceptance of assistive social agent technology by older adults: the almere model. Int J Soc Robot 2(4):361–375

    Article  Google Scholar 

  20. Hoffman ML (1991) Empathy, social cognition, and moral action. In: Kurtines WL, Gewirtz JL, Lamp JL (eds) Handbook of moral behavior and development. Bd, vol 1, Taylor & Francis, pp 275–301

  21. Kachouie R, Sedighadeli S, Khosla R, Chu M-T (2014) Socially assistive robots in elderly care: a mixed-method systematic literature review. Int J Hum Comput Interact 30(5):369–393

    Article  Google Scholar 

  22. Legaspi R, Kurihara S, Fukui K.-i, Moriyama K, Numao M (2008) An empathy learning problem for HSI: to be empathic, self-improving and ambient. In: 2008 conference on human system interactions. IEEE, pp 209–214

  23. Leite I, Martinho C, Paiva A (2013) Social robots for long-term interaction: a survey. Int J Soc Robot 5(2):291–308

    Article  Google Scholar 

  24. Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A (2011) Modelling empathy in social robotic companions. In: International conference on user modeling, adaptation, and personalization. Springer, pp 135–147

  25. Leite I, Pereira A, Mascarenhas S, Martinho C, Prada R, Paiva A (2013) The influence of empathy in human–robot relations. Int J Hum Comput Stud 71(3):250–260

    Article  Google Scholar 

  26. McQuiggan SW, Lester JC (2006) Learning empathy: a data-driven framework for modeling empathetic companion agents. In: Proceedings of the fifth international joint conference on autonomous agents and multiagent systems. ACM, pp 961–968

  27. McQuiggan SW, Robison JL, Phillips R, Lester JC (2008) Modeling parallel and reactive empathy in virtual agents: an inductive approach. In: Proceedings of the 7th international joint conference on autonomous agents and multiagent systems-volume 1. International foundation for autonomous agents and multiagent systems, pp 167–174

  28. Mehrabian A (2017) Nonverbal communication. Routledge, New York

    Book  Google Scholar 

  29. Niculescu A, van Dijk B, Nijholt A, Li H, See SL (2013) Making social robots more attractive: the effects of voice pitch, humor and empathy. Int J Soc Robot 5(2):171–191

    Article  Google Scholar 

  30. Pandey AK, Gelin R (2018) A mass-produced sociable humanoid robot: pepper: the first machine of its kind. IEEE Robot Autom Mag 25(3):40–48

    Article  Google Scholar 

  31. Platt FW, Keller VF (1994) Empathic communication. J Gen Intern Med 9(4):222–226

    Article  Google Scholar 

  32. Prendinger H, Ishizuka M (2005) The empathic companion: a character-based interface that addresses users’affective states. Applied Artif Intell 19(3–4):267–285

    Article  Google Scholar 

  33. Rasool Z, Masuyama N, Islam MN, Loo CK (2015) Empathic interaction using the computational emotion model. In: IEEE symposium series on computational intelligence. IEEE 2015, pp 109–116

  34. Rodrigues SH, Mascarenhas SF, Dias J, Paiva A (2009) “i can feel it too!”: Emergent empathic reactions between synthetic characters. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, pp 1–7

  35. Shaw LL, Batson CD, Todd RM (1994) Empathy avoidance: forestalling feeling for another in order to escape the motivational consequences. J Pers Soc Psychol 67(5):879

    Article  Google Scholar 

  36. Sidner CL, Kidd CD, Lee C, Lesh N (2004) Where to look: a study of human–robot engagement. In: Proceedings of the 9th international conference on intelligent user interfaces. ACM, pp 78–84

  37. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  38. Tapus A, Mataric MJ (2008) Socially assistive robots: the link between personality, empathy, physiological signals, and task performance. In: AAAI spring symposium: emotion, personality, and social behavior, pp 133–140

  39. Weisz E, Zaki J (2018) Motivated empathy: a social neuroscience perspective. Curr Opin Psychol 24:67–71

    Article  Google Scholar 

  40. Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Burlington

    Google Scholar 

  41. Zhang A (2017) Speech recognition (version 3.8). https://github.com/Uberi/speech_recognition/

Download references

Acknowledgements

The work leading to these results has received funding from Flanders Make Proud (PROgramming by User Demonstration) and the Flemish Government under the program Onderzoeksprogramma Artificiele Intelligentie (AI) Vlaanderen.

Author information

Authors and Affiliations

  1. Robotics and Multibody Mechanics Research Group, Vrije Universiteit Brussel and Flanders Make, Brussels, Belgium

    Elahe Bagheri, Hoang-Long Cao & Bram Vanderborght

  2. Artificial Intelligence Lab, Vrije Universiteit Brussel, Brussels, Belgium

    Oliver Roesler

Authors
  1. Elahe Bagheri
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Oliver Roesler
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Hoang-Long Cao
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Bram Vanderborght
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Elahe Bagheri.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bagheri, E., Roesler, O., Cao, HL. et al. A Reinforcement Learning Based Cognitive Empathy Framework for Social Robots. Int J of Soc Robotics 13, 1079–1093 (2021). https://doi.org/10.1007/s12369-020-00683-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-020-00683-4

Keywords