Skip to main content
Log in

A Recipe for Empathy

Integrating the Mirror System, Insula, Somatosensory Cortex and Motherese

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

Could a robot feel authentic empathy? What exactly is empathy, and why do most humans have it? We present a model which suggests that empathy is an emergent behavior with four main elements: a mirror neuron system, somatosensory cortices, an insula, and infant-directed “baby talk” or motherese. To test our hypothesis, we implemented a robot called MEI (multimodal emotional intelligence) with these functions, and allowed it to interact with human caregivers using comfort and approval motherese, the first kinds of vocalizations heard by infants at 3 and 6 months of age. The robot synchronized in real-time to the humans through voice and movement dynamics, while training statistical models associated with its low level gut feeling (“flourishing” or “distress”, based on battery or temperature). Experiments show that the post-interaction robot associates novel happy voices with physical flourishing 90 % of the time, sad voices with distress 84 % of the time. Our results also show that a robot trained with infant-directed “attention bids” can recognize adult fear voices. Importantly, this is the first emotion system to recognize adult emotional voices after training only with motherese, suggesting that this specific parental behavior may help build emotional intelligence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. http://www.aldebaran-robotics.com.

  2. http://www.merriam-webster.com/dictionary/authentic.

  3. http://www.hark.jp.

  4. http://pascal.kgw.tu-berlin.de/emodb/.

References

  1. de Waal FB (2007) Putting the altruism back into altruism: the evolution of empathy. Ann Rev Psychol 59:1–22

    Google Scholar 

  2. Shamay-Tsoory SG, Aharon-Peretz J, Perry D (2009) Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain 132(3):617–627

    Article  Google Scholar 

  3. Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. J Autism Develop Disord 34(2):163–175

    Article  Google Scholar 

  4. Juslin PN, Sloboda JA (2010) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UK

    Google Scholar 

  5. Clore GL, Palmer J (2009) Affective guidance of intelligent agents: how emotion controls cognition. Cognit Syst Res 10(1):21–30

    Article  Google Scholar 

  6. Gonsior B, Sosnowski S, Buß M, Wollherr D, Kuhnlenz K (2012) An emotional adaption approach to increase helpfulness towards a robot. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2429–2436. IEEE

  7. Cramer H, Goddijn J, Wielinga B, Evers V (2010) Effects of (in) accurate empathy and situational valence on attitudes towards robots. In: Proceedings of the 5th ACM/IEEE international conference on human–robot interaction, pp 141–142. IEEE, Piscataway, NJ, USA

  8. Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A (2012) Modelling empathy in social robotic companions. In: Ardissono L, Kuflik T (eds) Advances in user modeling. Springer, Heidelberg, pp 135–147

  9. Pereira A, Leite I, Mascarenhas S, Martinho C, Paiva A (2011) Using empathy to improve human–robot relationships. In: Human–robot personal relationships. Springer, Berlin, pp 130–138

  10. Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) Empathizing with robots: fellow feeling along the anthropomorphic spectrum. IEEE affective computing and intelligent interaction and workshops, pp 1–6

  11. Kwak SS, Kim Y, Kim E, Shin C, Cho K (2013) What makes people empathize with an emotional robot? The impact of agency and physical embodiment on human empathy for a robot. In: RO-MAN, 2013 IEEE, pp 180–185. IEEE

  12. Turkle S (2012) Alone together: why we expect more from technology and less from each other. Basic Books, New York

  13. Vesquez JD (1997) Modeling emotions and other motivations in synthetic agents. In: Proceedings of AAAI-97. AAAI, Providence, RI, USA, pp 1015

  14. Yamada T, Hashimoto H, Tosa N (1995) Pattern recognition of emotion with neural network. In Proceedings of IECON, pp 183–187

  15. Breazeal CL (2004) Designing sociable robots with CDROM. MIT press, Cambridge, MA

    Google Scholar 

  16. Mehrabian A (1996) Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol Develop Learn Person Soc 14:261–292

    Article  MathSciNet  Google Scholar 

  17. Nagai Y, Rohlfing J (2009) Computational analysis of motionese toward scaffolding robot action learning. IEEE Trans Auton Mental Develop 1(1):44–54

    Article  Google Scholar 

  18. Ishihara H, Yoshikawa Y, Asada M (2011) Realistic child robot Affetto for understanding the caregiver–child attachment relationship that guides the child development. Develop Learn (ICDL) 2:1–5

    Google Scholar 

  19. Asada M, Hosoda K, Kuniyoshi Y, Ishiguro H, Inui T, Yoshikawa Y, Ogino M, Yoshida C (2009) Cognitive developmental robotics: a Survey. IEEE Trans Auton Mental Dev 11:12–34

  20. Kozima H, Nakagawa C, Yano H (2004) Can a robot empathize with people? Artific Life Robot 8(1):83–88

    Article  Google Scholar 

  21. Shic F, Scassellati B (2006) How not to evaluate a developmental system. International joint conference on neural networks (IJCNN), Vancouver, 2006, pp 5218–5225

  22. Hatfield E, Cacioppo JT (1994) Emotional contagion. Cambridge University Press, Cambridge, UK

    Google Scholar 

  23. Baron-Cohen S (2011) Zero degrees of empathy: a new theory of human cruelty. Penguin, London

    Google Scholar 

  24. Edele A, Dziobek I, Keller M (2013) Explaining altruistic sharing in the dictator game: the role of affective empathy, cognitive empathy, and justice sensitivity. Learn Individ Differ 24:96–102

    Article  Google Scholar 

  25. Preston SD, de Waal F (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(01):1–20

    Google Scholar 

  26. Neumann R, Strack F (2000) Mood contagion: the automatic transfer of mood between persons. J Personal Soc Psychol 79(2):211–223

  27. Juslin PN, Simon L, Daniel V, Lars-Olov L (2010) How does music evoke emotions? Exploring the underlying mechanisms. In: Juslin PN, Sloboda JA (eds) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UK

    Google Scholar 

  28. Iacoboni M (2009) Imitation, empathy, and mirror neurons. Ann Rev Psychol 60(1):653–670

    Article  Google Scholar 

  29. Iacoboni M, Dapretto M (2006) The mirror neuron system and the consequences of its dysfunction. Nat Rev Neurosci 7(12):942–951

    Article  Google Scholar 

  30. Nummenmaa L, Hirvonen J, Parkkola R, Hietanen JK (2008) Is emotional contagion special? An fMRI study on neural systems for affective and cognitive empathy. NeuroImage 43(3):571–580

    Article  Google Scholar 

  31. di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exper Brain Res 91(1):176–180

    Article  Google Scholar 

  32. Carr L, Iacoboni M, Dubeau MC, Mazziotta JC, Lenzi GL (2003) Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas. Proc Nat Acad Sci 100(9):5497–5502

  33. Rizzolatti G, Craighero L (2004) The mirror neuron system. Ann Rev Neurosci 27(1):169–192

    Article  Google Scholar 

  34. Kohler E (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297(5582):846–848

    Article  Google Scholar 

  35. Koelsch S, Fritz T, von Cramon DY, Müller K, Friederici AD (2006) Investigating emotion with music: an fMRI study. Hum Brain Mapp 27(3):239–250

    Article  Google Scholar 

  36. Damasio AR (2004) Looking for Spinoza: joy, sorrow and the feeling brain. Random House, New York

  37. Dinstein I (2008) Human cortex: reflections of mirror neurons. Curr Biol 18(20):R956–R959

    Article  Google Scholar 

  38. Butti C, Hof PR (2010) The insular cortex: a comparative perspective. Brain Struct Funct 214(5–6):477–493

    Article  Google Scholar 

  39. Craig AD (2009) How do you feel—now? The anterior insula and human awareness. Nat Rev Neurosci 10:59–70

    Article  Google Scholar 

  40. Mutschler I, Schulze-Bonhage A, Glauche V, Demandt E, Speck O, Ball T (2007) A rapid sound–action association effect in human insular cortex. PLoS ONE 2(2):e259

    Article  Google Scholar 

  41. Adolphs R, Tranel D, Damasio AR (2003) Dissociable neural systems for recognizing emotions. Brain Cognit 52(1):61–69

    Article  Google Scholar 

  42. Calder AJ, Keane J, Manes F, Antoun N, Young AW (2000) Impaired recognition and experience of disgust following brain injury. Nat Neurosci 3(11):1077–1078

    Article  Google Scholar 

  43. Morrison I, Bjornsdotter M, Olausson H (2011) Vicarious responses to social touch in posterior insular cortex are tuned to pleasant caressing speeds. J Neurosci 31(26):9554–9562

    Article  Google Scholar 

  44. Kim P, Feldman R, Mayes LC, Eicher V, Thompson N, Leckman JF, Swain JE (2011) Breastfeeding, brain activation to own infant cry, and maternal sensitivity. J Child Psychol Psychiatry 52(8):907–915

    Article  Google Scholar 

  45. Pohl A, Anders S, Schulte-Rüther M, Mathiak K, Kircher T (2013) Positive facial affect—an fMRI study on the involvement of insula and amygdala. PLoS ONE 8(8):e69886

  46. Singer T (2004) Empathy for pain involves the affective but not sensory components of pain. Science 303(5661):1157–1162

    Article  Google Scholar 

  47. Bernhardt BC, Singer T (2012) The neural basis of empathy. Ann Rev Neurosci 35:1–23

    Article  Google Scholar 

  48. Adolphs R, Damasio H, Tranel D, Cooper G, Damasio AR (2000) A role for somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping. J Neurosci 20(7):2683–2690

    Google Scholar 

  49. Damasio A (1994) Descartes’ error: emotion, reason and the human mind. Grossett/Putnam, New York

    Google Scholar 

  50. Davis M, Whalen PJ (2001) The amygdala: vigilance and emotion. Mol Psychiatry 6(1):13–34

    Article  Google Scholar 

  51. Hein G, Singer T (2008) I feel how you feel but not always: the empathic brain and its modulation. Curr Opin Neurobiol 18(2):153–158

    Article  Google Scholar 

  52. Fellous JM, Ledoux JE (2005) Toward basic principles for emotional processing: what the fearful brain tells the robot. In: Fellous JM, Arbib MA (eds) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UK

    Chapter  Google Scholar 

  53. Walker-Andrews AS (1997) Infants’ perception of expressive behaviors: differentiation of multimodal information. Psychol Bull 121(3):437

    Article  Google Scholar 

  54. Grossmann T, Striano T, Friederici AD (2006) Crossmodal integration of emotional information from face and voice in the infant brain. Develop Sci 9(3):309–315

    Article  Google Scholar 

  55. Boucenna S, Gaussier P, Andry P, Hafemeister L (2010) Imitation as a communication tool for online facial expression learning and recognition. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), 2010, pp 5323–5328

  56. Watanabe A, Ogino M, Asada M (2007) Mapping facial expression to internal states based on intuitive parenting. J Robot Mechatron 19(3):315

    Google Scholar 

  57. Saint-Georges C, Chetouani M, Cassel R, Apicella F, Mahdhaoui A, Muratori F, Laznik MC, Cohen D (2013) Motherese in interaction at the cross-road of emotion and cognition? (a systematic review). PloS ONE 8(10):e78–e103

    Article  Google Scholar 

  58. Soken NH, Pick AD (1992) Intermodal perception of happy and angry expressive behaviors by seven-month-old infants. Child Develop 63(4):787–795

    Article  Google Scholar 

  59. Fernald A (1989) Intonation and communicative intent in mothers’ speech to infants: is the melody the message? Child Dev 60(6):1497–1510

  60. Trainor LJ, Austin CM, Desjardins RN (2000) Is infant-directed speech prosody a result of the vocal expression of emotion? Psychol Sci 11(3):188–195

    Article  Google Scholar 

  61. Gleason JB (1977) Talking to children: some notes on feedback. In: Snow CE, Ferguson CA (eds) Talking to children: language input and acquisition. Cambridge University Press, Cambridge, UK, pp 199–205

    Google Scholar 

  62. Fernald A, Simon T (1984) Expanded intonation contours in mothers’ speech to newborns. Develop Psychol 20(1):104

    Article  Google Scholar 

  63. Reissland N, Stephenson T (1999) Turn-taking in early vocal interaction: a comparison of premature and term infants vocal interaction with their mothers. Child: care health develop 25(6):447–456

    Google Scholar 

  64. Best CT (1997) Accommodation in mean f, during mother–infant and father–infant vocal interactions: a longitudinal case study. J Child Lang 4(7):9–736

    Google Scholar 

  65. Lim A, Ogata T, Okuno HG (2012) Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music. EURASIP J Audio Speech Music Process 2012(1):1–12

    Article  Google Scholar 

  66. Lim A, Ogata T, Okuno HG (2011) Converting emotional voice to motion for robot telepresence. In: 11th IEEE-RAS international conference on humanoid robots (humanoids), 2011, pp 472–479. IEEE

  67. Sievers B, Polansky L, Casey M, Wheatley T (2013) Music and movement share a dynamic structure that supports universal expressions of emotion. Proc Nat Acad Sci 110(1):70–75

    Article  Google Scholar 

  68. Fellous JM, Arbib MA (2005) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UK

    Book  Google Scholar 

  69. Lim A, Okuno HG (2014) The MEI robot: towards using Motherese to develop multimodal emotional intelligence. IEEE Trans Auton Mental Develop 6(2):126–138

  70. Schwarz G (1978) Estimating the dimension of a model. Ann Statist 6(2):461–464

    Article  MATH  MathSciNet  Google Scholar 

  71. Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Hoboken, NJ, USA

  72. Kitamura C, Lam C (2009) Age-specific preferences for infant-directed affective intent. Infancy 14(1):77–100

  73. Fernald A (1993) Approval and disapproval: infant responsiveness to vocal affect in familiar and unfamiliar languages. Child Dev 64(3):657–674

    Article  Google Scholar 

  74. Asada M, Nagai Y, Ishihara H (2012) Why not artificial sympathy? In: International conference on social robotics, pp 278–287

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Angelica Lim.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lim, A., Okuno, H.G. A Recipe for Empathy. Int J of Soc Robotics 7, 35–49 (2015). https://doi.org/10.1007/s12369-014-0262-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12369-014-0262-y

Keywords

Navigation