Abstract
Could a robot feel authentic empathy? What exactly is empathy, and why do most humans have it? We present a model which suggests that empathy is an emergent behavior with four main elements: a mirror neuron system, somatosensory cortices, an insula, and infant-directed “baby talk” or motherese. To test our hypothesis, we implemented a robot called MEI (multimodal emotional intelligence) with these functions, and allowed it to interact with human caregivers using comfort and approval motherese, the first kinds of vocalizations heard by infants at 3 and 6 months of age. The robot synchronized in real-time to the humans through voice and movement dynamics, while training statistical models associated with its low level gut feeling (“flourishing” or “distress”, based on battery or temperature). Experiments show that the post-interaction robot associates novel happy voices with physical flourishing 90 % of the time, sad voices with distress 84 % of the time. Our results also show that a robot trained with infant-directed “attention bids” can recognize adult fear voices. Importantly, this is the first emotion system to recognize adult emotional voices after training only with motherese, suggesting that this specific parental behavior may help build emotional intelligence.
Similar content being viewed by others
References
de Waal FB (2007) Putting the altruism back into altruism: the evolution of empathy. Ann Rev Psychol 59:1–22
Shamay-Tsoory SG, Aharon-Peretz J, Perry D (2009) Two systems for empathy: a double dissociation between emotional and cognitive empathy in inferior frontal gyrus versus ventromedial prefrontal lesions. Brain 132(3):617–627
Baron-Cohen S, Wheelwright S (2004) The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. J Autism Develop Disord 34(2):163–175
Juslin PN, Sloboda JA (2010) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UK
Clore GL, Palmer J (2009) Affective guidance of intelligent agents: how emotion controls cognition. Cognit Syst Res 10(1):21–30
Gonsior B, Sosnowski S, Buß M, Wollherr D, Kuhnlenz K (2012) An emotional adaption approach to increase helpfulness towards a robot. In: 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 2429–2436. IEEE
Cramer H, Goddijn J, Wielinga B, Evers V (2010) Effects of (in) accurate empathy and situational valence on attitudes towards robots. In: Proceedings of the 5th ACM/IEEE international conference on human–robot interaction, pp 141–142. IEEE, Piscataway, NJ, USA
Leite I, Pereira A, Castellano G, Mascarenhas S, Martinho C, Paiva A (2012) Modelling empathy in social robotic companions. In: Ardissono L, Kuflik T (eds) Advances in user modeling. Springer, Heidelberg, pp 135–147
Pereira A, Leite I, Mascarenhas S, Martinho C, Paiva A (2011) Using empathy to improve human–robot relationships. In: Human–robot personal relationships. Springer, Berlin, pp 130–138
Riek LD, Rabinowitch TC, Chakrabarti B, Robinson P (2009) Empathizing with robots: fellow feeling along the anthropomorphic spectrum. IEEE affective computing and intelligent interaction and workshops, pp 1–6
Kwak SS, Kim Y, Kim E, Shin C, Cho K (2013) What makes people empathize with an emotional robot? The impact of agency and physical embodiment on human empathy for a robot. In: RO-MAN, 2013 IEEE, pp 180–185. IEEE
Turkle S (2012) Alone together: why we expect more from technology and less from each other. Basic Books, New York
Vesquez JD (1997) Modeling emotions and other motivations in synthetic agents. In: Proceedings of AAAI-97. AAAI, Providence, RI, USA, pp 1015
Yamada T, Hashimoto H, Tosa N (1995) Pattern recognition of emotion with neural network. In Proceedings of IECON, pp 183–187
Breazeal CL (2004) Designing sociable robots with CDROM. MIT press, Cambridge, MA
Mehrabian A (1996) Pleasure-arousal-dominance: a general framework for describing and measuring individual differences in temperament. Curr Psychol Develop Learn Person Soc 14:261–292
Nagai Y, Rohlfing J (2009) Computational analysis of motionese toward scaffolding robot action learning. IEEE Trans Auton Mental Develop 1(1):44–54
Ishihara H, Yoshikawa Y, Asada M (2011) Realistic child robot Affetto for understanding the caregiver–child attachment relationship that guides the child development. Develop Learn (ICDL) 2:1–5
Asada M, Hosoda K, Kuniyoshi Y, Ishiguro H, Inui T, Yoshikawa Y, Ogino M, Yoshida C (2009) Cognitive developmental robotics: a Survey. IEEE Trans Auton Mental Dev 11:12–34
Kozima H, Nakagawa C, Yano H (2004) Can a robot empathize with people? Artific Life Robot 8(1):83–88
Shic F, Scassellati B (2006) How not to evaluate a developmental system. International joint conference on neural networks (IJCNN), Vancouver, 2006, pp 5218–5225
Hatfield E, Cacioppo JT (1994) Emotional contagion. Cambridge University Press, Cambridge, UK
Baron-Cohen S (2011) Zero degrees of empathy: a new theory of human cruelty. Penguin, London
Edele A, Dziobek I, Keller M (2013) Explaining altruistic sharing in the dictator game: the role of affective empathy, cognitive empathy, and justice sensitivity. Learn Individ Differ 24:96–102
Preston SD, de Waal F (2002) Empathy: its ultimate and proximate bases. Behav Brain Sci 25(01):1–20
Neumann R, Strack F (2000) Mood contagion: the automatic transfer of mood between persons. J Personal Soc Psychol 79(2):211–223
Juslin PN, Simon L, Daniel V, Lars-Olov L (2010) How does music evoke emotions? Exploring the underlying mechanisms. In: Juslin PN, Sloboda JA (eds) Handbook of music and emotion: theory, research, applications. Oxford University Press, Oxford, UK
Iacoboni M (2009) Imitation, empathy, and mirror neurons. Ann Rev Psychol 60(1):653–670
Iacoboni M, Dapretto M (2006) The mirror neuron system and the consequences of its dysfunction. Nat Rev Neurosci 7(12):942–951
Nummenmaa L, Hirvonen J, Parkkola R, Hietanen JK (2008) Is emotional contagion special? An fMRI study on neural systems for affective and cognitive empathy. NeuroImage 43(3):571–580
di Pellegrino G, Fadiga L, Fogassi L, Gallese V, Rizzolatti G (1992) Understanding motor events: a neurophysiological study. Exper Brain Res 91(1):176–180
Carr L, Iacoboni M, Dubeau MC, Mazziotta JC, Lenzi GL (2003) Neural mechanisms of empathy in humans: a relay from neural systems for imitation to limbic areas. Proc Nat Acad Sci 100(9):5497–5502
Rizzolatti G, Craighero L (2004) The mirror neuron system. Ann Rev Neurosci 27(1):169–192
Kohler E (2002) Hearing sounds, understanding actions: action representation in mirror neurons. Science 297(5582):846–848
Koelsch S, Fritz T, von Cramon DY, Müller K, Friederici AD (2006) Investigating emotion with music: an fMRI study. Hum Brain Mapp 27(3):239–250
Damasio AR (2004) Looking for Spinoza: joy, sorrow and the feeling brain. Random House, New York
Dinstein I (2008) Human cortex: reflections of mirror neurons. Curr Biol 18(20):R956–R959
Butti C, Hof PR (2010) The insular cortex: a comparative perspective. Brain Struct Funct 214(5–6):477–493
Craig AD (2009) How do you feel—now? The anterior insula and human awareness. Nat Rev Neurosci 10:59–70
Mutschler I, Schulze-Bonhage A, Glauche V, Demandt E, Speck O, Ball T (2007) A rapid sound–action association effect in human insular cortex. PLoS ONE 2(2):e259
Adolphs R, Tranel D, Damasio AR (2003) Dissociable neural systems for recognizing emotions. Brain Cognit 52(1):61–69
Calder AJ, Keane J, Manes F, Antoun N, Young AW (2000) Impaired recognition and experience of disgust following brain injury. Nat Neurosci 3(11):1077–1078
Morrison I, Bjornsdotter M, Olausson H (2011) Vicarious responses to social touch in posterior insular cortex are tuned to pleasant caressing speeds. J Neurosci 31(26):9554–9562
Kim P, Feldman R, Mayes LC, Eicher V, Thompson N, Leckman JF, Swain JE (2011) Breastfeeding, brain activation to own infant cry, and maternal sensitivity. J Child Psychol Psychiatry 52(8):907–915
Pohl A, Anders S, Schulte-Rüther M, Mathiak K, Kircher T (2013) Positive facial affect—an fMRI study on the involvement of insula and amygdala. PLoS ONE 8(8):e69886
Singer T (2004) Empathy for pain involves the affective but not sensory components of pain. Science 303(5661):1157–1162
Bernhardt BC, Singer T (2012) The neural basis of empathy. Ann Rev Neurosci 35:1–23
Adolphs R, Damasio H, Tranel D, Cooper G, Damasio AR (2000) A role for somatosensory cortices in the visual recognition of emotion as revealed by three-dimensional lesion mapping. J Neurosci 20(7):2683–2690
Damasio A (1994) Descartes’ error: emotion, reason and the human mind. Grossett/Putnam, New York
Davis M, Whalen PJ (2001) The amygdala: vigilance and emotion. Mol Psychiatry 6(1):13–34
Hein G, Singer T (2008) I feel how you feel but not always: the empathic brain and its modulation. Curr Opin Neurobiol 18(2):153–158
Fellous JM, Ledoux JE (2005) Toward basic principles for emotional processing: what the fearful brain tells the robot. In: Fellous JM, Arbib MA (eds) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UK
Walker-Andrews AS (1997) Infants’ perception of expressive behaviors: differentiation of multimodal information. Psychol Bull 121(3):437
Grossmann T, Striano T, Friederici AD (2006) Crossmodal integration of emotional information from face and voice in the infant brain. Develop Sci 9(3):309–315
Boucenna S, Gaussier P, Andry P, Hafemeister L (2010) Imitation as a communication tool for online facial expression learning and recognition. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), 2010, pp 5323–5328
Watanabe A, Ogino M, Asada M (2007) Mapping facial expression to internal states based on intuitive parenting. J Robot Mechatron 19(3):315
Saint-Georges C, Chetouani M, Cassel R, Apicella F, Mahdhaoui A, Muratori F, Laznik MC, Cohen D (2013) Motherese in interaction at the cross-road of emotion and cognition? (a systematic review). PloS ONE 8(10):e78–e103
Soken NH, Pick AD (1992) Intermodal perception of happy and angry expressive behaviors by seven-month-old infants. Child Develop 63(4):787–795
Fernald A (1989) Intonation and communicative intent in mothers’ speech to infants: is the melody the message? Child Dev 60(6):1497–1510
Trainor LJ, Austin CM, Desjardins RN (2000) Is infant-directed speech prosody a result of the vocal expression of emotion? Psychol Sci 11(3):188–195
Gleason JB (1977) Talking to children: some notes on feedback. In: Snow CE, Ferguson CA (eds) Talking to children: language input and acquisition. Cambridge University Press, Cambridge, UK, pp 199–205
Fernald A, Simon T (1984) Expanded intonation contours in mothers’ speech to newborns. Develop Psychol 20(1):104
Reissland N, Stephenson T (1999) Turn-taking in early vocal interaction: a comparison of premature and term infants vocal interaction with their mothers. Child: care health develop 25(6):447–456
Best CT (1997) Accommodation in mean f, during mother–infant and father–infant vocal interactions: a longitudinal case study. J Child Lang 4(7):9–736
Lim A, Ogata T, Okuno HG (2012) Towards expressive musical robots: a cross-modal framework for emotional gesture, voice and music. EURASIP J Audio Speech Music Process 2012(1):1–12
Lim A, Ogata T, Okuno HG (2011) Converting emotional voice to motion for robot telepresence. In: 11th IEEE-RAS international conference on humanoid robots (humanoids), 2011, pp 472–479. IEEE
Sievers B, Polansky L, Casey M, Wheatley T (2013) Music and movement share a dynamic structure that supports universal expressions of emotion. Proc Nat Acad Sci 110(1):70–75
Fellous JM, Arbib MA (2005) Who needs emotions? The brain meets the robot. Oxford University Press, Oxford, UK
Lim A, Okuno HG (2014) The MEI robot: towards using Motherese to develop multimodal emotional intelligence. IEEE Trans Auton Mental Develop 6(2):126–138
Schwarz G (1978) Estimating the dimension of a model. Ann Statist 6(2):461–464
Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Hoboken, NJ, USA
Kitamura C, Lam C (2009) Age-specific preferences for infant-directed affective intent. Infancy 14(1):77–100
Fernald A (1993) Approval and disapproval: infant responsiveness to vocal affect in familiar and unfamiliar languages. Child Dev 64(3):657–674
Asada M, Nagai Y, Ishihara H (2012) Why not artificial sympathy? In: International conference on social robotics, pp 278–287
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lim, A., Okuno, H.G. A Recipe for Empathy. Int J of Soc Robotics 7, 35–49 (2015). https://doi.org/10.1007/s12369-014-0262-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12369-014-0262-y