Skip to main content
SpringerLink
Log in

Impact of sensory preferences of individuals with autism on the recognition of emotions expressed by two robots, an avatar, and a human

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

We design a personalized human-robot environment for social learning for individuals with autism spectrum disorders (ASD). In order to define an individual’s profile, we posit that the individual’s reliance on proprioceptive and kinematic visual cues should affect the way the individual suffering from ASD interacts with a social agent (human/robot/virtual agent). In this paper, we assess the potential link between recognition performances of body/facial expressions of emotion of increasing complexity, emotion recognition on platforms with different visual features (two mini-humanoid robots, a virtual agent, and a human), and proprioceptive and visual cues integration of an individual. First, we describe the design of the EMBODI-EMO database containing videos of controlled body/facial expressions of emotions from various platforms. We explain how we validated this database with typically developed (TD) individuals. Then, we investigate the relationship between emotion recognition and proprioceptive and visual profiles of TD individuals and individuals with ASD. For TD individuals, our results indicate a relationship between profiles and emotion recognition. As expected, we show that TD individuals that rely more heavily on visual cues yield better recognition scores. However, we found that TD individuals relying on proprioception have better recognition scores, going against our hypothesis. Finally, participants with ASD relying more heavily on proprioceptive cues have lower emotion recognition scores on all conditions than participants relying on visual cues.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Database link: http://perso.ensta-paristech.fr/~tapus/eng/media/EMBODI-EMO.zip.

  2. Link to the database: http://perso.ensta-paristech.fr/~tapus/eng/media/EMBODI-EMO.zip.

References

  • Bar-Haim, Y., & Bart, O. (2006). Motor function and social participation in kindergarten children. Social Development, 15(2), 296–310.

    Article  Google Scholar 

  • Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997). Is there a” language of the eyes”? evidence from normal adults, and adults with autism or asperger syndrome. Visual Cognition, 4(3), 311–331.

    Article  Google Scholar 

  • Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “reading the mind in the eyes” test revised version: A study with normal adults, and adults with asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, 42(2), 241–251.

    Article  Google Scholar 

  • Begeer, S., Rieffe, C., Terwogt, M. M., & Stockmann, L. (2006). Attention to facial emotion expressions in children with autism. Autism, 10(1), 37–51.

    Article  Google Scholar 

  • Bondy, A. S., & Frost, L. A. (1994). The picture exchange communication system. Focus on Autism and Other Developmental Disabilities, 9(3), 1–19.

    Article  Google Scholar 

  • Bray, A., Subanandan, A., Isableu, B., Ohlmann, T., Golding, J. F., & Gresty, M. A. (2004). We are most aware of our place in the world when about to fall. Current Biology, 14(15), R609–R610.

    Article  Google Scholar 

  • Brown, C., & Dunn, W. (2002). Adolescent-adult sensory profile: User’s manual. San Antonio: Therapy Skill Builders.

    Google Scholar 

  • Buisine, S., Courgeon, M., Charles, A., Clavel, C., Martin, J. C., Tan, N., et al. (2014). The role of body postures in the recognition of emotions in contextually rich scenarios. International Journal of Human-Computer Interaction, 30(1), 52–62.

    Article  Google Scholar 

  • Burke, D., Hagbarth, K. E., Löfstedt, L., & Wallin, B. G. (1976). The responses of human muscle spindle endings to vibration of non-contracting muscles. The Journal of physiology, 261(3), 673–693.

    Article  Google Scholar 

  • Celani, G., Battacchi, M. W., & Arcidiacono, L. (1999). The understanding of the emotional meaning of facial expressions in people with autism. Journal of autism and developmental disorders, 29(1), 57–66.

    Article  Google Scholar 

  • Charman, T., Swettenham, J., Baron-Cohen, S., Cox, A., Baird, G., & Drew, A. (1997). Infants with autism: An investigation of empathy, pretend play, joint attention, and imitation. Developmental Psychology, 33(5), 781.

    Article  Google Scholar 

  • Chevalier, P., Isableu, B., Martin, J. C., & Tapus, A. (2016). Individuals with autism: Analysis of the first interaction with nao robot based on their proprioceptive and kinematic profiles. In T. Borangiu (Ed.), Advances in robot design and intelligent control (pp. 225–233). Berlin: Springer.

    Chapter  Google Scholar 

  • Chiari, L., Rocchi, L., & Cappello, A. (2002). Stabilometric parameters are affected by anthropometry and foot placement. Clinical Biomechanics, 17(9), 666–677.

    Article  Google Scholar 

  • Coates, S., Lord, M., & Jakabovics, E. (1975). Field dependence-independence, social-non-social play and sex differences in preschool children. Perceptual and Motor Skills, 40(1), 195–202.

    Article  Google Scholar 

  • Costa, S., Soares, F., Pereira, A. P., Santos, C., Hiolle, A. (2014). Building a game scenario to encourage children with autism to recognize and label emotions using a humanoid robot. In Robot and human interactive communication, 2014 RO-MAN: The 23rd IEEE international symposium on, IEEE, pp 820–825

  • Coulson, M. (2004). Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28(2), 117–139.

    Article  MathSciNet  Google Scholar 

  • Courgeon, M., & Clavel, C. (2013). Marc: A framework that features emotion models for facial animation during human-computer interaction. Journal on Multimodal User Interfaces, 7(4), 311–319.

    Article  Google Scholar 

  • Courgeon, M., Grynszpan, O., Buisine, S., Martin, J. (2012). Interactive expressive virtual characters: Challenges for conducting experimental studies about multimodal social interaction. In Proceedings of the 9th International Conference on Disability, Virtual Reality & Associated Technologies Laval, France, 10–12 Sept.

  • Dael, N., Mortillaro, M., & Scherer, K. R. (2012). Emotion expression in body action and posture. Emotion, 12(5), 1085.

    Article  Google Scholar 

  • Dautenhahn, K., Nehaniv, C. L., Walters, M. L., Robins, B., Kose-Bagci, H., Mirza, N. A., et al. (2009). Kaspar—A minimally expressive humanoid robot for human-robot interaction research. Applied Bionics and Biomechanics, 6(3–4), 369–397.

    Article  Google Scholar 

  • De Gelder, B., & Van den Stock, J. (2011). The bodily expressive action stimulus test (beast). construction and validation of a stimulus basis for measuring perception of whole body expression of emotions. Frontiers in Psychology, 2, 419.

    Article  Google Scholar 

  • Dunn W (1999) The sensory profile: Users manual (psychological corporation, San Antonio, TX)

  • Edin, B. B. (2001). Cutaneous afferents provide information about knee joint movements in humans. The Journal of Physiology, 531(1), 289–297.

    Article  Google Scholar 

  • Ekman, P., & Friesen, W. (1984). Emotion facial action coding system (em-facs). San Francisco: University of California.

    Google Scholar 

  • Erden, M. S. (2013). Emotional postures for the humanoid-robot nao. International Journal of Social Robotics, 5(4), 441–456.

    Article  Google Scholar 

  • Feil-Seifer, D., Mataric, M. J. (2005) Defining socially assistive robotics. In Rehabilitation robotics, 2005. ICORR 2005. 9th International conference on, IEEE, pp 465–468

  • Ferrell, W., Gandevia, S., & McCloskey, D. (1987). The role of joint receptors in human kinaesthesia when intramuscular receptors cannot contribute. The Journal of physiology, 386(1), 63–71.

    Article  Google Scholar 

  • Gepner, B., & Mestre, D. R. (2002). Brief report: Postural reactivity to fast visual motion differentiates autistic from children with asperger syndrome. Journal of Autism and Developmental disorders, 32(3), 231–238.

    Article  Google Scholar 

  • Gepner, B., Mestre, D., Masson, G., & de Schonen, S. (1995). Postural effects of motion vision in young autistic children. NeuroReport, 6(8), 1211–1214.

    Article  Google Scholar 

  • Golan, O., Ashwin, E., Granader, Y., McClintock, S., Day, K., Leggett, V., et al. (2010). Enhancing emotion recognition in children with autism spectrum conditions: An intervention using animated vehicles with real emotional faces. Journal of Autism and Developmental Disorders, 40(3), 269–279.

    Article  Google Scholar 

  • Goodwin, G. M., McCloskey, D. I., & Matthews, P. B. (1972). Proprioceptive illusions induced by muscle vibration: Contribution by muscle spindles to perception? Science, 175(4028), 1382–1384.

    Article  Google Scholar 

  • Gowen, E., & Hamilton, A. (2013). Motor abilities in autism: A review using a computational context. Journal of Autism and Developmental Disorders, 43(2), 323–344.

    Article  Google Scholar 

  • Greffou, S., Bertone, A., Hahler, E. M., Hanssens, J. M., Mottron, L., & Faubert, J. (2012). Postural hypo-reactivity in autism is contingent on development and visual environment: A fully immersive virtual reality study. Journal of Autism and Developmental Disorders, 42(6), 961–970.

    Article  Google Scholar 

  • Hart, M. (2005). Autism/excel study. In Proceedings of the 7th international ACM SIGACCESS conference on computers and accessibility, ACM, pp 136–141

  • Haswell, C. C., Izawa, J., Dowell, L. R., Mostofsky, S. H., & Shadmehr, R. (2009). Representation of internal models of action in the autistic brain. Nature Neuroscience, 12(8), 970–972.

    Article  Google Scholar 

  • Herbert, M. R., Ziegler, D. A., Makris, N., Filipek, P. A., Kemper, T. L., Normandin, J. J., et al. (2004). Localization of white matter volume increase in autism and developmental language disorder. Annals of Neurology, 55(4), 530–540.

    Article  Google Scholar 

  • Isableu, B., Fourre, B., Vuillerme, N., Giraudet, G., & Amorim, M. A. (2011). Differential integration of visual and kinaesthetic signals to upright stance. Experimental Brain Research, 212(1), 33–46.

    Article  Google Scholar 

  • Kim, E. S., Berkovits, L. D., Bernier, E. P., Leyzberg, D., Shic, F., Paul, R., et al. (2013). Social robots as embedded reinforcers of social behavior in children with autism. Journal of Autism and Developmental Disorders, 43(5), 1038–1049.

    Article  Google Scholar 

  • Kohen-Raz, R., Volkman, F. R., & Cohen, D. J. (1992). Postural control in children with autism. Journal of Autism and Developmental Disorders, 22(3), 419–432.

    Article  Google Scholar 

  • Kozima, H., Nakagawa, C., Yasuda, Y. (2005) Interactive robots for communication-care: A case-study in autism therapy. In Robot and human interactive communication, 2005. ROMAN 2005. IEEE International Workshop on, IEEE, pp. 341–346

  • Liu, W., & Chepyator-Thomson, J. R. (2009). Field dependence-independence and physical activity engagement among middle school students. Physical Education and Sport Pedagogy, 14(2), 125–136.

    Article  Google Scholar 

  • Meeren, H. K., van Heijnsbergen, C. C., & de Gelder, B. (2005). Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the United States of America, 102(45), 16,518–16,523.

    Article  Google Scholar 

  • Mehrabian, A. (1972). Nonverbal communication. Transaction Publishers.

  • Meltzoff, A. N., Brooks, R., Shon, A. P., & Rao, R. P. (2010). Social robots are psychological agents for infants: A test of gaze following. Neural Networks, 23(8), 966–972.

    Article  Google Scholar 

  • Qin, S., Nagai, Y., Kumagaya, S., Ayaya, S., Asada, M. (2014). Autism simulator employing augmented reality: A prototype. In Development and Learning and Epigenetic Robotics (ICDL-Epirob), 2014 Joint IEEE international conferences on, IEEE, pp 155–156

  • Roll, J., & Vedel, J. (1982). Kinaesthetic role of muscle afferents in man, studied by tendon vibration and microneurography. Experimental Brain Research, 47(2), 177–190.

    Article  Google Scholar 

  • Salter, T., Michaud, F., & Larouche, H. (2010). How wild is wild? a taxonomy to characterize the wildness of child-robot interaction. International Journal of Social Robotics, 2(4), 405–415.

    Article  Google Scholar 

  • Saracho, O. (2003). Matching teachers’ and students’ cognitive styles. Early Child Development and Care, 173(2–3), 161–173.

    Article  Google Scholar 

  • Scassellati, B., Admoni, H., & Mataric, M. (2012). Robots for use in autism research. Annual Review of Biomedical Engineering, 14, 275–294.

    Article  Google Scholar 

  • Scherer, K. R. (1995). Expression of emotion in voice and music. Journal of Voice, 9(3), 235–248.

    Article  Google Scholar 

  • Silver, M., & Oakes, P. (2001). Evaluation of a new computer intervention to teach people with autism or asperger syndrome to recognize and predict emotions in others. Autism, 5(3), 299–316.

    Article  Google Scholar 

  • Simmons, D. R., Robertson, A. E., McKay, L. S., Toal, E., McAleer, P., & Pollick, F. E. (2009). Vision in autism spectrum disorders. Vision Research, 49(22), 2705–2739.

    Article  Google Scholar 

  • Tapus, A., Mataric, M., Scassellati, B. (2007). Attention to facial emotion expressions in children with autism. In IEEE robotics and automation magazine (RAM), Special issue on Grand Challenges in Robotics, Vol. 14, No. 1

  • Tapus, A., Peca, A., Aly, A., Pop, C., Jisa, L., Pintea, S., et al. (2012). Children with autism social engagement in interaction with nao, an imitative robot-a series of single case experiments. Interaction Studies, 13(3), 315–347.

    Article  Google Scholar 

  • Thill, S., Pop, C. A., Belpaeme, T., Ziemke, T., & Vanderborght, B. (2012). Robot-assisted therapy for autism spectrum disorders with (partially) autonomous control: Challenges and outlook. Paladyn, Journal of Behavioral Robotics, 3(4), 209–217.

    Article  Google Scholar 

  • Wainer, J., Dautenhahn, K., Robins, B., Amirabdollahian, F. (2010). Collaborating with kaspar: Using an autonomous humanoid robot to foster cooperative dyadic play among children with autism (pp 631–638)

  • Wallbott, H. G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(6), 879–896.

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported by IdF Doctoral Fellowship 2013, HANDICAP theme and is done in collaboration with MAIA Autisme and FAM-La Lendemaine thanks to G. Lerigoleur and D. Mercier. We would like to thank our participants, their families, and their caretakers. We also thank Virginie Demulier for her help.

Author information

Authors and Affiliations

  1. Robotics and Computer Vision Lab, U2IS ENSTA ParisTech, Université Paris-Saclay, 828 bd des Maréchaux, 91762, Palaiseau cedex, France

    Pauline Chevalier & Adriana Tapus

  2. LIMSI, CNRS, Université Paris-Saclay, Bât 508, Campus Universitaire, 91405, Orsay, France

    Pauline Chevalier & Jean-Claude Martin

  3. CIAMS, Univ. Paris-Sud, Université Paris-Saclay, 91405, Orsay Cedex, France

    Pauline Chevalier & Brice Isableu

  4. CIAMS, Université d’Orléans, 45067, Orléans, France

    Pauline Chevalier & Brice Isableu

  5. I.M.E. Notre Ecole, 78955, Carrière sous Poissy, France

    Christophe Bazile

Authors
  1. Pauline Chevalier
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Claude Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Brice Isableu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christophe Bazile
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Adriana Tapus
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pauline Chevalier.

Additional information

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Assistive and Rehabilitation Robotics”.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chevalier, P., Martin, JC., Isableu, B. et al. Impact of sensory preferences of individuals with autism on the recognition of emotions expressed by two robots, an avatar, and a human. Auton Robot 41, 613–635 (2017). https://doi.org/10.1007/s10514-016-9575-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-016-9575-z

Keywords

Search

Navigation