Skip to main content
SpringerLink
Log in

Emotional State of Children with ASD and Intellectual Disabilities: Perceptual Experiment and Automatic Recognition by Video, Audio and Text Modalities

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14338))

Included in the following conference series:

  • 475 Accesses

Abstract

The paper presents the results of perceptual experiments (by humans) and automatic recognition of the emotional states of children with Autism Spectrum Disorders (ASD) and Intellectual Disabilities (ID) by video, audio and text modalities. The participants of the study were 50 children aged 5 - 15 years: 25 children with ASD, 25 children with ID, and 20 adults - the participants of the perceptual experiment. Automatic analysis of facial expression by video was performed using FaceReader software runs on the Microsoft Azure cloud platform and convolutional neural network. Automatic recognition of the emotional states of children by speech was carried out using a recurrent neural network. This study was conducted in accordance with the design developed in the study of the recognition of the emotional states of children with Down syndrome by facial expression, voice, and text. The results of the perceptual experiment showed a greater accuracy in recognizing the emotional states of children with ASD and ID in comparison with automatic classification. The emotions of children with ASD are more accurately recognized by the video modality, children with ID - by the text modality. The novelty of the research is the comparative results for groups of children with similar and overlapping symptoms of ASD and ID, and in setting tasks related to the analysis of the emotional sphere of children with atypical development.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Schadenberg, B.R., Reidsma, D., Heylen, D.K.J., Evers, V.: Differences in spontaneous interactions of autistic children in an interaction with an adult and humanoid robot. Front. Robot. AI 7(28), 1–19 (2020)

    Google Scholar 

  2. Garg, R., et al.: The last decade of HCI research on children and voice-based conversational agents. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022), Article 149, pp. 1–19. New York, NY, USA (2022)

    Google Scholar 

  3. Scassellati, B., et al.: Improving social skills in children with ASD using a long-term, in-home social robot. Sci. Robot. 3(21), eaat7544 (2018)

    Article  Google Scholar 

  4. Leung, F.Y.N., et al.: Emotion recognition across visual and auditory modalities in autism spectrum disorder: a systematic review and meta-analysis. Dev. Rev. 63(1), 101000 (2022)

    Article  Google Scholar 

  5. Vandevelde, S., et al.: The scale for emotional development-revised (SED-R) for persons with intellectual disabilities and mental health problems: development, description, and reliability. Int. J. Dev. Disabil. 62(1), 11–23 (2016)

    Article  Google Scholar 

  6. Sterkenburg, P.S., et al.: Scale of emotional development–short: reliability and validity in two samples of children with an intellectual disability. Res. Dev. Disabil. 108, 103821 (2021). https://doi.org/10.1016/j.ridd.2020.103821

    Article  Google Scholar 

  7. Fridenson-Hayo, S., et al.: Basic and complex emotion recognition in children with autism: cross-cultural findings. Mol. Autism 7, 52 (2016)

    Article  Google Scholar 

  8. Russell, J.A., Bachorowski, J.-A., Fernández-Dols, J.-M.: Facial and vocal expressions of emotion. Annu. Rev. Psychol. 54(1), 329–349 (2003). https://doi.org/10.1146/annurev.psych.54.101601.145102

    Article  Google Scholar 

  9. Wing, L.: The definition and prevalence of autism: a review. Eur. Child Adolesc. Psychiatry 2(1), 61–74 (1993)

    Article  Google Scholar 

  10. Jacques, C., Courchesne, V., Mineau, S., Dawson, M., Mottron, L.: Positive, negative, neutral or unknown? The perceived valence of emotions expressed by young autistic children in a novel context suited to autism. Autism 26(7), 1833–1848 (2022)

    Article  Google Scholar 

  11. des Portes, V.: Intellectual disability. In: Handbook of Clinical Neurology, vol. 174, pp. 113–126 (2020)

    Google Scholar 

  12. Frolova, O., Lyakso, E.: Communication skills of preschool children with mental retardation and developmental disorders. In: Abstract book of 19th European conference on Developmental Psychology. ECDP - 2019, p. 159, Greece, Athens (2019)

    Google Scholar 

  13. Frolova, O.V., Lyakso, E.E.: Perceptual features of speech and vocalizations of 5–8 years old children with autism spectrum disorders and intellectual disabilities: recognition of the child’s gender, age and state. In: Proceedings of International congress, Neuroscience for Medicine and Psychology, p. 486, Sudak, Russia (2020)

    Google Scholar 

  14. Lyakso, E., Frolova, O., Nikolaev, A.: Voice and speech features as diagnostic symptom. In: Pracana, C., Wang, M. (eds.) Psychological Applications and Trends, pp. 259–363. Science Press, Lisboa, Portugal (2021)

    Google Scholar 

  15. Lyakso, E., et al.: Recognition of the emotional state of children with down syndrome by video, audio and text modalities: human and automatic. LNAI 13721, 438–450 (2022)

    Google Scholar 

  16. Lyakso, E., Frolova, O., Kleshnev, E., Ruban, N., Mekala, M., Arulalan, K.V.: Approbation of the Child’s Emotional Development Method (CEDM). In: Companion Publication of the 2022 International Conference on Multimodal Interaction (ICMI ‘22 Companion), pp. 201–210. New York, NY, USA (2022)

    Google Scholar 

  17. Frolova, O., Kleshnev, E., Grigorev, A., Filatova, Y., Lyakso, E.: Assessment of the emotional sphere of children with typical development and autism spectrum disorders based on an interdisciplinary approach. Hum. Physiol. 49(3), 216–224 (2023)

    Article  Google Scholar 

  18. Likert, R.: A technique for the measurement of attitudes. Arch. Psychol. 22, 5–55 (1932)

    Google Scholar 

  19. Md Juremi, N.R., Zulkifley, M.A., Hussain, A., Zaki, W.M.D.: Inter-rater reliability of actual tagged emotion categories validation using Cohen’s Kappa coefficient. J. Theor. Appl. Inf. Technol. 95, 259–264 (2017)

    Google Scholar 

  20. Bobicev, V., Sokolova, M.: Inter-annotator agreement in sentiment analysis: machine learning perspective. In: Recent Advances in Natural Language Processing Meet Deep Learning, pp. 97–102. Varna, Bulgaria (2017)

    Google Scholar 

  21. Ekman, P.: Basic emotions. In: Dalgleish, T., Power M.J. (eds.) Handbook of Cognition and Emotion, pp. 45–60. Wiley, Hoboken (1999)

    Google Scholar 

  22. FFmpeg. https://ffmpeg.org. Accessed 13 Jul 2023

  23. Multi-task Cascaded Convolutional Networks (MTCNN) via Deepface. https://github.com/serengil/deepface. Accessed 13 Jul 2023

  24. Kaggle facial expression recognition challenge in 2013. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge. Accessed 13 Jul 2023

  25. Audacity. https://www.audacityteam.org. Accessed 13 Jul 2023

  26. Korobov, M.: Morphological analyzer and generator for Russian and Ukrainian languages. Anal. Images Soc. Netw. Texts 542, 320–332 (2015)

    Article  Google Scholar 

  27. LinisCrowd 2015 tone dictionary. http://linis-crowd.org/. Accessed 13 Jul 2023

  28. Dalianis, H: Evaluation Metrics and Evaluation, pp. 45–53. Springer, Cham (2018).https://doi.org/10.1007/978-3-319-78503-5_6

  29. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  MATH  Google Scholar 

  30. Matveev, Y., Lyakso, E., Matveev, A., Frolova, O., Grigorev, A., Nikolaev, A.: Automatic classification of the emotional state of atypically developing children. In: Proceedings of the 24th International Congress of Acoustics, ABS-0338, pp. 1–7. Gyeongju, Korea (2022). https://ica2022korea.org/

  31. Marchi, E., et al.: Typicality and emotion in the voice of children with autism spectrum condition: evidence across three languages. In: Interspeech, pp. 115–119. Dresden, Germany (2015)

    Google Scholar 

  32. Landowska, A., et al.: Automatic emotion recognition in children with autism: a systematic literature review. Sensors (Basel) 22(4), 1649 (2022)

    Article  Google Scholar 

  33. Wishart, J.G., Cebula, K.R., Willis, D.S., Pitcairn, T.K.: Understanding of facial expressions of emotion by children with intellectual disabilities of differing aetiology. J. Intellect. Disabil. Res. 51(Pt 7), 551–563 (2007)

    Article  Google Scholar 

  34. Hammann, T., et al.: The challenge of emotions — an experimental approach to assess the emotional competence of people with intellectual disabilities. Disabilities 2, 611–625 (2022)

    Article  Google Scholar 

  35. Barabanschikov, V.A., Korolkova, O.A., Lobodinskaya, E.A.: Perception of facial expressions during masking and apparent motion. Exp. Psychol. 8(1), 7–27 (2015)

    Google Scholar 

  36. Ambadar, Z., Schooler, J.W., Cohn, J.F.: Deciphering the enigmatic face: the importance of facial dynamics in interpreting subtle facial expressions. Psychol. Sci. 16(5), 403–410 (2005)

    Article  Google Scholar 

  37. Barabanschikov, V.A., Suvorova, E.V.: Human emotional state assessment based on a video portrayal. Exp. Psychol. 13(4), 4–24 (2020)

    Google Scholar 

  38. Lyakso, E.E., Frolova, O.V., Grigorev, A.S., Sokolova, V.D., Yarotskaya, K.A.: Recognition by adults of emotional state in typically developing children and children with autism spectrum disorders. Neurosci. Behav. Physiol. 47(9), 1051–1059 (2017)

    Article  Google Scholar 

  39. Pell, M.D., Kotz, S.A.: On the time course of vocal emotion recognition. PLoS ONE 6(11), e27256 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

This study is financially supported by the Russian Science Foundation (project 22–45-02007) - for Russian researches, DST/INT/RUS/RSF/P-57/2021 – for Indian researches.

Author information

Authors and Affiliations

  1. The Child Speech Research Group, St. Petersburg State University, St. Petersburg, Russia

    Elena Lyakso, Olga Frolova, Aleksandr Nikolaev, Severin Grechanyi, Anton Matveev & Yuri Matveev

  2. ITMO University, St. Petersburg, Russia

    Olesia Makhnytkina

  3. School of Electrical Engineering, Vellore Institute of Technology, Vellore, India

    Ruban Nersisson

Authors
  1. Elena Lyakso
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olga Frolova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aleksandr Nikolaev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Severin Grechanyi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anton Matveev
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yuri Matveev
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Olesia Makhnytkina
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ruban Nersisson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elena Lyakso .

Editor information

Editors and Affiliations

  1. St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia

    Alexey Karpov

  2. Koneru Lakshmaiah Education Foundation, Vaddeswaram, India

    K. Samudravijaya

  3. Indian Institute of Information Technology Dharwad, Dharwad, India

    K. T. Deepak

  4. Indian Institute of Technology Dharwad, Dharwad, India

    Rajesh M. Hegde

  5. KIIT Group of Colleges, Gurugram, India

    Shyam S. Agrawal

  6. Indian Institute of Technology Dharwad, Dharwad, India

    S. R. Mahadeva Prasanna

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lyakso, E. et al. (2023). Emotional State of Children with ASD and Intellectual Disabilities: Perceptual Experiment and Automatic Recognition by Video, Audio and Text Modalities. In: Karpov, A., Samudravijaya, K., Deepak, K.T., Hegde, R.M., Agrawal, S.S., Prasanna, S.R.M. (eds) Speech and Computer. SPECOM 2023. Lecture Notes in Computer Science(), vol 14338. Springer, Cham. https://doi.org/10.1007/978-3-031-48309-7_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-48309-7_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-48308-0

  • Online ISBN: 978-3-031-48309-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation