Skip to main content
SpringerLink
Log in

Recognition of the Emotional State of Children with Down Syndrome by Video, Audio and Text Modalities: Human and Automatic

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13721))

Included in the following conference series:

  • 812 Accesses

Abstract

The paper presents the results of perceptual experiments (by humans) and automatic recognition of the emotional states of children with Down syndrome (DS) by video, audio and text modalities. The participants of the study were 35 children with DS aged 5–16 years, and 30 adults – the participants of the perceptual experiment. Automatic analysis of facial expression by video was performed using FaceReader software runs on the Microsoft Azure cloud platform and convolutional neural network. Automatic recognition of the emotional states of children by speech was carried out using a recurrent neural network. Specifically for this project, we did not apply any additional transfer learning or fine-tuning as our goal was to investigate how the generic models perform for children with DS. The results of perceptual experiments showed that adults recognize the emotional states of children with DS by video better than by audio. Automatic classification of children’s emotional states by facial expression revealed better results for joy and neutral states than for sadness and anger; by audio the best results were shown for the neutral state, by the texts of children’s speech - for joy, the state of sadness was not recognized automatically. The study revealed the possibility of using the available software for classifying the neutral state and the state of joy, i.e. states with neutral and positive valence, and the need to develop an approach to determine the state of sadness and anger.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hippolyte, L., Barisnikov, K., Van der Linden, M.: Face processing and facial emotion recognition in adults with Down syndrome. Am. J. Ment. Retard. 113(4), 292–306 (2008)

    Article  Google Scholar 

  2. Carvajal, F., Fernández-Alcaraz, C., Rueda, M., Sarrión, L.: Processing of facial expressions of emotions by adults with Down syndrome and moderate intellectual disability. Res. Dev. Disabil. 33(3), 783–790 (2012)

    Article  Google Scholar 

  3. Virji-Babul, N., Watt, K., Nathoo, F., Johnson, P.: Recognition of facial expressions of emotion in adults with Down syndrome. Phys. Occup. Ther. Pediatr. 32(3), 333–343 (2012)

    Article  Google Scholar 

  4. Pochon, R., Declercq, C.: Emotion recognition by children with Down syndrome: a longitudinal study. J. Intellect. Dev. Disabil. 38(4), 332–343 (2013)

    Article  Google Scholar 

  5. Cebula, K.R., Wishart, J.G., Willis, D.S., Pitcairn, T.K.: Emotion recognition in children with Down syndrome: influence of emotion label and expression intensity. Am. J. Intellect. Dev. Disabil. 22(2), 138–155 (2017)

    Article  Google Scholar 

  6. Barisnikov, K., Thomasson, M., Stutzmann, J., Lejeune, F.: Relation between processing facial identity and emotional expression in typically developing school-age children and those with Down syndrome. Appl. Neuropsychol. Child 9(2), 179–192 (2020)

    Article  Google Scholar 

  7. Barisnikov, K., Theurel, A., Lejeune, F.: Emotion knowledge in neurotypical children and in those with down syndrome. Appl. Neuropsychol. Child 11(3), 197–211 (2022)

    Article  Google Scholar 

  8. Pochon, R., Touchet, C., Ibernon, L.: Recognition of basic emotions with and without the use of emotional vocabulary by adolescents with down syndrome. Behav. Sci. 12(6), 167 (2022). https://doi.org/10.3390/bs12060167

    Article  Google Scholar 

  9. Roch, M., Pesciarelli, F., Leo, I.: How individuals with down syndrome process faces and words conveying emotions? Evidence from a priming paradigm. Front. Psychol. 11, 692 (2020)

    Article  Google Scholar 

  10. Sorce, J.F., Emde, R.N.: The meaning of infant emotional expressions: regularities in caregiving responses in normal and Down’s syndrome infants. J. Child Psychol. Psychiatry 23(2), 145–158 (1982)

    Article  Google Scholar 

  11. Berger, J., Cunningham, C.C.: Aspects of early social smiling by infants with Down’s syndrome. Child Care Health Dev. 12(1), 13–24 (1986)

    Article  Google Scholar 

  12. Carvajal, F., Iglesias, J.: Judgements of facial and vocal signs of emotion in infants with Down syndrome. Dev. Psychobiol. 48(8), 644–652 (2006)

    Article  Google Scholar 

  13. Lyakso, E., Frolova, O., Gorodniy, V., Grigovev, A., Nikolaev, A., Matveev, Y.: Re-flection of the emotional state in the characteristics of voice and speech of children with Down syndrome. In: Proceedings SpeD 2019, 10th IEEE International Conference on Speech Technology and Human-Computer Dialogue, Timisoara, Romania, pp. 1–6 (2019)

    Google Scholar 

  14. Lyakso, E.E., Frolova, O.V., Matveev, Y.N.: Facial expression: psychophysiological study. In: Joseph, R.A.N., Mahesh, V.G.V., Nersisson, R. (eds.) Handbook of Research on Deep Learning-Based Image Analysis under Constrained and Unconstrained Environments, Hershey, PA, pp. 266–289. IGI Global (2021)

    Google Scholar 

  15. Dykens, E., Hodapp, R.M., Evans, D.W.: Profiles and development of adaptive behavior in children with Down syndrome. Am. J. Ment. Retard. 98(5), 580–587 (1994)

    Google Scholar 

  16. Carr, J.: Down’s Syndrome: Children Growing Up. Cambridge University Press, Cambridge (1995)

    Book  Google Scholar 

  17. Fidler, D.J.: The emerging Down syndrome behavioral phenotype in early childhood implications for practice. Infants Young Child. 18(2), 86–103 (2005)

    Article  Google Scholar 

  18. Izard, C.E., Youngstrom, E.A., Fine, S.E., Mostow, A.J., Trentacosta, C.J.: Emotions and developmental psychopathology. In: Cicchetti, D., Cohen, D.J. (eds.) Developmental psychopathology, pp. 244–292. Wiley, New York (2006)

    Google Scholar 

  19. Kim, E.S., Paul, R., Shic, F., Scassellati, B.: Bridging the research gap: making HRI useful to individuals with autism. J. Hum.-Robot Interact. 1, 26–54 (2012)

    Article  Google Scholar 

  20. Garg, R., et al.: The last decade of HCI research on children and voice-based conversational agents. In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI 2022), pp. 1–19. Association for Computing Machinery, New York (2022). Article: 149

    Google Scholar 

  21. Scassellati, B., et al.: Improving social skills in children with ASD using a long-term, in-home social robot. Sci. Robot. 3(21), eaat7544, 1–9 (2018)

    Google Scholar 

  22. Tsai, Y.T., Lin, W.A.: Design of an intelligent cognition assistant for people with cognitive impairment. In: IEEE 20th International Conference on High Performance Computing and Communications, 16th International Conference on Smart City and 4th International Conference on Data Science and Systems, HPCC/SmartCity/DSS 2018, Exeter, UK, pp. 1207–1212. IEEE (2018)

    Google Scholar 

  23. Bargagna, S., et al.: Educational robotics in down syndrome: a feasibility study. Technol. Knowl. Learn. 24(2), 315–323 (2018). https://doi.org/10.1007/s10758-018-9366-z

    Article  Google Scholar 

  24. González-González, C.S., et al.: Computational thinking and down syndrome: an exploratory study using the KIBO robot. Informatics 6(25), 1–20 (2019). https://doi.org/10.3390/informatics6020025

    Article  Google Scholar 

  25. Alemi, M., Bahramipour, S.: An innovative approach of incorporating a humanoid robot into teaching EFL learners with intellectual disabilities. Asian-Pac. J. Second Foreign Lang. Educ. 4, 10 (2019). https://doi.org/10.1186/s40862-019-0075-5

  26. Dalianis, H.: Evaluation metrics and evaluation. In: Dalianis, H. (ed.) Clinical Text Mining, pp. 45–53. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-78503-5_6

    Chapter  Google Scholar 

  27. Juremi, N.R.M., Zulkifley, M.A., Hussain, A., Zaki, W.M.D.: Inter-rater reliability of actual tagged emotion categories validation using Cohen’s Kappa coefficient. J. Theor. Appl. Inf. Technol. 95, 259–264 (2017)

    Google Scholar 

  28. Bobicev, V., Sokolova, M.: Inter-annotator agreement in sentiment analysis: machine learning perspective. In: Recent Advances in Natural Language Processing Meet Deep Learning, Varna, Bulgaria, pp. 97–102 (2017)

    Google Scholar 

  29. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  MATH  Google Scholar 

  30. Ekman, P.: Basic emotions. In: Dalgleish, T., Power, M.J. (eds.) Handbook of cognition and emotion, pp. 45–60. Wiley, Hoboken (1999)

    Google Scholar 

  31. Terzis, V., Moridis, C.N., Economides, A.A.: Measuring instant emotions during a self-assessment test: the use of FaceReader. In: 7th International Conference on Methods and Techniques in Behavioral Research, Eindhoven, The Netherlands, pp. 192–195 (2010)

    Google Scholar 

  32. FFmpeg. https://ffmpeg.org. Accessed 31 July 2022

  33. Multi-task Cascaded Convolutional Networks (MTCNN) via Deepface. https://github.com/serengil/deepface. Accessed 31 July 2022

  34. Kaggle facial expression recognition challenge in 2013. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge. Accessed 31 July 2022

  35. Audacity. https://www.audacityteam.org. Accessed 31 July 2022

  36. Korobov, M.: Morphological analyzer and generator for Russian and Ukrainian languages. In: Khachay, M.Y., Konstantinova, N., Panchenko, A., Ignatov, D.I., Labunets, V.G. (eds.) AIST 2015. CCIS, vol. 542, pp. 320–332. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-26123-2_31

    Chapter  Google Scholar 

  37. LinisCrowd 2015 tone dictionary. http://linis-crowd.org/. Accessed 31 July 2022

  38. Kumin, L.: Early Communication Skills for Children with Down Syndrome: A Guide for Parents and Professionals. Woodbine House, Bethesda (2003)

    Google Scholar 

  39. Sforza, C., Dellavia, C., Dolci, C., Donetti, E., Ferrario, V.F.: A quantitative three-dimensional assessment of abnormal variations in the facial soft tissues of individuals with Down syndrome. Cleft Palate-Craniofac. J. 42(4), 410–416 (2005)

    Article  Google Scholar 

  40. Zhao, Q., et al.: Automated Down syndrome detection using facial photographs. In: 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Osaka, Japan. IEEE Engineering in Medicine and Biology Society, pp. 3670–3673. IEEE (2013)

    Google Scholar 

  41. Mittal, A., Gaur, H., Mishra, M.: Detection of Down syndrome using deep facial recognition. In: Chaudhuri, B.B., Nakagawa, M., Khanna, P., Kumar, S. (eds.) Proceedings of 3rd International Conference on Computer Vision and Image Processing. AISC, vol. 1022, pp. 119–130. Springer, Singapore (2020). https://doi.org/10.1007/978-981-32-9088-4_11

    Chapter  Google Scholar 

  42. Kent, R.D., Vorperian, H.K.: Speech impairment in Down syndrome: a review. J. Speech Lang. Hear. Res. 56(1), 178–210 (2013)

    Article  Google Scholar 

  43. Kanamori, G., Witter, M., Brown, J., Williams-Smith, L.: Otolaryngologic manifestations of Down syndrome. Otolaryngol. Clin. North Am. 33(6), 1285–1292 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

This study is financially supported by the Russian Science Foundation (project 22-45-02007) for Russian researches and Department of Science and Technology (DST) (INTRUSRFBR382) - for Indian researchers.

Author information

Authors and Affiliations

  1. The Child Speech Research Group, St. Petersburg State University, St. Petersburg, Russia

    Elena Lyakso, Olga Frolova, Anton Matveev, Yuri Matveev & Aleksey Grigorev

  2. ITMO University, Saint Petersburg, 197101, Russia

    Olesia Makhnytkina

  3. School of Electrical Engineering, Vellore Institute of Technology, Vellore, India

    Nersisson Ruban

Authors
  1. Elena Lyakso
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olga Frolova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anton Matveev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuri Matveev
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aleksey Grigorev
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Olesia Makhnytkina
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Nersisson Ruban
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elena Lyakso .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Dharwad, Dharwad, India

    S. R. Mahadeva Prasanna

  2. St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia

    Alexey Karpov

  3. Koneru Lakshmaiah Education Foundation, Vaddeswaram, India

    K. Samudravijaya

  4. KIIT Group of Colleges, Gurugram, India

    Shyam S. Agrawal

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lyakso, E. et al. (2022). Recognition of the Emotional State of Children with Down Syndrome by Video, Audio and Text Modalities: Human and Automatic. In: Prasanna, S.R.M., Karpov, A., Samudravijaya, K., Agrawal, S.S. (eds) Speech and Computer. SPECOM 2022. Lecture Notes in Computer Science(), vol 13721. Springer, Cham. https://doi.org/10.1007/978-3-031-20980-2_38

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20980-2_38

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20979-6

  • Online ISBN: 978-3-031-20980-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation