Skip to main content
SpringerLink
Log in

A comparative study of Autistic Children Emotion recognition based on Spatio-Temporal and Deep analysis of facial expressions features during a Meltdown Crisis

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The recognition of human emotion is a significant contribution to many computer vision appli-cations. Despite its importance, this work is the first one towards an automatic Autistic Children emotion recognition system to ensure their security during meltdown crisis. The current solutions to handle a meltdown crisis are based on a preventive approach. Indeed, Meltdown symptoms are determined by abnormal facial expressions related to compound emotions. To provide for this correspondence, we experimentally evaluate, in this paper, hand-crafted Geometric Spatio-Temporal and Deep features of realistic autistic children facial expressions. Towards this end, we compared the Compound Emotion Recognition (CER) performance for different combinations of these features, and we determined the features that best distinguish a Compound Emotion (CE) of autistic children during a meltdown crisis from the normal state. We used “Meltdown crisis”1 dataset to conduct our experiments on realistic Meltdown / Normal scenarios of autistic children. In this evaluation, we show that the gathered features can lead to very encouraging performances through the use of Random Forest classifier (91.27%) with hand-crafted features. Moreover, classifiers trained on deep features from InceptionResnetV2 show higher performance (97.5%) with supervised learning techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Child’s eyes are covered for privacy

References

  1. Ahmed W, Mitra S, Chanda K, Mazumdar D (2013) Assisting the autistic with improved facial expression recognition from mixed expressions. In: 2013 Fourth national conference on computer vision, pattern recognition, image processing and graphics (NCVPRIPG). IEEE, pp 1–4

  2. Alaoui AEK (2017) Mathématiques pour l’enseignement: master 1. MEEF Ellipses

  3. Anwar S, Milanova M (2016) Real time face expression recognition of children with autism. In: IAEMR

  4. Application FB (2016) https://developer.microsoft.com/en-us/windows/Kinect

  5. Bennie M (2016) Tantrum vs autistic meltdown: What is the difference. Autism Awareness

  6. Chollet F (2017) Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  7. CIM-10: (2008). https://icd.who.int/browse10/2008/fr#/I

  8. Control D (2018) Prevention. https://www.cdc.gov/

  9. Dettmers T (2019) A full hardware guide to deep learning

  10. Documentation KS (2016) https://docs.microsoft.com/en-us/previous-versions/windows/kinect/

  11. DSM-V (2013) http://www.psychomedia.qc.ca/dsm-5/2013-05-22/guide-psychomedia

  12. Du S, Martinez AM (2015) Compound facial expressions of emotion: from basic research to clinical applications. Dialog Clin Neurosci 17(4):443

    Article  Google Scholar 

  13. Du S, Tao Y, Martinez AM (2014) Compound facial expressions of emotion. Proc Natl Acad Sci 111(15):E1454–E1462

    Article  Google Scholar 

  14. Durmuṡoġlu A, Kahraman Y (2016) Facial expression recognition using geometric features. In: 2016 International conference on systems, signals and image processing (IWSSIP). IEEE, pp 1–5

  15. Ekman P (2009) Lie catching and microexpressions. The philosophy of deception, pp 118–133

  16. Eyes F (2018) http://comprendrelautisme.com/lautisme/les-signes-de-lautisme/

  17. Grimaces of anger, S (2018). https://www.cairn.info

  18. Guha T, Yang Z, Ramakrishna A, Grossman RB, Hedley D, Lee S, Narayanan SS (2015) On quantifying facial expression-related atypicality of children with autism spectrum disorder. In: 2015 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 803–807

  19. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior JCSJ, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26391–26403

    Article  Google Scholar 

  20. Hall MA, Holmes G (2003) Benchmarking attribute selection techniques for discrete class data mining. IEEE Trans Knowl Data Eng 15(6):1437–1447

    Article  Google Scholar 

  21. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10–18

    Article  Google Scholar 

  22. Haque MIU, Valles D (2018) A facial expression recognition approach using dcnn for autistic children to identify emotions. In: 2018 IEEE 9Th annual information technology, electronics and mobile communication conference (IEMCON). IEEE, pp 546–551

  23. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  24. He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. In: European conference on computer vision. Springer, pp 630–645

  25. Hira ZM, Gillies DF (2015) A review of feature selection and feature extraction methods applied on microarray data Advances in bioinformatics 2015

  26. Ho TK (1995) Random decision forests. In: Proceedings of 3rd international conference on document analysis and recognition, vol 1. IEEE, pp 278–282

  27. Howard A, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  28. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  29. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size. arXiv:1602.07360

  30. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167

  31. Jaiswal S, Valstar MF, Gillott A, Daley D (2017) Automatic detection of adhd and asd from expressive behaviour in rgbd data. In: 2017 12Th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, pp 762–769

  32. Karkra JS, Kaur J (2016) Compound facial expression recognition through gabor filter and rbf network. International Journal of Computer Science and Mobile Computing, pp 576–583

  33. Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  34. LaPlante D, Ambady N (2000) Multiple messages: Facial recognition advantage for compound expressions. J Nonverbal Behav 24(3):211–224

    Article  Google Scholar 

  35. LeCun Y, Jackel L, Bottou L, Cortes C, Denker JS, Drucker H, Guyon I, Muller UA, Sackinger E, Simard P et al (1995) Learning algorithms for classification: a comparison on handwritten digit recognition. Neural Netw Stat Mech Perspect 261:276

    Google Scholar 

  36. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  37. Liawatimena S, Heryadi Y, Trisetyarso A, Wibowo A, Abbas BS, Barlian E, et al. (2018) A fish classification on images using transfer learning and matlab. In: 2018 Indonesian association for pattern recognition international conference (INAPR). IEEE, pp 108–112

  38. Liliana D, Basaruddin T, Widyanto M (2018) Mixed facial emotion recognition using active appearance model and hidden conditional random fields. Int J Pure Appl Math 118:3159–3167

    Google Scholar 

  39. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision. Springer, pp 740–755

  40. Martinez AM (2017) Visual perception of facial expressions of emotion. Curr Opin Psychol 17:27–33

    Article  Google Scholar 

  41. Masmoudi M, Jarraya SK, Hammami M (2019) Meltdowncrisis: Dataset of autistic children during meltdown crisis. In: 2019 15Th international conference on signal-image technology & internet-based systems (SITIS). IEEE, pp 239–246

  42. Mehmood I, Ullah A, Muhammad K, Deng DJ, Meng W, Al-Turjman F, Sajjad M, De Albuquerque VHC (2019) Efficient image recognition and retrieval on iot-assisted energy-constrained platforms from big data repositories. IEEE Internet Things J 6(6):9246–9255

    Article  Google Scholar 

  43. or crying by closing the eyes H (2018) https://www.asperansa.org/prob/_sensoriels.html

  44. or crying by closing the eyes H open the mouth: (2018). http://www.siwadam.com/hmm/a3.html

  45. Powers DM (2011) Evaluation: from precision, recall and f-measure to roc. informedness markedness and correlation

  46. Redmon J, Farhadi A (2017) Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7263–7271

  47. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv:1804.02767

  48. Saeys Y, Inza I, Larrañaga P (2007) A review of feature selection techniques in bioinformatics. Bioinformatics 23(19):2507–2517

    Article  Google Scholar 

  49. Sainath TN, Vinyals O, Senior A, Sak H (2015) Convolutional, long short-term memory, fully connected deep neural networks. In: 2015 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 4580–4584

  50. Sajjad M, Zahir S, Ullah A, Akhtar Z, Muhammad K (2019) Human behavior understanding in big multimedia data using cnn based facial expression recognition. Mob Netw Appl:1–11

  51. Salmam FZ, Madani A, Kissi M (2018) Emotion recognition from facial expression based on fiducial points detection and using neural network. Int J Electr Comput Eng 8(1):52

    Google Scholar 

  52. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  53. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  54. Smiles N (2018) http://www.autistessansfrontieres.com/lenfant-autiste-ou-tsa/

  55. Srivastava RK, Greff K, Schmidhuber J (2015) Highway networks. arXiv:1505.00387

  56. Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system-a case study. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 132–137

  57. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  58. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826

  59. Tan M, Chen B, Pang R, Vasudevan V, Sandler M, Howard A, Le QV (2019) Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2820–2828

  60. Tasnim N, Khwaja A, Rashid H (2018) Emotion recognition from facial expression of autism spectrum disordered children using image processing and machine learning algorithms. Ph.D. thesis, BRAC University

  61. Villacampa O (2015) Feature selection and classification methods for decision making: a comparative analysis

  62. Wang S, Chen CS, Rinsurongkawong V, Akdag F, Eick CF (2010) A polygon-based methodology for mining related spatial datasets. In: Proceedings of the 1st ACM SIGSPATIAL international workshop on data mining for geoinformatics. ACM, pp 1–8

  63. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833

  64. Zhang Y, Wang S, Phillips P, Ji G (2014) Binary pso with mutation operator for feature selection using decision tree applied to spam detection. Knowl-Based Syst 64:22–31

    Article  Google Scholar 

  65. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  66. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multiorientation fusion histogram. Mathematical Problems in Engineering 2017

  67. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognitin, pp 2921–2929

Download references

Acknowledgements

We wish to express our gratitude and appreciation to the staff and autistic children parents of “ASSAADA” center for their unconditional support and help. Our special gratitude goes to Mr. Zouhir Fourti the head of “ASSAADA” center for providing administrative services and facilities.

Author information

Authors and Affiliations

  1. Mir@cl Laboratory, University of Sfax, Sfax, Tunisia

    Salma Kammoun Jarraya, Marwa Masmoudi & Mohamed Hammami

  2. CS Department, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, KSA, Saudi Arabia

    Salma Kammoun Jarraya

  3. Department of Computer Science, Faculty of Science, Sfax University, Sfax, Tunisia

    Mohamed Hammami

Authors
  1. Salma Kammoun Jarraya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marwa Masmoudi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Hammami
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marwa Masmoudi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Features from dataset are available from the corresponding author on reasonable request

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jarraya, S.K., Masmoudi, M. & Hammami, M. A comparative study of Autistic Children Emotion recognition based on Spatio-Temporal and Deep analysis of facial expressions features during a Meltdown Crisis. Multimed Tools Appl 80, 83–125 (2021). https://doi.org/10.1007/s11042-020-09451-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09451-y

Keywords

Search

Navigation