Skip to main content
Springer Nature Link
Log in

Autism classification and monitoring from predicted categorical and dimensional emotions of video features

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Autism in children has been increasing at an alarming rate over the years, and currently 1% of children struggle with this disorder. It can be better managed via early diagnosis and treatment. Autistic children are characterised by deficiencies in communicative and social capabilities and are most commonly identified by their stimming behaviours. Therefore, it is helpful to understand their emotions when they are exhibiting this type of behaviour. However, most of the current affect recognition approaches majorly focus on predicting either exclusively on basic categories of emotion, or continuous emotions. We propose an approach which maps basic categories of emotion to continuous dimensional emotions, opening more avenues for understanding emotions of autistic children. In our approach, we first predict the basic emotion category with a convolutional neural network, followed by continuous emotion prediction by a deep regression model. Moreover, our method is deployed as a web application for visual video monitoring. For autism analysis, we performed image-based and video-based classification of stimming behaviours using the extracted behavioural and emotional features. Our emotion classifier was able to achieve a competitive F1-score, while our regression model performed excellently in terms of CCC and RMSE compared with existing methods. Image-based analysis of autism did not yield meaningful classification when using emotional features but it provided useful cues when dealing with textural features. In video-based autism analysis, our chosen clustering algorithm was able to classify stimming behaviours into different clusters, each cluster demonstrating a dominant emotion category.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Data availability

Datasets used in this paper can be accessed by getting in touch with authors of the following papers: Kollias, D., Kotsia, I., Hajiyev, E. & Zafeiriou, S. Analysing Affective Behaviour in the second ABAW2 Competition. CoRR. abs/2106.15318 (2021), https://arxiv.org/abs/2106.15318. Mollahosseini, A., Hasani, B. & Mahoor, M. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Transactions On Affective Computing. 10, 18–31 (2019,1), http://dx.doi.org/10.1109/TAFFC.2017.2740923. Rajagopalan, S., Dhall, A. & Goecke, R. Self-stimulatory behaviours in the wild for autism diagnosis. 2013 IEEE International Conference On Computer Vision Workshops. (2013).

References

  1. Ekman, P.: Basic emotions. Handbook Of Cognition And Emotion. pp. 45–60 (2005)

  2. Barrett, L., Bliss-Moreau, E.: Affect as a psychological primitive. Adv. Exp. Social Psychol. 41, 167–218 (2009)

    Article  Google Scholar 

  3. Kirby, M., Sirovich, L.: Application of the Karhunen–Loeve procedure for the characterization of human faces. IEEE Trans. Pattern Anal. Mach. Intell. 12, 103–108 (1990)

    Article  Google Scholar 

  4. Chellappa, R., Wilson, C., Sirohey, S.: Human and machine recognition of faces: a survey. Proc. IEEE 83, 705–741 (1995)

    Article  Google Scholar 

  5. Happy, S., George, A., Routray, A.A : real time facial expression classification system using Local Binary Patterns. In: 2012 4th International conference on intelligent human computer interaction (IHCI)

  6. Liang, S., Sabri, A., Alnajjar, F., Loo, C.: Autism spectrum self-stimulatory behaviors classification using explainable temporal coherency deep features and SVM classifier. IEEE Access 9, 34264–34275 (2021)

    Article  Google Scholar 

  7. Zhao, K., Chu, W., Zhang, H.: Deep region and multi-label learning for facial action unit detection. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR). pp. 3391–3399 (2016)

  8. Hasani, B., Mahoor, M.: Facial expression recognition using enhanced deep 3D convolutional neural networks. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW). pp. 2278–2288 (2017)

  9. Chao, L., Tao, J., Yang, M., Li, Y., Wen, Z.: Multi-scale temporal modeling for dimensional emotion recognition in video. In: Proceedings of the 4th international workshop on audio/visual emotion challenge. pp. 11–18 (2014)

  10. Yang, Z., Wu, B., Zheng, K., Wang, X., Lei, L.: A survey of collaborative filtering-based recommender systems for mobile internet applications. IEEE Access 4, 3273–3287 (2016)

    Article  Google Scholar 

  11. Gogna, A., Majumdar, A.: A comprehensive recommender system model: improving accuracy for both warm and cold start users. IEEE Access 3, 2803–2813 (2015)

    Article  Google Scholar 

  12. Nam, S., Kim, S., Kim, H., Yu, Y.: Comparative study of the performance of support vector machines with various kernels (2021)

  13. Landowska, A., Karpus, A., Zawadzka, T., Robins, B., Erol Barkana, D., Kose, H., Zorcec, T., Cummins, N.: Automatic emotion recognition in children with autism: a systematic literature review. Sensors 22, 1649 (2022)

    Article  Google Scholar 

  14. Ghoreishi, N., Goshvarpour, A., Zare-Molekabad, S., Khorshidi, N., Baratzade, S.: Classification of autistic children using polar-based lagged state-space indices of EEG signals. Signal Image Video Process. 15, 1805–1812 (2021)

    Article  Google Scholar 

  15. Hwooi, S., Othmani, A., Sabri, A.: Deep learning-based approach for continuous affect prediction from facial expression images in valence-arousal space. IEEE Access 10, 96053–96065 (2022)

    Article  Google Scholar 

  16. ukherjee, H., Salam, H., Othmani, A., Santosh, K.: How intense are your words? Understanding emotion intensity from speech. In: 2021 IEEE 21st international conference on communication technology (ICCT). pp. 1280–1286

  17. Ruta, D., Gabrys, B.: Classifier selection for majority voting. Inf. Fusion 6, 63–81 (2005)

    Article  Google Scholar 

  18. Derbeko, P., El-Yaniv, R., Meir, R.: Variance optimized bagging. In: European conference on machine learning. (2002)

  19. Jiang, T., Li, J., Zheng, Y., Sun, C.: Improved bagging algorithm for pattern recognition in UHF signals of partial discharges. Energies 4, 1087–1101 (2011)

    Article  Google Scholar 

  20. Ahmadzadeh, M., Petron, M., Sasikala, K.: The dempster-shafer combination rule as a tool to classifier combination. In: IGARSS 2000. IEEE 2000 international geoscience and remote sensing symposium. Taking the pulse of the planet: the role of remote sensing in managing the environment. Proceedings (Cat. No.00CH37120). 6 pp. 2429–2431 vol.6 (2000)

  21. Wolpert, D.: Stacked generalization. Neural Netw 5, 241–259 (1992)

    Article  Google Scholar 

  22. Mollahosseini, A., Hasani, B., Mahoor, M.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10, 18–31 (2019)

    Article  Google Scholar 

  23. Handrich, S., Dinges, L., Al-Hamadi, A., Werner, P., Aghbari, Z.: Simultaneous prediction of valence/arousal and emotions on AffectNet, Aff-Wild and AFEW-VA. Proc. Comput. Sci. 170, 634–641 (2020)

    Article  Google Scholar 

  24. Rajagopalan, S., Dhall, A., Goecke, R.: Self-stimulatory behaviours in the wild for autism diagnosis. In: 2013 IEEE international conference on computer vision workshops. (2013)

  25. Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Proceedings Of The 25th international conference on neural information processing systems—Volume 1. pp. 1097–1105 (2012)

  26. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)

    Article  Google Scholar 

  27. Microsoft cognitive services: Emotion API (2016) https://www.microsoft.com/cognitive-services/en-us/emotion-api

  28. Drucker, H., C, C., Kaufman, L., Smola, A., Vapnik, V.: Support vector regression machines. Advances in Neural Information Processing Systems.. 9 (2003)

  29. Kollias, D., Kotsia, I., Hajiyev, E., Zafeiriou, S.: Analysing affective behavior in the second ABAW2 Competition. CoRR. arXiv:2106.15318 (2021)

Download references

Acknowledgements

This work is funded under grant number IF040-2021 (MATCH2021: Malaysia France Bilateral Research Grant). It is also funded by the Ministry of Higher Education, Malaysia (JPT(BKPI)1000/016/018/25(58)) through Malaysia Big Data Research Excellence Consortium (BiDaREC), via the research grant managed by Universiti Malaya (Grant No.: KKP002-2021).

Funding

This work is funded under grant number IF040-2021 (MATCH2021: Malaysia France Bilateral Research Grant). It is also funded by the Ministry of Higher Education, Malaysia (JPT(BKPI)1000/016/018/25(58)) through Malaysia Big Data Research Excellence Consortium (BiDaREC), via the research grant managed by Universiti Malaya (Grant No.: KKP002-2021).

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence, Faculty of Computer Science and Information Technology, University of Malaya, 50603, Kuala Lumpur, Selangor, Malaysia

    Stephen Wen Hwooi Khor & Aznul Qalid Md Sabri

  2. LISSI, Université Paris-Est Créteil (UPEC), 94400, Vitry sur Seine, France

    Alice Othmani

Authors
  1. Stephen Wen Hwooi Khor
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Aznul Qalid Md Sabri
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Alice Othmani
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

AO and AQMS provided supervisory instruction as to the proper direction, structure, and content of the paper, while WHKs performed most of the writing and experiments under stated supervision. All authors reviewed the manuscript.

Corresponding author

Correspondence to Aznul Qalid Md Sabri.

Ethics declarations

Conflict of interest

Not applicable.

Ethical approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khor, S.W.H., Md Sabri, A.Q. & Othmani, A. Autism classification and monitoring from predicted categorical and dimensional emotions of video features. SIViP 18, 191–198 (2024). https://doi.org/10.1007/s11760-023-02699-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-023-02699-5

Keywords