Skip to main content
Log in

Integration of Facial Thermography in EEG-based Classification of ASD

  • Research Article
  • Published:
International Journal of Automation and Computing Aims and scope Submit manuscript

Abstract

Autism spectrum disorder (ASD) is a neurodevelopmental disorder affecting social, communicative, and repetitive behavior. The phenotypic heterogeneity of ASD makes timely and accurate diagnosis challenging, requiring highly trained clinical practitioners. The development of automated approaches to ASD classification, based on integrated psychophysiological measures, may one day help expedite the diagnostic process. This paper provides a novel contribution for classifing ASD using both thermographic and EEG data. The methodology used in this study extracts a variety of feature sets and evaluates the possibility of using several learning models. Mean, standard deviation, and entropy values of the EEG signals and mean temperature values of regions of interest (ROIs) in facial thermographic images were extracted as features. Feature selection is performed to filter less informative features based on correlation. The classification process utilizes Naïve Bayes, random forest, logistic regression, and multi-layer perceptron algorithms. The integration of EEG and thermographic features have achieved an accuracy of 94% with both logistic regression and multi-layer perceptron classifiers. The results have shown that the classification accuracies of most of the learning models have increased after integrating facial thermographic data with EEG.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 5th ed., London, UK: American Psychiatric Association, 2013.

    Google Scholar 

  2. G. Dawson, S. Rogers, J. Munson, M. Smith, J. Winter, J. Greenson, A. Donaldson, J. Varley. Randomized, controlled trial of an intervention for toddlers with autism: The Early Start Denver Model. Pediatrics, vol. 125, no. 1, pp. e17–e23, 2010. DOI: https://doi.org/10.1542/peds.2009-0958.

    Google Scholar 

  3. G. Dawson, E. J. H. Jones, K. Merkle, K. Venema, R. Lowy, S. Faja, D. Kamara, M. Murias, J. Greenson, J. Winter, M. Smith, S. J. Rogers, S. J. Webb. Early behavioral intervention is associated with normalized brain activity in young children with autism. Journal of the American Academy of Child & Adolescent Psychiatry, vol. 51, no. 11, pp. 1150–1159, 2012. DOI: https://doi.org/10.1016//j.jaac.2012.08.018.

    Google Scholar 

  4. G. Brihadiswaran, D. Haputhanthri, S. Gunathilaka, D. Meedeniya, S. Jayarathna. EEG-based processing and classification methodologies for autism spectrum disorder: A review. Journal of Computer Science, vol. 15, no. 8, pp. 1161–1183, 2019. DOI: https://doi.org/10.3844/jcssp.2019.1161.1183.

    Google Scholar 

  5. D. Haputhanthri, G. Brihadiswaran, S. Gunathilaka, D. Meedeniya, Y. Jayawardena, S. Jayarathna, M. Jaime. An EEG based channel optimized classification approach for autism spectrum disorder. In Proceedings of Moratuwa Engineering Research Conference, IEEE, Moratuwa, Sri Lanka, pp. 123–128, 2019. DOI: https://doi.org/10.1109/MERCon.2019.8818814.

    Google Scholar 

  6. S. D. Hill, E. A. Wagner, J. G. Shedlarski Jr., S. P. Sears. Diurnal cortisol and temperature variation of normal and autistic children. Developmental Psychobiology, vol. 10. no. 6, pp. 579–583, 1977. DOI: https://doi.org/10.1002/dev.420100612.

    Google Scholar 

  7. R. M. J. Cotterill. Fever in autistics. Nature, vol. 313, no. 6002, Article number 426, 1985. DOI: https://doi.org/10.1038/313426c0.

  8. L. K. Curran, C. J. Newschaffer, L. C. Lee, S. O. Crawford, M. V. Johnston, A. W. Zimmerman. Behaviors associated with fever in children with autism spectrum disorders. Pediatrics, vol. 120, no. 6, pp. e1386–e1392, 2007. DOI: https://doi.org/10.1542/peds.2007-0360.

    Google Scholar 

  9. C. Harshaw, M. S. Blumberg, J. R. Alberts. Thermoregulation, energetics, and behavior. In APA Handbook of Comparative Psychology: Basic Concepts, Methods, Neural Substrate, and Behavior, J. Call, G. M. Burghardt, I. M. Pepperberg, C. T. Snowdon, T. Zentall, Washington, USA: American Psychological Association, pp. 931–952, 2017. DOI: https://doi.org/10.1037/0000011-045.

    Google Scholar 

  10. T. Deboer, I. Tobler. Temperature dependence of EEG frequencies during natural hypothermia. Brain Research, vol. 670, no. 1, pp. 153–156, 1995. DOI: https://doi.org/10.1016/0006-8993(94)01299-W.

    Google Scholar 

  11. T. Deboer. Brain temperature dependent changes in the electroencephalogram power spectrum of humans and animals. Journal of Sleep Research, vol. 7, no. 4, pp. 254–262, 1998. DOI: https://doi.org/10.1046/j.1365-2869.1998.00125.x.

    Google Scholar 

  12. E. Salazar-López, E. Domínguez, V. Juárez Ramos, J. de la Fuente, A. Meins, O. Iborra, G. Gálvez, M. A. Rodríguez-Artacho, E. Gómez-Milán. The mental and subjective skin: Emotion, empathy, feelings and thermography. Consciousness and Cognition, vol. 34, pp. 149–162, 2015. DOI: https://doi.org/10.1016/j.concog.2015.04.003.

    Google Scholar 

  13. A. Truzzi, V. P. Senese, P. Setoh, C. Ripoli, G. Esposito. In utero testosterone exposure influences physiological responses to dyadic interactions in neurotypical adults. Acta Neuropsychiatrica, vol. 28, no. 5, pp. 304–309, 2016. DOI: https://doi.org/10.1017/neu.2016.15.

    Google Scholar 

  14. K. Gotham, S. Risi, A. Pickles, C. Lord. The autism diagnostic observation schedule: Revised algorithms for improved diagnostic validity. Journal of Autism and Developmental Disorders, vol. 37, no. 4, pp. 613–627, 2007. DOI: https://doi.org/10.1007/s10803-006-0280-1.

    Google Scholar 

  15. K. Gotham, A. Pickles, C. Lord. Standardizing ADOS scores for a measure of severity in autism spectrum disorders. Journal of Autism and Developmental Disorders, vol. 39, no. 5, pp. 693–705, 2009. DOI: https://doi.org/10.1007/s10803-008-0674-3.

    Google Scholar 

  16. D. S. Mandell, R. F. Ittenbach, S. E. Levy, J. A. Pinto-Martin. Disparities in diagnoses received prior to a diagnosis of autism spectrum disorder. Journal of Autism and Developmental Disorders, vol. 37, no. 9, pp. 1795–1802, 2007. DOI: https://doi.org/10.1007/s10803-006-0314-8.

    Google Scholar 

  17. BioHealth Group. Bio-health Informatics, [Online], Available: https://sites.google.com/cse.mrt.ac.lk/biohealth, September 06, 2019.

  18. D. A. Meedeniya, I. D. Rubasinghe. A review of supportive computational approaches for neurological disorder identification. Interdisciplinary Approaches to Altering Neurodevelopmental Disorders, T. Wadhera, B. R. Ambedkar, D. Kakkar, Eds., Hershey, USA: IGI Gloabal, 2020.

    Google Scholar 

  19. C. C. Chernecky, B. J. Berger. Laboratory Tests and Diagnostic Procedures, 6th ed. Saint Louis, USA: Elsevier, 2013.

    Google Scholar 

  20. S. Noachtar, J. Rémi. The role of EEG in epilepsy: a critical review. Epilepsy & Behavior, vol. 15, no. 1, pp. 22–33, 2009. DOI: https://doi.org/10.1016/j.yebeh.2009.02.035.

    Google Scholar 

  21. S. Leske, S. S. Dalal. Reducing power line noise in EEG and MEG data via spectrum interpolation. Neuroimage, vol. 189, pp. 763–776, 2019. DOI: https://doi.org/10.1016/j.neuroimage.2019.01.026.

    Google Scholar 

  22. S. Deyashi, D. Banerjee, B. Chakraborty, D. Ghosh, J. Debnath. Application of CBR on Viral Fever Detection System (VFDS). In Proceedings of the 9th IEEE International Conference on Industrial Informatics, IEEE, Caparica, Portugal, pp. 660–665, 2011. DOI: https://doi.org/10.1109/INDIN.2011.6035037.

    Google Scholar 

  23. F. G. Filip, C. B. Zamfirescu, C. Ciurea. Computer-Supported Collaborative Decision-Making, Cham, USA: Springer, pp. 1–69, 2017.

    Google Scholar 

  24. K. Kawamoto, C. A. Houlihan, E. A. Balas, D. F. Lobach. Improving clinical practice using clinical decision support systems: A systematic review of trials to identify features critical to success. BMJ, vol. 330, no. 7494, Article number 765, 2005. DOI: https://doi.org/10.1136/bmj.38398.500764.8F.

  25. S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna, A. M. P. Michalek, Gavindya Jayawardena. A rule-based system for ADHD identification using eye movement data. In Proceedings of 2019 Moratuwa Engineering Research Conference, IEEE, Moratuwa, Sri Lanka, pp. 538–543, 2019. DOI: https://doi.org/10.1109/MERCon.2019.8818865.

    Google Scholar 

  26. D. Rubasinghe, D. A. Meedeniya. Automated neuroscience decision support framework. Deep Learning Techniques for Biomedical and Health Informatics, B. Agarwal, V. E. Balas, L. C. Jain, R. C. Poonia, Manisha, Eds., Academic Press, pp. 305–326, 2020. DOI: https://doi.org/10.1016/B978-0-12-819061-6.00013-6.

  27. I. Rish. An empirical study of the naive Bayes classifier. In Proceedings of IJCAI Workshop on Empirical Methods in Artificial Intelligence, Morgan Kaufmann, Seattle, USA, pp. 41–46, 2001.

    Google Scholar 

  28. S. Thapaliya, S. Jayarathna, M. Jaime. EvalUATING THE EEG and eye movements for autism spectrum disorder. In Proceedings of IEEE International Conference on Big Data, IEEE, Seattle, WA, USA, pp. 2328–2336, 2018. DOI: https://doi.org/10.1109/BigData.2018.8622501.

    Google Scholar 

  29. E. Grossi, C. Olivieri, M. Buscema. Diagnosis of autism through EEG processed by advanced computational algorithms. Computer methods and programs in biomedicine, vol. 142, no. C, pp. 73–79, 2017. DOI: https://doi.org/10.1016/j.cmpb.2017.02.002.

    Google Scholar 

  30. S. Dreiseitl, L. Ohno-Machado. Logistic regression and artificial neural network classification models: A methodology review. Journal of Biomedical Informatics, vol. 35, no. 5–6, pp. 352–359, 2002. DOI: https://doi.org/10.1016/S1532-0464(03)00034-0.

    Google Scholar 

  31. A. Liaw, M. Wiener. Classification and regression by randomForest. R News, vol. 2, no. 3, pp. 18–22, 2002.

    Google Scholar 

  32. W. J. Bosl, H. Tager-Flusberg, C. A. Nelson. EEG analytics for early detection of autism spectrum disorder: A datadriven approach. Scientific Reports, vol. 8, no. 1, Article number 6828, 2018. DOI: https://doi.org/10.1038/s41598-018-24318-x.

  33. T. Poggio, H. Mhaskar, L. Rosasco, B. Miranda, Q. L. Liao. Why and when can deep-but not shallow-networks avoid the curse of dimensionality: A review. International Journal of Automation and Computing, vol. 14, no. 5, pp. 503–519, 2017. DOI: https://doi.org/10.1007/s11633-017-1054-2.

    Google Scholar 

  34. Z. J. Yao, J. Bi, Y. X. Chen. Applying deep learning to individual and community health monitoring data: A survey. International Journal of Automation and Computing, vol. 15, no. 6, pp. 643–655, 2018. DOI: https://doi.org/10.1007/s11633-018-1136-9.

    Google Scholar 

  35. M. Ahmadlou, H. Adeli, A. Adeli. Fuzzy Synchronization Likelihood-wavelet methodology for diagnosis of autism spectrum disorder. Journal of Neuroscience Methods, vol. 211, no. 2, pp. 203–209, 2012. DOI: https://doi.org/10.1016/j.jneumeth.2012.08.020.

    Google Scholar 

  36. R. Djemal, K. AlSharabi, S. Ibrahim, A. Alsuwailem. EEG-based computer aided diagnosis of autism spectrum disorder using wavelet, entropy, and ANN. BioMed Research International, vol. 2017, Article number 9816591, 2017. DOI: https://doi.org/10.1155/2017/9816591.

  37. S. Afrakhteh, M. R. Mosavi, M. Khishe, A. Ayatollahi. Accurate classification of EEG signals using neural networks trained by hybrid population-physic-based algorithm. International Journal of Automation and Computing, vol. 17, no. 1, pp. 108–122, 2020. DOI: https://doi.org/10.1007/s11633-018-1158-3.

    Google Scholar 

  38. S. Siuly, V. Bajaj, A. Sengur, Y. C. Zhang. An advanced analysis system for identifying alcoholic brain state through EEG signals. International Journal of Automation and Computing, vol. 16, no. 6, pp. 737–747, 2019. DOI: https://doi.org/10.1007/s11633-019-1178-7.

    Google Scholar 

  39. S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna. A survey of attention deficit hyperactivity disorder identification using psychophysiological data. International Journal of Online and Biomedical Engineering, vol. 15, no. 13, pp. 61–76, 2019. DOI: https://doi.org/10.3991/ijoe.v15i13.10744.

    Google Scholar 

  40. W. Bosl, A. Tierney, H. Tager-Flusberg, C. Nelson. EEG complexity as a biomarker for autism spectrum disorder risk. BMC Medicine, vol. 9, no. 1, Article number 18, 2011. DOI: https://doi.org/10.1186/1741-7015-9-18.

  41. S. Jayarathna, Y. Jayawardana, M. Jaime, S. Thapaliya. Electroencephalogram (EEG) for delineating objective measure of autism spectrum disorder. Computational Models for Biomedical Reasoning and Problem Solving, C. H. Chen, S. C. S. Cheung, Eds., Hershey, PA, USA: IGI Global, pp. 34–65, 2019.

    Google Scholar 

  42. M. Ahmadlou, H. Adeli, A. Adeli. Fractality and a wavelet-chaos-neural network methodology for EEG-based diagnosis of autistic spectrum disorder. Journal of Clinical Neurophysiology, vol. 27, no. 5, pp. 328–333, 2010. DOI: https://doi.org/10.1097/WNP.0b013e3181f40dc8.

    Google Scholar 

  43. M. Ahmadlou, H. Adeli, A. Adeli. Improved visibility graph fractality with application for the diagnosis of Autism Spectrum Disorder. Physica A: Statistical Mechanics and its Applications, vol. 391, no. 20, pp. 4720–4726, 2012. DOI: https://doi.org/10.1016/j.physa.2012.04.025.

    Google Scholar 

  44. V. Hus, C. Lord. The autism diagnostic observation schedule, module 4: Revised algorithm and standardized severity scores. Journal of Autism and Developmental Disorders, vol. 44, no. 8, pp. 1996–2012, 2014. DOI: https://doi.org/10.1007/s10803-014-2080-3.

    Google Scholar 

  45. A. McCrimmon, K. Rostad. Test review: Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) manual (Part II): Toddler module. Journal of Psychoeducational Assessment, vol. 32, no. 1, pp. 88–92, 2014. DOI: https://doi.org/10.1177/0734282913490916.

    Google Scholar 

  46. LiveAmp (32 channels)[Apparatus]. (2018). Gilching, Germany: Brain Products GmbH. [Online], Available: https://www.brainproducts.com, September 25, 2019.

  47. L. Leuchs. Choosing your reference — and why it matters. Brain Products Press Release, 2019. [Online], Available: https://pressrelease.brainproducts.com/referencing/#21, September 20, 2019.

  48. R. Kher. Signal processing techniques for removing noise from ECG signals. Journal of Biomedical Engineering and Research, vol. 1, no. 1, pp. 1–9, 2019.

    Google Scholar 

  49. R. Milo, P. Jorgensen, U. Moran, G. Weber, M. Springer. BioNumbers-the database of key numbers in molecular and cell biology. Nucleic Acids Research, vol. 38, no. Suppl 1, pp. D750–D753, 2009. DOI: https://doi.org/10.1093/nar/gkp889.

    Google Scholar 

  50. S. Sanei, J. A. Chambers. EEG Signal Processing, United Kingdom: John Wiley & Sons, 2013.

    Google Scholar 

  51. M. A. Hall. Correlation-based Feature Selection for Machine Learning, Ph. D. dissertation, University of Waikato, Hamilton, USA, 1999.

    Google Scholar 

  52. E. Y. K. Ng, G. J. L. Kawb, W. M. Chang. Analysis of IR thermal imager for mass blind fever screening. Microvascular Research, vol. 68, no. 2, pp. 104–109, 2004. DOI: https://doi.org/10.1016/j.mvr.2004.05.003.

    Google Scholar 

  53. A. C. Marinescu, S. Sharples, A. C. Ritchie, T. Sánchez López, M. McDowell, H. P. Morvan. Physiological parameter response to variation of mental workload. Human Factors: The Journal of the Human Factors and Ergonomics Society, vol. 60, no. 1, pp. 31–56, 2018. DOI: https://doi.org/10.1177/0018720817733101.

    Google Scholar 

  54. C. K. L. Or, V. Duffy. Development of a facial skin temperature-based methodology for non-intrusive mental workload measurement. Occupational Ergonomics, vol. 7, no. 2, pp. 83–94, 2007.

    Google Scholar 

  55. D. Shearn, E. Bergman, K. Hill, A. Abel, L. Hinds. Facial coloration and temperature responses in blushing. Psychophysiology, vol. 27, no. 6, pp. 687–693, 1990. DOI: https://doi.org/10.1111/j.1469-8986.1990.tb03194.x.

    Google Scholar 

  56. S. Bejerot, J. M. Eriksson, E. Mörtberg. Social anxiety in adult autism spectrum disorder. Psychiatry Research, vol. 220, no. 1–2, pp. 705–707, 2014. DOI: https://doi.org/10.1016/j.psychres.2014.08.030.

    Google Scholar 

  57. G. Bradski. The OpenCV library. Dr. Dobb’s Journal of Software Tools, vol. 25, no. 11, pp. 120–126, 2000.

    Google Scholar 

  58. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I. H. Witten. The WEKA data mining software: An update. ACM SIGKDD Explorations Newsletter, vol. 11, pp. 10–18, 2009. DOI: https://doi.org/10.1145/1656274.1656278.

    Google Scholar 

  59. M. Saltan, S. Terzi. Modeling deflection basin using artificial neural networks with cross-validation technique in back calculating flexible pavement layer moduli. Advances in Engineering Software, vol. 39, no. 7, pp. 588–592, 2008. DOI: https://doi.org/10.1016/j.advengsoft.2007.06.002.

    Google Scholar 

  60. G. Ariyarathne, S. De Silva, S. Dayarathna, D. Meedeniya, S. Jayarathna. ADHD identification using convolutional neural network with seed-based approach for fMRI data. In Proceedings of the 9th International Conference on Software and Computer Applications, ACM, Malaysia, pp. 31–35, 2020. (未找到本条文献信息, 请核对)

    Google Scholar 

  61. S. De Silva, S. Dayarathna, G. Ariyarathne, D. Meedeniya, S. Jayarathna. fMRI feature extraction model for ADHD classification using convolutional neural network. International Journal of E-Health and Medical Communications, vol. 12, no. 1, Article number 6, 2020.

  62. I. Rubasinghe, D. Meedeniya. Ultrasound nerve segmentation using deep probabilistic programming. Journal of ICT Research and Applications, vol. 13, no. 3, pp. 241–256, 2019. DOI: https://doi.org/10.5614/itbj.ict.res.appl.2019.13.3.5.

    Google Scholar 

Download references

Acknowledgments

This work was supported by Old Dominion University, Norfolk, Virginia and University of Moratuwa, Sri Lanka.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dulani Meedeniya.

Additional information

Dilantha Haputhanthri is an undergraduate at Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka.

His research interests include bioinformatics, machine learning, data analytics and algorithm development.

Gunavaran Brihadiswaran is an undergraduate at Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka.

His research interests include emerging memory technologies, computer architecture, data structures and algorithms, and computational biology.

Sahan Gunathilaka is an undergraduate in the Department of Computer Science and Engineering at the University of Moratuwa, Sri Lanka.

His research interests include bioinformatics, machine learning, and image processing.

Dulani Meedeniya received the Ph. D. degree in computer science from University of St Andrews, UK. She is a senior lecturer in Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka. She is a Fellow of HEA(UK), MIET, MIEEE, and a Charted Engineer at EC (UK).

Her research interests indude software modeling and design, workflow tool support for bioinformatics, data visualization and recommender systems.

Sampath Jayarathna received the Ph. D. degree in computer science from the Texas A&M University College Station, USA in 2016. He is an assistant professor of computer science at Old Dominion University, USA, where he is associated with Web Science and Digital Libraries (WS-DL) research group. He is a member of ACM, IEEE, and Sigma XI. His research interests include machine learning, information retrieval, data science, eye tracking, and brain-computer interfacing.

Mark Jaime received the Ph. D. degger in psychology from Florida International University, USA in 2008, where he was trained in developmental psychobiology. He then completed two post-doctoral fellowships specializing in the neurocognitive development of autism spectrum disorder (ASD) and childhood intersensory processing in Department of Psychology, University of Miami, USA and Department of Psychology and Neuroscience at Dalhousie University, Canada. He is an associate professor of psychology at the Columbus Center of Indiana University-Purdue University, USA. He is a full member of the International Society for Autism Research.

His research interests include characterizing psychophysiological mechanisms of social information processing in ASD and typical populations. He also engages in interdisciplinary collaborations focused on the development of neurocognitive markers of social impairment.

Christopher Harshaw received the Ph. D. degree in developmental Science from Florida International University, USA. He then completed a postdoc at Indiana University, USA with a focus on Behavioral Neuroscience and Developmental Psychobiology. He is an assistant professor at Department of Psychology, University of New Orleans, USA. He directs the Mechanisms Underlying Sociality (MUS) Laboratory, which is currently focused on how deficits in temperature regulation may relate to differences in social cognitive abilities in Autism Spectrum Disorders.

His research interest is gaining a better understanding of how bodily mechanisms influence cognition and behavior.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Haputhanthri, D., Brihadiswaran, G., Gunathilaka, S. et al. Integration of Facial Thermography in EEG-based Classification of ASD. Int. J. Autom. Comput. 17, 837–854 (2020). https://doi.org/10.1007/s11633-020-1231-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-020-1231-6

Keywords

Navigation