Skip to main content
SpringerLink
Log in

Explainable Multimodal Machine Learning for Engagement Analysis by Continuous Performance Test

  • Conference paper
  • First Online:
Universal Access in Human-Computer Interaction. User and Context Diversity (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13309))

Included in the following conference series:

Abstract

The human vision system assiduously looks for exciting regions in the real world, in images and videos, to reduce the search effort for various tasks, such as object detection and recognition. A spatial attention representation can divulge the exciting segments, blocks or regions in such images. The Conners’ continuous performance test is a visual assessment technique to evaluate the attention and the response inhibition component of executive control to assess attention deficit hyperactivity disorder (ADHD) and other neurological disorders. Artificial Intelligence and Machine Learning models are advancing ever more complex, going from shallow to deep learning over time. Thus, we can achieve higher accuracy and greater precision. However, this also tends to make these models ‘black boxes’, reducing the comprehensibility of the logic played out in the various predictions and outcomes. This raises an obvious question - how do we understand the prediction suggested or recommended by these machine learning models so that we can place trust in them? XAI attempts to make a trade-off between precision, accuracy and interpretability to achieve this. This research work presents an Explainable Artificial Intelligence (XAI) model for a continuous performance test, monitoring multisensor data and multimodal machine learning for engagement analysis. The sensor data considered included body pose, Electrocardiograph, eye gaze, interaction data and facial features via accurate labelling of engagement or disengagement for cognitive attention of a Seek-X type task execution. We used decision trees and XAI to visualize the multisensor multimodal data, which will help us assess the model’s accuracy intuitively and provide us with the explainability of engagement or disengagement for visual interactions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aci, C.I., Kaya, M., Mishchenko, Y.: Distinguishing mental attention states of humans via an EEG-based passive BCI using machine learning methods. Expert Syst. Appl. 134, 153–166 (2019). https://doi.org/10.1016/j.eswa.2019.05.057, https://www.sciencedirect.com/science/article/pii/S0957417419303926

  2. Adiba, F.I., Islam, T., Kaiser, M.S., Mahmud, M., Rahman, M.A.: Effect of corpora on classification of fake news using Naive Bayes classifier. Int. J. Autom. Artif. Intell. Mach. Learn. 1(1), 80–92 (2020). https://researchlakejournals.com/index.php/AAIML/article/view/45

  3. Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Patt. Anal. Mach. Intell. 41(2), 423–443 (2019). https://doi.org/10.1109/TPAMI.2018.2798607

  4. Belle, A., Hobson, R., Najarian, K.: A physiological signal processing system for optimal engagement and attention detection. In: 2011 IEEE International Conference on Bioinformatics and Biomedicine Workshops (BIBMW), pp. 555–561 (2011). https://doi.org/10.1109/BIBMW.2011.6112429

  5. Biswas, M., Kaiser, M.S., Mahmud, M., Al Mamun, S., Hossain, M.S., Rahman, M.A.: An XAI based autism detection: the context behind the detection. In: Mahmud, M., Kaiser, M.S., Vassanelli, S., Dai, Q., Zhong, N. (eds.) Brain Informatics, pp. 448–459. Lecture Notes in Computer Science. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-86993-9_40

  6. Bloom, E., Heath, N.: Recognition, expression, and understanding facial expressions of emotion in adolescents with nonverbal and general learning disabilities. J. Learn. Disabil. 43(2), 180–192 (2010)

    Article  Google Scholar 

  7. Bone, D., Goodwin, M.S., Black, M.P., Lee, C.C., Audhkhasi, K., Narayanan, S.: Applying machine learning to facilitate autism diagnostics: pitfalls and promises. J. Autism Dev. Disord. 45(5), 1121–1136 (2015). https://doi.org/10.1007/s10803-014-2268-6

  8. Brown, D., Sherkat, N., Taheri, M.: Modeling engagement with multimodal multisensor data: the continuous performance test as an objective tool to track flow. Int. J. Comput. Inf. Eng. 14(162), 197–208 (2020)

    Google Scholar 

  9. Brown, D.J., Kerr, S., Wilson, J.R.: Virtual environments in special-needs education. Commun. ACM 40(8), 72–75 (1997). https://doi.org/10.1145/257874.257891

  10. Brown, D.J., McHugh, D., Standen, P., Evett, L., Shopland, N., Battersby, S.: Designing location-based learning experiences for people with intellectual disabilities and additional sensory impairments. Comput. Educ. 56(1), 11–20 (2011). https://doi.org/10.1016/j.compedu.2010.04.014, https://www.sciencedirect.com/science/article/pii/S0360131510001211

  11. Chakraborty, P., Yousuf, M.A., Rahman, S.: Predicting level of visual focus of human’s attention using machine learning approaches. In: Kaiser, M.S., Bandyopadhyay, A., Mahmud, M., Ray, K. (eds.) Proceedings of International Conference on Trends in Computational and Cognitive Engineering, pp. 683–694. Advances in Intelligent Systems and Computing. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-4673-4-56

  12. Chen, J., Liao, M., Wang, G., Chen, C.: An intelligent multimodal framework for identifying children with autism spectrum disorder. Int. J. Appl. Math. Comput. Sci. (2020). 10.34768/AMCS-2020-0032. https://sciendo.com/article/10.34768/amcs-2020-0032

  13. Cukurova, M.: Learning analytics as AI extenders in education: multimodal machine learning versus multimodal learning analytics. In: Proceedings of the Artificial Intelligence and Adaptive Education Conference, pp. 1–3 (2019)

    Google Scholar 

  14. Das, S., Yasmin, M.R., Arefin, M., Taher, K.A., Uddin, M.N., Rahman, M.A.: Mixed Bangla-English spoken digit classification using convolutional neural network. In: Mahmud, M., Kaiser, M.S., Kasabov, N., Iftekharuddin, K., Zhong, N. (eds.) Applied Intelligence and Informatics, pp. 371–383. Communications in Computer and Information Science. Springer International Publishing, Cham (2021). https://doi.org/10.1007/978-3-030-82269-9_29

  15. Das, T.R., Hasan, S., Sarwar, S.M., Das, J.K., Rahman, M.A.: Facial spoof detection using support vector machine. In: Kaiser, M.S., Bandyopadhyay, A., Mahmud, M., Ray, K. (eds.) Proceedings of International Conference on Trends in Computational and Cognitive Engineering, pp. 615–625. Advances in Intelligent Systems and Computing. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-4673-4_50

  16. Duda, M., Kosmicki, J.A., Wall, D.P.: Testing the accuracy of an observation-based classifier for rapid detection of autism risk. Transl. Psychiat. 4(8), e424–e424 (2014). https://doi.org/10.1038/tp.2014.65, https://www.nature.com/articles/tp201465, number: 8 Publisher: Nature Publishing Group

  17. Ferdous, H., Siraj, T., Setu, S.J., Anwar, M.M., Rahman, M.A.: Machine learning approach towards satellite image classification. In: Kaiser, M.S., Bandyopadhyay, A., Mahmud, M., Ray, K. (eds.) Proceedings of International Conference on Trends in Computational and Cognitive Engineering, pp. 627–637. Advances in Intelligent Systems and Computing. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-4673-4_51

  18. Garzon, M., Mainali, S.: Deep structure of DNA for genomic analysis. Human Molecular Genetics (2021)

    Google Scholar 

  19. Han, Y., Li, L., Zhang, J.: A coordinated representation learning enhanced multimodal machine translation approach with multi-attention, pp. 571–577. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3372278.3390717

  20. Harrivel, A.R., et al.: Prediction of cognitive states during flight simulation using multimodal psychophysiological sensing–AIAA Information Systems-AIAA Infotech @ Aerospace (2017). https://doi.org/10.2514/6.2017-1135, https://arc.aiaa.org/doi/abs/10.2514/6.2017-1135, archive Location: world

  21. Hilbert, K., Lueken, U., Muehlhan, M., Beesdo-Baum, K.: Separating generalized anxiety disorder from major depression using clinical, hormonal, and structural MRI data: a multimodal machine learning study. Brain Behav. 7(3), e00633 (2017). https://doi.org/10.1002/brb3.633, https://onlinelibrary.wiley.com/doi/abs/10.1002/brb3.633

  22. Holder, H.B., Kirkpatrick, S.W.: Interpretation of emotion from facial expressions in children with and without learning disabilities. J. Learn. Disabil. 24(3), 170–177 (1991)

    Article  Google Scholar 

  23. Iovannone, R., Dunlap, G., Huber, H., Kincaid, D.: Effective educational practices for students with autism spectrum disorders. Focus Autism Dev. Disabil. 18(3), 150–165 (2003). https://doi.org/10.1177/10883576030180030301. SAGE Publications Inc

  24. Joachims, T.: Learning to classify text using support vector machines, vol. 668. Springer Science and Business Media (2002). https://doi.org/10.1007/978-1-4615-0907-3_2

  25. Kosmicki, J., Sochat, V., Duda, M., Wall, D.: Searching for a minimal set of behaviors for autism detection through feature selection-based machine learning. Transl. Psychiatr. 5(2), e514–e514 (2015)

    Article  Google Scholar 

  26. Kourou, K., Exarchos, T.P., Exarchos, K.P., Karamouzis, M.V., Fotiadis, D.I.: Machine learning applications in cancer prognosis and prediction. Comput. Struct. Biotechnol. J. 13, 8–17 (2015)

    Article  Google Scholar 

  27. Li, J., Ngai, G., Leong, H.V., Chan, S.C.F.: Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics. SIGAPP Appl. Comput. Rev. 16(3), 37–49 (2016). https://doi.org/10.1145/3015297.3015301

  28. Liu, W., Zhou, T., Zhang, C., Zou, X., Li, M.: Response to name: a dataset and a multimodal machine learning framework towards autism study. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 178–183 (2017). https://doi.org/10.1109/ACII.2017.8273597

  29. Mahmud, M., Kaiser, M.S., McGinnity, T.M., Hussain, A.: Deep learning in mining biological data. Cogn. Computat. 13(1), 1–33 (2021). https://doi.org/10.1007/s12559-020-09773-x

  30. Mahmud, M., Kaiser, M.S., Rahman, M.M., Rahman, M.A., Shabut, A., Al-Mamun, S., Hussain, A.: A brain-inspired trust management model to assure security in a cloud based IoT framework for neuroscience applications. Cogn. Comput. 10(5), 864–873 (2018)

    Article  Google Scholar 

  31. Metsala, J.L., Galway, T.M., Ishaik, G., Barton, V.E.: Emotion knowledge, emotion regulation, and psychosocial adjustment in children with nonverbal learning disabilities. Child Neuropsychol. 23(5), 609–629 (2017)

    Article  Google Scholar 

  32. Morency, L.P., Baltrušaitis, T.: Multimodal machine learning: integrating language, vision and speech. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 3–5 (2017)

    Google Scholar 

  33. Most, T., Greenbank, A.: Auditory, visual, and auditory-visual perception of emotions by adolescents with and without learning disabilities, and their relationship to social skills. Learn. Disabil. Res. Pract. 15(4), 171–178 (2000)

    Article  Google Scholar 

  34. Myrden, A., Chau, T.: A passive EEG-BCI for single-trial detection of changes in mental state. IEEE Trans. Neural Syst. Rehabil. Eng. 25(4), 345–356 (2017). https://doi.org/10.1109/TNSRE.2016.2641956

  35. Nasrin, F., Ahmed, N.I., Rahman, M.A.: Auditory attention state decoding for the quiet and hypothetical environment: a comparison between bLSTM and SVM. In: Kaiser, M.S., Bandyopadhyay, A., Mahmud, M., Ray, K. (eds.) Proceedings of International Conference on Trends in Computational and Cognitive Engineering, pp. 291–301. Advances in Intelligent Systems and Computing. Springer, Singapore (2021). https://doi.org/10.1007/978-981-33-4673-4_23

  36. Nuamah, J.K., Seong, Y.: Support vector machine (SVM) classification of cognitive tasks based on electroencephalography (EEG) engagement index. Brain-Comput. Interf. 5(1), 1–12 (2018). https://doi.org/10.1080/2326263X.2017.1338012

  37. Ouherrou, N., Elhammoumi, O., Benmarrakchi, F., El Kafi, J.: Comparative study on emotions analysis from facial expressions in children with and without learning disabilities in virtual learning environment. Educ. Inf. Technol. 24(2), 1777–1792 (2019). https://doi.org/10.1007/s10639-018-09852-5

  38. Petti, V.L., Voelker, S.L., Shore, D.L., Hayman-Abello, S.E.: Perception of nonverbal emotion cues by children with nonverbal learning disabilities. J. Dev. Phys. Disabil. 15(1), 23–36 (2003)

    Article  Google Scholar 

  39. Rahman, M.A.: Gaussian Process in Computational Biology: Covariance Functions for Transcriptomics. Phd, University of Sheffield, February 2018. https://etheses.whiterose.ac.uk/19460/

  40. Rokach, L., Maimon, O.: Decision trees. In: Maimon, O., Rokach, L. (eds.) Data mining and knowledge discovery handbook, pp. 165–192. Springer, US, Boston, MA (2005). https://doi.org/10.1007/0-387-25465-X-9

  41. Sadik, R., Reza, M.L., Al Noman, A., Al Mamun, S., Kaiser, M.S., Rahman, M.A.: COVID-19 pandemic: a comparative prediction using machine learning. Int. J. Autom. Artif. Intell. Mach. Learn. 1(1), 1–16 (2020)

    Google Scholar 

  42. Song, Y.Y., Lu, Y.: Decision tree methods: applications for classification and prediction. Shanghai Arch. Psychiatr. 27(2), 130–135 (2015). https://doi.org/10.11919/j.issn.1002-0829.215044, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4466856/

  43. Rokach, L., Maimon, O.: Decision trees. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 165–192. Springer US, Boston, MA (2005). https://doi.org/10.1007/0-387-25465-X-9

  44. Swanson, L.: Vigilance deficit in learning disabled children: a signal detection analysis. J. Child Psychol. Psychiatr. 22(4), 393–399 (1981)

    Article  Google Scholar 

  45. Tavabi, L.: Multimodal machine learning for interactive mental health therapy. In: 2019 International Conference on Multimodal Interaction, pp. 453–456. ICMI 2019. Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3340555.3356095

  46. Walambe, R., Nayak, P., Bhardwaj, A., Kotecha, K.: Employing multimodal machine learning for stress detection. J. Healthc. Eng. 2021, e9356452 (2021). https://doi.org/10.1155/2021/9356452, https://www.hindawi.com/journals/jhe/2021/9356452/, publisher: Hindawi

  47. Xu, K., et al.: Multimodal machine learning for automated ICD coding. In: Doshi-Velez, F., et al. (eds.) Proceedings of the 4th Machine Learning for Healthcare Conference. Proceedings of Machine Learning Research, vol. 106, pp. 197–215. PMLR, 9–10 Aug 2019. https://proceedings.mlr.press/v106/xu19a.html

  48. Zubair, M.S., Brown, D.J., Hughes-Roberts, T., Bates, M.: Designing accessible visual programming tools for children with autism spectrum condition. Universal Access in the Information Society, pp. 1–20 (2021)

    Google Scholar 

Download references

Acknowledgement

This research was co-funded by the Erasmus+ programme of the European Union in the projects Pathway+ (2017-1-UK01-KA201-036761) ‘A Mobile Pedagogical Assistant to develop meaningful pathways to personalised learning’ and ‘AI-TOP’ (2020-1-UK01-KA201-079167) ‘An AI Tool to Predict Engagement and ‘Meltdown’ Events in Students with Autism’.

Author information

Authors and Affiliations

  1. Interactive Systems Research Group, Computer Science and Informatics, Nottingham Trent University, Nottingham, UK

    Muhammad Arifur Rahman, David J. Brown, Nicholas Shopland, Andrew Burton & Mufti Mahmud

  2. Medical Technologies Innovation Facility, Nottingham Trent University, Nottingham, UK

    Mufti Mahmud

Authors
  1. Muhammad Arifur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David J. Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicholas Shopland
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andrew Burton
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mufti Mahmud
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Arifur Rahman .

Editor information

Editors and Affiliations

  1. Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  2. University of Crete and Foundation for Research and Technology - Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rahman, M.A., Brown, D.J., Shopland, N., Burton, A., Mahmud, M. (2022). Explainable Multimodal Machine Learning for Engagement Analysis by Continuous Performance Test. In: Antona, M., Stephanidis, C. (eds) Universal Access in Human-Computer Interaction. User and Context Diversity. HCII 2022. Lecture Notes in Computer Science, vol 13309. Springer, Cham. https://doi.org/10.1007/978-3-031-05039-8_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-05039-8_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-05038-1

  • Online ISBN: 978-3-031-05039-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation