Skip to main content
SpringerLink
Log in

AVCA: Autonomous Virtual Cognitive Assessment

  • Chapter
  • First Online:
Transactions on Computational Science XL

Part of the book series: Lecture Notes in Computer Science ((TCOMPUTATSCIE,volume 13850))

  • 139 Accesses

Abstract

Fast-paced and ever-growing advances in Artificial Intelligence (AI) and Deep Neural Network (DNN) models have initiated works on autonomous monitoring/screening systems to assess individuals’ cognitive state. In conventional cognitive assessment systems, a physician evaluates the mental abilities of the brain by rating the patient’s numerical, verbal, and logical responses. Development of an autonomous cognitive assessment system that assist the physician is both of significant importance and a critically challenging task. As a first step towards achieving this objective, in the paper an Automated Virtual Cognitive Assessment (\(\text {AVCA}\)) framework is proposed that integrates Natural Language Processing (NLP) and hand gesture recognition techniques. The proposed \(\text {AVCA}\) framework provides individual scores in the seven major cognitive domains, i.e., orientation, attention, language, contractual ability, memory, calculation, and reasoning. More specifically, the \(\text {AVCA}\) framework is an autonomous cognitive assessment system that receives audio and video signals in a real-time fashion, and performs semantic and synthetic analysis using NLP techniques and DNN models. Real-time video processing engines of the \(\text {AVCA}\) monitors hand motions and facilitate simpler engagement for visual evaluation. Additionally, we propose an efficient model to facilitate human-machine interactions from speech recognition to text classification. In particular, an unsupervised contrastive learning framework is proposed using Bidirectional Encoder Representations from Transformers (BERT) that outperforms its state-of-the-art unsupervised counterparts achieving an average of 77.84% Sparsman’s correlation on standard Semantic Textual Similarity (STS) tasks.

This Project was partially supported by Department of National Defence’s Innovation for Defence Excellence & Security (IDEaS), Canada.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Jung, H.T., et al.: Remote assessment of cognitive impairment level based on serious mobile game performance: an initial proof of concept. IEEE J. Biom. Health Inf. 23(3), 1269–1277 (2019)

    Article  Google Scholar 

  2. Greenberg, P.E., Fournier, A.-A., Sisitsky, T., Pike, C.T., Kessler, R.C.: The economic burden of adults with major depressive disorder in the united states (2005 and 2010). J. Clin. Psychiatry 76(2), 155–162 (2015)

    Article  Google Scholar 

  3. Canal, F.Z., et al.: A survey on facial emotion recognition techniques: a state-of-the-art literature review. Inf. Sci. 582, 593–617 (2022)

    Article  Google Scholar 

  4. Li, X., et al.: EEG based emotion recognition: a tutorial and review. ACM Comput. Surv. 55(4), 1–57 (2023)

    Article  Google Scholar 

  5. Zhang, J., Yin, Z., Chen, P., Nichele, S.: Emotion recognition using multi-modal data and machine learning techniques: a tutorial and review. Inf. Fusion 59, 103–126 (2020)

    Article  Google Scholar 

  6. Hassan, M.M., et al.: Human emotion recognition using deep belief network architecture. Inf. Fusion 51, 10–18 (2019)

    Google Scholar 

  7. Yousaf, A., et al.: Emotion recognition by textual tweets classification using voting classifier (LR-SGD). IEEE Access 9, 6286–6295 (2021)

    Article  Google Scholar 

  8. Dimou, F.M., Eckelbarger, D., Riall, T.S.: Surgeon burnout: a systematic review. J. Am. Coll. Surg. 222(6), 1230 (2016)

    Article  Google Scholar 

  9. Ekman, P., et al.: Universals and cultural differences in the judgments of facial expressions of emotion. J. Pers. Soc. Psychol. 53(4), 712 (1987)

    Article  Google Scholar 

  10. Ahn, S., et al.: Exploring neuro-physiological correlates of drivers, mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data. Front. Neurosci. 10, 1–14 (2016)

    Google Scholar 

  11. Han, S.Y., Kwak, N.S., Oh, T., Lee, S.W.: Classification of Pilots. Mental States using a Multimodal Deep Learning Network. Biocybernetics Biomed. Eng. 40(1), 324–336 (2020)

    Google Scholar 

  12. Cheng, B., et al.: Measuring and computing cognitive statuses of construction workers based on electroencephalogram: a critical review. IEEE Trans. Comput. Soc. Syst. 9(6), 1644–1659 (2022)

    Article  Google Scholar 

  13. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., Babiloni, F.: Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75 (2014)

    Article  Google Scholar 

  14. Castaldo, R., Melillo, P., Bracale, U., Caserta, M., Triassi, M., Pecchia, L.: Acute mental stress assessment via short term HRV analysis in healthy adults: a systematic review with meta-analysis. Biomed. Signal Process. Control 18, 370–377 (2015)

    Article  Google Scholar 

  15. Folstein, M.F., Folstein, S.E.: Mini-Mental State’’ a practical method for grading the cognitive state of patients for the clinician. J. Psychiatric Res. 12(3), 189–198 (1975)

    Article  Google Scholar 

  16. Nasreddine, Z.S., et al.: The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. J. Amer. Geriatrics Soc. 53, 695–699 (2005)

    Article  Google Scholar 

  17. Kiernan, R.J., Mueller, J., Langston, J.W., Van Dyke, C.: The neurobehavioral cognitive status examination: a brief but differentiated approach to cognitive assessment. Ann. Intern. Med. 1987(107), 481–5 (1987)

    Article  Google Scholar 

  18. Fields, S.D., Fulop, G., Sachs, C.J., Strain, J., Fillit, H.: Usefulness of the neurobehavioral cognitive status examination in the hospitalized elderly. Int. Psychogeriatr. 4, 93–102 (1992)

    Article  Google Scholar 

  19. Murakami, H., Fujita, K., Futamura, A., et al.: The Montreal Cognitive Assessment (MoCA) and Neurobehavioral Cognitive Status Examination (COGNISTAT) are useful for screening mild cognitive impairment in Japanese patients with Parkinson’s disease. Neurol. Clin. Neurosci. 1, 103–8 (2013)

    Article  Google Scholar 

  20. Osmon, D.C., Smet, I.C., Winegarden, B., Gandhavadi, B.: Neurobehavioral cognitive status examination: its use with unilateral stroke patients in a rehabilitation setting. Arch. Phys. Med. Rehabil. 73, 414–418 (1992)

    Google Scholar 

  21. Tangalos, E.G., et al.: The mini-mental state examination in general medical practice: clinical utility and acceptance. Mayo Clinic Proc. 71(9), 829–837 (1996)

    Article  Google Scholar 

  22. Brouillette, R.M., et al.: Feasibility, reliability, and validity of a smartphone-based application for the assessment of cognitive function in the elderly. PLoS One 8(6) (2013). Art. no e65925

    Google Scholar 

  23. Timmers, C., Maeghs, A., Vestjens, M., Bonnemayer, C., Hamers, H., Blokland, A.: Ambulant cognitive assessment using a smartphone. Appl. Neuropsychol. 21(2) (2013). Art. no. e112197

    Google Scholar 

  24. Schweitzer, P., et al.: Feasibility and validity of mobile cognitive testing in the investigation of age-related cognitive decline. Int. J. Methods Psychiatric Res. 26(3) (2017). Art. no. e1521

    Google Scholar 

  25. Koelstra, S., et al.: Deap: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3(1), 18–31 (2011)

    Article  Google Scholar 

  26. Finney, G.R., Minagar, A., Heilman, K.M.: Assessment of mental Status. Neurol. Clin. 34(1), 1–16 (2016)

    Article  Google Scholar 

  27. Tsoi, K.K., et al.: Cognitive tests to detect dementia: a systematic review and meta-analysis. JAMA Internal Med. 175, 1450–1458

    Google Scholar 

  28. Daroische, R., et al.: Cognitive impairment after COVID-19 - a review on objective test data. Front. Neurol. 12, 1238 (2021)

    Article  Google Scholar 

  29. Tombaugh, T.N., McIntyre, N.J.: The mini-mental state examination: a comprehensive review. J. Amer. Geriatrics Soc. 40(9), 922–935 (1992)

    Article  Google Scholar 

  30. Goyal, A.K., Metallinou, A., Matsoukas, S.: Fast and scalable expansion of natural language understanding functionality for intelligent agents. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers). Association for Computational Linguistics (2018)

    Google Scholar 

  31. Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with lstm (2000)

    Google Scholar 

  32. Chung, C.G., Cho, K., Bengio, Y.: Empirical evaluation of gated recurrent neural networks on sequence modelling. In NIPS 2014 Workshop on Deep Learning (2014)

    Google Scholar 

  33. Chiu, J.P.C., Nichols, E.: Named entity recognition with bidirectional lstm-cnns. arXiv preprint arXiv:1511.08308 (2015)

  34. Huang, Z., Xu, W., Yu, K.: Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.0199 (2015)

  35. Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019)

  36. Peters, M., et al.: Deep contextualized word representations. In: North American Association for Computational Linguistics (NAACL) (2018)

    Google Scholar 

  37. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding with unsupervised learning. Technical report, OpenAI

    Google Scholar 

  38. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237(2019)

  39. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional Transformers for language understanding. In: Proc. 2019 Conf. North Am. Chapter Assoc. Comput. Linguist. Hum. Lang. Technol., vol. 1, NAACL-HLT 2019, Minneapolis, MN, USA (2019), pp. 4171–4186. Google-AI Language, June 2–7, 2019

    Google Scholar 

  40. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 6000–6010 (2017)

    Google Scholar 

  41. Reimers, N., Gurevych, I.: Sentence-BERT: Sentence embeddings using siamese BERT-Networks. DOI (2019)

    Google Scholar 

  42. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering (2015)

    Google Scholar 

  43. Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an ASR corpus based on public domain audio books. In: Proceedings of ICASSP, pp. 5206–5210. IEEE (2015)

    Google Scholar 

  44. Graves, A., Fernández, S., Gomez, F.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of ICML (2006)

    Google Scholar 

  45. Gao, T., Yao, X., Chen, D.: SimCSE: simple contrastive learning of sentence embeddings. In: International Conference on Empirical Methods in Natural Language Processing, pp. 6894–6910 (2021)

    Google Scholar 

  46. Zhang, J., Zhao, Y., Saleh, M., Liu, P.J.: Pegasus: pre-training with extracted gap-sentences for abstractive summarization. ICML (2020)

    Google Scholar 

  47. Lample, G., Conneau, A.: Crosslingual language model pretraining. arXiv preprint arXiv:1901.07291 (2019)

  48. Gondala, S., Verwimp, L., Pusateri, E., Tsagkias, M., Van Gysel, C.: Error-Driven Pruning of Language Models for Virtual Assistants. arXiv:2102.07219 (2021)

  49. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR (2005)

    Google Scholar 

  50. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. (JMLR) 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  51. Kukacka, J., Golkov, V., Cremers, D.: Regularization for deep learning: a taxonomy. arXiv preprint arXiv:1710.10686 (2017)

  52. Shorten, C., Khoshgoftaar, T.M., Furht, B.: Text data augmentation for deep learning. J. Big Data 8(1), 1–34 (2021)

    Article  Google Scholar 

  53. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial NLI: a new benchmark for natural language understanding. Association for Computational Linguistics (ACL), pp. 4885–4901 (2020)

    Google Scholar 

  54. Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Conference on Empirical Methods in Natural Language Processing, pp. 632–642 (2015)

    Google Scholar 

  55. Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1, pp. 1112–1122 (2018)

    Google Scholar 

  56. Baevski, A., Zhou, Y., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations (2020)

    Google Scholar 

  57. Hsu, W.-N., Bolte, B., Hunert Tsai, Y.-H., Lakhotia, K., Salakhutdinov, R., Mohamed, A.: HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio Speech Lang. Process. 29, 3451–3460 (2021)

    Google Scholar 

  58. S. Jones, K.: IDF term weighting and IR research lessons. J. Docum. 60(6), 521–523 (2004)

    Google Scholar 

  59. Maiya, A.S.: ktrain: a low-code library for augmented machine learning. arXiv:2004.10703 (2020)

  60. Chen, T., Sun, Y., Shi, Y., Hong, L.: On sampling strategies for neural network based collaborative filtering. In: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 767–776 (2017)

    Google Scholar 

  61. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning (ICML), pp. 1597–1607 (2020)

    Google Scholar 

  62. Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A.: SemEval-2012 task 6: a pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385–393 (2012)

    Google Scholar 

  63. Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A., Guo, W.: *SEM 2013 shared task: Semantic textual similarity. In: Second Joint Conference on Lexical and Computational Semantics (*SEM), volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pp. 32–43 (2013)

    Google Scholar 

  64. Agirre, E.. et al.: SemEval-2014 task 10: Multilingual semantic textual similarity. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 81–91 (2014)

    Google Scholar 

  65. Agirre, E., et al.: SemEval-2015 task 2: semantic textual similarity, English, Spanish and pilot on interpretability. In: Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pp. 252–263 (2015)

    Google Scholar 

  66. Agirre, E., et al.: SemEval-2016 task 1: semantic textual similarity, monolingual and cross-lingual evaluation. In: Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), pp. 497–511. Association for Computational Linguistics (2016)

    Google Scholar 

  67. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: SemEval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 1–14 (2017)

    Google Scholar 

  68. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R.: A SICK cure for the evaluation of compositional distributional semantic models. In: International Conference on Language Resources and Evaluation (LREC), pp. 216–223 (2014)

    Google Scholar 

  69. Reimers, N., Beyer, P., Gurevych, I.: Task-oriented intrinsic evaluation of semantic textual similarity. In: International Conference on Computational Linguistics (COLING), pp. 87–96

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Concordia Institute of Information Systems Engineering (CIISE), Montreal, Canada

    Bahar Karimi & Arash Mohammadi

  2. Electrical and Computer Engineering, Concordia University, Montreal, Canada

    Soheil Zabihi

  3. NOVATEK International, Quebec, H4R -2E9, Canada

    Amir Keynia & Aram Montazami

Authors
  1. Bahar Karimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soheil Zabihi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amir Keynia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Aram Montazami
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Arash Mohammadi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arash Mohammadi .

Editor information

Editors and Affiliations

  1. University of Calgary, Calgary, AB, Canada

    Marina Gavrilova

  2. Sardina Systems OÜ, Tallinn, Estonia

    C. J. Kenneth Tan

  3. McGill University, Montreal, QC, Canada

    Mark Coates

  4. University of Calgary, Calgary, AB, Canada

    Yaoping Hu

  5. University of Calgary, Calgary, AB, Canada

    Henry Leung

  6. Concordia University, Montreal, QC, Canada

    Arash Mohammadi

  7. University of Toronto, Toronto, ON, Canada

    Konstantinos N. Plataniotis

  8. University of Calgary, Calgary, AB, Canada

    Helder Rodrigues de Oliveira

Ethics declarations

Conflicts of Interests/Competing Interests

The authors declare no competing interests.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer-Verlag GmbH, DE, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Karimi, B., Zabihi, S., Keynia, A., Montazami, A., Mohammadi, A. (2023). AVCA: Autonomous Virtual Cognitive Assessment. In: Gavrilova, M., et al. Transactions on Computational Science XL. Lecture Notes in Computer Science(), vol 13850. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-67868-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-67868-8_3

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-67867-1

  • Online ISBN: 978-3-662-67868-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation