Skip to main content
SpringerLink
Log in

A convolutional oculomotor representation to model parkinsonian fixational patterns from magnified videos

  • Original Article
  • Published:
Pattern Analysis and Applications Aims and scope Submit manuscript

Abstract

Oculomotor alterations are a promising biomarker to detect and characterize Parkinson’s disease (PD), even in prodromal stages. Nowadays, however, only global and simplified gaze trajectories are used to approximate the complex interactions between neuromotor commands and ocular muscles. Besides, the acquisition of such signals often requires sophisticated calibration and invasive settings. This work presents a novel imaging biomarker for PD assessment that models ocular fixational movements, recorded with conventional cameras. Firstly, a video acceleration magnification is performed to enhance small relevant fixation patterns on standard gaze video recordings. Hence, from each video are extracted a set of spatio-temporal slices, which thereafter are represented as convolutional feature maps, recovered as the first-layer responses of pre-trained CNN architectures. The feature maps are then efficiently encoded by means of covariance matrices to train a support vector machine and perform the disease classification. From a set of 130 recordings of 13 PD patients and 13 age-matched controls, the proposed approach achieved an average accuracy of 95.4% and an AUC of 0.984, following a leave-one-patient-out cross-validation scheme. The proposed imaging-based descriptor properly captures known disease tremor patterns, since PD classification performance is outstanding when augmented motion frequencies were fixed within tremor-related ranges. These results suggest a successful PD characterization from fixational eye motion patterns using ordinary videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. A preliminary version of this work appeared in [38]. In this extended version: (1) it was included a considerable extension of the experimental evaluation, (2) it was performed an exhaustive analysis and description of the proposed strategy, and (3) it was considered new CNN architectures for eye-fixation representation.

  2. Preliminar work [38] presented on CIARP 2019 was evaluated with a total of 6 PD patients and 6 control subjects.

  3. The ImageNet classification challenge, with a total training set of around 1.2 million samples.

References

  1. Abdulhay E, Arunkumar N, Narasimhan K, Vellaiappan E, Venkatraman V (2018) Gait and tremor investigation using machine learning techniques for the diagnosis of Parkinson disease. Future Gener Comput Syst 83:366–373

    Article  Google Scholar 

  2. Adhikari S, Stark DE (2017) Video-based eye tracking for neuropsychiatric assessment. Ann N Y Acad Sci 1387(1):145–152

    Article  Google Scholar 

  3. Ajay J, Song C, Wang A, Langan J, Li Z, Xu W (2018) A pervasive and sensor-free deep learning system for parkinsonian gait analysis. In: 2018 IEEE EMBS international conference on biomedical & health informatics (BHI). IEEE, pp 108–111

  4. Archibald NK, Hutton SB, Clarke MP, Mosimann UP, Burn DJ (2013) Visual exploration in Parkinson’s disease and Parkinson’s disease dementia. Brain 136(3):739–750

    Article  Google Scholar 

  5. Belić M, Bobić V, Badža M, Šolaja N, Đurić-Jovičić M, Kostić VS (2019) Artificial intelligence for assisting diagnostics and assessment of Parkinson’s disease—a review. Clin Neurol Neurosurg 184:105442

  6. Bhatia KP, Bain P, Bajaj N et al (2018) Consensus statement on the classification of tremors, from the task force on tremor of the international Parkinson and movement disorder society. Mov Disord 33(1):75–87

    Article  Google Scholar 

  7. Caramia C, Torricelli D, Schmid M, Muñoz-Gonzalez A, Gonzalez-Vargas J, Grandas F, Pons JL (2018) IMU-based classification of Parkinson’s disease from gait: a sensitivity analysis on sensor location and feature selection. IEEE J Biomed Health Inform 22(6):1765–1774

    Article  Google Scholar 

  8. Carson TB, Sutton SZ (2018) Application for smart phone or related devices for use in assessment of vestibulo-ocular reflex. US Patent App. 15/569,472

  9. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:27:1–27:27

    Article  Google Scholar 

  10. Chollet F (2017) Xception: deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1251–1258

  11. Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: International conference on machine learning, pp 647–655

  12. Ekker MS, Janssen S, Seppi K et al (2017) Ocular and visual disorders in Parkinson’s disease: common but frequently overlooked. Parkinsonism Relat Disord 40:1–10

    Article  Google Scholar 

  13. Ewenczyk C, Mesmoudi S, Gallea C, Welter ML, Gaymard B, Demain A, Cherif LY, Degos B, Benali H, Pouget P et al (2017) Antisaccades in Parkinson disease: a new marker of postural control? Neurology 88(9):853–861

    Article  Google Scholar 

  14. Fleet DJ, Jepson AD (1990) Computation of component image velocity from local phase information. Int J Comput Vis 5(1):77–104

    Article  Google Scholar 

  15. Fukushima K, Ito N, Barnes GR, Onishi S, Kobayashi N, Takei H, Olley PM, Chiba S, Inoue K, Warabi T (2015) Impaired smooth-pursuit in Parkinson’s disease: normal cue-information memory, but dysfunction of extra-retinal mechanisms for pursuit preparation and execution. Physiol Rep 3(3):e12361

    Article  Google Scholar 

  16. Gatys L, Ecker AS, Bethge M (2015) Texture synthesis using convolutional neural networks. In: Advances in neural information processing systems, pp 262–270

  17. Gitchel GT, Wetzel PA, Baron MS (2012) Pervasive ocular tremor in patients with Parkinson disease. Arch Neurol 69(8):1011–1017

    Article  Google Scholar 

  18. Gitchel GT, Wetzel PA, Qutubuddin A, Baron MS (2014) Experimental support that ocular tremor in Parkinson’s disease does not originate from head movement. Parkinsonism Relat Disord 20(7):743–747

    Article  Google Scholar 

  19. Goetz CG, Poewe W, Rascol O et al (2004) Movement disorder society task force report on the Hoehn and Yahr staging scale: status and recommendations the movement disorder society task force on rating scales for Parkinson’s disease. Mov Disord 19(9):1020–1028

    Article  Google Scholar 

  20. Goldberg ME, Walker MF (2013) The control of gaze. In: Kandel ER, Schwartz JH, Jessell TM, Siegelbaum SA, Hudspeth AJ (eds) Principles of neural science, 5th edn. McGraw-Hill, New York, pp 894–916

    Google Scholar 

  21. Gorges M, Müller HP, Lulé D et al (2016) The association between alterations of eye movement control and cerebral intrinsic functional connectivity in Parkinson’s disease. Brain Imaging Behav 10(1):79–91

    Article  Google Scholar 

  22. Hanuška J, Bonnet C, Rusz J et al (2015) Fast vergence eye movements are disrupted in Parkinson’s disease: a video-oculography study. Parkinsonism Relat Disord 21(7):797–799

    Article  Google Scholar 

  23. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  24. Hu K, Wang Z, Martens KE, Lewis S (2018) Vision-based freezing of gait detection with anatomic patch based representation. In: Asian conference on computer vision. Springer, Berlin, pp 564–576

  25. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  26. Iranzo A, Santamaria J, Tolosa E (2016) Idiopathic rapid eye movement sleep behaviour disorder: diagnosis, management, and the need for neuroprotective interventions. Lancet Neurol 15(4):405–419

    Article  Google Scholar 

  27. Jankovic J (2008) Parkinson’s disease: clinical features and diagnosis. J Neurol Neurosurg Psychiatry 79(4):368–376

    Article  Google Scholar 

  28. Kaski D, Saifee TA, Buckwell D, Bronstein AM (2013) Ocular tremor in Parkinson’s disease is due to head oscillation. Mov Disord 28(4):534–537

    Article  Google Scholar 

  29. Khosla A, Kim D (2015) Optical imaging devices: new technologies and applications. CRC Press, Boca Raton

    Google Scholar 

  30. Kubis A, Szymański A, Przybyszewski AW (2015) Fuzzy rough sets theory applied to parameters of eye movements can help to predict effects of different treatments in Parkinson’s patients. In: International conference on pattern recognition and machine intelligence. Springer, Berlin, pp 325–334

  31. Lai HY, Saavedra-Peña G, Sodini C, Heldt T, Sze V (2018) Enabling saccade latency measurements with consumer-grade cameras. In: 2018 25th IEEE international conference on image processing (ICIP). IEEE, pp 3169–3173

  32. Lal V, Truong D (2019) Eye movement abnormalities in movement disorders. Clin Parkinsonism Relat Disord 1:54–63

    Article  Google Scholar 

  33. Larrazabal A, Cena CG, Martínez C (2019) Video-oculography eye tracking towards clinical applications: a review. Comput Biol Med 108:57–66

    Article  Google Scholar 

  34. Poewe W, Seppi K, Tanner CM, Halliday GM, Brundin P, Volkmann J, Schrag AE, Lang AE (2017) Parkinson disease. Nat Rev Dis Primers 3:17013

    Article  Google Scholar 

  35. Portilla J, Simoncelli EP (2000) A parametric texture model based on joint statistics of complex wavelet coefficients. Int J Comput Vis 40(1):49–70

    Article  Google Scholar 

  36. Rizzo G, Copetti M, Arcuti S et al (2016) Accuracy of clinical diagnosis of Parkinson disease a systematic review and meta-analysis. Neurology 86(6):566–576

    Article  Google Scholar 

  37. Salat D, Noyce AJ, Schrag A, Tolosa E (2016) Challenges of modifying disease progression in prediagnostic Parkinson’s disease. Lancet Neurol 15(6):637–648

    Article  Google Scholar 

  38. Salazar I, Pertuz S, Contreras W, Martínez F (2019) Parkinsonian ocular fixation patterns from magnified videos and CNN features. In: 24th Iberoamerican congress on pattern recognition, vol 11896. Springer, Berlin, pp 740–750

  39. Sharif Razavian A, Azizpour H, Sullivan J, Carlsson S (2014) CNN features off-the-shelf: an astounding baseline for recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 806–813

  40. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556

  41. Sledzianowski A, Szymanski A, Drabik A, Szlufik S, Koziorowski DM, Przybyszewski AW (2019) Measurements of antisaccades parameters can improve the prediction of Parkinson’s disease progression. In: Asian conference on intelligent information and database systems. Springer, Berlin, pp 602–614

  42. Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Thirty-first AAAI conference on artificial intelligence

  43. Szymański A, Szlufik S, Koziorowski DM, Habela P, Przybyszewski AW (2016) Building intelligent classifiers for doctor-independent Parkinson’s disease treatments. In: Conference of information technologies in biomedicine. Springer, Berlin, pp 267–276

  44. Turcano P, Chen JJ, Bureau BL, Savica R (2018) Early ophthalmologic features of Parkinson’s disease: a review of preceding clinical and diagnostic markers. J Neurol 1–9

  45. Venuto CS, Potter NB, Ray Dorsey E, Kieburtz K (2016) A review of disease progression models of Parkinson’s disease and applications in clinical trials. Mov Disord 31(7):947–956

    Article  Google Scholar 

  46. Zhang Y, Pintea SL, Van Gemert JC (2017) Video acceleration magnification. In: Computer vision and pattern recognition

  47. Zhang Y, Yan A, Liu B, Wan Y, Zhao Y, Liu Y, Tan J, Song L, Gu Y, Liu Z (2018) Oculomotor performances are associated with motor and non-motor symptoms in Parkinson’s disease. Front Neurol 9:960

    Article  Google Scholar 

Download references

Acknowledgements

The authors express their gratitude to the Parkinson foundation FAMPAS (Fundación del Adulto Mayor y Parkinson Santander) and the local elderly institution Asilo San Rafael for making possible the recording of the dataset proposed in this work. Also, many thanks to the Vicerrectoría de Investigación y Extensión of the Universidad Industrial de Santander for supporting this research work by the project “Reconocimiento continuo de expresiones cortas del lenguaje de señas registrado en secuencias de video”, with SIVIE code 2430.

Author information

Authors and Affiliations

  1. Biomedical Imaging, Vision and Learning Laboratory (BIVL2ab), Universidad Industrial de Santander, Bucaramanga, Colombia

    Isail Salazar & Fabio Martínez

  2. Grupo de Investigación en Conectividad y Procesamiento de Señales (CPS), Universidad Industrial de Santander, Bucaramanga, Colombia

    Said Pertuz

  3. Departamento de Neurocirugía, Clínica FOSCAL, Bucaramanga, Colombia

    William Contreras

Authors
  1. Isail Salazar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Said Pertuz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William Contreras
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fabio Martínez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabio Martínez.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salazar, I., Pertuz, S., Contreras, W. et al. A convolutional oculomotor representation to model parkinsonian fixational patterns from magnified videos. Pattern Anal Applic 24, 445–457 (2021). https://doi.org/10.1007/s10044-020-00922-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10044-020-00922-4

Keywords

Search

Navigation