Skip to main content
SpringerLink
Log in

Deformation modeling and classification using deep convolutional neural networks for computerized analysis of neuropsychological drawings

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Drawing-based tests are cost-effective, noninvasive screening methods, popularly employed by psychologists for the early detection and diagnosis of various neuropsychological disorders. Computerized analysis of such drawings is a complex task due to the high degree of deformations present in the responses and reliance on extensive clinical manifestations for their inferences. Traditional rule-based approaches employed in visual analysis-based systems prove insufficient to model all possible clinical deformations. Meanwhile, procedural analysis-based techniques may contradict with the standard test conduction and evaluation protocols. Leveraging on the increasing popularity of convolutional neural networks (CNNs), we propose an effective technique for modeling and classifying dysfunction indicating deformations in drawings without modifying clinical standards. Contrary to conventional sketch recognition applications where CNNs are trained to diminish intra-shape class variations, we employ deformation-specific augmentation to enhance the presence of specific deviations that are defined by clinical practitioners. The performance of our proposed technique is evaluated using Lacks’ scoring of the Bender-Gestalt test, as a case study. The results of our experimentation substantiate that our proposed approach can represent domain knowledge sufficiently without extensive heuristics and can effectively identify drawing-based biomarkers for various neuropsychological disorders.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Mainland BJ, Shulman KI (2013) Clock drawing test. In: Larner AJ (ed) Cognitive screening instruments. Springer, New York, pp 79–109

    Google Scholar 

  2. Shin M-S, Park S-Y, Park S-R, Seol S-H, Kwon JS (2006) Clinical and empirical applications of the Rey–Osterrieth complex figure test. Nat Protoc 1(2):892–899

    Google Scholar 

  3. Bender L (1938) A visual motor gestalt test and its clinical use. Research monographs. American Orthopsychiatric Association, Chicago

    Google Scholar 

  4. Coates DR, Wagemans J, Sayim B (2017) Diagnosing the periphery: using the Rey–Osterrieth complex figure drawing test to characterize peripheral visual function. i-Perception 8(3):1–20

    Google Scholar 

  5. Price CC, Cunningham H, Coronado N, Freedland A, Cosentino S, Penney DL, Penisi A, Bowers D, Okun MS, Libon DJ (2011) Clock drawing in the montreal cognitive assessment: recommendations for dementia assessment. Dement Geriatr Cogn Disord 31(3):179–187

    Google Scholar 

  6. Moetesum M, Siddiqi I, Masroor U, Djeddi C (2015) Automated scoring of bender gestalt test using image analysis techniques. In: 13th international conference on document analysis and recognition (ICDAR). IEEE, pp 666–670

  7. Lipsman N, Schwartz ML, Huang Y, Lee L, Sankar T, Chapman M, Hynynen K, Lozano AM (2013) MR-guided focused ultrasound thalamotomy for essential tremor: a proof-of-concept study. Lancet Neurol 12(5):462–468

    Google Scholar 

  8. Drotár P, Mekyska J, Rektorová I, Masarová L, Smékal Z, Faundez-Zanuy M (2015) Decision support framework for Parkinson’s disease based on novel handwriting markers. IEEE Trans Neural Syst Rehabil Eng 23(3):508–516

    Google Scholar 

  9. Moetesum M, Siddiqi I, Vincent N, Cloppet F (2018) Assessing visual attributes of handwriting for prediction of neurological disorders—a case study on Parkinson’s disease. Pattern Recognit Lett 121:19–27

    Google Scholar 

  10. Hinton GE, Osindero S, Teh Y-W (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    MathSciNet  MATH  Google Scholar 

  11. LeCun Y, Boser B, Denker J, Henderson D, Howard R, Hubbard W, Jackel L (1989) Handwritten digit recognition with a back-propagation network. In: Neural information processing systems (NIPS)

  12. Yann Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Google Scholar 

  13. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  14. Battiato S, Farinella GM, Giudice O, Puglisi G (2016) Aligning shapes for symbol classification and retrieval. Multimed Tools Appl 75:5513–5531

    Google Scholar 

  15. Plamondon R, Srihari SN (2000) Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans Pattern Anal Mach Intell 22(1):63–84

    Google Scholar 

  16. Khalid PI, Yunus J, Adnan R (2010) Extraction of dynamic features from hand drawn data for the identification of children with handwriting difficulty. Res Dev Disabil 31(1):256–262

    Google Scholar 

  17. Rosenblum S, Samuel M, Zlotnik S, Erikh I, Schlesinger I (2013) Handwriting as an objective tool for Parkinson’s disease diagnosis. J Neurol 260(9):2357–2361

    Google Scholar 

  18. Likforman-Sulem L, Esposito A, Faundez-Zanuy M, Clémençon S, Cordasco G (2017) EMOTHAW: a novel database for emotional state recognition from handwriting and drawing. IEEE Trans Hum Mach Syst 47(2):273–284

    Google Scholar 

  19. Naseer A, Rani M, Naz S, Razzak MI, Imran M, Xu G (2019) Refining Parkinson’s neurological disorder identification through deep transfer learning. Neural Comput Appl. https://doi.org/10.1007/s00521-019-04069-0

    Article  Google Scholar 

  20. Fairhurst MC, Linnell T, Glenat S, Guest R, Heutte L, Paquet T (2008) Developing a generic approach to online automated analysis of writing and drawing tests in clinical patient profiling. Behav Res Methods 40(1):290–303

    Google Scholar 

  21. Smith SL, Hiller DL (1996) Image analysis of neuropsychological test responses. In: Medical imaging 1996: image processing. International Society for Optics and Photonics, vol 2710, pp 904–916

  22. Canham R, Smith S, Tyrrell A (2000) Automated scoring of a neuropsychological test: the Rey–Osterrieth complex figure. In: Proceedings of the 26th Euromicro conference. IEEE, vol 2, pp 406–413

  23. Canham R, Smith S, Tyrrell A (2005) Location of structural sections from within a highly distorted complex line drawing. IEE Proc Vis Image Signal Process 152(6):741–749

    Google Scholar 

  24. Bennasar M, Setchi R, Hicks Y, Bayer A (2014) Cascade classification for diagnosing dementia. In: IEEE international conference on systems, man and cybernetics (SMC). IEEE, pp 2535–2540

  25. Harbi Z, Hicks Y, Setchi R (2016) Clock drawing test digit recognition using static and dynamic features. Procedia Comput Sci 96:1221–1230

    Google Scholar 

  26. Harbi Z, Hicks Y, Setchi R (2017) Clock drawing test interpretation system. Procedia Comput Sci 112:1641–1650

    Google Scholar 

  27. Pullman SL (1998) Spiral analysis: a new technique for measuring tremor with a digitizing tablet. Mov Disord 13(S3):85–89

    Google Scholar 

  28. Pereira CR, Pereira DR, da Silva FA, Hook C, Weber SA, Pereira LA, Papa JP (2015) A step towards the automated diagnosis of parkinson’s disease: analyzing handwriting movements. In: IEEE 28th international symposium on computer-based medical systems (CBMS). IEEE, pp 171–176

  29. Lacks P (1999) Bender Gestalt screening for brain dysfunction. Wiley, New York

    Google Scholar 

  30. Glenat S, Heutte L, Paquet T, Guest R, Fairhurst M, Linnell T (2008) The development of a computer-assisted tool for the assessment of neuropsychological drawing tasks. Int J Inf Technol Decis Mak 7(04):751–767

    Google Scholar 

  31. Song Y, Davis R, Ma K, Penney DL (2016) Balancing appearance and context in sketch interpretation. In: Proceedings of the twenty-fifth international joint conference on artificial intelligence. AAAI Press, pp 2625–2632

  32. Smith SL, Cervantes BR (1998) Dynamic feature analysis of vector-based images for neuropsychological testing. In: Medical imaging 1998: physiology and function from multidimensional images. International Society for Optics and Photonics, vol 3337, pp 304–314

  33. Garbi A, Smith SL, Heseltine D, Thomson P (1999) Automated and enhanced assessment of unilateral visual neglect. In: IET conference proceedings, pp 382–386(4)

  34. Kornmeier J, Bach M (2005) The Necker cube—an ambiguous figure disambiguated in early visual processing. Vis Res 45(8):955–960

    Google Scholar 

  35. Smith SL, Lones MA (2009) Implicit context representation Cartesian genetic programming for the assessment of visuo-spatial ability. In: IEEE congress on evolutionary computation, pp 1072–1078

  36. Heinik J, Werner P, Dekel T, Gurevitz I, Rosenblum S (2010) Computerized kinematic analysis of the clock drawing task in elderly people with mild major depressive disorder: an exploratory study. Int Psychogeriatr 22(3):479–488

    Google Scholar 

  37. Werner P, Rosenblum S, Bar-On G, Heinik J, Korczyn A (2006) Handwriting process variables discriminating mild Alzheimer’s disease and mild cognitive impairment. J Gerontol Ser B Psychol Sci Soc Sci 61(4):P228–P236

    Google Scholar 

  38. Hazell A, Smith SL (2008) Towards an objective assessment of alzheimer’s disease: the application of a novel evolutionary algorithm in the analysis of figure copying tasks. In: Proceedings of the 10th annual conference companion on Genetic and evolutionary computation. ACM, pp 2073–2080

  39. Garre-Olmo J, Faúndez-Zanuy M, López-de Ipiña K, Calvó-Perxas L, Turró-Garriga O (2017) Kinematic and pressure features of handwriting and drawing: preliminary results between patients with mild cognitive impairment, Alzheimer disease and healthy controls. Curr Alzheimer Res 14(9):960–968

    Google Scholar 

  40. El Yacoubi M, Garcia-Salicetti S, Kahindo C, Rigaud A-S, Cristancho-Lacroix V (2018) From aging to early-stage Alzheimer’s: Uncovering handwriting multimodal behaviors by semi-supervised learning and sequential representation learning. Pattern Recognit 86:08

    Google Scholar 

  41. Plamondon R, O’Reilly C, Ouellet-Plamondon C (2014) Strokes against stroke—strokes for strides. Pattern Recognit 47(3):929–944

    Google Scholar 

  42. O’Reilly C, Plamondon R, Lebrun L-H (2014) Linking brain stroke risk factors to human movement features for the development of preventive tools. Front Aging Neurosci 6:150

    Google Scholar 

  43. Drotár P, Mekyska J, Rektorová I, Masarová L, Smékal Z, Faundez-Zanuy M (2013) A new modality for quantitative evaluation of parkinson’s disease: in-air movement. In: 13th IEEE international conference on bioinformatics and bioengineering (BIBE). IEEE, pp 1–4

  44. Drotár P, Mekyska J, Rektorová I, Masarová L, Smékal Z, Faundez-Zanuy M (2014) Analysis of in-air movement in handwriting: a novel marker for Parkinson’s disease. Comput Methods Programs Biomed 117(3):405–411

    Google Scholar 

  45. Drotár P, Mekyska J, Rektorová I, Masarová L, Smékal Z, Faundez-Zanuy M (2016) Evaluation of handwriting kinematics and pressure for differential diagnosis of Parkinson’s disease. Artif Intell Med 67:39–46

    Google Scholar 

  46. Pereira CR, Weber SA, Hook C, Rosa GH, Papa JP (2016) Deep learning-aided Parkinson. In: 29th SIBGRAPI conference on graphics, patterns and images (SIBGRAPI). IEEE, pp 340–346

  47. Taleb C, Khachab M, Mokbel C, Likforman-Sulem L (2017) Feature selection for an improved Parkinson’s disease identification based on handwriting. In: 1st international workshop on Arabic script analysis and recognition (ASAR). IEEE, pp 52–56

  48. Mucha J, Zvoncak V, Galaz Z, Faundez-Zanuy M, Mekyska J, Kiska T, Smekal Z, Brabenec L, Rektorova I, Lopez-de Ipina K (2018) Fractional derivatives of online handwriting: a new approach of parkinsonic dysgraphia analysis. In: 41st international conference on telecommunications and signal processing (TSP). IEEE, pp 1–4

  49. Rémi C, Frélicot C, Courtellemont P (2002) Automatic analysis of the structuring of children’s drawings and writing. Pattern Recognit 35(5):1059–1069

    MATH  Google Scholar 

  50. Chindaro S, Guest R, Fairhurst M, Potter J (2004) Assessing visuo-spatial neglect through feature selection from shape drawing performance and sequence analysis. Int J Pattern Recognit Artif Intell 18(07):1253–1266

    Google Scholar 

  51. Renau-Ferrer N, Rémi C (2010) A method for visuo-spatial classification of freehand shapes freely sketched. In: 14th international conference on image processing, computer vision, & pattern recognition, p 216

  52. Beuvens F, Vanderdonckt J (may 2012) Usigesture: An environment for integrating pen-based interaction in user interface development. In: 6th International conference on research challenges in information science (RCIS), pp 1–12

  53. De Pandis MF, Galli M, Vimercati S, Cimolin V, De Angelis MV, Albertini G (2010) A new approach for the quantitative evaluation of the clock drawing test: preliminary results on subjects with Parkinson’s disease. Neurol Res Int 2010:283890

    Google Scholar 

  54. Galli M, Vimercati SL, Stella G, Caiazzo G, Norveti F, Onnis F, Rigoldi C, Albertini G (2011) A new approach for the quantitative evaluation of drawings in children with learning disabilities. Res Dev Disabil 32(3):1004–1010

    Google Scholar 

  55. Vimercati S, Galli M, De Pandis M, Ancillao A, Stella G, Albertini G (2012) Quantitative evaluation of graphic gesture in subjects with Parkinson’s disease and in children with learning disabilities. Gait Posture 35:S23–S24

    Google Scholar 

  56. Tabatabaey-Mashadi N, Sudirman R, Guest RM, Khalid PI (2013) An evaluation of children’s structural drawing strategies. J Teknol [Sci Eng] 61(2):27–32

    Google Scholar 

  57. Tabatabaey-Mashadi N, Sudirman R, Guest RM, Khalid PI (2015) Analyses of pupils’ polygonal shape drawing strategy with respect to handwriting performance. Pattern Anal Appl 18(3):571–586

    MathSciNet  Google Scholar 

  58. Khalid PI, Yunus J, Adnan R, Harun M, Sudirman R, Mahmood NH (2010) The use of graphic rules in grade one to help identify children at risk of handwriting difficulties. Res Dev Disabil 31(6):1685–1693

    Google Scholar 

  59. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 807–814

  60. Szarvas M, Yoshizawa A, Yamamoto M, Ogata J (2005) Pedestrian detection with convolutional neural networks. In: Proceedings of the IEEE intelligent vehicles symposium. IEEE, pp 224–229

  61. Mori K, Matsugu M, Suzuki T (2005) Face recognition using SVM fed with intermediate output of CNN for face detection. In: MVA, pp 410–413

  62. Lauer F, Suen CY, Bloch G (2007) A trainable feature extractor for handwritten digit recognition. Pattern Recognit 40(6):1816–1824

    MATH  Google Scholar 

  63. Sheikhi S (2007) Clinical use of bender-gestalt test in brain lesions diagnosis and its comparison with magnetic resonance imaging (MRI). J Urmia Nurs Midwifery Fac 5(1):15–21

    Google Scholar 

  64. Santos AA, Jorge LM (2007) Bender test with dyslexics: comparison of two systems of punctuation. Psico-USF 12(1):13–21

    Google Scholar 

  65. Allen RA, Decker SL (2008) Utility of the bender visual-motor gestalt test second edition in the assessment of attention-deficit/hyperactivity disorder 1. Percept Motor Skills 107(3):663–675

    Google Scholar 

  66. Ferreira RB, Feil CF, Nunes MLT (2009) Bender visual-motor gestalt test in the children’s clinical assessment. Psico-USF 14(2):185–192

    Google Scholar 

  67. Keppeke L, Cintra I, Schoen TH (2013) Bender visual-motor gestalt test in adolescents: relationship between visual-motor development and the tanner stages 1. Percept Motor Skills 117(1):257–275

    Google Scholar 

  68. Koppitz EM (1964) The bender gestalt test for young children. Grune & Stratton, New York

    Google Scholar 

  69. Cavanaugh JJ (1973) Preventing reading failure: prediction, diagnosis, intervention. Pediatrics 52(1):157–158

    Google Scholar 

  70. Hutt ML, Monheit S (1985) Effectiveness of the hutt adaptation of the bender-gestalt test configuration scale in differentiating emotionally disturbed adolescents. Psychol Rep 56(2):439–443

    Google Scholar 

  71. Brannigan GG, Decker SL, Madsen DH (2004) Innovative features of the Bender–Gestalt Ii and expanded guidelines for the use of the global scoring system. Bender visual-motor Gestalt test, second edition assessment service bulletin, 1

  72. Moetesum M, Siddiqi I, Masroor U, Vincent N, Cloppet F (2016) Segmentation and classification of offline hand drawn images for the BGT neuropsychological screening test. In: 8th international conference on digital image processing (ICDIP 2016). International Society for Optics and Photonics

  73. Nazar H, Moetesum M, Ehsan S, Siddiqi I, Khurshid K, Vincent N, McDonald-Maier KD (2017) Classification of graphomotor impressions using convolutional neural networks—an application to automated neuropsychological screening tests. In: 14th international conference on document analysis and recognition (ICDAR)

  74. Niu X-X, Suen CY (2012) A novel hybrid CNN-SVM classifier for recognizing handwritten digits. Pattern Recognit 45(4):1318–1325

    Google Scholar 

  75. Shorten C, Khoshgoftaar TM (2019) A survey on image data augmentation for deep learning. J Big Data 6(1):60

    Google Scholar 

  76. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357

    MATH  Google Scholar 

  77. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 248–255

  78. Miles TR (1997) The Bangor dyslexia test. Learning Development Aids, Wisbech

    Google Scholar 

  79. Wilkinson GS, Robertson GJ (2006) Wide range achievement test (WRAT4). Psychological Assessment Resources, Lutz, FL

    Google Scholar 

  80. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  81. Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(<\)0.5 mb model size. arXiv preprint arXiv:1602.07360

  82. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1–9

  83. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 2818–2826

  84. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  85. Iandola F, Moskewicz M, Karayev S, Girshick R, Darrell T, Keutzer K (2014) DenseNet: implementing efficient ConvNet descriptor pyramids. arXiv preprint arXiv:1404.1869

  86. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp 3320–3328

  87. Agrawal P, Carreira J, Malik J (2015) Learning to see by moving. In: Proceedings of the IEEE international conference on computer vision, pp 37–45

Download references

Author information

Authors and Affiliations

  1. Bahria University, Islamabad, Pakistan

    Momina Moetesum & Imran Siddiqi

  2. University of Essex, Essex, UK

    Shoaib Ehsan

  3. Paris Descartes University, Paris, France

    Nicole Vincent

Authors
  1. Momina Moetesum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Imran Siddiqi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shoaib Ehsan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicole Vincent
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Imran Siddiqi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Moetesum, M., Siddiqi, I., Ehsan, S. et al. Deformation modeling and classification using deep convolutional neural networks for computerized analysis of neuropsychological drawings. Neural Comput & Applic 32, 12909–12933 (2020). https://doi.org/10.1007/s00521-020-04735-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-04735-8

Keywords

Search

Navigation