Yifei Chen
ABSTRACT
Parkinson's Disease (PD) is a commonly diagnosed movement disorder with more than 10 million patients worldwide. Its clinical evaluation relies on a rating system called MDS-UPDRS, which includes subjective and error-prone motor examinations. This paper proposes an objective and interpretable visual system (PD-Net ) to quantitatively evaluate motor function of PD patients using video footage. The PD-Net consists of three modules: 1) a pose detector to infer 21 hand keypoints directly from RGB videos, 2) a movement analysis module to study temporal patterns of hand keypoints and discover motor symptoms, and 3) a scoring module to predict MDS-UPDRS ratings with retrieved symptoms. Trained with an in-house clinical dataset, PD-Net can effectively handle the unique challenges of PD examination videos, such as clinically-defined gestures, distinct self-occlusion/foreshortening effect and contextual background. And it detects hand keypoints of PD patients with an average accuracy of 84.1%, a 32.9% improvement over OpenPose. When compared to the ratings of experienced clinicians, PD-Net achieves an overall MDS-UPDRS rating score accuracy of 87.6% and Cohen's kappa of 0.82 on a testing dataset of 509 examination videos at a level exceeding human raters. This study demonstrates a clinically applicable automated video analysis system for PD clinical evaluation, which can facilitate early detection, routine monitoring, and treatment assessment.
Supplemental Material
- Jerry Ajay, Chen Song, Aosen Wang, Jeanne Langan, Zhinan Li, and Wenyao Xu. 2018. A pervasive and sensor-free deep learning system for Parkinsonian gait analysis. In 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI). IEEE, 108--111.Google Scholar
Cross Ref
- Paulina JM Bank, Johan Marinus, Carel GM Meskers, Jurriaan H de Groot, and Jacobus J van Hilten. 2017. Optical Hand Tracking: A Novel Technique for the Assessment of Bradykinesia in Parkinson's Disease. Movement Disorders, Vol. 4, 6 (2017), 875--883.Google ScholarCross Ref
- AH Butt, E Rovini, C Dolciotti, G De Petris, P Bongioanni, MC Carboncini, and F Cavallo. 2018. Objective and automatic classification of Parkinson disease with Leap Motion controller. Biomedical Engineering Online, Vol. 17, 1 (2018), 168.Google ScholarCross Ref
- Yujun Cai, Liuhao Ge, Jianfei Cai, and Junsong Yuan. 2018. Weakly-supervised 3D hand pose estimation from monocular RGB images. In Proceedings of the European Conference on Computer Vision (ECCV). 666--682.Google ScholarDigital Library
- Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A Sheikh. 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).Google Scholar
- Chun-Min Chang, Yu-Lin Huang, Jui-Cheng Chen, and Chi-Chun Lee. 2019. Improving Automatic Tremor and Movement Motor Disorder Severity Assessment for Parkinson's Disease with Deep Joint Training. In 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 3408--3411.Google ScholarCross Ref
- Yifei Chen, Haoyu Ma, Deying Kong, Xiangyi Yan, Jianbao Wu, Wei Fan, and Xiaohui Xie. 2020. Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation. In The IEEE Winter Conference on Applications of Computer Vision (WACV) .Google ScholarCross Ref
- Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017. RMPE: Regional Multi-Person Pose Estimation. In The IEEE International Conference on Computer Vision (ICCV) .Google Scholar
- Claudia Ferraris, Roberto Nerino, Antonio Chimienti, Giuseppe Pettiti, Nicola Cau, Veronica Cimolin, Corrado Azzaro, Giovanni Albani, Lorenzo Priano, and Alessandro Mauro. 2018. A Self-Managed System for Automated Assessment of UPDRS Upper Limb Tasks in Parkinson's Disease. Sensors, Vol. 18, 10 (2018), 3523.Google ScholarCross Ref
- Brook Galna, Gillian Barry, Dan Jackson, Dadirayi Mhiripiri, Patrick Olivier, and Lynn Rochester. 2014. Accuracy of the Microsoft Kinect sensor for measuring movement in people with Parkinson's disease. Gait & Posture, Vol. 39, 4 (2014), 1062--1068.Google ScholarCross Ref
- Christopher G Goetz, Barbara C Tilley, Stephanie R Shaftman, Glenn T Stebbins, Stanley Fahn, Pablo Martinez-Martin, Werner Poewe, Cristina Sampaio, Matthew B Stern, Richard Dodel, et al. 2008. Movement Disorder Society-sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS-UPDRS): scale presentation and clinimetric testing results. Movement Disorders, Vol. 23, 15 (2008), 2129--2170.Google ScholarCross Ref
- Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The elements of statistical learning: data mining, inference, and prediction .Springer Science & Business Media.Google Scholar
- Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, et al. 2019. Panoptic Studio: A massively multiview system for social interaction capture. IEEE transactions on pattern analysis and machine intelligence, Vol. 41, 1 (2019), 190--204.Google Scholar
- Deying Kong, Yifei Chen, Haoyu Ma, Xiangyi Yan, and Xiaohui Xie. 2019. Adaptive Graphical Model Network for 2D Handpose Estimation. In British Machine Vision Conference (BMVC). BMVA Press.Google Scholar
- Deying Kong, Haoyu Ma, Yifei Chen, and Xiaohui Xie. 2020 b. Rotation-invariant Mixed Graphical Model Network for 2D Hand Pose Estimation. In The IEEE Winter Conference on Applications of Computer Vision. 1546--1555.Google ScholarCross Ref
- Deying Kong, Haoyu Ma, and Xiaohui Xie. 2020 a. Sia-gcn: A spatial information aware graph neural network with 2d convolutions for hand pose estimation. arXiv preprint arXiv:2009.12473 (2020).Google Scholar
- Michael H Li, Tiago A Mestre, Susan H Fox, and Babak Taati. 2018a. Automated assessment of levodopa-induced dyskinesia: Evaluating the responsiveness of video-based features. Parkinsonism & Related Disorders, Vol. 53 (2018), 42--45.Google ScholarCross Ref
- Michael H Li, Tiago A Mestre, Susan H Fox, and Babak Taati. 2018b. Vision-based assessment of parkinsonism and levodopa-induced dyskinesia with pose estimation. Journal of NeuroEngineering and Rehabilitation, Vol. 15, 1 (2018), 97.Google ScholarCross Ref
- Bo Lin, Wei Luo, Zhiling Luo, Bo Wang, Shuiguang Deng, Jianwei Yin, and Mengchu Zhou. 2020. Bradykinesia Recognition in Parkinson's Disease via Single RGB Video. ACM Transactions on Knowledge Discovery from Data (TKDD), Vol. 14, 2 (2020), 1--19.Google Scholar
- Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. 2017. VNect: Real-time 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics (TOG), Vol. 36, 4 (2017), 44.Google ScholarDigital Library
- Seyedeh Somayyeh Mousavi, Mohammad Firouzmand, Mostafa Charmi, Mohammad Hemmati, Maryam Moghadam, and Yadollah Ghorbani. 2019. Blood pressure estimation from appropriate and inappropriate PPG signals using A whole-based method. Biomedical Signal Processing and Control, Vol. 47 (2019), 196--206.Google ScholarCross Ref
- Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked Hourglass networks for human pose estimation. In European Conference on Computer Vision. Springer, 483--499.Google ScholarCross Ref
- Yan Pang, Jake Christenson, Feng Jiang, Tim Lei, Remy Rhoades, Drew Kern, John A Thompson, and Chao Liu. 2020. Automatic detection and quantification of hand movements toward development of an objective assessment of tremor and bradykinesia in Parkinson's disease. Journal of Neuroscience Methods (2020), 108576.Google Scholar
- Paschalis Panteleris, Iason Oikonomidis, and Antonis Argyros. 2018. Using a single RGB frame for real time 3D hand pose estimation in the wild. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 436--445.Google ScholarCross Ref
- Felix Scholkmann, Jens Boss, and Martin Wolf. 2012. An efficient algorithm for automatic peak detection in noisy periodic and quasi-periodic signals. Algorithms, Vol. 5, 4 (2012), 588--603.Google ScholarCross Ref
- Andrew Siderowf, Michael McDermott, Karl Kieburtz, Karen Blindauer, Sandra Plumb, Ira Shoulson, and Parkinson Study Group. 2002. Test--retest reliability of the unified Parkinson's disease rating scale in patients with early Parkinson's disease: results from a multicenter clinical trial. Movement disorders, Vol. 17, 4 (2002), 758--763.Google Scholar
- Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand keypoint detection in single images using multiview bootstrapping. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2.Google ScholarCross Ref
- Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep High-Resolution Representation Learning for Human Pose Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) .Google Scholar
- Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional pose machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4724--4732.Google ScholarCross Ref
- Bin Xiao, Haiping Wu, and Yichen Wei. 2018. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV). 466--481.Google ScholarDigital Library
- Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus, and Thomas Brox. 2019. FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images. In Proceedings of the IEEE International Conference on Computer Vision. 813--822.Google ScholarCross Ref
Index Terms
- PD-Net: Quantitative Motor Function Evaluation for Parkinson's Disease via Automated Hand Gesture Analysis
Recommendations
The PARK Framework for Automated Analysis of Parkinson's Disease Characteristics
There are about 900,000 people with Parkinson's disease (PD) in the United States. Even though there are benefits of early treatment, unfortunately, over 40% of individuals with PD over 65 years old do not see a neurologist. It is often very difficult ...
The design and evaluation of a wearable-based system for targeted tremor assessment in Parkinson's disease
MMSys '22: Proceedings of the 13th ACM Multimedia Systems ConferenceWearable sensors are worn by subjects to allow for continuous physiological monitoring. The use of wearable sensors for the quantification of movement within research communities has increased in recent years, with the purpose of objectively assessing ...
Quantitative evaluation of Parkinson's disease using sensor based smart glove
CBMS '11: Proceedings of the 2011 24th International Symposium on Computer-Based Medical SystemsParkinson's disease (PD) is a chronic neurodegenerative movement disorder. The motor status of patients with PD can be characterized by the Unified Parkinson's disease rating scale (UPDRS). However, the UPDRS evaluates the movement disorders on a ...
Comments