Abstract
The van der Heijde modification of the Sharp (SvdH) score is a widely used radiographic scoring method to quantify damage in Rheumatoid Arthritis (RA) in clinical trials. However, its complexity with a necessity to score each individual joint, and the expertise required limit its application in clinical practice, especially in disease progression measurement. In this work, we addressed this limitation by developing a bespoke, automated pipeline that is capable of predicting the SvdH score and RA severity from hand radiographs without the need to localise the joints first. Using hand radiographs from RA and suspected RA patients, we first investigated the performance of the state-of-the-art architectures in predicting the total SvdH score for hands and wrists and its corresponding severity class. Secondly, we leveraged publicly available data sets to perform transfer learning with different finetuning schemes and ensemble learning, which resulted in substantial improvement in model performance being on par with an experienced human reader. The best model for RA scoring achieved a Pearson’s correlation coefficient (PCC) of 0.925 and root mean squared error (RMSE) of 18.02, while the best model for RA severity classification achieved an accuracy of 0.358 and PCC of 0.859. Our score prediction model attained almost comparable accuracy with experienced radiologists (PCC = 0.97, RMSE = 18.75). Finally, using Grad-CAM, we showed that our models could focus on the anatomical structures in hands and wrists which clinicians deemed as relevant to RA progression in the majority of cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bird, A., et al.: Artificial intelligence and the future of radiographic scoring in rheumatoid arthritis: a viewpoint. Arthritis Res. Ther. 24, 1–10 (2022). https://doi.org/10.1186/S13075-022-02972-X
BMJ Best Practice: Rheumatoid arthritis - symptoms, diagnosis and treatment, April 2024. https://bestpractice.bmj.com/topics/en-gb/105
Boini, S., Guillemin, F.: Radiographic scoring methods as outcome measures in rheumatoid arthritis: properties and advantages. Ann. Rheumatic Diseases 60, 817–827 (2001). https://ard.bmj.com/content/60/9/817
Gilvaz, V.J., Reginato, A.M.: Artificial intelligence in rheumatoid arthritis: potential applications and future implications. Fronti. Med. 10 (2023). https://doi.org/10.3389/FMED.2023.1280312
Halabi, S.S., et al.: The RSNA pediatric bone age machine learning challenge. Radiology 290, 498–503 (2019). https://doi.org/10.1148/RADIOL.2018180736
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, December 2016, pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90
Hemalatha, R.J., Vijaybaskar, V., Thamizhvani, T.R.: Automatic localization of anatomical regions in medical ultrasound images of rheumatoid arthritis using deep learning. Proc. Inst. Mech. Eng. 233, 657–667 (2019). https://doi.org/10.1177/0954411919845747
Hirano, T., et al.: Development and validation of a deep-learning model for scoring of radiographic finger joint destruction in rheumatoid arthritis. Rheumatol. Adv. Pract. 3 (2019). https://doi.org/10.1093/RAP/RKZ047
Ho, S., Elamvazuthi, I., Lu, C.K.: Classification of rheumatoid arthritis using machine learning algorithms. In: 2018 IEEE 4th International Symposium in Robotics and Manufacturing Automation. ROMA 2018 (2018). https://doi.org/10.1109/ROMA46407.2018.8986700
Honda, S., Yano, K., Tanaka, E., Ikari, K., Harigai, M.: Development of a scoring model for the Sharp/van der Heijde score using convolutional neural networks and its clinical application. Rheumatology 62, 2272–2283 (2023). https://doi.org/10.1093/RHEUMATOLOGY/KEAC586
Imtiaz, M., Shah, S.A.A., ur Rehman, Z.: A review of arthritis diagnosis techniques in artificial intelligence era: current trends and research challenges. Neurosci. Inform. 2, 100079 (2022).https://doi.org/10.1016/J.NEURI.2022.100079
Landewé, R.B., et al.: Is radiographic progression in modern rheumatoid arthritis trials still a robust outcome? Experience from tofacitinib clinical trials. Arthritis Res. Ther. 18 (2016). https://doi.org/10.1186/s13075-016-1106-y
Maziarz, K., Krason, A., Wojna, Z.: Deep learning for rheumatoid arthritis: joint detection and damage scoring in x-rays (2022). https://doi.org/10.48550/arXiv.2104.13915
Mei, X., et al.: RadImageNet: an open radiologic deep learning research dataset for effective transfer learning. Radiol. Artif. Intell. 4 (2022). https://doi.org/10.1148/RYAI.210315
Muller, D., Soto-Rey, I., Kramer, F.: An analysis on ensemble learning optimized medical image classification with deep convolutional neural networks. IEEE Access 10 (2022). https://doi.org/10.1109/ACCESS.2022.3182399
NICE: Prevalence and incidence of rheumatoid arthritis, April 2024. https://cks.nice.org.uk/topics/rheumatoid-arthritis/background-information/prevalence-incidence/
Parashar, A., Rishi, R., Parashar, A., Rida, I.: Medical imaging in rheumatoid arthritis: a review on deep learning approach. Open Life Sci. 18 (2023). https://doi.org/10.1515/BIOL-2022-0611
Radu, A.F., Bungau, S.G.: Management of rheumatoid arthritis: an overview. Cells 10, 2857 (2021). https://doi.org/10.3390/CELLS10112857
Salaffi, F., Carotti, M., Beci, G., Carlo, M.D., Giovagnoni, A.: Radiographic scoring methods in rheumatoid arthritis and psoriatic arthritis. Radiologia Medica 124, 1071–1086 (2019). https://doi.org/10.1007/S11547-019-01001-3
Salehi, A.W., et al.: A study of CNN and transfer learning in medical imaging: advantages, challenges, future scope. Sustainability 15, 5930 (2023). https://doi.org/10.3390/SU15075930
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018). https://doi.org/10.1109/CVPR.2018.00474
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-CAM: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision , October 2017, pp. 618–626 (2017). https://doi.org/10.1109/ICCV.2017.74
Sun, D., et al.: A crowdsourcing approach to develop machine learning models to quantify radiographic joint damage in rheumatoid arthritis. JAMA Netw. Open 5(8), e2227423–e2227423 (2022). https://doi.org/10.1001/jamanetworkopen.2022.27423
Tan, Y.M., Quek, R., Chong, H., Hargreaves, C.A.: Rheumatoid arthritis: automated scoring of radiographic joint damage (2021). https://doi.org/10.48550/arXiv.2110.08812
van der Heijde, D., Boonen, A., Boers, M., Kostense, P., van der Linden, S.: Reading radiographs in chronological order, in pairs or as single films has important implications for the discriminative power of rheumatoid arthritis clinical trials. Rheumatology (Oxford, England) 38, 1213–1220 (1999). https://doi.org/10.1093/RHEUMATOLOGY/38.12.1213
van der Heijde, D., Dankert, T., Nieman, F., Rau, R., Boers, M.: Reliability and sensitivity to change of a simplification of the Sharp/van der Heijde radiological assessment in rheumatoid arthritis. Rheumatology 38, 941–947 (1999). https://doi.org/10.1093/RHEUMATOLOGY/38.10.941
Wang, H.J., et al.: Deep learning-based computer-aided diagnosis of rheumatoid arthritis with hand x-ray images conforming to modified total sharp/van der Heijde score. Biomedicines 10, pp. 1355 (2022). https://doi.org/10.3390/BIOMEDICINES10061355
Wang, Z., Liu, J., Gu, Z., Li, C.: An efficient CNN for hand x-ray overall scoring of rheumatoid arthritis. Complexity 2022 (2022). https://doi.org/10.1155/2022/5485606
Welsing, P.M., et al.: The relationship between disease activity and radiologic progression in patients with rheumatoid arthritis: a longitudinal analysis. Arthritis Rheumat. 50, 2082–2093 (2004). https://doi.org/10.1002/ART.20350
Acknowledgements
Zhiyan Bo is supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). We would like to thank the Oxford Biomedical Research Computing Facility for providing the computational resources.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bo, Z., Coates, L.C., Papież, B.W. (2024). Deep Learning Models to Automate the Scoring of Hand Radiographs for Rheumatoid Arthritis. In: Yap, M.H., Kendrick, C., Behera, A., Cootes, T., Zwiggelaar, R. (eds) Medical Image Understanding and Analysis. MIUA 2024. Lecture Notes in Computer Science, vol 14860. Springer, Cham. https://doi.org/10.1007/978-3-031-66958-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-66958-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-66957-6
Online ISBN: 978-3-031-66958-3
eBook Packages: Computer ScienceComputer Science (R0)