Assistance tools for the evaluation of machine learning algorithm performance: the decision tree-based tools developed by the Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c effort

Karen Drukker; Berkman Sahiner; Tingting Hu; Grace Hyun Kim; Heather M. Whitney; Natalie Baughan; Kyle J. Myers; Maryellen L. Giger; Michael McNitt-Gray

doi:10.1117/12.2654250

3 April 2023 Assistance tools for the evaluation of machine learning algorithm performance: the decision tree-based tools developed by the Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c effort

Karen Drukker, Berkman Sahiner, Tingting Hu, Grace Hyun Kim, Heather M. Whitney, Natalie Baughan, Kyle J. Myers, Maryellen L. Giger, Michael McNitt-Gray

Author Affiliations +

Proceedings Volume 12467, Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment; 124670E (2023) https://doi.org/10.1117/12.2654250
Event: SPIE Medical Imaging, 2023, San Diego, California, United States

Abstract

MIDRC was created to facilitate machine learning research for tasks including early detection, diagnosis, prognosis, and assessment of treatment response related to the COVID-19 pandemic and beyond. The purpose of the Technology Development Project (TDP) 3c is to create resources to assist researchers in evaluating the performance of their machine learning algorithms. An interactive decision tree has been developed, organized by the type of task that the machine learning algorithm is being trained to perform. The user can select information such as: (a) the type of task, (b) the nature of the reference standard, and (c) the type of the algorithm output. Based on the user responses, they can obtain recommendations regarding appropriate performance evaluation approaches and metrics, including literature references, short video tutorials, and links to available software. Five tasks have been identified for the decision tree: (a) classification, (b) detection/localization, (c) segmentation, (d) time-to-event analysis, and (e) estimation. As an example, the classification branch of the decision tree includes binary and multi-class classification tasks and provides suggestions for methods and metrics as well as software recommendations, and literature references for situations where the algorithm produces either binary or non-binary (e.g., continuous) output and for reference standards with negligible or non-negligible variability and unreliability. The decision tree has been made publicly available on the MIDRC website to assist researchers in conducting task-specific performance evaluations, including classification, detection/localization, segmentation, estimation, and time-to-event tasks.

Conference Presentation

Citation Download Citation

Karen Drukker, Berkman Sahiner, Tingting Hu, Grace Hyun Kim, Heather M. Whitney, Natalie Baughan, Kyle J. Myers, Maryellen L. Giger, and Michael McNitt-Gray "Assistance tools for the evaluation of machine learning algorithm performance: the decision tree-based tools developed by the Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c effort", Proc. SPIE 12467, Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment, 124670E (3 April 2023); https://doi.org/10.1117/12.2654250

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available