Using uncertainty quantification to improve reliability of video-based skill assessment metrics in central venous catheterization

Catherine Austin; Rebecca Hisey; Olivia O'Driscoll; Tamas Ungi; Gabor Fichtinger

doi:10.1117/12.2654419

3 April 2023 Using uncertainty quantification to improve reliability of video-based skill assessment metrics in central venous catheterization

Catherine Austin, Rebecca Hisey, Olivia O'Driscoll, Tamas Ungi, Gabor Fichtinger

Author Affiliations +

Proceedings Volume 12466, Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling; 124660C (2023) https://doi.org/10.1117/12.2654419
Event: SPIE Medical Imaging, 2023, San Diego, California, United States

Abstract

Computed-based skill assessment relies on accurate metrics to provide comprehensive feedback to trainees. Improving the accuracy of video-based metrics computed using object detection is generally done by improving the performance of the object detection network, however increasing its performance requires resources that cannot always be obtained. This study aims to improve the accuracy of metrics in central venous catheterization without requiring a high performing object detection network by removing false positive predictions identified using uncertainty quantification. The uncertainty for each bounding box was calculated using an entropy equation. The uncertainties were then compared to an uncertainty threshold computed using the optimal point of a Receiver Operating Characteristic curve. Predictions were removed if the uncertainty fell below the predefined threshold. 50 videos were recorded and annotated with ground truth bounding boxes. These bounding boxes were used to train an object detection network, which was used to produce predictive bounding boxes for the test set. This method was evaluated by computing metrics for the predictive bounding boxes with and without having false positives removed and comparing them to ground truth labels using a Pearson Correlation. The Pearson Correlations for the baseline comparisons and the comparisons made using the results calculated using false positive removal were 0.922 and 0.816 for syringe path lengths, 0.753 and 0.510 for ultrasound path lengths, 0.831 and 0.489 for ultrasound usage times, and 0.857 and 0.805 for syringe usage times. This method consistently reduced inflated metrics, making it promising for improving metric accuracy.

Conference Presentation

Citation Download Citation

Catherine Austin, Rebecca Hisey, Olivia O'Driscoll, Tamas Ungi, and Gabor Fichtinger "Using uncertainty quantification to improve reliability of video-based skill assessment metrics in central venous catheterization", Proc. SPIE 12466, Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling, 124660C (3 April 2023); https://doi.org/10.1117/12.2654419

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available